As the coronavirus pandemic spreads, LA’s IT department quickly developed a system to enable many city employees to work from home.
Before Feb. 15, most of the 48,000 government employees of Los Angeles had no official and productive infrastructure to enable them to work from home. But that has changed quickly with the swift arrival of the COVID-19 pandemic.
In just two days starting March 2, officials in the city’s information technology office brainstormed, designed, configured, and built a web-based work-from-home system from the ground up, and then tested it using its own staffers from their homes to be sure it worked as designed.
On March 13, the first 12 city IT workers were using and trying out the nascent platform, called Connect2LACity, and running it through its paces remotely, eating their own dog food, and learning their own lessons about its operation, just like other city workers would soon be doing. After those tests, it was released to other employees and within a week, was successfully being used by 2,159 city employees to work from home.
The system has continued to grow, with 11,600 users by April 1 and 18,170 total users by April 9. Some 25,000 workers or more could eventually be using it as needed.
The whirlwind effort began around Feb. 26, when the CDC issued a report that said it was no longer a question of the pandemic causing shut-downs across the nation. Ted Ross, the CIO for the city’s information technology office, said, “When I saw it, I said, ‘uh oh, we really need to be able to support a workforce that’s going to be working at home. We had no instructions, no orders. But we started discussing it as an executive team in the CIO’s office.”
SEE: Coronavirus: Critical IT policies and tools every business needs (TechRepublic Premium)
By March 2, the staff was brainstorming, looking at what they had available as resources, what would be needed, how many users would require access, and how it could all be brought together quickly. The team started with an initial target of 15,000 users.
None of it existed as the project began, Ross said.
“We had an old VPN solution that was used by less than 200 city employees,” but it was out of date. An early option was to take the old VPN system and expand it dramatically, but that was rejected because it was too inefficient and was not user friendly. The team then looked at systems where workers could be given access to their work computers from home and other options.
That seemed promising until discussions of how to provide access to software began, said Ross. One option was designating access to specific applications by specific workers, but that quickly became unwieldy and difficult to manage, and it was rejected.
“We quickly knew that a city of our size, with 42 departments, we realized it would be placing us in permissions purgatory to be trying to give and manage the right access rights for workers,” said Ross. “So we decided on giving workers access to their own office machines with all that is included, such as applications, settings, and shared drives, so they could use it all from their home computers.”
To make this work, the IT department acquired licenses for two pieces of software–identity management and secure remote access applications–from two vendors he declined to name for security reasons.
One of the most challenging questions raised by a senior deputy during the staff discussions was whether employees might have to give up some features of their in-office computer configurations, such as shared drive access, due to the work-from-home arrangements, said Ross.
“It became very important from our perspective to not let that happen, but to give the employees the same tools they had at work so we could make them just as effective working from home,” he said.
With lots of input and discussions, the Connect2LACity platform was created in two days and underwent testing on March 13. “I threw a lot of people at it,” said Ross. The package includes a web portal and security and productivity software for employees that they download at home and install on their home computers. The applications are available for desktop and laptop Windows and Mac machines, as well as for tablets and even Chromebooks. The system also includes multi-factor authentication, identity management, remote access capabilities, and more.
As the platform was being developed, the IT team worked with leaders in other city departments to identify the people who would be using the system so they could be given instructions on how it would work and how they would start using it.
“We didn’t know what we would need later, so we tried to build something out that could handle any scenario,” said Ross. Sending the original 12 IT department testers home with the platform to use was done to ensure that problems could be addressed before it was distributed to the rest of the workers.
“It had some access issues, it had some instructions issues,” said Ross. “It was all the kinds of things you want to learn before it is deployed to thousands of people.” The system is scalable so it can be expanded to 25,000 or more employees as needed.
The group that played the largest role in developing the platform was the IT department’s security team, which focused on user, network, and data center security for the project. “This was built with security in mind first and foremost,” said Ross.
See: Windows 10 security: A guide for business leaders (TechRepublic Premium)
Some 25 employees in the IT office worked on the project from brainstorming to implementation to get it completed quickly, and have been providing help desk support as well.
The implementation curve–getting all the 18,000-plus users online in just a few weeks, has been “staggering,” he said. “We had something like eight times the normal numbers of help desk tickets in the first two weeks. Normally, we have about 150 to 200 a week. We ended up at over 1,500 in the first week.”
Those help desk requests were largely pleas for help, rather than reports of problems.
The system is showing positive results so far. Before the work-from-home situation, for example, city employees were using Google Hangouts Meet for video conferencing about 100 to 200 times per week, said Ross. Using the remote platform, city workers are now using Google Hangouts Meet an average of more than 2,000 times a day.
The strategy to have workers use their own computers at home was chosen because the pandemic had already tightened the laptop market, making it almost impossible to quickly acquire, configure, and safely distribute enough new machines for workers at the spur of the moment.
“I would have loved to have provided a new laptop for every worker, but there are supply chain issues,” said Ross. “I say ‘hats off’ to city employees for being willing to use their own home computers. We took specific steps to sandbox their city work on those machines.”
The biggest problem that has arisen for some users has been connectivity problems at their homes, but the city has been successfully lobbying local internet providers to fix those issues and boost their services where needed, said Ross.
City workers who don’t have home internet access are being provided with new work smartphones as needed so they can use those devices as mobile hotspots for connectivity. Several hundred workers have been provided with portable hotspots to give them internet access.
The success of the project has been possible due to support from the mayor’s office and from the staff of the city’s IT department employees, as well as input from focus groups that provided their feedback and help, said Ross.
“Our strength is our ability to use focus groups,” he said. “We took people in my department who are not highly technical and had them review what we were doing. I think that was a great lesson for us to continue using. It’s a bit of a blur–how much had to happen in a short time to make it all work. It’s really the great work of my entire team and other teams.”
Meanwhile, just before all this happened, the city’s IT department had just completed a major datacenter project and had moved its mainframe to another location.
“It’s perfect that we have this new datacenter right now,” said Ross. “It helps us react to the pandemic situation. And we had just moved the mainframe up to the state of California offices. We had some major projects that we are glad we finished because they would be very difficult to complete now.”