Being the primary technology entity for the US’ most populated state is no easy task. With over 140 different departments and 200,000 employees, the California Department of Technology (CDT) keeps state departments up to date on the latest tech—but also ensures over 39 million residents can easily access public health and emergency services information.
CDT’s Office of Enterprise Technology (OET), which is responsible for providing statewide web, geospatial, and data services, navigates government regulations to get this information online, quickly. Every moment counts in an emergency or national disaster—including how many seconds it takes for a website to load.
“We need to have access to an environment that allows us to be very fast,” explained Skreakar Peddi, Cloud Security and Infrastructure Solution Architect. “If we deliver a website in an emergency 30 days late, the website won’t be helping our community.” And for several years now, the department’s goal has been just that—modernizing their infrastructure to support on-demand software development and delivery, confirmed CDT Chief Technology Innovation Officer Scott Gregory. “In the past, we’ve had residents that were relying on tools to know whether or not they have electricity, things at the very basic levels related to health and public safety,” Gregory said. “In order to provide that, we’ve been required to develop and deploy our solutions much more rapidly.”
But digital transformation can be easier said than done. In order to support their new mandate of rapid development and deployment, the team knew they had to turn to automation—something their current on-premises architecture couldn’t support. Code and scripts were all stored in data centers, or even in folders on individual’s PCs. “For all the CI builds, checking code, and security scanning, developers built their code on their desktop and did a copy and paste to the server,” Lead DevOps Engineer/Solutions Architect Shamal Siwan explained, opening their code up to critical vulnerabilities.
CDT racked a server into a data center with connecting power, added more servers, CPUs, and memory on top of that, and spent valuable time on manual maintenance. From Peddi’s point of view, that time and lack of security was non-negotiable. “Rapid application deployment and rapid website set up is mission-critical stuff.”
Going from on-premises infrastructure to infrastructure as code would take time, so the department built on what they were already using with Microsoft Azure DevOps—and GitHub was selected. “For us, GitHub is easy because Microsoft is hosting their core code in GitHub. It’s seamless for us to pull it to our repository, work it out, and monitor it,” said Siwan. As a first step, the department hosted their code on-premises, using GitHub Enterprise Server. “A developer committed to GitHub, then we wrapped up the code in a CI build that went through an automated release to the website,” said Siwan. They were able to see the change within minutes. Now CDT is able to turn out code, applications, and websites in a fraction of the amount of time that it was traditionally done in the past.
Still, with their goal being infrastructure as code, they knew their endpoint wouldn’t be anywhere but the cloud. As the department adopted an agile framework, they switched to GitHub Enterprise Cloud to help them build even more quickly—together. “Within our department, collaboration is easy, because we all have access to Azure DevOps,” said Peddi. “But now when we need to collaborate between departments, we give access to our GitHub repositories to collaborate and get caught up in the content.”
We reduced our deployment time significantly. To deliver quickly, using GitHub and Azure DevOps for our DevSecOps process, CI/CD, infrastructure, code, and automation was the key.
With this support for collaboration and built-in security, the department was able to extend DevOps to DevSecOps. “We wanted to make it stable to increase collaboration and we wanted to document our controls and security. Security is a major part of our work,” said Siwan. By combining GitHub and Azure DevOps, security is now a shared responsibility across the development process and built directly into the developer workflow. Before a developer commits code into GitHub, their code is pre-scanned and shown to developers as a “pre-commit.” After code is committed, it triggers their CI pipeline, funneling the code through Veracode—CDT’s static scan analysis tool—and SCAs. Developers simply point Veracode to one of their GitHub repositories and can immediately assess any reported code vulnerabilities.
“The goal here is to fail fast,” Siwan explained. “You fail fast, you fix it, and you move forward, because guess what? It takes you a minute to solve that security issue in development when it could potentially take you hours or days to fix in production. Today, you only need to lift a finger to commit code.”
Learning to fail and adapt along the way ensured CDT’s digital transformation efforts didn’t stumble when it mattered most: California wildfire season. Because of high winds, the state’s largest utility provider—PG&E—shut down power to avoid potential sparks. CDT jumped into action: working across departments, their developers were able to build and deploy online resources in less than 24 hours to help residents find emergency assistance. “We reduced our deployment time significantly,” Siwan said. “To deliver quickly, using GitHub and Azure DevOps for our DevSecOps process, CI/CD, infrastructure, code, and automation was the key.”
For Deputy Chief Technology Innovation Officer Manveer Bola, the department’s level of technical delivery and stability during the blackout wasn’t just above average. It was unprecedented. “A lot of people were surprised how—unlike traditionally—the government was able to perform better than a private entity in keeping these websites up and resilient with all the demands. The government actually delivered a much more stable solution much more quickly than a private entity could.”
Gregory agreed—recognizing that “more” may include more responsibility, but also more opportunities. “It’s begun to set a precedent that this organization is able to turn out code, applications, and websites in a fraction of the amount of time that traditionally it was done in the past.” And as CDT touches just about every department in the executive branch of the California government, their digital transformation can become a blueprint for other state organizations. “We’ve been able to build partnerships not only with industry but also internally with government partners to convince them that this is the future. We’re very proud of that.”