Everybody makes mistakes. That's never going to change. But thanks to technology, it's becoming easier and easier to catch those mistakes before they spiral into disasters.

Telus uses GitHub Secret Scanning to scour our repositories for any passwords, tokens, encryption keys, or other secrets that may have been accidentally committed to a repository (thus increasing the risk that they would find their way into production systems). As useful as this is, by default secret scanning only tells you after you've already committed a secret. This year, we also turned on push protection, which alerts developers when they try to commit code containing a potential secret.

In addition to catching secrets before they make it into production, we're able to learn from the cases where a developer tries to commit a secret and make changes to our processes to prevent the problem in the future. Ultimately, we don't want to rely on push protection to prevent secret sharing. We want to have processes that keep things from ever hitting push protection in the first place. We've already found places where it makes sense for us to use the GitHub Actions Secrets API to store and retrieve passwords and encryption keys so that there's little danger of accidentally including them in a publicly available script. We find it's much easier to have conversations about what went wrong when the bad thing—pushing a secret into a codebase—didn't actually happen than it is to have those conversations where the bad thing did happen. Heads are cooler, people are less defensive.

Enabling push protection for every repository across Telus was about as close to the "push button, receive value" ideal as you can get in enterprise software. But we never thought we would reach the point where we could click the "Enable All" button and apply push protection to every single repository created by every single developer at our company. But after only one day of planning, we activated push protection for everyone.

There are two main reasons it was so easy to adopt push protection as a safety net for our entire company. First, push protection is designed to be non-blocking, so we didn't have to worry that we'd accidentally stop developers from working. There's no need for additional training: GitHub walks developers through the process. Second, we built our development processes with GitHub as a single source of truth, which meant that we could be confident that push protection would check all the code we push.

What we realized is that push protection provides a fantastic model for how changes can be easily deployed and managed within large organizations, given the right tools and conditions. Hopefully, our experience will be useful for other organizations looking to activate push protection enterprise-wide or implement other changes such as "pre-flight" checklists or code quality tools.

In this Guide you will learn:

1. How automated tools can ensure the adoption of new technologies and processes in large organizations with minimal training required.

2. Why you need a “single source of truth” to implement organization-wide changes.

Overcoming the adoption problem

Adoption of new tools or processes is an age-old problem in enterprise IT. It comes down to two fundamental challenges: education and verification. You may have an elegant solution to a problem users are having across the organization, but it won't help anyone if they don't use it. First, you have to make users aware that the tool is available. Then they need to be trained on how to use it. Then, after an organization-wide education campaign, you'll want to check to see if the solution is actually being used—some tools make this easier to do than others.

One potential way to address the adoption problem is to force adoption through a technical fix. For example, you could automatically run some software, such as a virus/malware protection system, when users login to their computers. In our case, we run scans when developers try to commit code. But automation comes with its own risks and challenges.

One major issue is the potential for work stoppages or productivity blows. For example, you wouldn't want a false positive on an automatic malware scan to stop employees from using their computers entirely. Nor would we want secret scanning to stop developers from committing code with, say, a code example with a non-functioning key. If developers are blocked from doing their jobs, productivity decreases, and employees are aggravated. Blockages create toil, forcing developers to rope in additional personnel to help unblock them.

GitHub Secret Scanning avoids this scenario by making secret scanning non-blocking. If users need to commit a place-holder secret for tests, they can simply confirm that they're committing a test secret and move on. Likewise, if they run into a false positive—where what they're committing isn't a secret at all—they can report the false positive and continue their commit. There's also an "I'll fix it later" option that allows the user to move forward, but they must acknowledge that they understand the risks. Selecting this option also creates a security alert and notifies the repository's admins. Developers can keep working and we have a log of the decision they made, along with context they provide at the time. The worst-case scenario is that someone ignores the warning and pushes the secret anyway, but at least we have some accountability in that situation. The process is dead simple. GitHub prompts the user and provides clear explanations about what their options are. Even a developer who had never heard of push protection before can move forward without the need for training or complex documentation.  

This provides a clear example of how organization-wide solutions can work: Opt everyone in automatically, but give people an easy and clear way to move forward regardless.

How to turn this on

1. On GitHub.com, navigate to the main page of the organization.

2. Under your organization name, click “Settings.”

   

3. In the "Security" section of the sidebar, click “Code security and analysis.”

4. Under "Code security and analysis," find "GitHub Advanced Security."

5. Under "Secret scanning" and "Push protection," click “Enable all.”

   

Using secret scanning as a push protection from the command line

When you attempt to push a supported secret to a repository or organization with secret scanning as push protection–enabled, GitHub will block the push. You can remove the secret from your commit or follow a provided URL to allow the push.

Up to five detected secrets will be displayed at a time on the command line. If a particular secret has already been detected in the repository and an alert already exists, GitHub will not block that secret.

If you need to remove the secret from your latest commit (that is, HEAD) on the branch being pushed and any earlier commits that contain the secret, you can remove the secret from HEAD, then squash the commits between when the commit was introduced and the first version of HEAD for which the secret has been removed.

Allowing a blocked secret to be pushed

 If GitHub blocks a secret that you believe is safe to push, you can allow the secret and specify the reason why it should be allowed.

If you confirm a secret is real and you intend to fix it later, you should aim to remediate the secret as soon as possible. For example, you might revoke the secret and remove it from the repository's commit history. For more information, see this documentation on removing sensitive data from a repository.

When you allow a secret to be pushed, an alert is created in the "Security" tab. GitHub closes the alert and doesn't send a notification if you specify that the secret is a false positive or used only in tests. If you specify that the secret is real and that you will fix it later, GitHub keeps the security alert open and sends notifications to the author of the commit, as well as to repository administrators. Check out more information on managing alerts from secret scanning.

1. Visit the URL returned by GitHub when your push was blocked.

2. Choose the option that best describes why you should be able to push the secret. TIP: If the secret is only used in tests and poses no threat, choose “It's used in tests.” If the detected string is not a secret, choose “It's a false positive.” If the secret is real but you intend to fix it later, choose “I'll fix it later.”

3. Click “Allow me to push this secret.”

4. Reattempt the push on the command line within three hours. If you have not pushed within three hours, you will need to repeat this process.

Using secret scanning as a push protection from the web UI

When you use the web UI to attempt to commit a supported secret to a repository or organization with secret scanning as push protection–enabled, GitHub will block the commit. You will see a banner at the top of the page with information about the secret's location, and the secret will also be underlined in the file so you can easily find it.

You can remove the secret from the file using the web UI. Once you remove the secret, the banner at the top of the page will change and tell you that you can now commit your changes.

Finding your single source of truth

At Telus, we like to let people pick the tools they're most comfortable with. We trust our technical teams to make the right call about what the best tools are for any particular job. Perhaps paradoxically, standardizing on GitHub makes it possible for us to allow people to choose practically every other aspect of their tooling and stack—from language to IDE to cloud provider—because we still have a single source of truth about what code is being committed and deployed. One team might be writing code destined for Microsoft Azure using JetBrains IntelliJ while another is deploying code written in VS Code to AWS Lambda, but we know all that code will pass through GitHub before it goes into production. That makes GitHub the perfect place to handle secret scanning.

At your organization, or for a different solution, this single point of convergence might be something else. Maybe your engineers choose everything except the cloud provider, so the hosting platform becomes the single source of truth for your organization. The important thing is to identify a particular location from which you can get an organization-wide view of what your developers are doing so that you can add check-points and validation.

Thanks to this combination of non-blocking interventions and our investment in making GitHub a single source of truth for enterprise development at Telus, we were able to make a company-wide change with little effort and no toil. We haven't received a single complaint about push protection, and it's already prevented some serious headaches. What we've seen is that organizational change is not only possible, but simple when the conditions are right. You can keep this in mind as you architect your processes and as you purchase or build new software solutions.