According to research from GitGuardian and CyberArk, 79% of IT decision-makers reported having experienced a secrets leak, up from 75% in the previous year's report. At the same time, the number of leaked credentials has never been higher, with over 12.7 million hardcoded credentials in public GitHub repositories alone. One of the more troubling aspects of this report is that over 90% of valid secrets found and reported remained valid for more than 5 days.
According to the same research, on average, it takes organizations 27 days to remediate leaked credentials. Combine that with the fact that non-human identities outnumber human identities by at least 45:1, and it is easy to see why many organizations are realizing stopping secrets sprawl means finding a way to deal with this machine identity crisis. Unfortunately, the research also shows that many teams are confused about who owns the security of these identities. It is a perfect storm of risk.
Why Does Rotation Take So Long
So, why are we taking so long to rotate credentials if we know they are one of the easiest attack paths for adversaries? One major contributing factor is a lack of clarity on how our credentials are permissioned. Permissions are what authorize what specific things one entity, such as a Kubernetes workload or a microservice, can successfully request from another service or data source.
Let's remember what remediation of a secrets sprawl incident means: you need to safely replace a secret without breaking anything or granting new, too-wide permissions, which would potentially introduce more security risks to your company. If you already have full insight into the lifecycle of your non-human identities and their associated secrets, this is a fairly straightforward process of replacing them with new secrets with the same permissions. This can take considerable time if you don't already have that insight, as you need to hope the developer who originally created it is still there and has documented what was done.
Let's look at why permissions management is especially challenging in environments dominated by NHIs, examine the challenges developers and security teams face in balancing access control and productivity, and discuss how a shared responsibility model might help.
Who Really Owns Secrets Sprawl?
Secrets sprawl generally refers to the proliferation of access keys, passwords, and other sensitive credentials across development environments, repositories, and services like Slack or Jira. GitGuardian's latest Voice of the Practitioners report highlights that 65% of respondents place the responsibility for remediation squarely on the IT security teams. At the same time, 44% of IT leaders reported developers are not following best practices for secrets management.
Secrets sprawl and the underlying issues of over-permissioned long-lived credentials will continue to fall in this gap until we figure out how to better work together in a shared responsibility model.
The Developer's Perspective On Permissions
Developers face enormous pressure to build and deploy features quickly. However, managing permissions carefully, with security best practices, can be labor-intensive. Each project or application often has its own unique access requirements, which take time to research and properly set, almost feeling like a full-time job on top of the work making and deploying their applications. Best practices for creating and managing permissions too commonly do not get applied evenly across teams, are seldom documented appropriately, or are forgotten altogether after the developer gets the application working.
Compounding the issue, in too many cases, developers are simply granting too wide of permissions to these machine identities. One report found that only 2% of granted permissions are actually used. If we take a closer look at what they are up against, it is easy to see why.
For instance, think about managing permissions within Amazon Web Services. AWS's Identity and Access Management (IAM) policies are known for their flexibility but are also complex and confusing to navigate. IAM supports various policy types—identity-based, resource-based, and permission boundaries—all of which require precise configurations. AWS also offers multiple access paths for credentials, including IAM roles and KMS (Key Management Service) grants, which each come with its own unique access configurations. Learning this system is no small feat.
Another common example of a service where permissions can become difficult to manage is GitHub. API keys can grant permissions to repositories across various organizations, making it challenging to ensure appropriate access boundaries. A single key can unintentionally provide excessive access across environments when developers are members of multiple organizations. The pressure is on to get it right, while the clock is always ticking and the backlog keeps getting bigger.
Why Security Teams Alone Can't Fix This
It may seem logical to assign security teams responsibility for monitoring and rotating secrets; after all, this is a security concern. The reality is that these teams often lack the granular project-level knowledge needed to make changes safely. Security teams don't always have the context to understand what specific permissions are essential for keeping applications running. For instance, a seemingly minor permission change could break a CI/CD pipeline, disrupt production, or even cause a company-wide cascading failure if the wrong service disappears.
The dispersed nature of secrets management across teams and environments also increases the attack surface. With no one really in charge, it becomes much harder to maintain consistency in access controls and audit trails. This fragmentation often results in excessive or outdated credentials and their associated permissions remaining active for far too long, possibly forever. It can make it difficult to know who has legitimate or illegitimate access to which secrets at any given time.
A Shared Responsibility Model For Faster Rotation
Developers and security teams could help address these issues by meeting in the middle and building a shared responsibility model. In such a model, developers are more responsible for consistently managing their permissions through proper tooling, such as CyberArk's Conjur Secrets Manager or Vault by HashiCorp, while also better documenting the permissions and scope of the necessary permissions at the project level. Security teams should be helping developers by working to automate secrets rotation, investing in the proper observability tooling to gain clarity into the state of secrets, and working with IT to eliminate long-lived credentials altogether.
If developers clearly document which permissions are needed in their requirements, it could help security teams conduct faster and more precise audits and speed remediation. If security teams work to ensure that the easiest and fastest overall path toward implementing a new non-human identity secret is also the safest and most scalable route, then there are going to be far fewer incidents that require emergency rotation, and everyone wins.
The goal for developers should be to ensure that the security team can rotate or update credentials in their applications with confidence, on their own, knowing they're not jeopardizing production.
Key Questions to Address around Permissioning
When thinking through what needs to be documented, here are a few specific data points to help this cross-team effort flow more smoothly:
- Who Created the Credential? - Many organizations find it difficult to track credential ownership, especially when a key is shared or rotated. This knowledge is essential to understanding who is responsible for rotating or revoking credentials.
- What Resources Does It Access? - API keys can often access a range of services, from databases to third-party integrations, making it essential to limit permissions to the absolute minimum necessary.
- What Permissions Does It Grant? - Permissions vary widely depending on roles, resource-based policies, and policy conditions. For instance, in Jenkins, a user with `Overall/Read` permission can view general information, while `Overall/Administer` grants full control over the system.
- How Do We Revoke or Rotate It? - The ease of revocation varies by platform, and in many cases, teams must manually track down keys and permissions across systems, complicating remediation and prolonging exposure to threats.
- Is the Credential Active? - Knowing whether a credential is still in use is critical. When NHIs use long-lived API keys, these credentials may remain active indefinitely unless managed properly, creating persistent access risks.
Permissions Are Challenging, But We Can Manage Them Together As One Team
According to the GitGuardian report, while 75% of respondents expressed confidence in their secrets management capabilities, the reality is often much different. The average remediation time of 27 days reflects this gap between confidence and practice. It is time to rethink how we implement and communicate secrets and their permissions as an organization.
While developers work diligently to balance security and functionality, the lack of streamlined permissions processes and uncentralized or unstandardized documentation paths only amplify the risks. Security teams alone can't resolve these issues effectively due to their limited insight into project-specific needs. They need to work hand-in-hand with developers every step of the way.
GitGuardian is building the next generation of secrets security tooling, helping security and IT teams get a handle on secrets sprawl. Knowing what plaintext, long-lived credentials are exposed in your code and other environments is a needed first step to eliminating this threat. Start today with GitGuardian.
