Managing Secrets Across Continuous Delivery Pipelines

The recent news about uber being hacked was due to application source code containing unencrypted username/password information (which happened to be shared & stored on Github). It’s easy to be cynical about this breach but managing secrets across applications, environments and deployment pipelines can be extremely challenging. And more so than ever with Continuous Delivery and daily/hourly code changes.

What Are Secrets?

Secrets, as the word says, are things that exist but only very few should know about. In the context of continuous delivery, they are all your credentials, keys, config files or any passwords that you need to deploy your software across environments, 3rd party APIs, production infrastructure, and tools.

What Secrets are Relevant For Continuous Delivery?

When you want to integrate and implement CD in your organization, the following secrets are typically required:

  • Continuous Integration tools & repositories – Jenkins, Bamboo, JFrog, TeamCity etc
  • Cloud Providers – AWS keys, GCP config file etc
  • Infrastructure – SSH keys, load balancer credentials etc
  • Collaboration providers – Slack, SMTP, Jira, ServiceNow etc
  • Verification providers – Appdynamics, New Relic, Splunk etc

These days, organizations have multiple applications/services that may be deployed in different environments using different tools to manage code deployments. That means there are different secrets for each service, team, and tools with different access levels. This makes it essential for any CD platform to make secrets management core to its purpose. It becomes necessary to reuse secrets across multiple applications/environments and tightly control the access.

How Do Customers Currently Store Secrets Today?

From the customers we’ve spoken with, storing secrets is typically done using one of two solutions:
1. Hashicorp Vault
Vault is an on-premises solution that lets the user store secrets. The stored data is encrypted and access to that data is given by tokens. These tokens can be created using specific privileges for specific lease times giving access to specific secrets. The tokens can be revoked at any time making the secrets inaccessible. It’s a popular way for centralizing the sensitive credentials, and every piece that uses the secrets gets them at runtime via access tokens and discards them immediately–making sure that they are never available for longer than needed.
2. Amazon KMS
Amazon KMS is a popular way of encrypting and decrypting secrets. The user creates an encryption key within Amazon IAM and configures the access for it by services, locations, machines etc. This key reference is used as a master key for encryption and decryption. The steps which happen for encryption/decryption are following:

  • During data encryption, a new encryption key is asked from KMS which returns the key and the key’s encrypted value
  • Using the plain text key, the secret is encrypted
  • The encrypted key and encrypted secret then are stored
  • For decryption, the encrypted key is sent to KMS which in turn returns the plaintext key
  • Using the plain text key the secret is then decrypted and used

What this ensures is that at no point in time the plaintext data (key or value) are stored in the system. Also, for each secret that gets stored, the encryption key is different. The capability for encrypting/decryption keys can be control through AWS console, and if there’s ever a breach the master key can be disabled–making sure that no one gets to know the secret values.

What Harness Learned From Customers

Over the past six months, we’ve listened to hundreds of customers share their pain and challenges for managing secrets throughout their CD pipelines. Below are the most common requirements these customers asked for:

  • A central place to see all the secrets text/files that are used for CD
  • Fine-grained access management for each secret
  • See where secrets are being used across deployment workflows and pipelines e.g. ssh key being used in workflow X
  • See a changelog, e.g. when was the secret added, by who, and when all it has been changed or removed
  • See when and by what the secret was used at runtime (e.g, Jenkins password being used to download artifacts or ssh key being used to deploy the artifact to a server)
  • Deactivate a secret. This helps when a secret is deprecated and is supposed to be moved before being deleted or changed
  • Migrate existing secrets to a new service, e.g. moving secrets to a different vault setup or to a different Amazon KMS
  • Deactivate the secret at KMS or Vault level in case there is a threat of compromise

Managing Secrets With Harness Continuous Delivery

At Harness, management of secrets is core to our architecture and solution. If a user doesn’t configure any secrets management, then by default we already have an amazon KMS configured for all accounts which are used to encrypt/decrypt the secrets they create. Below is a diagram representing how secrets are used with Harness. As you can see, with Harness the secrets are always accessed/decrypted at the time when they are needed and at no time are they stored unencrypted. Also, the Harness manager doesn’t have access to your key management system, and the delegate sits in the private network of your setup so with Harness you never have to make your secret managers be accessible publically. This adds an important layer of security.

Configuring Secrets Management with Harness

With Harness, you can implement secrets management in minutes. The main menu has a tab “Continuous Security” that has an option for “Secrets Management.”

From here, simply click “Add Secret Manager”

The Secret Management page has an option for “Configure Secret Manager,” which lets you create a secret manager of either Amazon KMS or Hashicorp Vault.

A user can choose not to create any manager, and if that’s the case, the Harness default KMS kicks in to encrypt/decrypt your secrets:

Note: The secret manager shows how many secrets it has encrypted so far. Harness also supports the functionality to migrate your secrets to another secret manager in case there is a threat of compromise on any secret manager.

The two main ways a user can manage secrets is by either uploading a text or a file that is supposed to be encrypted. The default secret manager for your account will be used to encrypt the secret. This encrypted text or file is now available across your account to be used. This way you can upload a secret once and can use it everywhere and manage it in one place.

As you can see in the screenshot above, each of these secrets has information on where are they being referenced, who has uploaded or changed them, and where at runtime (e.g. workflow) they have been used so far. This gives you a centralized way to manage your secret texts and secret files, and enables you to see which workflow in which environment is using your secret–thus providing you with complete visibility.
With Harness, secrets for your Continuous Delivery pipelines can be implemented and managed in minutes.

The Engineer Types Header Image

The Engineer Types: Which Are You?

Ever wonder which type of engineer you are? Are you the Night Owl, the "It Worked On My Machine," the Perfectionist, or the Devil's Advocate?

EKS-Anywhere Launch

Harness the Power of AWS EKS-Anywhere

Harness is excited to be part of the launch of EKS-Anywhere. As an intelligent software delivery platform, we can meet the expectation of successful application delivery - anywhere.

Developer Experience

How to Prioritize the Developer Experience and Improve Output

Developer Experience (DX) is the overall interactions and feelings that the developer feels when working towards a goal.

The Women of DevOps: Stefania Chaplin

The Women of DevOps: Stefania Chaplin, Solutions Architect at Secure Code Warrior