5 ways to inject secrets from AWS into Jenkins pipelines

Jenkins provides first-class support for injecting Jenkins-defined credentials into pipelines and freestyle projects.

As you get more and more secrets to store though, this approach can become cumbersome to manage. In this article you’ll discover 5 better ways to manage secrets required by your Jenkins jobs, instead using AWS for the heavy lifting.

What’s the problem with Jenkins credentials?

Jenkins provides a helpful mechanism to manage secrets through the Manage Credentials area within Manage Jenkins. This provides a central place to add and update credentials, as well as see which jobs are using a particular credential. The traditional approach is to maintain these credentials manually. While this can work well for small setups, it has these disadvantages.

• the secret value is duplicated in several places e.g. you define a database password in CloudFormation and manually copy it into the Jenkins credential
• if the source secret changes, the Jenkins credential must be kept up to date e.g. if you change the database password you need to remember to update Jenkins
• it doesn’t use configuration-as-code best practices to configure Jenkins

In an environment such as AWS where infrastructure-as-code is frequently practiced with tools like CloudFormation and Terraform, the above approach seems outdated. Fortunately, Jenkins plugins exist allowing us to access the right services within AWS to better manage these credentials.

Secrets vs. credentials

While secret refers to any sensitive data, a credential is specifically the data that allows you to gain access to some service. In AWS secret is more commonly used, whereas in Jenkins it’s credential. Within this article these terms are used interchangeably.

Five better approaches to injecting secrets into Jenkins jobs

As expected, there’s more than one way to peel an orange 🍊, so here are 5 different approaches to getting secrets into Jenkins and your Jenkins jobs running within AWS. Which one is right for you will depend on these factors:

• whether you’re able or want to install additional Jenkins plugins
• whether you prefer to use AWS Secrets Manager or Parameter Store
• whether you’re able to grant the relevant permissions to Jenkins

To easily try these examples out yourself in your own Jenkins instance, deploy the below CloudFormation stack.

It provisions:

1. a new Virtual Private Cloud (VPC), subnets, and load balancer within your AWS account
2. a Jenkins master running in AWS Elastic Container Service (ECS)
3. a Jenkins pipeline job demonstrating each of the 5 techniques below

Note this is just a demonstration setup. To learn what’s required to make it production ready, check out Deploy your own production-ready Jenkins in AWS ECS.

You’ll need to provide these parameters to the CloudFormation stack:

1. CertificateArn which is the ARN of a Certificate Manager certificate to be used by the load balancer created by the stack, behind which Jenkins is served. The certificate must be valid for the JenkinsDomain.
2. JenkinsDomain which is the domain you’ll use to access the Jenkins instance. You can either pass the HostedZoneId for the stack to automatically register Jenkins into that Route 53 hosted zone, or once the stack has been created copy the LoadBalancerDNSName stack output which you can set in your own DNS CNAME record.

Once the stack has created, you can login to Jenkins. The default username is developer and the password is contained within an AWS Secrets Manager secret named JenkinsPasswordSecret.

Once you’re in, run the secret-super-pipeline to see the below 5 techniques in action!

All the following code snippets reference the same example CloudFormation template.

1) Secrets manager - injected via environment variable

AWS Secrets Manager has tight integration with ECS, allowing you to inject a secret value directly into a Jenkins container defined as an ECS task.

This is how that looks in a CloudFormation AWS::ECS::TaskDefinition resource, where ValueFrom refers to a secret ARN.

      ContainerDefinitions:
          Secrets:
            - Name: SECRET_FROM_ENVIRONMENT_VARIABLE_INJECTED_FROM_SECRETS_MANAGER
              ValueFrom: !Ref JenkinsPipelineSecret

Within a Jenkins pipeline definition this value can then be accessed directly through an environment variable whenever you need it.

            steps {
                echo "Secret value '$SECRET_FROM_ENVIRONMENT_VARIABLE_INJECTED_FROM_SECRETS_MANAGER'"
            }

To use this technique, the task execution role must be given the secretsmanager:GetSecretValue permission on the secret.

2) Secrets manager - injected via AWS Secrets Manager Credentials Provider plugin

The AWS Secrets Manager Credentials Provider plugin allows you within your pipeline definition to refer directly to a secret stored in Secrets Manager, using the credentials syntax. The credential name you provide is the name of the secret in AWS. In the background, the plugin makes all the necessary API calls and retrieves the secret value, assuming Jenkins has the necessary permissions.

In the pipeline definition, use this syntax.

            environment {
                SECRET_VALUE = credentials('JenkinsPipelineSecret')
            }

In this case, we’re referring to a Secrets Manager secret called JenkinsPipelineSecret. Nothing needs to be injected into the Jenkins container and no Jenkins credential needs to be created manually.

Within the Manage Credentials area of Jenkins, the value shows up automatically as a credential like this.

The credential is marked with the green icon to show it’s derived from the plugin.

To use this technique, the Jenkins role must have the secretsmanager:GetSecretValue permission to access the secret and secretsmanager:ListSecrets permission to list all secrets.

3) Secrets manager - injected via JCasC plugin + AWS Secrets Manager Credentials Provider plugin

The Jenkins Configuration as Code (JCasC) plugin allows you to define your Jenkins configuration in YAML. It has integration with the AWS Secrets Manager Credentials Provider plugin, allowing you to refer to the Secrets Manager secret directly within a credential declaration in the YAML file.

In the example below, which you can also try out with the CloudFormation template described above, a credential is being created based on the value of the Secrets Manager secret named JenkinsPipelineSecret.

credentials:
  system:
    domainCredentials:
      - credentials:
          - string:
              id: "jcasc-credential-from-secrets-manager"
              scope: GLOBAL
              secret: "${JenkinsPipelineSecret}"

With this in place you can then refer to the credential in the normal way in your pipeline definition.

            environment {
                SECRET_VALUE = credentials('jcasc-credential-from-secrets-manager')
            }

The permissions required are the same as option 2 above.

4. Parameter Store - injected environment variable

You can also store secrets in the AWS Parameter Store. This option is sometimes preferred as it can work out cheaper to store the values.

In a similar way to Secrets Manager, ECS allows you to inject Parameter Store values directly into the Jenkins container.

Here’s how the CloudFormation looks for the AWS::ECS::TaskDefinition resource, with ValueFrom taking the ARN of the Parameter Store parameter.

      ContainerDefinitions:
          Secrets:
            - Name: SECRET_FROM_ENVIRONMENT_VARIABLE_INJECTED_FROM_PARAMETER_STORE
              ValueFrom: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${JenkinsPipelineParameter}'

As expected, you can then access the value directly from within a pipeline definition.

            steps {
                echo "Secret value '$SECRET_FROM_ENVIRONMENT_VARIABLE_INJECTED_FROM_PARAMETER_STORE'"
            }

For this technique to work, the Jenkins execution role must have the ssm:GetParameters permission to retrieve the parameter. Note that this permission is different to ssm:GetParameter.

5. Parameter Store - injected via JCasC plugin + Configuration as Code AWS SSM plugin

As if you didn’t have enough Jenkins plugins already, the Configuration as Code AWS SSM plugin allows you to define a credential directly in the JCasC YAML file and refer to a value defined in a Parameter Store parameter.

In this example JCasC YAML file, we’re creating a credential whose value comes from the Parameter Store parameter named JenkinsPipelineParameter.

credentials:
  system:
    domainCredentials:
      - credentials:
          - string:
              id: "jcasc-credential-from-parameter-store"
              scope: GLOBAL
              secret: "${JenkinsPipelineParameter}"

We can then use the credential as normal within the pipeline.

            environment {
                SECRET_VALUE = credentials('jcasc-credential-from-parameter-store')
            }

To use this technique, the Jenkins role needs the ssm:GetParameter permission to access the parameter (not ssm:GetParameters).

Side-by-side comparison

To help you understand which approach is right for you, here’s a comparison.

Final thoughts

Hopefully you’ve got some ideas now of how you can improve your Jenkins credential management process within AWS. I’d be really interested to know what you think is the best option, so please leave a comment below.