Inject secrets with Vault-agent-injector

Securing sensitive information, like passwords, access tokens, and private keys, is really important in the modern software development lifecycle. If these secrets are compromised, it can lead to significant security vulnerabilities. It’s really important to have robust mechanisms for storing and managing these secrets securely. Among the various solutions available, Vault stands out as a comprehensive credential store offering a wide array of functionalities, ranging from basic credential storage to sophisticated PKI (Public Key Infrastructure) services.

What is Vault (hashicorp)?

Vault is a great way to store sensitive data securely. It can hold things like API encryption keys, passwords, and certificates. This solid platform not only makes it easy to store data securely, but also helps users manage access to their sensitive data. With Vault, users can easily create, update and delete secrets from one place, making it easier to manage important information.

Vault has a bunch of different ways for you to interact with its services, including the “vault-cli,” REST API, and user interface (UI). This means you can use it however you want, no matter what your needs are. Vault lets developers and system administrators keep sensitive information safe and sound throughout the software development and deployment process.

Vault just works with existing environments, including Kubernetes clusters, with very few dependencies. One of the first things you need to do is get a key-value database set up as a backend. We’d recommend using either Vault’s internal Raft storage or HashiCorp Consul.

You can get Vault in two versions: open source and enterprise. The platform has a bunch of different engines, like the PKI engine and the KV secrets engine, which are pretty popular.

The KV secrets engine offers a flexible key-value storage solution in two different versions. Version 1 works with a non-versioned backend, which makes the most of memory for each stored key. On the other hand, Version 2 uses a versioned backend, which stores a set number of versions of secrets (usually 10 by default). This means you can reset secrets if you need to.

Vault’s user interface (UI) and command-line interface (CLI) are pretty similar, which makes it easy to use and consistent. What’s also great is that any action you take in the UI can be replicated via the CLI, so you get a unified experience across both interfaces.

In Kubernetes, you can store sensitive data as k8s-secrets, but it’s only encoded in base64. Using the Vault-agent-injector gives us a way to avoid storing sensitive data in a k8s secret, which means we don’t have to worry about entries in the etcd database.

The Vault-agent-injector gets the job done by seamlessly retrieving sensitive data from Vault and mounting it directly into the container as a file. This means that there is no k8s secret that can be accessed by other users, which is great for security.

There are also some other benefits to this approach. When you’re storing sensitive files in a Git repository to deploy secrets-manifests, encryption is a must. This means you need to implement and manage encryption tools. This not only makes maintenance harder but also means you have to manage encryption and decryption for every creation, update or use of sensitive data.

But using Vault as a secret store with the Vault-agent-injector means you don’t have to manage encryption keys or store sensitive data in a Git repository.

You can integrate the Vault Agent injector with your application in two ways: either with an Init container on its own or with an additional sidecar container. The Init container gets things started before the main container kicks in, making it easy to mount secret(s) within the main container. At the same time, the sidecar container, which runs alongside the main container, keeps the secret(s) in sync. With the sidecar container, updating secrets within the main container is simple – any updates made in Vault are automatically updated by the sidecar container.

This article shows you how to add secrets to a container using the Vault-agent-injector.

You’ll need to have a Vault instance running first.

How does it work?

The main thing to remember about Vault and a Kubernetes cluster is that you need to activate the kubernetes-auth method within Vault. This involves setting up a serviceAccount in the cluster, which is responsible for getting secrets from Vault. You also need to bind the serviceAccount to the clusterRole “system:auth-delegator” through a clusterroleBinding. This cluster role gives the service account the right to create “token reviews” and “subject access reviews,” which are essential for verifying tokens and getting user info from the cluster. Once the service account has the right permissions, the kubernetes-auth method in Vault can start. To enable Vault to review tokens, you just need to put the jwt-token and the CA of the service account into Vault. This step is all about getting the information together and setting up the kubernetes-auth. Here’s an example of how it’s done.
Once that’s done, you can enable kubernetes-auth. This will create a kubernetes-auth-role to decide which serviceAccount in a given namespace can request secrets from Vault. While you can use wildcards, it’s better to restrict access to specific namespaces and serviceAccounts. This kubernetes-auth-role also includes a Vault policy. This defines the secret path and associated capabilities. Usually, only read permissions are needed for secret injection, but in some cases, additional permissions might be required. However, restricted access means greater security.
Once all the preparation steps are complete and the deployment with special Vault annotations is underway, the Vault-injector mutation-webhook alters the deployment by adding an init and sidecar container. The init container gets the Vault address and Vault role, then sends a request to Vault. Vault checks the JWT token of the service account, which is defined in the deployment, and responds with the secret if everything’s OK. The init container puts the secret straight into the deployment container as a file, without creating a Kubernetes secret. At the same time, the sidecar container checks the secret from time to time and moves it around in the deployment container if it detects any changes.
It’s also worth mentioning that since Kubernetes version 1.24, it’s necessary to mount serviceAccount tokens as needed. In our case, we can mount the token by setting “automountServiceAccountToken” to true in the spec section.

Install the vault-agent-injector

Initially, the installation of the Vault-agent-injector within your designated cluster for secret consumption from Vault is imperative. An approach to is through the utilization of Helm. https://github.com/hashicorp/vault-helm

injector:
  enabled: true
  
externalVaultAddr: "https://<EXTERNAL_IP_OR_FQDN>"

securityContext:
  container:
    seccompProfile:
      type: "RuntimeDefault"
    capabilities:
      drop: ["ALL"]
    privileged: false
    allowPrivilegeEscalation: false

Once you have prepared the short values file, you can continue with the installation.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm upgrade --install vault-secret-injector hashicorp/vault -f <YOUR_VALUES_FILE>

Enable kubernetes auth

In the subsequent steps, we will undertake practical actions. This includes enabling the kubernetes-auth method and establishing a kubernetes-auth-role, secret, and policy within Vault. Once our Vault environment is configured accordingly, we will proceed to create a deployment capable of consuming the secrets. Let’s dig into it.

Begin by creating a ServiceAccount within your cluster tasked with consuming secrets from Vault. This serviceAccount facilitates communication between Vault and the cluster. It is advisable to create this serviceAccount within the same namespace where your “vault-secrets-injector” is installed, for optimal organization.

vault_sa_name=<SA_NAME>
target_namespace=<NAMESPACE>

cat <<EOF> | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${vault_sa_name}
  namespace: ${target_namespace}
EOF
# since kubernetes version v1.24 it's necessary to create manually a serviceAccount token secret
cat <<EOF> | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: ${vault_sa_name}-token
  namespace: ${target_namespace}
  annotations:
    kubernetes.io/service-account.name: ${vault_sa_name}
type: kubernetes.io/service-account-token
EOF

Afterward you must give your SA the necessary permissions with a ClusterRoleBinding.

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ${vault_sa_name}-token-review-bind
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: ${vault_sa_name}
  namespace: ${target_namespace}
EOF

To continue, let’s prepare some variables and collect the necessary information. And afterward we can enable the Vault-k8s-auth.

vault_sa_name=<SA_NAME>
target_namespace=<NAMESPACE>

k8s_host=$(kubectl cluster-info | head -n 1 | awk '{ print $7 }' | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g")

sa_jwt_token=$(kubectl get -n ${target_namespace} secrets -ojson \
| jq -r --arg vault_sa_name "${vault_sa_name}" '.items[] | select(.type == "kubernetes.io/service-account-token") | . | select(.metadata.annotations."kubernetes.io/service-account.name" == $vault_sa_name ) | .data.token' \
| base64 -d)

sa_ca_cert=$(kubectl get -n ${target_namespace} secrets -ojson \
| jq -r --arg vault_sa_name "${vault_sa_name}" '.items[] | select(.type == "kubernetes.io/service-account-token") | . | select(.metadata.annotations."kubernetes.io/service-account.name" == $vault_sa_name ) | .data."ca.crt"' \
| base64 -d)

Log into your Vault instance and now enable the authentication in vault for Kubernetes.

export VAULT_ADDR=<YOUR_VAULT_ADDR>
vault login -method <YOUR_METHOD>

vault auth enable -max-lease-ttl=5 -default-lease-ttl=5 kubernetes

# output
# Success! Enabled kubernetes auth method at: kubernetes/

vault write auth/kubernetes/config \ # default is auth/kubernetes
  token_reviewer_jwt="$sa_jwt_token" \
  kubernetes_host="$k8s_host" \
  kubernetes_ca_cert="$sa_ca_cert" \
  disable_local_ca_jwt="true" \
  disable_iss_validation="false"


# output
# Success! Data written to: auth/kubernetes/config

# to verify your configuration use
vault read auth/kubernetes/config

“-max-lease-ttl” set the max ttl for the access tokens which will created with this auth

“-default-lease-ttl” set the default ttl for the access tokens which will created with this auth

I recommend setting this as short as necessary, as the default TTL may be too long. I mostly use a TTL of 5 seconds. Every time a serviceAccount requests a secret, a new token is created. In the worst case, your Vault backend cannot handle too many tokens and crashes.

You can see your Vault-k8s-auth under ; Vault-ui → “Access” → “Auth Methods” → YOUR_AUTH_METHOD

create a secret

Vault-kubernetes-auth is in place, so let’s create a secret.

# create a secret in Vault
vault secrets enable -path=secret/ -version=2 kv
vault kv put example-secret/dev/creds username="dbuser" password="mY-@w3some-pasSwoRd"

# output
# ======== Secret Path ========
# example-secret/data/dev/creds

# ======= Metadata =======
# Key                Value
# ---                -----
# created_time       2024-02-09T14:06:27.704636309Z
# custom_metadata    <nil>
# deletion_time      n/a
# destroyed          false
# version            1

vault kv get example-secret/dev/creds

# output
# ======== Secret Path ========
# example-secret/data/dev/creds

# ======= Metadata =======
# Key                Value
# ---                -----
# created_time       2024-02-09T14:06:27.704636309Z
# custom_metadata    <nil>
# deletion_time      n/a
# destroyed          false
# version            1

# ====== Data ======
# Key         Value
# ---         -----
# password    mY-@wesome-pasSwoRd
# username    dbuser

create a Vault policy

Create a restricted Vault-policy with only the necessary permission.

vault policy write dev-kv-ro - <<EOF
path "kv/data/secret/dev/*" {
   capabilities = ["read", "list"]
}
EOF

# output
# Success! Uploaded policy: dev-kv-ro

You can see your Vault-policy under ; Vault-ui → “Access” → “Policy” → YOUR_POLICY

create a kubernetes-auth-role

Next you must define a kubernetes-auth-role. With this role you bind your Vault kubernetes-auth with a serviceAccount, the Vault policy and a k8s namespace together.

# set up a Vault role for an k8s serviceAccount which can read secrets (k8s)
vault write auth/kubernetes/role/default-ns-role \
   bound_service_account_names=example-sa \
   bound_service_account_namespaces=default \
   policies=dev-kv-ro

# output
# Success! Data written to: auth/kubernetes/role/default-ns-role

You can see your Vault-role under ; Vault-ui → “Access” → “Auth Methods” → YOUR_AUTH_METHOD → “Roles”

Test Deployment

This is a simple deployment which you can use to test your secret injection.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: basic-secret
  labels:
    app: basic-secret
    namespace: default
spec:
  selector:
    matchLabels:
      app: basic-secret
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true" # enable Vault-injector
        vault.hashicorp.com/agent-json-patch: '[{"op": "replace", "path": "/securityContext/seccompProfile", "value": {"type": "RuntimeDefault"}}]' # is needed to patch the side-car container
        vault.hashicorp.com/agent-init-json-patch: '[{"op": "replace", "path": "/securityContext/seccompProfile", "value": {"type": "RuntimeDefault"}}]' # is needed to patch the init-container
        vault.hashicorp.com/role: "default-ns-role" # Vault-role to have access
        vault.hashicorp.com/tls-skip-verify: "true" # skip tls verification of your Vault instance, use only for testing true !
        vault.hashicorp.com/agent-inject-secret-example-secret: "example-secret/dev/creds" # Path to secret in Vault
        vault.hashicorp.com/agent-inject-template-example-secret: | # with this you can customize how should your secret look like
          {{- with secret "example-secret/dev/creds" -}}
          "username":"{{.Data.data.username}}"
          "password":"{{.Data.data.password}}"
          {{- end }}
      labels:
        app: basic-secret
    spec:
      serviceAccountName: example-sa # ServiceAccount which is allowed to request 
secrets
      automountServiceAccountToken: true
      containers:
      - name: secret-reader
        image: dengelhardt1/alpine-curl-non-root:v1.0.0 # this is a public image
        command:
          - /bin/sh
          - -c
        args:
          - |
            cat /vault/secrets/example-secret
            while true;do sleep 1;done
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          privileged: false
          seccompProfile:
            type: RuntimeDefault
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: example-sa
  namespace: default
---
apiVersion: v1
kind: Secret
metadata:
  name: example-sa-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: example-sa
type: kubernetes.io/service-account-token

Explanations:

vault.hashicorp.com/agent-inject: “true”

Set this annotation to true to enable the Vault-injector for this pod. Default value:false

vault.hashicorp.com/agent-json-patch: ‘[{“op”: “replace”, “path”: “/securityContext/seccompProfile”, “value”: {“type”: “RuntimeDefault”}}]’

vault.hashicorp.com/agent-init-json-patch: ‘[{“op”: “replace”, “path”: “/securityContext/seccompProfile”, “value”: {“type”: “RuntimeDefault”}}]’

This is needed to patch the Vault injector container with the additional securityContext settings.

vault.hashicorp.com/role: “<NAME_OF_ROLE>”

Enter your Vault-role, so you define what kind of access the injector use.

vault.hashicorp.com/agent-inject-secret-<SECRET>: “<PATH_TO_SECRET>/<NAME_OF_SECRET>”

Enter the path to the secret, which you’re going to mount into your container.

vault.hashicorp.com/agent-inject-template-<SECRET>: |

  {{- with secret “<PATH_TO_SECRET>/<NAME_OF_SECRET>” -}}

    {{ .Data.data.<SECRET_FIELD>}}

  {{- end }}

With this option, you can template your secret. You can define whether it should mount as json or customize it, however. In this example, it will mount only the value of your secret.

Additional important Annotations:

vault.hashicorp.com/auth-path: auth/<YOUR_AUTH_PATH>

Enter your auth-path, if you specified one. Default value:/auth/kubernetes <>

vault.hashicorp.com/tls-skip-verify: “false”

Set this to true to skip the verification of your Vault tls certificate. Default value:false

vault.hashicorp.com/tls-secret: “<K8S_SECRET_FOR_CA_CERT>”

Create a k8s-secret which contains only the custom ca-certificate from your Vault tls to verify the Vault tls certificate. This is mounted to “/vault/tls”

vault.hashicorp.com/ca-cert: “/vault/tls/<NAME_IN_K8S_SECRET>”

Enter path to custom ca-certificate <. This> depends on how you created the k8s secret within < the> custom ca-certificate.

Further annotations for Vault-injector: klick here

Deploy the test app

# apply your the test app
kubectl apply -f </PATH/TO/YAML>

# verfiy your installation
kubectl get pods

# verify the secret inside the container
kubectl logs -n <NAMEPACE> <POD_NAME>

Final considerations

The Vault-agent-injector is a powerful and secure solution for managing secrets in your Kubernetes cluster. The first configuration is not quite easy, but it has a bundle of security and flexibility. You have just one source of truth for your secrets, and didn’t store them in a git repository for your applications. And also you could decrease the number of Kubernetes secrets and with that especially the entries in the etcd database. The life-cycle of your secrets is only maintained in Vault. Changes of secrets will be picked up by Vault-agent-injectors sidecar container and rotated in your application. In the case of deleting the application, you haven’t to take care about clean up secrets inside your Kubernetes namespace, which is also a great benefit to use the injector.