Mounting a Kubernetes Secret as a single file inside a Pod

Recently I needed to mount an SSH private key used for one app to connect to another app into a running Pod, but to make sure it was done securely, we put the SSH key into a Kubernetes Secret, and then mounted the Secret into a file inside the Pod spec for a Deployment.

I wanted to document the process here because (a) I know I'm going to have to do it again and this will save me a few minutes' research, and (b) it's very slightly unintuitive (at least to me).

First I defined a secret in a namespace:

apiVersion: v1
kind: Secret
metadata:
  name: ssh-key
  namespace: acme
data:
  id_rsa: {{ secret_value_base64_encoded }}

Note the key of id_rsa for the secret data—I used this because when you mount a secret into a volume, the mount point will be a directory, and each file in that directory corresponds to a key in the Secret's data. So in this case, if I set a mount path of /var/my-app, then Kubernetes would place a file in there named id_rsa, with the value from the Secret. (Note that I'm using Ansible to template and apply manifests, so I'm actually using a value like {{ ansible_vault_encrypted_string | b64encode }}, which uses Ansible Vault to decrypt an encrypted private key in a playbook variable—though that's besides the point here).

To get that file to mount in the path /var/my-app/id_rsa, I add the volume like so in my Deployment spec:

spec:
  template:
    spec:
      containers:
      - image: "my-image:latest"
        name: my-app
        ...
        volumeMounts:
          - mountPath: "/var/my-app"
            name: ssh-key
            readOnly: true
      volumes:
        - name: ssh-key
          secret:
            secretName: ssh-key

Note that you can control the secrets files permissions using defaultMode in the volumes definition, or even individually per file (if there are multiple keys in the Secret's data), but that exercise is left up to the reader. See the Secrets documentation for more on that (specifically, the section on Secret files permissions).

Mounting a secret to a single file in an existing directory

One thing that is not supported, unfortunately, is mounting a single secret to a single file in a directory which already exists inside the container. This means secrets can't be mounted as files in the same way you'd do a file-as-volume-mount in Docker or mount a ConfigMap item into an existing directory. When you mount a secret to a directory (like /var/my-app in the above example), Kubernetes will mount the entire directory /var/my-app with only the contents of your secret / secretName items.

To overcome this issue, you can mount the secret somewhere else (e.g. /var/my-app-secrets), then use a postStart lifecycle hook to copy it into place:

        lifecycle:
          postStart:
            exec:
              command:
                - /bin/sh
                - -c
                - cp /var/my-app-secrets/id_rsa /var/my-app/id_rsa

That way, existing contents of the /var/my-app directory are be preserved.

Further reading

secrets
kubernetes
manifest
k8s
deployment

Comments

Hi, thank you for the informative tutorial.

Could you provide more information on how you are using Ansible to populate the values inside the secrets file?

This was useful, I stop by here to remind myself, because it's the simplest guide out there. Thanks!

In Linux I expect a mount to make anything in the directory be mounted onto to become unavailable. Here it looks like /var/my-app is not getting wiped out, but added to. Is that correct?

Be careful when moving secrets in containers, secret volumes are tmpfs by default, you may write them on non volatil device.
You'd better use a symlink in postStart.

You can use subPath to mount a configMap or secret key to a specific subdirectory without overwriting it's contents.

        volumeMounts:
          - mountPath: "/var/my-app/id_rsa"
              subPath: id_rsa
            name: ssh-key
            readOnly: true
      volumes:
        - name: ssh-key
          secret:
            secretName: ssh-key
            items:
              - key: id_rsa
                path: id_rsa

Thanks, this worked for me on k8s v1.19. Jeff you might want to consider confirming this/updating your article, seems it should work for you too.

I used a Secret with stringData and as long as subPath matched the 'filename' specified in the Secret it seems to work. e.g.

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
stringData:
  filename.txt: |
    mySecretGoesHere

You can use a link instead of copying the file, so you can get your file contact updated when the secret content is modified.

        lifecycle:
          postStart:
            exec:
              command:
                - /bin/sh
                - -c
                - ln -sf /var/my-app-secrets/settings.json /var/my-app/settings.json

What happens to id_rsa on mount? Does it decoded automatically? This my issue. It doesn't work for me. Is decoding manually done in separate command? k8s doc says it decodes on mount.

I don't think it will always work due to the fact that your entrypoint script and poststart are running async.

I just want to confirm. Does the data in the `data` field get decoded to plain text when the Secret get mounted to the pod by file or by environment variables? It seems like so in my testing. Is there a doc explaining this? Or is this a *Secret*?

Please don't mind the previous comment I made. The answer is yes. I think I read too much and become stupid. It was in the doc as well.

Thanks, man, I still struggle with using subpath but want to keep secret updated automatically. Let me verify your solution.

- name: config
mountPath: /path/to/config.json
subPath: config.json
readOnly: true

Using subPath you preserve the files content while mounting secret over file.

Hello Jeff,

Thank you for this information. I'm a beginner and I've a question on a related topic, could you please have a look?
I would like to have a secret available across the containers that are created in a specific namespace.
This is what I've done
1. Created a secret for a specific namespace
2. Added a imagePullSecrets to the serviceaccount of that namespace which creates the containers
3. Created a nginx container and when I described the pod, I was able to see the ImagePullSecrets with the secret name

However when I login to the container and navigate to /run/secrets/ - I don't find my secret added here.
Would it be possible for you to let me know what I'm doing wrong? Thank you.

Great tip on existing folders. One issue I had with K8s v1.24.16-eks-2d98532 is the `mountPath` could not be wrapped in quotes. Not sure why. Hope that helps someone.

Extremely helpful article for a newbee like me

I'm just learning about Kubernetes but I know a bit about SSH. My first thought when I saw the problem description was, why not just put a line like "IdentityFile /var/my-app-secrets/id_rsa" into ~/.ssh/config or /etc/ssh/ssh_config .