Skip to main content

Secure Secrets Management

Key Takeaways for AI & Readers
  • Beyond Base64: Kubernetes Secrets are merely base64-encoded (not encrypted) by default and are stored in plaintext in etcd. Anyone with API access or etcd access can read them. Robust secret management requires external solutions.
  • External Secrets Operator (ESO): The industry-standard pattern for syncing secrets from external vaults (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) into native Kubernetes Secrets. ESO handles rotation, templating, and multi-provider setups.
  • Secrets Store CSI Driver: Mounts secrets directly into Pods as files via a CSI volume, avoiding storage in etcd entirely. The secret exists only in the Pod's tmpfs filesystem for enhanced security.
  • Sealed Secrets for GitOps: Bitnami Sealed Secrets allow asymmetrically encrypted secrets to be safely committed to Git repositories. Only the in-cluster controller can decrypt them, enabling true GitOps for secret management.
  • Encryption at Rest: Kubernetes supports configuring encryption providers (aescbc, secretbox, KMS) for etcd data. This is a critical baseline control that should be enabled in every production cluster.
  • Secret Rotation: Secrets should be rotated regularly. Both ESO and the CSI Driver support automatic refresh intervals to pick up rotated values from the external provider without pod restarts.

By default, Kubernetes Secrets are just base64-encoded strings stored in etcd. Base64 is not encryption — it is a reversible encoding. Anyone with kubectl get secret -o yaml access or direct etcd access can read your database passwords, API keys, and TLS certificates in plain text. In a compromised cluster, native Secrets offer essentially zero protection.

In production, you must use a proper secret management strategy that addresses three concerns: where secrets are sourced (an external vault), how they are delivered (synced or mounted), and how they are protected at rest (encryption).

1. The Problem with Native Secrets​

Before diving into solutions, understand exactly what native Kubernetes Secrets provide and what they do not:

# This "Secret" is NOT secure — the value is just base64
apiVersion: v1
kind: Secret
metadata:
  name: database-creds
  namespace: production
type: Opaque
data:
  # echo -n "superSecretPassword" | base64
  password: c3VwZXJTZWNyZXRQYXNzd29yZA==
# Anyone with read access can decode it instantly
echo "c3VwZXJTZWNyZXRQYXNzd29yZA==" | base64 -d
# Output: superSecretPassword

The specific risks:

  • Secrets are stored unencrypted in etcd by default.
  • Secrets are transmitted in plaintext over the API (TLS protects the transport, but the value itself is not encrypted).
  • Anyone with RBAC permission to get secrets in a namespace can read all secrets in that namespace.
  • Secrets stored in Git manifests (even base64-encoded) are visible to anyone with repository access.
  • There is no built-in audit trail for secret access beyond API audit logging.
  • There is no built-in rotation mechanism.

2. The Sync Pattern: External Secrets Operator (ESO)​

The External Secrets Operator is the most widely adopted solution. It creates a bridge between external secret management systems and native Kubernetes Secrets.

HashiCorp Vault
🔐
Managed outside K8s
EXTERNAL SECRETS OPERATOR
🤖
K8s Secret
🤫
c3VwZXItc2VjcmV0...
Base64 Encoded
External Secrets automatically fetches data from Vault/AWS and populates native Kubernetes Secrets. This keeps credentials out of your Git repo.

Architecture​

ESO introduces two primary custom resources:

  • SecretStore / ClusterSecretStore: Defines how to connect to the external provider (endpoint, authentication method, region).
  • ExternalSecret: Declares which secrets to fetch from the provider and how to map them into a Kubernetes Secret.

The operator runs a reconciliation loop: it periodically fetches values from the external provider and creates or updates the target Kubernetes Secret. If the value changes in the provider, ESO updates the Secret automatically.

SecretStore Configuration​

# ClusterSecretStore — available to all namespaces
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
            # Uses IRSA (IAM Roles for Service Accounts) on EKS

ExternalSecret Resource​

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-creds
  namespace: production
spec:
  refreshInterval: 1h              # re-fetch from provider every hour
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-creds           # name of the K8s Secret to create
    creationPolicy: Owner          # ESO owns the Secret lifecycle
    template:                      # optional: transform the secret data
      type: Opaque
      data:
        # Use Go templating to construct a connection string
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.example.com:5432/mydb"
  data:
    - secretKey: username
      remoteRef:
        key: production/database    # path in AWS Secrets Manager
        property: username          # JSON key within the secret
    - secretKey: password
      remoteRef:
        key: production/database
        property: password

HashiCorp Vault Provider​

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"                 # KV v2 engine
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "production-app"
          serviceAccountRef:
            name: vault-auth-sa

GCP Secret Manager Provider​

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: my-gcp-project-123
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production
          clusterProjectID: my-gcp-project-123
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

3. Secrets Store CSI Driver​

The Secrets Store CSI Driver takes a fundamentally different approach: instead of syncing secrets into Kubernetes Secret objects, it mounts secrets directly into pods as files via a CSI volume. The secret never touches etcd.

# SecretProviderClass defines which secrets to mount
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: production
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "production-app"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/production/database"
        secretKey: "password"
  # Optional: also sync to a K8s Secret for env vars
  secretObjects:
    - secretName: database-creds-synced
      type: Opaque
      data:
        - objectName: db-password
          key: password
---
# Pod that mounts the secret as a file
apiVersion: v1
kind: Pod
metadata:
  name: store-api
  namespace: production
spec:
  serviceAccountName: store-api-sa
  containers:
    - name: store-api
      image: registry.example.com/store-api:v2.0.0
      volumeMounts:
        - name: secrets
          mountPath: "/mnt/secrets"
          readOnly: true
      # The secret is available at /mnt/secrets/db-password
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: vault-db-creds

When to use CSI Driver vs ESO: Use the CSI Driver when you want secrets to never exist as Kubernetes Secret objects (higher security posture). Use ESO when you need secrets available as environment variables or when multiple pods across namespaces need the same secret.

4. Sealed Secrets for GitOps​

If you follow a GitOps workflow where everything in the cluster is declared in Git, you face a challenge: you cannot commit plaintext secrets to a repository. Bitnami Sealed Secrets solves this with asymmetric encryption.

How It Works​

  1. The Sealed Secrets controller generates an RSA key pair inside the cluster.
  2. You use the kubeseal CLI to encrypt a Secret using the controller's public key.
  3. The resulting SealedSecret resource is safe to commit to Git.
  4. The controller decrypts the SealedSecret and creates a native Kubernetes Secret.
# Create a regular secret YAML (do NOT apply it)
kubectl create secret generic api-key \
  --from-literal=key=sk_live_abc123xyz \
  --dry-run=client -o yaml > secret.yaml

# Encrypt it using the controller's public key
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# The sealed-secret.yaml is safe to commit to Git
# This is safe to store in Git — only the in-cluster controller can decrypt it
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: api-key
  namespace: production
spec:
  encryptedData:
    key: AgB2s3... # RSA-encrypted value — cannot be decoded without the private key
  template:
    metadata:
      name: api-key
      namespace: production
    type: Opaque

Sealed Secrets Scopes​

Sealed Secrets supports three encryption scopes:

  • strict (default): The SealedSecret is bound to a specific name and namespace. It cannot be moved.
  • namespace-wide: The SealedSecret can be renamed within the same namespace.
  • cluster-wide: The SealedSecret can be decrypted in any namespace.

Always use strict scope unless you have a specific reason not to.

5. SOPS for Encrypting Files​

Mozilla SOPS (Secrets OPerationS) is a tool for encrypting YAML, JSON, and INI files. Unlike Sealed Secrets, SOPS encrypts individual values within a file while leaving keys and structure visible, making diffs readable.

# Encrypted with SOPS — keys are visible, values are encrypted
apiVersion: v1
kind: Secret
metadata:
  name: database-creds
type: Opaque
data:
  username: ENC[AES256_GCM,data:abc123...,type:str]
  password: ENC[AES256_GCM,data:def456...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:us-east-1:123456789:key/abc-def-123
  encrypted_regex: "^(data|stringData)$"

SOPS integrates with AWS KMS, GCP KMS, Azure Key Vault, and age/PGP for encryption. It pairs well with Flux CD, which has native SOPS decryption support.

6. Encryption at Rest​

Even with external secret management, you should enable encryption at rest for etcd. This protects against direct etcd access (e.g., from a compromised node or etcd backup).

# /etc/kubernetes/encryption-config.yaml on the API server
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # KMS provider (recommended for production)
      - kms:
          apiVersion: v2
          name: aws-encryption
          endpoint: unix:///run/kmsplugin/socket.sock
      # Fallback: aescbc with a local key (less secure than KMS)
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      # identity means "no encryption" — used for reading old unencrypted data
      - identity: {}

On managed Kubernetes (EKS, GKE, AKS), etcd encryption is typically configured at the cluster level:

  • EKS: Enable envelope encryption with a customer-managed KMS key.
  • GKE: Etcd is encrypted by default. You can add application-layer encryption with Cloud KMS.
  • AKS: Enable encryption at rest with customer-managed keys via Azure Key Vault.

7. Secret Rotation Strategies​

Secrets must be rotated regularly. The challenge in Kubernetes is that pods typically read secrets at startup and cache them. Rotation strategies include:

  1. ESO refreshInterval: ESO re-fetches the secret periodically. The Kubernetes Secret is updated, but running pods must be restarted to pick up the new value.
  2. CSI Driver auto-rotation: The CSI Driver can periodically re-mount updated secret files. Applications that read secrets from files on each request (rather than caching at startup) will pick up changes without restart.
  3. Reloader: Use a tool like Stakater Reloader that watches for Secret changes and triggers rolling restarts of Deployments that reference them.
  4. Application-level refresh: Design your application to periodically re-read secrets from the filesystem or re-fetch from the Kubernetes API.
# Stakater Reloader annotation — auto-restart on secret change
apiVersion: apps/v1
kind: Deployment
metadata:
  name: store-api
  annotations:
    reloader.stakater.com/auto: "true"    # restart on any referenced secret change
spec:
  # ...

Common Pitfalls​

  1. Committing plaintext secrets to Git: Even if you delete the commit, secrets remain in Git history. Use git-secrets or pre-commit hooks to prevent this.
  2. Overly broad RBAC for secrets: Granting get on secrets at the cluster level lets any service account read every secret. Scope RBAC to specific namespaces and specific secret names when possible.
  3. Forgetting to rotate the Sealed Secrets key: The Sealed Secrets controller rotates its key by default every 30 days, but old keys are retained for decryption. Back up the key (it is the only way to recover SealedSecrets) and plan for key rotation.
  4. Not enabling encryption at rest: Many teams assume managed Kubernetes encrypts etcd by default. On EKS, you must explicitly enable envelope encryption. Verify your cluster configuration.
  5. Using secrets in environment variables instead of files: Environment variables are visible in process listings (/proc/<pid>/environ), container inspection (kubectl exec), and crash dumps. Mounting secrets as files with restrictive permissions is more secure.
  6. Ignoring the immutable field: For secrets that should never change (e.g., TLS CA certificates), set immutable: true to prevent accidental modification and improve etcd performance.

Best Practices​

  1. Never commit plaintext secrets to Git. Use Sealed Secrets, SOPS, or an external secrets operator.
  2. Enable encryption at rest for etcd in every cluster. Use a KMS provider for production.
  3. Use RBAC to restrict who can get, list, and watch secrets. Audit secret access with Kubernetes audit logging.
  4. Prefer file mounts over environment variables for delivering secrets to applications.
  5. Implement secret rotation with automated refresh intervals and pod restart mechanisms.
  6. Use namespaced SecretStore resources instead of ClusterSecretStore to limit blast radius — a compromised namespace should not be able to access another namespace's secrets.
  7. Enable Kubernetes audit logging for secret access and integrate with your SIEM for alerting on unusual access patterns.
  8. Back up Sealed Secrets private keys — if the controller's key is lost, all SealedSecrets become undecryptable.
  9. Use short-lived credentials where possible (e.g., Vault dynamic secrets, AWS STS temporary credentials) rather than long-lived static secrets.

What's Next?​

  • Explore Pod Security to ensure pods cannot escalate privileges and access secrets they should not.
  • Learn about Policy as Code to enforce that all deployments use external secrets instead of inline secret values.
  • See RBAC and Access Control for fine-grained control over who can read secrets.
  • Understand Multi-Tenancy patterns for isolating secrets between teams sharing a cluster.