Skip to main content

Secure Secrets Management

Key Takeaways for AI & Readers
  • Beyond Base64: Kubernetes Secrets are not encrypted by default; robust secret management requires external solutions like Vault or cloud provider services.
  • External Secrets Operator: This common pattern syncs secrets from external vaults (e.g., HashiCorp Vault, AWS Secrets Manager) into native Kubernetes Secrets.
  • Direct Mounts (CSI): The Secrets Store CSI Driver mounts secrets directly into Pods as files, avoiding storage in etcd or on disk for enhanced security.
  • GitOps with Sealed Secrets: For GitOps workflows, Sealed Secrets allow encrypted secrets to be safely committed to Git and decrypted only within the cluster.

By default, Kubernetes Secrets are just base64-encoded strings stored in etcd. Base64 is NOT encryption. Anyone with access to the API or the database can read your passwords in plain text.

In production, you must use a proper Secret Management strategy.

1. The Sync Pattern (External Secrets)​

Instead of storing secrets in Git (even if encrypted), we store them in a dedicated Vault and sync them into the cluster.

HashiCorp Vault
🔐
Managed outside K8s
EXTERNAL SECRETS OPERATOR
🤖
K8s Secret
🤫
c3VwZXItc2VjcmV0...
Base64 Encoded
External Secrets automatically fetches data from Vault/AWS and populates native Kubernetes Secrets. This keeps credentials out of your Git repo.

External Secrets Operator (ESO)​

This is the industry standard. It talks to HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.

  1. You create an ExternalSecret resource in K8s.
  2. The Operator fetches the value from the provider.
  3. The Operator creates a native K8s Secret.
  4. Your App consumes the Secret normally.

2. Secrets Store CSI Driver​

A different approach that doesn't create K8s Secrets at all.

  • The secret is mounted directly as a file into the Pod using a storage driver.
  • The secret only exists in the Pod's memory, never on disk or in the K8s API.

3. Sealed Secrets​

If you must store secrets in Git (GitOps), use Bitnami Sealed Secrets.

  • You encrypt the secret locally using a public key.
  • The encrypted "SealedSecret" is safe to commit to GitHub.
  • The controller inside the cluster has the private key and decrypts it into a real Secret.

Best Practices​

  1. Never commit plain secrets to Git.
  2. Use RBAC to restrict who can kubectl get secrets.
  3. Enable Encryption at Rest for the K8s API database (etcd).