Security (AWS Secrets Manager)

Introduction to Secrets Management#

In Chapter 15, we built an RDS database. That database has a username and a password. Where do we put that password?

• If we put it in the source code (GitHub), a hacker will find it.
• If we put it in a raw Kubernetes YAML file, anyone with kubectl access can read it.

We need a secure, encrypted vault to store passwords, API keys, and certificates. AWS Secrets Manager is that vault.

Level 1 — Beginner#

What is AWS Secrets Manager?#

Imagine you have a master key that opens the front door of your company.

• The Bad Way: You tape the key to the front door so employees can use it. (This is like putting a password in GitHub).
• The Good Way: You put the key in a heavy steel safe. Only the CEO knows the combination. When an employee needs to open the door, they ask the CEO to open the safe.

AWS Secrets Manager is the steel safe. We put the RDS database password in the safe. When the Kubernetes API needs to connect to the database, it secretly asks AWS for the password.

ASCII Diagram: The Secrets Flow#

[ Developer ] --(Writes code, NO PASSWORDS)--> [ GitHub ]
                                                  |
[ AWS Secrets Manager ]                           |
          |                                       |
    (Holds Password)                        (Deploys Code)
          |                                       v
          +-----> [ Kubernetes Pod (Your API) ] <---+

Level 2 — Intermediate#

Why not just use Kubernetes Secrets?#

Kubernetes has a built-in object called a Secret. Why don't we just use that?

1. It's not actually a secret! By default, a Kubernetes Secret is just Base64 encoded. If you run echo "password" | base64, it outputs cGFzc3dvcmQ=. Anyone can instantly decode that. It is NOT encrypted at rest.
2. GitOps Conflict: If we use ArgoCD, all our Kubernetes YAML files must be in GitHub. We cannot put a Base64 encoded Kubernetes Secret in GitHub.

How AWS Secrets Manager Fixes This#

AWS Secrets Manager encrypts the data using AWS KMS (Key Management Service). It uses military-grade AES-256 encryption. Even if a rogue AWS employee breaks into the physical data center and steals the hard drive containing your secrets, they cannot read them without the KMS encryption keys.

Level 3 — Advanced#

Analyzing the Integration (External Secrets Operator)#

How does a Kubernetes Pod actually get the password out of AWS? We use a tool called the External Secrets Operator (ESO).

Look at this theoretical Custom Resource inside our cluster:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: backend
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: rds-secret
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/rds/mysql
      property: password

Line-by-Line Breakdown:

• refreshInterval: "1h": ESO connects to AWS every 1 hour to check if the password changed. If it did, it updates Kubernetes automatically.
• secretStoreRef:: Points to a configuration telling ESO to talk to AWS (instead of Azure or HashiCorp Vault).
• remoteRef: key: prod/rds/mysql: This is the exact path inside AWS Secrets Manager where the vault is located.
• target: name: rds-secret: This is the magic. ESO pulls the encrypted password from AWS, decrypts it in memory, and automatically creates a native Kubernetes Secret named rds-secret. The Pod can now mount this secret securely.

Password Rotation#

If a developer leaves the company, you must change the database password. AWS Secrets Manager has a feature called Automatic Rotation. You can configure it to change the RDS password every 30 days automatically. Because ESO checks AWS every hour, Kubernetes will automatically pull down the new password without human intervention.

Level 4 — Enterprise#

Enterprise Patterns: IRSA (IAM Roles for Service Accounts)#

Wait. If the External Secrets Operator needs to connect to AWS Secrets Manager to download the password... how does it prove to AWS who it is? Does ESO have its own password? NO.

In an enterprise AWS environment, we use IRSA. We link a Kubernetes ServiceAccount directly to an AWS IAM Role. When the ESO Pod starts, AWS injects a temporary OIDC (OpenID Connect) web token into the Pod. ESO trades this token to AWS STS for temporary AWS credentials. Result: The Pod can access AWS Secrets Manager securely, and there are absolutely zero static passwords anywhere in the entire architecture.

Alternatives: HashiCorp Vault#

While AWS Secrets Manager is excellent, it only works in AWS. If a Fortune 500 company has a Hybrid-Cloud architecture (some servers in AWS, some in Google Cloud, some in an On-Premises basement), they cannot use AWS Secrets Manager for everything. Instead, they deploy HashiCorp Vault. Vault is cloud-agnostic. It provides the exact same functionality (encryption, dynamic secrets, rotation) but works universally across all clouds. The External Secrets Operator can be configured to pull from Vault instead of AWS with a 1-line code change.

Interview Questions#

Beginner#

Q: What is the difference between encryption and Base64 encoding? A: Base64 encoding is just a way to translate text into a format computers can easily transmit; anyone can instantly translate it back without a key. Encryption (like AES-256) scrambles the data mathematically; you cannot read the data unless you possess the specific cryptographic key used to lock it. Kubernetes Secrets use Base64. AWS Secrets Manager uses Encryption.

Intermediate#

Q: How does the External Secrets Operator (ESO) solve the GitOps secrets problem? A: GitOps requires all YAML to be in Git. ESO allows us to store an ExternalSecret YAML in Git that contains pointers to AWS (e.g., key: prod/rds), rather than the actual passwords. ArgoCD syncs the pointers, and ESO dynamically fetches the real passwords from AWS at runtime.

Senior#

Q: You configured AWS Secrets Manager to automatically rotate the RDS database password every 30 days. However, your Java Spring Boot application in Kubernetes crashed with an "Access Denied" database error immediately after the rotation. Why, and how do you fix it? A: While ESO successfully pulled the new password and updated the Kubernetes Secret, the Java Spring Boot application only reads environment variables or mounted files at startup. It does not hot-reload secrets. You must architect a solution to restart the Pod when the Secret changes. Tools like Reloader (Stakater) watch Kubernetes Secrets; when a Secret updates, Reloader automatically performs a rolling restart of the associated Deployment, ensuring the Java app reads the new password without downtime.

Principal/Architect#

Q: Explain how OIDC federation (IRSA) establishes a chain of trust between a Kubernetes cluster and AWS IAM without storing static AWS access keys in the cluster. A: The Kubernetes API Server is configured as an OpenID Connect (OIDC) Identity Provider. AWS IAM is configured to trust this specific OIDC provider endpoint. When a Pod (with a designated ServiceAccount) is scheduled, the kubelet injects a cryptographically signed JWT into the Pod's filesystem. The AWS SDK inside the Pod reads this JWT and sends an AssumeRoleWithWebIdentity API call to AWS STS. STS verifies the JWT's signature against the Kubernetes API Server's public OIDC discovery keys. If valid, STS returns temporary AWS session credentials to the Pod. This entire chain relies on cryptographic trust, eliminating the need to ever create or rotate long-lived static AWS access keys. Contents | 19 — Container Orchestration (Kubernetes) |