AWS Secrets Manager Integration

Overview

AWS Secrets Manager is a managed service for storing, retrieving, and rotating application secrets like database credentials, API keys, and OAuth tokens. It eliminates the practice of hardcoding credentials in source code or stuffing them into environment variables that get committed to version control. If you are running anything in production on AWS, Secrets Manager should be your default choice for credential management.

Prerequisites

• An AWS account with IAM permissions for Secrets Manager
• Node.js v16 or later installed
• AWS CLI v2 configured with credentials (aws configure)
• Basic familiarity with AWS SDK for JavaScript v3
• An Express.js application (for the complete example)

Install the required packages:

npm install @aws-sdk/client-secrets-manager @aws-sdk/client-lambda express pg

Creating and Managing Secrets

Secrets Manager stores secrets as key-value pairs or plaintext strings. The most common pattern is a JSON object containing multiple related values. You can create secrets through the AWS Console, CLI, or SDK.

Creating a Secret via CLI

aws secretsmanager create-secret \
  --name production/myapp/database \
  --description "Production database credentials" \
  --secret-string '{"username":"app_user","password":"S3cur3P@ss!","host":"mydb.cluster-abc123.us-east-1.rds.amazonaws.com","port":5432,"dbname":"myapp_prod"}'

I use a hierarchical naming convention with forward slashes: environment/application/purpose. This maps cleanly to IAM policy conditions and makes it trivial to grant access by environment or application.

Creating a Secret with Node.js

var AWS = require("@aws-sdk/client-secrets-manager");

var client = new AWS.SecretsManagerClient({ region: "us-east-1" });

function createSecret(name, secretValue) {
  var command = new AWS.CreateSecretCommand({
    Name: name,
    Description: "Application database credentials",
    SecretString: JSON.stringify(secretValue),
    Tags: [
      { Key: "Environment", Value: "production" },
      { Key: "Application", Value: "myapp" }
    ]
  });

  return client.send(command);
}

var credentials = {
  username: "app_user",
  password: "S3cur3P@ss!",
  host: "mydb.cluster-abc123.us-east-1.rds.amazonaws.com",
  port: 5432,
  dbname: "myapp_prod"
};

createSecret("production/myapp/database", credentials)
  .then(function(result) {
    console.log("Secret created:", result.ARN);
  })
  .catch(function(err) {
    console.error("Failed to create secret:", err.message);
  });

Updating a Secret

function updateSecret(name, newValue) {
  var command = new AWS.PutSecretValueCommand({
    SecretId: name,
    SecretString: JSON.stringify(newValue)
  });

  return client.send(command);
}

Every time you call PutSecretValue, Secrets Manager creates a new version. The previous version is still accessible through version IDs and staging labels. This is important for rotation, which I will cover shortly.

Retrieving Secrets with Node.js SDK

Retrieving secrets is the operation you will perform most frequently. Here is a straightforward retrieval function:

var AWS = require("@aws-sdk/client-secrets-manager");

var client = new AWS.SecretsManagerClient({ region: "us-east-1" });

function getSecret(secretName) {
  var command = new AWS.GetSecretValueCommand({
    SecretId: secretName
  });

  return client.send(command).then(function(response) {
    if (response.SecretString) {
      return JSON.parse(response.SecretString);
    }
    // Binary secrets are returned as a Buffer
    var buff = Buffer.from(response.SecretBinary, "base64");
    return buff.toString("utf-8");
  });
}

getSecret("production/myapp/database").then(function(secret) {
  console.log("Host:", secret.host);
  console.log("Username:", secret.username);
});

This works but has a problem: every call hits the Secrets Manager API. At $0.05 per 10,000 API calls plus the latency of a network round trip on every request, this adds up fast in a high-traffic application.

Caching Secrets for Performance

The official AWS recommendation is to cache secrets locally and refresh them periodically. AWS provides a caching client for some languages, but for Node.js you are better off building your own. It is simple and gives you full control over cache invalidation.

var AWS = require("@aws-sdk/client-secrets-manager");

var client = new AWS.SecretsManagerClient({ region: "us-east-1" });
var cache = {};
var DEFAULT_TTL = 300000; // 5 minutes in milliseconds

function getCachedSecret(secretName, ttl) {
  var cacheTtl = ttl || DEFAULT_TTL;
  var now = Date.now();
  var cached = cache[secretName];

  if (cached && (now - cached.timestamp) < cacheTtl) {
    return Promise.resolve(cached.value);
  }

  var command = new AWS.GetSecretValueCommand({
    SecretId: secretName
  });

  return client.send(command).then(function(response) {
    var value = JSON.parse(response.SecretString);
    cache[secretName] = {
      value: value,
      timestamp: now,
      versionId: response.VersionId
    };
    return value;
  });
}

function invalidateCache(secretName) {
  if (secretName) {
    delete cache[secretName];
  } else {
    cache = {};
  }
}

module.exports = {
  getSecret: getCachedSecret,
  invalidateCache: invalidateCache
};

A five-minute TTL is a reasonable default. It means that after a secret rotates, your application will pick up the new value within five minutes. For most applications that is acceptable. If you need tighter control, reduce the TTL or use CloudWatch Events to trigger cache invalidation.

Secret Rotation with Lambda

Automatic rotation is the feature that separates Secrets Manager from simpler alternatives. You configure a Lambda function that Secrets Manager invokes on a schedule to generate new credentials, update the target service, and store the new values.

How Rotation Works

Rotation follows a four-step process, each triggered by Secrets Manager calling your Lambda with a different Step value:

1. createSecret - Generate a new credential and store it as AWSPENDING
2. setSecret - Apply the new credential to the target service (e.g., update the database password)
3. testSecret - Verify the new credential works
4. finishSecret - Move the AWSCURRENT label to the new version

Rotation Lambda for RDS

var AWS = require("@aws-sdk/client-secrets-manager");
var crypto = require("crypto");
var pg = require("pg");

var secretsClient = new AWS.SecretsManagerClient({ region: "us-east-1" });

exports.handler = function(event, context, callback) {
  var secretId = event.SecretId;
  var step = event.Step;
  var token = event.ClientRequestToken;

  console.log("Rotation step:", step, "for secret:", secretId);

  var stepHandlers = {
    createSecret: createSecret,
    setSecret: setSecret,
    testSecret: testSecret,
    finishSecret: finishSecret
  };

  var handler = stepHandlers[step];
  if (!handler) {
    return callback(new Error("Unknown step: " + step));
  }

  handler(secretId, token)
    .then(function() { callback(null); })
    .catch(function(err) { callback(err); });
};

function createSecret(secretId, token) {
  // Get current secret
  var getCommand = new AWS.GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSCURRENT"
  });

  return secretsClient.send(getCommand).then(function(current) {
    var currentSecret = JSON.parse(current.SecretString);

    // Generate new password
    var newPassword = crypto.randomBytes(32).toString("base64").slice(0, 40);
    currentSecret.password = newPassword;

    // Store as pending
    var putCommand = new AWS.PutSecretValueCommand({
      SecretId: secretId,
      ClientRequestToken: token,
      SecretString: JSON.stringify(currentSecret),
      VersionStages: ["AWSPENDING"]
    });

    return secretsClient.send(putCommand);
  });
}

function setSecret(secretId, token) {
  // Get the pending secret
  var getCommand = new AWS.GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSPENDING",
    VersionId: token
  });

  return secretsClient.send(getCommand).then(function(pending) {
    var secret = JSON.parse(pending.SecretString);

    // Connect with current (admin) credentials and update password
    var dbClient = new pg.Client({
      host: secret.host,
      port: secret.port,
      database: secret.dbname,
      user: secret.username,
      password: secret.password
    });

    return dbClient.connect().then(function() {
      var query = "ALTER USER " + secret.username + " WITH PASSWORD '" + secret.password + "'";
      return dbClient.query(query);
    }).then(function() {
      return dbClient.end();
    });
  });
}

function testSecret(secretId, token) {
  var getCommand = new AWS.GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSPENDING",
    VersionId: token
  });

  return secretsClient.send(getCommand).then(function(pending) {
    var secret = JSON.parse(pending.SecretString);

    var dbClient = new pg.Client({
      host: secret.host,
      port: secret.port,
      database: secret.dbname,
      user: secret.username,
      password: secret.password
    });

    return dbClient.connect().then(function() {
      return dbClient.query("SELECT 1");
    }).then(function() {
      return dbClient.end();
    });
  });
}

function finishSecret(secretId, token) {
  // Get current version
  var describeCommand = new AWS.DescribeSecretCommand({
    SecretId: secretId
  });

  return secretsClient.send(describeCommand).then(function(metadata) {
    var versions = metadata.VersionIdsToStages;
    var currentVersionId;

    Object.keys(versions).forEach(function(versionId) {
      if (versions[versionId].indexOf("AWSCURRENT") !== -1) {
        currentVersionId = versionId;
      }
    });

    // Move AWSCURRENT to new version, AWSPREVIOUS to old
    var updateCommand = new AWS.UpdateSecretVersionStageCommand({
      SecretId: secretId,
      VersionStage: "AWSCURRENT",
      MoveToVersionId: token,
      RemoveFromVersionId: currentVersionId
    });

    return secretsClient.send(updateCommand);
  });
}

Enabling Rotation via CLI

aws secretsmanager rotate-secret \
  --secret-id production/myapp/database \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:RotateMyAppSecret \
  --rotation-rules '{"AutomaticallyAfterDays": 30}'

Thirty days is a common rotation interval. For highly sensitive credentials, consider seven days. The tradeoff is that more frequent rotation increases the chance of transient connection failures during the rotation window.

RDS Credential Rotation

For RDS databases, AWS provides pre-built rotation Lambda functions. This is the easiest path and what I recommend for most teams.

# Enable rotation with the built-in RDS single-user rotation template
aws secretsmanager rotate-secret \
  --secret-id production/myapp/rds-credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRDSPostgreSQLRotation \
  --rotation-rules '{"ScheduleExpression": "rate(30 days)"}'

The single-user rotation strategy updates the password for the same user. The multi-user strategy alternates between two users so that one set of credentials is always valid. Use multi-user if you cannot tolerate any connection failures during rotation.

Multi-User Rotation Secret Format

For multi-user rotation, the secret must include a reference to a master secret:

{
  "username": "app_user",
  "password": "currentPassword123",
  "host": "mydb.cluster-abc123.us-east-1.rds.amazonaws.com",
  "port": 5432,
  "dbname": "myapp_prod",
  "masterarn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/myapp/rds-master"
}

Secrets Manager vs Parameter Store

This is a question I get asked constantly. Here is my honest take:

Feature	Secrets Manager	Parameter Store (SecureString)
Cost	$0.40/secret/month + $0.05/10K API calls	Free tier for standard params; $0.05/10K API calls for advanced
Rotation	Built-in automatic rotation	Manual only
Cross-account	Native support via resource policies	Requires IAM roles and custom solutions
Binary secrets	Yes	No
Versioning	Automatic with staging labels	Version history but no staging labels
Max size	64 KB	8 KB (advanced tier)
CloudFormation	Dynamic references supported	Dynamic references supported

My recommendation: Use Secrets Manager for anything that needs rotation (database passwords, API keys with expiry) and for cross-account scenarios. Use Parameter Store for static configuration values, feature flags, and non-sensitive config. Do not overthink this. The cost difference is negligible at scale.

Secret Versioning and Staging Labels

Every time you update a secret, Secrets Manager creates a new version with a unique version ID. Staging labels control which version is returned by default:

• AWSCURRENT - The active version returned by default
• AWSPREVIOUS - The previous version (automatically assigned during rotation)
• AWSPENDING - The version being prepared during rotation

You can also create custom staging labels:

function getSecretByStage(secretName, stage) {
  var command = new AWS.GetSecretValueCommand({
    SecretId: secretName,
    VersionStage: stage || "AWSCURRENT"
  });

  return client.send(command).then(function(response) {
    return {
      value: JSON.parse(response.SecretString),
      versionId: response.VersionId,
      versionStages: response.VersionStages
    };
  });
}

// Retrieve the previous version (useful for rollback)
getSecretByStage("production/myapp/database", "AWSPREVIOUS")
  .then(function(result) {
    console.log("Previous version:", result.versionId);
  });

You can add custom staging labels to implement blue-green deployment patterns for secrets:

aws secretsmanager update-secret-version-stage \
  --secret-id production/myapp/database \
  --version-stage BLUE \
  --move-to-version-id "abc-123-def-456"

Cross-Account Secret Sharing

Sharing secrets across AWS accounts is a common requirement in organizations with multiple accounts for different environments or teams. Secrets Manager supports this through resource-based policies.

Granting Cross-Account Access

aws secretsmanager put-resource-policy \
  --secret-id production/myapp/shared-api-key \
  --resource-policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::987654321098:role/ConsumerAppRole"
        },
        "Action": [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ],
        "Resource": "*"
      }
    ]
  }'

Retrieving a Cross-Account Secret

From the consuming account, reference the secret by its full ARN:

var client = new AWS.SecretsManagerClient({ region: "us-east-1" });

function getCrossAccountSecret() {
  var command = new AWS.GetSecretValueCommand({
    SecretId: "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/myapp/shared-api-key-AbCdEf"
  });

  return client.send(command).then(function(response) {
    return JSON.parse(response.SecretString);
  });
}

The consuming account's IAM role also needs secretsmanager:GetSecretValue permission. Both the resource policy on the secret and the identity policy on the role must allow the action. This is the standard AWS dual-authorization model.

If the secret is encrypted with a customer-managed KMS key, you must also grant kms:Decrypt permission to the consuming account on that KMS key.

CloudFormation Integration

You can reference Secrets Manager secrets directly in CloudFormation templates using dynamic references. This keeps credentials out of your templates entirely.

AWSTemplateFormatVersion: '2010-09-09'
Description: Application stack with Secrets Manager integration

Resources:
  AppSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: production/myapp/database
      Description: Database credentials for MyApp
      GenerateSecretString:
        SecretStringTemplate: '{"username": "app_user"}'
        GenerateStringKey: password
        PasswordLength: 40
        ExcludeCharacters: '"@/\'

  AppDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: db.t3.medium
      Engine: postgres
      MasterUsername: !Sub '{{resolve:secretsmanager:${AppSecret}:SecretString:username}}'
      MasterUserPassword: !Sub '{{resolve:secretsmanager:${AppSecret}:SecretString:password}}'
      DBName: myapp_prod

  RotationSchedule:
    Type: AWS::SecretsManager::RotationSchedule
    Properties:
      SecretId: !Ref AppSecret
      RotationLambdaARN: !GetAtt RotationFunction.Arn
      RotationRules:
        AutomaticallyAfterDays: 30

  SecretTargetAttachment:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId: !Ref AppSecret
      TargetId: !Ref AppDatabase
      TargetType: AWS::RDS::DBInstance

The SecretTargetAttachment resource is important. It automatically updates the secret with the database endpoint, port, and engine information after the RDS instance is created. Without it, you would need to manually add connection details to the secret.

Lambda Layer for Secret Caching

For Lambda functions that need secrets, packaging a caching layer avoids repeated API calls across invocations within the same execution environment.

// layer/nodejs/secretsCache.js
var AWS = require("@aws-sdk/client-secrets-manager");

var client = new AWS.SecretsManagerClient({
  region: process.env.AWS_REGION
});
var cache = {};

function getSecret(secretName) {
  if (cache[secretName]) {
    return Promise.resolve(cache[secretName]);
  }

  var command = new AWS.GetSecretValueCommand({
    SecretId: secretName
  });

  return client.send(command).then(function(response) {
    var value = JSON.parse(response.SecretString);
    cache[secretName] = value;
    return value;
  });
}

function clearCache() {
  cache = {};
}

module.exports = {
  getSecret: getSecret,
  clearCache: clearCache
};

Lambda execution environments persist between invocations, so the in-memory cache survives across requests. The cache resets when the execution environment is recycled, which naturally provides periodic refreshes. For Lambda, this is usually sufficient. If you need explicit TTL-based expiry in Lambda, use the same pattern from the caching section above.

Deploy the layer:

cd layer
zip -r secrets-cache-layer.zip nodejs/
aws lambda publish-layer-version \
  --layer-name secrets-cache \
  --zip-file fileb://secrets-cache-layer.zip \
  --compatible-runtimes nodejs18.x nodejs20.x

Monitoring Secret Access with CloudTrail

Every Secrets Manager API call is logged in CloudTrail. This is critical for auditing who accessed which secrets and when.

Key Events to Monitor

• GetSecretValue - Someone or something retrieved a secret
• PutSecretValue - A secret was updated
• DeleteSecret - A secret was deleted
• RotationStarted - Automatic rotation began
• RotationSucceeded / RotationFailed - Rotation completed or failed

CloudWatch Alarm for Unauthorized Access

aws cloudwatch put-metric-alarm \
  --alarm-name "SecretsManager-UnauthorizedAccess" \
  --metric-name "Errors" \
  --namespace "AWS/SecretsManager" \
  --statistic "Sum" \
  --period 300 \
  --threshold 5 \
  --comparison-operator "GreaterThanThreshold" \
  --evaluation-periods 1 \
  --alarm-actions "arn:aws:sns:us-east-1:123456789012:security-alerts"

CloudTrail Log Query with Athena

SELECT
  eventtime,
  useridentity.arn AS caller,
  requestparameters AS params,
  errorcode,
  errormessage
FROM cloudtrail_logs
WHERE eventsource = 'secretsmanager.amazonaws.com'
  AND eventname = 'GetSecretValue'
  AND errorcode IS NOT NULL
ORDER BY eventtime DESC
LIMIT 50;

This query surfaces failed secret retrieval attempts, which could indicate misconfigured applications or unauthorized access attempts.

Complete Working Example

Here is a production-ready Express application that demonstrates Secrets Manager integration with caching, rotation handling, and graceful fallback.

// app.js
var express = require("express");
var AWS = require("@aws-sdk/client-secrets-manager");
var pg = require("pg");

var app = express();
var PORT = process.env.PORT || 3000;

// --- Secrets Manager Client ---

var secretsClient = new AWS.SecretsManagerClient({
  region: process.env.AWS_REGION || "us-east-1"
});

// --- Secret Cache ---

var secretCache = {};
var CACHE_TTL = 300000; // 5 minutes

function getCachedSecret(secretName) {
  var now = Date.now();
  var cached = secretCache[secretName];

  if (cached && (now - cached.timestamp) < CACHE_TTL) {
    return Promise.resolve(cached.value);
  }

  var command = new AWS.GetSecretValueCommand({
    SecretId: secretName
  });

  return secretsClient.send(command)
    .then(function(response) {
      var value = JSON.parse(response.SecretString);
      secretCache[secretName] = {
        value: value,
        timestamp: now,
        versionId: response.VersionId
      };
      return value;
    })
    .catch(function(err) {
      // If we have a stale cached value, return it during API failures
      if (cached) {
        console.warn(
          "Secrets Manager API failed, using stale cache for:",
          secretName,
          "Error:",
          err.message
        );
        return cached.value;
      }
      throw err;
    });
}

function invalidateSecret(secretName) {
  delete secretCache[secretName];
}

// --- Database Connection Pool with Rotation Handling ---

var pool = null;
var DB_SECRET_NAME = process.env.DB_SECRET_NAME || "production/myapp/database";

function getDbPool() {
  if (pool) {
    return Promise.resolve(pool);
  }

  return getCachedSecret(DB_SECRET_NAME).then(function(dbSecret) {
    pool = new pg.Pool({
      host: dbSecret.host,
      port: dbSecret.port,
      database: dbSecret.dbname,
      user: dbSecret.username,
      password: dbSecret.password,
      max: 20,
      idleTimeoutMillis: 30000,
      connectionTimeoutMillis: 5000
    });

    pool.on("error", function(err) {
      console.error("Unexpected pool error:", err.message);

      // If authentication fails, the password may have rotated
      if (err.message.indexOf("authentication") !== -1 ||
          err.message.indexOf("password") !== -1) {
        console.log("Possible credential rotation detected. Refreshing pool.");
        invalidateSecret(DB_SECRET_NAME);
        pool.end();
        pool = null;
      }
    });

    return pool;
  });
}

function queryWithRetry(sql, params, retries) {
  var maxRetries = retries || 1;

  return getDbPool()
    .then(function(dbPool) {
      return dbPool.query(sql, params);
    })
    .catch(function(err) {
      // Retry once on auth failure (credential rotation)
      if (maxRetries > 0 && (
        err.code === "28P01" || // PostgreSQL invalid password
        err.message.indexOf("authentication") !== -1
      )) {
        console.log("Auth failure detected. Invalidating cache and retrying.");
        invalidateSecret(DB_SECRET_NAME);
        pool.end();
        pool = null;
        return queryWithRetry(sql, params, maxRetries - 1);
      }
      throw err;
    });
}

// --- API Key Retrieval ---

var API_KEYS_SECRET = process.env.API_KEYS_SECRET || "production/myapp/api-keys";

function getApiKey(keyName) {
  return getCachedSecret(API_KEYS_SECRET).then(function(keys) {
    if (!keys[keyName]) {
      throw new Error("API key not found: " + keyName);
    }
    return keys[keyName];
  });
}

// --- Express Routes ---

app.get("/health", function(req, res) {
  queryWithRetry("SELECT 1 AS ok")
    .then(function(result) {
      res.json({ status: "healthy", db: "connected" });
    })
    .catch(function(err) {
      res.status(503).json({
        status: "unhealthy",
        db: "disconnected",
        error: err.message
      });
    });
});

app.get("/users", function(req, res) {
  queryWithRetry("SELECT id, name, email FROM users LIMIT 50")
    .then(function(result) {
      res.json(result.rows);
    })
    .catch(function(err) {
      console.error("Query failed:", err.message);
      res.status(500).json({ error: "Internal server error" });
    });
});

app.get("/external-data", function(req, res) {
  getApiKey("stripe")
    .then(function(apiKey) {
      // Use the API key for an external service call
      res.json({
        message: "API key retrieved successfully",
        keyPrefix: apiKey.substring(0, 7) + "..."
      });
    })
    .catch(function(err) {
      console.error("Failed to get API key:", err.message);
      res.status(500).json({ error: "Configuration error" });
    });
});

// --- Graceful Shutdown ---

process.on("SIGTERM", function() {
  console.log("SIGTERM received. Shutting down gracefully.");
  if (pool) {
    pool.end().then(function() {
      console.log("Database pool closed.");
      process.exit(0);
    });
  } else {
    process.exit(0);
  }
});

// --- Start Server ---

app.listen(PORT, function() {
  console.log("Server running on port", PORT);

  // Pre-warm the secret cache on startup
  getCachedSecret(DB_SECRET_NAME)
    .then(function() {
      console.log("Database secret cached successfully.");
    })
    .catch(function(err) {
      console.error("WARNING: Failed to pre-cache database secret:", err.message);
      console.error("Application will retry on first request.");
    });
});

Key design decisions in this example:

1. Stale cache fallback - If the Secrets Manager API is unavailable, we return the last known value rather than failing. This prevents API outages from cascading into application failures.
2. Rotation-aware retry - When a database query fails with an authentication error, we invalidate the cached credentials, rebuild the connection pool, and retry once. This handles the brief window during rotation when cached credentials may be stale.
3. Pre-warming - The secret cache is populated at startup so the first request does not incur the latency of a Secrets Manager API call.
4. Graceful shutdown - The connection pool is drained cleanly on SIGTERM, which is essential for container environments.

Common Issues and Troubleshooting

1. AccessDeniedException on GetSecretValue

AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/MyAppRole/i-0abc123
is not authorized to perform: secretsmanager:GetSecretValue on resource:
arn:aws:secretsmanager:us-east-1:123456789012:secret:production/myapp/database-AbCdEf

This means your IAM role lacks the secretsmanager:GetSecretValue permission. Add it to the role's policy:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/myapp/*"
}

If the secret is encrypted with a customer-managed KMS key, you also need kms:Decrypt on that key.

2. ResourceNotFoundException During Rotation

ResourceNotFoundException: Secrets Manager can't find the specified secret.

This typically occurs when the rotation Lambda cannot find the pending secret version. Verify that the ClientRequestToken in PutSecretValue matches the token passed to the Lambda. Also confirm the Lambda has network access to the Secrets Manager endpoint, especially if it runs inside a VPC. You need either a NAT Gateway or a VPC endpoint for Secrets Manager.

3. Connection Reset After Rotation

Error: Connection terminated unexpectedly
error: password authentication failed for user "app_user"

Your application is using cached credentials that were invalidated by rotation. Implement the retry-on-auth-failure pattern shown in the complete example above. Also ensure your cache TTL is shorter than your rotation window.

4. Rotation Lambda Timeout

Task timed out after 30.00 seconds

The default Lambda timeout of 30 seconds is often not enough for rotation, especially if the Lambda needs to connect to a database in a VPC. Increase the timeout to 120 seconds minimum. Also check that the Lambda's security group allows outbound traffic to both the Secrets Manager endpoint and the target database.

5. DecryptionFailure with KMS

DecryptionFailure: The ciphertext refers to a customer master key that does not exist,
does not exist in this region, or you are not allowed to access.

This occurs when you move secrets across regions or when KMS key policies change. Ensure the KMS key used to encrypt the secret exists in the same region and that the calling identity has kms:Decrypt permission on that specific key ARN.

Best Practices

• Never log secret values. It sounds obvious, but I have seen production systems dump database credentials into CloudWatch Logs via overly verbose error handlers. Log the secret name and version ID, never the value.

• Use resource-based policies for cross-account access instead of sharing IAM credentials. Resource policies are auditable, revocable, and follow the principle of least privilege.

• Tag every secret with at least Environment, Application, and Owner. Tags enable cost allocation, IAM policy conditions, and quick identification during incident response.

• Set up CloudWatch alarms for rotation failures. A failed rotation can leave your application with credentials that will expire. Monitor the RotationFailed CloudTrail event and alert your on-call team immediately.

• Use VPC endpoints for Secrets Manager if your application runs in a VPC. This avoids routing API calls through the public internet and eliminates the need for a NAT Gateway on that traffic path. The endpoint type is com.amazonaws.{region}.secretsmanager.

• Implement graceful degradation with stale cache. A Secrets Manager API outage should not bring down your application if you have recently cached credentials. Return stale values with a warning log rather than failing the request.

• Rotate secrets on a schedule, not just when compromised. Regular rotation limits the blast radius of any credential leak. Even if a credential is exfiltrated, it becomes useless within the rotation window.

• Use separate secrets for separate concerns. Do not stuff database credentials, API keys, and OAuth tokens into a single secret. Separate secrets allow granular IAM policies and independent rotation schedules.

• Prefer GenerateSecretString in CloudFormation over hardcoded passwords. Let Secrets Manager generate strong random passwords that never appear in your template source code or CloudFormation state.

• Test rotation in staging before enabling it in production. Rotation failures can lock you out of your database. Run at least three successful rotation cycles in a non-production environment before enabling it on production secrets.

References

• AWS Secrets Manager User Guide
• AWS SDK for JavaScript v3 - SecretsManagerClient
• Rotate AWS Secrets Manager secrets
• AWS Secrets Manager pricing
• VPC Endpoints for Secrets Manager
• CloudFormation Dynamic References
• AWS Secrets Manager vs Systems Manager Parameter Store