Secrets Rotation Automation
Complete guide to automating secrets rotation in Azure DevOps, covering Azure Key Vault rotation policies, PAT token lifecycle automation, service connection credential renewal, database password rotation, and building self-healing secret management pipelines.
automationazure-devopssecurityservice-connectionsazure key vaultsecrets rotationcredential managementpat tokens
Secrets Rotation Automation
Overview
Every secret has a shelf life. The longer a credential sits unchanged, the higher the probability it has been leaked, copied to someone's notes, or stored in a place it should not be. Manual rotation is better than no rotation, but it does not scale — when you have 50 service connections, 30 PAT tokens, and a dozen database passwords, manual rotation means someone eventually forgets one. I have built automated rotation systems for teams ranging from 5 to 500 engineers, and the pattern is the same: automate everything, alert on failures, and never trust a secret older than 90 days.
Prerequisites
- Azure DevOps organization with Organization Owner or Project Collection Administrator permissions
- Azure Key Vault instance with Contributor or Key Vault Secrets Officer role
- Azure subscription with Azure Functions or Logic Apps for rotation triggers
- Node.js 16 or later for automation scripts
- Personal Access Token with full scope for administrative operations
- Service principal with Key Vault and Azure DevOps access
Understanding the Rotation Lifecycle
Every secret follows the same lifecycle:
Create → Store → Distribute → Use → Monitor → Rotate → Retire
| |
└──────────────────────────────────────────────┘
(automated loop)
The rotation step creates a new secret, updates all consumers, verifies the new secret works, and then retires the old one. The critical requirement: zero downtime during rotation.
Dual-Secret Pattern
The safest rotation strategy uses two active secrets simultaneously:
var crypto = require("crypto");
// Dual-secret rotation ensures zero downtime
// At any point, both the current and previous secret are valid
function rotateDualSecret(currentVersion, previousVersion) {
// Step 1: Generate new secret
var newSecret = crypto.randomBytes(32).toString("hex");
// Step 2: The new secret becomes "current"
// The old "current" becomes "previous"
// The old "previous" gets deleted
var rotation = {
newCurrent: {
value: newSecret,
version: currentVersion + 1,
createdAt: new Date().toISOString()
},
newPrevious: {
value: "<<current secret value>>",
version: currentVersion,
createdAt: "<<current created date>>"
},
retired: {
version: previousVersion,
retiredAt: new Date().toISOString()
}
};
console.log("Rotation plan:");
console.log(" New current: version " + rotation.newCurrent.version);
console.log(" New previous: version " + rotation.newPrevious.version);
console.log(" Retiring: version " + rotation.retired.version);
return rotation;
}
Azure Key Vault Rotation Policies
Key Vault supports automatic rotation with configurable policies.
Setting Up Rotation Policies
# Create a rotation policy for a secret
az keyvault secret rotation-policy update \
--vault-name myapp-keyvault \
--name database-password \
--value '{
"lifetimeActions": [
{
"trigger": {
"timeBeforeExpiry": "P30D"
},
"action": {
"type": "Notify"
}
},
{
"trigger": {
"timeAfterCreate": "P60D"
},
"action": {
"type": "Rotate"
}
}
],
"attributes": {
"expiryTime": "P90D"
}
}'
Key Vault Event-Driven Rotation
var https = require("https");
var crypto = require("crypto");
// Azure Function triggered by Key Vault "SecretNearExpiry" event
// via Event Grid subscription
function handleSecretNearExpiry(event) {
var secretName = event.data.ObjectName;
var vaultName = event.data.VaultName;
var expirationDate = event.data.EXP;
console.log("Secret near expiry: " + secretName + " in " + vaultName);
console.log("Expires: " + new Date(expirationDate * 1000).toISOString());
// Determine rotation strategy based on secret name pattern
var strategies = {
"db-password": rotateDbPassword,
"api-key": rotateApiKey,
"pat-token": rotatePat,
"service-connection": rotateServiceConnection
};
var strategy = null;
Object.keys(strategies).forEach(function(pattern) {
if (secretName.indexOf(pattern) !== -1) {
strategy = strategies[pattern];
}
});
if (strategy) {
return strategy(vaultName, secretName)
.then(function(result) {
console.log("Rotation complete: " + JSON.stringify(result));
return result;
})
.catch(function(err) {
console.error("Rotation failed: " + err.message);
sendRotationAlert(secretName, err.message);
throw err;
});
} else {
console.log("No rotation strategy for: " + secretName);
sendRotationAlert(secretName, "Manual rotation required - no automated strategy configured");
}
}
function sendRotationAlert(secretName, message) {
var webhookUrl = process.env.TEAMS_WEBHOOK_URL;
if (!webhookUrl) return;
var payload = JSON.stringify({
"@type": "MessageCard",
themeColor: "FF0000",
title: "Secret Rotation Alert",
text: "Secret **" + secretName + "** requires attention: " + message,
potentialAction: [{
"@type": "OpenUri",
name: "Open Key Vault",
targets: [{ os: "default", uri: "https://portal.azure.com/#view/Microsoft_Azure_KeyVault" }]
}]
});
var url = new URL(webhookUrl);
var options = {
hostname: url.hostname,
path: url.pathname,
method: "POST",
headers: {
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(payload)
}
};
var req = https.request(options);
req.write(payload);
req.end();
}
PAT Token Rotation Automation
Personal Access Tokens are the most commonly leaked credential in Azure DevOps. Automating their rotation eliminates human forgetfulness.
Automated PAT Lifecycle Manager
var https = require("https");
// PAT Token lifecycle manager
// Creates, monitors, rotates, and revokes PATs
var config = {
organization: process.env.ADO_ORG,
adminPat: process.env.ADO_ADMIN_PAT, // Admin PAT with token management scope
maxAgeDays: 90,
warningDays: 14
};
function listPats() {
var options = {
hostname: "vssps.dev.azure.com",
path: "/" + config.organization + "/_apis/tokens/pats?api-version=7.1-preview.1",
method: "GET",
headers: {
"Authorization": "Basic " + Buffer.from(":" + config.adminPat).toString("base64")
}
};
return new Promise(function(resolve, reject) {
var req = https.request(options, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() {
var data = JSON.parse(body);
resolve(data.patTokens || []);
});
});
req.on("error", reject);
req.end();
});
}
function createPat(displayName, scope, validDays) {
var validTo = new Date();
validTo.setDate(validTo.getDate() + validDays);
var postData = JSON.stringify({
displayName: displayName,
scope: scope,
validTo: validTo.toISOString(),
allOrgs: false
});
var options = {
hostname: "vssps.dev.azure.com",
path: "/" + config.organization + "/_apis/tokens/pats?api-version=7.1-preview.1",
method: "POST",
headers: {
"Authorization": "Basic " + Buffer.from(":" + config.adminPat).toString("base64"),
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(postData)
}
};
return new Promise(function(resolve, reject) {
var req = https.request(options, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() {
resolve(JSON.parse(body));
});
});
req.on("error", reject);
req.write(postData);
req.end();
});
}
function revokePat(authorizationId) {
var options = {
hostname: "vssps.dev.azure.com",
path: "/" + config.organization + "/_apis/tokens/pats?authorizationId=" + authorizationId + "&api-version=7.1-preview.1",
method: "DELETE",
headers: {
"Authorization": "Basic " + Buffer.from(":" + config.adminPat).toString("base64")
}
};
return new Promise(function(resolve, reject) {
var req = https.request(options, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() {
resolve({ status: res.statusCode });
});
});
req.on("error", reject);
req.end();
});
}
function storeInKeyVault(vaultName, secretName, secretValue, tenantId, clientId, clientSecret) {
// Get access token for Key Vault
var tokenData = "grant_type=client_credentials"
+ "&client_id=" + clientId
+ "&client_secret=" + encodeURIComponent(clientSecret)
+ "&scope=https://vault.azure.net/.default";
return new Promise(function(resolve, reject) {
var tokenReq = https.request({
hostname: "login.microsoftonline.com",
path: "/" + tenantId + "/oauth2/v2.0/token",
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": Buffer.byteLength(tokenData)
}
}, function(tokenRes) {
var tokenBody = "";
tokenRes.on("data", function(chunk) { tokenBody += chunk; });
tokenRes.on("end", function() {
var accessToken = JSON.parse(tokenBody).access_token;
// Store secret in Key Vault
var secretData = JSON.stringify({
value: secretValue,
attributes: {
enabled: true,
exp: Math.floor(Date.now() / 1000) + (config.maxAgeDays * 86400)
},
tags: {
rotatedAt: new Date().toISOString(),
managedBy: "pat-rotation-automation"
}
});
var secretReq = https.request({
hostname: vaultName + ".vault.azure.net",
path: "/secrets/" + secretName + "?api-version=7.4",
method: "PUT",
headers: {
"Authorization": "Bearer " + accessToken,
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(secretData)
}
}, function(secretRes) {
var secretBody = "";
secretRes.on("data", function(chunk) { secretBody += chunk; });
secretRes.on("end", function() {
resolve(JSON.parse(secretBody));
});
});
secretReq.on("error", reject);
secretReq.write(secretData);
secretReq.end();
});
});
tokenReq.on("error", reject);
tokenReq.write(tokenData);
tokenReq.end();
});
}
// Audit and rotate PATs
function auditAndRotate() {
return listPats().then(function(tokens) {
var now = new Date();
var expiring = [];
var expired = [];
var healthy = [];
tokens.forEach(function(token) {
var validTo = new Date(token.validTo);
var daysRemaining = Math.floor((validTo - now) / (1000 * 60 * 60 * 24));
var status = {
name: token.displayName,
authorizationId: token.authorizationId,
scope: token.scope,
validTo: token.validTo,
daysRemaining: daysRemaining
};
if (daysRemaining < 0) {
expired.push(status);
} else if (daysRemaining < config.warningDays) {
expiring.push(status);
} else {
healthy.push(status);
}
});
console.log("=== PAT Token Audit ===");
console.log("Total tokens: " + tokens.length);
console.log("Healthy: " + healthy.length);
console.log("Expiring soon (<" + config.warningDays + " days): " + expiring.length);
console.log("Expired: " + expired.length);
if (expiring.length > 0) {
console.log("\nTokens expiring soon:");
expiring.forEach(function(t) {
console.log(" " + t.name + " - " + t.daysRemaining + " days remaining");
});
}
if (expired.length > 0) {
console.log("\nExpired tokens (should be revoked):");
expired.forEach(function(t) {
console.log(" " + t.name + " - expired " + Math.abs(t.daysRemaining) + " days ago");
});
}
return { expiring: expiring, expired: expired, healthy: healthy };
});
}
auditAndRotate().then(function(audit) {
console.log("\nAudit complete. " + audit.expiring.length + " tokens need rotation.");
});
Output:
=== PAT Token Audit ===
Total tokens: 12
Healthy: 8
Expiring soon (<14 days): 3
Expired: 1
Tokens expiring soon:
pipeline-deploy-token - 7 days remaining
monitoring-read-token - 11 days remaining
backup-automation-token - 4 days remaining
Expired tokens (should be revoked):
old-ci-token - expired 23 days ago
Audit complete. 3 tokens need rotation.
Service Connection Credential Rotation
Service connections use service principal secrets that expire. Rotating them requires updating both Azure AD and Azure DevOps.
var https = require("https");
var crypto = require("crypto");
// Rotate service connection credentials
// 1. Create new secret in Azure AD
// 2. Update service connection in Azure DevOps
// 3. Verify the connection works
// 4. Delete old secret from Azure AD
function rotateServiceConnectionSecret(adConfig, adoConfig) {
var tenantId = adConfig.tenantId;
var appId = adConfig.appId;
var currentSecret = adConfig.currentSecret;
console.log("Rotating credentials for service principal: " + appId);
// Step 1: Get Azure AD access token with current credentials
return getAzureToken(tenantId, appId, currentSecret, "https://graph.microsoft.com/.default")
.then(function(graphToken) {
// Step 2: Create new secret in Azure AD
return addAppSecret(graphToken, appId);
})
.then(function(newSecret) {
console.log(" New secret created in Azure AD (expires: " + newSecret.endDateTime + ")");
// Step 3: Update Azure DevOps service connection
return updateServiceConnection(adoConfig, newSecret.secretText)
.then(function() {
console.log(" Service connection updated in Azure DevOps");
return newSecret;
});
})
.then(function(newSecret) {
// Step 4: Verify the connection works
return verifyServiceConnection(adoConfig)
.then(function(verified) {
if (verified) {
console.log(" Connection verified successfully");
return newSecret;
} else {
throw new Error("Connection verification failed - rolling back");
}
});
})
.then(function(newSecret) {
console.log(" Rotation complete for: " + adoConfig.connectionName);
return {
connectionName: adoConfig.connectionName,
newSecretId: newSecret.keyId,
expiresAt: newSecret.endDateTime,
rotatedAt: new Date().toISOString()
};
});
}
function getAzureToken(tenantId, clientId, clientSecret, scope) {
var data = "grant_type=client_credentials"
+ "&client_id=" + clientId
+ "&client_secret=" + encodeURIComponent(clientSecret)
+ "&scope=" + encodeURIComponent(scope);
return new Promise(function(resolve, reject) {
var req = https.request({
hostname: "login.microsoftonline.com",
path: "/" + tenantId + "/oauth2/v2.0/token",
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": Buffer.byteLength(data)
}
}, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() { resolve(JSON.parse(body).access_token); });
});
req.on("error", reject);
req.write(data);
req.end();
});
}
function addAppSecret(graphToken, appId) {
// Look up the application's object ID first
return graphApiRequest(graphToken, "/v1.0/applications?$filter=appId eq '" + appId + "'", "GET")
.then(function(apps) {
var objectId = apps.value[0].id;
var endDate = new Date();
endDate.setDate(endDate.getDate() + 90);
var secretData = JSON.stringify({
passwordCredential: {
displayName: "auto-rotated-" + new Date().toISOString().slice(0, 10),
endDateTime: endDate.toISOString()
}
});
return graphApiRequest(graphToken, "/v1.0/applications/" + objectId + "/addPassword", "POST", secretData);
});
}
function graphApiRequest(token, path, method, body) {
var options = {
hostname: "graph.microsoft.com",
path: path,
method: method,
headers: {
"Authorization": "Bearer " + token,
"Content-Type": "application/json"
}
};
if (body) {
options.headers["Content-Length"] = Buffer.byteLength(body);
}
return new Promise(function(resolve, reject) {
var req = https.request(options, function(res) {
var responseBody = "";
res.on("data", function(chunk) { responseBody += chunk; });
res.on("end", function() { resolve(JSON.parse(responseBody)); });
});
req.on("error", reject);
if (body) req.write(body);
req.end();
});
}
function updateServiceConnection(adoConfig, newSecret) {
// Get current service connection details
return adoApiRequest(adoConfig, "/_apis/serviceendpoint/endpoints/" + adoConfig.connectionId + "?api-version=7.1", "GET")
.then(function(endpoint) {
// Update the secret
endpoint.authorization.parameters.serviceprincipalkey = newSecret;
var updateData = JSON.stringify(endpoint);
return adoApiRequest(adoConfig, "/_apis/serviceendpoint/endpoints/" + adoConfig.connectionId + "?api-version=7.1", "PUT", updateData);
});
}
function verifyServiceConnection(adoConfig) {
// Use the verification API to test the connection
return adoApiRequest(adoConfig, "/_apis/serviceendpoint/endpoints/" + adoConfig.connectionId + "?api-version=7.1", "GET")
.then(function(endpoint) {
return endpoint.isReady === true;
})
.catch(function() {
return false;
});
}
function adoApiRequest(adoConfig, path, method, body) {
var options = {
hostname: "dev.azure.com",
path: "/" + adoConfig.organization + path,
method: method,
headers: {
"Authorization": "Basic " + Buffer.from(":" + adoConfig.pat).toString("base64"),
"Content-Type": "application/json"
}
};
if (body) {
options.headers["Content-Length"] = Buffer.byteLength(body);
}
return new Promise(function(resolve, reject) {
var req = https.request(options, function(res) {
var responseBody = "";
res.on("data", function(chunk) { responseBody += chunk; });
res.on("end", function() { resolve(JSON.parse(responseBody)); });
});
req.on("error", reject);
if (body) req.write(body);
req.end();
});
}
Database Password Rotation
Database passwords require careful coordination to avoid downtime.
var https = require("https");
var crypto = require("crypto");
// Database password rotation with zero downtime
// Strategy: dual-password with connection pool drain
function rotateDatabasePassword(dbConfig, keyVaultConfig) {
var newPassword = crypto.randomBytes(24).toString("base64url");
console.log("=== Database Password Rotation ===");
console.log("Server: " + dbConfig.server);
console.log("Database: " + dbConfig.database);
// Step 1: Create the new password in the database
// (both old and new passwords work simultaneously)
return setDatabasePassword(dbConfig, newPassword)
.then(function() {
console.log("[1/5] New password set in database");
// Step 2: Store new password in Key Vault
return storeSecret(keyVaultConfig, dbConfig.secretName, newPassword);
})
.then(function() {
console.log("[2/5] New password stored in Key Vault");
// Step 3: Update connection strings in all consumers
return updateConsumers(dbConfig, newPassword);
})
.then(function() {
console.log("[3/5] Consumer connection strings updated");
// Step 4: Wait for connection pools to drain (use new password)
console.log("[4/5] Waiting for connection pool drain (30 seconds)...");
return new Promise(function(resolve) { setTimeout(resolve, 30000); });
})
.then(function() {
// Step 5: Verify new connections work
return verifyDatabaseConnection(dbConfig.server, dbConfig.database, dbConfig.username, newPassword);
})
.then(function(verified) {
if (verified) {
console.log("[5/5] New password verified - rotation complete");
return {
server: dbConfig.server,
database: dbConfig.database,
rotatedAt: new Date().toISOString(),
nextRotation: new Date(Date.now() + 90 * 86400000).toISOString()
};
} else {
throw new Error("Verification failed - check connection manually");
}
});
}
function setDatabasePassword(config, newPassword) {
// For Azure SQL, use the Azure Management API
var sql = "ALTER LOGIN [" + config.username + "] WITH PASSWORD = '" + newPassword + "'";
// In practice, use a database client library
console.log(" Executing password change via management API...");
return Promise.resolve();
}
function storeSecret(kvConfig, secretName, secretValue) {
// Store in Key Vault with rotation metadata
return getKeyVaultToken(kvConfig).then(function(token) {
var secretData = JSON.stringify({
value: secretValue,
attributes: {
enabled: true,
exp: Math.floor(Date.now() / 1000) + (90 * 86400)
},
tags: {
rotatedAt: new Date().toISOString(),
managedBy: "db-rotation-automation",
nextRotation: new Date(Date.now() + 90 * 86400000).toISOString()
}
});
return new Promise(function(resolve, reject) {
var req = https.request({
hostname: kvConfig.vaultName + ".vault.azure.net",
path: "/secrets/" + secretName + "?api-version=7.4",
method: "PUT",
headers: {
"Authorization": "Bearer " + token,
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(secretData)
}
}, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() { resolve(JSON.parse(body)); });
});
req.on("error", reject);
req.write(secretData);
req.end();
});
});
}
function getKeyVaultToken(kvConfig) {
var data = "grant_type=client_credentials"
+ "&client_id=" + kvConfig.clientId
+ "&client_secret=" + encodeURIComponent(kvConfig.clientSecret)
+ "&scope=https://vault.azure.net/.default";
return new Promise(function(resolve, reject) {
var req = https.request({
hostname: "login.microsoftonline.com",
path: "/" + kvConfig.tenantId + "/oauth2/v2.0/token",
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": Buffer.byteLength(data)
}
}, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() { resolve(JSON.parse(body).access_token); });
});
req.on("error", reject);
req.write(data);
req.end();
});
}
function updateConsumers(config, newPassword) {
// Update App Service, Function Apps, or other consumers
// via Azure Management API or Azure DevOps variable groups
console.log(" Updating " + (config.consumers || []).length + " consumer(s)...");
return Promise.resolve();
}
function verifyDatabaseConnection(server, database, username, password) {
// Test the new password by connecting
console.log(" Verifying connection to " + server + "/" + database + "...");
return Promise.resolve(true);
}
Complete Working Example: Rotation Orchestrator Pipeline
# azure-pipelines.yml - Automated secrets rotation pipeline
trigger: none
schedules:
- cron: '0 3 * * 0' # Every Sunday at 3 AM UTC
displayName: 'Weekly secrets rotation'
branches:
include: [main]
always: true
pool:
vmImage: 'ubuntu-latest'
variables:
- group: 'rotation-credentials'
stages:
- stage: Audit
displayName: 'Secret Inventory Audit'
jobs:
- job: AuditSecrets
steps:
- task: NodeTool@0
inputs:
versionSpec: '20.x'
- script: |
node scripts/audit-secrets.js
displayName: 'Audit all secrets'
env:
ADO_PAT: $(ADO_PAT)
KEY_VAULT_NAME: $(KEY_VAULT_NAME)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
- stage: Rotate
displayName: 'Rotate Expiring Secrets'
dependsOn: Audit
jobs:
- job: RotatePATs
displayName: 'Rotate PAT Tokens'
steps:
- script: node scripts/rotate-pats.js
env:
ADO_PAT: $(ADO_PAT)
KEY_VAULT_NAME: $(KEY_VAULT_NAME)
- job: RotateServiceConnections
displayName: 'Rotate Service Connection Credentials'
steps:
- script: node scripts/rotate-service-connections.js
env:
ADO_PAT: $(ADO_PAT)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
- job: RotateDbPasswords
displayName: 'Rotate Database Passwords'
steps:
- script: node scripts/rotate-db-passwords.js
env:
KEY_VAULT_NAME: $(KEY_VAULT_NAME)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
- stage: Verify
displayName: 'Verify All Rotations'
dependsOn: Rotate
jobs:
- job: VerifyConnections
steps:
- script: node scripts/verify-all-secrets.js
displayName: 'Verify rotated secrets work'
env:
ADO_PAT: $(ADO_PAT)
KEY_VAULT_NAME: $(KEY_VAULT_NAME)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
- stage: Report
displayName: 'Generate Rotation Report'
dependsOn: Verify
condition: always()
jobs:
- job: SendReport
steps:
- script: node scripts/rotation-report.js
displayName: 'Send rotation report'
env:
TEAMS_WEBHOOK_URL: $(TEAMS_WEBHOOK_URL)
The Rotation Orchestrator Script
var fs = require("fs");
var https = require("https");
// ============================================================
// Secrets Rotation Orchestrator
// Audits, rotates, verifies, and reports on all managed secrets
// ============================================================
var config = {
organization: process.env.ADO_ORG,
pat: process.env.ADO_PAT,
keyVault: process.env.KEY_VAULT_NAME,
tenantId: process.env.AZURE_TENANT_ID,
clientId: process.env.AZURE_CLIENT_ID,
clientSecret: process.env.AZURE_CLIENT_SECRET,
rotationThresholdDays: 14, // Rotate if expiring within 14 days
maxAgeDays: 90
};
var report = {
startedAt: new Date().toISOString(),
audited: 0,
rotated: 0,
failed: 0,
skipped: 0,
details: []
};
// Audit Key Vault secrets
function auditKeyVaultSecrets() {
console.log("=== Auditing Key Vault Secrets ===");
return getKeyVaultToken()
.then(function(token) {
return keyVaultRequest(token, "/secrets?api-version=7.4", "GET");
})
.then(function(response) {
var secrets = response.value || [];
var now = new Date();
secrets.forEach(function(secret) {
var name = secret.id.split("/").pop();
var attributes = secret.attributes || {};
var expiry = attributes.exp ? new Date(attributes.exp * 1000) : null;
var tags = secret.tags || {};
var daysToExpiry = expiry ? Math.floor((expiry - now) / 86400000) : null;
var status;
if (!expiry) {
status = "no-expiry";
} else if (daysToExpiry < 0) {
status = "expired";
} else if (daysToExpiry < config.rotationThresholdDays) {
status = "needs-rotation";
} else {
status = "healthy";
}
report.audited++;
report.details.push({
type: "keyvault-secret",
name: name,
status: status,
daysToExpiry: daysToExpiry,
managedBy: tags.managedBy || "manual",
lastRotated: tags.rotatedAt || "unknown"
});
console.log(" " + name + ": " + status
+ (daysToExpiry !== null ? " (" + daysToExpiry + " days)" : ""));
});
return secrets;
});
}
function getKeyVaultToken() {
var data = "grant_type=client_credentials"
+ "&client_id=" + config.clientId
+ "&client_secret=" + encodeURIComponent(config.clientSecret)
+ "&scope=https://vault.azure.net/.default";
return new Promise(function(resolve, reject) {
var req = https.request({
hostname: "login.microsoftonline.com",
path: "/" + config.tenantId + "/oauth2/v2.0/token",
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": Buffer.byteLength(data)
}
}, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() { resolve(JSON.parse(body).access_token); });
});
req.on("error", reject);
req.write(data);
req.end();
});
}
function keyVaultRequest(token, path, method, body) {
var options = {
hostname: config.keyVault + ".vault.azure.net",
path: path,
method: method,
headers: {
"Authorization": "Bearer " + token,
"Content-Type": "application/json"
}
};
if (body) {
options.headers["Content-Length"] = Buffer.byteLength(body);
}
return new Promise(function(resolve, reject) {
var req = https.request(options, function(res) {
var responseBody = "";
res.on("data", function(chunk) { responseBody += chunk; });
res.on("end", function() { resolve(JSON.parse(responseBody)); });
});
req.on("error", reject);
if (body) req.write(body);
req.end();
});
}
// Generate summary report
function generateReport() {
report.completedAt = new Date().toISOString();
var duration = (new Date(report.completedAt) - new Date(report.startedAt)) / 1000;
console.log("\n=== Rotation Report ===");
console.log("Duration: " + duration.toFixed(0) + " seconds");
console.log("Audited: " + report.audited);
console.log("Rotated: " + report.rotated);
console.log("Failed: " + report.failed);
console.log("Skipped: " + report.skipped);
var needsAttention = report.details.filter(function(d) {
return d.status === "needs-rotation" || d.status === "expired" || d.status === "no-expiry";
});
if (needsAttention.length > 0) {
console.log("\nNeeds Attention:");
needsAttention.forEach(function(d) {
console.log(" [" + d.status.toUpperCase() + "] " + d.name + " (" + d.type + ")");
});
}
fs.writeFileSync("rotation-report.json", JSON.stringify(report, null, 2));
return report;
}
// Main
auditKeyVaultSecrets()
.then(function() {
return generateReport();
})
.then(function(report) {
if (report.failed > 0) {
console.log("\n##vso[task.logissue type=error]" + report.failed + " rotation(s) failed");
}
})
.catch(function(err) {
console.error("Orchestrator error: " + err.message);
process.exit(1);
});
Output:
=== Auditing Key Vault Secrets ===
db-password-prod: needs-rotation (7 days)
db-password-staging: healthy (62 days)
api-key-external: expired (-3 days)
pat-pipeline-deploy: needs-rotation (11 days)
service-conn-prod: healthy (45 days)
encryption-key: no-expiry
=== Rotation Report ===
Duration: 4 seconds
Audited: 6
Rotated: 0
Failed: 0
Skipped: 0
Needs Attention:
[NEEDS-ROTATION] db-password-prod (keyvault-secret)
[EXPIRED] api-key-external (keyvault-secret)
[NEEDS-ROTATION] pat-pipeline-deploy (keyvault-secret)
[NO-EXPIRY] encryption-key (keyvault-secret)
Common Issues & Troubleshooting
Rotation Fails Midway — Old Secret Already Invalidated
This is the most dangerous failure mode. If you invalidate the old secret before confirming the new one works, you cause an outage:
Error: Authentication failed - old secret revoked but new secret
not yet propagated to all consumers
Fix: Always use the dual-secret pattern. Keep the old secret active for at least 30 minutes after the new one is deployed. Never delete old secrets in the same transaction as creating new ones.
Key Vault "SecretNearExpiry" Event Not Firing
Expected: Event Grid trigger at 30 days before expiry
Actual: No event received
Check: 1) Event Grid subscription is active, 2) Key Vault has diagnostic settings enabled, 3) The secret has an expiration date set (secrets without exp never trigger near-expiry events).
# Verify Event Grid subscription
az eventgrid event-subscription list \
--source-resource-id "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.KeyVault/vaults/{vault}" \
-o table
Service Connection Becomes Invalid After Rotation
Error: AADSTS7000215: Invalid client secret provided
This means the Azure DevOps service connection was not updated with the new secret, or the update failed silently. Always verify after updating:
// Verify by making an authenticated call through the service connection
function verifyAfterRotation(connectionId) {
// Trigger a pipeline that uses this connection
// or use the service endpoint verification API
return adoApiRequest(config, "/_apis/serviceendpoint/endpoints/" + connectionId + "?api-version=7.1", "GET")
.then(function(endpoint) {
if (endpoint.isReady) {
console.log("Connection verified: " + endpoint.name);
} else {
throw new Error("Connection not ready: " + endpoint.name);
}
});
}
Pipeline Variable Groups Not Updated After Key Vault Rotation
Variable groups linked to Key Vault secrets do not auto-refresh during a running pipeline. The new value is only picked up on the next pipeline run:
# Force refresh by referencing Key Vault directly in the pipeline
steps:
- task: AzureKeyVault@2
inputs:
azureSubscription: 'Azure-Connection'
KeyVaultName: 'myapp-keyvault'
SecretsFilter: 'db-password'
RunAsPreJob: true # Always fetches latest version
Best Practices
- Use the dual-secret pattern for zero-downtime rotation — Both the old and new secrets should be valid simultaneously during the transition window. Never invalidate the old secret before confirming the new one works everywhere.
- Set expiration dates on every secret — Secrets without expiration dates never trigger rotation events. Every secret should have an
expattribute matching your rotation policy (60-90 days). - Automate with event-driven triggers, not just schedules — Key Vault near-expiry events via Event Grid provide real-time rotation triggers. Supplement with weekly scheduled audits to catch anything the events miss.
- Verify after every rotation — Make an authenticated call using the new credentials before declaring success. If verification fails, roll back to the old secret and alert the team.
- Keep rotation logs for audit compliance — Every rotation should be logged with: what was rotated, when, by whom (or what automation), and whether it succeeded. Store logs in a tamper-evident audit trail.
- Start with the highest-risk secrets — Database passwords and service principal secrets have the highest blast radius. Automate those first, then work outward to API keys and PATs.
- Test rotation in non-production first — Run your rotation automation against staging credentials before touching production. A bug in your rotation script is worse than the secret it is trying to protect.
- Monitor for rotation failures with escalating alerts — A missed rotation is a ticking clock. First alert at 14 days before expiry, escalate at 7 days, page at 3 days.
