Datadog Integration with Azure DevOps
A comprehensive guide to integrating Datadog with Azure DevOps for deployment tracking, pipeline monitoring, CI visibility, deployment gates based on Datadog monitors, and correlating release events with production metrics and APM data.
Datadog Integration with Azure DevOps
Overview
Datadog is the monitoring platform I see most often alongside Azure DevOps in organizations that are not all-in on the Azure ecosystem. Teams use Azure DevOps for CI/CD but Datadog for infrastructure monitoring, APM, and log management because Datadog's multi-cloud support and visualization capabilities are hard to beat. The integration between the two turns your deployment pipeline into a feedback loop — Datadog tracks when deployments happen, monitors their impact on production, and can even gate deployments based on real-time health metrics. This article covers setting up that integration end to end.
Prerequisites
- Datadog account (any tier — some features require Pro or Enterprise)
- Azure DevOps organization with Pipelines enabled
- Datadog API key and Application key (from Organization Settings > API Keys / Application Keys)
- Node.js 16 or later for scripting examples
- At least one service instrumented with Datadog APM or infrastructure monitoring
- Familiarity with Datadog monitors, dashboards, and event API
Deployment Tracking with Datadog Events
The simplest integration sends deployment events to Datadog. These appear as overlays on dashboards and in the event stream, letting you correlate deployments with metric changes.
Sending Events from Pipeline Steps
# azure-pipelines.yml
steps:
- script: |
curl -s -X POST "https://api.datadoghq.com/api/v1/events" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: $(DATADOG_API_KEY)" \
-d '{
"title": "Deployment: $(Build.DefinitionName) #$(Build.BuildNumber)",
"text": "Deployed build $(Build.BuildNumber) to production from branch $(Build.SourceBranchName). Triggered by $(Build.RequestedFor).",
"priority": "normal",
"tags": [
"environment:production",
"service:$(Build.Repository.Name)",
"build_number:$(Build.BuildNumber)",
"branch:$(Build.SourceBranchName)",
"source:azure-devops"
],
"alert_type": "info",
"source_type_name": "azure_devops"
}'
displayName: "Send deployment event to Datadog"
env:
DATADOG_API_KEY: $(DatadogApiKey)
Deployment Event Helper Script
For richer deployment events with timing, change tracking, and links back to Azure DevOps:
// scripts/datadog-deployment-event.js
var https = require("https");
var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";
var BUILD_NUMBER = process.env.BUILD_BUILDNUMBER || "local";
var BUILD_URL = (process.env.SYSTEM_TEAMFOUNDATIONCOLLECTIONURI || "") +
(process.env.SYSTEM_TEAMPROJECT || "") +
"/_build/results?buildId=" + (process.env.BUILD_BUILDID || "0");
var DEFINITION_NAME = process.env.BUILD_DEFINITIONNAME || "unknown";
var BRANCH = (process.env.BUILD_SOURCEBRANCH || "").replace("refs/heads/", "");
var REQUESTED_BY = process.env.BUILD_REQUESTEDFOR || "manual";
var REPO_NAME = process.env.BUILD_REPOSITORY_NAME || "unknown";
var COMMIT_SHA = process.env.BUILD_SOURCEVERSION || "unknown";
var ENVIRONMENT = process.env.DEPLOY_ENVIRONMENT || "production";
var SERVICE_NAME = process.env.DD_SERVICE || REPO_NAME;
function sendEvent(event, callback) {
var body = JSON.stringify(event);
var options = {
hostname: "api." + DD_SITE,
path: "/api/v1/events",
method: "POST",
headers: {
"Content-Type": "application/json",
"DD-API-KEY": DD_API_KEY,
"Content-Length": Buffer.byteLength(body)
}
};
var req = https.request(options, function (res) {
var data = "";
res.on("data", function (chunk) { data += chunk; });
res.on("end", function () {
if (res.statusCode >= 200 && res.statusCode < 300) {
var parsed = JSON.parse(data);
callback(null, parsed);
} else {
callback(new Error("Datadog API error " + res.statusCode + ": " + data));
}
});
});
req.on("error", callback);
req.write(body);
req.end();
}
var eventText = "%%%\n" +
"### Deployment Details\n\n" +
"| Field | Value |\n" +
"|-------|-------|\n" +
"| **Service** | " + SERVICE_NAME + " |\n" +
"| **Environment** | " + ENVIRONMENT + " |\n" +
"| **Build** | [#" + BUILD_NUMBER + "](" + BUILD_URL + ") |\n" +
"| **Branch** | `" + BRANCH + "` |\n" +
"| **Commit** | `" + COMMIT_SHA.substring(0, 8) + "` |\n" +
"| **Triggered by** | " + REQUESTED_BY + " |\n" +
"| **Pipeline** | " + DEFINITION_NAME + " |\n" +
"\n%%%";
var event = {
title: "Deployed " + SERVICE_NAME + " #" + BUILD_NUMBER + " to " + ENVIRONMENT,
text: eventText,
priority: "normal",
tags: [
"environment:" + ENVIRONMENT,
"service:" + SERVICE_NAME,
"build_number:" + BUILD_NUMBER,
"branch:" + BRANCH,
"source:azure-devops",
"pipeline:" + DEFINITION_NAME
],
alert_type: "info",
source_type_name: "azure_devops",
aggregation_key: SERVICE_NAME + "-deploy"
};
sendEvent(event, function (err, result) {
if (err) {
console.error("Failed to send Datadog event: " + err.message);
process.exit(0); // Don't fail the pipeline over a notification
}
console.log("Datadog deployment event sent (ID: " + result.event.id + ")");
console.log("URL: " + result.event.url);
});
Datadog CI Visibility
Datadog CI Visibility provides native pipeline monitoring with trace-level detail for every pipeline run. It shows stage durations, failure patterns, flaky tests, and resource utilization.
Enabling CI Visibility for Azure Pipelines
Install the Datadog CI Azure DevOps extension from the Marketplace. Then add the Datadog task to your pipeline:
# azure-pipelines-dd-ci.yml
trigger:
branches:
include:
- main
pool:
vmImage: "ubuntu-latest"
steps:
- task: DatadogCISetup@1
displayName: "Setup Datadog CI"
inputs:
apiKey: $(DATADOG_API_KEY)
site: "datadoghq.com"
- script: npm ci
displayName: "Install dependencies"
- script: npm test
displayName: "Run tests"
env:
DD_CIVISIBILITY_AGENTLESS_ENABLED: "true"
DD_API_KEY: $(DATADOG_API_KEY)
DD_SITE: "datadoghq.com"
DD_SERVICE: "my-app"
DD_ENV: "ci"
- script: npm run build
displayName: "Build"
With CI Visibility enabled, every pipeline execution appears in Datadog's CI section with:
- Pipeline execution timeline with stage and step durations
- Test results with pass/fail/skip counts
- Flaky test detection across multiple runs
- Custom metrics and spans from instrumented tests
Custom Pipeline Metrics
Send custom metrics from your pipeline to track build-specific data:
// scripts/dd-pipeline-metrics.js
var https = require("https");
var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";
function sendMetric(metricName, value, tags, callback) {
var now = Math.floor(Date.now() / 1000);
var body = JSON.stringify({
series: [{
metric: metricName,
type: 0, // gauge
points: [[now, value]],
tags: tags
}]
});
var options = {
hostname: "api." + DD_SITE,
path: "/api/v1/series",
method: "POST",
headers: {
"Content-Type": "application/json",
"DD-API-KEY": DD_API_KEY,
"Content-Length": Buffer.byteLength(body)
}
};
var req = https.request(options, function (res) {
var data = "";
res.on("data", function (chunk) { data += chunk; });
res.on("end", function () {
callback(null, res.statusCode);
});
});
req.on("error", callback);
req.write(body);
req.end();
}
var SERVICE = process.env.DD_SERVICE || process.env.BUILD_REPOSITORY_NAME || "unknown";
var PIPELINE = process.env.BUILD_DEFINITIONNAME || "unknown";
var BRANCH = (process.env.BUILD_SOURCEBRANCH || "").replace("refs/heads/", "");
var RESULT = process.env.AGENT_JOBSTATUS || "Unknown";
var tags = [
"service:" + SERVICE,
"pipeline:" + PIPELINE,
"branch:" + BRANCH,
"result:" + RESULT.toLowerCase(),
"source:azure-devops"
];
// Calculate build duration
var startTime = new Date(process.env.SYSTEM_PIPELINESTARTTIME || Date.now());
var durationSec = Math.round((Date.now() - startTime.getTime()) / 1000);
var pending = 2;
function done() { pending--; if (pending === 0) { console.log("All metrics sent."); } }
sendMetric("ci.pipeline.duration", durationSec, tags, function (err) {
if (err) { console.error("Duration metric error: " + err.message); }
else { console.log("Sent pipeline duration: " + durationSec + "s"); }
done();
});
sendMetric("ci.pipeline.completed", 1, tags, function (err) {
if (err) { console.error("Completed metric error: " + err.message); }
else { console.log("Sent pipeline completion metric"); }
done();
});
Deployment Gates with Datadog Monitors
Use Datadog monitors as deployment gates — the pipeline checks a Datadog monitor before allowing deployment to proceed. If the monitor is in alert state, the deployment is blocked.
Checking Monitor Status from Pipeline
// scripts/dd-gate-check.js
var https = require("https");
var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_APP_KEY = process.env.DATADOG_APP_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";
var MONITOR_IDS = (process.env.DD_MONITOR_IDS || "").split(",").filter(function (id) { return id.trim(); });
if (MONITOR_IDS.length === 0) {
console.error("No monitor IDs specified. Set DD_MONITOR_IDS environment variable.");
process.exit(1);
}
function getMonitorStatus(monitorId, callback) {
var options = {
hostname: "api." + DD_SITE,
path: "/api/v1/monitor/" + monitorId,
method: "GET",
headers: {
"DD-API-KEY": DD_API_KEY,
"DD-APPLICATION-KEY": DD_APP_KEY,
"Accept": "application/json"
}
};
var req = https.request(options, function (res) {
var data = "";
res.on("data", function (chunk) { data += chunk; });
res.on("end", function () {
if (res.statusCode === 200) {
callback(null, JSON.parse(data));
} else {
callback(new Error("Monitor API error: " + res.statusCode));
}
});
});
req.on("error", callback);
req.end();
}
console.log("Checking " + MONITOR_IDS.length + " Datadog monitor(s)...\n");
var completed = 0;
var allHealthy = true;
var results = [];
MONITOR_IDS.forEach(function (id) {
getMonitorStatus(id.trim(), function (err, monitor) {
if (err) {
console.error("Failed to check monitor " + id + ": " + err.message);
allHealthy = false;
results.push({ id: id, name: "ERROR", status: "error" });
} else {
var status = monitor.overall_state;
var healthy = status === "OK" || status === "No Data";
var icon = healthy ? "OK" : "ALERT";
console.log("[" + icon + "] Monitor #" + id + ": " + monitor.name);
console.log(" Status: " + status);
console.log(" Type: " + monitor.type);
if (!healthy) {
allHealthy = false;
console.log(" !! This monitor is blocking deployment");
}
console.log("");
results.push({ id: id, name: monitor.name, status: status });
}
completed++;
if (completed === MONITOR_IDS.length) {
console.log("=== Gate Result: " + (allHealthy ? "PASSED" : "BLOCKED") + " ===");
if (!allHealthy) {
var blocked = results.filter(function (r) { return r.status !== "OK" && r.status !== "No Data"; });
console.error("\nDeployment blocked by " + blocked.length + " monitor(s):");
blocked.forEach(function (r) {
console.error(" - #" + r.id + " " + r.name + " (" + r.status + ")");
});
process.exit(1);
}
}
});
});
Pipeline usage:
- script: node scripts/dd-gate-check.js
displayName: "Datadog deployment gate check"
env:
DATADOG_API_KEY: $(DatadogApiKey)
DATADOG_APP_KEY: $(DatadogAppKey)
DD_MONITOR_IDS: "12345678,23456789,34567890"
Datadog Service Catalog Integration
Datadog's Service Catalog can track deployment frequency and status per service. Register your services and their pipelines:
# datadog-service-catalog.yaml (checked into repo)
schema-version: v2
dd-service: my-api-service
team: backend-team
contacts:
- type: slack
contact: "#backend-alerts"
- type: email
contact: [email protected]
repos:
- url: https://dev.azure.com/your-org/your-project/_git/my-api
provider: azure-devops
ci-cd:
- ci_provider: azurepipelines
pipeline_id: "42"
pipeline_url: https://dev.azure.com/your-org/your-project/_build?definitionId=42
links:
- name: Runbook
type: runbook
url: https://wiki.company.com/runbooks/my-api
- name: Architecture Diagram
type: doc
url: https://wiki.company.com/architecture/my-api
tags:
- "env:production"
- "tier:1"
- "language:nodejs"
Push this to Datadog's service catalog API:
- script: |
curl -s -X POST "https://api.datadoghq.com/api/v2/services/definitions" \
-H "DD-API-KEY: $(DATADOG_API_KEY)" \
-H "DD-APPLICATION-KEY: $(DATADOG_APP_KEY)" \
-H "Content-Type: application/json" \
-d @datadog-service-catalog.yaml
displayName: "Update Datadog Service Catalog"
Complete Working Example: Full Observability Pipeline
This pipeline integrates Datadog at every stage — pre-deployment gate checks, deployment events, post-deployment monitoring, and automatic rollback on degraded metrics:
// scripts/dd-post-deploy-check.js
// Checks Datadog APM metrics after deployment to verify service health
var https = require("https");
var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_APP_KEY = process.env.DATADOG_APP_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";
var SERVICE = process.env.DD_SERVICE || "my-app";
var ENVIRONMENT = process.env.DD_ENV || "production";
function queryMetric(query, from, to, callback) {
var params = "query=" + encodeURIComponent(query) +
"&from=" + from + "&to=" + to;
var options = {
hostname: "api." + DD_SITE,
path: "/api/v1/query?" + params,
method: "GET",
headers: {
"DD-API-KEY": DD_API_KEY,
"DD-APPLICATION-KEY": DD_APP_KEY,
"Accept": "application/json"
}
};
var req = https.request(options, function (res) {
var data = "";
res.on("data", function (chunk) { data += chunk; });
res.on("end", function () {
if (res.statusCode === 200) {
callback(null, JSON.parse(data));
} else {
callback(new Error("Query failed: " + res.statusCode + " " + data));
}
});
});
req.on("error", callback);
req.end();
}
function getLatestValue(series) {
if (!series || !series.series || series.series.length === 0) { return null; }
var points = series.series[0].pointlist;
if (!points || points.length === 0) { return null; }
return points[points.length - 1][1];
}
var now = Math.floor(Date.now() / 1000);
var fiveMinAgo = now - 300;
console.log("Post-deployment health check for " + SERVICE + " (" + ENVIRONMENT + ")\n");
var checks = [
{
name: "Error Rate",
query: "sum:trace." + SERVICE + ".errors{env:" + ENVIRONMENT + "}.as_rate() / sum:trace." + SERVICE + ".hits{env:" + ENVIRONMENT + "}.as_rate() * 100",
threshold: 5,
unit: "%"
},
{
name: "P95 Latency",
query: "p95:trace." + SERVICE + ".request.duration{env:" + ENVIRONMENT + "}",
threshold: 2000,
unit: "ms"
},
{
name: "Request Rate",
query: "sum:trace." + SERVICE + ".hits{env:" + ENVIRONMENT + "}.as_rate()",
threshold: 0, // Alert if zero (service is down)
unit: "req/s",
alertBelow: true
}
];
var completed = 0;
var allPassed = true;
checks.forEach(function (check) {
queryMetric(check.query, fiveMinAgo, now, function (err, result) {
completed++;
if (err) {
console.error("[ERROR] " + check.name + ": " + err.message);
allPassed = false;
} else {
var value = getLatestValue(result);
if (value === null) {
console.log("[WARN] " + check.name + ": No data available");
} else {
var passed;
if (check.alertBelow) {
passed = value > check.threshold;
} else {
passed = value <= check.threshold;
}
var icon = passed ? "OK" : "FAIL";
var roundedValue = Math.round(value * 100) / 100;
console.log("[" + icon + "] " + check.name + ": " + roundedValue + check.unit +
" (threshold: " + (check.alertBelow ? ">" : "<=") + " " + check.threshold + check.unit + ")");
if (!passed) { allPassed = false; }
}
}
if (completed === checks.length) {
console.log("\n=== Post-Deploy Result: " + (allPassed ? "HEALTHY" : "DEGRADED") + " ===");
if (!allPassed) {
console.error("Service health degraded after deployment!");
process.exit(1);
}
}
});
});
Pipeline YAML using the full integration:
# azure-pipelines-datadog-full.yml
trigger:
branches:
include:
- main
pool:
vmImage: "ubuntu-latest"
variables:
- group: datadog-credentials
- name: serviceName
value: "my-api"
- name: deployEnv
value: "production"
stages:
- stage: Build
jobs:
- job: Build
steps:
- script: npm ci && npm test && npm run build
displayName: "Build and test"
- task: PublishPipelineArtifact@1
inputs:
targetPath: dist
artifact: app
- stage: PreDeployGate
displayName: "Pre-Deploy Gate (Datadog)"
dependsOn: Build
jobs:
- job: GateCheck
steps:
- script: node scripts/dd-gate-check.js
displayName: "Check Datadog monitors"
env:
DATADOG_API_KEY: $(DatadogApiKey)
DATADOG_APP_KEY: $(DatadogAppKey)
DD_MONITOR_IDS: $(GATE_MONITOR_IDS)
- stage: Deploy
dependsOn: PreDeployGate
jobs:
- deployment: Production
environment: production
strategy:
runOnce:
deploy:
steps:
- script: echo "Deploying $(serviceName)..."
displayName: "Deploy"
- script: node scripts/datadog-deployment-event.js
displayName: "Send Datadog deployment event"
env:
DATADOG_API_KEY: $(DatadogApiKey)
DD_SERVICE: $(serviceName)
DEPLOY_ENVIRONMENT: $(deployEnv)
- stage: PostDeployVerify
displayName: "Post-Deploy Verification"
dependsOn: Deploy
jobs:
- job: Verify
steps:
- script: sleep 120
displayName: "Stabilization period (2 min)"
- script: node scripts/dd-post-deploy-check.js
displayName: "Datadog health check"
env:
DATADOG_API_KEY: $(DatadogApiKey)
DATADOG_APP_KEY: $(DatadogAppKey)
DD_SERVICE: $(serviceName)
DD_ENV: $(deployEnv)
- script: node scripts/dd-pipeline-metrics.js
displayName: "Send pipeline metrics"
condition: always()
env:
DATADOG_API_KEY: $(DatadogApiKey)
DD_SERVICE: $(serviceName)
Common Issues and Troubleshooting
Datadog events not appearing in the event stream
HTTP 403: Forbidden
Verify your API key is correct and has not been revoked. Datadog API keys are organization-wide — if your org has multiple Datadog accounts, ensure the key matches the account where you expect events. Also check the DD_SITE value: US customers use datadoghq.com, EU customers use datadoghq.eu, and other regions have different sites.
CI Visibility pipeline traces are missing or incomplete
The Datadog CI extension must be set up before any build steps run. If you add the DatadogCISetup task after other tasks, those earlier tasks are not traced. Also verify the pipeline agent has outbound HTTPS access to api.datadoghq.com — corporate firewalls may block it.
Monitor gate check returns "No Data" status
A monitor in "No Data" state means Datadog has not received metrics for the monitor's metric query within the evaluation window. This often happens for new monitors or after a metric name change. Decide whether "No Data" should block deployments (conservative) or allow them (permissive). The gate check script above treats "No Data" as passing — adjust if your risk tolerance differs.
Deployment events have wrong timestamps
Datadog events use the server-side receive time if no timestamp is specified in the payload. If your pipeline agent's clock is skewed or you are sending events from a post-deployment step that runs much later, the event may not align with the actual deployment time. Always include an explicit timestamp in the event payload using the pipeline's start time variable.
Rate limiting when sending many metrics
HTTP 429: Too Many Requests
Datadog's metric submission API accepts up to 500 series per request and has per-minute rate limits based on your plan. Batch metrics into a single API call rather than sending one metric per request. For CI pipelines running hundreds of jobs concurrently, consider using the Datadog Agent running on the build agents instead of direct API calls.
Best Practices
Send deployment events for every environment, not just production. Tracking staging and dev deployments helps identify issues before they reach production. Tag events with
environment:stagingto differentiate them on dashboards.Use Datadog monitors as deployment gates for critical services. Define monitors for error rate, latency, and request volume. If any are in alert state, block the deployment. This prevents deploying into an already degraded environment.
Include Azure DevOps build URLs in Datadog events. When investigating an issue in Datadog, a direct link back to the pipeline build log saves time. Include the build URL as a link in the event markdown text.
Create a Datadog dashboard for CI/CD metrics. Combine pipeline duration trends, deployment frequency, failure rates, and post-deployment health into a single dashboard. This gives engineering leadership visibility into delivery performance.
Tag all Datadog submissions consistently. Use the same
service,environment, andteamtags across events, metrics, and APM traces. Consistent tagging enables cross-referencing between CI data and production monitoring.Set up Datadog monitors that trigger on deployment-correlated degradation. Use anomaly detection monitors that alert when error rates or latency deviate from the baseline within minutes of a deployment event. This automates the "was it the deploy?" question.
Store Datadog API keys as secret pipeline variables. Never hard-code keys in YAML or scripts. Use Azure DevOps variable groups with secret variables, and reference them through
$(DatadogApiKey)in pipelines.