Integrations

Datadog Integration with Azure DevOps

A comprehensive guide to integrating Datadog with Azure DevOps for deployment tracking, pipeline monitoring, CI visibility, deployment gates based on Datadog monitors, and correlating release events with production metrics and APM data.

Datadog Integration with Azure DevOps

Overview

Datadog is the monitoring platform I see most often alongside Azure DevOps in organizations that are not all-in on the Azure ecosystem. Teams use Azure DevOps for CI/CD but Datadog for infrastructure monitoring, APM, and log management because Datadog's multi-cloud support and visualization capabilities are hard to beat. The integration between the two turns your deployment pipeline into a feedback loop — Datadog tracks when deployments happen, monitors their impact on production, and can even gate deployments based on real-time health metrics. This article covers setting up that integration end to end.

Prerequisites

  • Datadog account (any tier — some features require Pro or Enterprise)
  • Azure DevOps organization with Pipelines enabled
  • Datadog API key and Application key (from Organization Settings > API Keys / Application Keys)
  • Node.js 16 or later for scripting examples
  • At least one service instrumented with Datadog APM or infrastructure monitoring
  • Familiarity with Datadog monitors, dashboards, and event API

Deployment Tracking with Datadog Events

The simplest integration sends deployment events to Datadog. These appear as overlays on dashboards and in the event stream, letting you correlate deployments with metric changes.

Sending Events from Pipeline Steps

# azure-pipelines.yml
steps:
  - script: |
      curl -s -X POST "https://api.datadoghq.com/api/v1/events" \
        -H "Content-Type: application/json" \
        -H "DD-API-KEY: $(DATADOG_API_KEY)" \
        -d '{
          "title": "Deployment: $(Build.DefinitionName) #$(Build.BuildNumber)",
          "text": "Deployed build $(Build.BuildNumber) to production from branch $(Build.SourceBranchName). Triggered by $(Build.RequestedFor).",
          "priority": "normal",
          "tags": [
            "environment:production",
            "service:$(Build.Repository.Name)",
            "build_number:$(Build.BuildNumber)",
            "branch:$(Build.SourceBranchName)",
            "source:azure-devops"
          ],
          "alert_type": "info",
          "source_type_name": "azure_devops"
        }'
    displayName: "Send deployment event to Datadog"
    env:
      DATADOG_API_KEY: $(DatadogApiKey)

Deployment Event Helper Script

For richer deployment events with timing, change tracking, and links back to Azure DevOps:

// scripts/datadog-deployment-event.js
var https = require("https");

var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";

var BUILD_NUMBER = process.env.BUILD_BUILDNUMBER || "local";
var BUILD_URL = (process.env.SYSTEM_TEAMFOUNDATIONCOLLECTIONURI || "") +
    (process.env.SYSTEM_TEAMPROJECT || "") +
    "/_build/results?buildId=" + (process.env.BUILD_BUILDID || "0");
var DEFINITION_NAME = process.env.BUILD_DEFINITIONNAME || "unknown";
var BRANCH = (process.env.BUILD_SOURCEBRANCH || "").replace("refs/heads/", "");
var REQUESTED_BY = process.env.BUILD_REQUESTEDFOR || "manual";
var REPO_NAME = process.env.BUILD_REPOSITORY_NAME || "unknown";
var COMMIT_SHA = process.env.BUILD_SOURCEVERSION || "unknown";
var ENVIRONMENT = process.env.DEPLOY_ENVIRONMENT || "production";
var SERVICE_NAME = process.env.DD_SERVICE || REPO_NAME;

function sendEvent(event, callback) {
    var body = JSON.stringify(event);

    var options = {
        hostname: "api." + DD_SITE,
        path: "/api/v1/events",
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "DD-API-KEY": DD_API_KEY,
            "Content-Length": Buffer.byteLength(body)
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            if (res.statusCode >= 200 && res.statusCode < 300) {
                var parsed = JSON.parse(data);
                callback(null, parsed);
            } else {
                callback(new Error("Datadog API error " + res.statusCode + ": " + data));
            }
        });
    });

    req.on("error", callback);
    req.write(body);
    req.end();
}

var eventText = "%%%\n" +
    "### Deployment Details\n\n" +
    "| Field | Value |\n" +
    "|-------|-------|\n" +
    "| **Service** | " + SERVICE_NAME + " |\n" +
    "| **Environment** | " + ENVIRONMENT + " |\n" +
    "| **Build** | [#" + BUILD_NUMBER + "](" + BUILD_URL + ") |\n" +
    "| **Branch** | `" + BRANCH + "` |\n" +
    "| **Commit** | `" + COMMIT_SHA.substring(0, 8) + "` |\n" +
    "| **Triggered by** | " + REQUESTED_BY + " |\n" +
    "| **Pipeline** | " + DEFINITION_NAME + " |\n" +
    "\n%%%";

var event = {
    title: "Deployed " + SERVICE_NAME + " #" + BUILD_NUMBER + " to " + ENVIRONMENT,
    text: eventText,
    priority: "normal",
    tags: [
        "environment:" + ENVIRONMENT,
        "service:" + SERVICE_NAME,
        "build_number:" + BUILD_NUMBER,
        "branch:" + BRANCH,
        "source:azure-devops",
        "pipeline:" + DEFINITION_NAME
    ],
    alert_type: "info",
    source_type_name: "azure_devops",
    aggregation_key: SERVICE_NAME + "-deploy"
};

sendEvent(event, function (err, result) {
    if (err) {
        console.error("Failed to send Datadog event: " + err.message);
        process.exit(0); // Don't fail the pipeline over a notification
    }
    console.log("Datadog deployment event sent (ID: " + result.event.id + ")");
    console.log("URL: " + result.event.url);
});

Datadog CI Visibility

Datadog CI Visibility provides native pipeline monitoring with trace-level detail for every pipeline run. It shows stage durations, failure patterns, flaky tests, and resource utilization.

Enabling CI Visibility for Azure Pipelines

Install the Datadog CI Azure DevOps extension from the Marketplace. Then add the Datadog task to your pipeline:

# azure-pipelines-dd-ci.yml
trigger:
  branches:
    include:
      - main

pool:
  vmImage: "ubuntu-latest"

steps:
  - task: DatadogCISetup@1
    displayName: "Setup Datadog CI"
    inputs:
      apiKey: $(DATADOG_API_KEY)
      site: "datadoghq.com"

  - script: npm ci
    displayName: "Install dependencies"

  - script: npm test
    displayName: "Run tests"
    env:
      DD_CIVISIBILITY_AGENTLESS_ENABLED: "true"
      DD_API_KEY: $(DATADOG_API_KEY)
      DD_SITE: "datadoghq.com"
      DD_SERVICE: "my-app"
      DD_ENV: "ci"

  - script: npm run build
    displayName: "Build"

With CI Visibility enabled, every pipeline execution appears in Datadog's CI section with:

  • Pipeline execution timeline with stage and step durations
  • Test results with pass/fail/skip counts
  • Flaky test detection across multiple runs
  • Custom metrics and spans from instrumented tests

Custom Pipeline Metrics

Send custom metrics from your pipeline to track build-specific data:

// scripts/dd-pipeline-metrics.js
var https = require("https");

var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";

function sendMetric(metricName, value, tags, callback) {
    var now = Math.floor(Date.now() / 1000);

    var body = JSON.stringify({
        series: [{
            metric: metricName,
            type: 0, // gauge
            points: [[now, value]],
            tags: tags
        }]
    });

    var options = {
        hostname: "api." + DD_SITE,
        path: "/api/v1/series",
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "DD-API-KEY": DD_API_KEY,
            "Content-Length": Buffer.byteLength(body)
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            callback(null, res.statusCode);
        });
    });

    req.on("error", callback);
    req.write(body);
    req.end();
}

var SERVICE = process.env.DD_SERVICE || process.env.BUILD_REPOSITORY_NAME || "unknown";
var PIPELINE = process.env.BUILD_DEFINITIONNAME || "unknown";
var BRANCH = (process.env.BUILD_SOURCEBRANCH || "").replace("refs/heads/", "");
var RESULT = process.env.AGENT_JOBSTATUS || "Unknown";

var tags = [
    "service:" + SERVICE,
    "pipeline:" + PIPELINE,
    "branch:" + BRANCH,
    "result:" + RESULT.toLowerCase(),
    "source:azure-devops"
];

// Calculate build duration
var startTime = new Date(process.env.SYSTEM_PIPELINESTARTTIME || Date.now());
var durationSec = Math.round((Date.now() - startTime.getTime()) / 1000);

var pending = 2;
function done() { pending--; if (pending === 0) { console.log("All metrics sent."); } }

sendMetric("ci.pipeline.duration", durationSec, tags, function (err) {
    if (err) { console.error("Duration metric error: " + err.message); }
    else { console.log("Sent pipeline duration: " + durationSec + "s"); }
    done();
});

sendMetric("ci.pipeline.completed", 1, tags, function (err) {
    if (err) { console.error("Completed metric error: " + err.message); }
    else { console.log("Sent pipeline completion metric"); }
    done();
});

Deployment Gates with Datadog Monitors

Use Datadog monitors as deployment gates — the pipeline checks a Datadog monitor before allowing deployment to proceed. If the monitor is in alert state, the deployment is blocked.

Checking Monitor Status from Pipeline

// scripts/dd-gate-check.js
var https = require("https");

var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_APP_KEY = process.env.DATADOG_APP_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";
var MONITOR_IDS = (process.env.DD_MONITOR_IDS || "").split(",").filter(function (id) { return id.trim(); });

if (MONITOR_IDS.length === 0) {
    console.error("No monitor IDs specified. Set DD_MONITOR_IDS environment variable.");
    process.exit(1);
}

function getMonitorStatus(monitorId, callback) {
    var options = {
        hostname: "api." + DD_SITE,
        path: "/api/v1/monitor/" + monitorId,
        method: "GET",
        headers: {
            "DD-API-KEY": DD_API_KEY,
            "DD-APPLICATION-KEY": DD_APP_KEY,
            "Accept": "application/json"
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            if (res.statusCode === 200) {
                callback(null, JSON.parse(data));
            } else {
                callback(new Error("Monitor API error: " + res.statusCode));
            }
        });
    });

    req.on("error", callback);
    req.end();
}

console.log("Checking " + MONITOR_IDS.length + " Datadog monitor(s)...\n");

var completed = 0;
var allHealthy = true;
var results = [];

MONITOR_IDS.forEach(function (id) {
    getMonitorStatus(id.trim(), function (err, monitor) {
        if (err) {
            console.error("Failed to check monitor " + id + ": " + err.message);
            allHealthy = false;
            results.push({ id: id, name: "ERROR", status: "error" });
        } else {
            var status = monitor.overall_state;
            var healthy = status === "OK" || status === "No Data";
            var icon = healthy ? "OK" : "ALERT";

            console.log("[" + icon + "] Monitor #" + id + ": " + monitor.name);
            console.log("     Status: " + status);
            console.log("     Type: " + monitor.type);

            if (!healthy) {
                allHealthy = false;
                console.log("     !! This monitor is blocking deployment");
            }
            console.log("");

            results.push({ id: id, name: monitor.name, status: status });
        }

        completed++;
        if (completed === MONITOR_IDS.length) {
            console.log("=== Gate Result: " + (allHealthy ? "PASSED" : "BLOCKED") + " ===");

            if (!allHealthy) {
                var blocked = results.filter(function (r) { return r.status !== "OK" && r.status !== "No Data"; });
                console.error("\nDeployment blocked by " + blocked.length + " monitor(s):");
                blocked.forEach(function (r) {
                    console.error("  - #" + r.id + " " + r.name + " (" + r.status + ")");
                });
                process.exit(1);
            }
        }
    });
});

Pipeline usage:

- script: node scripts/dd-gate-check.js
  displayName: "Datadog deployment gate check"
  env:
    DATADOG_API_KEY: $(DatadogApiKey)
    DATADOG_APP_KEY: $(DatadogAppKey)
    DD_MONITOR_IDS: "12345678,23456789,34567890"

Datadog Service Catalog Integration

Datadog's Service Catalog can track deployment frequency and status per service. Register your services and their pipelines:

# datadog-service-catalog.yaml (checked into repo)
schema-version: v2
dd-service: my-api-service
team: backend-team
contacts:
  - type: slack
    contact: "#backend-alerts"
  - type: email
    contact: [email protected]
repos:
  - url: https://dev.azure.com/your-org/your-project/_git/my-api
    provider: azure-devops
ci-cd:
  - ci_provider: azurepipelines
    pipeline_id: "42"
    pipeline_url: https://dev.azure.com/your-org/your-project/_build?definitionId=42
links:
  - name: Runbook
    type: runbook
    url: https://wiki.company.com/runbooks/my-api
  - name: Architecture Diagram
    type: doc
    url: https://wiki.company.com/architecture/my-api
tags:
  - "env:production"
  - "tier:1"
  - "language:nodejs"

Push this to Datadog's service catalog API:

- script: |
    curl -s -X POST "https://api.datadoghq.com/api/v2/services/definitions" \
      -H "DD-API-KEY: $(DATADOG_API_KEY)" \
      -H "DD-APPLICATION-KEY: $(DATADOG_APP_KEY)" \
      -H "Content-Type: application/json" \
      -d @datadog-service-catalog.yaml
  displayName: "Update Datadog Service Catalog"

Complete Working Example: Full Observability Pipeline

This pipeline integrates Datadog at every stage — pre-deployment gate checks, deployment events, post-deployment monitoring, and automatic rollback on degraded metrics:

// scripts/dd-post-deploy-check.js
// Checks Datadog APM metrics after deployment to verify service health
var https = require("https");

var DD_API_KEY = process.env.DATADOG_API_KEY;
var DD_APP_KEY = process.env.DATADOG_APP_KEY;
var DD_SITE = process.env.DD_SITE || "datadoghq.com";
var SERVICE = process.env.DD_SERVICE || "my-app";
var ENVIRONMENT = process.env.DD_ENV || "production";

function queryMetric(query, from, to, callback) {
    var params = "query=" + encodeURIComponent(query) +
        "&from=" + from + "&to=" + to;

    var options = {
        hostname: "api." + DD_SITE,
        path: "/api/v1/query?" + params,
        method: "GET",
        headers: {
            "DD-API-KEY": DD_API_KEY,
            "DD-APPLICATION-KEY": DD_APP_KEY,
            "Accept": "application/json"
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            if (res.statusCode === 200) {
                callback(null, JSON.parse(data));
            } else {
                callback(new Error("Query failed: " + res.statusCode + " " + data));
            }
        });
    });

    req.on("error", callback);
    req.end();
}

function getLatestValue(series) {
    if (!series || !series.series || series.series.length === 0) { return null; }
    var points = series.series[0].pointlist;
    if (!points || points.length === 0) { return null; }
    return points[points.length - 1][1];
}

var now = Math.floor(Date.now() / 1000);
var fiveMinAgo = now - 300;

console.log("Post-deployment health check for " + SERVICE + " (" + ENVIRONMENT + ")\n");

var checks = [
    {
        name: "Error Rate",
        query: "sum:trace." + SERVICE + ".errors{env:" + ENVIRONMENT + "}.as_rate() / sum:trace." + SERVICE + ".hits{env:" + ENVIRONMENT + "}.as_rate() * 100",
        threshold: 5,
        unit: "%"
    },
    {
        name: "P95 Latency",
        query: "p95:trace." + SERVICE + ".request.duration{env:" + ENVIRONMENT + "}",
        threshold: 2000,
        unit: "ms"
    },
    {
        name: "Request Rate",
        query: "sum:trace." + SERVICE + ".hits{env:" + ENVIRONMENT + "}.as_rate()",
        threshold: 0, // Alert if zero (service is down)
        unit: "req/s",
        alertBelow: true
    }
];

var completed = 0;
var allPassed = true;

checks.forEach(function (check) {
    queryMetric(check.query, fiveMinAgo, now, function (err, result) {
        completed++;

        if (err) {
            console.error("[ERROR] " + check.name + ": " + err.message);
            allPassed = false;
        } else {
            var value = getLatestValue(result);

            if (value === null) {
                console.log("[WARN]  " + check.name + ": No data available");
            } else {
                var passed;
                if (check.alertBelow) {
                    passed = value > check.threshold;
                } else {
                    passed = value <= check.threshold;
                }

                var icon = passed ? "OK" : "FAIL";
                var roundedValue = Math.round(value * 100) / 100;

                console.log("[" + icon + "]  " + check.name + ": " + roundedValue + check.unit +
                    " (threshold: " + (check.alertBelow ? ">" : "<=") + " " + check.threshold + check.unit + ")");

                if (!passed) { allPassed = false; }
            }
        }

        if (completed === checks.length) {
            console.log("\n=== Post-Deploy Result: " + (allPassed ? "HEALTHY" : "DEGRADED") + " ===");
            if (!allPassed) {
                console.error("Service health degraded after deployment!");
                process.exit(1);
            }
        }
    });
});

Pipeline YAML using the full integration:

# azure-pipelines-datadog-full.yml
trigger:
  branches:
    include:
      - main

pool:
  vmImage: "ubuntu-latest"

variables:
  - group: datadog-credentials
  - name: serviceName
    value: "my-api"
  - name: deployEnv
    value: "production"

stages:
  - stage: Build
    jobs:
      - job: Build
        steps:
          - script: npm ci && npm test && npm run build
            displayName: "Build and test"
          - task: PublishPipelineArtifact@1
            inputs:
              targetPath: dist
              artifact: app

  - stage: PreDeployGate
    displayName: "Pre-Deploy Gate (Datadog)"
    dependsOn: Build
    jobs:
      - job: GateCheck
        steps:
          - script: node scripts/dd-gate-check.js
            displayName: "Check Datadog monitors"
            env:
              DATADOG_API_KEY: $(DatadogApiKey)
              DATADOG_APP_KEY: $(DatadogAppKey)
              DD_MONITOR_IDS: $(GATE_MONITOR_IDS)

  - stage: Deploy
    dependsOn: PreDeployGate
    jobs:
      - deployment: Production
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploying $(serviceName)..."
                  displayName: "Deploy"

                - script: node scripts/datadog-deployment-event.js
                  displayName: "Send Datadog deployment event"
                  env:
                    DATADOG_API_KEY: $(DatadogApiKey)
                    DD_SERVICE: $(serviceName)
                    DEPLOY_ENVIRONMENT: $(deployEnv)

  - stage: PostDeployVerify
    displayName: "Post-Deploy Verification"
    dependsOn: Deploy
    jobs:
      - job: Verify
        steps:
          - script: sleep 120
            displayName: "Stabilization period (2 min)"

          - script: node scripts/dd-post-deploy-check.js
            displayName: "Datadog health check"
            env:
              DATADOG_API_KEY: $(DatadogApiKey)
              DATADOG_APP_KEY: $(DatadogAppKey)
              DD_SERVICE: $(serviceName)
              DD_ENV: $(deployEnv)

          - script: node scripts/dd-pipeline-metrics.js
            displayName: "Send pipeline metrics"
            condition: always()
            env:
              DATADOG_API_KEY: $(DatadogApiKey)
              DD_SERVICE: $(serviceName)

Common Issues and Troubleshooting

Datadog events not appearing in the event stream

HTTP 403: Forbidden

Verify your API key is correct and has not been revoked. Datadog API keys are organization-wide — if your org has multiple Datadog accounts, ensure the key matches the account where you expect events. Also check the DD_SITE value: US customers use datadoghq.com, EU customers use datadoghq.eu, and other regions have different sites.

CI Visibility pipeline traces are missing or incomplete

The Datadog CI extension must be set up before any build steps run. If you add the DatadogCISetup task after other tasks, those earlier tasks are not traced. Also verify the pipeline agent has outbound HTTPS access to api.datadoghq.com — corporate firewalls may block it.

Monitor gate check returns "No Data" status

A monitor in "No Data" state means Datadog has not received metrics for the monitor's metric query within the evaluation window. This often happens for new monitors or after a metric name change. Decide whether "No Data" should block deployments (conservative) or allow them (permissive). The gate check script above treats "No Data" as passing — adjust if your risk tolerance differs.

Deployment events have wrong timestamps

Datadog events use the server-side receive time if no timestamp is specified in the payload. If your pipeline agent's clock is skewed or you are sending events from a post-deployment step that runs much later, the event may not align with the actual deployment time. Always include an explicit timestamp in the event payload using the pipeline's start time variable.

Rate limiting when sending many metrics

HTTP 429: Too Many Requests

Datadog's metric submission API accepts up to 500 series per request and has per-minute rate limits based on your plan. Batch metrics into a single API call rather than sending one metric per request. For CI pipelines running hundreds of jobs concurrently, consider using the Datadog Agent running on the build agents instead of direct API calls.

Best Practices

  • Send deployment events for every environment, not just production. Tracking staging and dev deployments helps identify issues before they reach production. Tag events with environment:staging to differentiate them on dashboards.

  • Use Datadog monitors as deployment gates for critical services. Define monitors for error rate, latency, and request volume. If any are in alert state, block the deployment. This prevents deploying into an already degraded environment.

  • Include Azure DevOps build URLs in Datadog events. When investigating an issue in Datadog, a direct link back to the pipeline build log saves time. Include the build URL as a link in the event markdown text.

  • Create a Datadog dashboard for CI/CD metrics. Combine pipeline duration trends, deployment frequency, failure rates, and post-deployment health into a single dashboard. This gives engineering leadership visibility into delivery performance.

  • Tag all Datadog submissions consistently. Use the same service, environment, and team tags across events, metrics, and APM traces. Consistent tagging enables cross-referencing between CI data and production monitoring.

  • Set up Datadog monitors that trigger on deployment-correlated degradation. Use anomaly detection monitors that alert when error rates or latency deviate from the baseline within minutes of a deployment event. This automates the "was it the deploy?" question.

  • Store Datadog API keys as secret pipeline variables. Never hard-code keys in YAML or scripts. Use Azure DevOps variable groups with secret variables, and reference them through $(DatadogApiKey) in pipelines.

References

Powered by Contentful