Connecting Azure DevOps to Azure Monitor

Overview

The gap between deploying code and knowing whether it works in production is where most teams lose time. Azure DevOps handles the build and deploy pipeline; Azure Monitor handles the production telemetry. Connecting them closes that gap — deployment annotations mark when releases happened on your monitoring dashboards, release gates check production health before promoting builds, and alerts automatically create work items when things break. I have set up this integration for teams where the mean time to detect issues dropped from hours to minutes because the pipeline was finally aware of what was happening in production.

Prerequisites

• Azure DevOps organization with Pipelines and Boards enabled
• Azure subscription with Azure Monitor and Application Insights resources
• Azure service connection configured in Azure DevOps
• Application Insights instrumentation key or connection string
• Azure Monitor workspace (Log Analytics)
• Familiarity with Kusto Query Language (KQL) for Azure Monitor queries

Deployment Annotations

Deployment annotations mark the exact moment a release happened on your Application Insights timeline. When investigating a production issue, seeing a deployment marker at the point where error rates spiked immediately tells you the deployment caused it.

Adding Annotations from Azure Pipelines

# azure-pipelines.yml
steps:
  - script: |
      echo "Deploying application..."
      # Your deployment steps here

  - task: AzureCLI@2
    displayName: "Create Deployment Annotation"
    inputs:
      azureSubscription: "your-azure-subscription"
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        APPINSIGHTS_ID=$(az monitor app-insights component show \
          --app your-app-insights \
          --resource-group your-rg \
          --query id --output tsv)

        ANNOTATION_JSON=$(cat <<EOF
        {
          "Id": "$(Build.BuildId)",
          "AnnotationName": "Release $(Build.BuildNumber)",
          "EventTime": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
          "Category": "Deployment",
          "Properties": "{\"BuildNumber\":\"$(Build.BuildNumber)\",\"Branch\":\"$(Build.SourceBranch)\",\"ReleaseName\":\"$(Release.ReleaseName)\",\"Environment\":\"production\",\"RequestedBy\":\"$(Build.RequestedFor)\"}"
        }
        EOF
        )

        az rest --method put \
          --url "${APPINSIGHTS_ID}/Annotations?api-version=2015-05-01" \
          --body "$ANNOTATION_JSON"

        echo "Deployment annotation created for build $(Build.BuildNumber)"

Annotations with the REST API Directly

For more control, use the Application Insights REST API from a Node.js script:

// scripts/create-annotation.js
var https = require("https");

var APPINSIGHTS_APP_ID = process.env.APPINSIGHTS_APP_ID;
var APPINSIGHTS_API_KEY = process.env.APPINSIGHTS_API_KEY;
var BUILD_NUMBER = process.env.BUILD_BUILDNUMBER || "local";
var BUILD_BRANCH = (process.env.BUILD_SOURCEBRANCH || "").replace("refs/heads/", "");
var REQUESTED_BY = process.env.BUILD_REQUESTEDFOR || "manual";

var annotation = {
    Id: Date.now().toString(),
    AnnotationName: "Release " + BUILD_NUMBER,
    EventTime: new Date().toISOString(),
    Category: "Deployment",
    Properties: JSON.stringify({
        BuildNumber: BUILD_NUMBER,
        Branch: BUILD_BRANCH,
        RequestedBy: REQUESTED_BY,
        Timestamp: new Date().toISOString()
    })
};

var body = JSON.stringify([annotation]);

var options = {
    hostname: "aigs1.eus.ext.azure.com",
    path: "/api/v1/apps/" + APPINSIGHTS_APP_ID + "/annotations",
    method: "PUT",
    headers: {
        "Content-Type": "application/json",
        "X-AIGS-ApiKey": APPINSIGHTS_API_KEY,
        "Content-Length": Buffer.byteLength(body)
    }
};

var req = https.request(options, function (res) {
    var data = "";
    res.on("data", function (chunk) { data += chunk; });
    res.on("end", function () {
        if (res.statusCode >= 200 && res.statusCode < 300) {
            console.log("Annotation created successfully");
        } else {
            console.error("Annotation failed (" + res.statusCode + "): " + data);
        }
    });
});

req.on("error", function (err) {
    console.error("Annotation request failed: " + err.message);
});

req.write(body);
req.end();

Release Gates with Azure Monitor

Release gates query Azure Monitor before allowing a deployment to proceed to the next stage. If error rates are high or the application is unhealthy, the gate blocks the promotion.

Azure Monitor Alert Gate

Configure a release gate that checks Azure Monitor alerts:

1. In your release pipeline, select a stage
2. Click the gate icon (pre-deployment or post-deployment)
3. Add Query Azure Monitor alerts
4. Configure:
   • Azure subscription: your subscription
   • Resource group: your-rg
   • Resource type: Application Insights
   • Resource name: your-app-insights
   • Alert filter: Severity Sev0, Sev1
   • Time range: 5 minutes

If any Sev0 or Sev1 alerts are active, the gate fails and the deployment waits.

Custom KQL Query Gate

For more sophisticated health checks, use the Invoke REST API gate with a KQL query:

# In a YAML pipeline, use the InvokeRestAPI task
steps:
  - task: InvokeRestAPI@1
    displayName: "Check Application Health"
    inputs:
      connectionType: "connectedServiceName"
      serviceConnection: "azure-monitor-connection"
      method: POST
      urlSuffix: "/v1/workspaces/$(WORKSPACE_ID)/query"
      body: |
        {
          "query": "requests | where timestamp > ago(5m) | summarize errorRate = todouble(countif(success == false)) / count() * 100 | project healthy = errorRate < 5"
        }
      waitForCompletion: "true"

Post-Deployment Health Gate

After deploying, wait for the application to stabilize, then check health before marking the deployment as successful:

# azure-pipelines-with-gates.yml
stages:
  - stage: Deploy
    jobs:
      - deployment: Production
        environment: "production"
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploying..."
                  displayName: "Deploy application"

            postRouteTraffic:
              steps:
                - script: |
                    echo "Waiting 2 minutes for deployment to stabilize..."
                    sleep 120
                  displayName: "Stabilization period"

                - task: AzureCLI@2
                  displayName: "Check post-deployment health"
                  inputs:
                    azureSubscription: "your-azure-subscription"
                    scriptType: bash
                    scriptLocation: inlineScript
                    inlineScript: |
                      # Query Application Insights for error rate
                      ERROR_RATE=$(az monitor app-insights query \
                        --app your-app-insights \
                        --resource-group your-rg \
                        --analytics-query "requests | where timestamp > ago(5m) | summarize errorRate = round(todouble(countif(success == false)) / count() * 100, 2)" \
                        --query "tables[0].rows[0][0]" --output tsv)

                      echo "Post-deployment error rate: ${ERROR_RATE}%"

                      if (( $(echo "$ERROR_RATE > 5.0" | bc -l) )); then
                        echo "##vso[task.logissue type=error]Error rate ${ERROR_RATE}% exceeds 5% threshold"
                        echo "##vso[task.complete result=Failed;]High error rate after deployment"
                      else
                        echo "Error rate ${ERROR_RATE}% is within acceptable range"
                      fi

Alert-Driven Work Item Creation

When Azure Monitor detects an issue in production, automatically create a work item in Azure DevOps so the team can track and resolve it.

Azure Monitor Action Group

Create an Action Group that calls an Azure Function or Logic App, which then creates a work item:

// alert-to-workitem/index.js
// Azure Function triggered by Azure Monitor alert webhook
var https = require("https");

var AZURE_ORG = process.env.AZURE_DEVOPS_ORG;
var AZURE_PROJECT = process.env.AZURE_DEVOPS_PROJECT;
var AZURE_PAT = process.env.AZURE_DEVOPS_PAT;

function createWorkItem(alertData, callback) {
    var auth = Buffer.from(":" + AZURE_PAT).toString("base64");
    var severity = alertData.data.essentials.severity;
    var priority = severity === "Sev0" ? 1 : (severity === "Sev1" ? 2 : 3);

    var title = "[Alert] " + alertData.data.essentials.alertRule;
    var description = "<h3>Azure Monitor Alert</h3>" +
        "<p><strong>Severity:</strong> " + severity + "</p>" +
        "<p><strong>Fired at:</strong> " + alertData.data.essentials.firedDateTime + "</p>" +
        "<p><strong>Description:</strong> " + (alertData.data.essentials.description || "No description") + "</p>" +
        "<p><strong>Monitor condition:</strong> " + alertData.data.essentials.monitorCondition + "</p>" +
        "<p><strong>Target resource:</strong> " + alertData.data.essentials.targetResourceName + "</p>";

    if (alertData.data.alertContext && alertData.data.alertContext.condition) {
        var conditions = alertData.data.alertContext.condition.allOf || [];
        conditions.forEach(function (c) {
            description += "<p><strong>Metric:</strong> " + c.metricName +
                " " + c.operator + " " + c.threshold +
                " (actual: " + c.metricValue + ")</p>";
        });
    }

    var patchDoc = [
        { op: "add", path: "/fields/System.Title", value: title },
        { op: "add", path: "/fields/System.Description", value: description },
        { op: "add", path: "/fields/Microsoft.VSTS.Common.Priority", value: priority },
        { op: "add", path: "/fields/System.Tags", value: "azure-monitor-alert;auto-created;" + severity }
    ];

    var body = JSON.stringify(patchDoc);

    var options = {
        hostname: "dev.azure.com",
        path: "/" + AZURE_ORG + "/" + AZURE_PROJECT + "/_apis/wit/workitems/$Bug?api-version=7.1",
        method: "POST",
        headers: {
            "Authorization": "Basic " + auth,
            "Content-Type": "application/json-patch+json",
            "Content-Length": Buffer.byteLength(body)
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            if (res.statusCode >= 200 && res.statusCode < 300) {
                var result = JSON.parse(data);
                callback(null, result);
            } else {
                callback(new Error("Work item creation failed: " + res.statusCode + " " + data));
            }
        });
    });

    req.on("error", callback);
    req.write(body);
    req.end();
}

module.exports = function (context, req) {
    context.log("Alert webhook received");

    var alertData = req.body;

    if (!alertData || !alertData.data || !alertData.data.essentials) {
        context.res = { status: 400, body: "Invalid alert payload" };
        context.done();
        return;
    }

    var monitorCondition = alertData.data.essentials.monitorCondition;
    if (monitorCondition === "Resolved") {
        context.log("Alert resolved — skipping work item creation");
        context.res = { status: 200, body: "Alert resolved, no action needed" };
        context.done();
        return;
    }

    createWorkItem(alertData, function (err, result) {
        if (err) {
            context.log.error("Failed to create work item: " + err.message);
            context.res = { status: 500, body: err.message };
        } else {
            context.log("Created work item #" + result.id + ": " + result.fields["System.Title"]);
            context.res = { status: 200, body: { workItemId: result.id } };
        }
        context.done();
    });
};

Deduplication

Alert rules can fire repeatedly. Avoid creating duplicate work items by checking if one already exists:

function findExistingWorkItem(alertRule, callback) {
    var wiql = {
        query: "SELECT [System.Id] FROM WorkItems " +
            "WHERE [System.Title] CONTAINS '[Alert] " + alertRule.replace(/'/g, "''") + "' " +
            "AND [System.State] <> 'Closed' " +
            "AND [System.Tags] CONTAINS 'azure-monitor-alert' " +
            "AND [System.CreatedDate] >= @today - 7"
    };

    var auth = Buffer.from(":" + AZURE_PAT).toString("base64");
    var body = JSON.stringify(wiql);

    var options = {
        hostname: "dev.azure.com",
        path: "/" + AZURE_ORG + "/" + AZURE_PROJECT + "/_apis/wit/wiql?api-version=7.1",
        method: "POST",
        headers: {
            "Authorization": "Basic " + auth,
            "Content-Type": "application/json",
            "Content-Length": Buffer.byteLength(body)
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            var result = JSON.parse(data);
            var existing = result.workItems && result.workItems.length > 0;
            callback(null, existing ? result.workItems[0].id : null);
        });
    });

    req.on("error", callback);
    req.write(body);
    req.end();
}

Check before creating. If a work item already exists for this alert rule, add a comment instead of creating a duplicate.

Pipeline Telemetry to Application Insights

Send custom telemetry from your pipelines to Application Insights for build analytics:

// scripts/pipeline-telemetry.js
var https = require("https");

var INSTRUMENTATION_KEY = process.env.APPINSIGHTS_INSTRUMENTATION_KEY;
var INGESTION_ENDPOINT = "https://dc.services.visualstudio.com/v2/track";

function sendTelemetry(eventName, properties, measurements) {
    var envelope = {
        name: "Microsoft.ApplicationInsights." + INSTRUMENTATION_KEY.replace(/-/g, "") + ".Event",
        time: new Date().toISOString(),
        iKey: INSTRUMENTATION_KEY,
        tags: {
            "ai.cloud.role": "azure-pipelines",
            "ai.cloud.roleInstance": process.env.BUILD_DEFINITIONNAME || "unknown"
        },
        data: {
            baseType: "EventData",
            baseData: {
                ver: 2,
                name: eventName,
                properties: properties || {},
                measurements: measurements || {}
            }
        }
    };

    var body = JSON.stringify([envelope]);
    var url = new URL(INGESTION_ENDPOINT);

    var options = {
        hostname: url.hostname,
        path: url.pathname,
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "Content-Length": Buffer.byteLength(body)
        }
    };

    var req = https.request(options, function (res) {
        var data = "";
        res.on("data", function (chunk) { data += chunk; });
        res.on("end", function () {
            if (res.statusCode === 200) {
                console.log("Telemetry sent: " + eventName);
            } else {
                console.error("Telemetry failed (" + res.statusCode + "): " + data);
            }
        });
    });

    req.on("error", function (err) {
        console.error("Telemetry error: " + err.message);
    });

    req.write(body);
    req.end();
}

// Send build metrics
var buildStart = new Date(process.env.SYSTEM_PIPELINESTARTTIME || Date.now());
var durationMs = Date.now() - buildStart.getTime();

sendTelemetry("PipelineCompleted", {
    buildNumber: process.env.BUILD_BUILDNUMBER || "local",
    definitionName: process.env.BUILD_DEFINITIONNAME || "unknown",
    branch: (process.env.BUILD_SOURCEBRANCH || "").replace("refs/heads/", ""),
    result: process.env.AGENT_JOBSTATUS || "Unknown",
    reason: process.env.BUILD_REASON || "manual",
    repository: process.env.BUILD_REPOSITORY_NAME || "unknown"
}, {
    durationMs: durationMs,
    durationMinutes: Math.round(durationMs / 60000 * 100) / 100
});

Use this in your pipeline:

- script: node scripts/pipeline-telemetry.js
  displayName: "Send pipeline telemetry"
  condition: always()
  env:
    APPINSIGHTS_INSTRUMENTATION_KEY: $(APPINSIGHTS_KEY)

Now you can query pipeline metrics in Azure Monitor:

customEvents
| where name == "PipelineCompleted"
| extend buildNumber = tostring(customDimensions.buildNumber),
         definition = tostring(customDimensions.definitionName),
         result = tostring(customDimensions.result),
         branch = tostring(customDimensions.branch),
         durationMin = todouble(customMeasurements.durationMinutes)
| summarize
    totalBuilds = count(),
    avgDuration = round(avg(durationMin), 1),
    successRate = round(countif(result == "Succeeded") * 100.0 / count(), 1),
    failedCount = countif(result == "Failed")
    by definition, bin(timestamp, 1d)
| order by timestamp desc

Complete Working Example: Observability-Driven Deployment Pipeline

This pipeline integrates Azure Monitor at every stage — pre-deployment health checks, deployment annotations, post-deployment verification, and rollback triggers:

# azure-pipelines-observable.yml
trigger:
  branches:
    include:
      - main

pool:
  vmImage: "ubuntu-latest"

variables:
  - group: azure-monitor-config
  - name: appInsightsName
    value: "ai-myapp-prod"
  - name: resourceGroup
    value: "rg-myapp-prod"

stages:
  - stage: Build
    jobs:
      - job: BuildAndTest
        steps:
          - script: npm ci && npm test
            displayName: "Build and test"

          - task: PublishPipelineArtifact@1
            inputs:
              targetPath: "$(System.DefaultWorkingDirectory)"
              artifact: "app"

  - stage: PreDeployCheck
    displayName: "Pre-Deployment Health Check"
    dependsOn: Build
    jobs:
      - job: CheckHealth
        steps:
          - task: AzureCLI@2
            displayName: "Verify production is healthy before deploy"
            inputs:
              azureSubscription: "your-azure-subscription"
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                ERROR_RATE=$(az monitor app-insights query \
                  --app $(appInsightsName) \
                  --resource-group $(resourceGroup) \
                  --analytics-query "requests | where timestamp > ago(10m) | summarize round(todouble(countif(success == false)) / count() * 100, 2)" \
                  --query "tables[0].rows[0][0]" --output tsv)

                echo "Current error rate: ${ERROR_RATE}%"

                if (( $(echo "$ERROR_RATE > 10.0" | bc -l) )); then
                  echo "##vso[task.logissue type=error]Production error rate is ${ERROR_RATE}% — too high to deploy"
                  exit 1
                fi

                ACTIVE_ALERTS=$(az monitor alert list \
                  --resource-group $(resourceGroup) \
                  --query "length([?properties.essentials.monitorCondition=='Fired' && (properties.essentials.severity=='Sev0' || properties.essentials.severity=='Sev1')])" \
                  --output tsv 2>/dev/null || echo "0")

                echo "Active Sev0/Sev1 alerts: $ACTIVE_ALERTS"
                if [ "$ACTIVE_ALERTS" -gt "0" ]; then
                  echo "##vso[task.logissue type=error]Active critical alerts — deployment blocked"
                  exit 1
                fi

                echo "Production is healthy. Proceeding with deployment."

  - stage: Deploy
    dependsOn: PreDeployCheck
    jobs:
      - deployment: Production
        environment: "production"
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploying build $(Build.BuildNumber)..."
                  displayName: "Deploy"

                - task: AzureCLI@2
                  displayName: "Create deployment annotation"
                  inputs:
                    azureSubscription: "your-azure-subscription"
                    scriptType: bash
                    scriptLocation: inlineScript
                    inlineScript: |
                      az monitor app-insights component update-tags \
                        --app $(appInsightsName) \
                        --resource-group $(resourceGroup) \
                        --tags "lastDeployment=$(Build.BuildNumber)"

                      echo "Deployment annotation created"

  - stage: PostDeployVerify
    displayName: "Post-Deployment Verification"
    dependsOn: Deploy
    jobs:
      - job: Verify
        steps:
          - script: sleep 120
            displayName: "Wait for stabilization (2 min)"

          - task: AzureCLI@2
            displayName: "Post-deployment health check"
            inputs:
              azureSubscription: "your-azure-subscription"
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                echo "Checking post-deployment metrics..."

                RESULTS=$(az monitor app-insights query \
                  --app $(appInsightsName) \
                  --resource-group $(resourceGroup) \
                  --analytics-query "
                    requests
                    | where timestamp > ago(5m)
                    | summarize
                        errorRate = round(todouble(countif(success == false)) / count() * 100, 2),
                        avgDuration = round(avg(duration), 0),
                        p95Duration = round(percentile(duration, 95), 0),
                        totalRequests = count()
                  " --output json)

                echo "$RESULTS" | jq '.'

                ERROR_RATE=$(echo "$RESULTS" | jq -r '.tables[0].rows[0][0]')
                AVG_DURATION=$(echo "$RESULTS" | jq -r '.tables[0].rows[0][1]')
                P95_DURATION=$(echo "$RESULTS" | jq -r '.tables[0].rows[0][2]')

                echo "Error rate: ${ERROR_RATE}%"
                echo "Avg response time: ${AVG_DURATION}ms"
                echo "P95 response time: ${P95_DURATION}ms"

                if (( $(echo "$ERROR_RATE > 5.0" | bc -l) )); then
                  echo "##vso[task.logissue type=error]Post-deployment error rate ${ERROR_RATE}% exceeds threshold"
                  exit 1
                fi

                echo "Post-deployment verification PASSED"

          - script: node scripts/pipeline-telemetry.js
            displayName: "Send deployment telemetry"
            condition: always()
            env:
              APPINSIGHTS_INSTRUMENTATION_KEY: $(APPINSIGHTS_KEY)

Common Issues and Troubleshooting

Deployment annotation not appearing in Application Insights

The annotations API has changed across Application Insights versions. Classic Application Insights uses the /Annotations endpoint while workspace-based Application Insights uses a different mechanism. Verify your Application Insights resource type and use the corresponding API. Also check that the annotation timestamp falls within the time range displayed on the dashboard.

Release gate times out waiting for Azure Monitor query

Gate evaluation timed out

Azure Monitor queries can take 10-30 seconds, and gates have a short evaluation window. Increase the gate timeout and evaluation interval in the gate configuration. Also simplify your KQL query — complex queries with multiple joins or large time ranges take longer to execute.

Application Insights query returns "Forbidden" from pipeline

ERROR: (AuthorizationFailed) The client does not have authorization to perform action 'microsoft.insights/components/query/read'

The service connection's Service Principal needs the Monitoring Reader role on the Application Insights resource. Assign the role at the resource group level to cover all monitoring resources. The default Contributor role does not include monitoring query permissions.

Alert webhook payload format changed unexpectedly

Azure Monitor has two alert payload schemas: common alert schema and the legacy schema. If you enabled the common alert schema on an action group but your webhook handler expects the legacy format, parsing fails. Always check the action group's "Use common alert schema" setting and handle both formats in your webhook receiver.

Best Practices

• Create deployment annotations on every production deployment. Annotations are the fastest way to correlate production issues with deployments. When someone reports a bug, the first question is "did anything deploy recently?" — annotations answer that instantly.

• Use pre-deployment gates to block deploys during incidents. If production is already on fire, deploying new code makes it worse. A simple gate that checks for active Sev0/Sev1 alerts prevents piling new changes on top of existing problems.

• Send custom pipeline telemetry to Application Insights. Track build duration, success rates, and failure patterns over time. This turns Application Insights into a single dashboard for both application and CI/CD health.

• Deduplicate alert-driven work items. Alert rules fire repeatedly while a condition persists. Without deduplication, a single production incident can create dozens of identical bugs. Check for existing open work items with the same alert rule name before creating new ones.

• Set post-deployment verification windows based on your traffic patterns. A 2-minute stabilization period works for high-traffic applications that generate enough telemetry quickly. Low-traffic services may need 10-15 minutes for meaningful error rate calculations.

• Use KQL queries for sophisticated health checks. Simple HTTP health endpoints tell you the app is running. KQL queries against Application Insights tell you the app is running well — error rates, latency percentiles, dependency failures, and exception trends.

References

• Azure Monitor Overview
• Application Insights Annotations
• Release Gates in Azure Pipelines
• Azure Monitor Action Groups
• Kusto Query Language (KQL) Reference
• Azure CLI Monitor Commands