Load Testing with Azure Load Testing Service

Overview

Azure Load Testing is a managed service that runs Apache JMeter scripts at scale without requiring you to provision or manage load generation infrastructure. You upload a JMeter test plan, specify the number of virtual users and engine instances, and Azure spins up the infrastructure, runs the test, and gives you response time percentiles, throughput graphs, and error rates. If your application runs on Azure, it also correlates server-side metrics from Application Insights and Azure Monitor directly in the test results.

I have been running load tests against Node.js APIs and microservices for years, first with self-managed JMeter clusters, then with various SaaS tools, and now primarily with Azure Load Testing. The managed service eliminates the operational overhead of maintaining load generators, and the pipeline integration means performance regressions are caught in CI/CD before they reach production. This article covers creating load tests from JMeter scripts, configuring them in Azure, integrating with Azure Pipelines, analyzing results, and setting threshold-based pass/fail criteria.

Prerequisites

• An Azure subscription with permissions to create resources
• An Azure DevOps organization with Azure Pipelines enabled
• Apache JMeter installed locally (5.5+ recommended) for test plan authoring
• A target application or API to test
• Node.js 18+ for helper scripts
• Basic familiarity with JMeter test plans (.jmx files)

Creating a JMeter Test Plan

Azure Load Testing executes standard JMeter .jmx files. Author your test plan locally in JMeter's GUI, then upload it to the service.

Basic HTTP Load Test

Here is a JMeter test plan for a Node.js REST API:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="API Load Test">
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments">
        <collectionProp name="Arguments.arguments">
          <elementProp name="BASE_URL" elementType="Argument">
            <stringProp name="Argument.name">BASE_URL</stringProp>
            <stringProp name="Argument.value">${__P(base_url,https://api.example.com)}</stringProp>
          </elementProp>
          <elementProp name="DURATION" elementType="Argument">
            <stringProp name="Argument.name">DURATION</stringProp>
            <stringProp name="Argument.value">${__P(duration,300)}</stringProp>
          </elementProp>
        </collectionProp>
      </elementProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="API Users">
        <intProp name="ThreadGroup.num_threads">50</intProp>
        <intProp name="ThreadGroup.ramp_time">60</intProp>
        <boolProp name="ThreadGroup.scheduler">true</boolProp>
        <stringProp name="ThreadGroup.duration">${DURATION}</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController">
          <boolProp name="LoopController.continue_forever">true</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </elementProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="GET /api/health">
          <stringProp name="HTTPSampler.domain">${BASE_URL}</stringProp>
          <stringProp name="HTTPSampler.path">/api/health</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <stringProp name="HTTPSampler.protocol">https</stringProp>
        </HTTPSamplerProxy>
        <hashTree/>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="GET /api/users">
          <stringProp name="HTTPSampler.domain">${BASE_URL}</stringProp>
          <stringProp name="HTTPSampler.path">/api/users?page=1&amp;limit=20</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <stringProp name="HTTPSampler.protocol">https</stringProp>
        </HTTPSamplerProxy>
        <hashTree/>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="POST /api/users">
          <stringProp name="HTTPSampler.domain">${BASE_URL}</stringProp>
          <stringProp name="HTTPSampler.path">/api/users</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <stringProp name="HTTPSampler.protocol">https</stringProp>
          <boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments">
            <collectionProp name="Arguments.arguments">
              <elementProp name="" elementType="HTTPArgument">
                <stringProp name="Argument.value">{"name":"LoadTest User ${__threadNum}","email":"loadtest${__threadNum}@example.com"}</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
        </HTTPSamplerProxy>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Headers">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="Content-Type" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">application/json</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Save this as load-test.jmx. The ${__P(base_url,...)} syntax allows overriding the target URL from Azure Load Testing without modifying the JMX file.

Validating Locally

Run a quick validation with a small number of users:

jmeter -n -t load-test.jmx -Jbase_url=https://api.example.com -Jduration=30 -l results.jtl

The -n flag runs in non-GUI mode. Check results.jtl for errors before uploading to Azure.

Setting Up Azure Load Testing

Creating a Load Testing Resource

# Create a resource group (if needed)
az group create --name rg-loadtesting --location eastus

# Create the Azure Load Testing resource
az load create \
  --name my-load-tests \
  --resource-group rg-loadtesting \
  --location eastus

Creating a Test via the Portal

1. Navigate to the Azure Load Testing resource in the Azure portal
2. Click Tests > Create > Upload a JMeter script
3. Upload your .jmx file
4. Configure:
   • Engine instances: Number of parallel JMeter engines (each runs the full thread count)
   • Test duration: Override the JMX duration
   • Environment variables: Pass variables like base_url to the JMX script
5. Optionally add server-side monitoring by selecting Azure resources (App Service, Functions, AKS) to monitor during the test
6. Click Run to execute

Understanding Engine Instances

Each engine instance runs an independent copy of your JMeter test plan. If your test plan has 50 threads and you use 4 engine instances, the total virtual users is 200 (50 x 4). This is how you scale beyond what a single JMeter instance can handle.

For Node.js APIs, I typically start with:

• 50 threads per engine for API endpoints
• 2 engine instances for baseline tests
• 10+ engine instances for peak load simulation

Configuring Test Parameters

Environment Variables and Secrets

Pass configuration to your JMeter scripts through environment variables:

# load-test-config.yaml
testPlan: load-test.jmx
engineInstances: 4
env:
  - name: base_url
    value: https://api-staging.example.com
  - name: duration
    value: "300"
secrets:
  - name: api_key
    value: https://my-keyvault.vault.azure.net/secrets/api-key

The secrets reference Azure Key Vault secrets, so sensitive values like API keys never appear in configuration files.

CSV Data Files

JMeter tests often need CSV files for data-driven scenarios (user credentials, test data). Upload them alongside the JMX file:

username,password
[email protected],TestP@ss1
[email protected],TestP@ss2
[email protected],TestP@ss3

Reference the CSV in your JMX file with a CSV Data Set Config element. Azure Load Testing distributes the CSV across engine instances automatically.

Pipeline Integration

The AzureLoadTest Task

Azure Pipelines has a dedicated task for running load tests:

trigger:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

steps:
  - task: AzureLoadTest@1
    inputs:
      azureSubscription: my-azure-connection
      loadTestConfigFile: tests/load/load-test-config.yaml
      loadTestResource: my-load-tests
      resourceGroup: rg-loadtesting
    displayName: Run load test

  - task: PublishBuildArtifacts@1
    inputs:
      pathToPublish: $(System.DefaultWorkingDirectory)/loadTest
      artifactName: load-test-results
    condition: always()
    displayName: Publish load test results

The task runs the load test, waits for completion, and downloads the results. If pass/fail criteria are configured and the test fails, the task returns a non-zero exit code.

Load Test Configuration File

The configuration file defines the test plan, engine count, and pass/fail criteria:

# load-test-config.yaml
version: v0.1
testId: api-performance-test
testPlan: load-test.jmx
engineInstances: 4
failureCriteria:
  - avg(response_time_ms) > 500
  - percentage(error) > 5
  - p99(response_time_ms) > 2000
env:
  - name: base_url
    value: https://api-staging.example.com
  - name: duration
    value: "300"

Pass/Fail Criteria

The failureCriteria section defines conditions that fail the pipeline:

Metric	Description	Example
avg(response_time_ms)	Average response time	> 500
p90(response_time_ms)	90th percentile response time	> 1000
p95(response_time_ms)	95th percentile response time	> 1500
p99(response_time_ms)	99th percentile response time	> 2000
percentage(error)	Error rate percentage	> 5
requests_per_second	Throughput	< 100

When any criterion is violated, the load test is marked as failed, and the pipeline task returns a failure status.

Full Pipeline with Deploy + Load Test

trigger:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

stages:
  - stage: Deploy
    jobs:
      - job: DeployStaging
        steps:
          - task: AzureWebApp@1
            inputs:
              azureSubscription: my-azure-connection
              appType: webAppLinux
              appName: my-api-staging
              package: $(System.DefaultWorkingDirectory)/dist
            displayName: Deploy to staging

          - script: |
              sleep 30
              HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://my-api-staging.azurewebsites.net/health)
              if [ "$HTTP_STATUS" != "200" ]; then
                echo "Health check failed: $HTTP_STATUS"
                exit 1
              fi
            displayName: Wait and verify health

  - stage: LoadTest
    dependsOn: Deploy
    jobs:
      - job: RunLoadTest
        steps:
          - task: AzureLoadTest@1
            inputs:
              azureSubscription: my-azure-connection
              loadTestConfigFile: tests/load/load-test-config.yaml
              loadTestResource: my-load-tests
              resourceGroup: rg-loadtesting
              env: |
                [
                  { "name": "base_url", "value": "https://my-api-staging.azurewebsites.net" }
                ]
            displayName: Run performance test

          - task: PublishBuildArtifacts@1
            inputs:
              pathToPublish: $(System.DefaultWorkingDirectory)/loadTest
              artifactName: load-test-results
            condition: always()

  - stage: PromoteToProduction
    dependsOn: LoadTest
    condition: succeeded()
    jobs:
      - deployment: DeployProd
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: my-azure-connection
                    appType: webAppLinux
                    appName: my-api-production
                    package: $(Pipeline.Workspace)/drop

This pipeline deploys to staging, runs a load test against staging, and only promotes to production if the load test passes all criteria.

Analyzing Results

Azure Portal Dashboard

After a test run, the Azure Load Testing dashboard shows:

• Response time percentiles (p50, p90, p95, p99) over time
• Throughput (requests per second) over time
• Error rate over time
• Virtual users ramp-up and steady state
• Per-request breakdown showing which API endpoints are slow

Server-Side Metrics

When you enable Application Insights integration, the results dashboard also shows:

• CPU and memory utilization of the target application
• Database query performance
• Dependency call durations
• Exception rates correlated with load patterns

This is the killer feature. You can see that the p99 response time spiked at minute 3, and at the same time, the database CPU hit 95%. Without server-side correlation, you would be guessing about the cause.

Exporting Results

Download the raw JMeter .jtl results file for custom analysis:

// analyze-results.js -- Parse JMeter JTL results
var fs = require("fs");
var readline = require("readline");

var resultsFile = process.argv[2] || "results.jtl";

if (!fs.existsSync(resultsFile)) {
  console.error("Results file not found: " + resultsFile);
  process.exit(1);
}

var stats = {
  total: 0,
  success: 0,
  failure: 0,
  responseTimes: [],
  byEndpoint: {}
};

var rl = readline.createInterface({
  input: fs.createReadStream(resultsFile),
  crlfDelay: Infinity
});

var isFirstLine = true;

rl.on("line", function(line) {
  if (isFirstLine) { isFirstLine = false; return; } // skip CSV header

  var fields = line.split(",");
  if (fields.length < 12) return;

  var elapsed = parseInt(fields[1]);
  var label = fields[2];
  var success = fields[7] === "true";

  stats.total++;
  if (success) stats.success++;
  else stats.failure++;

  stats.responseTimes.push(elapsed);

  if (!stats.byEndpoint[label]) {
    stats.byEndpoint[label] = { count: 0, total: 0, failures: 0, times: [] };
  }
  stats.byEndpoint[label].count++;
  stats.byEndpoint[label].total += elapsed;
  stats.byEndpoint[label].times.push(elapsed);
  if (!success) stats.byEndpoint[label].failures++;
});

rl.on("close", function() {
  stats.responseTimes.sort(function(a, b) { return a - b; });

  function percentile(arr, p) {
    var idx = Math.ceil(arr.length * p / 100) - 1;
    return arr[Math.max(0, idx)];
  }

  console.log("Load Test Results Analysis");
  console.log("==========================");
  console.log("Total requests: " + stats.total);
  console.log("Successful: " + stats.success);
  console.log("Failed: " + stats.failure);
  console.log("Error rate: " + (stats.failure / stats.total * 100).toFixed(2) + "%");
  console.log("");
  console.log("Response Times:");
  console.log("  Average: " + Math.round(stats.responseTimes.reduce(function(a, b) { return a + b; }, 0) / stats.total) + "ms");
  console.log("  p50: " + percentile(stats.responseTimes, 50) + "ms");
  console.log("  p90: " + percentile(stats.responseTimes, 90) + "ms");
  console.log("  p95: " + percentile(stats.responseTimes, 95) + "ms");
  console.log("  p99: " + percentile(stats.responseTimes, 99) + "ms");
  console.log("  Max: " + stats.responseTimes[stats.responseTimes.length - 1] + "ms");
  console.log("");
  console.log("By Endpoint:");

  Object.keys(stats.byEndpoint).forEach(function(label) {
    var ep = stats.byEndpoint[label];
    ep.times.sort(function(a, b) { return a - b; });
    var avg = Math.round(ep.total / ep.count);
    console.log("  " + label);
    console.log("    Requests: " + ep.count + " | Errors: " + ep.failures + " | Avg: " + avg + "ms | p95: " + percentile(ep.times, 95) + "ms");
  });
});

node analyze-results.js loadTest/results.jtl

# Output:
# Load Test Results Analysis
# ==========================
# Total requests: 45000
# Successful: 44820
# Failed: 180
# Error rate: 0.40%
#
# Response Times:
#   Average: 142ms
#   p50: 98ms
#   p90: 285ms
#   p95: 412ms
#   p99: 890ms
#   Max: 3421ms
#
# By Endpoint:
#   GET /api/health
#     Requests: 15000 | Errors: 0 | Avg: 12ms | p95: 25ms
#   GET /api/users
#     Requests: 15000 | Errors: 30 | Avg: 198ms | p95: 520ms
#   POST /api/users
#     Requests: 15000 | Errors: 150 | Avg: 216ms | p95: 690ms

Complete Working Example

A complete load testing setup with a JMeter test plan, configuration file, pipeline, and results analysis:

Test Configuration

# tests/load/load-test-config.yaml
version: v0.1
testId: api-regression-load-test
testPlan: load-test.jmx
engineInstances: 2
failureCriteria:
  - avg(response_time_ms) > 300
  - p95(response_time_ms) > 1000
  - p99(response_time_ms) > 2000
  - percentage(error) > 2
env:
  - name: base_url
    value: https://api-staging.example.com
  - name: duration
    value: "180"

Pipeline

# azure-pipelines-load-test.yml
trigger: none  # Run manually or on schedule

schedules:
  - cron: "0 2 * * 1-5"
    displayName: Nightly load test
    branches:
      include:
        - main
    always: true

pool:
  vmImage: ubuntu-latest

steps:
  - task: AzureLoadTest@1
    inputs:
      azureSubscription: my-azure-connection
      loadTestConfigFile: tests/load/load-test-config.yaml
      loadTestResource: my-load-tests
      resourceGroup: rg-loadtesting
    displayName: Run nightly load test

  - task: PublishBuildArtifacts@1
    inputs:
      pathToPublish: $(System.DefaultWorkingDirectory)/loadTest
      artifactName: load-test-results
    condition: always()
    displayName: Archive results

Common Issues and Troubleshooting

1. JMeter Script Fails with "Connection Refused"

Error:

java.net.ConnectException: Connection refused

The target URL is not reachable from the Azure Load Testing infrastructure. Common causes: the target is behind a firewall, the URL is wrong, or the application is not deployed. For private endpoints, configure VNet integration in the Azure Load Testing resource.

2. Test Runs But Shows 0 Requests

Error: The test completes but the dashboard shows zero requests.

The JMeter script has an error -- typically a misconfigured server name or path. Validate the script locally with jmeter -n -t test.jmx before uploading. Check that environment variable references (${__P(base_url,...)}) resolve correctly.

3. Pass/Fail Criteria Not Evaluated

Error: The test runs but criteria are not applied.

The failureCriteria section in the config YAML must use the exact metric names. Common mistakes: using response_time instead of response_time_ms, or error_rate instead of percentage(error). Check the Azure Load Testing documentation for exact metric names.

4. Engine Instance Count Causes Target Overload

Error: The target application crashes or returns 503 errors during the test.

You are generating more load than the application can handle. Start with 1-2 engine instances and increase gradually. Use the ramp-up period in your JMeter thread group to avoid sudden spikes. Monitor server-side CPU and memory during the test.

5. Results File Too Large to Download

Error: The pipeline times out trying to download results.

Long-running tests with high throughput produce large .jtl files. Configure JMeter to sample results instead of recording every request, or increase the pipeline task timeout. In the JMX file, add a Simple Data Writer with a sample interval instead of logging every request.

Best Practices

1. Run load tests on every deploy to staging. Do not wait for manual performance testing. Automated load tests in the pipeline catch regressions before they reach production.

2. Set realistic failure criteria based on SLOs. If your SLO is 200ms p95 response time, set the criterion to p95(response_time_ms) > 200. Do not guess -- derive thresholds from your production SLAs.

3. Use parameterized JMeter scripts. Pass the target URL, duration, and thread count through environment variables. This lets you reuse the same JMX file across staging, pre-production, and production environments.

4. Start small and scale up. Begin with 50 virtual users on 1 engine instance. Validate the test works, then increase to your target load. Jumping to 1,000 users immediately masks configuration problems.

5. Enable server-side metrics. Correlating client-side response times with server CPU, memory, database latency, and dependency performance is essential for diagnosing bottlenecks. Configure Application Insights integration.

6. Run baseline tests before making changes. Establish a performance baseline on the current release. Run the same test after changes and compare. Without a baseline, you cannot tell if performance improved or degraded.

7. Schedule nightly load tests. Use pipeline schedules to run load tests against staging every night. This catches performance regressions from code changes merged during the day.

8. Test with realistic data volumes. A load test against an empty database is meaningless. Seed your staging database with production-like data volumes before running tests.

9. Use ramp-up periods to simulate realistic traffic. Real traffic does not jump from 0 to 1,000 users instantly. Use a 60-120 second ramp-up to simulate gradual load increase.

10. Archive results for trend analysis. Publish load test results as build artifacts. Compare results across builds to track performance trends over weeks and months.

References

• Azure Load Testing Documentation
• Create a JMeter-Based Load Test
• Azure Load Testing Pipeline Task
• Pass/Fail Criteria
• Apache JMeter Documentation
• Server-Side Metrics in Azure Load Testing
• VNet Integration for Private Endpoints