Quality Metrics and Test Coverage Tracking
Track software quality with code coverage, test pass rates, defect density, and traceability metrics in Azure DevOps
Quality Metrics and Test Coverage Tracking
Measuring software quality without data is guesswork. Quality metrics give engineering teams concrete numbers to track improvements, identify regressions, and make informed decisions about release readiness. Azure DevOps provides a rich set of tools for collecting code coverage, tracking test results, mapping requirements to tests, and building dashboards that surface the metrics that actually matter.
This article covers the full pipeline from collecting raw coverage data in your CI builds to building a custom quality scorecard that aggregates everything into a single view. We will use Node.js throughout, with Istanbul/nyc for coverage collection and the Azure DevOps REST APIs for pulling metrics programmatically.
Prerequisites
- An Azure DevOps organization with a project configured
- A Node.js application with unit tests (Mocha, Jest, or similar)
- Azure Pipelines configured for your repository
- A Personal Access Token (PAT) with read access to Test Management, Analytics, and Build
- Node.js 18 or later installed locally
- Basic familiarity with YAML pipeline syntax
Defining Quality Metrics for Software Teams
Before instrumenting anything, you need to decide what you are measuring and why. Too many teams collect coverage numbers without understanding what those numbers represent. Here are the metrics that matter for most engineering organizations.
Code Coverage measures the percentage of your source code exercised by automated tests. It comes in several flavors: line coverage, branch coverage, function coverage, and statement coverage. Branch coverage is the most meaningful because it tells you whether both sides of every conditional have been tested. Line coverage alone can be misleading because a single line with a ternary operator might have two paths that line coverage treats as one.
Test Pass Rate is the ratio of passing tests to total tests over a given period. A consistent 100% pass rate on your main branch is the baseline expectation. What matters more is the trend on feature branches and the flakiness rate, which is the percentage of tests that intermittently fail without code changes.
Defect Density measures the number of bugs per unit of code, typically per thousand lines. This tells you which modules are problematic and where your testing strategy has gaps.
Mean Time to Detect (MTTD) is the average elapsed time between when a bug is introduced and when it is discovered. Shorter MTTD means your testing pipeline catches issues quickly.
Mean Time to Resolve (MTTR) is the average time from bug discovery to fix deployment. Together with MTTD, this gives you the full picture of your defect lifecycle.
Requirement Coverage maps test cases to requirements or user stories. A requirement without linked test cases is a gap in your quality strategy. Azure Test Plans makes this traceability explicit.
Code Coverage Collection in Azure Pipelines
The first step is generating coverage data in your CI pipeline. For Node.js projects, Istanbul (via its command-line interface nyc) is the standard tool.
Istanbul/nyc for Node.js Coverage
Install nyc as a dev dependency:
npm install --save-dev nyc
Configure nyc in your package.json:
{
"nyc": {
"reporter": ["text", "cobertura", "html"],
"include": ["src/**/*.js"],
"exclude": ["src/**/*.test.js", "src/**/*.spec.js"],
"branches": 80,
"lines": 80,
"functions": 80,
"statements": 80,
"check-coverage": true,
"all": true
},
"scripts": {
"test": "mocha --recursive ./test",
"test:coverage": "nyc npm test"
}
}
The all: true setting is important. Without it, nyc only reports coverage for files that are actually loaded during tests. If you have a utility module that no test imports, it will not appear in the report at all, giving you an inflated coverage number. With all: true, every file matching the include pattern shows up, even at 0% coverage.
The cobertura reporter generates an XML file that Azure Pipelines knows how to parse. The html reporter gives you a browsable local report. The text reporter prints a summary to the console during the build.
Run it locally to verify:
npx nyc npm test
You will see output like:
----------|---------|----------|---------|---------|
File | % Stmts | % Branch | % Funcs | % Lines |
----------|---------|----------|---------|---------|
All files | 87.32 | 74.19 | 91.67 | 87.32 |
src/ | 87.32 | 74.19 | 91.67 | 87.32 |
api.js | 92.11 | 83.33 | 100.00 | 92.11 |
db.js | 78.57 | 60.00 | 80.00 | 78.57 |
util.js | 95.00 | 85.71 | 100.00 | 95.00 |
----------|---------|----------|---------|---------|
Publishing Coverage Reports in Azure Pipelines
Here is a YAML pipeline that runs tests, collects coverage, and publishes the results to Azure DevOps:
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool@0
inputs:
versionSpec: '20.x'
displayName: 'Install Node.js'
- script: npm ci
displayName: 'Install dependencies'
- script: npx nyc --reporter=cobertura --reporter=text npm test
displayName: 'Run tests with coverage'
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-results.xml'
mergeTestResults: true
displayName: 'Publish test results'
condition: always()
- task: PublishCodeCoverageResults@2
inputs:
summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml'
displayName: 'Publish code coverage'
condition: always()
For the test results step, you need your test runner to output JUnit XML. With Mocha, add the mocha-junit-reporter:
npm install --save-dev mocha-junit-reporter
Then update your test script:
{
"scripts": {
"test": "mocha --recursive ./test --reporter mocha-junit-reporter --reporter-options mochaFile=test-results.xml"
}
}
After the pipeline runs, the Build Summary page in Azure DevOps shows a Code Coverage tab with your line, branch, function, and statement percentages. It also renders the HTML report inline so you can drill into individual files.
Coverage Trends and Thresholds
A single coverage number is not useful. What matters is the trend. Azure DevOps automatically tracks coverage across builds, so you can see whether your team is improving or regressing over time.
To enforce thresholds, use the nyc check-coverage flag we configured earlier. If coverage drops below 80% on any metric, the build fails:
npx nyc check-coverage --branches 80 --lines 80 --functions 80 --statements 80
Put this as a separate pipeline step so it runs after coverage collection:
- script: npx nyc check-coverage --branches 80 --lines 80 --functions 80 --statements 80
displayName: 'Enforce coverage thresholds'
A failing build because coverage dropped is a clear signal. It forces the team to either write tests for new code or have a deliberate conversation about lowering the threshold.
Some teams set a ratchet policy: the threshold is automatically raised to match the current coverage level minus a small buffer. This prevents coverage from ever going down. You can implement this with a script that reads the current coverage from the Cobertura XML and updates a config file.
Requirement-Based Test Coverage in Azure Test Plans
Code coverage tells you whether your tests exercise the code. Requirement coverage tells you whether your tests verify the requirements. These are fundamentally different concerns.
Azure Test Plans lets you link test cases to work items (user stories, requirements, or features). A test case linked to a user story provides traceability. If every user story has at least one linked test case that passes, you have requirement coverage.
To set this up:
- Create test plans and test suites in Azure Test Plans
- Create test cases as work items
- Link each test case to the relevant requirement using the "Tested By / Tests" link type
- Run the test cases (manually or via automated test runs)
- View the Requirements Quality widget on your dashboard
Traceability Matrices
A traceability matrix maps requirements to test cases to test results. Azure DevOps does not generate a formal matrix view out of the box, but you can build one using the REST APIs or OData queries.
The matrix answers three questions:
- Which requirements have no test cases? (coverage gap)
- Which test cases are not linked to any requirement? (orphaned tests)
- Which requirements have failing test cases? (quality risk)
We will build this programmatically in the complete working example below.
Quality Dashboards with Widgets
Azure DevOps dashboards support built-in widgets for quality metrics:
- Test Results Trend shows pass/fail/not run counts over time
- Code Coverage shows the coverage percentage for the latest build
- Requirements Quality shows requirements with linked test outcomes
- Chart for Test Plans visualizes test plan progress
- Burndown can show bug burndown for a sprint
To create a quality dashboard:
- Navigate to Overview > Dashboards
- Create a new dashboard named "Quality Metrics"
- Add the Test Results Trend widget (configure it to show the last 14 builds)
- Add the Code Coverage widget (point it at your build pipeline)
- Add the Requirements Quality widget (select your test plan)
- Add a Query Results widget showing active bugs by severity
Custom Quality Gates
Beyond the built-in widgets, you can create custom quality gates that block deployments based on metric thresholds. Use a pipeline stage with conditions:
stages:
- stage: QualityGate
displayName: 'Quality Gate'
jobs:
- job: CheckMetrics
steps:
- script: |
node scripts/check-quality-gate.js
displayName: 'Evaluate quality metrics'
env:
AZURE_DEVOPS_PAT: $(AzureDevOpsPAT)
AZURE_DEVOPS_ORG: $(System.CollectionUri)
PROJECT: $(System.TeamProject)
BUILD_ID: $(Build.BuildId)
The check-quality-gate.js script queries the Azure DevOps APIs and exits with a non-zero code if any metric fails:
var https = require("https");
var url = require("url");
var pat = process.env.AZURE_DEVOPS_PAT;
var org = process.env.AZURE_DEVOPS_ORG;
var project = process.env.PROJECT;
var buildId = process.env.BUILD_ID;
var thresholds = {
lineCoverage: 80,
branchCoverage: 75,
testPassRate: 100,
maxActiveCriticalBugs: 0
};
function makeRequest(apiUrl, callback) {
var parsed = url.parse(apiUrl);
var options = {
hostname: parsed.hostname,
path: parsed.path,
headers: {
"Authorization": "Basic " + Buffer.from(":" + pat).toString("base64"),
"Content-Type": "application/json"
}
};
https.get(options, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() {
callback(null, JSON.parse(body));
});
}).on("error", function(err) {
callback(err);
});
}
function checkCoverage(callback) {
var apiUrl = org + project + "/_apis/test/codecoverage?buildId=" + buildId + "&api-version=7.1";
makeRequest(apiUrl, function(err, data) {
if (err) return callback(err);
if (!data.coverageData || data.coverageData.length === 0) {
return callback(new Error("No coverage data found for build " + buildId));
}
var modules = data.coverageData[0].modules;
var totalLines = 0;
var coveredLines = 0;
var totalBranches = 0;
var coveredBranches = 0;
modules.forEach(function(mod) {
mod.statistics.forEach(function(stat) {
if (stat.label === "Lines") {
totalLines += stat.total;
coveredLines += stat.covered;
}
if (stat.label === "Branches") {
totalBranches += stat.total;
coveredBranches += stat.covered;
}
});
});
var linePct = totalLines > 0 ? (coveredLines / totalLines * 100).toFixed(2) : 0;
var branchPct = totalBranches > 0 ? (coveredBranches / totalBranches * 100).toFixed(2) : 0;
console.log("Line coverage: " + linePct + "%");
console.log("Branch coverage: " + branchPct + "%");
callback(null, {
lineCoverage: parseFloat(linePct),
branchCoverage: parseFloat(branchPct)
});
});
}
function checkTestPassRate(callback) {
var apiUrl = org + project + "/_apis/test/runs?buildId=" + buildId + "&api-version=7.1";
makeRequest(apiUrl, function(err, data) {
if (err) return callback(err);
if (!data.value || data.value.length === 0) {
return callback(new Error("No test runs found for build " + buildId));
}
var totalPassed = 0;
var totalTests = 0;
var remaining = data.value.length;
data.value.forEach(function(run) {
totalPassed += run.passedTests || 0;
totalTests += run.totalTests || 0;
remaining--;
if (remaining === 0) {
var passRate = totalTests > 0 ? (totalPassed / totalTests * 100).toFixed(2) : 0;
console.log("Test pass rate: " + passRate + "% (" + totalPassed + "/" + totalTests + ")");
callback(null, parseFloat(passRate));
}
});
});
}
function evaluateGate() {
var failures = [];
checkCoverage(function(err, coverage) {
if (err) {
console.error("Coverage check failed:", err.message);
process.exit(1);
}
if (coverage.lineCoverage < thresholds.lineCoverage) {
failures.push("Line coverage " + coverage.lineCoverage + "% is below threshold " + thresholds.lineCoverage + "%");
}
if (coverage.branchCoverage < thresholds.branchCoverage) {
failures.push("Branch coverage " + coverage.branchCoverage + "% is below threshold " + thresholds.branchCoverage + "%");
}
checkTestPassRate(function(err, passRate) {
if (err) {
console.error("Test pass rate check failed:", err.message);
process.exit(1);
}
if (passRate < thresholds.testPassRate) {
failures.push("Test pass rate " + passRate + "% is below threshold " + thresholds.testPassRate + "%");
}
if (failures.length > 0) {
console.error("\nQuality gate FAILED:");
failures.forEach(function(f) { console.error(" - " + f); });
process.exit(1);
}
console.log("\nQuality gate PASSED");
process.exit(0);
});
});
}
evaluateGate();
OData Queries for Quality Metrics
Azure DevOps Analytics provides an OData endpoint for querying historical data. This is how you build trend reports that go beyond what the built-in widgets offer.
The Analytics OData endpoint is at:
https://analytics.dev.azure.com/{org}/{project}/_odata/v4.0-preview/
Useful entities for quality metrics:
TestResultsDailyfor test pass/fail trendsTestRunsfor individual test run detailsWorkItemsfor bug counts and defect densityPipelineRunsfor build success rates
Example OData query to get daily test results for the last 30 days:
https://analytics.dev.azure.com/{org}/{project}/_odata/v4.0-preview/TestResultsDaily?
$filter=DateSK ge 20260115 and Workflow/WorkflowName eq 'Build'
&$apply=groupby((DateSK, Outcome), aggregate($count as Count))
&$orderby=DateSK
You can call this from Node.js to feed a custom dashboard or alerting system.
Defect Density Tracking
Defect density is calculated as: (Number of defects / Size of the code module) * 1000
Size can be measured in lines of code (LOC), function points, or story points. LOC is the simplest and most common for engineering teams.
Track defect density per module over time to identify trouble spots. A module with rising defect density needs attention, whether that is refactoring, better test coverage, or a code review process change.
Here is a utility that calculates defect density from Azure DevOps work items and a lines-of-code count:
var https = require("https");
function getActiveBugs(org, project, pat, areaPath, callback) {
var wiql = JSON.stringify({
query: "SELECT [System.Id] FROM WorkItems WHERE " +
"[System.WorkItemType] = 'Bug' AND " +
"[System.State] <> 'Closed' AND " +
"[System.AreaPath] UNDER '" + areaPath + "'"
});
var options = {
hostname: "dev.azure.com",
path: "/" + org + "/" + project + "/_apis/wit/wiql?api-version=7.1",
method: "POST",
headers: {
"Authorization": "Basic " + Buffer.from(":" + pat).toString("base64"),
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(wiql)
}
};
var req = https.request(options, function(res) {
var body = "";
res.on("data", function(chunk) { body += chunk; });
res.on("end", function() {
var data = JSON.parse(body);
callback(null, data.workItems ? data.workItems.length : 0);
});
});
req.on("error", function(err) { callback(err); });
req.write(wiql);
req.end();
}
function calculateDefectDensity(bugCount, linesOfCode) {
if (linesOfCode === 0) return 0;
return (bugCount / linesOfCode * 1000).toFixed(2);
}
module.exports = {
getActiveBugs: getActiveBugs,
calculateDefectDensity: calculateDefectDensity
};
Mean Time to Detect and Resolve Metrics
MTTD and MTTR require tracking timestamps on bug work items. The key fields are:
- Created Date: when the bug was filed (detection time)
- Changed Date when State moved to "Resolved" or "Closed": resolution time
- Custom field or tag for when the bug was introduced (if you track this)
A practical approximation: MTTD is the time between the commit that introduced the bug (approximated by the sprint start date or a tagged commit) and the Created Date. MTTR is the time between Created Date and the date the State changed to "Resolved."
You can query these from the Analytics OData endpoint:
https://analytics.dev.azure.com/{org}/{project}/_odata/v4.0-preview/WorkItems?
$filter=WorkItemType eq 'Bug' and State eq 'Closed' and ClosedDate ge 2026-01-01Z
&$select=WorkItemId,CreatedDate,ClosedDate,ResolvedDate,Severity
Then calculate the averages in your Node.js dashboard code.
Complete Working Example: Quality Metrics Dashboard
This Node.js application aggregates code coverage, test pass rates, defect density, and traceability data from Azure DevOps APIs into a single quality scorecard.
var https = require("https");
var http = require("http");
var url = require("url");
// Configuration
var config = {
org: process.env.AZURE_DEVOPS_ORG || "myorg",
project: process.env.AZURE_DEVOPS_PROJECT || "myproject",
pat: process.env.AZURE_DEVOPS_PAT,
buildDefinitionId: process.env.BUILD_DEFINITION_ID || "1",
testPlanId: process.env.TEST_PLAN_ID || "1",
areaPath: process.env.AREA_PATH || "myproject",
port: process.env.PORT || 3000
};
var baseUrl = "https://dev.azure.com/" + config.org + "/" + config.project;
var analyticsUrl = "https://analytics.dev.azure.com/" + config.org + "/" + config.project;
function apiRequest(requestUrl, method, body, callback) {
if (typeof body === "function") {
callback = body;
body = null;
method = "GET";
}
if (typeof method === "function") {
callback = method;
method = "GET";
body = null;
}
var parsed = url.parse(requestUrl);
var options = {
hostname: parsed.hostname,
path: parsed.path,
method: method || "GET",
headers: {
"Authorization": "Basic " + Buffer.from(":" + config.pat).toString("base64"),
"Content-Type": "application/json"
}
};
if (body) {
var bodyStr = JSON.stringify(body);
options.headers["Content-Length"] = Buffer.byteLength(bodyStr);
}
var req = https.request(options, function(res) {
var chunks = "";
res.on("data", function(chunk) { chunks += chunk; });
res.on("end", function() {
try {
callback(null, JSON.parse(chunks));
} catch (e) {
callback(new Error("Failed to parse response: " + chunks.substring(0, 200)));
}
});
});
req.on("error", function(err) { callback(err); });
if (body) req.write(JSON.stringify(body));
req.end();
}
// Fetch latest build coverage
function getLatestCoverage(callback) {
var buildsUrl = baseUrl + "/_apis/build/builds?definitions=" +
config.buildDefinitionId + "&$top=1&statusFilter=completed&api-version=7.1";
apiRequest(buildsUrl, function(err, builds) {
if (err) return callback(err);
if (!builds.value || builds.value.length === 0) {
return callback(null, { lineCoverage: 0, branchCoverage: 0, buildId: null });
}
var buildId = builds.value[0].id;
var coverageUrl = baseUrl + "/_apis/test/codecoverage?buildId=" + buildId + "&api-version=7.1";
apiRequest(coverageUrl, function(err, data) {
if (err) return callback(err);
if (!data.coverageData || data.coverageData.length === 0) {
return callback(null, { lineCoverage: 0, branchCoverage: 0, buildId: buildId });
}
var totalLines = 0, coveredLines = 0;
var totalBranches = 0, coveredBranches = 0;
data.coverageData[0].modules.forEach(function(mod) {
mod.statistics.forEach(function(stat) {
if (stat.label === "Lines") {
totalLines += stat.total;
coveredLines += stat.covered;
}
if (stat.label === "Branches") {
totalBranches += stat.total;
coveredBranches += stat.covered;
}
});
});
callback(null, {
lineCoverage: totalLines > 0 ? (coveredLines / totalLines * 100).toFixed(2) : 0,
branchCoverage: totalBranches > 0 ? (coveredBranches / totalBranches * 100).toFixed(2) : 0,
buildId: buildId,
totalLines: totalLines,
coveredLines: coveredLines
});
});
});
}
// Fetch test pass rates for recent builds
function getTestPassRates(numBuilds, callback) {
var buildsUrl = baseUrl + "/_apis/build/builds?definitions=" +
config.buildDefinitionId + "&$top=" + numBuilds +
"&statusFilter=completed&api-version=7.1";
apiRequest(buildsUrl, function(err, builds) {
if (err) return callback(err);
if (!builds.value || builds.value.length === 0) {
return callback(null, []);
}
var results = [];
var remaining = builds.value.length;
builds.value.forEach(function(build) {
var runsUrl = baseUrl + "/_apis/test/runs?buildUri=" +
encodeURIComponent(build.uri) + "&api-version=7.1";
apiRequest(runsUrl, function(err, runs) {
var passed = 0, total = 0;
if (!err && runs.value) {
runs.value.forEach(function(run) {
passed += run.passedTests || 0;
total += run.totalTests || 0;
});
}
results.push({
buildNumber: build.buildNumber,
buildId: build.id,
date: build.finishTime,
passedTests: passed,
totalTests: total,
passRate: total > 0 ? (passed / total * 100).toFixed(2) : "N/A"
});
remaining--;
if (remaining === 0) {
results.sort(function(a, b) {
return new Date(a.date) - new Date(b.date);
});
callback(null, results);
}
});
});
});
}
// Fetch defect metrics
function getDefectMetrics(callback) {
var wiqlBody = {
query: "SELECT [System.Id], [System.CreatedDate], [Microsoft.VSTS.Common.ResolvedDate], " +
"[Microsoft.VSTS.Common.Severity], [System.State] " +
"FROM WorkItems WHERE [System.WorkItemType] = 'Bug' AND " +
"[System.AreaPath] UNDER '" + config.areaPath + "' " +
"ORDER BY [System.CreatedDate] DESC"
};
var wiqlUrl = baseUrl + "/_apis/wit/wiql?api-version=7.1";
apiRequest(wiqlUrl, "POST", wiqlBody, function(err, wiqlResult) {
if (err) return callback(err);
if (!wiqlResult.workItems || wiqlResult.workItems.length === 0) {
return callback(null, { totalBugs: 0, activeBugs: 0, avgMttr: 0, bugsBySeverity: {} });
}
var ids = wiqlResult.workItems.slice(0, 200).map(function(wi) {
return wi.id;
});
var batchUrl = baseUrl + "/_apis/wit/workitemsbatch?api-version=7.1";
var batchBody = {
ids: ids,
fields: [
"System.Id", "System.State", "System.CreatedDate",
"Microsoft.VSTS.Common.ResolvedDate",
"Microsoft.VSTS.Common.Severity"
]
};
apiRequest(batchUrl, "POST", batchBody, function(err, workItems) {
if (err) return callback(err);
var activeBugs = 0;
var resolvedBugs = [];
var bugsBySeverity = {};
workItems.value.forEach(function(wi) {
var fields = wi.fields;
var state = fields["System.State"];
var severity = fields["Microsoft.VSTS.Common.Severity"] || "Unspecified";
var created = fields["System.CreatedDate"];
var resolved = fields["Microsoft.VSTS.Common.ResolvedDate"];
bugsBySeverity[severity] = (bugsBySeverity[severity] || 0) + 1;
if (state === "Active" || state === "New") {
activeBugs++;
}
if (resolved && created) {
var mttr = new Date(resolved) - new Date(created);
resolvedBugs.push(mttr);
}
});
var avgMttr = 0;
if (resolvedBugs.length > 0) {
var totalMttr = resolvedBugs.reduce(function(sum, val) { return sum + val; }, 0);
avgMttr = totalMttr / resolvedBugs.length;
}
var mttrHours = (avgMttr / (1000 * 60 * 60)).toFixed(1);
var mttrDays = (avgMttr / (1000 * 60 * 60 * 24)).toFixed(1);
callback(null, {
totalBugs: ids.length,
activeBugs: activeBugs,
resolvedCount: resolvedBugs.length,
avgMttrHours: mttrHours,
avgMttrDays: mttrDays,
bugsBySeverity: bugsBySeverity
});
});
});
}
// Fetch requirement traceability
function getRequirementTraceability(callback) {
var testPlanUrl = baseUrl + "/_apis/testplan/Plans/" + config.testPlanId +
"/Suites?api-version=7.1";
apiRequest(testPlanUrl, function(err, suites) {
if (err) return callback(err);
if (!suites.value || suites.value.length === 0) {
return callback(null, { totalRequirements: 0, coveredRequirements: 0, coverage: 0 });
}
// Query requirements with test case links
var wiqlBody = {
query: "SELECT [System.Id] FROM WorkItems WHERE " +
"[System.WorkItemType] = 'User Story' AND " +
"[System.AreaPath] UNDER '" + config.areaPath + "' AND " +
"[System.State] <> 'Removed'"
};
var wiqlUrl = baseUrl + "/_apis/wit/wiql?api-version=7.1";
apiRequest(wiqlUrl, "POST", wiqlBody, function(err, result) {
if (err) return callback(err);
var totalRequirements = result.workItems ? result.workItems.length : 0;
var coveredRequirements = 0;
var uncoveredIds = [];
if (totalRequirements === 0) {
return callback(null, {
totalRequirements: 0,
coveredRequirements: 0,
coverage: 0,
uncoveredIds: []
});
}
var remaining = Math.min(totalRequirements, 100);
var ids = result.workItems.slice(0, 100);
ids.forEach(function(wi) {
var relUrl = baseUrl + "/_apis/wit/workitems/" + wi.id +
"?$expand=relations&api-version=7.1";
apiRequest(relUrl, function(err, item) {
var hasTestCase = false;
if (!err && item.relations) {
item.relations.forEach(function(rel) {
if (rel.rel === "Microsoft.VSTS.Common.TestedBy-Forward") {
hasTestCase = true;
}
});
}
if (hasTestCase) {
coveredRequirements++;
} else {
uncoveredIds.push(wi.id);
}
remaining--;
if (remaining === 0) {
callback(null, {
totalRequirements: totalRequirements,
coveredRequirements: coveredRequirements,
coverage: totalRequirements > 0
? (coveredRequirements / Math.min(totalRequirements, 100) * 100).toFixed(2)
: 0,
uncoveredIds: uncoveredIds
});
}
});
});
});
});
}
// Build the scorecard
function buildScorecard(callback) {
var scorecard = {};
var pending = 4;
function checkDone() {
pending--;
if (pending === 0) {
scorecard.timestamp = new Date().toISOString();
scorecard.overallHealth = calculateHealth(scorecard);
callback(null, scorecard);
}
}
getLatestCoverage(function(err, coverage) {
scorecard.coverage = err ? { error: err.message } : coverage;
checkDone();
});
getTestPassRates(10, function(err, rates) {
scorecard.testPassRates = err ? { error: err.message } : rates;
checkDone();
});
getDefectMetrics(function(err, defects) {
scorecard.defects = err ? { error: err.message } : defects;
checkDone();
});
getRequirementTraceability(function(err, traceability) {
scorecard.traceability = err ? { error: err.message } : traceability;
checkDone();
});
}
function calculateHealth(scorecard) {
var score = 100;
var reasons = [];
// Coverage checks
if (scorecard.coverage && !scorecard.coverage.error) {
if (parseFloat(scorecard.coverage.lineCoverage) < 80) {
score -= 20;
reasons.push("Line coverage below 80%");
}
if (parseFloat(scorecard.coverage.branchCoverage) < 75) {
score -= 15;
reasons.push("Branch coverage below 75%");
}
}
// Test pass rate check (latest build)
if (scorecard.testPassRates && scorecard.testPassRates.length > 0) {
var latest = scorecard.testPassRates[scorecard.testPassRates.length - 1];
if (latest.passRate !== "N/A" && parseFloat(latest.passRate) < 100) {
score -= 25;
reasons.push("Test pass rate below 100%");
}
}
// Defect check
if (scorecard.defects && !scorecard.defects.error) {
if (scorecard.defects.activeBugs > 10) {
score -= 15;
reasons.push("More than 10 active bugs");
}
if (parseFloat(scorecard.defects.avgMttrDays) > 7) {
score -= 10;
reasons.push("Average MTTR exceeds 7 days");
}
}
// Traceability check
if (scorecard.traceability && !scorecard.traceability.error) {
if (parseFloat(scorecard.traceability.coverage) < 90) {
score -= 15;
reasons.push("Requirement coverage below 90%");
}
}
var status = "HEALTHY";
if (score < 70) status = "AT RISK";
if (score < 50) status = "CRITICAL";
return { score: Math.max(0, score), status: status, reasons: reasons };
}
// HTTP server
var server = http.createServer(function(req, res) {
if (req.url === "/api/scorecard" && req.method === "GET") {
buildScorecard(function(err, scorecard) {
if (err) {
res.writeHead(500, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: err.message }));
return;
}
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify(scorecard, null, 2));
});
} else if (req.url === "/" && req.method === "GET") {
res.writeHead(200, { "Content-Type": "text/html" });
res.end(
"<html><head><title>Quality Scorecard</title></head><body>" +
"<h1>Quality Metrics Dashboard</h1>" +
"<p>GET <a href='/api/scorecard'>/api/scorecard</a> for the full quality scorecard.</p>" +
"</body></html>"
);
} else {
res.writeHead(404);
res.end("Not found");
}
});
server.listen(config.port, function() {
console.log("Quality metrics dashboard running on port " + config.port);
});
Run it:
AZURE_DEVOPS_PAT=your-pat AZURE_DEVOPS_ORG=myorg AZURE_DEVOPS_PROJECT=myproject node dashboard.js
Then visit http://localhost:3000/api/scorecard to see the aggregated quality metrics. The response includes coverage data, test pass rate trends, defect counts with MTTR, requirement traceability percentages, and an overall health score.
Common Issues and Troubleshooting
Coverage report not appearing in Azure DevOps.
The most common cause is a wrong file path in the PublishCodeCoverageResults task. The Cobertura XML must exist at the path you specify. Run ls -la coverage/ in a pipeline script step to verify the file exists. Also check that nyc is configured with the cobertura reporter.
Coverage percentage is suspiciously high.
You probably do not have all: true in your nyc configuration. Without it, files that are never imported during testing are excluded entirely from the report. A project with 50 source files where tests only touch 10 will report coverage only for those 10 files, making 80% coverage look like full coverage when it actually covers a fraction of the codebase.
Test results show zero tests in Azure DevOps.
The PublishTestResults task needs JUnit or NUnit XML format. If your test runner outputs a different format or the file path is wrong, you will see zero results. Double check the testResultsFiles glob pattern. Also make sure the test results file is generated before the publish step runs, and that the condition: always() is set so it publishes even if tests fail.
OData queries return 401 or 403 errors. The PAT needs the Analytics (read) scope. Some organizations disable Analytics entirely. Check the Organization Settings > Extensions page to verify Analytics is installed. Also confirm the PAT has not expired; they silently stop working without clear error messages.
Quality gate blocks deployment but coverage data is stale.
This happens when the coverage query uses a build ID from a previous pipeline run. Make sure you are passing the current $(Build.BuildId) to your quality gate script. If you are running the gate in a separate stage, the build ID from the triggering build must be passed explicitly as a pipeline parameter.
Requirement traceability shows zero coverage despite linked test cases. The link type matters. The correct relationship is "Tested By" (from requirement to test case). If you use a generic "Related" link, it will not show up in traceability queries. In the work item form, use the "Add link" dialog and select "Tested By" specifically.
Best Practices
Track branch coverage, not just line coverage. Line coverage can reach 90% while leaving half your conditional logic untested. Branch coverage catches this. Set your pipeline thresholds on branch coverage as the primary gate.
Set coverage thresholds on new code, not just overall. A legacy codebase at 40% coverage cannot realistically reach 80% overnight. Instead, require that every pull request maintains or improves the current level. Use tools like
nyc check-coveragewith a dynamically calculated threshold based on the previous build.Make quality dashboards visible to the whole team. Put the dashboard on a monitor in the team area, or post the scorecard to your team channel daily. Metrics that nobody sees have no effect on behavior. Visibility creates accountability.
Distinguish between flaky tests and real failures. A test that passes on retry is not a passing test. Track flakiness as a separate metric. Azure DevOps marks test results as flaky if they oscillate between pass and fail across builds. Surface this metric in your dashboard and fix flaky tests with the same urgency as real bugs.
Use requirement traceability to find untested features. Run the traceability query before each release. Any user story without a linked passing test case is a risk. This does not mean you need manual test cases for everything, but you should have at least one automated test that exercises the core scenario.
Measure MTTR by severity level. A critical bug that takes 14 days to resolve is a fundamentally different problem than a low-severity cosmetic issue taking the same time. Break your MTTR reporting down by severity so you can set different SLAs for different levels.
Automate the quality scorecard generation. Do not rely on people manually checking dashboards. Run the scorecard as a scheduled pipeline and send alerts when the health score drops below a threshold. Integrate it into your release approval workflow so releases cannot proceed when quality metrics are below acceptable levels.
Review metrics regularly in retrospectives. Quality metrics are most valuable when they drive action. Include them in your sprint retrospectives. If defect density is rising in a particular module, that is a signal to prioritize refactoring or pair programming on that area.
