Decision Support Systems with LLMs
Build decision support systems with LLM-powered analysis, multi-criteria evaluation, and explainable recommendations in Node.js.
Decision Support Systems with LLMs
Overview
Decision support systems have existed since the 1960s, but LLMs fundamentally change what they can do. Instead of rigid rule-based engines that spit out a number, you can now build systems that gather heterogeneous data, reason across multiple criteria, explain their logic in plain language, and adapt to novel situations they were never explicitly programmed for. This article walks through how to design and implement a production decision support service in Node.js that uses Claude for multi-criteria analysis, generates structured recommendations with confidence scores, and tracks outcomes so the system actually gets better over time.
Prerequisites
- Node.js v18+ installed
- Working knowledge of Express.js and async patterns
- An Anthropic API key (Claude API access)
- PostgreSQL (for decision audit trail and outcome tracking)
- Familiarity with REST API design
- Basic understanding of decision analysis concepts (weighted criteria, risk matrices)
Install the core dependencies:
npm install express anthropic pg uuid body-parser
What Decision Support Systems Do
A decision support system (DSS) is not a decision-making system. That distinction matters. The goal is to augment human judgment, not replace it. In practice, a DSS does four things:
- Gathers context — pulls in data from multiple sources so the decision-maker does not have to manually compile it.
- Analyzes trade-offs — evaluates options against weighted criteria and surfaces conflicts the human might miss.
- Generates recommendations — proposes a course of action with supporting reasoning.
- Explains itself — tells you why it recommends what it recommends, including what it is uncertain about.
The LLM component replaces what would traditionally be a complex rules engine or a statistical model. Instead of spending months encoding domain knowledge into IF/THEN rules, you describe the decision context in natural language and let the model reason about it. The trade-off is latency and cost versus flexibility — and in most business decision scenarios, a few seconds of latency is completely acceptable.
Designing a Decision Support Architecture
The architecture follows a pipeline pattern. Data flows through four stages, and each stage produces artifacts that the next stage consumes.
┌──────────────┐ ┌──────────────┐ ┌────────────────┐ ┌──────────────┐
│ Data │───>│ Analysis │───>│ Recommendation │───>│ Explanation │
│ Gathering │ │ Engine │ │ Generator │ │ & Audit │
└──────────────┘ └──────────────┘ └────────────────┘ └──────────────┘
│ │ │ │
External APIs LLM + Scoring Structured Output Decision Log
Internal DBs Multi-criteria Confidence Scores Outcome Track
User Input Risk Analysis Alternatives Feedback Loop
Here is the foundational module that ties this pipeline together:
var Anthropic = require("@anthropic-ai/sdk");
var { v4: uuidv4 } = require("uuid");
var anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
var SYSTEM_PROMPT = [
"You are an expert decision analyst. Your role is to help humans make better decisions",
"by analyzing data, evaluating trade-offs, and providing structured recommendations.",
"Always quantify uncertainty. Always present alternatives. Never pretend to have certainty",
"you do not possess. When data is missing, say so explicitly.",
"Respond ONLY with valid JSON matching the requested schema."
].join(" ");
function createDecisionContext(decisionId, title, description, options, criteria, data) {
return {
id: decisionId || uuidv4(),
title: title,
description: description,
options: options,
criteria: criteria,
contextData: data || {},
createdAt: new Date().toISOString()
};
}
module.exports = {
anthropic: anthropic,
SYSTEM_PROMPT: SYSTEM_PROMPT,
createDecisionContext: createDecisionContext
};
Implementing Multi-Criteria Analysis with LLMs
Multi-criteria decision analysis (MCDA) is the backbone of any serious DSS. You define criteria, assign weights, and score each option against each criterion. Traditionally this requires a domain expert to build scoring functions. With an LLM, you can describe the criteria in natural language and have the model do the scoring — as long as you structure the output carefully.
var { anthropic, SYSTEM_PROMPT } = require("./decision-core");
function buildAnalysisPrompt(context) {
var criteriaList = context.criteria.map(function(c) {
return "- " + c.name + " (weight: " + c.weight + "): " + c.description;
}).join("\n");
var optionsList = context.options.map(function(o) {
return "- " + o.name + ": " + o.description;
}).join("\n");
var dataSection = Object.keys(context.contextData).length > 0
? "\n\nSupporting data:\n" + JSON.stringify(context.contextData, null, 2)
: "";
return [
"Decision: " + context.title,
"Description: " + context.description,
"",
"Options to evaluate:",
optionsList,
"",
"Evaluation criteria (weights sum to 1.0):",
criteriaList,
dataSection,
"",
"Score each option against each criterion on a scale of 1-10.",
"Provide a weighted total score for each option.",
"Return JSON with this schema:",
"{",
' "analysis": {',
' "scores": [{ "option": "string", "criteriaScores": [{ "criterion": "string", "score": number, "reasoning": "string" }], "weightedTotal": number }],',
' "dominantOption": "string",',
' "scoreSensitivity": "string",',
' "dataGaps": ["string"]',
" }",
"}"
].join("\n");
}
function analyzeDecision(context) {
var prompt = buildAnalysisPrompt(context);
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
var text = response.content[0].text;
var parsed = JSON.parse(text);
parsed.analysis.decisionId = context.id;
parsed.analysis.analyzedAt = new Date().toISOString();
return parsed.analysis;
});
}
module.exports = { analyzeDecision: analyzeDecision };
The key detail here is the scoreSensitivity field. This tells the decision-maker whether the top option wins by a wide margin or whether a small change in weights could flip the recommendation. That kind of meta-analysis is something LLMs handle well because they can reason about the robustness of their own scoring.
Structured Decision Frameworks
Beyond raw scoring, you often need structured frameworks — SWOT analysis, pros/cons lists, risk matrices. Rather than building separate modules for each framework, you can have the LLM generate them on demand based on the decision context.
function generateFramework(context, frameworkType) {
var frameworkPrompts = {
"swot": "Perform a SWOT analysis (Strengths, Weaknesses, Opportunities, Threats) for each option. Return JSON: { \"swot\": [{ \"option\": \"string\", \"strengths\": [\"string\"], \"weaknesses\": [\"string\"], \"opportunities\": [\"string\"], \"threats\": [\"string\"] }] }",
"pros-cons": "Generate a detailed pros and cons list for each option. Return JSON: { \"proscons\": [{ \"option\": \"string\", \"pros\": [{ \"point\": \"string\", \"impact\": \"high|medium|low\" }], \"cons\": [{ \"point\": \"string\", \"impact\": \"high|medium|low\" }] }] }",
"risk-matrix": "Build a risk assessment matrix for each option. Evaluate likelihood (1-5) and impact (1-5) for each identified risk. Return JSON: { \"risks\": [{ \"option\": \"string\", \"risks\": [{ \"description\": \"string\", \"likelihood\": number, \"impact\": number, \"riskScore\": number, \"mitigation\": \"string\" }] }] }",
"cost-benefit": "Perform a cost-benefit analysis for each option. Quantify where possible, estimate where not. Return JSON: { \"costBenefit\": [{ \"option\": \"string\", \"costs\": [{ \"item\": \"string\", \"estimate\": \"string\", \"confidence\": \"high|medium|low\" }], \"benefits\": [{ \"item\": \"string\", \"estimate\": \"string\", \"confidence\": \"high|medium|low\" }], \"netAssessment\": \"string\" }] }"
};
var frameworkPrompt = frameworkPrompts[frameworkType];
if (!frameworkPrompt) {
return Promise.reject(new Error("Unknown framework: " + frameworkType));
}
var userMessage = [
"Decision: " + context.title,
"Description: " + context.description,
"Options: " + context.options.map(function(o) { return o.name; }).join(", "),
"",
frameworkPrompt
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: userMessage }]
}).then(function(response) {
return JSON.parse(response.content[0].text);
});
}
I have shipped systems where the risk matrix alone justified the entire project. A product manager once told me that seeing AI-generated risk mitigations for each option saved the team two days of meetings. That is the real value — not replacing the discussion, but front-loading the analysis so the discussion is productive.
Integrating Real-Time Data Feeds
A DSS is only as good as its data. Static analysis is useful, but live data makes recommendations dramatically more relevant. Here is a pattern for gathering data from multiple sources before running analysis:
var http = require("http");
var https = require("https");
function DataGatherer() {
this.sources = [];
}
DataGatherer.prototype.addSource = function(name, fetchFn) {
this.sources.push({ name: name, fetch: fetchFn });
return this;
};
DataGatherer.prototype.gather = function() {
var fetches = this.sources.map(function(source) {
return source.fetch()
.then(function(data) {
return { source: source.name, data: data, status: "success" };
})
.catch(function(err) {
console.error("Data source " + source.name + " failed:", err.message);
return { source: source.name, data: null, status: "error", error: err.message };
});
});
return Promise.all(fetches).then(function(results) {
var gathered = {};
var failures = [];
results.forEach(function(result) {
if (result.status === "success") {
gathered[result.source] = result.data;
} else {
failures.push(result.source + ": " + result.error);
}
});
return {
data: gathered,
failures: failures,
completeness: ((results.length - failures.length) / results.length * 100).toFixed(1) + "%"
};
});
};
// Example: gather market data for a vendor selection decision
function createVendorDataGatherer(vendorNames) {
var gatherer = new DataGatherer();
gatherer.addSource("pricing", function() {
// In production, this calls your pricing API or scrapes vendor pages
return Promise.resolve(
vendorNames.map(function(v) { return { vendor: v, monthlyRate: Math.floor(Math.random() * 5000) + 1000 }; })
);
});
gatherer.addSource("reviews", function() {
// Pull from G2, Capterra, etc.
return Promise.resolve(
vendorNames.map(function(v) { return { vendor: v, avgRating: (Math.random() * 2 + 3).toFixed(1), reviewCount: Math.floor(Math.random() * 500) + 50 }; })
);
});
gatherer.addSource("uptime", function() {
// Pull from status page APIs
return Promise.resolve(
vendorNames.map(function(v) { return { vendor: v, uptimePercent: (99 + Math.random()).toFixed(3) }; })
);
});
return gatherer;
}
module.exports = { DataGatherer: DataGatherer, createVendorDataGatherer: createVendorDataGatherer };
The key pattern here is graceful degradation. If one data source fails, the analysis continues with whatever data is available. The completeness percentage is passed to the LLM so it can factor data quality into its confidence scoring.
Recommendation Engine with Explanations
Recommendations without explanations are useless in a business context. Nobody trusts a black box. The recommendation engine needs to produce not just "choose Option A" but a structured explanation of why, what assumptions were made, and what could change the recommendation.
function generateRecommendation(context, analysis) {
var prompt = [
"Based on the following multi-criteria analysis, generate a structured recommendation.",
"",
"Decision: " + context.title,
"Analysis results: " + JSON.stringify(analysis, null, 2),
"",
"Return JSON with this schema:",
"{",
' "recommendation": {',
' "primaryChoice": "string",',
' "confidence": number,',
' "confidenceFactors": [{ "factor": "string", "effect": "increases|decreases", "magnitude": "strong|moderate|weak" }],',
' "explanation": {',
' "summary": "string (2-3 sentences)",',
' "keyReasons": ["string"],',
' "assumptions": ["string"],',
' "caveats": ["string"]',
" },",
' "alternatives": [{ "option": "string", "scenario": "string (when this option would be better)" }],',
' "reversibilityScore": number,',
' "timeToDecide": "string (how urgent is this decision)"',
" }",
"}"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
var result = JSON.parse(response.content[0].text);
result.recommendation.decisionId = context.id;
result.recommendation.generatedAt = new Date().toISOString();
return result.recommendation;
});
}
The reversibilityScore is something I started adding after a few deployments. It answers the question: "If this turns out to be wrong, how hard is it to undo?" A score of 0.9 means the decision is easily reversible (try a SaaS tool for a month), while 0.1 means you are locked in (sign a 3-year enterprise contract). This single number changes how people interact with recommendations — they are far more willing to act quickly on reversible decisions.
Confidence Scoring
Confidence scoring is where most DSS implementations fall flat. They either do not include confidence at all, or they give a meaningless number. Here is how to make confidence scores meaningful:
function calibrateConfidence(rawConfidence, dataCompleteness, criteriaCount, optionSpread) {
var dataFactor = parseFloat(dataCompleteness) / 100;
var criteriaFactor = Math.min(criteriaCount / 5, 1.0); // more criteria = better calibrated
var spreadFactor = optionSpread > 0.15 ? 1.0 : optionSpread / 0.15; // clear winner vs. close call
var calibrated = rawConfidence * 0.4 + dataFactor * 0.25 + criteriaFactor * 0.15 + spreadFactor * 0.2;
return {
overall: Math.round(calibrated * 100) / 100,
breakdown: {
modelConfidence: rawConfidence,
dataQuality: dataFactor,
criteriaAdequacy: criteriaFactor,
optionDifferentiation: spreadFactor
},
interpretation: calibrated > 0.8 ? "High confidence — act on this recommendation"
: calibrated > 0.6 ? "Moderate confidence — consider gathering more data"
: calibrated > 0.4 ? "Low confidence — significant uncertainty, investigate further"
: "Very low confidence — insufficient data for a reliable recommendation"
};
}
The optionSpread measures how far apart the top two options are in weighted score. If Option A scores 8.2 and Option B scores 8.1, the spread is tiny and the confidence should drop. The model might be confident in its analysis, but the decision itself is a coin flip.
Presenting Trade-offs and Alternatives
Every recommendation should come with a "what if" section. Decision-makers need to see not just the best option, but what they are giving up by not choosing alternatives.
function generateTradeoffMatrix(context, analysis) {
var prompt = [
"Given this decision analysis, create a trade-off matrix showing what the decision-maker",
"gains and loses with each option compared to every other option.",
"",
"Decision: " + context.title,
"Analysis: " + JSON.stringify(analysis, null, 2),
"",
"Return JSON:",
"{",
' "tradeoffs": [',
' { "choosingOption": "string", "overOption": "string", "gains": ["string"], "losses": ["string"], "netAssessment": "string" }',
" ],",
' "dealbreakers": [{ "option": "string", "dealbreaker": "string", "severity": "critical|major|minor" }],',
' "synergies": [{ "options": ["string"], "synergy": "string" }]',
"}"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
return JSON.parse(response.content[0].text);
});
}
Decision Audit Trail
Every decision your system helps with should be logged. This is not optional. In regulated industries it is a compliance requirement, and in every industry it is how you improve the system over time.
var { Pool } = require("pg");
var pool = new Pool({ connectionString: process.env.POSTGRES_CONNECTION_STRING });
function DecisionAuditLog() {}
DecisionAuditLog.prototype.logDecision = function(decisionId, context, analysis, recommendation) {
var query = [
"INSERT INTO decision_audit_log",
"(decision_id, title, context, analysis, recommendation, created_at)",
"VALUES ($1, $2, $3, $4, $5, NOW())"
].join(" ");
return pool.query(query, [
decisionId,
context.title,
JSON.stringify(context),
JSON.stringify(analysis),
JSON.stringify(recommendation)
]);
};
DecisionAuditLog.prototype.logOutcome = function(decisionId, chosenOption, outcome, notes) {
var query = [
"UPDATE decision_audit_log",
"SET chosen_option = $2, outcome = $3, outcome_notes = $4,",
"outcome_recorded_at = NOW()",
"WHERE decision_id = $1"
].join(" ");
return pool.query(query, [decisionId, chosenOption, outcome, notes]);
};
DecisionAuditLog.prototype.getDecisionHistory = function(limit) {
var query = [
"SELECT decision_id, title, recommendation->>'primaryChoice' as recommended,",
"chosen_option, outcome, created_at, outcome_recorded_at",
"FROM decision_audit_log",
"ORDER BY created_at DESC LIMIT $1"
].join(" ");
return pool.query(query, [limit || 50]).then(function(result) { return result.rows; });
};
DecisionAuditLog.prototype.getAccuracyMetrics = function() {
var query = [
"SELECT",
"COUNT(*) as total_decisions,",
"COUNT(outcome) as outcomes_recorded,",
"COUNT(CASE WHEN recommendation->>'primaryChoice' = chosen_option THEN 1 END) as recommendation_followed,",
"COUNT(CASE WHEN outcome = 'positive' THEN 1 END) as positive_outcomes,",
"COUNT(CASE WHEN outcome = 'positive' AND recommendation->>'primaryChoice' = chosen_option THEN 1 END) as positive_when_followed",
"FROM decision_audit_log",
"WHERE outcome IS NOT NULL"
].join(" ");
return pool.query(query).then(function(result) {
var row = result.rows[0];
return {
totalDecisions: parseInt(row.total_decisions),
outcomesRecorded: parseInt(row.outcomes_recorded),
recommendationFollowRate: row.outcomes_recorded > 0
? (parseInt(row.recommendation_followed) / parseInt(row.outcomes_recorded) * 100).toFixed(1) + "%"
: "N/A",
positiveOutcomeRate: row.outcomes_recorded > 0
? (parseInt(row.positive_outcomes) / parseInt(row.outcomes_recorded) * 100).toFixed(1) + "%"
: "N/A",
positiveWhenFollowed: parseInt(row.recommendation_followed) > 0
? (parseInt(row.positive_when_followed) / parseInt(row.recommendation_followed) * 100).toFixed(1) + "%"
: "N/A"
};
});
};
module.exports = { DecisionAuditLog: DecisionAuditLog };
The accuracy metrics are gold. After a few months of operation, you can show stakeholders: "When people followed the system's recommendation, 78% of outcomes were positive. When they did not, 52% were positive." That builds trust faster than any demo.
Handling Uncertainty and Ambiguity
LLMs can hallucinate confidence. You need guardrails that force the system to be honest about what it does not know.
function uncertaintyAnalysis(context, analysis) {
var prompt = [
"Analyze the uncertainty in this decision analysis. Be ruthlessly honest about what we do NOT know.",
"",
"Decision: " + context.title,
"Analysis: " + JSON.stringify(analysis, null, 2),
"",
"Return JSON:",
"{",
' "uncertainties": [',
' { "area": "string", "description": "string", "impact": "could_flip_decision|significant|minor",',
' "reducible": boolean, "howToReduce": "string" }',
" ],",
' "unknownUnknowns": "string (what categories of risk might we not be seeing at all)",',
' "worstCase": { "scenario": "string", "likelihood": "string", "mitigation": "string" },',
' "informationValue": [{ "dataPoint": "string", "expectedImpact": "string", "costToObtain": "string" }]',
"}"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
return JSON.parse(response.content[0].text);
});
}
The informationValue section is particularly powerful. It tells the decision-maker: "If you could get this one piece of data, it would significantly change the analysis, and here is how much it would cost to get it." That is decision science at its best — knowing what you need to know before you decide.
What-If Scenarios
Interactive exploration lets decision-makers test how changes to assumptions affect the recommendation. This is where an LLM-powered DSS truly outshines traditional systems.
function whatIfAnalysis(context, analysis, scenario) {
var prompt = [
"The user wants to explore a what-if scenario for this decision.",
"",
"Original decision: " + context.title,
"Original analysis: " + JSON.stringify(analysis, null, 2),
"",
"What-if scenario: " + scenario,
"",
"Re-evaluate the decision under this scenario. Show how scores change and whether",
"the recommendation would flip. Return JSON:",
"{",
' "scenarioImpact": {',
' "description": "string",',
' "originalRecommendation": "string",',
' "newRecommendation": "string",',
' "recommendationChanged": boolean,',
' "scoreChanges": [{ "option": "string", "originalScore": number, "newScore": number, "delta": number }],',
' "keyInsight": "string"',
" }",
"}"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
return JSON.parse(response.content[0].text);
});
}
Group Decision Facilitation
When multiple stakeholders are involved, the DSS can moderate by synthesizing different viewpoints and surfacing areas of agreement and disagreement.
function facilitateGroupDecision(context, stakeholderInputs) {
var inputSummary = stakeholderInputs.map(function(s) {
return "- " + s.name + " (" + s.role + "): Priorities: " + s.priorities.join(", ") + ". Preferred option: " + s.preferredOption + ". Concerns: " + s.concerns;
}).join("\n");
var prompt = [
"Multiple stakeholders have weighed in on this decision. Facilitate alignment.",
"",
"Decision: " + context.title,
"Options: " + context.options.map(function(o) { return o.name; }).join(", "),
"",
"Stakeholder inputs:",
inputSummary,
"",
"Return JSON:",
"{",
' "consensus": {',
' "areasOfAgreement": ["string"],',
' "areasOfDisagreement": [{ "topic": "string", "positions": [{ "stakeholder": "string", "position": "string" }], "suggestedResolution": "string" }],',
' "compositeRecommendation": "string",',
' "compromiseOptions": [{ "description": "string", "satisfies": ["string"], "sacrifices": ["string"] }],',
' "nextSteps": ["string"]',
" }",
"}"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
return JSON.parse(response.content[0].text);
});
}
Complete Working Example
Here is a full Express.js decision support service that ties everything together. This is production-ready scaffolding — you would add authentication, rate limiting, and input validation for a real deployment.
Database Schema
CREATE TABLE decision_audit_log (
id SERIAL PRIMARY KEY,
decision_id UUID UNIQUE NOT NULL,
title VARCHAR(500) NOT NULL,
context JSONB NOT NULL,
analysis JSONB,
recommendation JSONB,
chosen_option VARCHAR(255),
outcome VARCHAR(50),
outcome_notes TEXT,
created_at TIMESTAMP DEFAULT NOW(),
outcome_recorded_at TIMESTAMP
);
CREATE INDEX idx_decision_audit_created ON decision_audit_log(created_at DESC);
CREATE INDEX idx_decision_audit_outcome ON decision_audit_log(outcome) WHERE outcome IS NOT NULL;
Express Application
// decision-support-server.js
var express = require("express");
var bodyParser = require("body-parser");
var { v4: uuidv4 } = require("uuid");
var Anthropic = require("@anthropic-ai/sdk");
var { Pool } = require("pg");
var app = express();
app.use(bodyParser.json());
var anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
var pool = new Pool({ connectionString: process.env.POSTGRES_CONNECTION_STRING });
var SYSTEM_PROMPT = [
"You are an expert decision analyst. Analyze data, evaluate trade-offs, provide structured recommendations.",
"Always quantify uncertainty. Always present alternatives. Respond ONLY with valid JSON matching the requested schema."
].join(" ");
// --- Core Analysis Function ---
function runAnalysis(context) {
var criteriaList = context.criteria.map(function(c) {
return "- " + c.name + " (weight: " + c.weight + "): " + c.description;
}).join("\n");
var optionsList = context.options.map(function(o) {
return "- " + o.name + ": " + o.description;
}).join("\n");
var prompt = [
"Decision: " + context.title,
"Description: " + context.description,
"",
"Options:", optionsList,
"",
"Criteria:", criteriaList,
"",
"Context data: " + JSON.stringify(context.contextData || {}),
"",
"Perform multi-criteria analysis. Score each option 1-10 on each criterion.",
"Then generate a recommendation with confidence score (0-1), explanation,",
"assumptions, caveats, alternatives, and trade-offs.",
"",
"Return JSON:",
"{",
' "analysis": {',
' "scores": [{ "option": "string", "criteriaScores": [{ "criterion": "string", "score": number, "reasoning": "string" }], "weightedTotal": number }],',
' "dataGaps": ["string"]',
" },",
' "recommendation": {',
' "primaryChoice": "string",',
' "confidence": number,',
' "summary": "string",',
' "keyReasons": ["string"],',
' "assumptions": ["string"],',
' "caveats": ["string"],',
' "alternatives": [{ "option": "string", "scenario": "string" }]',
" },",
' "tradeoffs": [{ "choosingOption": "string", "overOption": "string", "gains": ["string"], "losses": ["string"] }]',
"}"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 8192,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
return JSON.parse(response.content[0].text);
});
}
// --- Routes ---
// POST /decisions — Submit a new decision for analysis
app.post("/decisions", function(req, res) {
var body = req.body;
if (!body.title || !body.options || !body.criteria) {
return res.status(400).json({ error: "Missing required fields: title, options, criteria" });
}
if (body.options.length < 2) {
return res.status(400).json({ error: "At least 2 options are required" });
}
var weightSum = body.criteria.reduce(function(sum, c) { return sum + c.weight; }, 0);
if (Math.abs(weightSum - 1.0) > 0.01) {
return res.status(400).json({ error: "Criteria weights must sum to 1.0, got " + weightSum });
}
var decisionId = uuidv4();
var context = {
id: decisionId,
title: body.title,
description: body.description || "",
options: body.options,
criteria: body.criteria,
contextData: body.contextData || {}
};
runAnalysis(context)
.then(function(result) {
// Log to audit trail
var query = "INSERT INTO decision_audit_log (decision_id, title, context, analysis, recommendation) VALUES ($1, $2, $3, $4, $5)";
return pool.query(query, [
decisionId,
context.title,
JSON.stringify(context),
JSON.stringify(result.analysis),
JSON.stringify(result.recommendation)
]).then(function() {
return res.json({
decisionId: decisionId,
analysis: result.analysis,
recommendation: result.recommendation,
tradeoffs: result.tradeoffs
});
});
})
.catch(function(err) {
console.error("Analysis failed:", err);
res.status(500).json({ error: "Analysis failed", message: err.message });
});
});
// POST /decisions/:id/what-if — Explore a scenario
app.post("/decisions/:id/what-if", function(req, res) {
var decisionId = req.params.id;
var scenario = req.body.scenario;
if (!scenario) {
return res.status(400).json({ error: "Missing required field: scenario" });
}
pool.query("SELECT context, analysis FROM decision_audit_log WHERE decision_id = $1", [decisionId])
.then(function(result) {
if (result.rows.length === 0) {
return res.status(404).json({ error: "Decision not found" });
}
var row = result.rows[0];
var context = row.context;
var analysis = row.analysis;
var prompt = [
"Original decision: " + context.title,
"Original analysis: " + JSON.stringify(analysis),
"",
"What-if scenario: " + scenario,
"",
"Re-evaluate. Return JSON:",
"{ \"scenarioImpact\": { \"originalRecommendation\": \"string\", \"newRecommendation\": \"string\",",
"\"recommendationChanged\": boolean, \"scoreChanges\": [{ \"option\": \"string\", \"originalScore\": number, \"newScore\": number }],",
"\"keyInsight\": \"string\" } }"
].join("\n");
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
res.json(JSON.parse(response.content[0].text));
});
})
.catch(function(err) {
console.error("What-if analysis failed:", err);
res.status(500).json({ error: "What-if analysis failed", message: err.message });
});
});
// POST /decisions/:id/outcome — Record the outcome
app.post("/decisions/:id/outcome", function(req, res) {
var decisionId = req.params.id;
var body = req.body;
if (!body.chosenOption || !body.outcome) {
return res.status(400).json({ error: "Missing required fields: chosenOption, outcome" });
}
if (["positive", "negative", "neutral", "mixed"].indexOf(body.outcome) === -1) {
return res.status(400).json({ error: "outcome must be: positive, negative, neutral, or mixed" });
}
var query = "UPDATE decision_audit_log SET chosen_option = $2, outcome = $3, outcome_notes = $4, outcome_recorded_at = NOW() WHERE decision_id = $1";
pool.query(query, [decisionId, body.chosenOption, body.outcome, body.notes || null])
.then(function(result) {
if (result.rowCount === 0) {
return res.status(404).json({ error: "Decision not found" });
}
res.json({ message: "Outcome recorded", decisionId: decisionId });
})
.catch(function(err) {
res.status(500).json({ error: "Failed to record outcome", message: err.message });
});
});
// GET /decisions/metrics — System accuracy metrics
app.get("/decisions/metrics", function(req, res) {
var query = [
"SELECT",
"COUNT(*) as total,",
"COUNT(outcome) as with_outcomes,",
"COUNT(CASE WHEN recommendation->>'primaryChoice' = chosen_option THEN 1 END) as followed,",
"COUNT(CASE WHEN outcome = 'positive' THEN 1 END) as positive,",
"COUNT(CASE WHEN outcome = 'positive' AND recommendation->>'primaryChoice' = chosen_option THEN 1 END) as positive_followed",
"FROM decision_audit_log WHERE outcome IS NOT NULL"
].join(" ");
pool.query(query)
.then(function(result) {
var r = result.rows[0];
var total = parseInt(r.with_outcomes) || 1;
var followed = parseInt(r.followed) || 0;
res.json({
totalDecisions: parseInt(r.total),
outcomesRecorded: parseInt(r.with_outcomes),
recommendationFollowRate: (followed / total * 100).toFixed(1) + "%",
positiveOutcomeRate: (parseInt(r.positive) / total * 100).toFixed(1) + "%",
positiveWhenFollowed: followed > 0
? (parseInt(r.positive_followed) / followed * 100).toFixed(1) + "%"
: "N/A"
});
})
.catch(function(err) {
res.status(500).json({ error: "Failed to get metrics", message: err.message });
});
});
// GET /decisions/:id/framework/:type — Generate analysis framework
app.get("/decisions/:id/framework/:type", function(req, res) {
var validFrameworks = ["swot", "pros-cons", "risk-matrix", "cost-benefit"];
var frameworkType = req.params.type;
if (validFrameworks.indexOf(frameworkType) === -1) {
return res.status(400).json({ error: "Valid frameworks: " + validFrameworks.join(", ") });
}
pool.query("SELECT context FROM decision_audit_log WHERE decision_id = $1", [req.params.id])
.then(function(result) {
if (result.rows.length === 0) {
return res.status(404).json({ error: "Decision not found" });
}
var context = result.rows[0].context;
var frameworkPrompts = {
"swot": "Perform a SWOT analysis for each option. Return JSON: { \"swot\": [{ \"option\": \"string\", \"strengths\": [\"string\"], \"weaknesses\": [\"string\"], \"opportunities\": [\"string\"], \"threats\": [\"string\"] }] }",
"pros-cons": "Generate pros/cons for each option. Return JSON: { \"proscons\": [{ \"option\": \"string\", \"pros\": [{ \"point\": \"string\", \"impact\": \"high|medium|low\" }], \"cons\": [{ \"point\": \"string\", \"impact\": \"high|medium|low\" }] }] }",
"risk-matrix": "Risk assessment for each option. Return JSON: { \"risks\": [{ \"option\": \"string\", \"risks\": [{ \"description\": \"string\", \"likelihood\": number, \"impact\": number, \"riskScore\": number, \"mitigation\": \"string\" }] }] }",
"cost-benefit": "Cost-benefit analysis. Return JSON: { \"costBenefit\": [{ \"option\": \"string\", \"costs\": [{ \"item\": \"string\", \"estimate\": \"string\" }], \"benefits\": [{ \"item\": \"string\", \"estimate\": \"string\" }], \"netAssessment\": \"string\" }] }"
};
var prompt = "Decision: " + context.title + "\nOptions: " + context.options.map(function(o) { return o.name; }).join(", ") + "\n\n" + frameworkPrompts[frameworkType];
return anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: prompt }]
}).then(function(response) {
res.json(JSON.parse(response.content[0].text));
});
})
.catch(function(err) {
res.status(500).json({ error: "Framework generation failed", message: err.message });
});
});
var PORT = process.env.PORT || 3000;
app.listen(PORT, function() {
console.log("Decision support service running on port " + PORT);
});
Example API Call
curl -X POST http://localhost:3000/decisions \
-H "Content-Type: application/json" \
-d '{
"title": "Cloud Provider Selection for New Microservices Platform",
"description": "Selecting between AWS, GCP, and Azure for a new microservices platform serving 10M requests/day",
"options": [
{ "name": "AWS", "description": "Amazon Web Services - mature ecosystem, largest market share" },
{ "name": "GCP", "description": "Google Cloud - strong Kubernetes, competitive pricing" },
{ "name": "Azure", "description": "Microsoft Azure - enterprise integration, hybrid cloud strengths" }
],
"criteria": [
{ "name": "Cost", "weight": 0.25, "description": "Total cost of ownership over 3 years" },
{ "name": "Kubernetes Support", "weight": 0.20, "description": "Native K8s tooling and managed services" },
{ "name": "Team Expertise", "weight": 0.20, "description": "Current team familiarity and hiring pool" },
{ "name": "Service Breadth", "weight": 0.15, "description": "Range of managed services available" },
{ "name": "Vendor Lock-in Risk", "weight": 0.10, "description": "Ease of migration away from provider" },
{ "name": "Compliance", "weight": 0.10, "description": "HIPAA/SOC2/FedRAMP certifications" }
],
"contextData": {
"teamSize": 12,
"currentStack": "Docker + ECS on AWS",
"budget": "$50K/month",
"timeline": "6 months to production"
}
}'
Example Response
{
"decisionId": "a3f8c1e2-7b4d-4e9a-b5c6-d8e9f0a1b2c3",
"analysis": {
"scores": [
{
"option": "AWS",
"criteriaScores": [
{ "criterion": "Cost", "score": 6, "reasoning": "Premium pricing but team already has reserved instances and cost management in place" },
{ "criterion": "Kubernetes Support", "score": 7, "reasoning": "EKS is solid but GKE leads in managed K8s" },
{ "criterion": "Team Expertise", "score": 9, "reasoning": "Team already runs ECS on AWS — lowest switching cost" },
{ "criterion": "Service Breadth", "score": 9, "reasoning": "Widest range of managed services" },
{ "criterion": "Vendor Lock-in Risk", "score": 5, "reasoning": "Deep AWS-specific services increase lock-in" },
{ "criterion": "Compliance", "score": 9, "reasoning": "Most comprehensive compliance certifications" }
],
"weightedTotal": 7.45
}
],
"dataGaps": ["No performance benchmarks for specific workload", "Missing SLA comparison data"]
},
"recommendation": {
"primaryChoice": "AWS",
"confidence": 0.72,
"summary": "AWS is recommended primarily due to team expertise and migration cost. The team already operates on AWS, and switching providers introduces 3-4 months of migration overhead that outweighs GCP's Kubernetes advantages.",
"keyReasons": [
"Team has 2+ years of AWS operational experience",
"Migration risk is non-trivial given the 6-month timeline",
"AWS service breadth supports future requirements"
],
"assumptions": [
"Team size remains stable at 12",
"Budget of $50K/month is firm",
"No regulatory requirement forcing specific provider"
],
"caveats": [
"If the team doubles in size, GCP's Kubernetes tooling may become more valuable",
"AWS costs tend to grow faster than GCP at scale"
],
"alternatives": [
{ "option": "GCP", "scenario": "If timeline extends to 12 months and team has capacity for migration" },
{ "option": "Azure", "scenario": "If enterprise SSO/Active Directory integration becomes a hard requirement" }
]
}
}
Common Issues and Troubleshooting
1. JSON Parse Errors from LLM Response
SyntaxError: Unexpected token 'T' at position 0
at JSON.parse (<anonymous>)
This happens when the model returns a text explanation instead of JSON. The fix is to validate and retry:
function safeParseResponse(response) {
var text = response.content[0].text;
// Strip markdown code fences if present
text = text.replace(/^```json\s*\n?/, "").replace(/\n?```\s*$/, "");
try {
return JSON.parse(text);
} catch (e) {
// Try to extract JSON from within the text
var match = text.match(/\{[\s\S]*\}/);
if (match) {
return JSON.parse(match[0]);
}
throw new Error("Failed to parse LLM response as JSON: " + text.substring(0, 200));
}
}
2. Criteria Weights Do Not Sum to 1.0
Error: Criteria weights must sum to 1.0, got 0.9500000000000001
Floating point arithmetic causes rounding issues. Always use a tolerance threshold:
var weightSum = criteria.reduce(function(sum, c) { return sum + c.weight; }, 0);
if (Math.abs(weightSum - 1.0) > 0.01) {
// Normalize weights instead of rejecting
criteria = criteria.map(function(c) {
return Object.assign({}, c, { weight: c.weight / weightSum });
});
}
3. Token Limit Exceeded for Large Decision Contexts
Error: max_tokens must be less than or equal to 8192
anthropic.BadRequestError: prompt is too long: 210432 tokens > 200000 maximum
When context data is large, you need to summarize before sending to the LLM:
function truncateContextData(data, maxChars) {
var serialized = JSON.stringify(data);
if (serialized.length <= maxChars) {
return data;
}
// Keep structure but truncate arrays
var truncated = {};
Object.keys(data).forEach(function(key) {
if (Array.isArray(data[key]) && data[key].length > 10) {
truncated[key] = data[key].slice(0, 10);
truncated[key + "_note"] = "Truncated from " + data[key].length + " items";
} else {
truncated[key] = data[key];
}
});
return truncated;
}
4. PostgreSQL Connection Pool Exhaustion
Error: sorry, too many clients already
at Connection.parseE (node_modules/pg/lib/connection.js:614:13)
This happens under heavy concurrent analysis load. Configure the pool correctly and add connection timeout handling:
var pool = new Pool({
connectionString: process.env.POSTGRES_CONNECTION_STRING,
max: 20,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000
});
pool.on("error", function(err) {
console.error("Unexpected pool error:", err);
});
5. Inconsistent Scoring Across Repeated Analyses
The same decision context can produce different scores on different runs due to LLM non-determinism. Mitigate this by setting temperature to 0 and averaging multiple runs for high-stakes decisions:
function stableAnalysis(context, runs) {
runs = runs || 3;
var promises = [];
for (var i = 0; i < runs; i++) {
promises.push(
anthropic.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
temperature: 0,
system: SYSTEM_PROMPT,
messages: [{ role: "user", content: buildPrompt(context) }]
}).then(function(r) { return JSON.parse(r.content[0].text); })
);
}
return Promise.all(promises).then(function(results) {
// Average the weighted totals across runs
var averaged = results[0];
averaged.analysis.scores.forEach(function(score, idx) {
var totalSum = results.reduce(function(sum, r) {
return sum + r.analysis.scores[idx].weightedTotal;
}, 0);
score.weightedTotal = Math.round(totalSum / runs * 100) / 100;
});
averaged.analysis.stabilityNote = "Averaged across " + runs + " runs";
return averaged;
});
}
Best Practices
Always validate criteria weights on the server. Do not trust the client to send weights that sum to 1.0. Normalize them silently or reject with a clear error. Floating point arithmetic will bite you otherwise.
Set temperature to 0 for scoring, higher for brainstorming. Multi-criteria scoring should be deterministic. Use
temperature: 0for analysis and scoring prompts, buttemperature: 0.7for what-if exploration and creative scenario generation where diversity of thought is valuable.Never present AI confidence as certainty. Always show the confidence score prominently, and always accompany it with the breakdown (data quality, criteria adequacy, option differentiation). A 0.72 confidence score means "we are 72% sure," not "this is the right answer."
Log every decision, every recommendation, every outcome. The audit trail is not just for compliance. It is the training data for improving your system. After 100 decisions with recorded outcomes, you can start showing empirical accuracy rates that build real stakeholder trust.
Implement circuit breakers for the LLM. If the Anthropic API is down or slow, your decision support service should not hang. Set aggressive timeouts (30 seconds for analysis, 15 seconds for frameworks) and return partial results with a degraded-service indicator rather than failing entirely.
Separate data gathering from analysis. The data gathering phase should complete entirely before analysis begins. This lets you measure data completeness, handle source failures gracefully, and cache data for what-if scenarios that re-analyze the same context under different assumptions.
Version your prompts. Store the prompt template version alongside each decision in the audit log. When you update prompts (and you will, frequently), you need to know which version produced which recommendations. This is essential for understanding changes in system accuracy over time.
Present alternatives as first-class outputs, not afterthoughts. Decision-makers often choose an option that was not the primary recommendation, especially when the confidence score is moderate. Make the "when would this alternative be better" section prominent and actionable.
Rate-limit analysis endpoints aggressively. Each decision analysis costs real money in API tokens. Implement per-user and per-organization rate limits. A complex decision with multiple frameworks can cost $0.50-$2.00 in API calls — that adds up fast if someone automates requests against your endpoint.
