DynamoDB Design Patterns for Application Developers
Master DynamoDB single-table design patterns with Node.js including GSI strategies, transactions, and event-driven architectures
DynamoDB Design Patterns for Application Developers
DynamoDB is a fully managed NoSQL database from AWS that delivers single-digit millisecond performance at any scale, but only if you design your data model correctly. Unlike relational databases where you normalize first and optimize later, DynamoDB requires you to understand your access patterns upfront and model your data around them. This article covers the design patterns that matter most for application developers building real systems with Node.js.
Prerequisites
- An AWS account with DynamoDB access
- Node.js v14 or later installed
- Basic understanding of NoSQL concepts
- AWS SDK for JavaScript v2 or v3 installed
- Familiarity with Express.js (for the complete example)
Install the AWS SDK:
npm install aws-sdk
Single-Table Design Philosophy
The most counterintuitive aspect of DynamoDB for developers coming from relational databases is single-table design. Instead of creating a table per entity (users, orders, products), you store everything in one table and use composite keys to differentiate entity types.
Why would you do this? Because DynamoDB charges per request and per table, and joins do not exist. Every query hits a single table. If your application needs data from "users" and "orders" in a single page load, that means two separate queries against two tables. With single-table design, you can retrieve related entities in a single query using the sort key.
Here is a concrete example. Consider an e-commerce application with users and their orders:
PK | SK | Data
----------------|---------------------|------------------
USER#u001 | PROFILE | {name, email, ...}
USER#u001 | ORDER#2024-001 | {total, status, ...}
USER#u001 | ORDER#2024-002 | {total, status, ...}
USER#u002 | PROFILE | {name, email, ...}
USER#u002 | ORDER#2024-003 | {total, status, ...}
With this layout, a single query on PK = USER#u001 returns the user profile and all their orders. You filter by SK = PROFILE to get just the user, or SK begins_with ORDER# to get just their orders.
Single-table design is not always the right call. If you have completely unrelated entities with no shared access patterns, separate tables are fine. The pattern shines when entities are accessed together frequently.
Partition Key and Sort Key Strategies
Your partition key (PK) determines how DynamoDB distributes data across storage partitions. A bad partition key leads to hot partitions, throttling, and miserable performance. The sort key (SK) determines ordering within a partition and enables range queries.
Partition Key Rules:
- High cardinality is essential. A boolean field is a terrible partition key. A user ID is a good one.
- Even distribution matters. If 80% of requests go to the same partition key value, you have a hot partition.
- Composite keys solve most problems. Prefix your keys with entity types:
USER#123,ORG#456.
Sort Key Patterns:
// Hierarchical sort keys enable flexible queries
var items = [
{ PK: "ORG#acme", SK: "METADATA" },
{ PK: "ORG#acme", SK: "USER#[email protected]" },
{ PK: "ORG#acme", SK: "USER#[email protected]" },
{ PK: "ORG#acme", SK: "ROLE#admin" },
{ PK: "ORG#acme", SK: "ROLE#viewer" },
{ PK: "ORG#acme", SK: "INVITE#2024-01-15#inv001" }
];
// Query all users in org: SK begins_with "USER#"
// Query all roles: SK begins_with "ROLE#"
// Query invites after a date: SK begins_with "INVITE#2024-01"
// Get org metadata: SK = "METADATA"
Timestamp-based sort keys are powerful for time-series data. Use ISO 8601 format so lexicographic sorting matches chronological sorting:
var AWS = require("aws-sdk");
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "AppTable",
KeyConditionExpression: "PK = :pk AND SK BETWEEN :start AND :end",
ExpressionAttributeValues: {
":pk": "LOGS#service-api",
":start": "2024-01-01T00:00:00Z",
":end": "2024-01-31T23:59:59Z"
}
};
docClient.query(params, function(err, data) {
if (err) {
console.error("Query failed:", err);
} else {
console.log("Found", data.Count, "log entries");
}
});
GSI and LSI Design
Global Secondary Indexes (GSIs) are the workhorse of DynamoDB design. They let you query your data using different key combinations without duplicating anything manually. A GSI is essentially a full copy of your table with a different partition key and sort key.
Local Secondary Indexes (LSIs) share the same partition key as the base table but provide an alternate sort key. They must be created at table creation time and share the partition's 10 GB limit.
The practical rule: use GSIs almost exclusively. LSIs have too many constraints for most use cases.
The Inverted Index Pattern:
One of the most useful GSI patterns is the inverted index, where your GSI swaps the base table's PK and SK:
Base Table: PK = USER#u001, SK = ORDER#2024-001
GSI1: PK = ORDER#2024-001, SK = USER#u001
This lets you query "all orders for a user" on the base table and "which user placed this order" on the GSI.
The Overloaded GSI Pattern:
Instead of creating a GSI per access pattern, you add generic GSI1PK and GSI1SK attributes to your items and point a single GSI at them:
// User item
var userItem = {
PK: "USER#u001",
SK: "PROFILE",
GSI1PK: "EMAIL#[email protected]",
GSI1SK: "USER#u001",
name: "Shane Larson",
email: "[email protected]"
};
// Order item
var orderItem = {
PK: "USER#u001",
SK: "ORDER#2024-001",
GSI1PK: "STATUS#pending",
GSI1SK: "2024-01-15T10:30:00Z",
total: 59.99,
status: "pending"
};
Now your GSI1 supports two completely different queries: "find user by email" and "find all pending orders sorted by date." This is the real power of single-table design with overloaded GSIs.
GSI Cost Considerations:
Every GSI doubles your write costs for the attributes it projects. If you project all attributes into three GSIs, you pay four times for every write (base table plus three GSIs). Be deliberate about which attributes you project. Use KEYS_ONLY projection when you only need the keys, and INCLUDE to project specific attributes.
Query vs Scan Operations
This distinction is critical to DynamoDB performance and cost.
Query uses the partition key (and optionally the sort key) to find items. It reads only the items that match. This is what you should use 99% of the time.
Scan reads every item in the entire table and then filters. It consumes read capacity proportional to the full table size, regardless of how many items match your filter.
var AWS = require("aws-sdk");
var docClient = new AWS.DynamoDB.DocumentClient();
// GOOD: Query - reads only matching items
function getUserOrders(userId, callback) {
var params = {
TableName: "AppTable",
KeyConditionExpression: "PK = :pk AND begins_with(SK, :sk)",
ExpressionAttributeValues: {
":pk": "USER#" + userId,
":sk": "ORDER#"
}
};
docClient.query(params, callback);
}
// BAD: Scan with filter - reads entire table
function getUserOrdersScan(userId, callback) {
var params = {
TableName: "AppTable",
FilterExpression: "PK = :pk AND begins_with(SK, :sk)",
ExpressionAttributeValues: {
":pk": "USER#" + userId,
":sk": "ORDER#"
}
};
docClient.scan(params, callback);
}
The query reads a few KB. The scan reads the entire table, which could be terabytes. Your AWS bill will reflect the difference.
When scans are acceptable: Data exports, one-time migrations, analytics jobs on small tables, or parallel scans with worker processes during off-peak hours.
Filter expressions on queries are still useful. They do not reduce the amount of data read from the partition, but they reduce the amount of data returned to your application, saving network bandwidth:
var params = {
TableName: "AppTable",
KeyConditionExpression: "PK = :pk AND begins_with(SK, :sk)",
FilterExpression: "#s = :status",
ExpressionAttributeNames: {
"#s": "status"
},
ExpressionAttributeValues: {
":pk": "USER#u001",
":sk": "ORDER#",
":status": "shipped"
}
};
Batch Operations and Transactions
BatchGetItem retrieves up to 100 items across multiple tables in a single call. Items are retrieved in parallel, making this significantly faster than individual GetItem calls:
var params = {
RequestItems: {
"AppTable": {
Keys: [
{ PK: "USER#u001", SK: "PROFILE" },
{ PK: "USER#u002", SK: "PROFILE" },
{ PK: "USER#u003", SK: "PROFILE" }
]
}
}
};
docClient.batchGet(params, function(err, data) {
if (err) {
console.error("BatchGet failed:", err);
return;
}
// Always check for unprocessed keys
var unprocessed = data.UnprocessedKeys;
if (unprocessed && Object.keys(unprocessed).length > 0) {
console.log("Retrying unprocessed keys...");
docClient.batchGet({ RequestItems: unprocessed }, function(retryErr, retryData) {
// Handle retry
});
}
console.log("Users:", data.Responses.AppTable);
});
BatchWriteItem handles up to 25 put or delete operations per call. It does not support updates:
var params = {
RequestItems: {
"AppTable": [
{
PutRequest: {
Item: { PK: "USER#u004", SK: "PROFILE", name: "New User" }
}
},
{
DeleteRequest: {
Key: { PK: "USER#u003", SK: "PROFILE" }
}
}
]
}
};
docClient.batchWrite(params, function(err, data) {
if (err) {
console.error("BatchWrite failed:", err);
return;
}
if (data.UnprocessedItems && Object.keys(data.UnprocessedItems).length > 0) {
console.log("Retrying unprocessed items...");
// Implement exponential backoff retry
}
});
TransactWriteItems provides ACID transactions across up to 100 items. This is essential for operations that must be atomic:
var params = {
TransactItems: [
{
Put: {
TableName: "AppTable",
Item: {
PK: "USER#u001",
SK: "ORDER#2024-005",
total: 149.99,
status: "confirmed"
}
}
},
{
Update: {
TableName: "AppTable",
Key: { PK: "USER#u001", SK: "PROFILE" },
UpdateExpression: "SET orderCount = orderCount + :inc",
ExpressionAttributeValues: { ":inc": 1 }
}
},
{
Update: {
TableName: "AppTable",
Key: { PK: "PRODUCT#p100", SK: "METADATA" },
UpdateExpression: "SET stock = stock - :dec",
ConditionExpression: "stock >= :dec",
ExpressionAttributeValues: { ":dec": 1 }
}
}
]
};
docClient.transactWrite(params, function(err, data) {
if (err) {
if (err.code === "TransactionCanceledException") {
console.error("Transaction cancelled. Reasons:", err.message);
// Check which condition failed
} else {
console.error("Transaction error:", err);
}
} else {
console.log("Transaction committed successfully");
}
});
Transactions cost twice the WCU/RCU of non-transactional operations. Use them when you need atomicity, not as a default.
DynamoDB Streams for Event-Driven Patterns
DynamoDB Streams capture a time-ordered sequence of item-level changes. Every insert, update, and delete produces a stream record. This enables event-driven architectures where downstream services react to data changes.
Common use cases:
- Replicating data to Elasticsearch for full-text search
- Sending notifications when order status changes
- Maintaining aggregate counters in a separate item
- Cross-region replication
- Audit logging
// Lambda function processing DynamoDB Stream events
var AWS = require("aws-sdk");
var ses = new AWS.SES();
exports.handler = function(event, context, callback) {
var promises = event.Records.map(function(record) {
if (record.eventName === "MODIFY") {
var newImage = AWS.DynamoDB.Converter.unmarshall(record.dynamodb.NewImage);
var oldImage = AWS.DynamoDB.Converter.unmarshall(record.dynamodb.OldImage);
// Detect order status change
if (newImage.SK && newImage.SK.startsWith("ORDER#")) {
if (oldImage.status !== newImage.status && newImage.status === "shipped") {
return sendShippingNotification(newImage);
}
}
}
return Promise.resolve();
});
Promise.all(promises)
.then(function() { callback(null, "Success"); })
.catch(function(err) { callback(err); });
};
function sendShippingNotification(order) {
var params = {
Destination: { ToAddresses: [order.customerEmail] },
Message: {
Subject: { Data: "Your order has shipped!" },
Body: {
Text: { Data: "Order " + order.SK.replace("ORDER#", "") + " is on its way." }
}
},
Source: "[email protected]"
};
return ses.sendEmail(params).promise();
}
Stream view types determine what data is included in stream records:
KEYS_ONLY- Only the key attributesNEW_IMAGE- The entire item after modificationOLD_IMAGE- The entire item before modificationNEW_AND_OLD_IMAGES- Both before and after (most flexible, highest cost)
Use NEW_AND_OLD_IMAGES when you need to detect what changed. Use KEYS_ONLY when you just need to know which items were affected.
TTL for Automatic Expiry
Time to Live (TTL) lets DynamoDB automatically delete items after a specified timestamp. The TTL attribute must contain a Unix epoch timestamp (seconds, not milliseconds). Items are typically deleted within 48 hours of expiration, not instantly.
function createSession(userId, sessionData, callback) {
var now = Math.floor(Date.now() / 1000);
var ttl = now + (24 * 60 * 60); // Expire in 24 hours
var params = {
TableName: "AppTable",
Item: {
PK: "SESSION#" + sessionData.sessionId,
SK: "USER#" + userId,
createdAt: new Date().toISOString(),
ttl: ttl, // DynamoDB TTL attribute
data: sessionData
}
};
docClient.put(params, callback);
}
TTL is free. You do not pay for the delete operations that TTL performs. This makes it ideal for:
- Session data
- Temporary tokens and OTPs
- Cache entries
- Event logs with a retention policy
- Shopping cart items that expire after inactivity
Important: TTL deletions produce stream records with eventName: "REMOVE" and a userIdentity field set to dynamodb.amazonaws.com. Use this to distinguish TTL deletions from application-initiated deletes in your stream processor.
Capacity Modes
On-Demand Mode charges per request. No capacity planning required. DynamoDB scales instantly to handle any traffic volume. This is the right choice when:
- Traffic is unpredictable or spiky
- You are building a new application and do not know your traffic patterns yet
- You prefer simplicity over cost optimization
- Your workload has significant idle periods
Provisioned Mode charges for reserved read and write capacity units (RCUs and WCUs). You specify how much throughput you need, and DynamoDB reserves it. Auto-scaling can adjust capacity based on utilization, but it reacts slowly (minutes, not seconds).
The cost math is straightforward. On-demand costs roughly 6-7x more per request than provisioned capacity at steady-state utilization. If your table has consistent, predictable traffic, provisioned mode with auto-scaling saves significant money.
// Creating a table with provisioned capacity and auto-scaling
var dynamodb = new AWS.DynamoDB();
var params = {
TableName: "AppTable",
KeySchema: [
{ AttributeName: "PK", KeyType: "HASH" },
{ AttributeName: "SK", KeyType: "RANGE" }
],
AttributeDefinitions: [
{ AttributeName: "PK", AttributeType: "S" },
{ AttributeName: "SK", AttributeType: "S" },
{ AttributeName: "GSI1PK", AttributeType: "S" },
{ AttributeName: "GSI1SK", AttributeType: "S" }
],
BillingMode: "PROVISIONED",
ProvisionedThroughput: {
ReadCapacityUnits: 25,
WriteCapacityUnits: 25
},
GlobalSecondaryIndexes: [
{
IndexName: "GSI1",
KeySchema: [
{ AttributeName: "GSI1PK", KeyType: "HASH" },
{ AttributeName: "GSI1SK", KeyType: "RANGE" }
],
Projection: { ProjectionType: "ALL" },
ProvisionedThroughput: {
ReadCapacityUnits: 10,
WriteCapacityUnits: 10
}
}
]
};
dynamodb.createTable(params, function(err, data) {
if (err) console.error("Create table failed:", err);
else console.log("Table created:", data.TableDescription.TableName);
});
My recommendation: Start with on-demand mode. Switch to provisioned with auto-scaling once your traffic patterns stabilize and cost optimization becomes a priority.
Node.js DocumentClient Patterns
The DocumentClient is the recommended way to interact with DynamoDB from Node.js. It handles type marshalling automatically, so you work with native JavaScript types instead of DynamoDB's JSON format.
Retry and Error Handling:
var AWS = require("aws-sdk");
var docClient = new AWS.DynamoDB.DocumentClient({
maxRetries: 3,
retryDelayOptions: {
base: 200 // Base delay in ms, SDK uses exponential backoff
}
});
function getItemWithRetry(key, retries, callback) {
var params = {
TableName: "AppTable",
Key: key
};
docClient.get(params, function(err, data) {
if (err) {
if (err.code === "ProvisionedThroughputExceededException" && retries > 0) {
var delay = Math.pow(2, 3 - retries) * 100;
console.log("Throttled. Retrying in " + delay + "ms...");
setTimeout(function() {
getItemWithRetry(key, retries - 1, callback);
}, delay);
} else {
callback(err);
}
} else {
callback(null, data.Item);
}
});
}
Conditional Writes for Optimistic Locking:
function updateUserProfile(userId, updates, expectedVersion, callback) {
var params = {
TableName: "AppTable",
Key: { PK: "USER#" + userId, SK: "PROFILE" },
UpdateExpression: "SET #name = :name, email = :email, version = :newVersion",
ConditionExpression: "version = :expectedVersion",
ExpressionAttributeNames: {
"#name": "name"
},
ExpressionAttributeValues: {
":name": updates.name,
":email": updates.email,
":newVersion": expectedVersion + 1,
":expectedVersion": expectedVersion
},
ReturnValues: "ALL_NEW"
};
docClient.update(params, function(err, data) {
if (err) {
if (err.code === "ConditionalCheckFailedException") {
callback(new Error("Conflict: item was modified by another process"));
} else {
callback(err);
}
} else {
callback(null, data.Attributes);
}
});
}
Pagination Strategies
DynamoDB returns a maximum of 1 MB of data per query. If there are more results, it returns a LastEvaluatedKey that you use as the ExclusiveStartKey for the next page.
Cursor-Based Pagination (Recommended):
function getOrdersPage(userId, pageSize, lastKey, callback) {
var params = {
TableName: "AppTable",
KeyConditionExpression: "PK = :pk AND begins_with(SK, :sk)",
ExpressionAttributeValues: {
":pk": "USER#" + userId,
":sk": "ORDER#"
},
Limit: pageSize,
ScanIndexForward: false // Newest first
};
if (lastKey) {
params.ExclusiveStartKey = JSON.parse(
Buffer.from(lastKey, "base64").toString("utf8")
);
}
docClient.query(params, function(err, data) {
if (err) {
callback(err);
return;
}
var nextCursor = null;
if (data.LastEvaluatedKey) {
nextCursor = Buffer.from(
JSON.stringify(data.LastEvaluatedKey)
).toString("base64");
}
callback(null, {
items: data.Items,
nextCursor: nextCursor,
count: data.Count
});
});
}
// Express route handler
app.get("/api/users/:userId/orders", function(req, res) {
var pageSize = parseInt(req.query.limit) || 20;
var cursor = req.query.cursor || null;
getOrdersPage(req.params.userId, pageSize, cursor, function(err, result) {
if (err) {
return res.status(500).json({ error: "Failed to fetch orders" });
}
res.json(result);
});
});
Response:
{
"items": [
{ "PK": "USER#u001", "SK": "ORDER#2024-005", "total": 149.99 },
{ "PK": "USER#u001", "SK": "ORDER#2024-004", "total": 89.50 }
],
"nextCursor": "eyJQSyI6IlVTRVIjdTAwMSIsIlNLIjoiT1JERVIjMjAyNC0wMDQifQ==",
"count": 2
}
Encode the LastEvaluatedKey as base64 to create an opaque cursor. Never expose raw DynamoDB keys to clients since they reveal your internal data model.
Complete Working Example: Multi-Tenant SaaS Application
This example implements a single-table design for a SaaS application with organizations, users, and role-based permissions.
Table Design:
Entity | PK | SK | GSI1PK | GSI1SK
---------------|-----------------|---------------------|---------------------------|------------------
Organization | ORG#<orgId> | METADATA | - | -
User Profile | ORG#<orgId> | USER#<email> | EMAIL#<email> | ORG#<orgId>
Role | ORG#<orgId> | ROLE#<roleName> | - | -
Permission | ORG#<orgId> | PERM#<userId>#<res> | USERID#<userId> | PERM#<resource>
Invitation | ORG#<orgId> | INVITE#<email> | EMAIL#<email> | INVITE#<orgId>
API Key | ORG#<orgId> | APIKEY#<keyId> | APIKEY#<hashedKey> | ORG#<orgId>
Full Implementation:
var AWS = require("aws-sdk");
var crypto = require("crypto");
var express = require("express");
var docClient = new AWS.DynamoDB.DocumentClient({
region: process.env.AWS_REGION || "us-east-1"
});
var TABLE_NAME = process.env.DYNAMODB_TABLE || "SaasApp";
var app = express();
app.use(express.json());
// ==========================================
// Organization Operations
// ==========================================
function createOrganization(orgName, ownerEmail, callback) {
var orgId = crypto.randomBytes(8).toString("hex");
var now = new Date().toISOString();
var params = {
TransactItems: [
{
Put: {
TableName: TABLE_NAME,
Item: {
PK: "ORG#" + orgId,
SK: "METADATA",
orgId: orgId,
name: orgName,
plan: "free",
createdAt: now,
ownerEmail: ownerEmail,
entityType: "Organization"
},
ConditionExpression: "attribute_not_exists(PK)"
}
},
{
Put: {
TableName: TABLE_NAME,
Item: {
PK: "ORG#" + orgId,
SK: "USER#" + ownerEmail,
GSI1PK: "EMAIL#" + ownerEmail,
GSI1SK: "ORG#" + orgId,
email: ownerEmail,
role: "owner",
joinedAt: now,
entityType: "User"
}
}
},
{
Put: {
TableName: TABLE_NAME,
Item: {
PK: "ORG#" + orgId,
SK: "ROLE#owner",
permissions: ["*"],
entityType: "Role"
}
}
},
{
Put: {
TableName: TABLE_NAME,
Item: {
PK: "ORG#" + orgId,
SK: "ROLE#member",
permissions: ["read:projects", "write:projects", "read:settings"],
entityType: "Role"
}
}
}
]
};
docClient.transactWrite(params, function(err) {
if (err) {
callback(err);
} else {
callback(null, { orgId: orgId, name: orgName });
}
});
}
// ==========================================
// User Operations
// ==========================================
function addUserToOrg(orgId, email, role, callback) {
var now = new Date().toISOString();
var params = {
TableName: TABLE_NAME,
Item: {
PK: "ORG#" + orgId,
SK: "USER#" + email,
GSI1PK: "EMAIL#" + email,
GSI1SK: "ORG#" + orgId,
email: email,
role: role,
joinedAt: now,
entityType: "User"
},
ConditionExpression: "attribute_not_exists(PK)"
};
docClient.put(params, function(err) {
if (err && err.code === "ConditionalCheckFailedException") {
callback(new Error("User already exists in this organization"));
} else {
callback(err);
}
});
}
function getUserOrgs(email, callback) {
var params = {
TableName: TABLE_NAME,
IndexName: "GSI1",
KeyConditionExpression: "GSI1PK = :pk AND begins_with(GSI1SK, :sk)",
ExpressionAttributeValues: {
":pk": "EMAIL#" + email,
":sk": "ORG#"
}
};
docClient.query(params, function(err, data) {
if (err) {
callback(err);
} else {
callback(null, data.Items);
}
});
}
function getOrgMembers(orgId, callback) {
var params = {
TableName: TABLE_NAME,
KeyConditionExpression: "PK = :pk AND begins_with(SK, :sk)",
ExpressionAttributeValues: {
":pk": "ORG#" + orgId,
":sk": "USER#"
}
};
docClient.query(params, function(err, data) {
if (err) {
callback(err);
} else {
callback(null, data.Items);
}
});
}
// ==========================================
// API Key Operations
// ==========================================
function createApiKey(orgId, keyName, callback) {
var rawKey = crypto.randomBytes(32).toString("hex");
var hashedKey = crypto.createHash("sha256").update(rawKey).digest("hex");
var keyId = crypto.randomBytes(4).toString("hex");
var prefix = rawKey.substring(0, 8);
var params = {
TableName: TABLE_NAME,
Item: {
PK: "ORG#" + orgId,
SK: "APIKEY#" + keyId,
GSI1PK: "APIKEY#" + hashedKey,
GSI1SK: "ORG#" + orgId,
keyId: keyId,
keyPrefix: prefix,
keyName: keyName,
createdAt: new Date().toISOString(),
entityType: "ApiKey"
}
};
docClient.put(params, function(err) {
if (err) {
callback(err);
} else {
// Return the raw key only on creation
callback(null, {
keyId: keyId,
apiKey: rawKey,
prefix: prefix
});
}
});
}
function authenticateApiKey(rawKey, callback) {
var hashedKey = crypto.createHash("sha256").update(rawKey).digest("hex");
var params = {
TableName: TABLE_NAME,
IndexName: "GSI1",
KeyConditionExpression: "GSI1PK = :pk",
ExpressionAttributeValues: {
":pk": "APIKEY#" + hashedKey
}
};
docClient.query(params, function(err, data) {
if (err) {
callback(err);
} else if (data.Items.length === 0) {
callback(new Error("Invalid API key"));
} else {
var keyItem = data.Items[0];
var orgId = keyItem.PK.replace("ORG#", "");
callback(null, { orgId: orgId, keyId: keyItem.keyId });
}
});
}
// ==========================================
// Express Routes
// ==========================================
app.post("/api/organizations", function(req, res) {
createOrganization(req.body.name, req.body.ownerEmail, function(err, org) {
if (err) {
return res.status(500).json({ error: err.message });
}
res.status(201).json(org);
});
});
app.get("/api/organizations/:orgId/members", function(req, res) {
getOrgMembers(req.params.orgId, function(err, members) {
if (err) {
return res.status(500).json({ error: err.message });
}
res.json({ members: members });
});
});
app.post("/api/organizations/:orgId/members", function(req, res) {
addUserToOrg(req.params.orgId, req.body.email, req.body.role || "member", function(err) {
if (err) {
var status = err.message.includes("already exists") ? 409 : 500;
return res.status(status).json({ error: err.message });
}
res.status(201).json({ message: "User added" });
});
});
app.get("/api/users/:email/organizations", function(req, res) {
getUserOrgs(req.params.email, function(err, orgs) {
if (err) {
return res.status(500).json({ error: err.message });
}
res.json({ organizations: orgs });
});
});
app.post("/api/organizations/:orgId/api-keys", function(req, res) {
createApiKey(req.params.orgId, req.body.name, function(err, key) {
if (err) {
return res.status(500).json({ error: err.message });
}
res.status(201).json(key);
});
});
var port = process.env.PORT || 3000;
app.listen(port, function() {
console.log("SaaS API running on port " + port);
});
This single-table design supports all of the following access patterns with efficient queries:
- Get organization details (PK =
ORG#<id>, SK =METADATA) - List all members of an org (PK =
ORG#<id>, SK begins_withUSER#) - Find all orgs a user belongs to (GSI1: PK =
EMAIL#<email>, SK begins_withORG#) - Authenticate an API key (GSI1: PK =
APIKEY#<hash>) - List all API keys for an org (PK =
ORG#<id>, SK begins_withAPIKEY#) - Get org roles (PK =
ORG#<id>, SK begins_withROLE#)
Common Issues and Troubleshooting
1. ValidationException: One or more parameter values were invalid
ValidationException: One or more parameter values were invalid:
Condition parameter type does not match schema type
This happens when your key attribute types do not match the table schema. DynamoDB partition and sort keys are defined as S (String), N (Number), or B (Binary). If your schema says S but you pass a number, you get this error. The DocumentClient helps, but you can still hit this with null or undefined values:
// This will fail if orderId is undefined
var params = {
TableName: "AppTable",
Key: { PK: "USER#u001", SK: "ORDER#" + orderId }
};
// Always validate inputs
if (!orderId) {
return callback(new Error("orderId is required"));
}
2. ProvisionedThroughputExceededException
ProvisionedThroughputExceededException: Rate exceeded for table AppTable
Your table cannot keep up with the request rate. This happens with provisioned mode when you exhaust your capacity. Solutions: enable auto-scaling, switch to on-demand mode, or implement exponential backoff. Also check for hot partitions by examining CloudWatch metrics per partition key.
3. TransactionConflictException
TransactionConflictException: Transaction is ongoing for the item
Two transactions are trying to modify the same item simultaneously. DynamoDB uses optimistic concurrency control for transactions. One will succeed and the other will fail. Your application must catch this and retry:
function retryTransaction(params, maxRetries, callback) {
var attempt = 0;
function tryOnce() {
docClient.transactWrite(params, function(err, data) {
if (err && err.code === "TransactionConflictException" && attempt < maxRetries) {
attempt++;
var delay = Math.pow(2, attempt) * 50;
setTimeout(tryOnce, delay);
} else {
callback(err, data);
}
});
}
tryOnce();
}
4. Item Size Exceeds 400 KB Limit
ValidationException: Item size has exceeded the maximum allowed size of 400 KB
A single DynamoDB item cannot exceed 400 KB. This catches people who store large JSON blobs or arrays that grow without bounds. Solutions: store large payloads in S3 and keep a reference in DynamoDB, split the item into multiple items using sort key segments, or compress the data before storage.
5. Query Key Condition Not Supported
ValidationException: Query key condition not supported
You used a filter expression operator in the KeyConditionExpression. Key conditions only support =, <, <=, >, >=, BETWEEN, and begins_with. You cannot use contains, size, or attribute_exists in key conditions. Move those to FilterExpression instead.
Best Practices
Design for access patterns first. List every query your application needs before creating the table. DynamoDB punishes you for schema changes far more than relational databases do.
Use composite sort keys for flexible queries. A sort key like
STATUS#shipped#2024-01-15lets you query by status, by status and date range, or by exact status and date. Design your sort keys to support the most queries with the fewest indexes.Never scan in production request paths. Scans read the entire table. Put a query behind every API endpoint. If you cannot express your access pattern as a query, you are missing a GSI.
Implement exponential backoff with jitter for retries. The AWS SDK handles basic retries, but your application-level retries should add randomized jitter to prevent thundering herd problems when many clients retry simultaneously.
Use projection expressions to reduce response size. If you only need three attributes from an item with fifty, specify
ProjectionExpressionto reduce network transfer and speed up responses:
var params = {
TableName: "AppTable",
Key: { PK: "USER#u001", SK: "PROFILE" },
ProjectionExpression: "#name, email, #role",
ExpressionAttributeNames: {
"#name": "name",
"#role": "role"
}
};
Set TTL on temporary data from day one. Session tokens, invite links, temporary locks, and cache entries should all have TTL attributes. Free automatic cleanup prevents table bloat and eliminates the need for batch cleanup jobs.
Use DynamoDB Local for development and testing. Running the downloadable version of DynamoDB locally means your development environment does not depend on AWS connectivity and does not incur costs. Your tests run faster and your CI pipeline stays simple.
Monitor with CloudWatch and set alarms. Track
ConsumedReadCapacityUnits,ConsumedWriteCapacityUnits,ThrottledRequests, andSystemErrors. Set alarms at 80% capacity utilization so you have time to react before throttling begins.Keep GSI count low. Each GSI adds write cost and storage cost. The overloaded GSI pattern lets you serve multiple access patterns from a single index. Aim for two or three GSIs maximum per table.
Use consistent reads only when necessary. Eventually consistent reads cost half as much as strongly consistent reads and are sufficient for most use cases. Reserve consistent reads for operations where stale data causes correctness issues, like checking a user's balance before processing a payment.
