Retries Will Happen: Writing Idempotent AWS Lambda Handlers

It is easy to write a Lambda handler as if each event arrives exactly once. Not every event does. Lambda and the services around it retry on your behalf, and a handler that sends an email, charges a card or calls a partner API will happily do it twice. You want those retries; the goal is to make a repeated event harmless.

Where the duplicates come from

• Synchronous (API Gateway, RequestResponse): Lambda doesn't retry function errors, but callers can. The AWS SDK retries client timeouts and 5xx errors, so a slow invocation can run twice.
• Asynchronous (S3, SNS, EventBridge, InvocationType: 'Event'): on a function error, Lambda tries twice more, one and then two minutes later. The docs also warn that an event can arrive more than once even when nothing failed.
• Event source mappings: SQS standard queues are at-least-once, and if an invocation fails, every message in the batch becomes visible again, including those you already handled. Kinesis and DynamoDB Streams retry a failed batch until it succeeds or the records expire.

A timeout counts as a function error, even if the side effect completed just before the function was stopped. And if the SQS visibility timeout is shorter than your processing time, a second invocation can pick up a message the first is still working on. AWS recommends at least six times the function timeout.

Claim the key before the side effect

Before any work, create a record for the event's idempotency key with a DynamoDB conditional put. Exactly one invocation can create the item; everyone else gets a ConditionalCheckFailedException. A "read, then write if missing" check can't promise that, since two concurrent invocations can both read "missing".

The table has a string partition key pk and TTL on expiresAt. Using the v2 SDK that ships with the Node.js 14 runtime:

import { AWSError, DynamoDB } from 'aws-sdk'

const db = new DynamoDB.DocumentClient()
const TableName = process.env.IDEMPOTENCY_TABLE as string
const KEEP_FOR = 24 * 60 * 60 // seconds

export class StillInProgressError extends Error {}

export async function runOnce<T>(key: string, lockSeconds: number, work: () => Promise<T>): Promise<T> {
  const now = Math.floor(Date.now() / 1000)

  try {
    await db
      .put({
        TableName,
        Item: { pk: key, status: 'IN_PROGRESS', expiresAt: now + lockSeconds },
        // a new key, or a previous record that has expired
        ConditionExpression: 'attribute_not_exists(pk) OR expiresAt < :now',
        ExpressionAttributeValues: { ':now': now },
      })
      .promise()
  } catch (err) {
    if ((err as AWSError).code !== 'ConditionalCheckFailedException') throw err

    const { Item } = await db.get({ TableName, Key: { pk: key }, ConsistentRead: true }).promise()
    if (Item?.status === 'COMPLETED') return Item.result as T // duplicate: replay the stored result
    throw new StillInProgressError(`${key} is being processed by another invocation`)
  }

  let result: T
  try {
    result = await work()
  } catch (err) {
    await db.delete({ TableName, Key: { pk: key } }).promise() // let the retry start over
    throw err
  }

  await db
    .update({
      TableName,
      Key: { pk: key },
      UpdateExpression: 'SET #status = :done, #result = :result, expiresAt = :keepUntil',
      ExpressionAttributeNames: { '#status': 'status', '#result': 'result' },
      ExpressionAttributeValues: { ':done': 'COMPLETED', ':result': result, ':keepUntil': now + KEEP_FOR },
    })
    .promise()

  return result
}

Notes on the code:

• status and result are DynamoDB reserved words, hence the aliases.
• TTL takes epoch seconds, and deletion is lazy: expired items typically disappear within 48 hours and still show up in reads until then. So the condition checks expiresAt itself.
• While the work runs, expiresAt is a short lock: if the function crashes or times out halfway, it runs out and a retry can take over. On success, the record is kept for a day.
• The read after a failed condition is strongly consistent, so it sees the record just written.
• A duplicate that finds IN_PROGRESS throws: for SQS and async invocations that means "retry later", and behind API Gateway you return a 409. A client retrying after the first request finished gets the stored result.

Wiring it into an SQS handler:

import { SQSHandler } from 'aws-lambda'
import { runOnce } from './idempotency'
import { sendConfirmationEmail } from './email'

export const handler: SQSHandler = async (event, context) => {
  for (const record of event.Records) {
    const order = JSON.parse(record.body)
    // lock the key for as long as this invocation can run
    const lockSeconds = Math.ceil(context.getRemainingTimeInMillis() / 1000)

    await runOnce(`order-confirmation#${order.orderId}`, lockSeconds, () => sendConfirmationEmail(order))
  }
}

What to key on

The key should identify the operation, not the delivery:

• Not context.awsRequestId: a redelivered SQS message arrives in a new invocation with a new request ID.
• The SQS messageId survives redeliveries, but a producer that retries SendMessage creates a second message with a new ID.
• Usually best: the action plus a business ID, like order-confirmation# and the order ID. For HTTP APIs, accept an idempotency key header from the client, as many payment APIs do.
• If nothing natural exists, hash the fields that define "the same request".

Takeaways

• Treat async and queue- or stream-driven invocations as at-least-once.
• Claim the key with a conditional write, never read-then-write.
• Store the result, so a retried caller gets the same answer.
• This narrows the window without closing it: if the work succeeds and saving the result fails, a retry can repeat it. When a downstream API accepts an idempotency key, pass yours along too.