Batching

Lower AWS Lambda invocation costs by processing multiple queue or stream records per function execution with safe batching settings.

Overview

Batching lets a single Lambda invocation process multiple records at once (from SQS, Kinesis, or DynamoDB Streams), so you pay for fewer invocations while maintaining throughput.

Combined with event filtering, batching is one of the most effective ways to lower Lambda costs for high-volume event-driven workloads.

When batching makes sense

  • High-volume, non-latency-sensitive workloads (analytics, logs, ETL).
  • Workloads where per-record isolation is not critical (you can tolerate batch-level retries).
  • Streams or queues where producers already aggregate events naturally.

Avoid very large batches when:

  • You need low latency per record.
  • A single bad record should not cause the entire batch to be retried repeatedly.
  • Your function has heavy per-batch initialization and can’t handle large payloads.
  • You haven’t configured partial-failure handling and DLQs/on-failure destinations to isolate bad records.

SQS batch configuration

For SQS, configure a batch size so each invocation processes multiple messages:

aws lambda create-event-source-mapping \
  --function-name my-function \
  --event-source-arn arn:aws:sqs:region:account:queue \
  --batch-size 10

Tune --batch-size based on:

  • Average processing time per message.
  • Error rate (large batches + high error rate can increase retries).
  • Max payload size your function can handle comfortably.
  • Partial failures: Use reportBatchItemFailures for SQS so one bad message doesn’t poison the whole batch and you can re-drive only failed records.

Kinesis and DynamoDB Streams batching

For Kinesis and DynamoDB Streams, Lambda already polls and delivers records in batches. You can adjust:

  • Batch size: max number of records per batch.
  • Batch window: how long to wait to accumulate a full batch.
  • Partial failures: Enable BisectBatchOnFunctionError and on-failure destinations/streams so problematic records are isolated and retried without reprocessing entire batches.

Example (Kinesis):

aws lambda create-event-source-mapping \
  --function-name my-stream-processor \
  --event-source-arn arn:aws:kinesis:region:account:stream/my-stream \
  --batch-size 100 \
  --maximum-batching-window-in-seconds 5

Use a small batching window for near-real-time processing and a larger window when you can trade a bit of latency for better batching and lower cost.

Monitoring and tuning

  • Track ApproximateNumberOfMessagesVisible (SQS) or GetRecords throughput (Kinesis/DynamoDB Streams) to ensure you’re keeping up.
  • Watch IteratorAge for streams to confirm batching is not causing unacceptable lag.
  • Log batch sizes and per-record processing time to identify optimal configuration.
  • Configure DLQs or on-failure destinations for all batch-consuming functions so any records that still fail after retries are captured for later inspection instead of being retried indefinitely.

Remember that batching and event filtering stack extremely well: filter out noise at the source, then batch the remaining high-value events to minimize invocations and cost.

Start with conservative batch sizes, observe behavior in production, then increase incrementally to capture more savings.