Skip to content

Lambda DynamoDB Streams event source mapping with StartingPosition=LATEST can skip records after control-plane update without CloudTrail trace #487

@WindAzure

Description

@WindAzure

I am trying to understand the exact behavioral contract of Lambda event source mappings for DynamoDB Streams, specifically around what constitutes an “update” or “re-application” of the mapping, and how that affects event delivery guarantees.

During an investigation of a small number of missing downstream effects, I observed a scenario where DynamoDB Stream records exist and are fully readable via the Streams API, but are never delivered to the Lambda function.

https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html#dyanmo-db-stream-poll
The documentation mentions that records may be skipped depending on the starting position, but it does not clearly explain when or why this can occur, nor what actions or systems are responsible for triggering such behavior.

Observed facts:

  • DynamoDB Stream records are present and readable using GetRecords with TRIM_HORIZON.
  • The Lambda event source mapping is configured with StartingPosition = LATEST.
  • The event source mapping reports:
    • State = Enabled
    • StateTransitionReason = "User action"
    • LastProcessingResult = "OK"
  • The records’ ApproximateCreationDateTime falls shortly before the event source mapping’s LastModified timestamp.
  • These records are never delivered to the Lambda function, and no corresponding invocation or log entry is observed.
  • Lambda metrics (Invocations, Errors, IteratorAge) appear normal.
  • There was no intentional IaC deployment, no manual console change, and no known infrastructure or configuration change during this time window.
  • CloudTrail shows no CreateEventSourceMapping or UpdateEventSourceMapping API calls, and no IAM principal associated with any update.
  • From the perspective of the DynamoDB Streams API, the records are intact and accessible.

From a user’s perspective, this raises a critical documentation and observability question: the mapping appears to have been updated or re-applied (StateTransitionReason = "User action"), yet there is no visible actor, API call, or infrastructure change that explains who or what performed this update.

Questions:

  1. What specific actions or conditions are considered an “update” or “re-application” of a Lambda event source mapping? Does this include internal AWS control-plane reconciliations or other automated processes?
  2. Under which of these conditions is StartingPosition reinterpreted, potentially causing existing DynamoDB Stream records to be skipped when StartingPosition = LATEST?
  3. Is "User action" intended to include internal AWS control-plane operations that do not have a CloudTrail-visible principal?
  4. If such re-applications can occur without any user-visible action or audit trail, what guarantees (if any) does Lambda provide about event completeness?
  5. Is using a self-managed or managed stream consumer (for example, via Kinesis / KCL) the only officially supported way to guarantee no gaps in DynamoDB Stream processing?

The documentation currently states that records may be skipped, but it does not clearly define when this can happen, who or what triggers it, or how users should reason about these risks. Clarifying the exact update mechanisms and their impact on StartingPosition would help users understand the true guarantees of Lambda event source mappings and design appropriate safeguards.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions