Bump amazon-kinesis-client from 1.10.0 to 1.14.2

[dependabot-preview]

Bumps amazon-kinesis-client from 1.10.0 to 1.14.2.

Release notes

Sourced from amazon-kinesis-client's releases.

Release 1.14.2 of the Amazon Kinesis Client Library for Java

Latest Release (1.14.2 - February 24, 2021)

• Milestone#57
• #790 Fixing a bug that caused paginated ListShards calls with the ShardFilter parameter to fail when the lease table was being initialized.

Release 1.14.1 of the Amazon Kinesis Client Library for Java

Latest Release (1.14.1 - January 27, 2021)

• Milestone#56

• Fix for cross DDB table interference when multiple KCL applications are run in same JVM.

• Fix and guards to avoid potential checkpoint rewind during shard end, which may block children shard processing.

• Fix for thread cycle wastage on InitializeTask for deleted shard.

• Improved logging in LeaseCleanupManager that would indicate why certain shards are not cleaned up from the lease table.

Release v1.14.0 of the Amazon Kinesis Client Library for Java

Release 1.14.0 (August 17, 2020)

Milestone#50

• Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.

• When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.

  • Currently, the following shard filters are supported:
    • AT_TRIM_HORIZON - the response includes all the shards that were open at TRIM_HORIZON.
    • AT_LATEST - the response includes only the currently open shards of the data stream.
    • AT_TIMESTAMP - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
  • ShardFilter is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at KinesisClientLibConfiguration#initialPositionInStreamExtended.
  • For more information about ShardFilter, see the official AWS documentation on ShardFilter.

• Introducing support for the ChildShards response of the GetRecords API to perform lease/shard synchronization that happens at SHARD_END for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.

  • For KCL 1.x applications, this uses the ChildShards response of the GetRecords API.
  • For more information, see the official AWS Documentation on GetRecords and ChildShard.

• KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When KinesisClientLibConfiguration#shardSyncStrategyType is set to ShardSyncStrategyType.SHARD_END, PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When KinesisClientLibConfiguration#shardSyncStrategyType is set to ShardSyncStrategyType.PERIODIC, leasesRecoveryAuditorInconsistencyConfidenceThreshold is ignored.

  • New configuration options are available to configure PeriodicShardSyncManager in KinesisClientLibConfiguration

  Name	Default	Description
  leasesRecoveryAuditorInconsistencyConfidenceThreshold	3	Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for ShardSyncStrategyType.SHARD_END.

  • New CloudWatch metrics are also now emitted to monitor the health of PeriodicShardSyncManager:

  Name	Description
  NumStreamsWithPartialLeases	Number of streams that had holes in their hash ranges.
  NumStreamsToSync	Number of streams which underwent a full shard sync.

• Introducing deferred lease cleanup. Leases will be deleted asynchronously by LeaseCleanupManager upon reaching SHARD_END, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.

  • New configuration options are available to configure LeaseCleanupManager.

  | Name | Default | Description |

... (truncated)

Changelog

Sourced from amazon-kinesis-client's changelog.

Latest Release (1.14.2 - February 24, 2021)

• Milestone#57
• #790 Fixing a bug that caused paginated ListShards calls with the ShardFilter parameter to fail when the lease table was being initialized.

Release (1.14.1 - January 27, 2021)

• Milestone#56

• Fix for cross DDB table interference when multiple KCL applications are run in same JVM.

• Fix and guards to avoid potential checkpoint rewind during shard end, which may block children shard processing.

• Fix for thread cycle wastage on InitializeTask for deleted shard.

• Improved logging in LeaseCleanupManager that would indicate why certain shards are not cleaned up from the lease table.

Release (1.14.0 - August 17, 2020)

• Milestone#50

• Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.

• When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.

  • Currently, the following shard filters are supported:
    • AT_TRIM_HORIZON - the response includes all the shards that were open at TRIM_HORIZON.
    • AT_LATEST - the response includes only the currently open shards of the data stream.
    • AT_TIMESTAMP - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
  • ShardFilter is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at KinesisClientLibConfiguration#initialPositionInStreamExtended.
  • For more information about ShardFilter, see the official AWS documentation on ShardFilter.

• Introducing support for the ChildShards response of the GetRecords API to perform lease/shard synchronization that happens at SHARD_END for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.

  • For KCL 1.x applications, this uses the ChildShards response of the GetRecords API.
  • For more information, see the official AWS Documentation on GetRecords and ChildShard.

• KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When KinesisClientLibConfiguration#shardSyncStrategyType is set to ShardSyncStrategyType.SHARD_END, PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When KinesisClientLibConfiguration#shardSyncStrategyType is set to ShardSyncStrategyType.PERIODIC, leasesRecoveryAuditorInconsistencyConfidenceThreshold is ignored.

  • New configuration options are available to configure PeriodicShardSyncManager in KinesisClientLibConfiguration

  Name	Default	Description
  leasesRecoveryAuditorInconsistencyConfidenceThreshold	3	Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for ShardSyncStrategyType.SHARD_END.

  • New CloudWatch metrics are also now emitted to monitor the health of PeriodicShardSyncManager:

  Name	Description
  NumStreamsWithPartialLeases	Number of streams that had holes in their hash ranges.
  NumStreamsToSync	Number of streams which underwent a full shard sync.

• Introducing deferred lease cleanup. Leases will be deleted asynchronously by LeaseCleanupManager upon reaching SHARD_END, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.

  • New configuration options are available to configure LeaseCleanupManager.

  Name	Default	Description
  leaseCleanupIntervalMillis	1 minute	Interval at which to run lease cleanup thread.

... (truncated)

Commits

• f38dd18 Preparing for 1.14.2 release (#793)
• a13ed13 Fixing bug where ShardFilter parameter for ListShards was being passed in for...
• b4f6f8f Merge pull request #782 from awslabs/kcl_1.14.1_release
• 8ba93e6 Release notes update
• 4f6f4e5 KCL 1.14.1 release
• 539550b Merge pull request #781 from awslabs/dead_shard_initialize_cycle_fix
• 9ed1e10 Fixing unit test
• 5142ce9 Fixing exception type
• b04e93e Merge pull request #779 from awslabs/shard_end_corruption_issues_1
• 530fd47 Fixing record processor processing deleted leases in cycle
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in the .dependabot/config.yml file in this repo:

• Update frequency
• Automerge options (never/patch/minor, and dev/runtime dependencies)
• Out-of-range updates (receive only lockfile updates, if desired)
• Security updates (receive only security updates, if desired)

[dependabot-preview]

Superseded by #589.