[DO NOT MERGE]: Leverage x-ms-cosmos-hub-region-processing-only for 404 Read Session Not Available cross-region retry scenarios.

[jeet1995]

Motivation

The pull request serves as the first iteration which integrates x-ms-cosmos-hub-region-processing-only header. Setting the value of this header to true will allow a Cosmos DB backend node to return a 403:3 in case the backend node belongs to a non-hub physical partition.

Using this setup, the CosmosClient instance can determine partition-set level hub which in the first iteration helps in region detection of 404 Read Session Not Available cross-region detection for Single-Writer accounts. This is needed in particular when failover happens in a rolling-manner partition-set by partition-set and in Per-Partition Automatic Failover cases where hub is a partition-set granular notion. Simply relying on LocationCache to provide account-level hub region is incorrect.

Scope

In this pull request, the focus is on how 404 Read Session Not Available cross-region retry handling happens for Single-Writer accounts.

Critical Changes

The approach taken here is to pin the x-ms-cosmos-hub-region-processing-only once a request hits a 404 Read Session Not Available. This ensures an operation (a construct which encapsulates several I/O calls) is sticky to the hub region.

The other change, as a result of keeping the header set, a non-write operation can now see 403 Write Forbidden. As the goal is to determine hub, 403 Write Forbidden handling when such header is set is to ensure cycle through of available read regions as maintained by LocationCache.

Testing

Per-Partition Automatic Failover

The approach was to set a naming configuration (simulateRevokeLocalWriteStatusOfPartition) consumed by the service fabric process mapped to the original hub region (say North Central US) for a particular physical partition.

Post that, a 404 Read Session Not Available is injected into the same partition for which the write privilege was revoked (North Central US).

Using a "pure-read" workload, the goal is to assert whether the read (a readItem operation) gets a 200 status code from the partition-set specific hub region.

As "reads" can get a 403 Write Forbidden status code, these "reads" can update partition-set level hub which future reads and writes can use.

Pending item: extend this test to Query and ChangeFeed operations.

Single-Writer accounts with no PPAF enabled

Pending item: The expected test setup is to execute a write region change on an account with a physical partition-set count in the order of ~2000 (typical in our DR drills) and to subject the account to a "read-only" workload and see how hub-region stickiness holds up.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[FabianMeiswinkel]

Why is this getting removed?

[jeet1995]

@FabianMeiswinkel Ignore this change - I forgot for a moment why I added the per-partition set failover at both places. Immediately after the 403-3 is detected and also in shouldRetryOnEndpointFailureAsync (it can be relevant in Gateway Endpoint Unavailability).

[FabianMeiswinkel]

LGTM except for one question whether one change is intended - and if so why?

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).