From bb7b2e94c247fad3ee2c4af47c3469b744bb1d91 Mon Sep 17 00:00:00 2001 From: danielresnick Date: Wed, 29 Oct 2025 14:47:35 +1300 Subject: [PATCH 1/5] resumable: clarify handling of conflicts and retry policies See #3275 for discussion. --- draft-ietf-httpbis-resumable-upload.md | 75 ++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 9 deletions(-) diff --git a/draft-ietf-httpbis-resumable-upload.md b/draft-ietf-httpbis-resumable-upload.md index 4d140d1444..9f90be1f47 100644 --- a/draft-ietf-httpbis-resumable-upload.md +++ b/draft-ietf-httpbis-resumable-upload.md @@ -390,7 +390,7 @@ If the client received a final response with a - `2xx (Successful)` status code and the entire representation data was transferred in the request content, the upload is complete and the response comes from the targeted resource processing the representation. - `2xx (Successful)` status code and the entire representation data was not transferred in the request content, the `Location` response header field points the client to the created upload resource. The client can continue appending representation data to it ({{upload-appending}}). - `4xx (Client Error)` status code, the client SHOULD NOT attempt to retry or resume the upload, unless the semantics of the response allow or recommend the client to retry the request. -- `5xx (Server Error)` status code or no final response at all due to connectivity issues, the client MAY automatically attempt upload resumption by retrieving the current offset ({{offset-retrieving}}) if it received the URI of the upload resource in a `104 (Upload Resumption Supported)` interim response. +- `5xx (Server Error)` status code or no final response at all due to connectivity issues, the client SHOULD automatically attempt upload resumption by retrieving the current offset ({{offset-retrieving}}) if it received the URI of the upload resource in a `104 (Upload Resumption Supported)` interim response. ### Server Behavior @@ -511,11 +511,13 @@ If the client received a response with a - `2xx (Successful)` status code, the client can continue appending representation data to it ({{upload-appending}}) if the upload is not complete yet. - `307 (Temporary Redirect)` or `308 (Permanent Redirect)` status code, the client MAY retry retrieving the offset from the new URI. - `4xx (Client Error)` status code, the client SHOULD NOT attempt to retry or resume the upload, unless the semantics of the response allow or recommend the client to retry the request. -- `5xx (Server Error)` status code or no final response at all due to connectivity issues, the client MAY retry retrieving the offset. +- `5xx (Server Error)` status code or no final response at all due to connectivity issues, the client SHOULD retry retrieving the offset. ### Server Behavior {#offset-retrieving-server} -A successful response to a `HEAD` request against an upload resource +Upon receiving a `HEAD` request, a server MAY treat it as a hint to abort a previous, possibly stalled, upload creation or append request for the same resource. See {{concurrency}} for more details on handling concurrency. + +A successful response to a `HEAD` request against an upload resource: - MUST include the offset in the `Upload-Offset` header field ({{upload-offset}}), - MUST include the completeness state in the `Upload-Complete` header field ({{upload-complete}}), @@ -577,10 +579,9 @@ If the client received a final response with the `Upload-Complete: ?0` header fi - A `2xx (Successful)` status code indicates that representation data was appended but the upload is not complete. The client can continue appending representation data. - For a `307 (Temporary Redirect)` or `308 (Permanent Redirect)` status code, the client MAY retry appending to the new URI. -- For a `4xx (Client Error)` status code, the client SHOULD NOT attempt to retry or resume the upload, unless the semantics of the response allow or recommend the client to retry the request. -- For a `5xx (Server Error)` status code, the client MAY automatically attempt upload resumption by retrieving the current offset ({{offset-retrieving}}). - -If no final response was received at all due to connectivity issues, the client MAY automatically attempt upload resumption by retrieving the current offset ({{offset-retrieving}}). +- For a `409 (Conflict)` status code, the client SHOULD attempt to resume the upload by using the offset from the `Upload-Offset` response header field. +- For other `4xx (Client Error)` status codes, the client SHOULD NOT attempt to retry or resume the upload, unless the semantics of the response allow or recommend the client to retry the request. +- For a `5xx (Server Error)` status code or if no final response was received at all due to connectivity issues, the client SHOULD automatically attempt upload resumption by retrieving the current offset ({{offset-retrieving}}). ### Server Behavior @@ -653,6 +654,56 @@ Content-Type: application/json } ~~~ +C) The following example shows a client attempting to resume an upload with an incorrect offset. The server rejects the request with a `409 (Conflict)` status code and includes the correct offset in the `Upload-Offset` header field. The client then retries the request with the correct offset. + +~~~ http-message +PATCH /upload/e422fcae4 HTTP/1.1 +Host: example.com +Upload-Complete: ?1 +Upload-Offset: 1230000001 +Content-Length: 4567890 +Content-Type: application/partial-upload + +[content (4567890 bytes)] +~~~ + +~~~ http-message +HTTP/1.1 409 Conflict +Upload-Offset: 1230000000 +Content-Type: application/problem+json + +{ + "type":"https://iana.org/assignments/http-problem-types#\ + mismatching-upload-offset", + "title": "offset from request does not match offset of resource", + "expected-offset": 1230000000, + "provided-offset": 1230000001 +} +~~~ + +~~~ http-message +PATCH /upload/e422fcae4 HTTP/1.1 +Host: example.com +Upload-Complete: ?1 +Upload-Offset: 1230000000 +Content-Length: 4567891 +Content-Type: application/partial-upload + +[content (4567891 bytes)] +~~~ + +~~~ http-message +HTTP/1.1 200 OK +Upload-Complete: ?1 +Content-Type: application/json + +{ + "metadata": { + [...] + } +} +~~~ + ## Upload Cancellation {#upload-cancellation} ### Client Behavior @@ -686,9 +737,15 @@ Resumable uploads, as defined in this document, do not permit uploading represen Even if the client is well-behaved and doesn't send concurrent requests, network interruptions can occur in such a way that the client considers a request as failed while the server is unaware of the problem and considers the request still ongoing. The client might then try to resume the upload with the best intentions, resulting in concurrent requests from the server's perspective. Therefore, the server MUST take measures to prevent race conditions, data loss and corruption from concurrent requests to append representation data ({{upload-appending}}) and/or cancellation ({{upload-cancellation}}) to the same upload resource. In addition, the server MUST NOT send outdated information in responses when retrieving the offset ({{offset-retrieving}}). This means that the offset sent by the server MUST be accepted in a subsequent request to append representation data if no other request to append representation data or cancel was received in the meantime. In other words, clients have to be able to use received offsets. -The RECOMMENDED approach is as follows: If an upload resource receives a new request to retrieve the offset ({{offset-retrieving}}), append representation data ({{upload-appending}}), or cancel the upload ({{upload-cancellation}}) while a previous request for creating the upload ({{upload-creation}}) or appending representation data ({{upload-appending}}) is still ongoing, the resource SHOULD prevent race conditions, data loss, and corruption by terminating the previous request before processing the new request. Due to network delay and reordering, the resource might still be receiving representation data from an ongoing transfer for the same upload resource, which in the client's perspective has failed. Since the client is not allowed to perform multiple transfers in parallel, the upload resource can assume that the previous attempt has already failed. Therefore, the server MAY abruptly terminate the previous HTTP connection or stream. +To meet these requirements, a server can use various strategies. Three common approaches are: + +1. **Latest Request Wins**: If the server receives a new request while a previous one is in-flight, it abruptly terminates the previous HTTP connection or stream before processing the new request. + +2. **Exclusive Locking**: The server processes requests sequentially, effectively creating a lock. A new request is only processed after the previous one completes. This can be simpler to implement but may lead to delays if a request hangs from the server's perspective. + +3. **Optimistic Concurrency Control**: By maintaining the upload's state in a strongly consistent datastore, the server can atomically check if the `Upload-Offset` in a request matches the resource's current state. If they do not match, the server can reject the request with a `409 (Conflict)` status code. -Since implementing this approach is not always technically possible or feasible, other measures can be considered as well. A simpler approach is that the server only processes a new request to retrieve the offset ({{offset-retrieving}}), append representation data ({{upload-appending}}), or cancellation ({{upload-cancellation}}) once all previous requests have been processed. This effectively implements exclusive access to the upload resource through an access lock. However, since network interruptions can occur in ways that cause the request to hang from the server's perspective, it might take the server significant time to realize the interruption and time out the request. During this period, the client will be unable to access the resource and resume the upload, causing friction for the end users. Therefore, the recommended approach is to terminate previous requests to enable quick resumption of uploads. +Servers SHOULD choose a strategy that best fits their architecture while fulfilling the requirements of this section. Regardless of the chosen strategy, clients MUST be prepared to handle a `409 (Conflict)` response as a recoverable error, as described in {{upload-appending}}. # Status Code `104 (Upload Resumption Supported)` {#status-code-104} From 0e45118d6cefadd9b0d555b780f6924cc3850eb9 Mon Sep 17 00:00:00 2001 From: Daniel Resnick Date: Wed, 29 Oct 2025 15:27:42 +1300 Subject: [PATCH 2/5] Update changelog. --- draft-ietf-httpbis-resumable-upload.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/draft-ietf-httpbis-resumable-upload.md b/draft-ietf-httpbis-resumable-upload.md index 9f90be1f47..f844280dbf 100644 --- a/draft-ietf-httpbis-resumable-upload.md +++ b/draft-ietf-httpbis-resumable-upload.md @@ -1049,7 +1049,10 @@ Reference: ## Since draft-ietf-httpbis-resumable-upload-10 {:numbered="false"} -None yet. +* Clarify client and server handling of `409 (Conflict)` responses, requiring clients to treat them as recoverable and adding an example. +* Refine the "Concurrency" section to outline different server strategies (Latest Request Wins, Exclusive Locking, Optimistic Concurrency Control). +* Strengthen the recommendation for retrying `5xx` errors and connection issues to `SHOULD`. +* Clarify that a `HEAD` request in the "Offset Retrieval" section can serve as a hint for servers to abort prior in-flight requests. ## Since draft-ietf-httpbis-resumable-upload-09 {:numbered="false"} From 67464e80476e5f2c2b97d0b69f89c7b9367ba563 Mon Sep 17 00:00:00 2001 From: Daniel Resnick Date: Thu, 30 Oct 2025 10:26:12 +1300 Subject: [PATCH 3/5] - Improved wording throughout. - Addressed feedback around inconsistencies (HAVE vs MUST). - Removed example 409 conflict recovery. --- draft-ietf-httpbis-resumable-upload.md | 60 +++----------------------- 1 file changed, 5 insertions(+), 55 deletions(-) diff --git a/draft-ietf-httpbis-resumable-upload.md b/draft-ietf-httpbis-resumable-upload.md index f844280dbf..d01022a7ff 100644 --- a/draft-ietf-httpbis-resumable-upload.md +++ b/draft-ietf-httpbis-resumable-upload.md @@ -504,7 +504,7 @@ If the client wants to resume the upload after an interruption, it has to know t The offset can be less than or equal to the number of bytes of representation data that the client has already sent. The client MAY reject an offset which is greater than the number of bytes it has already sent during this upload by not continuing the upload and canceling the upload resource ({{upload-cancellation}}). The client is expected to handle backtracking of a reasonable length. If the offset is invalid for this upload, or if the client cannot backtrack to the offset and reproduce the same representation data it has already sent, the upload MUST be considered a failure. The client then MUST NOT continue the upload and SHOULD cancel the upload ({{upload-cancellation}}). -The client MUST NOT perform offset retrieval while creation ({{upload-creation}}) or appending ({{upload-appending}}) is in progress as this can cause the previous request to be terminated by the server as described in {{concurrency}}. +The client MUST NOT perform offset retrieval while creation ({{upload-creation}}) or appending ({{upload-appending}}) is in progress as this may cause the previous request to be terminated by the server as described in {{concurrency}}. If the client received a response with a @@ -579,7 +579,7 @@ If the client received a final response with the `Upload-Complete: ?0` header fi - A `2xx (Successful)` status code indicates that representation data was appended but the upload is not complete. The client can continue appending representation data. - For a `307 (Temporary Redirect)` or `308 (Permanent Redirect)` status code, the client MAY retry appending to the new URI. -- For a `409 (Conflict)` status code, the client SHOULD attempt to resume the upload by using the offset from the `Upload-Offset` response header field. +- For a `409 (Conflict)` status code, the client SHOULD attempt to resume the upload by either retrieving the current offset ({{offset-retrieving}}) or using the value from the response's `Upload-Offset` header field. - For other `4xx (Client Error)` status codes, the client SHOULD NOT attempt to retry or resume the upload, unless the semantics of the response allow or recommend the client to retry the request. - For a `5xx (Server Error)` status code or if no final response was received at all due to connectivity issues, the client SHOULD automatically attempt upload resumption by retrieving the current offset ({{offset-retrieving}}). @@ -654,56 +654,6 @@ Content-Type: application/json } ~~~ -C) The following example shows a client attempting to resume an upload with an incorrect offset. The server rejects the request with a `409 (Conflict)` status code and includes the correct offset in the `Upload-Offset` header field. The client then retries the request with the correct offset. - -~~~ http-message -PATCH /upload/e422fcae4 HTTP/1.1 -Host: example.com -Upload-Complete: ?1 -Upload-Offset: 1230000001 -Content-Length: 4567890 -Content-Type: application/partial-upload - -[content (4567890 bytes)] -~~~ - -~~~ http-message -HTTP/1.1 409 Conflict -Upload-Offset: 1230000000 -Content-Type: application/problem+json - -{ - "type":"https://iana.org/assignments/http-problem-types#\ - mismatching-upload-offset", - "title": "offset from request does not match offset of resource", - "expected-offset": 1230000000, - "provided-offset": 1230000001 -} -~~~ - -~~~ http-message -PATCH /upload/e422fcae4 HTTP/1.1 -Host: example.com -Upload-Complete: ?1 -Upload-Offset: 1230000000 -Content-Length: 4567891 -Content-Type: application/partial-upload - -[content (4567891 bytes)] -~~~ - -~~~ http-message -HTTP/1.1 200 OK -Upload-Complete: ?1 -Content-Type: application/json - -{ - "metadata": { - [...] - } -} -~~~ - ## Upload Cancellation {#upload-cancellation} ### Client Behavior @@ -739,13 +689,13 @@ Even if the client is well-behaved and doesn't send concurrent requests, network To meet these requirements, a server can use various strategies. Three common approaches are: -1. **Latest Request Wins**: If the server receives a new request while a previous one is in-flight, it abruptly terminates the previous HTTP connection or stream before processing the new request. +1. **Preemptive Cancellation**: If the server receives a new request for an upload resource while a previous request for the same resource is in-flight, it abruptly terminates the previous HTTP connection or stream before processing the new request. -2. **Exclusive Locking**: The server processes requests sequentially, effectively creating a lock. A new request is only processed after the previous one completes. This can be simpler to implement but may lead to delays if a request hangs from the server's perspective. +2. **Pessimistic Locking**: The server processes requests for a given upload resource sequentially, effectively creating an exclusive lock on that resource. A new request is only processed after the previous one completes. This can be simpler to implement but may lead to delays if a request hangs from the server's perspective. 3. **Optimistic Concurrency Control**: By maintaining the upload's state in a strongly consistent datastore, the server can atomically check if the `Upload-Offset` in a request matches the resource's current state. If they do not match, the server can reject the request with a `409 (Conflict)` status code. -Servers SHOULD choose a strategy that best fits their architecture while fulfilling the requirements of this section. Regardless of the chosen strategy, clients MUST be prepared to handle a `409 (Conflict)` response as a recoverable error, as described in {{upload-appending}}. +Servers SHOULD choose a strategy that best fits their architecture while fulfilling the requirements of this section. Regardless of the chosen strategy, clients SHOULD be prepared to handle a `409 (Conflict)` response as a recoverable error, as described in {{upload-appending}}. # Status Code `104 (Upload Resumption Supported)` {#status-code-104} From 7946da4f74fff81216df0441903db3a61698096b Mon Sep 17 00:00:00 2001 From: Daniel Resnick Date: Thu, 30 Oct 2025 10:34:18 +1300 Subject: [PATCH 4/5] Updated changelog to reflect latest revision. --- draft-ietf-httpbis-resumable-upload.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft-ietf-httpbis-resumable-upload.md b/draft-ietf-httpbis-resumable-upload.md index d01022a7ff..4942dc1ad3 100644 --- a/draft-ietf-httpbis-resumable-upload.md +++ b/draft-ietf-httpbis-resumable-upload.md @@ -999,8 +999,8 @@ Reference: ## Since draft-ietf-httpbis-resumable-upload-10 {:numbered="false"} -* Clarify client and server handling of `409 (Conflict)` responses, requiring clients to treat them as recoverable and adding an example. -* Refine the "Concurrency" section to outline different server strategies (Latest Request Wins, Exclusive Locking, Optimistic Concurrency Control). +* Clarify client and server handling of `409 (Conflict)` responses, strongly recommending clients to treat them as recoverable. +* Refine the "Concurrency" section to outline different server strategies. * Strengthen the recommendation for retrying `5xx` errors and connection issues to `SHOULD`. * Clarify that a `HEAD` request in the "Offset Retrieval" section can serve as a hint for servers to abort prior in-flight requests. From bb14f212e9396ab5cc239ac7ae9f0b0def83c57b Mon Sep 17 00:00:00 2001 From: Daniel Resnick Date: Wed, 5 Nov 2025 09:56:29 +1300 Subject: [PATCH 5/5] Update concurrency section & offset retrieval based on feedback. --- draft-ietf-httpbis-resumable-upload.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/draft-ietf-httpbis-resumable-upload.md b/draft-ietf-httpbis-resumable-upload.md index 4942dc1ad3..2339f3bb8e 100644 --- a/draft-ietf-httpbis-resumable-upload.md +++ b/draft-ietf-httpbis-resumable-upload.md @@ -515,9 +515,7 @@ If the client received a response with a ### Server Behavior {#offset-retrieving-server} -Upon receiving a `HEAD` request, a server MAY treat it as a hint to abort a previous, possibly stalled, upload creation or append request for the same resource. See {{concurrency}} for more details on handling concurrency. - -A successful response to a `HEAD` request against an upload resource: +A successful response to a `HEAD` request against an upload resource - MUST include the offset in the `Upload-Offset` header field ({{upload-offset}}), - MUST include the completeness state in the `Upload-Complete` header field ({{upload-complete}}), @@ -525,6 +523,8 @@ A successful response to a `HEAD` request against an upload resource: - MUST indicate the limits in the `Upload-Limit` header field ({{upload-limit}}), and - SHOULD include the `Cache-Control` header field with the value `no-store` to prevent HTTP caching ({{CACHING}}). +Upon receiving a `HEAD` request, a server MAY treat it as a hint to abort a previous, possibly stalled, upload creation or append request for the same resource. See {{concurrency}} for more details on handling concurrency. + The resource SHOULD NOT generate a response with the `301 (Moved Permanently)` and `302 (Found)` status codes because clients might follow the redirect without preserving the `HEAD` method. ### Examples {#offset-retrieving-example} @@ -693,7 +693,7 @@ To meet these requirements, a server can use various strategies. Three common ap 2. **Pessimistic Locking**: The server processes requests for a given upload resource sequentially, effectively creating an exclusive lock on that resource. A new request is only processed after the previous one completes. This can be simpler to implement but may lead to delays if a request hangs from the server's perspective. -3. **Optimistic Concurrency Control**: By maintaining the upload's state in a strongly consistent datastore, the server can atomically check if the `Upload-Offset` in a request matches the resource's current state. If they do not match, the server can reject the request with a `409 (Conflict)` status code. +3. **Optimistic Concurrency Control**: The server detects concurrent modifications by atomically checking the resource's state before applying changes. If a conflict is detected, the server rejects the request with a 409 (Conflict) status code, ensuring that only one of multiple parallel requests can succeed. Servers SHOULD choose a strategy that best fits their architecture while fulfilling the requirements of this section. Regardless of the chosen strategy, clients SHOULD be prepared to handle a `409 (Conflict)` response as a recoverable error, as described in {{upload-appending}}.