GH-46315 [C#] Apache Arrow Flight Middleware #46316

HackPoint · 2025-05-05T09:44:43Z

Enhancement: Apache Arrow Flight Middleware in C#

Overview

This Pull Request enhances middleware support for Apache Arrow Flight using C#, focusing on improved metadata header management and propagation for better observability and extensibility. It also provides handling for HTTP/HTTPS communication.

Rationale for this Change

Effective middleware is critical for managing metadata headers, ensuring accurate request/response handling, and simplifying debugging in distributed systems. By improving middleware capabilities, we enhance reliability and observability, significantly benefiting developers and operational teams managing complex Flight-based applications.

What's Included in this PR?

Middleware enhancements supporting complete metadata header propagation.
Middleware lifecycle hooks for better request/response management.
Comprehensive integration tests validating middleware functionality.
Documentation updates reflecting middleware improvements.

Key Features

Complete Header Propagation: Ensures accurate propagation of gRPC metadata headers throughout middleware lifecycle events.
HTTP/HTTPS Handling: Supports middleware integration and metadata propagation for HTTP and HTTPS communications.
Middleware Lifecycle Management: Supports reliable middleware hooks (OnBeforeSendingHeaders, OnHeadersReceived, OnCallCompleted).
Enhanced Testing: Adds comprehensive integration tests to verify correct middleware behavior.

Impact

Improves middleware reliability and simplifies debugging.
Enhances transparency in gRPC and HTTP/S communication within Flight-based applications.

Are These Changes Tested?

Testing Overview

Unit Tests:

Added tests for middleware lifecycle event execution (e.g., OnBeforeSendingHeaders, OnHeadersReceived, OnCallCompleted).
Verified internal logic for capturing and storing gRPC metadata headers.

Integration Tests:

Tested end-to-end with a real Flight client and in-memory server setup.
Validated propagation of custom headers (e.g., x-server-header, Set-Cookie) between client and server.

End-to-End Tests:

Simulated real-world Flight requests to ensure headers are processed consistently across middleware layers.
Confirmed correct invocation order and middleware behavior under different server responses.

Example Test Cases:

Verify that OnHeadersReceived correctly captures server-sent headers.
Ensure custom client middleware modifies request headers as expected.
Validate that OnCallCompleted is triggered on both success and error cases.

Checklist

Implementation completed
Tests added and passing

GitHub Issue: [C#] Apache Arrow Flight Middleware #46315

…ing headers and cookie values - Implemented tests verifying that CookieMiddleware correctly captures and persists values from both headers and cookies. - Added CapturingCookieMiddleware and CapturingCookieMiddlewareFactory for validating middleware lifecycle. - Ensured middleware correctly processes cookie strings and validates persistence throughout the request lifecycle.

…s it will add up and force more GC calls)

github-actions · 2025-05-05T09:45:11Z

⚠️ GitHub issue #46315 has been automatically assigned in GitHub to PR creator.

…nto feat/flight-csharp-middleware # Conflicts: # csharp/Directory.Build.props # csharp/src/Apache.Arrow.Flight/Middleware/ClientCookieMiddleware.cs # csharp/src/Apache.Arrow.Flight/Middleware/ClientCookieMiddlewareFactory.cs # csharp/src/Apache.Arrow.Flight/Middleware/CookieManager.cs # csharp/src/Apache.Arrow.Flight/Middleware/Extensions/FlightMethodParser.cs # csharp/src/Apache.Arrow.Flight/Middleware/Extensions/StatusUtils.cs # csharp/src/Apache.Arrow.Flight/Middleware/Interceptors/ClientInterceptorAdapter.cs # csharp/test/Apache.Arrow.Flight.Tests/MiddlewareTests/ClientCookieMiddlewareTests.cs # csharp/test/Apache.Arrow.Flight.Tests/MiddlewareTests/Stubs/ClientCookieMiddlewareMock.cs

…r for cookie middleware - Remove unused and redundant files

CurtHagenlocher · 2025-07-22T23:47:54Z

@HackPoint can you please rebase this change now that the other change is checked-in?

CurtHagenlocher · 2025-07-24T12:27:14Z

@HackPoint can you please rebase this change now that the other change is checked-in?

Alternatively, is it okay if we fork the C# implementation to a separate repo now? Rebasing this change might be complicated enough that we get no real benefit from continuing to use the same repo and this change is the last "pending" one before the proposed fork.

HackPoint · 2025-07-24T12:29:45Z

@HackPoint can you please rebase this change now that the other change is checked-in?

Alternatively, is it okay if we fork the C# implementation to a separate repo now? Rebasing this change might be complicated enough that we get no real benefit from continuing to use the same repo and this change is the last "pending" one before the proposed fork.

Than what's your preferred thing to do? Should I continue with merge and rebase or not?

CurtHagenlocher · 2025-07-24T12:37:35Z

Than what's your preferred thing to do? Should I continue with merge and rebase or not?

Merge and rebase is fine; I wasn't sure if you were working on this.

HackPoint · 2025-07-24T12:40:07Z

Than what's your preferred thing to do? Should I continue with merge and rebase or not?

Merge and rebase is fine; I wasn't sure if you were working on this.

Sir, I was so glad that you merged the pr , I started the moment you have mentioned this rebase. I almost done and started testing it against our db. The moment I will finish, I'll push the update.

…ddleware

…ddleware # Conflicts: # csharp/src/Apache.Arrow.Flight.Sql/Client/FlightSqlClient.cs # csharp/src/Apache.Arrow.Flight.Sql/DoPutResult.cs # csharp/src/Apache.Arrow.Flight.Sql/PreparedStatement.cs # csharp/src/Apache.Arrow.Flight.Sql/SqlActions.cs # csharp/src/Apache.Arrow.Flight.Sql/Transaction.cs # csharp/test/Apache.Arrow.Flight.Sql.Tests/FlightSqlClientTests.cs

HackPoint · 2025-07-31T19:27:50Z

@CurtHagenlocher Hi sir, it's completed and all the integration has been done and tested. Please merge it it should be working fine.

HackPoint · 2025-08-13T21:24:36Z

Than what's your preferred thing to do? Should I continue with merge and rebase or not?

Merge and rebase is fine; I wasn't sure if you were working on this.
@CurtHagenlocher Hi, can you please review the PR? Seems like it's missing functionality for our needs.
UPD:
@jeremyosterhoudt

CurtHagenlocher

Thanks! I left some feedback and some questions.

csharp/src/Apache.Arrow.Flight/Middleware/MetadataAdapter.cs

csharp/src/Apache.Arrow.Flight/Middleware/ClientCookieMiddlewareFactory.cs

csharp/src/Apache.Arrow.Flight/Middleware/ClientCookieMiddleware.cs

csharp/src/Apache.Arrow.Flight/Middleware/ClientCookieMiddlewareFactory.cs

csharp/src/Apache.Arrow.Flight/Middleware/Interceptors/ClientInterceptorAdapter.cs

…iddleware handling - Switched to file-scoped namespace style for consistency - Ensured exceptions from ResponseHeadersAsync are explicitly propagated - Lifted CallHeaders allocation out of loop to avoid redundant allocations - Captured getStatus() and getTrailers() once per call instead of per-middleware - Fixed double invocation of OnCallCompleted when a middleware throws - Extracted NotifyCompletionOnce as a private helper to keep HandleResponse DRY - Preserved behavior while ensuring OnCallCompleted is invoked at most once

raulcd · 2025-10-16T07:45:46Z

The dotnet implementation was moved to https://github.com/apache/arrow-dotnet
This should be moved there. Should we close the PR for this repository?

CurtHagenlocher · 2025-10-20T14:58:35Z

Superceded by apache/arrow-dotnet#139.

HackPoint · 2025-10-20T15:06:10Z

Superceded by apache/arrow-dotnet#139.

Hi Curt, what does it means? Did you merged the Middleware into the c# code ?

CurtHagenlocher · 2025-10-20T15:11:02Z

Hi Curt, what does it means? Did you merged the Middleware into the c# code ?

Not yet, I ported your PR to the new repository for the C# implementation. I want to make sure the tests are still passing, and it's been long enough that I'll take another look at it before committing.

HackPoint · 2025-10-20T15:49:07Z

Hi Curt, what does it means? Did you merged the Middleware into the c# code ?

Not yet, I ported your PR to the new repository for the C# implementation. I want to make sure the tests are still passing, and it's been long enough that I'll take another look at it before committing.

Sure, if you need me to fix something or help just tell me. And can you please say, why I'm unable to see myself as contributor?

CurtHagenlocher · 2025-10-20T16:12:14Z

I tried to change the commit message a few different ways but nothing I did made you show as a contributor. Feel free to submit a new PR to this repo if you like and then I'll close mine. (If you do this, consider starting with what's in my branch as I've fixed a bunch of lint issues.)

HackPoint · 2025-10-20T16:15:24Z

I tried to change the commit message a few different ways but nothing I did made you show as a contributor. Feel free to submit a new PR to this repo if you like and then I'll close mine. (If you do this, consider starting with what's in my branch as I've fixed a bunch of lint issues.)

I will fork your PR and re-push it. Starting immediately on it.

HackPoint and others added 7 commits April 15, 2025 19:11

feat: headers + cookie

26fa098

feat: DoPut

a14a845

feat: Middleware implementation

81ad726

fix: remove redundant memory allocation (minor, but for a lot of call…

da5a7e6

…s it will add up and force more GC calls)

chore: explicitly return Array.Empty<Cookie>

6f41d63

Merge branch 'apache:main' into feat/flight-csharp-middleware

ca8cc4f

HackPoint requested a review from CurtHagenlocher as a code owner May 5, 2025 09:44

github-actions bot added Component: C# awaiting review Awaiting review labels May 5, 2025

HackPoint and others added 4 commits May 8, 2025 15:00

Merge branch 'apache:main' into feat/flight-csharp-middleware

9cd47eb

fix(Middleware): re-implemented case in-sensitive and response handle…

3983768

…r for cookie middleware - Remove unused and redundant files

Merge branch 'main' into feat/flight-csharp-middleware

7dd907b

HackPoint added 3 commits July 28, 2025 16:05

Merge remote-tracking branch 'origin/main' into feat/flight-csharp-mi…

e8e251d

…ddleware

fix: failing test

18f79de

chore: adding apache license notes

440fbf7

CurtHagenlocher requested changes Aug 14, 2025

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 14, 2025

Merge branch 'apache:main' into feat/flight-csharp-middleware

67360a1

CurtHagenlocher mentioned this pull request Oct 20, 2025

feat: Implement Arrow Flight Middleware support apache/arrow-dotnet#139

Merged

2 tasks

CurtHagenlocher closed this Oct 20, 2025

GH-46315 [C#] Apache Arrow Flight Middleware #46316

GH-46315 [C#] Apache Arrow Flight Middleware #46316

Uh oh!

Conversation

HackPoint commented May 5, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Enhancement: Apache Arrow Flight Middleware in C#

Overview

Rationale for this Change

What's Included in this PR?

Key Features

Impact

Are These Changes Tested?

Checklist

Uh oh!

github-actions bot commented May 5, 2025

Uh oh!

CurtHagenlocher commented Jul 22, 2025

Uh oh!

CurtHagenlocher commented Jul 24, 2025

Uh oh!

HackPoint commented Jul 24, 2025

Uh oh!

CurtHagenlocher commented Jul 24, 2025

Uh oh!

HackPoint commented Jul 24, 2025

Uh oh!

HackPoint commented Jul 31, 2025

Uh oh!

HackPoint commented Aug 13, 2025

Uh oh!

CurtHagenlocher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Oct 16, 2025

Uh oh!

CurtHagenlocher commented Oct 20, 2025

Uh oh!

HackPoint commented Oct 20, 2025

Uh oh!

CurtHagenlocher commented Oct 20, 2025

Uh oh!

HackPoint commented Oct 20, 2025

Uh oh!

CurtHagenlocher commented Oct 20, 2025

Uh oh!

HackPoint commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HackPoint commented May 5, 2025 •

edited by github-actions bot

Loading