[ADBC][FlightSQL][Java] 'Encountered end-of-stream mid-frame' upgrading from Arrow 17 -> 18

[jarohen]

Describe the bug, including details regarding any error messages, version, and platform.

Hey folks - I seem to have a regression upgrading to Arrow 18 in an E2E ADBC (via FlightSQL) test:

AdbcException{message=Encountered end-of-stream mid-frame, status=INTERNAL, sqlState='null', vendorCode=0, cause=null, details=0}
	at org.apache.arrow.adbc.driver.flightsql.FlightSqlDriverUtil.fromFlightException(FlightSqlDriverUtil.java:109)
	at org.apache.arrow.adbc.driver.flightsql.FlightSqlStatement.execute(FlightSqlStatement.java:210)
	at org.apache.arrow.adbc.driver.flightsql.FlightSqlStatement.executeFlightInfo(FlightSqlStatement.java:221)
	at org.apache.arrow.adbc.driver.flightsql.FlightSqlStatement.executeQuery(FlightSqlStatement.java:248)
	at xtdb.AdbcTest.test simple round-trip(AdbcTest.kt:69)

(unfortunately this is the only stack trace I'm getting atm)

• Regressing test
• Deps before: Java 21+, Arrow 17.0.0, ADBC 0.14.0; after: Arrow 18.0.0, otherwise unchanged.
• Our FlightSqlProducer implementation
• Our FlightSqlClient tests (unaffected by the upgrade)

Our implementation of the FlightSQL server is pretty bare-bones right now so entirely possible it's something we're doing wrong - raising here just in case anyone else is running into it (feel free to close if not!)

Cheers,

James

(our issue: xtdb/xtdb#3833)

Component(s)

FlightRPC, Java, Other

[lidavidm]

I missed this, sorry. This is only happening via ADBC, you say, not via the regular FlightSqlClient?

[lidavidm]

Also, what version of grpc-netty do you end up pulling in? grpc/grpc-java#11284

[jarohen]

aha, 1.60.0 atm. will see what's pulling that in

[jarohen]

upgrading to 18.1.0, ADBC 0.15.0 (and hence grpc-netty 1.65.0) gives me a different error, at least 😅

Through FlightSqlClient:

1. Unhandled java.lang.IndexOutOfBoundsException
   (No message)

               Buffer.java:  757  java.nio.Buffer$1/apply
               Buffer.java:  754  java.nio.Buffer$1/apply
        Preconditions.java:  213  jdk.internal.util.Preconditions$4/apply
        Preconditions.java:  210  jdk.internal.util.Preconditions$4/apply
        Preconditions.java:   98  jdk.internal.util.Preconditions/outOfBounds
        Preconditions.java:  106  jdk.internal.util.Preconditions/outOfBoundsCheckIndex
        Preconditions.java:  302  jdk.internal.util.Preconditions/checkIndex
               Buffer.java:  779  java.nio.Buffer/checkIndex
       HeapByteBuffer.java:  502  java.nio.HeapByteBuffer/getInt
                Table.java:  301  com.google.flatbuffers.Table/__reset
              Message.java:   41  org.apache.arrow.flatbuf.Message/__init
              Message.java:   42  org.apache.arrow.flatbuf.Message/__assign
              Message.java:   40  org.apache.arrow.flatbuf.Message/getRootAsMessage
              Message.java:   39  org.apache.arrow.flatbuf.Message/getRootAsMessage
    MessageSerializer.java:  727  org.apache.arrow.vector.ipc.message.MessageSerializer/readMessage
    MessageSerializer.java:  205  org.apache.arrow.vector.ipc.message.MessageSerializer/deserializeSchema
      FlightSqlClient.java: 1306  org.apache.arrow.flight.sql.FlightSqlClient$PreparedStatement/deserializeSchema
      FlightSqlClient.java: 1282  org.apache.arrow.flight.sql.FlightSqlClient$PreparedStatement/getParameterSchema
      FlightSqlClient.java: 1334  org.apache.arrow.flight.sql.FlightSqlClient$PreparedStatement/execute
       flight_sql_test.clj:  135  xtdb.flight-sql-test/fn/fn

Currently wondering whether this is the 0.14.0 -> 0.15.0 upgrade, if there was a breaking change there... 👀

I'll close this one for now and keep digging, given I'm not seeing the original error - but if you happen to know off-hand what might be causing this other one I'm all ears 😆

[jarohen]

Sorted - needed to migrate Schema/.toByteArray to Schema/.serializeAsMessage

[lidavidm]

Ah! Thanks for reporting back