Will arrow-java support overall compression at the FieldVector level in the future, rather than separately compressing each ArrowBuffer within the FieldVector?

[swuferhong]

Describe the enhancement requested

Currently, to do compression for one ArrowRecordBatch, in VectorUnloader, it will separately compress each ArrowBuffer within the FieldVector instead of compress at the FieldVector level. From the compression rate perspective, larger batches generally result in higher compression rates. Additionally, calling compress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) multiple times may consume more CPU than call once.

Therefore, I would like to ask if there will be support for overall compression at the FieldVector level, which could improve the compression ratio without affecting the ability to read individual columns.

[lidavidm]

This is a question for the format level (buffer-level compression is part of the Arrow spec), so I'd encourage you to ask on the mailing list instead.

[swuferhong]

This is a question for the format level (buffer-level compression is part of the Arrow spec), so I'd encourage you to ask on the mailing list instead.

Thanks @lidavidm, I'll go try sending an email.