feature partity: fsspec vs pyarrow

[kevinjqliu]

Feature Request / Improvement

During the community call, we were talking about the feature partity for fsspec and pyarrow. Specifically the lack of boundary/abstraction between FileIO/FileSystem and read/write features.
I think a lot of features are only implemented in pyarrow as of now.

Would be great to document the feature gap and implement those in fsspec (somehow)

Starting this issue to get the conversation started!

[jayceslesar]

Nice, yeah this is IMO blocking #1958. I can also take a stab at documenting this discrepancy in supported features

[jayceslesar]

What if we just made our own file system class on top of opendal?

[Fokko]

Can we elaborate on this? FileIO is designed to be the right abstraction, most of the listing operations on object-stores are performing very poorly because of the paged respones. Therefore the list-operation is left out intentionally since every non-filesystem (Iceberg is designed for object stores), requires different optimizations.

[jayceslesar]

nevermind this is not a blocker, see latest commit #1958