Skip to content

feat(python): refactor cython buffer to use c++ buffer #3219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
chaokunyang merged 4 commits into from
Jan 27, 2026
Merged

feat(python): refactor cython buffer to use c++ buffer #3219

chaokunyang merged 4 commits into from
Jan 27, 2026

Conversation

@chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented Jan 27, 2026 •
edited
Loading

Why?

  • Python Cython buffer implementation duplicated C++ buffer logic and error handling.
  • Centralizing buffer reads/writes in the C++ buffer reduces duplication and keeps behavior aligned.

What does this PR do?

  • Refactors python/pyfory/buffer.pyx to delegate reads/writes, varint/tagged encoding, and bounds checks to fory::Buffer.
  • Adds C++ Buffer int24 helpers and exposes missing buffer APIs/errors to Cython via libutil.pxd.
  • Introduces Python error mapping helpers and updates row/collection code to pass C++ buffer pointers correctly.

Related issues

Closes #3218

#3216
#1017
#3220

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

@chaokunyang chaokunyang self-assigned this Jan 27, 2026
@chaokunyang chaokunyang mentioned this pull request Jan 27, 2026
Open
17 tasks
@chaokunyang chaokunyang force-pushed the refactor_python_buffer branch from 5441b50 to 7d7cf71 Compare January 27, 2026 05:28
@chaokunyang chaokunyang merged commit 9b8c1ea into apache:main Jan 27, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@urlyy urlyy urlyy approved these changes

@pandalee99 pandalee99 Awaiting requested review from pandalee99

@PragmaTwice PragmaTwice Awaiting requested review from PragmaTwice PragmaTwice is a code owner

@theweipeng theweipeng Awaiting requested review from theweipeng

@LiangliangSui LiangliangSui Awaiting requested review from LiangliangSui

Assignees

@chaokunyang chaokunyang

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Python] refactor cython Buffer to wrap encoding to c++ Buffer

2 participants

@chaokunyang @urlyy