Skip to content

Conversation

@tompng
Copy link
Member

@tompng tompng commented Jan 29, 2026

BigDecimal's code was mixing up BigDecimal.double_fig(must be exactly 16) and internal extra calculation prec(could be any number around 10).
This change will clearly split this two essentially different constant usage.

Examples

This example should use double_fig because Float has double_fig precision. It must be 16.

initial_y = BigDecimal(Math.cbrt(x.to_f))
initial_prec = BigDecimal.double_fig * safe_factor
do_newton_method_iteration while prec < required_precision

This example should use EXTRA_PREC, a safe factor to reduce division/rounding error.
The value is completely unrelated to Float's internal precision. It can be reduced to 12, 10, 8 or something else in the future.

tan = sin(x, prec + EXTRA_PREC).div(cos(x, prec + EXTRA_PREC), prec)

BigDecimal's code was mixing up BigDecimal.double_fig(exactly 16) and internal extra calculation prec(Could be any number around 10).
This change will clearly split this two essentially different constant usage.
@tompng tompng merged commit 57cdef6 into ruby:master Jan 30, 2026
85 of 87 checks passed
@tompng tompng deleted the extraprec_const branch January 30, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant