Skip to content

Conversation

@wild0ne
Copy link
Contributor

@wild0ne wild0ne commented Jan 30, 2026

Ratio and RatioSD summary values are fixed for baseline (1.0 and 0.0, respectively)

@wild0ne

This comment was marked as off-topic.

@wild0ne
Copy link
Contributor Author

wild0ne commented Feb 7, 2026

@AndreyAkinshin Hey Andrey! It'd be great to have your thoughts on this PR. Thanks!

Copy link
Member

@AndreyAkinshin AndreyAkinshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR addresses a real user-facing problem — the baseline ratio deviating from 1.00 due to the all-pairs division approach — and doesn't introduce any regressions for non-baseline ratios. The fix is scoped and safe, so let's merge it.

Status update on the broader problem:

I'm working on a new statistical engine called Pragmastat. The latest version already includes Ratio and RatioBounds functions designed to replace the current BenchmarkDotNet ratio implementation. That implementation is quite flawed — when I originally wrote it, I wasn't aware of better solutions and picked an approach that felt reasonable at the time. After several years of research, I've arrived at a much better way to estimate ratios between benchmarks, which you can explore in the current Pragmastat.

The underlying research work is finished. What remains is implementing a few additional estimators and reworking the summary tables before it can be merged into BenchmarkDotNet. After that, we can completely drop the current ratio and statistics implementation, since it has multiple flaws and can be misleading in certain cases.

No ETA, but I hope to finish in the upcoming months. I've given similar estimates before, but this time I'm powered by an LLM-based coding agent, so progress has gotten a huge acceleration boost.

Bottom line: this PR is good to merge as-is. Further polishing of the current ratio logic isn't worth the investment — it will all be replaced by the Pragmastat-based engine.

@timcassell timcassell merged commit 84fcd87 into dotnet:master Feb 9, 2026
20 of 21 checks passed
@timcassell timcassell linked an issue Feb 9, 2026 that may be closed by this pull request
@timcassell timcassell added this to the v0.16.0 milestone Feb 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ratio value is not 1.00 for baseline

3 participants