perf: speed up LP file writing (2.5-3.9x on large models, no regressions on small)#564
Open
FBumann wants to merge 16 commits intoPyPSA:masterfrom
Open
perf: speed up LP file writing (2.5-3.9x on large models, no regressions on small)#564FBumann wants to merge 16 commits intoPyPSA:masterfrom
FBumann wants to merge 16 commits intoPyPSA:masterfrom
Conversation
Extract _format_and_write() helper that uses lazy().collect(engine="streaming") with automatic fallback, replacing 7 instances of df.select(concat_str(...)).write_csv(...).
Replace the vertical concat + sort approach in Constraint.to_polars() with an inner join, so every row has all columns populated. This removes the need for the group_by validation step in constraints_to_file() and simplifies the formatting expressions by eliminating null checks on coeffs/vars columns.
…r short DataFrame - Skip group_terms_polars when _term dim size is 1 (no duplicate vars) - Build the short DataFrame (labels, rhs, sign) directly with numpy instead of going through xarray.broadcast + to_polars - Add sign column via pl.lit when uniform (common case), avoiding costly numpy string array → polars conversion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…e vars Check n_unique before running the expensive group_by+sum. When all variable references are unique (common case for objectives), this saves ~31ms per 320k terms. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace np.unique with faster numpy equality check for sign uniformity. Eliminate redundant filter_nulls_polars and check_has_nulls_polars on the short DataFrame by applying the labels mask directly during construction. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Guard against IndexError when sign_flat is empty (no valid labels) by checking len(sign_flat) > 0 before accessing sign_flat[0]. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…for duplicate (labels, vars) pairs before calling group_terms_polars. Use it in both Constraint.to_polars() and LinearExpression.to_polars() to avoid expensive group_by when terms already reference distinct variables
FBumann
changed the title
FBumann
changed the title
Collaborator
|
Wonderful @FBumann ! This is very much welcome! |
Contributor
Author
|
@FabianHofmann Should a fix the codecov stuff? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this Pull Request
Speed up LP file writing by up to 3.9x on large models, with consistent improvements across all problem sizes. Includes a benchmark script for reproducibility.
Added a lp-file benchmarking script, which is a majority of the lines changed
Performance optimizations
concat_str+write_csvvia new_format_and_write()helper (with automatic fallback + warning)concat+sortwithjoinfor constraint assemblymaybe_group_terms_polars()to skip expensivegroup_bywhen terms already reference distinct variablesBug fixes
IndexErroron empty constraint slices insign_flatcheckBenchmark results
Reproduce with
python dev-scripts/benchmark_lp_writer.py --model basic -o results.json --label "my run".Synthetic model (2×N² vars, 2×N² constraints)
No regressions on small models, speedup grows with problem size up to 3.9x at 8M variables.
PyPSA SciGrid-DE (realistic power system model, 24–1000 snapshots)
Consistent 2.5–2.7x speedup across all sizes, reaching 7.0s → 2.7s at 2.5M variables / 6M constraints.
Checklist
doc.doc/release_notes.rstof the upcoming release is included.