Closed
Conversation
Two files changed: linopy/expressions.py: - Added LazyLinearExpression class (inherits LinearExpression) that stores a list of un-merged Dataset _parts instead of a single concatenated Dataset - Key overrides: data (lazy materialization), const, flat (iterates parts directly), __add__/__sub__/__neg__ (propagate laziness), all with fallback to parent when not lazy - Modified merge() to return LazyLinearExpression when dim == TERM_DIM and cls is a LinearExpression subclass - Protected lazy expressions from premature materialization in merge's data extraction and override detection linopy/objective.py: - Changed is_linear/is_quadratic from type(x) is LinearExpression identity checks to isinstance checks, so LazyLinearExpression is correctly identified as linear Performance (different-coordinate variables, 5 × 200×200×50): ┌───────────────────────┬─────────┬─────────┬───────────────┐ │ Metric │ Before │ After │ Change │ ├───────────────────────┼─────────┼─────────┼───────────────┤ │ Expression build time │ 0.31s │ 0.09s │ 3.4x faster │ ├───────────────────────┼─────────┼─────────┼───────────────┤ │ flat export time │ 0.57s │ 0.17s │ 3.4x faster │ ├───────────────────────┼─────────┼─────────┼───────────────┤ │ Peak RSS at flat │ 1337 MB │ 1186 MB │ -151 MB (11%) │ └───────────────────────┴─────────┴─────────┴───────────────┘ Same-coordinate variables see no regression — materialization occurs at to_constraint time with the same override join as before. Phase 2 (lazy constraints) would extend savings to that path.
1. _compact() — Groups parts by their coordinate signature and merges same-coord groups using join="override" (no padding). Keeps part count low after many chained additions. 2. to_polars() — Lazy override that converts each part to a polars DataFrame independently and concatenates, same pattern as flat. 3. rename() — Per-part dispatch that only renames dims/vars present in each part, avoiding errors for parts that lack the target dimension. 4. diff() — Per-part dispatch that applies diff only to parts containing the target dimension. 5. sel() and shift() — Fall back to materialized path since their semantics (indexing, fill values) need a consistent coordinate space. Still deferred (Phase 2+): - Lazy constraint propagation (to_constraint currently materializes) - Lazy _sum() for dimension reduction - These would require changes to the Constraint class and solver IO paths
Contributor
Author
|
For now continued in FBumann#12 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes # (if applicable).
Changes proposed in this Pull Request
Checklist
doc.doc/release_notes.rstof the upcoming release is included.