Skip to content

Conversation

@doorgan
Copy link
Contributor

@doorgan doorgan commented Jan 31, 2026

Code like %{a do :ok end | b c, d => e} is ambiguous because it can be interpreted as:

%{(... | b(c, d)) => e} - what Spitfires parsed
%{... | b(c, d) => e} - what Elixir chooses

Elixir is aware of this ambiguity and warns about it:

warning: missing parentheses on expression following operator "|", you must add parentheses to avoid ambiguities

Since this is ambiguous syntax, Spitfire it not entirely wrong here, but we should follow what Elixir produces anyways.

The only way I got this to trigger(from examples in the property tests in Lukasz prs) is with the lhs of | being a do/end block, I'm not 100% sure why, but withut it the Elixir parser straight out returns a syntax error.

We were allowing => to be consumed inside the no-parens call arguments and the map entry parser never got a chance to see it. The fix marks map entry contexts (map/struct literals and map updates) with map_context and blocks => from being consumed inside nested expressions. That keeps => at the entry level and allows the existing ambiguity handling in parse_pipe_op/2 to match Elixir’s choice. last_assoc_meta context is used to reconstruct the ast after resolving the ambiguity, and put the assoc metadata in the right node.

I added a new with_state/4 macro to make it easier to manage parser context. Setting and remember to unset parser context is very error prone, so this should help.

@doorgan doorgan marked this pull request as draft January 31, 2026 04:08
@doorgan doorgan force-pushed the doorgan/ambiguous-map-update branch 2 times, most recently from 448f4dc to 566b271 Compare January 31, 2026 07:50
@doorgan doorgan marked this pull request as ready for review January 31, 2026 08:02
@doorgan doorgan force-pushed the doorgan/ambiguous-map-update branch from 566b271 to 3344e33 Compare January 31, 2026 16:54
#
# This section rewrites the parse attempt to match Elixir's behavior.
case pairs do
[{{call, call_meta, [_, _ | _] = args}, value}] ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment that has an example of code that matches each clause? I'm getting confused cuz this says pairs but it's a list of one element.

Copy link
Contributor Author

@doorgan doorgan Jan 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. It was called pairs because it's the key/value pairs for a => b, c => d. It may also be a single element(the second clause) so I renamed this to entries instead.
I added a comment with the shape each clause handles.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I think I was getting confused at to what the ast for this is. The args are the pairs and the "call" is the pipe?

Copy link
Contributor Author

@doorgan doorgan Feb 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this pattern [{{call, call_meta, [_, _ | _] = args}, value}] matches this interpretation: %{... | (b(c, d)) => e}
Where:

  • call is b(c, d)
  • The ambiguity happens with a 2+ args call(as the comma separating the args causes the ambiguity when the source code has no parens), so [_, _ | _] = args is checking for c, d, ... in that call
  • value is e
  • There is no :"=>" ast node, this pair represents that

The pipe parsing happens in parse_expression when the flag is_map is true, then parse_pipe_op -> parse_map_update_pairs -> parse_comma_list. The last function there is what produces a list of entries for the map, and from that list is where this "pair" comes from if it finds an assoc_op(=>) token.

Then, in parse_pipe_op(which is only called for map updates, not list cons like [head | tail], we call this function that sees the ambiguity and resolves it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going over this again I realize if there's more entries like %{a do :ok end | b c, d => e, f => g} then there's still a mismatch, so I'll have to fix that too

Use the `:capture` option to return a map of selected keys as they were
at the end of the block, before state is restored.
"""
defmacro with_state(parser, updates, opts \\ [], do: block) do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you write a few unit tests for this?

Code like `%{a do :ok end | b c, d => e}` is ambiguous because it can be
interpreted as:

`%{(... | b(c, d)) => e}` - what Spitfires parsed
`%{... | b(c, d) => e}` - what Elixir chooses

Elixir is aware of this ambiguity and warns about it:
```
warning: missing parentheses on expression following operator "|", you must add parentheses to avoid ambiguities
```

Since this is ambiguous syntax, Spitfire it not entirely wrong here, but
we should follow what Elixir produces anyways.

The only way I got this to trigger(from examples in the property tests
in Lukasz prs) is with the lhs of `|` being a do/end block, I'm not 100%
sure why, but withut it the Elixir parser straight out returns a syntax error.

We were allowing `=>` to be consumed inside the no-parens call arguments
and the map entry parser never got a chance to see it.
The fix marks map entry contexts (map/struct literals and map updates) with
`map_context` and blocks `=>` from being consumed inside nested expressions.
That keeps `=>` at the entry level and allows the existing ambiguity handling
in `parse_pipe_op/2` to match Elixir’s choice.
@doorgan doorgan force-pushed the doorgan/ambiguous-map-update branch from 454ab8e to 725126e Compare February 1, 2026 02:48
@doorgan doorgan marked this pull request as draft February 2, 2026 14:59
@doorgan
Copy link
Contributor Author

doorgan commented Feb 2, 2026

I'm moving this to draft, extending the current fix to handle %{... | a b, c => d, f => g} is too hacky and doing it properly is requiring a larger refactor to explicitly parse matched/unmatched/no_parens expressions.

@doorgan
Copy link
Contributor Author

doorgan commented Feb 5, 2026

Closing in favor of #93, the approach here is not enough and doesn't fix the root cause.

@doorgan doorgan closed this Feb 5, 2026
mhanberg pushed a commit that referenced this pull request Feb 7, 2026
)

Supercedes #87 

I had originally intended to break down this into multiple PRs like #87,
but changing one part of the parser often broke another or didn't fully
fix the issues, so I apologize for the large diff. Tackling the root
cause ended up being a larger change but it fixed a whole family of
parser issues that would have taken 100 PRs applying bandaids. This is
also the reason this PR includes some tests from that PR.

This PR brings the `with_context` helper from that PR to help scope the
parser context to specific parsing contexts.

The main issue this PR tackles is how matched, unmatched and no-parens
expressions are handled.
The elixir grammar distinguishes three kinds of expressions:

- "matched": expressions with clear boundaries, like literals,
parenthesized calls, tuples, lists, maps, identifiers, modules.
The key property of these is that they can safely appear as an operand
without ambiguity.

- "unmatched": expressions with do-end blocks. These are "unmatched"
because their boundary is the end keyword, not a closing delimiter.
You cannot nest an unmatched expression inside another unmatched
expression without parentheses:
`if if true do true else false end do` - invalid
`if(if true do true else false end) do` - valid

Operators can take unmatched expressions as their RHS (e.g., x = if true
do 1 end), which is why `unmatched_op_expr` exists separately from
`matched_op_expr` in elixir_parser.yrl

- "no_parens": multi-argument calls without parentheses like `foo a, b,
c`. These cannot be nested.
No-parens expressions are further subdivided:
  - `no_parens_one`: single unambiguous arg (`f a, f key: val`)
  - `no_parens_many`: multiple args (`f a, b, c`)
- `no_parens_one_ambig`: single arg that is itself a no_parens_many (`f
g a, b -> f(g(a, b))`)

The grammar enforces that operators with a no_parens RHS can only
combine with matched/unmatched LHS — not another no_parens expression.

In Spitfire we don't work with grammar rules but we rather consume
tokens as we go, so to achieve the same we have to resort to flags like
`nesting`, `in_map`, `stab_state` etc on top of precedence checks). This
PR expands that to mimic the grammar rules behavior in elixir. I clarify
this because to explain the parser behavior I need to refer to the
elixir grammar, but when you look at the diff in this PR it's mostly
flags and extra checks, not explicit grammar rules.

Some examples of the issues this fixes:

- No-parens calls tended to consume too many tokens
Given `%{foo a, b => c}`, Elixir parses this as `%{foo(a, b) => c}`
because `foo a, b` is a `no_parend_many` and `=>` is an operator that
expects `matched_expr` on both sides.
Spitfire would parse this as `{foo(a, b =>c)}` because it parsed
no-parens calls at the `@lowest` precedence regardless of context.
The fix uses `assoc_op` precedence when `is_map == true` to force
`foo(a, b)` to complete before continuing parsing the rhs of `=>`

- `->` would be parsed as infix in the wrong contexts
`->` should only be allowed when wrapped around parens, `do`-`end` or
`fn`-`end`, but *never* as a regular infix operator.
Spitfire would treat `->` as an infix operator controlled by `is_stab`
but this flag was not scoped.
In code like `fn a, b -> (c -> d) end`, `is_stab` from the `fn` body
leaked into the grouped `(c -> d)` or viceversa.
Adding `stop_before_stab_op?` and scoping it with `with_context` each
boundary is managed independently

- keyword lists not stopping at stab boundary
There is a grammar rule `call_args_no_parens_kw_expr -> kw_eol
matched_expr`, it basically says the value of a keyword pair in
no-parens must be a matched expression, so in this code:
`fn key: expr -> body end`
`expr` stops at `->` because `->` can't appear in a matched expression

Spitfire would continue consuming tokens after `->` because it wasn't
aware of stabs in that context. This PR adds stab-aware functions to
deal with this scenario. Also, differently to #87, this does so more
generally so this takes care of the `%{a do :ok end | b c, d => e, f =>
g}` scenario that is broken in that PR.

Some other issues that aren't strictly related to
matched/unmatched/no-parens but were still necessary:

- `.` no-parens calls didn't parse args
`foo.bar 1, 2` is a valid no-parens call, but Spitfire parsed it as
`foo.bar()` followed by dangling `1, 2`. Checking `no_parens` metadata
like elixir does in `build_identifier`) fixes that

- `block_identifier` not treated as terminal
In the Elixir grammar, `block_identifier` (`else`, `rescue`, `catch`,
`after`) only appears in rules within do-blocks. In Spitfire,
`block_identifier` wasn't in `@terminals` or `@peeks` so the parser
could consume `else`/`rescue` as expression tokens instead of stopping
at block boundaries. For example, a stab body inside a do-block could
run past `else`.

---

A lot of those changes don't have corresponding tests in the PR
specifically. I was working on fixing the issues in property tests added
in #78, #80 and #81. The way I tested all this was to copy the new tests
from those files into my branch and running them locally.

After the changes here:
- Only 6 failures in tests from #78 
- All tests from #80 pass, so that PR could be merged after this
- Only 3 failures in tests from #81 (which I believe are unrelated to
this PR)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants