Plans for a Node creation API

[KaanOzkan]

We're currently adding Prism support to Sorbet for type checking. As part of this work, we need to programmatically create Prism nodes to translate RBS comments into equivalent Sorbet sig method calls.

Currently, we're using the exposed structs and functions directly: https://github.com/Shopify/sorbet/blob/73a82ff66355946c5e5e011bc5432e6228b84266/parser/prism/Factory.cc

Given the issue to reduce the public API scope, we wanted to check: would a public node creation API be something you'd be open to? We're happy to contribute to the design and implementation if it fits Prism's direction.

More context can be found in the Sorbet PR that introduces node creation: sorbet/sorbet#9667 🙇

[kddnewton]

So there's a bit of context here that I'm not sure if I've ever written down, so let me try to articulate it here.

In theory, the group of folks working on the parse.y parser have been attempting to replicate the prism API. This means whatever our public API actually looks like. It's not entirely clear to anyone what our public API actually is, so this has been a point of difficulty for them.

There's a tradeoff here where making the nodes opaque pointers would in theory may make maintenance easier going forward, but it would drastically increase the number of functions (and/or macros, depending on how they were implemented) because accessing a field would have to go through a function or macro. So a couple of ways forward that I'm thinking about this are either we create an opaque pointer API and then ship an internal header that CRuby includes in order to access the fields directly, or we give up on that future because it's just not practical. Truthfully I'm leaning toward the latter.

So as to the public node creation API: I'm generally amenable to it, but I would definitely want it to be generated off config.yml (probably in an ast.c file). Once that is done, I would want to replace the existing functions we have in prism.c with these generated functions. The biggest piece of logic that each of these functions does today is that it calculates the location of the nodes. We will need to decide a good, standard way to pass this information. Ideally we really shouldn't need to put any more information in config.yml to do this.

So yes, I'm open to it, I wouldn't mind it provided it is used at inception inside prism.c (as opposed to just a public API that we do not use), and am happy to review once stuff is ready. As a first step, I think a good proof of concept would be to create a new template, write out the creation function for a single node, then show what the diff would look like inside prism.c to get a handle of if we're going in the right direction.

[kddnewton]

As a follow-up here: it seems like you want an API for initializing nodes, but not necessarily creating them right? Presuming we have an arena allocation, there will already be memory allocated, so this is about setting the right fields?

If that's the case, I think we can generate macros instead of functions, which will reduce the overall code bloat coming from a bunch of generated functions.

[amomchilov]

We'd need to allocate brand new nodes out of the arena, and then initialize them