feat: support local development using Buck2

[bolinfest]

This is an experiment to get the Rust portion of the Codex repo setup to build with Buck2.

The motivation is to get build time benefits, both for local/incremental builds as well as in CI, though of note:

• For reference, general advantages of Buck2: https://buck2.build/docs/getting_started/what_is_buck2/#why-use-buck2-key-advantages
• Locally, Buck2 runs a daemon, which helps speed up incremental builds.
• Using Buck2 means we can ultimately leverage distributed builds
• Because Buck2 is stricter on running builds and tests in an "isolated" environment, it holds us to a higher standard to ensure builds are hermetic
• Buck2 has a complementary project, Reindeer, which is used to create BUCK build files for existing crates: https://github.com/facebookincubator/reindeer (I am not aware of a comparable offering for Bazel)

Unfortunately, buck2 test does not currently cache test results as Bazel does:

facebook/buck2#183

This is pretty important, as we want incremental invocations of:

./scripts/buck2 test //codex-rs/a

to be as fast as possible, so I'll follow up on this.

[dzbarsky]

@bolinfest Hi! I was pointed to this PR by the bazel community, some folks were a bit surprised to see this as Openai is generally using Bazel internally for many projects :)

I don't want to derail the experiments, but I was also curious why not use Bazel, given the feature gaps you mentioned in Buck?

Using Buck2 means we can ultimately leverage distributed builds

We can do this in Bazel from day 1 using Buildbuddy free tier :)

Because Buck2 is stricter on running builds and tests in an "isolated" environment, it holds us to a higher standard to ensure builds are hermetic

Last time I checked, Buck2 did not support any sandboxing for local execution, only remote. Has that been fixed?

Buck2 has a complementary project, Reindeer, which is used to create BUCK build files for existing crates: https://github.com/facebookincubator/reindeer (I am not aware of a comparable offering for Bazel)

There's a slightly different approach with Bazel - it integrates with Cargo.lock/Cargo.toml and generates external repositories with BUILD files. I maintain a ruleset that does this. It sounds like this was the missing piece you may have been unaware of?

The advantage of using Bazel is that it seems like the rust/cargo support may be more mature - rules_rs and rules_rust go through a lot of effort to match cargo's build script execution environment very closely, which should make the reindeer fixups in this PR unnecessary.

If you're open to it, I'm happy to prototype a Bazel integration in parallel to this Buck one - it should support test caching and remote build/test out-of-the-box, and I'm hopeful it will also support remote cross-builds to simplify the release story (though it depends on the complexity of the dep tree, we'll see). WDYT?

[bolinfest]

@dzbarsky Hello!

Honestly, I decided to try Buck2 first because I have personal contacts with the Buck2 and Reindeer folks (incidentally, I also created Buck "1"). Also, Buck2 is written in Rust, so I would certainly expect its Rust build rules to be performant.

I'm not keen to require folks to install a JVM to be able to run Bazel. As you can see in this PR, the DotSlash file for buck2 makes it straightforward to run it with minimal extra work (we already require DotSlash for one of our integration tests).

Last time I checked, Buck2 did not support any sandboxing for local execution, only remote. Has that been fixed?

I haven't looked, tbh.

Though what sort of "sandboxing" does it leverage? Is it using micro-VMs/containers or something else?

There's a slightly different approach with Bazel - it integrates with Cargo.lock/Cargo.toml and generates external repositories with BUILD files.

External repositories? That sounds a bit complicated...

If you're open to it, I'm happy to prototype a Bazel integration in parallel to this Buck one - it should support test caching and remote build/test out-of-the-box, and I'm hopeful it will also support remote cross-builds to simplify the release story (though it depends on the complexity of the dep tree, we'll see). WDYT?

If you're willing to give it a shot, I'm certainly interested! For sure, distributed builds and caching [however we get it] should materially speed things up for both iterative development and CI jobs.

[dzbarsky]

@dzbarsky Hello!

Honestly, I decided to try Buck2 first because I have personal contacts with the Buck2 and Reindeer folks (incidentally, I also created Buck "1"). Also, Buck2 is written in Rust, so I would certainly expect its Rust build rules to be performant.

I mean build rules are starlark, so it comes down to which starlark interpreter is faster :)

I'm not keen to require folks to install a JVM to be able to run Bazel. As you can see in this PR, the DotSlash file for buck2 makes it straightforward to run it with minimal extra work (we already require DotSlash for one of our integration tests).

You don't need a JVM to run bazel, it ships as a self-contained executable. All you need is a .bazelversion file to point at the right version and bazelisk installed on the system (or this script checked-in if you want to avoid any prereq)

Last time I checked, Buck2 did not support any sandboxing for local execution, only remote. Has that been fixed?

I haven't looked, tbh.

Though what sort of "sandboxing" does it leverage? Is it using micro-VMs/containers or something else?

Depends on the platform. OSX uses sandbox-exec which can block network. Linux uses fs/network namespaces. Both tend to create symlink trees for actions to isolate them from inputs, etc. It's also possible to spawn docker containers per action but the overhead is much higher and not worth it in my experience.

There's a slightly different approach with Bazel - it integrates with Cargo.lock/Cargo.toml and generates external repositories with BUILD files.

External repositories? That sounds a bit complicated...

Sorry, that's a bazelism. It just means "code that is pulled from elsewhere instead of vendored", but it's completely transparent. The user experience is just bazel build //target and bazel manages the download, build file generation, etc as part of that single command.

If you're open to it, I'm happy to prototype a Bazel integration in parallel to this Buck one - it should support test caching and remote build/test out-of-the-box, and I'm hopeful it will also support remote cross-builds to simplify the release story (though it depends on the complexity of the dep tree, we'll see). WDYT?

If you're willing to give it a shot, I'm certainly interested! For sure, distributed builds and caching [however we get it] should materially speed things up for both iterative development and CI jobs.

Cool, i'll give it a try, hopefully it shouldn't be too bad. It sounds from the above that you are (understandly) much more familiar with Buck than Bazel so hopefully a working demo should show what I mean a lot more clearly :)