Consider gems under `platform: :windows` filter in Gemfile when running on Windows with ARM architecture #8428

deivid-rodriguez · 2025-01-23T12:31:23Z

What was the end-user or developer problem that led to this PR?

Ruby now has arm64 support on Windows. However, the :windows value for the platform option in the Gemfile DSL doesn't select the gem when running on the new platform (aarch64-mingw-ucrt).

What is your fix for the problem, implemented in this PR?

While reviewing the proposed fix in #8378, I noticed several issues in how Windows is currently handled, so I started introducing some improvements that I eventually extracted to this separate PR.

The main fix to support the new platform is 96496e3, which adds a new value to the list of Windows platforms that the :windows value matches, and changes the way the matching is done to use Gem::Platform#===.

Make sure the following tasks are checked

Describe the problem / feature
Write tests for features and bug fixes
Write code to solve the problem
Make sure you follow the current code style and write meaningful commit messages without tags

bundler/spec/other/ext_spec.rb

I don't think any supported platform has these names, so the mapping should be unnecessary.

It's always going to be either the first or the second platform in the list so no need to keep an explicit list of all platforms.

I think they add unnecessary indirection and inconsistency to the specs.

For better debuggability.

We need to move platform monkeypatching to happen earlier because otherwise `Bundler::GemHelpers` will use the constants before they have actually been defined.

Co-authored-by: Johnny Shields <johnny.shields@gmail.com>

johnnyshields · 2025-01-24T16:16:33Z

Hi David, this looks very good at first glance. Did you also see my other PR #8379?

ntkme · 2025-01-24T16:22:01Z

Look good to me.

deivid-rodriguez · 2025-01-24T16:41:48Z

Thanks for having a look!

I did read the description of #8379 and seems reasonable, but did not look into the implementation yet. I was hoping these changes would make getting that one ready easier. Do you mind rebasing that and adapting it to what I'm doing here? I can have a look next week if tests are still giving trouble.

I will merge this one now.

johnnyshields · 2025-01-25T03:37:10Z

@deivid-rodriguez @ntkme @larskanis @MSP-Greg I just want to make sure we're all on the same page here.

(Correct if wrong) it seems like after this change that:

All mingw (regardless of x86, x64, aarch64 architecture) now selects a generic platform universal-mingw, and
that -ucrt is now implicit for universal.

I interpreted @ntkme's comment to mean that universal-mingw should only be used for ARM64EC and ARM64X compilation, both of which will run on both x64 and aarch64 but not x86 (32-bit)

I think we need to ensure that:

rake-compiler reserves universal-mingw only for ARM64EC/ARM64X + UCRT builds.
- AFAIK rake-compiler is not yet supporting ARM64EC/ARM64X builds (raised issue Support Window ARM64EC and ARM64X compilation modes for universal-mingw rake-compiler/rake-compiler#246)
- (Theoretically, universal-mingw-msvcrt could also be possible, but unlikely to be used in wild.)
Only aarch64 and x64 + UCRT select universal-mingw gems.
- Conversely, x86 arch and msvcrt/non-ucrt runtimes do not select universal-mingw.
If a universal-mingw build is not available, then we fallback to x64-mingw-ucrt, aarch-mingw-ucrt etc.
- For historical reasons, for all mingw other than universal, the ucrt is not implicit (msvcrt is implicit)

If these conditions are all met today with test cases, then no issue :)

johnnyshields · 2025-01-25T04:07:33Z

bundler/spec/other/ext_spec.rb

-    expect(generic(pl("x86-mingw32"))).to eq(pl("x86-mingw32"))
+  it "converts 32-bit mingw platform variants into universal-mingw" do
+    expect(generic(pl("i386-mingw32"))).to eq(pl("universal-mingw"))
+    expect(generic(pl("x86-mingw32"))).to eq(pl("universal-mingw"))


Pending discussion above, I think 32-bit should not select universal.

I also think we need a case that x64-mingw-msvcrt does not select universal-mingw, but does select universal-mingw-msvcrt

ntkme · 2025-01-25T04:11:58Z

I checked again and found that windows arm64 is super convoluted:

If ruby binary is native arm64, it can only load native arm64 DLL.
If ruby binary is ARM64EC, it can load ARM64EC/x86_64 DLL, but not native arm64 DLL
If ruby binary is x86_64, it cannot load ARM64EC or native arm64 DLL
ARM64X is a fat binary that combing ARM64EC and native arm64. This means ARM64X DLL can be loaded by ARM64EC executable or ARM64 executable, but not x86_86 executable.

@deivid-rodriguez So this is somewhat messed up right now.

Regarding ucrt being implicit, I think it’s not an issue unless someone manually build ruby >=3.1 with msvcrt because:

Ruby installer uses ucrt for all ruby >=3.1
Current rubygem/bunlder no longer support ruby <3.1

ntkme · 2025-01-25T04:21:47Z

In other words:

If ruby binary is ARM64EC, or ARM64X running on x64, the following kinds of DLL can be loaded:
- ARM64X
- ARM64EC
- x64
If ruby binary is native aarch64, or ARM64X running on aarch64:
- ARM64X
- aarch64
If ruby binary is native x64:
- x64

ntkme · 2025-01-25T04:38:56Z

RubyInstaller is only publishing native x64 and native aarch64. Currently, there is no ARM64EC or ARM64X RubyInstaller distribution, which means universal isn’t universal as of now - with the limited flavors of windows ruby distribution, ARM64EC DLL won’t not work anywhere, and ARM64X DLL only works on x64.

For now what we need is just to have aarch64-mingw-ucrt treated as a supported platform, and native gem published with this exact platform can be installed. For example, sass-embedded gem has been publishing native aarch64-mingw-ucrt prebuilt since April 2024.

johnnyshields · 2025-01-25T04:43:14Z

@ntkme I agree the simplest for now is to just do separate builds of x64-mingw-ucrt and aarch64-mingw-ucrt, and rollback universal. (@deivid-rodriguez -- looks like this PR should be reverted and reworked,)

If we do go with universal in the future, it could be something like:

universal is arm64x only (not arm64ec, since it doesn't run on aarch64 in non-emulation mode.)
We could extend the "current ruby" architecture values to be something like x64+arm64x-mingw-ucrt and aarch64+arm64x-mingw-ucrt.
- The Windows installer could make a single arm64x Ruby build, and then the x64+ or aarch64+ prefix is evaluated at runtime.
- These would then be eligible for universal-mingw-ucrt gems, but perhaps even then it would less confusing to call these gems arm64x-mingw-ucrt or universal+arm64x as they are not truly "universal".

However, maybe all this is more trouble than it's worth, as just treating x64 and aarch64 as separate architectures with separate Ruby/gem binaries is not a huge deal.

@larskanis ⬆️

ntkme · 2025-01-25T05:00:54Z

call these gems arm64x-mingw-ucrt as they are not truly "universal"

Right, I was under the impression that arm64x works similar to how universal binary works on macOS. I was totally wrong about that.

@deivid-rodriguez I think most of the changes in this PR still looks good, and perhaps we should just revert the universal-mingw and add explicit aarch64-mingw-ucrt for now.

larskanis · 2025-01-27T09:58:30Z

ARM64EC is an ABI with "emulation compatible" ARM64 code, which can not be executed on a real AMD or Intel CPU. ARM64X is native ARM64 code (which is very similar to the ARM64 Linux ABI) and "emulation compatible" ARM64 code combined into one DLL or EXE file. Neither of them can be executed on a AMD or Intel CPU. It makes most sense for system DLLs on Windows on ARM. The details are described here.

From a RubyInstaller perspective neither ARM64EC nor ARM64X makes sense to support:

ARM64EC would allow to run ruby and C extensions at almost native speed and still load emulated x64 DLLs into the same ruby process. But it can not load native ARM64 DLLs into the process. In the benchmarks which I executed, the emulated ruby had roughly 80% of the speed of the native ARM64 version. So the emulation runs x64-ruby very well. When the use case requires to run x64 DLLs into the ruby process, than it's best to use the emulated x64 RubyInstaller. If native ARM64 DLLs shall be loaded then the RubyInstaller ARM64 version can be used. It's not possible to mix x64 and native ARM64 into one process.
ARM64X could make sense for the libruby DLL. It would allow to embed ruby into an native ARM64 process as well as into an emulated x64 process. But embedded ruby is not a use case for RubyInstaller. If someone want's to embed ruby, he can compile libruby on its own for the desired architecture/ABI.

So universal-mingw or universal-windows in rubygems doesn't make much sense to me, since Windows doesn't have such a executable format usable on different processor architectures.

IMHO #8378 is all that's needed. It adjusts platform: :windows filter to include aarch64-mingw-ucrt. Loading specific platform gems for aarch64-mingw-ucrt already works for instance with the sass-embedded gem. There's no need to change anything.

In general I think mingw platform should not be treated very differently to windows platform. mingw should also not be promoted. This is because the intension of mingw has always been to follow mswin as much as possible and my experience is that 90% of all issues are related to windows in general and not specific to mingw or mswin implementations.

deivid-rodriguez · 2025-01-27T11:32:23Z

Thank you for all your comments. I will review this as soon as possible and revert if necessary.

deivid-rodriguez · 2025-01-27T16:07:28Z

Ok, so to clarify, this code does not change how RubyGems treats Windows platforms when choosing gem package variants for installation, it only changes the way platform: :windows in a Gemfile decides whether to include gems for resolution under a specific platform.

To give an example, if gem "tzinfo-data", platforms: :windows is included in a Gemfile, Bundler will now consider the tzinfo-data gem for resolution when running on an arm windows platform while before the gem would simply be ignored.

But it will still select the most specific match for installation when available, so if a arm64-mingw-ucrt variant is available, it will prefer to install that over a universal-mingw variant. That behavior does not change with this PR.

deivid-rodriguez · 2025-01-27T16:13:02Z

IMHO #8378 is all that's needed. It adjusts platform: :windows filter to include aarch64-mingw-ucrt. Loading specific platform gems for aarch64-mingw-ucrt already works for instance with the sass-embedded gem. There's no need to change anything.

That's what this PR does. Except that if in the future any other mingw architecture becomes a thing, it will be automatically matched by platform: :windows as well.

johnnyshields · 2025-01-27T18:11:25Z

@deivid-rodriguez thanks for the clarification, I took a second look and it should be good as-is. @larskanis and others please kindly test 🙇‍♂️

(And sorry for causing confusion here, the generic platform stuff tripped me up. Thank you everyone for the productive discussion on this thread, I learned quite a bit!)

deivid-rodriguez · 2025-01-27T18:13:38Z

No, no need to apologize, the code base is pretty confusing in this area and we use the same terminology for completely different things at times.

johnnyshields · 2025-01-27T18:17:54Z

Thanks again. By the way, this PR basically achieves what I was intending to do in PR #8379. I added a few deprecations there which I can split out into a new PR. I was also thinking to make the generic platform as universal-windows to better align with platform: :windows (and to be neutral as to mingw / mswin) -- would you accept that change as well?

deivid-rodriguez · 2025-01-27T18:30:28Z

Yes, please do rework #8379 or split out the deprecations to a separate PR 👍.

The idea of "universal-windows" sounds nice (it would fix this same issue for mswin on arm, right?), but it seems tricky to implement because currently Gem::Platform knows nothing about "windows" as a platform identifier, and this PR started using Gem::Platform#=== for checking whether the :windows platform DSL value matches the current platform. But I'm happy if you want to keep exploring this kind of improvement, just be aware that there may be dragons!

larskanis · 2025-01-27T19:31:55Z

@larskanis and others please kindly test 🙇‍♂️

master branch works fine on aarch64-mingw-ucrt. gem "tzinfo-data", platforms: %i[ windows jruby ] installs tzinfo-data gem and it selects the right architecture Installing sass-embedded 1.83.4 (aarch64-mingw-ucrt) and it works.

…ws-support Support installing arm native gems on Windows (cherry picked from commit 969f320)

deivid-rodriguez force-pushed the deivid-rodriguez/add-arm-windows-support branch from 9b72cb6 to fb2a3c3 Compare January 23, 2025 12:32

deivid-rodriguez marked this pull request as draft January 23, 2025 14:19

deivid-rodriguez force-pushed the deivid-rodriguez/add-arm-windows-support branch 3 times, most recently from e896819 to 5d540ed Compare January 23, 2025 18:12

This comment was marked as spam.

Sign in to view

deivid-rodriguez force-pushed the deivid-rodriguez/add-arm-windows-support branch 2 times, most recently from f6b85ee to 3ffa16a Compare January 24, 2025 07:02

deivid-rodriguez mentioned this pull request Jan 24, 2025

Support "mingw" generically as a Windows OS architecture (support for aarch64-mingw-ucrt) #8378

Closed

larskanis reviewed Jan 24, 2025

View reviewed changes

bundler/spec/other/ext_spec.rb Outdated Show resolved Hide resolved

deivid-rodriguez added 9 commits January 24, 2025 11:10

Remove unnecessary mapping

6b1bdfc

I don't think any supported platform has these names, so the mapping should be unnecessary.

Refactor specs to use platform strings directly

d24c0c2

Improve simulating Windows during Bundler specs

b48168b

Remove unnecessary test code

5d3cdb3

It's always going to be either the first or the second platform in the list so no need to keep an explicit list of all platforms.

Remove unnecessary platform helpers

609924d

I think they add unnecessary indirection and inconsistency to the specs.

Use RubyGems helpers for finding a non local tag for specs

a1d4ec1

Define Bundler::SpecSet#to_s

21d252f

For better debuggability.

Remove duplicated bundle install

1774ac5

Reuse platform constants

086c343

We need to move platform monkeypatching to happen earlier because otherwise `Bundler::GemHelpers` will use the constants before they have actually been defined.

deivid-rodriguez force-pushed the deivid-rodriguez/add-arm-windows-support branch from 3ffa16a to dd82a9a Compare January 24, 2025 10:11

deivid-rodriguez marked this pull request as ready for review January 24, 2025 10:12

Support installing arm native gems on Windows

96496e3

Co-authored-by: Johnny Shields <johnny.shields@gmail.com>

deivid-rodriguez force-pushed the deivid-rodriguez/add-arm-windows-support branch from dd82a9a to 96496e3 Compare January 24, 2025 13:02

deivid-rodriguez added the bundler: enhancement label Jan 24, 2025

deivid-rodriguez merged commit 969f320 into master Jan 24, 2025
91 checks passed

deivid-rodriguez deleted the deivid-rodriguez/add-arm-windows-support branch January 24, 2025 16:42

johnnyshields mentioned this pull request Jan 25, 2025

Support Window ARM64EC and ARM64X compilation modes for universal-mingw rake-compiler/rake-compiler#246

Closed

johnnyshields reviewed Jan 25, 2025

View reviewed changes

deivid-rodriguez changed the title ~~Support installing arm native gems on Windows~~ Consider gems under platform: :windows filter in Gemfile when running on Windows under ARM architecture Jan 27, 2025

deivid-rodriguez changed the title ~~Consider gems under platform: :windows filter in Gemfile when running on Windows under ARM architecture~~ Consider gems under platform: :windows filter in Gemfile when running on Windows with ARM architecture Jan 27, 2025

deivid-rodriguez mentioned this pull request Feb 11, 2025

Fix legacy Windows :platform values no longer working #8482

Merged

4 tasks

deivid-rodriguez added a commit that referenced this pull request Feb 17, 2025

Merge pull request #8428 from rubygems/deivid-rodriguez/add-arm-windo…

58bcab5

…ws-support Support installing arm native gems on Windows (cherry picked from commit 969f320)

Uh oh!

Consider gems under platform: :windows filter in Gemfile when running on Windows with ARM architecture #8428

Consider gems under platform: :windows filter in Gemfile when running on Windows with ARM architecture #8428

Uh oh!

Conversation

deivid-rodriguez commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was the end-user or developer problem that led to this PR?

What is your fix for the problem, implemented in this PR?

Make sure the following tasks are checked

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

johnnyshields commented Jan 24, 2025

Uh oh!

ntkme commented Jan 24, 2025

Uh oh!

deivid-rodriguez commented Jan 24, 2025

Uh oh!

Uh oh!

johnnyshields commented Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnnyshields Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ntkme commented Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntkme commented Jan 25, 2025

Uh oh!

ntkme commented Jan 25, 2025

Uh oh!

johnnyshields commented Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntkme commented Jan 25, 2025

Uh oh!

larskanis commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deivid-rodriguez commented Jan 27, 2025

Uh oh!

deivid-rodriguez commented Jan 27, 2025

Uh oh!

deivid-rodriguez commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnnyshields commented Jan 27, 2025

Uh oh!

deivid-rodriguez commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnnyshields commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deivid-rodriguez commented Jan 27, 2025

Uh oh!

larskanis commented Jan 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Consider gems under `platform: :windows` filter in Gemfile when running on Windows with ARM architecture #8428

Consider gems under `platform: :windows` filter in Gemfile when running on Windows with ARM architecture #8428

deivid-rodriguez commented Jan 23, 2025 •

edited

Loading

johnnyshields commented Jan 25, 2025 •

edited

Loading

johnnyshields Jan 25, 2025 •

edited

Loading

ntkme commented Jan 25, 2025 •

edited

Loading

johnnyshields commented Jan 25, 2025 •

edited

Loading

larskanis commented Jan 27, 2025 •

edited

Loading

deivid-rodriguez commented Jan 27, 2025 •

edited

Loading

deivid-rodriguez commented Jan 27, 2025 •

edited

Loading

johnnyshields commented Jan 27, 2025 •

edited

Loading