feat(macOS): Capture audio on macOS using Tap API#4209
feat(macOS): Capture audio on macOS using Tap API#4209ThomVanL wants to merge 32 commits intoLizardByte:masterfrom
Conversation
ThomVanL
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Bundle ReportChanges will increase total bundle size by 121 bytes (0.01%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sunshine-esmAssets Changed:
Files in
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4209 +/- ##
==========================================
- Coverage 12.09% 12.09% -0.01%
==========================================
Files 87 87
Lines 17612 17613 +1
Branches 8097 8097
==========================================
Hits 2131 2131
- Misses 14579 14580 +1
Partials 902 902
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Thanks for the PR. I have built this on my M1 Pro MBP and have been trying to get it to work, but haven't had much success so far. I am not able to get any sound to be captured and/or sent. So far I am only testing with the simplest use case of audio playing out of my MBP speakers. I edited the code to make the tap and aggregate non-private, so I could try to view the tap/aggregate with Apple's sample app I can see the tap, but not the aggregate. I'm not sure what's wrong. Can you detail your testing process? I do seem to be getting OK log entries and since I'm running from iTerm, all my permissions seem to be in order (iTerm has access to many things). I had to make the following changes to get it to build: plus a fix in our input.cpp that breaks the latest Xcode 16.4, I'm surprised if you didn't run into this one. I am running Xcode 16.4 clang-1700.0.13.5). |
|
Posting this here for reference in case it's needed for permissions of unit tests. We used to need something like this for macports, but it was never necessary for homebrew. Sunshine/.github/workflows/CI.yml Lines 841 to 898 in b662b8e |
|
Hey @andygrundman, thanks for taking the time to test the PR. Sorry to hear it’s not working correctly. Here’s my setup on an M4 MBP and what I did. clang --version
Apple clang version 17.0.0 (clang-1700.0.13.5)
Target: arm64-apple-darwin24.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/binI made the changes in VS Code and followed the steps in mkdir build
cmake -B build -G Ninja -S .
ninja -C buildFor testing I did the following steps:
So at one point I'm running three streams simultaneously and the audio plays through the devices. My audio_sink = Steam Streaming Speakers
macos_system_wide_audio_tap = true
origin_web_ui_allowed = pc
stream_audio = true
wan_encryption_mode = 2With I’m familiar with the sample app! It might be the case that the aggregate device settings are probably still marked as private. You’ll need to flip those to Sunshine/src/platform/macos/av_audio.mm Line 604 in 6404705 Sunshine/src/platform/macos/av_audio.mm Line 649 in 6404705 Then the tap will show up in Apple's sample app's UI. 
And the aggregate device shows up as well. 
But even with those values flipped to |
|
Thanks for the detailed info. I forgot my log only had Info level, when using Debug my log does look exactly like yours. I feel like I must just have a permission issue, maybe using iTerm as the permission "owner" isn't correct. Do you see both of these items? What process names are they using? I only get a System Audio item when recording a test file in AudioTapSample. Here's basically what my Screen & System Audio Recording settings look like: If this is the issue, I wonder how Sunshine can detect that it doesn't actually have permission, hmm. |
|
You're right, because I ran into a similar issue with VS Code. Granting it "Screen & System Audio Recording" wasn’t enough; I had to explicitly allow "System Audio Recording Only." From what I found online, the VS Code app bundle itself might be causing the problem. I also tried running 
I did not even notice the little privacy notice at the top until just now, thanks for that. Here's what it's like on my end. 
When I launched Edit: just to be clear, my dev loop consists of launching sunshine builds through the VS Code integrated terminal. |
Yeah, I wondered about ARC too and did a quick search for Good to hear that lines up with your findings as well. |
ThomVanL
force-pushed
the
users/thomasvanlaere/feat-macos-ca-taps
branch
from
d75fde0 to
db3d2df
Compare
…e/feat-macos-ca-taps
|
|
Hi @ReenigneArcher and @andygrundman, happy new year! 🎉 Also, my apologies for the delay in regards to this PR. I've removed some of the unit tests that turned out to be integration tests in disguise (particularly the TCC ones). After poking around for best practices on running integration tests in a CI/CD pipeline with TCC, I mostly found workarounds that felt a little hacky, so I decided to strip those integration tests out and stick with pure unit tests instead. Not sure if that sits right with you both, but feel free to let me know if it needs to be changed. |
This comment was marked as outdated.
This comment was marked as outdated.
|
This is looking pretty good, thanks! What I did was create a branch from yours, cherry-pick in some helper things (my build script, input fixes, my previous compile fix (how does this build for you in Tahoe?), and moved a bunch of noisy logging from debug to verbose). With all that out of the way, the main commit in my branch has a bunch of improvements for you to consider. Here's the items in that commit: https://github.com/andygrundman/Sunshine/commits/andyg.feat-macos-ca-taps-review/ I'm not married to these changes, so feel free to counter any of them. I'm not sure what the right buffer size should be, but I felt like 4096 * channels seemed small and just as arbitrary. The public tap thing is neat to look at, especially check out the Apple utility HALLab which exposes all the internals of audio devices. I worry that public taps may not get cleaned up in some situations such as a crash, although maybe private taps get automatically cleaned up. Using an env var for devs is a bit ugly but there's no other way to view the aggregate device. Re: resampling: do we need to have so much or any resampling code? Doesn't CoreAudio handle a lot of this behind the scenes? I couldn't get the system tap to ever need resampling (I think because it's not using a specific device), and when using BlackHole 16ch sink directly, the resampling code doesn't seem to be called anyway. Even if I run BlackHole at 192k or something, it seems to work fine. I did a trace in Activity Monitor and you can find the bit where the OS is resampling from 192k -> 48k if you search for This all works quite well for stereo. For surround, I guess we need to think about it some more. I just stumbled across a project called SoundPusher that creates a virtual 5.1 device and does on-demand AC3 encoding. I want to study it some more. https://codeberg.org/q-p/SoundPusher I found it via https://www.maven.de/2025/04/coreaudio-taps-for-dummies/ by the way. Edit: I think I just realized the conversion code is mostly there for multichannel right? I honestly have not even tested surround because I think that still requires a virtual audio device like what SoundPusher includes, or BlackHole. I'm not sure it works with Tap. This PR should probably just focus on making stereo taps work in the most user-friendly way possible. |
|
Another edit: I had a few moments of panic while testing this, you know the feeling: the audio is skipping and glitching out real bad, and your first thought is there's some bug in the realtime buffer code. Nope, it's all just AWDL. This code basically becomes unusable on wifi devices that don't have N1 chips. The iPad I have been using as a client still had 6ghz 6E mode enabled which is a death sentence for Moonlight. My N1-equipped iPhone Air streams it perfectly. Thanks Apple. |
| list(APPEND SUNSHINE_EXTERNAL_LIBRARIES | ||
| ${APP_KIT_LIBRARY} | ||
| ${APP_SERVICES_LIBRARY} | ||
| ${AV_FOUNDATION_LIBRARY} | ||
| ${CORE_MEDIA_LIBRARY} | ||
| ${CORE_VIDEO_LIBRARY} | ||
| ${FOUNDATION_LIBRARY} | ||
| ${AUDIO_TOOLBOX_LIBRARY} | ||
| ${AUDIO_UNIT_LIBRARY} | ||
| ${CORE_AUDIO_LIBRARY} | ||
| ${VIDEO_TOOLBOX_LIBRARY}) |
There was a problem hiding this comment.
Could this list be alphabetized?
| if(OPENSSL_FOUND) | ||
| include_directories(SYSTEM ${OPENSSL_INCLUDE_DIR}) | ||
| endif() |
There was a problem hiding this comment.
This shouldn't be necessary?
| include_directories(/opt/homebrew/opt/opus/include) | ||
| link_directories(/opt/homebrew/opt/opus/lib) |
There was a problem hiding this comment.
These paths depend on the architecture of homebrew, and can't be hardcoded.
| Sunshine supports native system audio capture on macOS 14.0 (Sonoma) and newer via Apple’s Audio Tap API. | ||
| To use it, simply leave the **Audio Sink** setting blank. | ||
|
|
||
| If you are running macOS 13 (Ventura) or earlier—or if you prefer to manage your own loopback device—you can still use [Soundflower](https://github.com/mattingalls/Soundflower) or [BlackHole](https://github.com/ExistentialAudio/BlackHole) and enter its device name in the **Audio Sink** field. |
There was a problem hiding this comment.
| If you are running macOS 13 (Ventura) or earlier—or if you prefer to manage your own loopback device—you can still use [Soundflower](https://github.com/mattingalls/Soundflower) or [BlackHole](https://github.com/ExistentialAudio/BlackHole) and enter its device name in the **Audio Sink** field. | |
| If you prefer to manage your own loopback device—you can still use [Soundflower](https://github.com/mattingalls/Soundflower) or [BlackHole](https://github.com/ExistentialAudio/BlackHole) and enter its device name in the **Audio Sink** field. |
macos-13 isn't supported anymore
| Sunshine now supports system audio capture natively on macOS 14.0 (Sonoma) and later, | ||
| using the built-in Core Audio Tap API. | ||
|
|
||
| On macOS 13 or earlier, or if you prefer a virtual loopback device, | ||
| you can still use "Soundflower" or "BlackHole" for system audio capture. | ||
|
|
||
| Gamepads are not currently supported on macOS. |
There was a problem hiding this comment.
| Sunshine now supports system audio capture natively on macOS 14.0 (Sonoma) and later, | |
| using the built-in Core Audio Tap API. | |
| On macOS 13 or earlier, or if you prefer a virtual loopback device, | |
| you can still use "Soundflower" or "BlackHole" for system audio capture. | |
| Gamepads are not currently supported on macOS. | |
| Gamepads are not currently supported on macOS. |
Since we don't need to worry about macos-13 now, and audio will work out of the box... we can remove the comment about audio altogether.
| @@ -8,5 +8,8 @@ | |||
| <string>Sunshine</string> | |||
| <key>NSMicrophoneUsageDescription</key> | |||
| <string>This app requires access to your microphone to stream audio.</string> | |||
There was a problem hiding this comment.
Maybe this comment should be updated?
| venv/ | ||
|
|
||
| # Caches | ||
| .cache/ |
There was a problem hiding this comment.
What, specifically, is generating this directory?
| @@ -0,0 +1,388 @@ | |||
| /** | |||
| * @file tests/unit/platform/test_macos_av_audio.mm | |||
There was a problem hiding this comment.
| * @file tests/unit/platform/test_macos_av_audio.mm | |
| * @file tests/unit/platform/macos/test_av_audio.mm |
Can you rename the file to be in a macos specific folder. We also have a folder for windows there now as well.
| TEST_F(AVAudioTest, ObjectLifecycle) { | ||
| AVAudio *avAudio = [[AVAudio alloc] init]; | ||
| EXPECT_NE(avAudio, nil); // Should create successfully | ||
| [avAudio release]; // Should not crash |
There was a problem hiding this comment.
If this can crash, I think it should be wrapped in a try/catch. If we hit the catch, then it should trigger GTEST to fail.
Same for anywhere else it could crash. This is because when a test crashes it crashes the whole test run.
| "audio_sink": "Audio Sink", | ||
| "audio_sink_desc_linux": "The name of the audio sink used for Audio Loopback. If you do not specify this variable, pulseaudio will select the default monitor device. You can find the name of the audio sink using either command:", | ||
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Sunshine can only access microphones on macOS due to system limitations. To stream system audio using Soundflower or BlackHole.", | ||
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Leave this blank to use the built-in system audio capture (requires macOS 14.0 or later, using the Core Audio Tap API). Alternatively, specify a virtual device such as Soundflower or BlackHole if you prefer or are running an older version of macOS.", |
There was a problem hiding this comment.
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Leave this blank to use the built-in system audio capture (requires macOS 14.0 or later, using the Core Audio Tap API). Alternatively, specify a virtual device such as Soundflower or BlackHole if you prefer or are running an older version of macOS.", | |
| "audio_sink_desc_macos": "The name of the audio sink used for Audio Loopback. Leave this blank to use the built-in system audio capture. Alternatively, specify a virtual device such as Soundflower or BlackHole if you prefer or are running an older version of macOS.", |
ReenigneArcher
added
roadmap
3 participants
Description
This PR adds system-wide audio tap support for macOS. The implementation introduces:
Additional changes
cmakefiles for compatibility with Homebrew-based setup.opensslandopusare found automatically. This replaces the need to run manuallncommands, but I’m not sure if this is the best long-term approach. Feedback welcome.cmake/compile_definitions/unix.cmaketo ensureSUNSHINE_ASSETS_DIRresolves correctly.src_assets/macos/assets/Info.plistto prepare for required macOS permission prompts.macos_system_wide_audio_tapconfig option to theaudio_tstruct.Testing
Notes
setupMicrophonefunctionality as it is also a viable option!macos_system_wide_audio_tapsetting.If the AI involvement is a blocker for merging, no worries. I can totally understand! This was also a learning project for me and I had a lot of fun building it.
Screenshot
Web UI – Audio/Video Configuration
New option for enabling system-wide audio recording on macOS. Disables the audio sink option when checked.
macOS Permission Prompt
System permission request when Sunshine first tries to access system audio:
System Settings – Screen & System Audio Recording
macOS privacy settings showing Sunshine/Terminal access to "System Audio Recording Only":
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage