Get changes merged upstream

[Zialus]

What is the current progress on upstreaming these changes?

Related topics:
Support macOS Catalina (10.15)
Support macOS Mojave (10.14)
Support macOS High Sierra (10.13)

[LouisBrunner]

Hi @Zialus,

I would love to get these changes upstream but there doesn't seem to be much appetite/time for macOS support... (or potentially no confidence in my fixes as macOS support has been broken for so long that most of the regression test suite is failing)

I have already posted a few times patches (e.g. multithreading fix) or even my full repository (here) but nothing has been merged.

Feel free to make noise and bring attention to that issue!

[Cloudef]

Thank you for your hard work.

[paulfloyd]

We will be discussing this on Friday (10th February 2023). See the Valgrind developers mailing list for details.

[paulfloyd]

I've merged most of the 10.13 changes upstream today. I noticed that the two necp syscalls marked for 10.15 and later are on 10.13 (haven't tried 10.12, not sure if I can access a machine).

I also pushed a bunch of suppressions for DRD.

[paulfloyd]

A quick (maybe) update here. I recently got a mac mini M4 at home, so eventually I should be able to play around with the darwin arm64 code.

Also I've been working upstream to try to clean up 10.13 support as much as possible, starting with memcheck. See bugzilla item 511461 that is linked to the 10.13 item in the first post for this issue.

I've merged changes from a couple of files in this repo (coregrind/m_syswrap/syscall-amd64-darwin.S and coregrind/m_syswrap/syswrap-amd64-darwin.c). Added some redirs for _chk variants of str* and mem* functions. There were a few fixes and updates to filters and expecteds. The biggest thing that I've done is to rework the handling of syscall arguments. This was broken for syscall SYS_syscall, which also meant that the scalar tests were severely broken. I've also fixed a few syscall wrappers. I'd like to clean up this code some more to make Darwin and FreeBSD share as much as possible. I think that the only remaining difference is that on FreeBSD putSyscallArgsIntoGuestState puts arguments back in the original SYS_syscall order if that was the original syscall whilst on Darwin putSyscallArgsIntoGuestState puts tham back in the canonical order. From a user perspective the result should be the same, just that Valgrind does the suffling rather than the kernel.

I'm now getting about 25 memcheck regtest fails. Most of these are related to

• the very large number of leaks/reachables in Apple libc
• missing debuginfo

I'm not too worried about either of those. The debuginfo may be better with later versions of the LLVM compilers and there is not much we can do about the libc leaks.

That leaves only 3 kinds of error that are worth investigating further.

[LouisBrunner]

A quick (maybe) update here. I recently got a mac mini M4 at home, so eventually I should be able to play around with the darwin arm64 code.

That's great news!

* missing debuginfo

Depending what is missing, it might be fixed in this repo, I made a lot of changes to get all symbols to be loaded, etc.

[paulfloyd]

I'm still seeing issues with exec. The thread_alloca testcase fails intermittently but quite often, this repo and upstream and on macOS 10.13 and 13

==33358== Process terminating with default action of signal 11 (SIGSEGV)
==33358==  Access not within mapped region at address 0x70000651F01A
==33358==    at 0x1006754A3: _pthread_join_cleanup (in /usr/lib/system/libsystem_pthread.dylib)
==33358==    by 0x10067786F: _pthread_join (in /usr/lib/system/libsystem_pthread.dylib)
==33358==    by 0x100000BE5: main (thread_alloca.c:50)
==33358==  If you believe this happened as a result of a stack
==33358==  overflow in your program's main thread (unlikely but
==33358==  possible), you can try to increase the size of the
==33358==  main thread stack using the --main-stacksize= flag.
==33358==  The main thread stack size used in this run was 8388608.

The mapping is in here somewhere

--33376:1: aspacem 295: ANON 70000641d000-70000651cfff 1048576 rwx--
--33376:1: aspacem 296: ANON 70000651d000-70000651efff    8192 -----

--33376:1: aspacem 297:      70000651f000-70000659ffff  528384

^^^ this segment. no RWX bits

--33376:1: aspacem 298: file 7000065a0000-7000065a1fff    8192 rw--- d=0x000 i=30      o=0       (48,2024)
--33376:1: aspacem 299: ANON 7000065a2000-7000065a5fff   16384 rwx--
--33376:1: aspacem 300: ANON 7000065a6000-7000065a7fff    8192 -----
--33376:1: aspacem 301: ANON 7000065a8000-7000066a7fff 1048576 rwx--

[paulfloyd]

I think that I'm getting close to having 10.13 working as well as possible - there are some issues that I'm going to ignore/hope that they go away with a more recent compiler.

Outstanding things:
I don't understand what is going on in the none fdleak_cmsg test. Valgrind seems confused about what file descriptor 5 is doing.
Memcheck and massif - only isses that I'm going to postpone.
DRD and Helgrind - just started looking at them. Helgrind is just about totally broken, I'd like to understand why. DRD about 1/4 of the tests fail. Some are DWARF reading errors. If I can fix the Helgrind fundamental issue then I should be able to clean up their regtests a fair bit. Some of these tests hang.
DHAT - haven't analysed.
Callgrind, cachegrind and lackey - all tests pass.

The last thing that I'd like to do is to be able to run GUI apps. At the moment if I try to run TextEditor with --topol=none I get

==36819== Nulgrind, the minimal Valgrind tool
==36819== Copyright (C) 2002-2024, and GNU GPL'd, by Nicholas Nethercote et al.
==36819== Using Valgrind-3.27.0.GIT and LibVEX; rerun with -h for copyright info
==36819== Command: /Applications/TextEdit.app//Contents/MacOS/TextEdit
==36819==
--36819-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--36819-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--36819-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--36819-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
--36819-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 16 times)

valgrind: m_syswrap/syswrap-darwin.c:519 (void assign_port_name(mach_port_t, const HChar *)): Assertion 'i' failed.

host stacktrace:
==36819==    at 0x2580108B9: ??? (m_libcassert.c:426)
==36819==    by 0x258010C2F: ??? (m_libcassert.c:497)
==36819==    by 0x258010C04: ??? (m_libcassert.c:564)
==36819==    by 0x2580DBDBC: ??? (syswrap-darwin.c:519)
==36819==    by 0x2580CB872: ??? (syswrap-main.c:2400)
==36819==    by 0x2580C9FBE: ??? (scheduler.c:1214)
==36819==    by 0x2580C7960: ??? (scheduler.c:1588)
==36819==    by 0x2580DBA32: ??? (syswrap-darwin.c:117)
==36819==    by 0x2580DBCEA: ??? (syswrap-darwin.c:370)

sched status:
  running_tid=2

Thread 1: status = VgTs_WaitSys syscall unix:5 (lwpid 771)
==36819==    at 0x10005B512: open (in /usr/lib/dyld)
==36819==    by 0x10003A056: dyld::my_open(char const*, int, int) (in /usr/lib/dyld)
==36819==    by 0x100034108: dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==36819==    by 0x100033D65: dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==36819==    by 0x100033AE5: dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==36819==    by 0x1000332B3: dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==36819==    by 0x10002D7FE: dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==36819==    by 0x10002D4A1: dyld::load(char const*, dyld::LoadContext const&, unsigned int&) (in /usr/lib/dyld)
==36819==    by 0x100038009: dlopen (in /usr/lib/dyld)
==36819==    by 0x107BE1D85: dlopen (in /usr/lib/system/libdyld.dylib)
==36819==    by 0x1000FB083: ___NSFrameworkHandleForLibrarianSymbols_block_invoke (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==36819==    by 0x107B6CDB7: _dispatch_client_callout (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B6CD6A: dispatch_once_f (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x1000FB069: ___initializeLibrarianFunctions_block_invoke (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==36819==    by 0x107B6CDB7: _dispatch_client_callout (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B6CD6A: dispatch_once_f (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x1000DB9AE: -[NSFileManager _registerForUbiquityAccountChangeNotifications] (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==36819==    by 0x100261224: standardDefaultCenter (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==36819==    by 0x1000E6167: -[NSUserDefaults(NSUserDefaults) registerDefaults:] (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==36819==    by 0x100D56E48: +[NSApplication _initializeRegisteredDefaults] (in /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit)
==36819==    by 0x100D5544E: +[NSApplication initialize] (in /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit)
==36819==    by 0x1007BFE3B: CALLING_SOME_+initialize_METHOD (in /usr/lib/libobjc.A.dylib)
==36819==    by 0x1007B13B7: _class_initialize (in /usr/lib/libobjc.A.dylib)
==36819==    by 0x1007B0E8A: lookUpImpOrForward (in /usr/lib/libobjc.A.dylib)
==36819==    by 0x1007B0913: _objc_msgSend_uncached (in /usr/lib/libobjc.A.dylib)
==36819==    by 0x100D58879: NSApplicationMain (in /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit)
==36819==    by 0x107BE0014: start (in /usr/lib/system/libdyld.dylib)
client stack range: [0x1040C8000 0x1048C7FFF] client SP: 0x1048C69E8
valgrind stack range: [0x70000068E000 0x70000078DFFF] top usage: 6696 of 1048576

Thread 2: status = VgTs_Runnable syscall mach:31 (lwpid 4107)
==36819==    at 0x107FE41FA: mach_msg_trap (in /usr/lib/system/libsystem_kernel.dylib)
==36819==    by 0x107FE3713: mach_msg (in /usr/lib/system/libsystem_kernel.dylib)
==36819==    by 0x107FD6A7E: host_request_notification (in /usr/lib/system/libsystem_kernel.dylib)
==36819==    by 0x107B73891: _dispatch_mach_host_notify_update (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B6CDB7: _dispatch_client_callout (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B6CD6A: dispatch_once_f (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B8BA7E: _dispatch_event_loop_timer_arm (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B73B77: _dispatch_timers_program (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x107B8595A: _dispatch_kevent_worker_thread (in /usr/lib/system/libdispatch.dylib)
==36819==    by 0x108023FF5: _pthread_wqthread (in /usr/lib/system/libsystem_pthread.dylib)
==36819==    by 0x108023BE8: start_wqthread (in /usr/lib/system/libsystem_pthread.dylib)
==36819==    by 0x6D68537365636E64: ???
==36819==    by 0x7865646E496D64: ???
==36819==    by 0x43500002FFF: ???
==36819==    by 0x72504642FFFFFFFF: ???
==36819==    by 0x65636E6572656664: ???
==36819==    by 0x74536D656D685372: ???
==36819==    by 0x300000657460: ???
==36819==    by 0xEFFFFFFFE: ???
==36819==    by 0xFFF: ???
==36819==    by 0x3B: ???
client stack range: ??????? client SP: 0x70000A0E27E8
valgrind stack range: [0x70000417D000 0x70000427CFFF] top usage: 8848 of 1048576

Thread 3: status = VgTs_WaitSys syscall unix:368 (lwpid 10243)
==36819==    at 0x107FEE28A: __workq_kernreturn (in /usr/lib/system/libsystem_kernel.dylib)
==36819==    by 0x108024008: _pthread_wqthread (in /usr/lib/system/libsystem_pthread.dylib)
==36819==    by 0x108023BE8: start_wqthread (in /usr/lib/system/libsystem_pthread.dylib)
client stack range: ??????? client SP: 0x70000A165A58

Thread 4: status = VgTs_WaitSys syscall unix:368 (lwpid 8451)
==36819==    at 0x107FEE28A: __workq_kernreturn (in /usr/lib/system/libsystem_kernel.dylib)
==36819==    by 0x10802420D: _pthread_wqthread (in /usr/lib/system/libsystem_pthread.dylib)
==36819==    by 0x108023BE8: start_wqthread (in /usr/lib/system/libsystem_pthread.dylib)
client stack range: ??????? client SP: 0x70000A1E8EE8
valgrind stack range: [0x700004385000 0x700004484FFF] top usage: 4792 of 1048576

Do you know if there are things that I haven;t merged yet that could fix this?

[LouisBrunner]

Do you know if there are things that I haven;t merged yet that could fix this?

Most issues with GUI apps are linked to wqthread. Not sure if you have copied all of those fixes yet.
This specific assert doesn't ring a bell however.

Outside of that, running it using heavy logging will usually points to the issue fairly reliably.
My goto is usually -d -d -d -v -v -v -v -v --trace-children=yes --trace-syscalls=yes --trace-flags=10000001 --vex-guest-max-insns=1 --trace-malloc=yes --trace-signals=yes --track-origins=yes, it should show what is dyld doing to trigger the issue. This is usually enough to know why this is happening. It's very slow and verbose however.

[paulfloyd]

It's definitely a wqthread issue. The stack with a debug only build is

    frame #0: 0x000000025800a88d none-amd64-darwin`vgPlain_debugLog(level=28672, modulename="", format="") at m_debuglog.c:1350
  * frame #1: 0x0000000258055dd4 none-amd64-darwin`vgPlain_am_alloc_VgStack(initial_sp=0x000070000d04d9e8) at aspacemgr-common.c:580
    frame #2: 0x000000025815e048 none-amd64-darwin`allocstack(tid=2) at syswrap-darwin.c:158
    frame #3: 0x00000002581a0ff6 none-amd64-darwin`wqthread_hijack(self=123145520734208, kport=3083, stackaddr=123145520209920, workitem=123145520733056, reuse=3932193, kevent_count=1, sp=123145520732928) at syswrap-amd64-darwin.c:519

The crash is right at the start of VG_(debugLog). It looks like a problem with the stack.

With a normal -O2 -g the picture is quite messy. I'm seeing a lot of crashes in VG_(apply_StackTrace)

I think that I have all of the wqthread changes. The last large set of changes are in readmacho.c (for the moment I'm only merging and testing on 10.13 x86 and amd64).

[LouisBrunner]

Is this only for x86, amd64 or both?