do an editing pass

Author: cfbolz

posts/2025/02/allocation_sampling.md

@@ -1,7 +1,7 @@
 <!--
-.. title: Low Overhead Allocation Sampling in a Garbage Collected Virtual Machine
+.. title: Low Overhead Allocation Sampling with VMProf in PyPy's GC
 .. slug: pypy-gc-sampling
-.. date: 2025-01-26 14:38:00 UTC
+.. date: 2025-02-28 14:38:00 UTC
 .. tags: gc, profiling, vmprof
 .. category:
 .. link:
@@ -14,16 +14,22 @@
 
 There are many time-based statistical profilers around (like VMProf or py-spy
 just to name a few). They allow the user to pick a trade-off between profiling
-precision and runtime overhead. On the other hand there are memory profilers,
-which profile every allocation, resulting in precise profiling, but larger
-overhead (e.g. memray). In this post we describe our experimental approach to
-low overhead statistical memory profiling tightly integrated into VMProf and
-the PyPy Garbage Collector. The main technical insight is that the check,
-whether an allocation should be sampled, can be folded into the bump pointer
-allocator checks, PyPy’s GC uses to find out if it should start a minor
-collection. This makes the fast path with and without memory sampling the same.
-A tool like this can be handy for discovering functions, allocating lots of
-memory, leading to longer GC pauses caused by collections.
+precision and runtime overhead.
+
+On the other hand there are memory profilers
+such as [memray](https://github.com/bloomberg/memray). They can be handy for
+finding leaks or for discovering functions that allocate a lot of memory.
+Memory profilers typlically save every single allocation a program does. This
+results in precise profiling, but larger overhead.
+
+In this post we describe our experimental approach to low overhead statistical
+memory profiling. Instead of saving every single allocation a program does, it
+only saves every nth allocated byte. We have tightly integrated VMProf and the
+PyPy Garbage Collector to achieve this. The main technical insight is that the
+check whether an allocation should be sampled can be made free. This is done by
+folding it into the bump pointer allocator check that the PyPy’s GC uses to
+find out if it should start a minor collection. In this way the fast path with
+and without memory sampling are exactly the same.
 
 ## Background
 
@@ -33,12 +39,11 @@ both of them first.
 ### VMProf
 
 [VMProf](https://github.com/vmprof/vmprof-python) is a statistical time-based
-profiler for PyPy. This means, VMProf samples the stack a certain
-user-configured number of times per second. By adjusting this number, the
+profiler for PyPy. VMProf samples the stack of currently running Python
+functions a certain user-configured number of times per second. By adjusting
+this number, the
 overhead of profiling can be modified to pick the correct trade-off between
-overhead and precision of the profile. Additionally, VMProf has the ability to
-record JIT traces, giving an insight what the JIT does to the executed
-functions of the program. In the resulting profile, functions with huge runtime
+overhead and precision of the profile. In the resulting profile, functions with huge runtime
 stand out the most, functions with shorter runtime less so. If you want to get
 a little more introduction to VMProf and how to use it with PyPy, you may look
 at [this blog
@@ -51,24 +56,23 @@ two spaces for allocated objects, the nursery and the old-space. Freshly
 allocated objects will be allocated into the nursery. When the nursery is full
 at some point, it will be collected and all objects that survive will be
 tenured i.e. moved into the old-space. The old-space is much larger than the
-nursery and is collected less frequent and incrementally (not completely
-collected in one go, but step-by-step).
-
-This is still a quite simplified explanation of PyPy’s GC, but we don't want to
-go over every detail right now. Instead, we will take a look at the fast path
-(allocations in the nursery) and how the nursery is collected.
+nursery and is collected less frequently and
+[incrementally](/posts/2024/03/fixing-bug-incremental-gc.html) (not completely
+collected in one go, but step-by-step). The old space collection is not
+relevant for the rest of the post though. We will now take a look at nursery
+allocations and how the nursery is collected.
 
 ### Bump Pointer Allocation in the Nursery
 
-The nursery (a small continuous memory area) utilizes pointers, to keep track
-from where on the nursery is free and where it ends, called `nursery_free` and
+The nursery (a small continuous memory area) utilizes two pointers to keep track
+from where on the nursery is free and where it ends. They are called `nursery_free` and
 `nursery_top`. When memory is allocated, the GC checks if there is enough space
 in the nursery left. If there is enough space, the `nursery_free` pointer will
-be returned as the start address for the new allocated memory, and
+be returned as the start address for the newly allocated memory, and
 `nursery_free` will be moved forward by the amount of allocated memory.
 
 
-<img src="../../../images/2025_02_allocation_sampling_images/nursery_allocation.svg">
+<img src="/images/2025_02_allocation_sampling_images/nursery_allocation.svg">
 
 
 ``` Python
@@ -83,6 +87,11 @@ def allocate(totalsize):
       result = collect_and_reserve(totalsize)
   # result is a pointer into the nursery, obj will be allocated there
   return result
+
+def collect_and_reserve(size_of_allocation):
+    # do a minor collection and return the start of the nursery afterwards
+    minor_collection()
+    return gc.nursery_free
 ```
 
 Understanding this is crucial for our allocation sampling approach, so let us
@@ -91,7 +100,7 @@ go through this step-by-step.
 We already saw an example on how an allocation into a non-full nursery will
 look like. But what happens, if the nursery is (too) full?
 
-<img src="../../../images/2025_02_allocation_sampling_images/nursery_full.svg">
+<img src=/images/2025_02_allocation_sampling_images/nursery_full.svg">
 
 
 As soon as an object doesn't fit into the nursery anymore, it will be
@@ -100,13 +109,15 @@ old-space, so that the nursery is free afterwards, and the requested allocation
 can be made.
 
 
-<img src="../../../images/2025_02_allocation_sampling_images/nursery_collected.svg">
-
+<img src="/images/2025_02_allocation_sampling_images/nursery_collected.svg">
 
-Note that this is still a bit of a simplification.
+(Note that this is still a bit of a simplification.)
 
 ## Sampling Approach
 
+The last section described how the nursery allocation works normally. Now we'll
+talk how we integrate the new allocation sampling approach into it.
+
 To decide whether the GC should trigger a sample, the sampling logic is
 integrated into the bump pointer allocation logic. Usually, when there is not
 enough space in the nursery left to fulfill an allocation request, the nursery
@@ -116,20 +127,21 @@ is calculated by `sample_point = nursery_free + sample_n_bytes` where
 `sample_n_bytes` is the number of bytes allocated before a sample is made (i.e.
 our sampling rate).
 
-Image we'd have a nursery of 2MB and want to sample every 512KB allocated, then
+Imagine we'd have a nursery of 2MB and want to sample every 512KB allocated, then
 you could imagine our nursery looking like that:
 
-<img src="../../../images/2025_02_allocation_sampling_images/nursery_sampling.svg">
+<img src="/images/2025_02_allocation_sampling_images/nursery_sampling.svg">
 
-And now here comes our secret trick. We use the sample point as `nursery_top`,
+We use the sample point as `nursery_top`,
 so that allocating a chunk of 512KB would exceed the nursery top and start a
 nursery collection. But of course we don't want to do a minor collection just
 then, so before starting a collection, we need to check if the nursery is
 actually full or if that is just an exceeded sample point. The latter will then
-trigger a sample via VMProf's C-interface.
+trigger a VMprof stack sample. Afterwards we don't actually do a minor
+collection, but change `nursery_top` and immediately return to the caller.
 
-Now I got to tell you, that the last picture is only a half-truth. There do not
-exist more than one sample point at any time. After we created the sampling
+The last picture is a conceptual simplification. Only one sampling point exists
+at any given time. After we created the sampling
 point, it will be used as nursery top, if exceeded at some point, we will just
 add `sample_n_bytes` to that sampling point, i.e. move it forward.
 
@@ -141,58 +153,47 @@ def collect_and_reserve(size_of_allocation):
     if gc.nursery_top == gc.sample_point:
         # One allocation could exceed multiple sample points
         # Sample, move sample_point forward
-        while gc.sample_point < gc.nursery_free + size_of_allocation:
-            vmprof.sample_now()
-            gc.sample_point += sample_n_bytes
-            
+        vmprof.sample_now()
+        gc.sample_point += sample_n_bytes
+
         # Set sample point as new nursery_top if it fits into the nursery
         if sample_point <= gc.real_nursery_top:
-          gc.nursery_top = sample_point
+            gc.nursery_top = sample_point
         # Or use the real nursery top if it does not fit
         else:
-          gc.nursery_top = gc.real_nursery_top
+            gc.nursery_top = gc.real_nursery_top
 
         # Is there enough memory left inside the nursery
         if gc.nursery_free + size_of_allocation <= gc.nursery_top:
-          # Yes => move nursery_free forward
-          gc.nursery_free += size_of_allocation
-          return gc.nursery_free  
-    
+            # Yes => move nursery_free forward
+            gc.nursery_free += size_of_allocation
+            return gc.nursery_free
+
     # We did not exceed a sampling point and must do a minor collection, or
-    # we exceeded a sample point but we needed to do a minor collection anways
+    # we exceeded a sample point but we needed to do a minor collection anway
     minor_collection()
-    return gc.nursery_free 
+    return gc.nursery_free
 ```
 
 ## Why is the Overhead ‘low’
 
-The tight integration of sampling into the nursery's bump-pointer logic does
-add only slight overhead.
-
-Every time an allocation exceeds the `sample_point`, `collect_and_reserve` is
-called to sample over the size of the allocation (as described in previous
-section). The resulting overhead is directly controlled by the sampling rate,
-as the amount of samples is `size_of_allocation / sample_n_bytes`.
+The most important property of our approach is that the bump-pointer fast path
+is not changed at all. If sampling is turned off, the slow path in
+`collect_and_reserve` has three extra instructions for the if at the beginning,
+but are only a very small amount of overhead, compared to doing a minor
+collection.
 
+When sampling is on, the extra logic in `collect_and_reserve` gets executed.
+Every time an allocation exceeds the `sample_point`, `collect_and_reserve` will
+sample the Python functions currently executing. The resulting overhead is
+directly controlled by `sample_n_bytes`.
 After sampling, the `sample_point` and `nursery_top` must be set accordingly.
 This will be done once after sampling in `collect_and_reserve`.
-
-Setting a pointer like `sampling_point`, `nursery_top`, etc. due to sampling,
-are additions, subtractions, loads and stores on the lowest level. Those are
-not expensive on their own, but rather when excecuted very often.
-
 At some point a nursery collection will free the nursery and set the new
-`sample_point` afterwards. 
-
-The total number of times sampling overhead is introduced is:
-- Some pointer arithmetic and a call to VMProf's c-interface every time the
-  `sample_point` is exceeded
-- Some pointer arithmetic after a nursery collection
-- Some pointer arithmetic when enabling or disabling sampling (but that is of
-  course constant overhead)
+`sample_point` afterwards.
 
-That means that the overhead mostly depends on the sampling rate and the amount
-of memory allocated by the user program, as the combination of those two
+That means that the overhead mostly depends on the sampling rate and the rate
+at which the user program allocates memory, as the combination of those two
 factors determines the amount of samples.
 
 Since the sampling rate can be adjusted from as low as 64 Byte to a theoretical
@@ -324,7 +325,8 @@ Modified VMProf with allocation sampling support
 
 ## Future Work
 
-There are multiple points we’d like to address in the future.
+We have a bunch of ideas of features that could be added to VMProf in the
+future.
 
 One very important thing when it comes to profiling is the overhead. One idea
 on how to decrease the overhead per sample is, to not walk the entire stack
@@ -344,7 +346,7 @@ pypylog as we do for vmprof, but this could introduce more overhead. A better
 way of associating them, could be to record the `TSC` with each sample so we’d
 get an approximate alignment of logged events and samples.
 
-Another Idea would be extracting information about allocations from the GC,
+Another idea would be extracting information about allocations from the GC,
 e.g. type of object to be allocated, object size or even some statistics about
 lifetime (if possible). For example PyPy could also log jitted functions, so we
 can see what function got jitted at what time. Together with the sampled