Snapshotter does not account for dynamic memory scaling (e.g., K8s burstable QoS)

[erandagan]

Description

In environments like Kubernetes, workloads often scale vertically. A Pod may start with a specific memory request (e.g., 1GiB) but be allowed to burst up to a limit (e.g., 8GiB) if cluster resources permit.

Currently, Crawlee's Snapshotter captures the memory limit only once during initialization. If the runtime allocates more memory to the process later, the Snapshotter remains "locked" to the initial baseline, leading to inaccurate autoscaling decisions and under-utilization of available resources.

Root Cause

• max_memory_size is set once in the constructor.

• Subsequent snapshot creation uses this cached value rather than querying the current system state.

Proposed Solution

Modify _snapshot_memory to fetch the max_memory_size value dynamically when a memory event is processed, ensuring the autoscaler respects the current ceiling.