Expand Beam Python heap dump with process/native memory stats (#39244)

[uditjainstjis]

Addresses #39244

What

Expands the SDK worker heap dump (enabled by --experiments=enable_heap_dump) with a new memory_stats() section so operators can distinguish memory growth on the native (C) heap from Python-object allocations, and can spot native-heap fragmentation — the two things the issue asks for.

The dump previously contained only the guppy Python-object heap. memory_stats() adds:

• Process memory — peak RSS (ru_maxrss, unit-corrected: bytes on macOS/BSD, KiB on Linux) and, on Linux, current RSS from /proc/self/statm.
• Python allocations — sys.getallocatedblocks() and per-generation gc collection stats.
• glibc malloc (native heap) — mallinfo2() arena/hblkhd/uordblks/fordblks/keepcost, plus a fordblks/(arena+hblkhd) fragmentation ratio. fordblks (free space the allocator retains rather than returning to the OS) growing relative to uordblks is the fragmentation signal referenced in the issue.

mallinfo2 is used rather than the legacy mallinfo, whose int fields overflow past 2GB and would misreport exactly when memory is the concern; if the symbol is absent it reports unavailable (needs glibc >= 2.33). No new dependencies — stdlib only (resource, gc, sys, ctypes, /proc).

Every collector degrades gracefully and never raises when a source is unavailable, and memory_stats() is emitted even when guppy is not importable.

Example output (Linux, glibc 2.39)

========== MEMORY STATS ==========
--- Process memory ---
peak RSS (ru_maxrss): 10485760 bytes
current RSS (/proc/self/statm): 10948608 bytes
--- Python allocations ---
sys.getallocatedblocks: 13689
gc.get_count (gen0, gen1, gen2): (442, 6, 0)
gc gen0: collections=6 collected=31 uncollectable=0
gc gen1: collections=0 collected=0 uncollectable=0
gc gen2: collections=0 collected=0 uncollectable=0
--- glibc malloc (native heap) ---
arena (non-mmapped bytes from sbrk): 1052672
hblkhd (mmapped bytes): 401408
uordblks (in-use bytes): 950368
fordblks (free bytes retained by allocator): 102304
keepcost (releasable top-most free bytes): 99808
fragmentation (fordblks / (arena + hblkhd)): 7.04%
==============================

Testing

• New unit tests in worker_status_test.py: section presence, memory stats emitted without guppy, and graceful degradation of the glibc section on non-glibc platforms.
• Full worker_status_test.py passes locally (9 passed).
• The mallinfo2 ctypes path (by-value struct restype) was executed on real Linux glibc 2.39 to confirm the struct layout and that it does not segfault; values are internally consistent (uordblks + fordblks == arena).
• yapf==0.43.0 clean, pylint 10.00/10.

This is a first cut; happy to also wire the section outside the guppy-gated dump or add malloc_info() XML if maintainers prefer.

• Update CHANGES.md with noteworthy changes.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[github-actions]

Assigning reviewers:

R: @shunping for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).