Memory leak in HWP PMX agent

[BrianJKoopman]

Ever since maintenance last week it seems there is now a memory leak in the HWP PMX agent. During maintenance we updated from v0.5.2 to v0.5.2-30-ga569783-dev. This still needs investigation.

Here's a plot of docker memory usage in percent for daq-satp2-ocs-hwp-pmx-1:

[BrianJKoopman]

This may be a wider underlying problem as the HWP gripper is also running away with memory usage:

Usage for the gripper agent on satp2 as a percentage again. (Max is 7.751 GiB, so this is ~1.7G usage.)

Logs are filled with this message every second:

2025-02-18T14:56:58+0000 Have not received 'ok' in 5.00 minutes. Issuing shutdown
2025-02-18T14:56:58+0000 start called for shutdown
2025-02-18T14:56:58+0000 shutdown:408349 Status is now "starting".
2025-02-18T14:56:58+0000 shutdown:408349 Status is now "running".
2025-02-18T14:56:58+0000 Enabling shutdown-mode, which will prevent grip operations from being performed
2025-02-18T14:56:58+0000 shutdown:408349 Shutdown completed
2025-02-18T14:56:58+0000 shutdown:408349 Status is now "done".
2025-02-18T14:56:59+0000 Have not received 'ok' in 5.00 minutes. Issuing shutdown
2025-02-18T14:56:59+0000 start called for shutdown
2025-02-18T14:56:59+0000 shutdown:408350 Status is now "starting".
2025-02-18T14:56:59+0000 shutdown:408350 Status is now "running".
2025-02-18T14:56:59+0000 Enabling shutdown-mode, which will prevent grip operations from being performed
2025-02-18T14:56:59+0000 shutdown:408350 Shutdown completed
2025-02-18T14:56:59+0000 shutdown:408350 Status is now "done".
2025-02-18T14:57:00+0000 Have not received 'ok' in 5.00 minutes. Issuing shutdown
2025-02-18T14:57:00+0000 start called for shutdown
2025-02-18T14:57:00+0000 shutdown:408351 Status is now "starting".
2025-02-18T14:57:00+0000 shutdown:408351 Status is now "running".
2025-02-18T14:57:00+0000 Enabling shutdown-mode, which will prevent grip operations from being performed
2025-02-18T14:57:00+0000 shutdown:408351 Shutdown completed
2025-02-18T14:57:00+0000 shutdown:408351 Status is now "done".