subprocess.POPEN waits forever

[SiqiLii]

I was running eureka.py and have output:

python eureka.py env=shadow_hand sample=4 iteration=2 model=gpt-4-0314
[2023-10-24 23:13:24,730][root][INFO] - Workspace: /home/123/isaacgym/python/Eureka/eureka/outputs/eureka/2023-10-24_23-13-24
[2023-10-24 23:13:24,730][root][INFO] - Project Root: /home/123/isaacgym/python/Eureka/eureka
[2023-10-24 23:13:24,730][root][INFO] - Using LLM: gpt-4-0314
[2023-10-24 23:13:24,730][root][INFO] - Task: ShadowHand
[2023-10-24 23:13:24,730][root][INFO] - Task description: to make the shadow hand spin the object to a target orientation
[2023-10-24 23:13:24,758][root][INFO] - Iteration 0: Generating 4 samples with gpt-4-0314
[2023-10-24 23:14:01,699][root][INFO] - Iteration 0: Prompt Tokens: 1735, Completion Tokens: 1254, Total Tokens: 2989
[2023-10-24 23:14:01,699][root][INFO] - Iteration 0: Processing Code Run 0
[2023-10-24 23:14:13,181][root][INFO] - Iteration 0: Code Run 0 successfully training!
[2023-10-24 23:14:13,181][root][INFO] - Iteration 0: Processing Code Run 1
[2023-10-24 23:14:26,009][root][INFO] - Iteration 0: Code Run 1 successfully training!
[2023-10-24 23:14:26,009][root][INFO] - Iteration 0: Processing Code Run 2
[2023-10-24 23:14:26,009][root][INFO] - Iteration 0: Code Run 2 cannot parse function signature!
[2023-10-24 23:14:26,009][root][INFO] - Iteration 0: Processing Code Run 3
[2023-10-24 23:14:40,169][root][INFO] - Iteration 0: Code Run 3 successfully training!

Then the process got stuck at rl.communication() and didn't move forward anymore

with Traceback:
Traceback (most recent call last):
File "eureka.py", line 397, in
main()
File "/home/.local/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/homer/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/home/.local/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 119, in run
ret = run_job(
File "/home/.local/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "eureka.py", line 211, in main
rl_run.communicate()
File "/home/anaconda3/envs/HarryVae/lib/python3.8/subprocess.py", line 1020, in communicate
self.wait()
File "/home/anaconda3/envs/HarryVae/lib/python3.8/subprocess.py", line 1083, in wait
return self._wait(timeout=timeout)
File "/home/anaconda3/envs/HarryVae/lib/python3.8/subprocess.py", line 1822, in _wait
(pid, sts) = self._try_wait(0)
File "/home/anaconda3/envs/HarryVae/lib/python3.8/subprocess.py", line 1780, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)

[ggao22]

Hi, were you eventually able to resolve this issue?