multithread error

[katherine-shiqi]

Hi group,


I tried to use thread parameter in the pulp function, but it sometimes would give me error of pulp.apis.core.PulpSolverError: Pulp: Error while trying to execute, use msg=True for more details/home/XXXX/.conda/envs/zone_env/lib/python3.7/site-packages/pulp/apis/../solverdir/cbc/linux/64/cbc

If I removed the thread (thread=False), then the algorithm would give me feasible results.



The problem I have is a mixed integer programming:

x = pulp.LpVariable.dicts("x", ((i, j, k) for i, j, k in for_all_ijk), cat='Continuous') y = pulp.LpVariable.dicts("y", ((i, j) for i, j in for_all_ij), cat='Binary')


##1. With thread=False
Result - Optimal solution found

Objective value: 28145.75399588
Enumerated nodes: 1192
Total iterations: 35023
Time (CPU seconds): 268.02
Time (Wallclock seconds): 269.49

Option for printingOptions changed from normal to all
Total time (CPU seconds): 268.85 (Wallclock seconds): 270.35

##2. With thread = 64

Cbc0030I Thread 29 used 11 times, waiting to start 89.512264, 57 locks, 0.080677509 locked, 0.12807012 waiting for locks
Cbc0030I Thread 30 used 10 times, waiting to start 93.806582, 49 locks, 0.053248167 locked, 0.086375237 waiting for locks
Cbc0030I Thread 31 used 9 times, waiting to start 91.730973, 41 locks, 0.11667466 locked, 0.12240982 waiting for locks
Cbc0030I Thread 32 used 7 times, waiting to start 89.428064, 35 locks, 0.059940338 locked, 0.044867516 waiting for locks
Cbc0030I Thread 33 used 10 times, waiting to start 91.933537, 46 locks, 0.060267925 locked, 0.089924097 waiting for locks
Cbc0030I Thread 34 used 7 times, waiting to start 90.59779, 39 locks, 0.082733631 locked, 0.10458612 waiting for locks
Cbc0030I Thread 35 used 7 times, waiting to start 97.596597, 32 locks, 0.08664012 locked, 0.086414099 waiting for locks
Cbc0030I Thread 36 used 8 times, waiting to start 97.017808, 37 locks, 0.14031792 locked, 0.10879588 waiting for locks
Cbc0030I Thread 37 used 6 times, waiting to start 95.043709, 29 locks, 0.088786125 locked, 0.11181641 waiting for locks
Cbc0030I Thread 38 used 7 times, waiting to start 98.152656, 36 locks, 0.081135988 locked, 0.15073299 waiting for locks
Cbc0030I Thread 39 used 5 times, waiting to start 101.4105, 24 locks, 0.040215254 locked, 0.1369307 waiting for locks
Cbc0030I Thread 40 used 4 times, waiting to start 99.877512, 18 locks, 0.040210247 locked, 0.031095982 waiting for locks
Cbc0030I Thread 41 used 6 timultiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/uwv999/zone_opt/python_scripts/zone_opt.py", line 31, in zone_opt
min_diam=min_diam)
File "/home/uwv999/zone_opt/python_scripts/optimizer.py", line 99, in optimize_pulp
model.solve(pulp.PULP_CBC_CMD(fracGap=0.0001,threads=64,msg=True)) #To specify the accuracy of the solver, you can use the fracGap argument of the selected solver.
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/site-packages/pulp/pulp.py", line 1737, in solve
status = solver.actualSolve(self, **kwargs)
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/site-packages/pulp/apis/coin_api.py", line 101, in actualSolve
return self.solve_CBC(lp, **kwargs)
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/site-packages/pulp/apis/coin_api.py", line 155, in solve_CBC
self.path)
pulp.apis.core.PulpSolverError: Pulp: Error while trying to execute, use msg=True for more details/home/uwv999/.conda/envs/zone_env/lib/python3.7/site-packages/pulp/apis/../solverdir/cbc/linux/64/cbc
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "opt_driver.py", line 177, in
for terr_id in iterable])).reset_index()
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/uwv999/.conda/envs/zone_env/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
pulp.apis.core.PulpSolverError: Pulp: Error while trying to execute, use msg=True for more details/home/uwv999/.conda/envs/zone_env/lib/python3.7/site-packages/pulp/apis/../solverdir/cbc/linux/64/cbc

[pchtsp]

Maybe we should add a warning on this. The thread parameter for CBC does not work with the CBC that comes with PuLP. If I remember correctly it was something related to how it was compiled but I'm not 100% sure. Maybe @tkralphs or @stumitchell can shed some light on this. Here is a link with an old thread about it: https://groups.google.com/g/pulp-or-discuss/c/IKE--kT6c5U/m/Y0OUJCqoBQAJ

[katherine-shiqi]

Thank you @pchtsp for your information. The old thread link you provided doesn't work, showing Content unavailable error.
Basically, you are saying that the threading for CBC that comes with PuLP doesn't work so whether I use False or 16 or 64 doesn't matter? What are the solvers that would have thread parameter working with PuLP? Is there a documentation on it?

Thanks,
Katherine

[stumitchell]

Hi going back a bit here, but I don't think the windows builds of cbc that we use have multithreading enabled as cbc depends on some linux library support (lpthread) to work. I would go to the cbc project https://github.com/coin-or/Cbc and ask for a windows build that does multi threading. and if they have one you can use it with pulp. And we can probably add it to the project. Stu Stuart Mitchell PhD Engineering Science Extraordinary Freelance Programmer and Optimisation Guru www.stuartmitchell.com

…

On Tue, Apr 6, 2021 at 2:47 AM Katherine Li ***@***.***> wrote: Thank you @pchtsp <https://github.com/pchtsp> for your information. The old thread link you provided doesn't work, showing Content unavailable error. Basically, you are saying that the threading for CBC that comes with PuLP doesn't work so whether I use False or 16 or 64 doesn't matter? What are the solvers that would have thread parameter working with PuLP? Is there a documentation on it? Thanks, Katherine — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFUIWJ4UIUZZQUSRCXTAZ3THHEOHANCNFSM42GAZK3A> .

[katherine-shiqi]

Hi Stu @stumitchell I am using the linux server to run the script so based on your description, it seems that linux builds of cbc that has multithreading enabled? if so, do you know the reason for the above errors? thank you!!

[stumitchell]

Hi I have read back to your original issue, Multi threading sometmes brings up issues in the cbc code. I have submitted a pr to update the cbc versions of pulp. are you able to try using those executables?

Can you see if this fixes the behaviour

Stu

[tkralphs]

If you're seeing those lines of output, you are definitely using a multi-threaded Cbc. I doubt the upgrade to 2.10 will change the behavior. 64 threads is pretty ambitious unless you have a very beefy server. How many cores do you have? How much memory? Have you tried running with fewer threads? Parallel branch-and-bound does not scale that well. It's doubtful you'll have good performance with this many threads.

It's a little hard to interpret the error. It's coming from the multiprocessing package of Python, but I'm a little confused by that, since it is Cbc that is doing the threading, not PuLP. Are you running multiple PuLP models in parallel, as well as running Cbc in parallel? That's what it sort of looks like. If so, you're probably running out of resources on some level, but I don't really know where.

If you want to see if Cbc is the problem, you should just run the instance through Cbc on the command line outside of PuLP and see if the behavior is the same.

[stumitchell]

Good spotting @tkralphs it appears that @katherine-shiqi is solving multiple pulp problems at the same time as well as using multiple threads in cbc.
I have never tried this myself and am not sure what may happen.

[katherine-shiqi]

Thank you team! I have commented out the multiprocessing pulp code but still sometimes will encounter multithreading issue. I have 96 cores and 383 GB memories in the server. I have tried 16 threads and it worked fine but when I tried 64 threads it failed again. I have monitored the memory utilization and it didn't exceed.
Also, i found out it just utilized one core instead of 96 cores. Is the multithreading in CBC set to run in single core only?
Thank you,
Katherine

[katherine-shiqi]

Hi @stumitchell After trying your newer cbc version, 64 threading works! Thank you!

[katherine-shiqi]

With the update, it also utilized multi-core instead of single core as well! I will use the updated version going forward. Thank you for helping team!!

[katherine-shiqi]

If you can approve the PR, it would be easier for us to utilize the latest version of PuLP. Thank you!