particles refinement stuck in the middle of running

[ariel234156]

Hello,
I have a problem with the particle refinement step. When I use up to 4 tomograms, the process runs well, and I get a result.
but when I use a bigger dataset, the process runs slow and gets stuck after 3 tomograms without error.
I have more than 6Tb in my nvme, so I don't think that happened because of a lack of storage.

My computer is relatively strong, 127v with 750Gb of RAM.

[abartesaghi]

Can you provide more details about the resource settings you are using for this? For example, how many Threads per task and Memory per task are you using? Also, is the merge job running at all? If so, what resources is it using and does it give any errors?

[ariel234156]

I used 100 Threads per task and 500GB of RAM
you can see the details in the photo:

There is no error, and in the "htop" command, it looks like the process continues, but if I check the log file, I can see no progress in the last 12 hours.

[abartesaghi]

With these settings, you are really only running one tilt-series at a time and the merge job will only run once the last tilt-series has finished processing. This means that the jobs should not be interfering with each other at all. Can you share the full log for the job that takes longer to run than usual? Also, the merge and launch tasks are usually lightweight processes and you can get away using more modest resources, e.g., 6 threads/24GB for the merge task and 1 thread/4GB for the launch task.

Note that the "streaming" log will only work for the first tilt-series in your dataset. The logs for the rest of the tilt-series will only show up once each process finishes running. Could that be the reason why you think your logs are not updating?

[ariel234156]

I looked into the output folder to check if the process was stuck. The process always stuck in this position-

i abort the running after 6h in same position at log file, i checked also difreenet process to be sure the log file updated lively.
The weird thing is that if I use the same tomograms but in a 4-tomogram dataset, it runs well, but it's stuck in the third tomogram with the bigger dataset.

[abartesaghi]

Particle extraction is taking 6 minutes to run, which tells me that this tomogram probably has a lot of particles (or the box size is very big). Depending on what type of search you are doing, refinement may take several hours and the progress bar will not update until the first batch of alignments is finished, which could well take several hours. That, however, does not explain why the same tomogram works fine when used as part of a smaller dataset. If you see all 99 copies of csp running and taking up 100% of the CPU, that means the program is running correctly, and it probably just needs more time to finish. Could the extraction or search parameters be different between the the small and large runs?

[ariel234156]

I tried to reduce my tomogram size and the box size, and now the process running well,
thank you.