Problem with sort-depth?

[mira523]

Hi, very new to GRSISort and I'm running into an issue when I try to sort some of the files from the experiment I'm working on. When I run GRSISort on the .mid files using the --recommended settings flag, I get a warning at the end of the sorting process stating "52 fragments were out of order, maximum entry difference was 44291381! Please consider increasing the sort depth with --sort-depth=44291381" (with slightly different numbers for different files). If I then try to sort the file again with the recommended sort-depth, the sorting process ends up being killed partway through, before the sorting is complete.

I am wondering, i) what is sort-depth?, ii) why might my files be having an issue with the sort-depth?, and iii) why using the recommended sort-depth would cause the sorting to be killed partway through?

Thanks in advance!

[VinzenzBildstein]

Hi, welcome to using GRSISort.

That is a very common thing to encounter I think, so I will try to give an explanation here. The way the data is being collected from the digitizers we can not be guaranteed that the different hits from the digitizers (called fragments in GRSISort) are sorted in time. So before we start building events from those hits, we need to sort them by time.

This is done using a c++ STL container that does the ordering for us (by whatever criterium is appropriate for the data set, can be sorted by CFD corrected time, LED time, trigger ID, or not sorted at all). Whenever that container reaches a specific size (default is 10000) we start building events by taking the first hit in time in the container and checking wether we can add it do the event we are currently building (depending on conditions on the CFD corrected time, LED time, or trigger ID).

But if the time of this hit is smaller (before) the time of the event, than this means the size of the container we use was selected to be too small to sort ALL hits correctly in time. This hit is then written to the OutOfOrder tree, and the information is added to the sorting diagnostics class. That class will at the end print out the warning you see. The "recommended" sorting depth it shows is a very conservative attempt to estimate how big the container would have to be to get all hits time ordered.

The sorting depth the code suggests is often way too large, which can lead to problems because the size of the container doesn't fit into the memory of the computer you are using to sort the data. If that computer runs out of memory the operating system will start trying to free memory by killing processes that use a lot of it, i.e. the operating system will kill GRSISort. I'm not sure what the exact size of one fragment is, but sizeof reports 208 bytes, so the size recommended in the example you gave would require 8.6 GiB of memory (plus all the other memory the OS and GRSISort need).

Now getting these warning is very common, and it essentially requires your judgment on whether it makes sense to increase the sorting depth (and to what size), or not. There are a number of things to consider:

• How many fragments were out of order compared to the overall number of fragments (you can find that by either looking at the size of the fragment tree, or at the end of the sort how many good fragment were written. In one example sort I have here 33 fragments were out of order out of 48M fragments, so less than one per million. For most types of experiments this would be a loss of statistics/hits that can easily be ignored.
• You can further investigate by looking at the TSortingDiagnostics object on the analysis root file (called "SortingDiagnostics"), e.g by using SortingDiagnostics->FragmentsOutOfTimeOrder() in a GRSISort session. This will bring out an unordered map of the time of the event as key, and a pair of the time difference and the entry number of the hit that was out of order.
• An other thing you can check is the TTree object called "OutOfOrderTree" which contains all fragments that were out of order (i.e. all fragments you are missing in the analysis). You can then check these fragments to see if there is anything that might suggest some issue for your analysis (like all coming from one channel or something like that).

In most cases (unless you are analysing a high precision experiment or there are a lot of fragments out of order), you can ignore the warning and just proceed analysing the data.

I hope this answered any question you (and I'm guessing a lot of other people) have about this, let me know if there is more I can explain about it.

[mira523]

Yes, this helps a ton, thank you!