Improve performance with reduced execution time and lower CPU/system resource usage

[abraunegg]

• Use global variables for logging verbose|debug logging state
• Only perform debug logging steps including calculations and/or output if debugging is actually enabled
• Optimise code execution and reduce resource usage by ~12% in real time, ~65% in CPU time, and ~87% in system time.

Calculations:

Command:

for i in {1..5}; do echo "Run #$i"; time ./onedrive -s --resync --resync-auth; echo ""; done

This PR shows significant improvements over 'master':

• The real execution time is about 12.5% faster, meaning the overall process completes quicker.
• The user time has decreased by nearly 65%, indicating that the new code base is more efficient in CPU utilisation.
• The sys time has dropped by approximately 87%, which suggests the new code has reduced system overhead.

[abraunegg]

Master Flamegraph

PR Flamegraph - debug only fixed

PR Flamegraph - verbose also fixed

[abraunegg]

@bpozdena

Is there any opportunity for you to run your tests / use this PR for some feedback please ?

[bpozdena]

@bpozdena

Is there any opportunity for you to run your tests / use this PR for some feedback please ?

I probably can, but I don't expect much of a difference in the actual sync cycle duration as I'm always heavily limited by the Graph API rate limit.

Will something like a graph of the CPU utilization for the onedrive process be enough for comparison?

[abraunegg]

@bpozdena
Is there any opportunity for you to run your tests / use this PR for some feedback please ?

I probably can, but I don't expect much of a difference in the actual sync cycle duration as I'm always heavily limited by the Graph API rate limit.

Will something like a graph of the CPU utilization for the onedrive process be enough for comparison?

Anything you can do would be greatly appreciated

[bpozdena]

I noticed my system became very sluggish for several seconds with the new build. It's because I'm using 16 onedrive threads and most of them had very high CPU utilization. But I never noticed this behavior with the previous builds.

Unfortunately my original test did not monitor all CPU cores, so I have to re-do it again, which will take time.

Also the new build stopped before completing the sync with the bellow message without any additional details.

ERROR: Microsoft OneDrive API returned an error with the following message:
  Error Message:    HTTP request returned status code 400 (Bad Request)
  Error Reason:     Invalid request
  Error Code:       invalidRequest
  Calling Function: createLocalPathStructure()


Attempting to perform a database vacuum to optimise database
Database vacuum is complete

I will investigate all issues further, it just takes time because each resync cycle takes me over an hour to complete.

[abraunegg]

@bpozdena

I noticed my system became very sluggish for several seconds with the new build. It's because I'm using 16 onedrive threads and most of them had very high CPU utilization. But I never noticed this behavior with the previous builds.

Very ood I would say - as #2880 / 'master' - as all that PR does is less execution when not using --verbose or debug (double --verbose) ... essentially having less processing .. so I would say something very odd with that.

Also the new build stopped before completing the sync with the bellow message without any additional details.

If you can reproduce that at all - that would be good - i will go back and look at the change and see if I miss-edited something with that function specifically.

[bpozdena]

• The total re-sync time is always the same as it's mainly limited by the API response times.
• The CPU utilization is noticeably improved in the non-verbose mode from average CPU utilization of 61% to 46%.
• Unfortunately, there is a noticeable regression in the verbose mode (-v) from Average CPU utilization of 52% to 72%.

Non-verbose

v2.5.2 release - non-verbose resync with 12 threads:
Duration: 86 minutes
Average CPU utilization: 48%
Average CPU (idle excluded): 61%

v2.5.2-6-g59007d5 - non-verbose resync with 12 threads:
Duration: 83 minutes
Average CPU utilization: 36%
Average CPU (idle excluded): 46%

Verbose (-v)

v2.5.2 release - verbose resync with 12 threads:
Duration: 84 minutes
Average CPU utilization: 40%
Average CPU (idle excluded): 52%

v2.5.2-6-g59007d5 - verbose resync with 12 threads:
Duration: 86 minutes
Average CPU : 57%
Average CPU (idle excluded): 72%

[abraunegg]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.