Disk force bundling #235
Open
Disk force bundling #235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…operations in a certain time and only does the force en-bloc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR intends to add a 'disk-force-bundling' mode to the FileSystemRepository as a performance optimization.
In concurrent scenarios, when there are a lot of concurrent transactions, in some of our usecases we have seen substantial contention on FileSystemRepository put operations, the bottleneck (in our cases at least) being the capability of the disks to serve the force operation (channel.force()). This is particularly true if we use spinning disks (but also with SSDs). By collecting all pending disk-forces in a given timeframe we can reduce the load on the disks and increase throughput substantially (in our cases from a couple of 100 tx/s to a couple of 1000 tx/s).
The general idea of the implementation is that the caller threads not directly wait on the previously synchronized put/writeToFile operations but instead get back a latch to wait on. The put/write operations put the operational data onto a queue that is then processed by a separate Thread in a loop, collecting all the writes and only doing a single force per 'bundle' of writes.
The implementation was done in a way that this 'bundling mode' is optional, per default the previous behavior is enabled.
The testcase is added on the JDBC module as it's a kind of integration test, it should demonstrate functional equivalence of the new mode to the previous behavior and also serve as indicator of the throughput improvements that can be expected.
Looking forward to feedback!
Regards, Leo
PS: All previous unit tests except PooledAlarmTimerTestJUnit are still working, but on my end at least PooledAlarmTimerTestJUnit was already not working in the base 6.0.1-SNAPSHOT version.