New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How the library works and is clustering supported? #62

Open

ngocdaothanh opened this issue Oct 1, 2019 · 0 comments

ngocdaothanh commented Oct 1, 2019 •

edited

Loading

Please update the README:

To outline how the library works.
To clarify if clustering is supported or not.

After reading the source code:

I guess clustering (running tasks on many machines) is not supported yet, because:

On startup, all running tasks are forcefully resumed. The code doesn't care that the tasks may be being taken by other machines.
When taking new tasks, the code only care about newly inserted tasks. It doesn't care about the tasks that have been marked as taken by other machines, but other machines may have died.

To support clustering with "at least once" semantic, these little modifications can be made ("little" compared to the existing great work):

Add visibility timeout column to the tasks DB table: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
At startup and when taking new tasks, use the visibility timeout to figure out if a machine should take over tasks that have been marked as taken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment