Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queued element not processed #65

Open
thorin8k opened this issue Nov 18, 2019 · 3 comments
Open

Queued element not processed #65

thorin8k opened this issue Nov 18, 2019 · 3 comments

Comments

@thorin8k
Copy link

thorin8k commented Nov 18, 2019

I have been using this job system since a while and i recently discovered one bug that occurs when the app restarts while an elements its been processing.

If the queue starts with some element 'locked' it keeps the last one without processing it until restat.

Steps to reproduce

  • Create a Simple queue in which every element takes a decent time to process
  • Add elements to the queue
  • Stop the script while processing one item
  • The element is now in the database with a lock value
  • Start the application again and wait for all tasks to complete.

Result

  • The last added item is still in the queue.

Even if you add some element (without restarting) the queue always keep the last element

Expected Result

  • All elements to be processed.

Aditional Data
In order to reproduce this the queue needs to be persisted. I have been using the postgresql store.

Workaround
Remove the values of 'lock' field before initializing the queue.

@turbopasi
Copy link

I encountered the same bug while using my own local store following the better-queue-memory example. When restarting the app, the last item in queue will be stuck and not be processed. @thorin8k do you have more details about what I can do to work around this bug ?

@thorin8k
Copy link
Author

thorin8k commented Jun 3, 2020

I encountered the same bug while using my own local store following the better-queue-memory example. When restarting the app, the last item in queue will be stuck and not be processed. @thorin8k do you have more details about what I can do to work around this bug ?

I was using better-queue with postgreSQL so my workaround was to empty the field 'lock' after every start. With this i ensured that any pending job gets reprocessed after the app starts.

Finally, as the app was getting bigger, we moved from better-queue to RabbitMQ so i haven't any additional info regarding the issue, sorry.

@AdelUnito
Copy link

I have the same issue even when failed tasks get retried they keep the old lock, I'll test your workaround

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants