Unexpected Auto-restart Video Job currently processing

[lovgrandma]

I have actually been building a production video upload ETL pipeline with this repository and its been working fine. Just wanted to share something I had stooped upon as I'm not sure if this is intended functionality.

I have a pipeline which transcodes video, audio, subtitles, thumbtracks to build a mpd and hls manifest and database record of all those files after theyre put onto S3. I have been transcoding multiple videos non stop anywhere from 15 seconds to 30 minutes. Working fine. But I did realize some very strange behaviour. I put an episode of Succession (hour and 3 minutes long) through the pipeline which takes long. Its ffmpeg transcoding this high resolution video from resolutions 1080p and down so it could take 45 minutes or more. Half way through the job was being started again on another thread 😕 so I was seeing these errors like the original uploaded video was being used by another process, cant start another ffmpeg job on the same file right.

So I just added an extra check to grab the record from the db and not restart if its already started. So two things:

• Is this intended functionality? For the job to start again on another thread if the original one started hasnt been completed after an hour or so?
• For anyone that is experiencing this, make sure to initially make a record on your db before you provision the job with "waiting" or something of the sort and then update it to signify it is in progress when the job has actually been picked up. If it starts again randomly you can just check the db and return on the job function call since its already running. This fixed my issue completely. But again, maybe there is a setting to prevent this.

[hibiken]

@lovgrandma Thank you for opening this discussion.

From the description above, it seems that you are using the default timeout of 30m (See the default behavior described here). If you don't specify Deadline or Timeout option, the task will have timeout of 30m and will fail when it reaches the deadline (deadline is either the absolute value specified by the Deadline or time.Now().Add(timeout) where timeout is the value specified by Timeout and time.Now() is the time when task is delivered to your Handler).

It sounds like in your use case, it'd be sufficient to set a higher timeout value (e.g. 2h).

Let me know if that you have follow up questions!

[lovgrandma]

This is clear. I updated my code to enqueue the task like so:
jobClient.Enqueue(task, asynq.Timeout(5 * time.Hour))

Appreciate the quick response.