[DPR2-1742] add status check #36
Conversation
| } | ||
| } | ||
| else -> { | ||
| Thread.sleep(WAIT_TIME) |
There was a problem hiding this comment.
Do we really want to be waiting for the query to finish? Each report could take up to 15 minutes, and a single lambda run is limited to 15 minutes. Even running these in parallel would be a bit dicey.
There was a problem hiding this comment.
yeah its only going to run as long as the time out above, which is currently set to max 120 secs but should probably be a lot less, this was really about capturing failures, which should hopefully happen quite quickly
There was a problem hiding this comment.
I mean in QUERY_STARTED ->, so its waiting for it to start but not necessarily to finish
There was a problem hiding this comment.
I've updated the timeout so it will finish 5 secs after the query has started. this will hopefully catch most of the initial errors
There was a problem hiding this comment.
Could we look at waiting for the query to be STARTED ?
There was a problem hiding this comment.
yes that what the logic does, but it waits based on the configurable timeout. Its currently 5 secs after the query has started, but you think remove the timeout all together or reduce it down further
There was a problem hiding this comment.
Ideally as the long term solution as discussed in our call we would want to have this lambda just executing the queries and storing the execution ID without waiting.
Then another lambda can pick up these stored execution ids and check the status periodically.
This is currently implementing all the logic in one lambda but as discussed previously the future state would be to split the lambdas up, so one does the searching / scheduling, then sends events to another lambda which can do the actually work of generating the dataset and checking the status, so there would be 1 lambda execution per dataset generation. Currently this lambda will timeout 5 secs after the query has started.