Support long running queries

[tim-hm]

Background

Some aggregations/queries are long running, and before their completion, the http connection will timeout (e.g. ALB defaults to 60s). In these cases, there is no endpoint to retrieve the output after the connection is terminated.

As a first step to solving this issue, we should support adding indexes to increase query performance (re #102), however, some queries may still take longer than 60s to return.

To address this, we want to allow the queuing query execution in one request and retrieving the results in another. Below is a starting point straw man implementation.

Proposal

1. Leave the existing /api/v1/queries/execute as is.
2. Add an endpoint to queue a query for execution:

   # Request
   POST /api/v1/queries/queue
   
   {
     "id": "<query-id>",
     "variables": { ... }
   }
   
   # Response:
   {
       "data": "<job-id>"
   }
   
   

3. Now the client should periodically check for the output:

   # Request
   GET /api/v1/queries/queue/:job-id
   
   # Pending response:
   {
     "data": {
       "status": "pending"
     }
   }
   
   # Finished response:
   {
     "data": {
       "status": "success" | "failure",
       "result": { ... }
     }
   }
   

Notes

• Completed jobs would have a 1 hour TTL