Integrate some part of fractal-demos

[tcompa]

Related to fractal-analytics-platform/fractal-demos#37.

Proposal: let's set up one more container with fractal-client, and let's have it as ready as possible to run example 01 from fractal-demos. Ideally all should be automated, meaning that make run should trigger a bunch of operations that are over only once example 01 is over.

Note that there will be a few expected gotcha on the way: e.g. paths need to be updated to something in /home/fractal-share, and we should wait for task collection to be over (through a script that checks every N seconds), ...

Longer-term perspective: this could represent one way to automatically re-run a bunch of fractal-demos examples. TBD what kind of checks could be included in the output (either just successful-execution assertions, or something more related to the output data - ref fractal-analytics-platform/fractal-demos#37).

[tcompa]

A broader review, planned with @mfranzon.

Versions

Expose env variables for versions (all defaulting to latest pypi version, if not specified) of

• fractal-server
• factal-tasks-core
• fractal-client
• fractal-web (latest release on github)
• fractal-demos (maybe either latest on github or specific commit - to be used with git checkout)

Each variable gets used in the right container.

Server container and DB container

• Use versions
• Make volumes non persistent.

Web client

• Use versions

demos container

• Use versions (both in Dockerfile and run_all_demos.sh)
• Move as many statements as possible from run_all_demos.sh to Dockerfile.
• Pip install a minimal set of requirements for check_result.py
• Split run_all_demos.sh in setup (up to task collection is over), example01, .. -> and/or collect all in a single script..
• Add batch option for workflow-apply command fractal-client#552
• Catch exit code of run_example and log its state somewhere
• After run_example.sh, add a new layer that waits for when job is over
• Catch job status and log it somewhere
• Add an example Python script for checking output, based on Add logic to validate the outcome of the examples fractal-demos#37
• Run check_results.py and log its output somewhere

Expected: a log file like

example 01
API calls -> success (or failed, here is the log..)
Job status -> done (or failed, here is the log..)
check-results -> success (or failed, here is the log..)

github action

• Add all of this as a github action (without setting versions, in the first version), and use workflow_dispatch

[tcompa]

Here is a proposal about versions:

For fractal-client (and the same holds for fractal-server) we may have one of the following env variable: FRACTAL_CLIENT_RELEASE_VERSION or FRACTAL_CLIENT_GIT_VERSION. The logic could be like:

1. If they are both unset, then we use pip install fractal-client.
2. If they are both set, we fail.
3. If FRACTAL_CLIENT_RELEASE_VERSION is set, then we use it as part of a pip install command.
4. If FRACTAL_CLIENT_GIT_VERSION is set (this could be a commit hash, or a tag, or a branch), then we go through git-clone and then git-checkout.

For fractal-tasks-core, we have fewer cases:

1. If FRACTAL_TASKS_CORE_RELEASE_VERSION is set, we pass it to the fractal-task-collect command.
2. If it is not set, we do not include it in the fractal-task-collect command.

For fractal-web, we could stick with FRACTAL_WEB_GIT_VERSION (a commit hash, a branch, or a tag):

1. If it's unset, we git-clone and stick with main
2. If it's set, we git-clone and then git-checkout.

[tcompa]

Add all of this as a github action (without setting versions, in the first version), and use workflow_dispatch

For this, we would need some logic to shutdown everything after the demos service is over.

Runs like this one are actually successful, see logs below. But then remained hanging.

2023-10-03T11:29:07.0375449Z fractal-server     | INFO:     172.18.0.4:58454 - "POST /api/v1/project/1/dataset/ HTTP/1.1" 201 Created
2023-10-03T11:29:07.0783398Z fractal-demos      | DS_OUT_ID: 2
2023-10-03T11:29:07.2770501Z fractal-server     | INFO:     127.0.0.1:37454 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:07.3675899Z fractal-server     | INFO:     172.18.0.4:58470 - "PATCH /api/v1/project/1/dataset/2 HTTP/1.1" 200 OK
2023-10-03T11:29:07.3713534Z fractal-demos      | {
2023-10-03T11:29:07.3714535Z fractal-demos      |   "name": "output-ds-cardiac-tiny",
2023-10-03T11:29:07.3714921Z fractal-demos      |   "type": "zarr",
2023-10-03T11:29:07.3715247Z fractal-demos      |   "meta": {},
2023-10-03T11:29:07.3726039Z fractal-demos      |   "read_only": false,
2023-10-03T11:29:07.3754438Z fractal-demos      |   "id": 2,
2023-10-03T11:29:07.3755046Z fractal-demos      |   "resource_list": [],
2023-10-03T11:29:07.3755557Z fractal-demos      |   "project_id": 1
2023-10-03T11:29:07.3758084Z fractal-demos      | }
2023-10-03T11:29:07.6938263Z fractal-server     | INFO:     172.18.0.4:58472 - "POST /api/v1/project/1/dataset/2/resource/ HTTP/1.1" 201 Created
2023-10-03T11:29:07.6971165Z fractal-demos      | {
2023-10-03T11:29:07.6971758Z fractal-demos      |   "path": "/home/fractal_share/fractal-demos/examples/01_cardio_tiny_dataset/output_cardiac-tiny",
2023-10-03T11:29:07.6972231Z fractal-demos      |   "id": 2,
2023-10-03T11:29:07.6972549Z fractal-demos      |   "dataset_id": 2
2023-10-03T11:29:07.6972855Z fractal-demos      | }
2023-10-03T11:29:08.0220125Z fractal-server     | INFO:     172.18.0.4:58476 - "POST /api/v1/project/1/workflow/ HTTP/1.1" 201 Created
2023-10-03T11:29:08.0635447Z fractal-demos      | WF_ID: 1
2023-10-03T11:29:08.3608941Z fractal-server     | INFO:     172.18.0.4:58482 - "GET /api/v1/task/ HTTP/1.1" 200 OK
2023-10-03T11:29:08.3850973Z fractal-server     | INFO:     172.18.0.4:58482 - "POST /api/v1/project/1/workflow/1/wftask/?task_id=1 HTTP/1.1" 201 Created
2023-10-03T11:29:08.3875595Z fractal-demos      | 1
2023-10-03T11:29:08.7139646Z fractal-server     | INFO:     172.18.0.4:58486 - "POST /api/v1/project/1/workflow/1/wftask/?task_id=2 HTTP/1.1" 201 Created
2023-10-03T11:29:08.7168084Z fractal-demos      | 2
2023-10-03T11:29:09.0413257Z fractal-server     | INFO:     172.18.0.4:60794 - "POST /api/v1/project/1/workflow/1/wftask/?task_id=3 HTTP/1.1" 201 Created
2023-10-03T11:29:09.0439754Z fractal-demos      | 3
2023-10-03T11:29:09.3657402Z fractal-server     | INFO:     172.18.0.4:60804 - "POST /api/v1/project/1/workflow/1/wftask/?task_id=4 HTTP/1.1" 201 Created
2023-10-03T11:29:09.3684489Z fractal-demos      | 4
2023-10-03T11:29:09.6921439Z fractal-server     | INFO:     172.18.0.4:60806 - "POST /api/v1/project/1/workflow/1/wftask/?task_id=5 HTTP/1.1" 201 Created
2023-10-03T11:29:09.6948986Z fractal-demos      | 5
2023-10-03T11:29:10.0195543Z fractal-server     | INFO:     172.18.0.4:60814 - "POST /api/v1/project/1/workflow/1/wftask/?task_id=7 HTTP/1.1" 201 Created
2023-10-03T11:29:10.0226695Z fractal-demos      | 6
2023-10-03T11:29:10.3614156Z fractal-server     | INFO:     172.18.0.4:60828 - "POST /api/v1/project/1/workflow/1/apply/?input_dataset_id=1&output_dataset_id=2 HTTP/1.1" 202 Accepted
2023-10-03T11:29:10.3649831Z fractal-demos      | 1
2023-10-03T11:29:10.3841733Z fractal-server     | 2023-10-03 11:29:10,383 - WF1_job1 - INFO - Start execution of workflow "Workflow cardiac-tiny"; more logs at /home/fractal_share/artifacts/proj_0000001_wf_0000001_job_0000001_20231003_112910/workflow.log
2023-10-03T11:29:10.4057931Z fractal-demos      | RUN EXAMPLE 01 - END
2023-10-03T11:29:10.4063383Z fractal-demos      | ALL GOOD
2023-10-03T11:29:10.5985507Z fractal-demos exited with code 0
2023-10-03T11:29:12.3886232Z fractal-server     | INFO:     127.0.0.1:55126 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:17.4835641Z fractal-server     | INFO:     127.0.0.1:55136 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:22.5791319Z fractal-server     | INFO:     127.0.0.1:55626 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:23.5608173Z fractal-db         | 2023-10-03 11:29:23.560 UTC [51] LOG:  checkpoint complete: wrote 199 buffers (1.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=19.656 s, sync=0.002 s, total=19.660 s; sync files=123, longest=0.001 s, average=0.001 s; distance=887 kB, estimate=887 kB
2023-10-03T11:29:27.6847156Z fractal-server     | INFO:     127.0.0.1:55642 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:32.7922978Z fractal-server     | INFO:     127.0.0.1:41600 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:37.9144614Z fractal-server     | INFO:     127.0.0.1:41604 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:43.0087622Z fractal-server     | INFO:     127.0.0.1:56064 - "GET /api/alive/ HTTP/1.1" 200 OK
2023-10-03T11:29:47.5637534Z fractal-server     | 2023-10-03 11:29:47,563 - WF1_job1 - INFO - End execution of workflow "Workflow cardiac-tiny"; more logs at /home/fractal_share/artifacts/proj_0000001_wf_0000001_job_0000001_20231003_112910/workflow.log
2023-10-03T11:29:48.1126583Z fractal-server     | INFO:     127.0.0.1:56078 - "GET /api/alive/ HTTP/1.1" 200 OK

[tcompa]

For this, we would need some logic to shutdown everything after the demos service is over.

See --abort-on-container-exit flag

[jluethi]

For fractal-tasks-core, we have fewer cases:

If FRACTAL_TASKS_CORE_RELEASE_VERSION is set, we pass it to the fractal-task-collect command.
If it is not set, we do not include it in the fractal-task-collect command.

Wouldn't we want to be able to use this to e.g. test a new task version or a task branch? Or is the focus on testing changes to server & web server with the released tasks?

[tcompa]

Wouldn't we want to be able to use this to e.g. test a new task version or a task branch? Or is the focus on testing changes to server & web server with the released tasks?

Brief answer: yes, this can be implemented.
It requires a slightly different approach from the other cases above (which only require the appropriate git clone/checkout and then pip install). Given a git commit hash (or branch), we would git-clone the tasks repo, then git-checkout, then install poetry, then build the wheel file with poetry, then ask fractal-server to collect tasks.

Higher-level: as soon as we have a first working version of this end-to-end tests, we need to decide when/where we want it to run.

1. One obvious use case will be locally: I want to debug something strange, and check that server 1.3.4 works fine with tasks 5.6.7.
2. Another relevant use case will be to run a bunch of automated tests in this repo. These could be always run with the latest versions, periodically (e.g. every night) and upon direct triggering by one of us.
3. We may also want to run these tests as part of other repositories' PRs, but for the moment I'd consider it too much of an overhead (to be discussed). This is the reason why supporting a tasks github branch is maybe not a priority, but it doesn't cost much to add the feature anyway.

[tcompa]

Closed with #21