Data harvester documentation

[lsetiawan]

In CAVA meetings we have had the same pairs of Q&As repeatedly coming up over how the harvester actually functions. This calls for a documentation of the design and behavior of the harvester.

While the harvester behaviors and features naturally evolve over time, it is imperative to keep this document up-to-date with PRs in the code. Doing so will help other team members understand current limitation of the CAVA system and provide recommendations on how to solve communication problems (as in requesting data) with the OOI CI.

The documentation should include at the minimum the following items:

• the time and frequency of which data are requested from the OOI CI
• the instruments and sites (via both reference designator and common names) from which data are requested
• details of data variables/streams requested by harvester
• step-by-step dissection of harvester behavior of each daily request
• step-by-step dissection of harvester behavior when any of the requests fail, and provide the reasons (or best guess of reasons) why the requests fail
• any change of the harvested data, including numerical values or dimension/coordinate variable names
• data provenance retrieved from OOI CI and any additional info added by CAVA
• an executable notebook that checks the exact correspondence between data on CAVA THREDDS/Postgres and data requested directly from OOI CI (Upgrade to Prefect 2.0 #21)

Comments

Michael Vardaro @ 2021-01-07T18:27:20

This still needs to happen.

Disclaimer

This issue was imported from https://github.com/cormorack/io2_portal/issues/22.

Note: Old repository might be a private repository.