"Integration" with Open Science Data Federation

[yarikoptic]

Proposed change

yet to be figured out, but a user asked

Please clarify whether data saved in DANDI can be reached and used through OSDF.
OSDF: https://osg-htc.org/services/osdf

From https://osg-htc.org/docs/data/stashcache/overview/: The OSG operates the Open Science Data Federation (OSDF), which provides organizations with a method to distribute their data in a scalable manner to thousands of jobs without needing to pre-stage data at each site.

Some caching solutions are part of the framework: https://osg-htc.org/docs/data/stashcache/install-cache/

Overall -- it sounds like a solution allowing making (public or private ) local data resources available to remote compute resources through unified interface with some caching to make it efficient. So, if someone tackles it I guess OSDF DANDI users would benefit form

• uniform access to DANDI data via OSDF interfaces
• caching
• may be discovery of some kind of a resource

Wasn't sure where to seek clarification yet... or what it would entail for a DANDI resource to expose itself as part of OSDF. If there was some other example of a public resource exposed to OSDF, might have provided a template to follow. Searching github got to https://github.com/PerilousApricot/docs/blob/de67fc125efb334ca67b011bde179689f035ffda/docs/data/stashcache/vo-data.md?plain=1#L3 so I will email help@opensciencegrid.org for either such integration makes sense if there are examples

Alternative options

regular means to access from S3 directly, via DANDI API, via datalad/git-annex, dandi-cli

Who would use this feature?

Users of OSDF

(Optional): Suggest a solution

yet to be figured out if there is a problem to be solved ;)

[yarikoptic]

Current solution approach: development of efficient WEBDAV interface which could then be used as an already supported "standard" protocol to implement desired by OSDF caching etc. Current webdav "frontend" is developed at https://github.com/dandi/dandidav and deployed at https://dandi.centerforopenneuroscience.org