Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CID blockstore #5787

Merged
merged 19 commits into from
Mar 1, 2025
Merged

CID blockstore #5787

merged 19 commits into from
Mar 1, 2025

Conversation

jorgee
Copy link
Contributor

@jorgee jorgee commented Feb 13, 2025 •
edited
Loading

M1 Implementation of CID store for provenance

Changes:

  • CID store is specified by workflow.data.store.location
  • Workflow Hash is created based on the workflow and parameters description
  • workflow, tasks and outputs metadata are stored in <cid.store.location>/.meta
  • references to other cid metadata are cid://<workflow_hash|task_hash/output_target_path
  • CID NIO Filesystem to access data based on CIS URLs
  • nextflow cid command to log, show and get lineage from CID store metadata

Known Limitations:

  • Outputs which are not published in absolutePaths or URLs which are not subfolders both the outputDir, we can not infer the relative output target path. They are not currently tracked in the CID store. We could create a hash for the parent directory of the URL or absolute path and use it as relative folder.

@jorgee jorgee mentioned this pull request Feb 13, 2025
Draft
@jorgee jorgee marked this pull request as draft February 13, 2025 11:09
@jorgee
M0 implementation
a2139e3 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
fix tests
fddc5f7 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
fix tests
fe780a8 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
first M1 updates
f9f7ed2 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
fix tests
0c2492e 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
update descriptions
41ac817 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
fix test
cdc3116 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@bentsherman bentsherman changed the title Cid store m0 CID blockstore Feb 21, 2025
@jorgee
First commit to M1 implementation
82b1ccd 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
fix NPE in tests
edfaf5b 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@bentsherman bentsherman mentioned this pull request Feb 27, 2025
Open
jorgee and others added 5 commits February 27, 2025 19:51
@jorgee
Fix NPE in tests
c207d92 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Add CidStore factory
f4b9031 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
fix cid paht hash validation
b89cdf1 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Merge branch 'cid-store' into cid-store-m0
fdbda27
@rmzelle
Typo fix in container.md (#5825) [ci skip]
cfb27d0 
"paths are configure" > "paths are configured"

Signed-off-by: Rintze M. Zelle, PhD <rintze.zelle@gmail.com>
@jorgee jorgee marked this pull request as ready for review February 28, 2025 12:42
@jorgee jorgee requested a review from pditommaso February 28, 2025 12:54
jorgee and others added 3 commits February 28, 2025 11:30
@jorgee @bentsherman
Fix Google Batch autoRetryExitCodes bug (#5828)
cfeedd6 
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Co-authored-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
Use parallelization in Gradle build (#5830) [ci fast]
c02fe91 
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@pditommaso
Merge branch 'master' into cid-store-m0
84ee951
@pditommaso pditommaso requested a review from a team as a code owner March 1, 2025 13:57
@pditommaso
Copy link
Member

Looks excellent. Let's merge and continue in the baseline branch

@pditommaso pditommaso merged commit 0d1a74f into cid-store Mar 1, 2025
3 of 6 checks passed
@pditommaso pditommaso deleted the cid-store-m0 branch March 1, 2025 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bentsherman bentsherman bentsherman left review comments

@pditommaso pditommaso Awaiting requested review from pditommaso

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jorgee @pditommaso @bentsherman @rmzelle