Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic rclone guide + config #183

Merged
merged 2 commits into from
Oct 14, 2024
Merged

Conversation

jbusecke
Copy link
Contributor

Adding basic rclone operations in the data guide + initial instructions for authenticated access with fsspec or rclone.

Copy link

github-actions bot commented Oct 14, 2024

👋 Thanks for opening this PR! The Cookbook will be automatically built with GitHub Actions. To see the status of your deployment, click below.
🔍 Git commit SHA: 73fc7fb
✅ Deployment Preview URL: https://leap-stc.github.io/_preview/183

`````

:::\{warning}
Ideally we want to store these secrets only in one central location. The natural place for these seems to be in an [AWS cli profiles](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html#cli-configure-files-format), which can also be used for fsspec. There however seem to be multiple issues ([here](https://forum.rclone.org/t/shared-credentials-file-is-not-recognised/46993)) around this feature in rclone, and so far we have not succeeded in using AWS profiles in rclone.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spent quite some time to try to reconcile this. Ideally we only want to store one config file (aws profile) and then reference that from the rclone config, but I did not manage to get this to work at all so far. Might be worth tracking in an issue.

@jbusecke
Copy link
Contributor Author

jbusecke commented Oct 14, 2024

Hmmm seems like the addition of sphinx-design has introduce new warnings that fail the build:

WARNING: while setting up extension sphinx_design: node class 'dropdown_main' is already registered, its visitors will be overridden
WARNING: while setting up extension sphinx_design: node class 'dropdown_title' is already registered, its visitors will be overridden
WARNING: while setting up extension sphinx_design: directive 'dropdown' is already registered, it will be overridden
WARNING: while setting up extension sphinx_design: role 'fa' is already registered, it will be overridden
WARNING: while setting up extension sphinx_design: node class 'fontawesome' is already registered, its visitors will be overridden

Let me try to ignore warnings following this pattern and these config values.

@jbusecke jbusecke merged commit 6a7709e into main Oct 14, 2024
2 checks passed
@jbusecke jbusecke deleted the authenticated-access-to-cloud-storage branch October 14, 2024 16:24
github-actions bot pushed a commit that referenced this pull request Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant