Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

working docker-compose: solved .mage_data persistance issue #30

Merged
merged 1 commit into from
Oct 21, 2024

Conversation

vnbl
Copy link
Collaborator

@vnbl vnbl commented Oct 21, 2024

docker-compose.yml file:

  • added volume for .mage_data folder (needed for past logs, storing keys and pipeline variables)
  • added MAGE_DATA_DIR variable to file and .env.example.

etl-pipeline/metadata.yaml file:

  • ensured project points to correct .mage_data folder

requirements.txt

  • pip freeze from working container

Notes:

  • After researching a bit I found that mage doesn't support directly setting up secrets from a .env file or using Github Actions outside of the predetermined services in io_config.yaml file (postgres and mysql are fine).
  • Secrets can only be created in Mage-UI and are stored and encrypted in a sqlite database in folder .mage_data in a secure location at the local machine / container.
  • If we don't want to store secrets this way we have to use a service that is predetermined in io_config.yaml file (options are AWS, Azure Key Vault or GCP Secret Manager, check io_config.yaml file for more info). For this solution we'll probably have to change the way secrets are accessed in code.

@vnbl vnbl merged commit 8556737 into dev Oct 21, 2024
2 checks passed
@vnbl vnbl deleted the docker-compose-setup branch October 23, 2024 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants