Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ETL-670] Adjust cleanup job to run on main directory of dev env #126

Merged
merged 3 commits into from
Jul 15, 2024
Merged

[ETL-670] Adjust cleanup job to run on main directory of dev env #126

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b300e3
Adjust cleanup job to run always, but skip steps if `main`
BryanFauble Jul 13, 2024
4430539
Update when cleanup is ran
BryanFauble Jul 15, 2024
6e36626
Add guard condition for main and non-dev bucket
BryanFauble Jul 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/upload-and-deploy.yaml
View file
Open in desktop
Copy link
Contributor

@rxu17 rxu17 Jul 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually making me rethink the logic of integration-test-develop as well. integration-test-develop is just copying the recent 2 weeks of exports and should just be triggering the branch's specific namespaced jobs to run on that subset of exports but it shouldn't be doing it for the main namespace in the dev project when we merge to main because main will always run on all of the data in dev to mirror what it does in prod (which it's currently doing)

I am thinking now that these are the updates we could make and it makes a bit more sense:

  • integration-test-develop needs the if: github.ref_name != 'main' line at the highest level of the GH job
  • integration-test-develop-cleanup needs the if: github.ref_name != 'main' line at the highest level of the GH job
  • sceptre-deploy-staging will have the update:needs: sceptre-deploy-develop

I think this will give the logic we need. That the integration test and cleanup for when we are in dev environment to only run when we are working on the namespaced branch, and doesn't run when we merge to main.

Thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the latest comments on this PR, but I think we want the same behavior when there are changes on dev/main as any other dev/* namespace.

Original file line number Diff line number Diff line change
Expand Up @@ -262,30 +262,33 @@ jobs:
name: Cleanup non-main branch data before integration tests
runs-on: ubuntu-latest
needs: sceptre-deploy-develop
if: github.ref_name != 'main'
environment: develop
# These permissions are needed to interact with GitHub's OIDC Token endpoint.
permissions:
id-token: write
contents: read
steps:
- name: Setup code, pipenv, aws
if: github.ref_name != 'main'
uses: Sage-Bionetworks/action-pipenv-aws-setup@v3
with:
role_to_assume: ${{ vars.AWS_CREDENTIALS_IAM_ROLE }}
role_session_name: GitHubActions-${{ github.repository_owner }}-${{ github.event.repository.name }}-${{ github.run_id }}
python_version: ${{ env.PYTHON_VERSION }}

- name: Set namespace for non-default branch
if: github.ref_name != 'main'
run: echo "NAMESPACE=$GITHUB_REF_NAME" >> $GITHUB_ENV

- name: Clean input data bucket
if: github.ref_name != 'main'
run: >
pipenv run python src/scripts/manage_artifacts/clean_for_integration_test.py
--bucket $DEV_INPUT_BUCKET
--bucket_prefix "${{ env.NAMESPACE }}/"

- name: Clean intermediate data bucket
if: github.ref_name != 'main'
run: >
pipenv run python src/scripts/manage_artifacts/clean_for_integration_test.py
--bucket $DEV_INTERMEDIATE_BUCKET
Expand Down
Loading