Skip to content

build-on-aws/ci-cd-serverless-spark

Serverless Spark CI/CD on AWS with GitHub Actions

Sample code repo that shows how to run GitHub Actions with Amazon EMR Serverless. Forms the basis of the following tutorial:

The code here represents what your repository should look like after following the tutorial.

Pre-requisites

  • An AWS Account with Admin privileges
  • GitHub OIDC Provider in AWS
  • S3 Bucket(s)
  • EMR Serverless Spark application(s)
  • IAM Roles for GitHub and EMR Serverless

You can create all of these, including some sample data, using the included CloudFormation template.

Warning 💰 The CloudFormation template creates EMR Serverless applications that you will be charged for when integration tests AND the scheduled workflow runs.

Note The IAM roles created in the template are very tightly scoped to the relevant S3 Buckets and EMR Serverless applications created by the stack.

Other resources

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.