Serverless Data Lake Framework 2.5.0-alpha.2

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-foundations] generic deployspec.yaml by @cnfait in #366
• [sdlf-team] seedfarmer support by @cnfait in #367
• [sdlf-dataset] seedfarmer support by @cnfait in #369
• pin cfn-lint to <1 in github workflow by @cnfait in #370
• disable w12 check on modulestack where exception is documented by @cnfait in #371
• remove hash from glue job cfn reference & script name by @cnfait in #372
• [sdlf-cicd] glue job deployer: support for custom templates by @cnfait in #373
• [sdlf-cicd] github support as an alternative to codecommit by @cnfait in #374

Full Changelog: 2.5.0-alpha.1...2.5.0-alpha.2

Serverless Data Lake Framework 2.5.0-alpha.1

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• SeedFarmer Support by @cnfait in #365

Full Changelog: 2.5.0-alpha.0...2.5.0-alpha.1

Serverless Data Lake Framework 2.5.0-alpha.0

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• new stage: sdlf-stage-lambda by @cnfait in #338
• force use of cfn-lint 0.87.7 since release 1.3.0 break cloudformation-cli by @cnfait in #341
• new stage: sdlf-stage-ecsfargate by @cnfait in #342
• new stage: emrserverless by @cnfait in #344
• new stage: sdlf-stage-glue by @cnfait in #345
• [sdlf-stage-*] generic get_transform_details in routing lambda by @cnfait in #346
• pin cfn-lint to <1 instead of 0.87.7 by @cnfait in #349
• [sdlf-pipeline] support for pipelines with no source event by @cnfait in #350
• [sdlf-stage-*] put metadata in transform field, handle decimal metadata by @cnfait in #351
• [vpc] support shared vpc subnets by @cnfait in #348
• Preview: CDK Python by @cnfait in #353
• build stage-lambda and stage-glue by default by @cnfait in #355
• [sdlf-team][cdk] lakeformation data_location_access permissions by @cnfait in #356
• [sdlf-dataset][cdk] lake formation resources by @cnfait in #357
• avoid code duplication in domain-cicd by @cnfait in #359
• [sdlf-team][cdk] lakeformation tag and tag permissions by @cnfait in #360
• [sdlf-stage-emrserverless/ecsfargate] CDK Python by @cnfait in #361
• [sdlf-foundations/team][cdk] kms removal policy by @cnfait in #362
• [sdlf-*][cdk] set removal policy on stateful resources by @cnfait in #363
• [sdlf-*][cdk] extends Construct instead of Stack, add requirements.txt by @cnfait in #364

Full Changelog: 2.4.0...2.5.0-alpha.0

Serverless Data Lake Framework 2.4.1

This is a bugfix release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• bugfix force use of cfn-lint 0.87.7 since release 1.3.0 breaks cloudformation-cli by @cnfait 4badb63 and #341

Full Changelog: 2.4.0...2.4.1

Serverless Data Lake Framework 2.4.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• Support for disabling stages by @cnfait in #326
• sdlf-cicd: support for gitlab as an alternative to codecommit by @cnfait in #331
• use mkdocs instead of sphinx for documentation website by @cnfait in #332
• general documentation update by @cnfait in #333
• diagrams for sdlf stages and sdlf-monitoring by @cnfait in #335
• Fix #272: Replace logs:TagLogGroup permission with logs:TagResource by @IAMDAVID0920 in #330
• replace uses of /SDLF/S3/CentralBucket with /SDLF/S3/RawBucket by @cnfait in #336
• add /SDLF/Dynamo/Datasets SSM parameter for datasets dynamodb table by @cnfait in 0f2c16c
• doc update: sdlf in-a-nutshell diagram by @cnfait in #337

New Contributors

• @IAMDAVID0920 made their first contribution in #330

Full Changelog: 2.3.0...2.4.0

Serverless Data Lake Framework 2.3.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• parameterize team main repository prefix by @cnfait in #290
• add ssm parameter for octagon-DataSchemas table by @cnfait in #291
• get dataschemas table name from ssm in replicate lambda by @cnfait in #292
• restrict sdlf-catalog lambda to its dedicated sqs queues by @cnfait in #293
• use RetainExceptOnCreate deletion policy for KMS keys and aliases by @cnfait in #294
• stageA/stageB: replace hardcoded suffix -a/-b with stage name by @cnfait in #295
• Glue security config SSM by @cnfait in #296
• add ssm parameter for Glue Crawler name by @cnfait in #297
• single role (per stage) for all metadata steps by @cnfait in #299
• legislators deploy.sh linting by @cnfait in #301
• add flags for optional features to deploy.sh multi-account deployment by @kangsoon-dev in #289
• add support for nested stacks in datasets template by @cnfait in #302
• glue job deployer: job/connection Update permissions by @cnfait in #304
• S3 Storage Lens permissions by @cnfait in #305
• update ruff linter settings by @gwenika in #306
• remove bogus SQS DeleteMessageBatch/SendMessageBatch permissions by @cnfait in #307
• Enforce TLS on devops S3 access logs bucket by @cnfait in #313
• Rework rMain CodePipeline by @cnfait in #314
• Glue Job Deployer: use /service-role/ path for role by @cnfait in #315
• Glue Job Deployer rework by @cnfait in #316
• do not set MaxItems when paginating by @cnfait in #318
• Glue Job Deployer: handle multiple Glue Job Connections, and their AZ by @cnfait in #319
• Glue Job Deployer CodeBuild: use crossaccount role by @cnfait in #320
• use proper DynamoDB JSON structure for items in sdlf-foundations replicate lambda by @cnfait in #321
• use proper DynamoDB JSON structure for items in sdlf-foundations catalog lambda by @cnfait in #322

Thanks

We thank all the contributors/users for their work on this release, in particular @gwenika and @kangsoon-dev!

Full Changelog: 2.2.0...2.3.0

Serverless Data Lake Framework 2.2.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• replace deprecated utcnow() with now(datetime.timezone.utc) for expli… by @denxychen in #274
• update now() with timezone alias #271 by @levo-777 in #273
• fix team removal workflow by @cnfait in #278
• Changed all the project to use Python 3.12 -- Lambdas and CodeBuild by @Neuw84 in #280
• remove support for single-bucket setup by @cnfait in #279
• Ruff format by @tomaszwrzonski in #285
• Allow adding of non-sdlf layers by arn + bug fix on custom stages by @kangsoon-dev in #277
• remove unused pEntryPoint parameter in stages by @cnfait in #288

New Contributors

• @denxychen made their first contribution in #274
• @levo-777 made their first contribution in #273
• @Neuw84 made their first contribution in #280
• @kangsoon-dev made their first contribution in #277

Thanks

We thank all the contributors/users for their work on this release, in particular @denxychen, @kangsoon-dev, @levo-777, @Neuw84 and @tomaszwrzonski!

Full Changelog: 2.1.1...2.2.0

Serverless Data Lake Framework 2.1.1

Full Changelog: 2.1.0...2.1.1

Serverless Data Lake Framework 2.1.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

SDLF 2.1 brings an example stage for data quality with Glue (thank you @mureddy19!), improvements to monitoring with S3 Storage Lens, Lake Formation tag support for teams, plus a number of smaller improvements and bug fixes.

What's Changed

• Fixed incorrectly named variable in the for loop which deletes legacy… by @waltmayfield in #243
• Transform data quality steps into a proper sdlf stage (sdlf-stage-dataquality) by @cnfait in #244
• Lake Formation tag (LF-Tag) support for sdlf pipelines by @cnfait in #247
• add DescribeStackEvents to CodePipeline role permissions (#245) by @cnfait in #248
• dedicated event bus per team in child accounts (#165) by @cnfait in #249
• rename pOrganizationName parameter to pOrg to be consistent across all SDLF by @cnfait in #250
• add kms encryption for codecommit repositories by @gwenika in #253
• Update GitHub actions to the latest actions version by @fleventy-5 in #254
• add on-demand github workflow testing sdlf deployment by @gwenika in #255
• make sure lf-tag is created before using it in permissions by @cnfait in #257
• create team repository in a separate stack from team codepipeline by @cnfait in #258
• add policies to retain KMS key aliases on update/delete by @gwenika in #259
• sdlf-monitoring rewrite, add storage lens by @cnfait in #260
• event rule to stop codepipeline first run (rMain only for now) by @cnfait in #261
• create sdlf repositories in a separate stack from sdlf-main codepipeline by @cnfait in #262
• create sdlf-main-{domain}-{team} repository with default content and branches by @cnfait in #263
• update user agent by @cnfait in #264
• update event rule to stop rDomain CodePipeline first run by @cnfait in #265

New Contributors

• @waltmayfield made their first contribution in #243
• @fleventy-5 made their first contribution in #254

Thanks

We thank all the contributors/users for their work on this release, in particular @fleventy-5, @gwenika, @mureddy19 and @waltmayfield!

Full Changelog: 2.0.1...2.1.0

Serverless Data Lake Framework 1.6.0

What's Changed

• remove duplicate EventSelectors in cloudtrail stack by @cnfait in f75307d
• update all requirements.txt with latest versions by @cnfait in 99d7746
• update external layer/wheel example with latest version of aws-sdk-pandas (awswrangler) by @cnfait in 7b85c16
• remove awscli v1 from codebuild pipLibraries project, use v2 instead by @cnfait in 8d143b8
• move from Python 3.9 to Python 3.11 across all SDLF by @cnfait in 6c2ff8b

Full Changelog: 1.5.2...1.6.0