Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingestion): Implement DB migrations & Ingestion IAC #4

Merged
merged 4 commits into from
Jun 27, 2024
Merged

Conversation

hibare
Copy link
Collaborator

@hibare hibare commented Jun 27, 2024

This PR introduces several key enhancements to our project:

  1. Database Migration using Alembic: Implements a new database migration to modify the schema and ensure compatibility with new data ingestion requirements.
    • Added new Alembic migration scripts located in the migrations/db/versions directory for findings, scans and jobs table.
    • The migration script alters the database schema to include new tables and columns required for the ingestion process.
  2. Infrastructure as Code (IAC) for Data Ingestion: Sets up the infrastructure to handle data ingestion using AWS services.
    • Added Terraform configuration files in the infrastructure/ingestion/aws directory.
    • The configuration provisions necessary AWS resources, step function, Lambda functions, and IAM roles.
  3. Lambda Functions for Data Ingestion: Includes Lambda functions to ingest data from S3 into the database tables.
    • Implemented Lambda functions in the infrastructure/ingestion/aws/lambda directory.
    • They process incoming data files and insert the data into the appropriate database tables.
  4. Ingestions: Implement findings data ingestion.

@hibare hibare marked this pull request as ready for review June 27, 2024 06:25
Copy link
Member

@yilmi yilmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@yilmi yilmi merged commit 5488ae2 into main Jun 27, 2024
1 check passed
@yilmi yilmi deleted the ingestion branch June 27, 2024 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants