Stable Data

Stable data is a full stack web app that allows users to create and manage data upload pipelines to meet their specific requirements. Stable Data allows users to specify data constraints, upload frequency requirements, and pipeline viewers, uploaders, and managers.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

This project requires docker and docker-compose to build

Installation

1. Clone the repo

   git clone https://github.com/jenniferNeto/ColPal.git

2. Create a .env file in the root directory Required attributes are:

   DB_HOST
   DB_NAME
   DB_USER
   DB_PASS
   
   POSTGRES_DB
   POSTGRES_USER
   POSTGRES_PASSWORD

   • DB_HOST must be 'db'
   • DB_x must match POSTGRES_x

3. Build the docker containers

   docker-compose up --build

Usage

Roadmap

• Pipeline create
  • Create pipeline groups: viewer, uploader, manager
    • Add groups to pipelines
• Pipeline update, admin/manager vs uploader
• Django test cases for actions
• Basic file upload
  • Upload temporary file to cloud storage
  • Generate schema for csv files
    • Generate schema for google sheets
  • Validate file against pipeline constraints
  • Upload approved file to snowflake

See the open issues for a full list of proposed features (and known issues).

License

Distributed under the MIT License. See LICENSE for more information.