Data lake pipeline based on APACHE Spark Streaming following the similar approach of delta lake (bronze, silver, gold) by databricks
- Contains all the ingested packet without any transformation along with ingestion information
- Contains a transformed version of bronze table payload with necessary information for gold table
- Primary Data store
- Search Store