New repository: sdlf-<team>-ingestionLibrary

[BradBender]

One of the challenges we've been facing is SDLF-teams have nowhere to store/deploy CloudFormation templates for ingestions. One option is to shoehorn templates into one of the existing sdlf- repos, but none of them are quite suitable:
sdlf-{team}-pipeline
sdlf-{team}-datalakeLibrary
sdlf-{team}-stageA
sdlf-{team}-stageB
sdlf-{team}-dataset
sdlf-{team}-pipLibrary

It would be useful to have a skeleton repository to store/deploy custom CloudFormation templates, specific to an individual SDLF-team. It would be up to each team to develop their own ingestion-mechanism(s), but we should at a minimum provide them a framework to build within. Teams who cannot ingest their own data will slow our data lake adoption :-(

[jaidisido]

Hi @BradBender, I agree with you that having a separate repository for ingestion is the way to go, and that is the strategy that I have seen most users follow.

We have made the deliberate choice of not creating one simply because we did not want this framework to increase too much in scope beyond what it is supposed to be doing.

That being said, adding a repository to a team would be as simple as these steps:

1. Create a sdlf-ingestion repository as a baseline. You can use this example as a template for instance
2. Modify this team deploy.sh to reference the sdlf-ingestion repository as well

Hope this helps