Ability to add dbt packages in CLI without meltano add transform ...

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3536

Originally created by @aaronsteers on 2022-05-26 20:17:07

---

Long-term, I'd like to replace this in sample scripts...

meltano add transform dbt-tap-gitlab

...with this...

meltano invoke dbt-snowflake:add-dependency --repo=https://gitlab.com/meltano/dbt-tap-gitlab

Then we can refer users to hub.getdbt.com for transform packages, while still having nice inline-able sample code. This won't happen in the short term though, and to keep samples working I think we have to accept meltano add transform dbt-tap-gitlab as sourced from the hub for a little while longer.

This also opens up a very broad range of utilities and transformation packages, both from the dbt community as well as the Meltano community. It also makes installing things like dbt-utils package much easier and faster for Meltano users.

Also logged as feature request here:

• [CT-690] Ability to add package dependencies via the CLI dbt-labs/dbt-core#5302

If added as a feature to dbt CLI, then we would not need to built into our own CLI.

[MeltyBot]

View 2 previous comments from the original issue on GitLab

[aaronsteers]

Originally this was authored before we had dbt-ext. Now that we have that in place, this should be a quick win.

We basically already have it written in meltano:

show code

    def add_to_packages(self, plugin: ProjectPlugin) -> None:
        """Add the plugin's package to the project's `packages.yml` file.

        Args:
            plugin: The plugin to add to the project.

        Raises:
            ValueError: If the plugin is missing the git repo URL.
        """
        if not os.path.exists(self.packages_file):
            self.packages_file.touch()

        package_yaml = yaml.load(self.packages_file) or {"packages": []}

        git_repo = plugin.pip_url
        if not git_repo:
            raise ValueError(f"Missing pip_url for transform plugin '{plugin.name}'")

        revision: str | None = None
        if len(git_repo.split("@")) == 2:
            git_repo, revision = git_repo.split("@")
        for package in package_yaml["packages"]:
            same_ref = (
                package.get("git", "") == git_repo
                and package.get("revision", None) == revision
            )
            if same_ref:
                return

        package_ref = {"git": git_repo}
        if revision:
            package_ref["revision"] = revision
        package_yaml["packages"].append(package_ref)

        with open(self.packages_file, "w") as f:
            yaml.dump(package_yaml, f)

[BuzzCutNorman]

If added as a feature to dbt CLI, then we would not need to built into our own CLI.

DBT added this in dbt-core 1.7, I just tried and it with dbt-postgres and it worked for me.

The --add-package flag allows you to add a package to the packages.yml with configurable --version and --source information. The --dry-run flag, when set to False(default), recompiles the package-lock.yml file after a new package is added to the packages.yml file. Set the flag to True for the changes to not persist.

https://docs.getdbt.com/reference/commands/deps

cc @edgarrmondragon

[edgarrmondragon]

@BuzzCutNorman that's nice! Did you use it with meltano invoke dbt-postgres deps --add-package?

cc @pnadolny13

Maybe relevant:

• Unpin dbt-* dependencies hub#1642

[BuzzCutNorman]

@BuzzCutNorman that's nice! Did you use it with meltano invoke dbt-postgres deps --add-package?

@edgarrmondragon Yes sir I did. Here is the command I used.

meltano invoke dbt-postgres deps --add-package tuva-health/the_tuva_project@">=0.5.0,<1.0.0"

I manually set dbt-postgres as the following in the project's meltano.yml file.

  utilities:
  - name: dbt-postgres
    variant: dbt-labs
    pip_url: dbt-core~=1.7.4 dbt-postgres~=1.7.4 git+https://github.com/meltano/dbt-ext.git@main

cc @pnadolny13

[edgarrmondragon]

Thanks for confirming @BuzzCutNorman 🙂

I've bumped the dbt utilites in meltano/hub#1644, so folks should be getting the latest features like deps --add-package.