Use CreatePipeline instead of PipelineSpec for resources.Pipeline struct

[andrewnester]

Changes

CreatePipeline is a more complete structure (superset of PipelineSpec one) which enables support of additional fields such as run_as and allow_duplicate_names in DABs configuration. Note: these fields are subject to support in TF in order to correctly work.

Tests

Existing tests pass + no fields are removed from JSON schema

[denik]

Is it possible to use new API stubbing functionality (example: https://github.com/databricks/cli/blob/main/acceptance/workspace/jobs/create/test.toml#L6) to add a couple of acceptance tests:

• one that uses run_as attribute (the one we want)
• one that uses dry_run attribute (the one that we don't want to support)

[shreyas-goenka]

Both run_as and dry_run don't seem to be supported in terraform? For run_as we should add support, but for dry_run we can either add support or remove it from the bundle schema.

Note, we can also switch over to using CreatePipeline in TF as well.

[andrewnester]

run_as will be supported in the next TF release, the rest indeed is not

[pietern]

I agree it would be nice to drop dry_run and allow_duplicate_names from the schema, but we have examples of other resources with properties we don't intend to support so we can deal with that separately.

Just so that we can omit these properties later, could you make sure that they're not forwarded to TF as part of tfdyn? That way we can remove them later (when we also take a closer look at properties included for other resources) and not regress anything.