Breaking changes
- When using
LoadMode.DBT_LS
, Cosmos will now attempt to use thedbtRunner
as opposed to subprocess to rundbt ls
. While this represents significant performance improvements (half the vCPU usage and some memory consumption improvement), this may not work in scenarios where users had multiple Python virtual environments to manage different versions of dbt and its adaptors. In those cases, please, setRenderConfig(invocation_mode=InvocationMode.SUBPROCESS)
to have the same behaviour Cosmos had in previous versions. Additional information here and here.
Features
- Use
dbtRunner
in the DAG Processor when usingLoadMode.DBT_LS
ifdbt-core
is available by @tatiana in #1484. Additional information here. - Allow users to opt-out of
dbtRunner
during DAG parsing withInvocationMode.SUBPROCESS
by @tatiana in #1495. Check out the documentation. - Add structure to support multiple db for async operator execution by @pankajastro in #1483
- Support overriding the
profile_config
per dbt node or folder using config by @tatiana in #1492. More information here.
Bug Fixes
- Fix select complex intersection of three tag-based graph selectors by @tatiana in #1466
Enhancement
- Fix OpenLineage deprecation warning by @CorsettiS in #1449
- Move
DbtRunner
related functions intodbt/runner.py
module by @tatiana in #1480
Others
- GitHub Actions Dependabot: #1487
- Pre-commit updates: #1473, #1493
Bug Fixes
- Fix
httpx.get
exception handling while emitting telemetry by @tatiana in #1439 - Fix (not) rendering detached tests in
TestBehavior.NONE
andAFTER_ALL
by @tatiana in #1463 - Fix detached test tasks names so they do not exceed 250 chars by @tatiana in #1464
Enhancement
- Allow users to opt-in or out (default) of detached test nodes by @tatiana in #1470. Learn more about this here.
Docs
- Docs: Fix broken links and rendering by @pankajastro in #1437
- Update
operator args
docs to includeinstall_deps
by @tatiana in #1456 - Improve Cosmos
select
docs to include latest graph operator support by @tatiana in #1467
Others
- Upgrade GitHub action artifacts upload-artifact & download-artifact to v4 by @pankajkoti in #1445
- Enable Depandabot to scan outdated Github Actions dependencies by @tatiana in #1446
- Pre-commit hook updates in #1459, #1441
- Dependabot Github action updates in #1451, #1452, #1453, #1454, #1455
Bug Fixes
- Fix rendering dbt tests with multiple parents by @tatiana in #1433
- Add
kwargs
param in DocsOperator methodupload_to_cloud_storage
by @pankajastro in #1422
Docs
- Improve OpenLineage documentation by @tatiana in #1431
Others
- Enable Docs DAG in CI leveraging existing CI connections by @pankajkoti in #1428
- Install providers with airflow by @pankajkoti in #1432
- Remove unused docs dependency by @pankajastro in #1414
- Pre-commit hook updates in #1424
New Features
- Support customizing Airflow operator arguments per dbt node by @wornjs in #1339. More information.
- Support uploading dbt artifacts to remote cloud storages via callback by @pankajkoti in #1389. Read more.
- Add support to
TestBehavior.BUILD
by @tatiana in #1377. Documentation. - Add support for the "at" operator when using
LoadMode.DBT_MANIFEST
orCUSTOM
by @benjy44 in #1372 - Add dbt clone operator by @pankajastro in #1326, as documented in here.
- Support rendering tasks with non-ASCII characters by @t0momi219 in #1278 Read more.
- Add warning callback on source freshness by @pankajastro in #1400 Read more.
- Add Oracle Profile mapping by @slords and @pankajkoti in #1190 and #1404
- Emit telemetry to Scarf during DAG run by @tatiana in #1397
- Save tasks map as
DbtToAirflowConverter
property by @internetcoffeephone and @hheemskerk in #1362
Bug Fixes
- Fix the mock value of port in
TrinoBaseProfileMapping
to be an integer by @dwolfeu #1322 - Fix access to the
dbt docs
menu item outside of Astro cloud by @tatiana in #1312 - Add missing
DbtSourceGcpCloudRunJobOperator
in modulecosmos.operators.gcp_cloud_run_job
by @anai-s in #1290 - Support building
DbtDag
without setting paths inProjectConfig
by @tatiana in #1307 - Fix parsing dbt ls outputs that contain JSONs that are not dbt nodes by @tatiana in #1296
- Fix Snowflake Profile mapping when using AWS default region by @tatiana in #1406
- Fix dag rendering for taskflow + DbtTaskGroup combo by @pankajastro in #1360
Enhancements
- Improve dbt command execution logs to troubleshoot
None
values by @tatiana in #1392 - Add logging of stdout to dbt graph run_command by @KarolGongola in #1390
- Add
profile_config
for Docker by @andrewhlui in #1347 - Support rendering build operator task-id with non-ASCII characters by @pankajastro in #1415
Docs
- Remove extra ` char from docs by @pankajastro in #1345
- Add limitation about copying target dir files to remote by @pankajkoti in #1305
- Generalise example from README by @ReadytoRocc in #1311
- Add security policy by @tatiana, @chaosmaw and @lzdanski in # 1385
- Mention in documentation that the callback functionality is supported in
ExecutionMode.VIRTUALENV
by @pankajkoti in #1401
Others
- Restore Jaffle Shop so that
basic_cosmos_dag
works as documented by @tatiana in #1374 - Remove Pytest durations from tests scripts by @tatiana in #1383
- Remove typing-extensions as dependency by @pankajastro in #1381
- Pin dbt-databricks version to < 1.9 by @pankajastro in #1376
- Refactor
dbt-sqlite
tests to usedbt-postgres
by @pankajastro in #1366 - Remove 'dbt-core<1.8.9' pin by @tatiana in #1371
- Remove dependency
eval_type_backport
by @tatiana in #1370 - Enable kubernetes tests for dbt>=1.8 by @pankajastro #1364
- CI Workaround: Pin dbt-core, Disable SQLite Tests, and Correctly Ignore Clone Test to Pass CI by @pankajastro in #1337
- Enable Azure task in the remote store manifest example DAG by @pankajkoti in #1333
- Enable GCP remote manifest task by @pankajastro in #1332
- Add exempt label option in GH action stale job by @pankajastro in #1328
- Add integration test for source node rendering by @pankajastro in #1327
- Fix vulnerability issue on docs dependency by @tatiana in #1313
- Add postgres pod status check for k8s tests in CI by @pankajkoti in #1320
- [CI] Reduce the amount taking to run tests in the CI from 5h to 11min by @tatiana in #1297
- Enable secret detection precommit check by @pankajastro in #1302
- Fix security vulnerability, by not pinning Airflow 2.10.0 by @tatiana in #1298
- Fix Netlify build timeouts by @tatiana in #1294
- Add stalebot to label/close stale PRs and issues by @tatiana in #1288
- Unpin dbt-databricks version by @pankajastro in #1409
- Fix source resource type tests by @pankajastro in #1405
- Increase performance tests models by @tatiana in #1403
- Drop running 1000 models in the CI by @pankajkoti in #1411
- Fix releasing package to PyPI by @tatiana in #1396
- Address review comments on PR 1347 regarding profile_config for ExecutionMode.Docker by @pankajkoti in #1413
- Pre-commit hook updates in #1394, #1373, #1358, #1340, #1331, #1314, #1301
Bug fixes
- Fix
DbtVirtualenvBaseOperator
to use correct virtualenv Python path by @kesompochy in #1252 - Fix displaying dbt docs as menu item in Astro by @tatiana in #1280
- Fix: Replace login by user for clickhouse profile by @petershenri in #1255
Enhancements
- Improve dbt Docs Hosting Debugging -- Update dbt_docs_not_set_up.html by @johnmcochran in #1250
- Minor refactor on VirtualenvOperators & add test for PR #1253 by @tatiana in #1286
Docs
- Add Welcome Section and "What Is Cosmos" Blurb to Home Page by @cmarteepants and @yanmastin-astro in #1251
- Update the URL for sample dbt docs hosted in Astronomer S3 bucket by @pankajkoti in #1283
- Add dedicated scarf tracking pixel to readme by @cmarteepants in #1256
Others
- Update
CODEOWNERS
to track all files by @pankajkoti in #1284 - Fix release after the
raw
rst directive was disabled in PyPI by @tatiana in #1282 - Update issue template
bug.yml
- cosmos version update in the dropdown by @pankajkoti in #1275 - Pre-commit hook updates in #1285, #1274, #1254, #1244
New Features
Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is a first step in this journey and we would really appreciate feedback from the community.
For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental
This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/.
Support using
DatasetAlias
and fix orphaning unreferenced dataset by @tatiana in #1217 #1240Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling
Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153
Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html
Enhancements
- Create single virtualenv when
DbtVirtualenvBaseOperator
hasvirtualenv_dir=None
andis_virtualenv_dir_temporary=True
by @kesompochy in #1200 - Consistently handle build and imports in
cosmos/__init__.py
by @tatiana in #1215 - Add enum constants to init for direct import by @fabiomx in #1184
Bug fixes
- URL encode dataset names to support multibyte characters by @t0momi219 in #1198
- Fix invalid argument (
full_refresh
) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 - Fix
printer_width
arg type inDbtProfileConfigVars
by @jessicaschueler in #1191 - Fix task owner fallback by @jmaicher in #1195
Docs
- Add scarf to readme and docs for website analytics by @cmarteepants in #1221
- Add
virtualenv_dir
param toExecutionConfig
docs by @pankajkoti in #1173 - Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174
- Refactor docs for async mode execution by @pankajkoti in #1241
Others
- Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233
- Fix CI wrt broken coverage upload artifact @pankajkoti in #1210
- Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208
- Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182
- CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237
- Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231
New Features
- Add support for loading manifest from cloud stores using Airflow Object Storage by @pankajkoti in #1109
- Cache
package-lock.yml
file by @pankajastro in #1086 - Support persisting the
LoadMode.VIRTUALENV
directory @LennartKloppenburg and @tatiana in #1079 and #611 - Add support to store and fetch
dbt ls
cache in remote stores by @pankajkoti in #1147 - Add default source nodes rendering by @arojasb3 in #1107
- Add Teradata
ProfileMapping
by @sc250072 in #1077
Enhancements
- Add
DatabricksOauthProfileMapping
profile by @CorsettiS in #1091 - Use
dbt ls
as the default parser whenprofile_config
is provided by @pankajastro in #1101 - Add task owner to dbt operators by @wornjs in #1082
- Extend Cosmos custom selector to support + when using paths and tags by @mvictoria in #1150
- Simplify logging by @dwreeves in #1108
Bug fixes
- Fix Teradata
ProfileMapping
target invalid issue by @sc250072 in #1088 - Fix empty tag in case of custom parser by @pankajastro in #1100
- Fix
dbt deps
ofLoadMode.DBT_LS
should useProjectConfig.dbt_vars
by @tatiana in #1114 - Fix import handling by lazy loading hooks introduced in PR #1109 by @dwreeves in #1132
- Fix Airflow 2.10 regression and add Airflow 2.10 in test matrix by @pankajastro in #1162
Docs
- Fix typo in azure-container-instance docs by @pankajastro in #1106
- Use Airflow trademark as it has been registered by @pankajastro in #1105
Others
- Run some example DAGs in Kubernetes execution mode in CI by @pankajastro in #1127
- Install requirements.txt by default during dev env spin up by @@CorsettiS in #1099
- Remove
DbtGraph.current_version
dead code by @tatiana in #1111 - Disable test for Airflow-2.5 and Python-3.11 combination in CI by @pankajastro in #1124
- Pre-commit hook updates in #1074, #1113, #1125, #1144, #1154, #1167
Bug fixes
- Fix getting temporary AWS credentials with assume_role by @piotrkubicki in #1081
- Fix issue 'No such file or directory' by @tatiana in #1097
Others
- Change Cosmos dev status from alpha to prod by @tatiana in #1098
- Pre-commit hook updates in #1083, #1092
New Features
- Speed up
LoadMode.DBT_LS
by caching dbt ls output in Airflow Variable by @tatiana in #1014 - Support to cache profiles created via
ProfileMapping
by @pankajastro in #1046 - Support for running dbt tasks in AWS EKS in #944 by @VolkerSchiewe
- Add Clickhouse profile mapping by @roadan and @pankajastro in #353 and #1016
- Add node config to TaskInstance Context by @linchun3 in #1044
Bug fixes
- Support partial parsing when cache is disabled by @tatiana in #1070
- Fix disk permission error in restricted env by @pankajastro in #1051
- Add CSP header to iframe contents by @dwreeves in #1055
- Stop attaching log adaptors to root logger to reduce logging costs by @glebkrapivin in #1047
Enhancements
- Support
static_index.html
docs by @dwreeves in #999 - Support deep linking dbt docs via Airflow UI by @dwreeves in #1038
- Add ability to specify host/port for Snowflake connection by @whummer in #1063
Docs
- Fix rendering for env
enable_cache_dbt_ls
by @pankajastro in #1069
Others
- Update documentation for DbtDocs generator by @arjunanan6 in #1043
- Use uv in CI by @dwreeves in #1013
- Cache hatch folder in the CI by @tatiana in #1056
- Change example DAGs to use
example_conn
as opposed toairflow_db
by @tatiana in #1054 - Mark plugin integration tests as integration by @tatiana in #1057
- Ensure compliance with linting rule D300 by using triple quotes for docstrings by @pankajastro in #1049
- Pre-commit hook updates in #1039, #1050, #1064
- Remove duplicates in changelog by @jedcunningham in #1068
Bug fixes
- Bring back
dataset
as a required field for BigQuery profile by @pankajkoti in #1033
Enhancements
- Only run
dbt deps
when there are dependencies by @tatiana and @AlgirdasDubickas in #1030
Docs
- Fix docs so it does not reference non-existing
get_dbt_dataset
by @tatiana in #1034
Bug fixes
- Fix the invocation mode for
ExecutionMode.VIRTUALENV
by @marco9663 in #1023 - Fix Cosmos
enable_cache
setting by @tatiana in #1025 - Make
GoogleCloudServiceAccountDictProfileMapping
dataset profile arg optional by @oliverrmaa and @pankajastro in #839 and #1017 - Athena profile mapping set
aws_session_token
in profile only if it exists by @pankajastro in #1022
Others
- Update dbt and Airflow conflicts matrix by @tatiana in #1026
- Enable Python 3.12 unittest by @pankajastro in #1018
- Improve error logging in
DbtLocalBaseOperator
by @davidsteinar in #1004 - Add GitHub issue templates for bug reports and feature request by @pankajkoti in #1009
- Add more fields in bug template to reduce turnaround in issue triaging by @pankajkoti in #1027
- Fix
dev/Dockerfile
+ Adduv pip install
for faster build time by @dwreeves in #997 - Drop support for Airflow 2.3 by @pankajkoti in #994
- Update Astro Runtime image by @RNHTTR in #988 and #989
- Enable ruff F linting by @pankajastro in #985
- Move Cosmos Airflow configuration to settings.py by @pankajastro in #975
- Fix CI Issues by @tatiana in #1005
- Pre-commit hook updates in #1000, #1019
Bug fixes
- Fix manifest testing behavior by @chris-okorodudu in #955
- Handle ValueError when unpacking partial_parse.msgpack by @tatiana in #972
Others
- Enable pre-commit run and fix type-check job by @pankajastro in #957
- Clean databricks credentials in test/CI by @tatiana in #969
- Update CODEOWNERS by @tatiana in #969 x
- Update emeritus contributors list by @tatiana in #961
- Promote @dwreeves to committer by @tatiana in #960
- Pre-commit hook updates in #956
Features
- Add dbt docs natively in Airflow via plugin by @dwreeves in #737
- Add support for
InvocationMode.DBT_RUNNER
for local execution mode by @jbandoro in #850 - Support partial parsing to render DAGs faster when using
ExecutionMode.LOCAL
,ExecutionMode.VIRTUALENV
andLoadMode.DBT_LS
by @dwreeves in #800 - Improve performance by 22-35% or more by caching partial parse artefact by @tatiana in #904
- Add Azure Container Instance as Execution Mode by @danielvdende in #771
- Add dbt build operators by @dylanharper-qz in #795
- Add dbt profile config variables to mapped profile by @ykuc in #794
- Add more template fields to
DbtBaseOperator
by @dwreeves in #786 - Add
pip_install_options
argument to operators by @octiva in #808
Bug fixes
- Make
PostgresUserPasswordProfileMapping
schema argument optional by @FouziaTariq in #683 - Fix
folder_dir
not showing on logs forDbtDocsS3LocalOperator
by @PrimOox in #856 - Improve
dbt ls
parsing resilience to missing tags/config by @tatiana in #859 - Fix
operator_args
modified in place in Airflow converter by @jbandoro in #835 - Fix Docker and Kubernetes operators execute method resolution by @jbandoro in #849
- Fix
TrinoBaseProfileMapping
required parameter for non method authentication by @AlexandrKhabarov in #921 - Fix global flags for lists by @ms32035 in #863
- Fix
GoogleCloudServiceAccountDictProfileMapping
when getting values from the Airflow connectionextra__
keys by @glebkrapivin in #923 - Fix using the dag as a keyword argument as
specific_args_keys
in DbtTaskGroup by @tboutaour in #916 - Fix ACI integration (
DbtAzureContainerInstanceBaseOperator
) by @danielvdende in #872 - Fix setting dbt project dir to the tmp dir by @dwreeves in #873
- Fix dbt docs operator to not use
graph.gpickle
file when--no-write-json
is passed by @dwreeves in #883 - Make Pydantic a required dependency by @pankajkoti in #939
- Gracefully error if users try to
emit_datasets
withAirflow 2.9.0
or2.9.1
by @tatiana in #948 - Fix parsing tests that have no parents in #933 by @jlaneve
- Correct
root_path
in partial parse cache by @pankajkoti in #950
Docs
- Fix docs homepage link by @jlaneve in #860
- Fix docs
ExecutionConfig.dbt_project_path
by @jbandoro in #847 - Fix typo in MWAA getting started guide by @jlaneve in #846
- Fix typo related to exporting docs to GCS by @tboutaour in #922
- Improve partial parsing docs by @tatiana in #898
- Improve docs for datasets for airflow >= 2.4 by @SiddiqueAhmad in #879
- Improve test behaviour docs to highlight
warning
feature in thevirtualenv
mode by @mc51 in #910 - Fix docs typo by @SiddiqueAhmad in #917
- Improve Astro docs by @RNHTTR in #951
Others
- Add performance integration tests by @jlaneve in #827
- Enable
append_env
inoperator_args
by default by @tatiana in #899 - Change default
append_env
behaviour depending on CosmosExecutionMode
by @pankajkoti and @pankajastro in #954 - Expose the
dbt
graph in theDbtToAirflowConverter
class by @tommyjxl in #886 - Improve dbt docs plugin rendering padding by @dwreeves in #876
- Add
connect_retries
to databricks profile to fix expensive integration failures by @jbandoro in #826 - Add import sorting (isort) to Cosmos by @jbandoro in #866
- Add Python 3.11 to CI/tests by @tatiana and @jbandoro in #821, #824 and #825
- Fix failing
test_created_pod
forapache-airflow-providers-cncf-kubernetes
after v8.0.0 update by @jbandoro in #854 - Extend
DatabricksTokenProfileMapping
test to include session properties by @tatiana in #858 - Fix broken integration test uncovered from Pytest 8.0 update by @jbandoro in #845
- Add Apache Airflow 2.9 to the test matrix by @tatiana in #940
- Replace deprecated
DummyOperator
byEmptyOperator
if Airflow >=2.4.0 by @tatiana in #900 - Improve logs to troubleshoot issue in 1.4.0a2 with astro-cli by @tatiana in #947
- Fix issue when publishing a new release to PyPI by @tatiana in #946
- Pre-commit hook updates in #820, #834, #843 and #852, #890, #896, #901, #905, #908, #919, #931, #941
Bug fixes
- Fix: ensure
DbtGraph.update_node_dependency
is called for all load methods by @jbandoro in #803 - Fix: ensure operator
execute
method is consistent across all execution base subclasses by @jbandoro in #805 - Fix custom selector when
test
node has nodepends_on
values by @tatiana in #814 - Fix forwarding selectors to test task when using
TestBehavior.AFTER_ALL
by @tatiana in #816
Others
- Docs: Remove incorrect docstring from
DbtLocalBaseOperator
by @jakob-hvitnov-telia in #797 - Add more logs to troubleshoot custom selector by @tatiana in #809
- Fix OpenLineage integration documentation by @tatiana in #810
- Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806
- Use Airflow constraint file for test environment setup by @jbandoro in #812
- pre-commit updates in #799, #807
Bug fixes
- Fix disable event tracking throwing error by @jbandoro in #784
- Fix support for string path for
LoadMode.DBT_LS_FILE
and docs by @flinz in #788 - Remove stack trace to disable unnecessary K8s error by @tatiana in #790
Others
- Update examples to use the astro-runtime 10.0.0 by @RNHTTR in #777
- Docs: add missing imports for mwaa getting started by @Benjamin0313 in #792
- Refactor common executor constructors with test coverage by @jbandoro in #774
- pre-commit updates in #789
Features
- Add new parsing method
LoadMode.DBT_LS_FILE
by @woogakoki in #733 (documentation). - Add support to select using (some) graph operators when using
LoadMode.CUSTOM
andLoadMode.DBT_MANIFEST
by @tatiana in #728 (documentation) - Add support for dbt
selector
arg for DAG parsing by @jbandoro in #755 (documentation). - Add
ProfileMapping
for Vertica by @perttus in #540, #688 and #741 (documentation). - Add
ProfileMapping
for Snowflake encrypted private key path by @ivanstillfront in #608 (documentation). - Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649
- Add
DbtDocsGCSOperator
for uploading dbt docs to GCS by @jbandoro in #616, (documentation). - Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648 (documentation).
- Add operator_args
full_refresh
as a templated field by @joppevos in #623 - Expose environment variables and dbt variables in
ProjectConfig
by @jbandoro in #735 (documentation). - Support disabling event tracking when using Cosmos profile mapping by @jbandoro in #768 (documentation).
Enhancements
- Make Pydantic an optional dependency by @pixie79 in #736
- Create a symbolic link to
dbt_packages
whendbt_deps
is False when usingLoadMode.DBT_LS
by @DanMawdsleyBA in #730 - Add
aws_session_token
for Athena mapping by @benjamin-awd in #663 - Retrieve temporary credentials from
conn_id
for Athena by @octiva in #758 - Extend
DbtDocsLocalOperator
with static flag by @joppevos in #759
Bug fixes
- Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in #772
Others
- Replace flake8 for Ruff by @joppevos in #743
- Reduce code complexity to 8 by @joppevos in #738
- Speed up integration tests by @jbandoro in #732
- Fix README quickstart link in by @RNHTTR in #776
- Add package location to work with hatchling 1.19.0 by @jbandoro in #761
- Fix type check error in
DbtKubernetesBaseOperator.build_env_args
by @jbandoro in #766 - Improve
DBT_MANIFEST
documentation by @dwreeves in #757 - Update conflict matrix between Airflow and dbt versions by @tatiana in #731 and #779
- pre-commit updates in #775, #770, #762
Bug fixes
- Fix running models that use alias while supporting dbt versions by @binhnq94 in #662
- Make
profiles_yml_path
optional forExecutionMode.DOCKER
andKUBERNETES
by @MrBones757 in #681 - Prevent overriding dbt profile fields with profile args of "type" or "method" by @jbandoro in #702
- Fix
LoadMode.DBT_LS
fail when dbt outputsWarnErrorOptions
by @adammarples in #692 - Add support for env vars in
RenderConfig
for dbt ls parsing by @jbandoro in #690 - Add support for Kubernetes
on_warning_callback
by @david-mag in #673 - Fix
ExecutionConfig.dbt_executable_path
to usedefault_factory
by @jbandoro in #678
Others
- Docs fix: example DAG in the README and docs/index by @tatiana in #705
- Docs improvement: highlight DAG examples in README by @iancmoritz and @jlaneve in #695
Bug fixes
- Store
compiled_sql
even when task fails by @agreenburg in #671 - Refactor
LoadMethod.LOCAL
to use symlinks instead of copying directory by @jbandoro in #660 - Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666
- Fix installing deps when using
profile_mapping
&ExecutionMode.LOCAL
by @joppevos in #659
Others
- Docs: add execution config to MWAA code example by @ugmuka in #674
- Docs: highlight DAG examples in docs by @iancmoritz and @jlaneve in #695
Bug fix
- Fix reusing config across TaskGroups/DAGs by @tatiana in #664
Bug fixes
- Support
ProjectConfig.dbt_project_path = None
& different paths for Rendering and Execution by @MrBones757 in #634 - Fix adding test nodes to DAGs built using
LoadMethod.DBT_MANIFEST
andLoadMethod.CUSTOM
by @edgga in #615
Others
- Add pre-commit hook for McCabe max complexity check and fix errors by @jbandoro in #629
- Update contributing docs for running integration tests by @jbandoro in #638
- Fix CI issue running integration tests by @tatiana in #640 and #644
- pre-commit updates in #637
Bug fixes
- Resolve errors occurring when
dbt_project_path
is str and partial supportdbt_project_path=None
by @MrBones757 in #605 - Fix running dbt tests that depend on multiple models (support
--indirect-selection buildable
) by @david-mag in #613 - Add tests to sources, snapshots and seeds when using
TestBehavior.AFTER_EACH
by @tatiana in #599 - Fix custom selector when select has a subset of tags of the models' tags by @david-mag in #606
- Fix
LoadMode.AUTOMATIC
behaviour to useLoadMode.DBT_LS
whenProfileMapping
is used by @tatiana in #625 - Fix failure if
openlineage-common
raises a jinja exception by @tatiana in #626
Others
- Update contributing guide docs by @raffifu in #591
- Remove unnecessary stack trace from Cosmos initialization by @tatiana in #624
- Fix running test that validates manifest-based DAGs by @tatiana in #619
- pre-commit updates in #604 and #621
Features
- Add support to model versioning available since dbt 1.6 by @binhnq94 in #516
- Add AWS Athena profile mapping by @benjamin-awd in #578
- Support customizing how dbt nodes are converted to Airflow by @tatiana in #503
- Make the arg
dbt_project_path
in theProjectConfig
optional by @MrBones757 in #581
Bug fixes
- Fix Cosmos custom selector to support filtering a single model by @jlaneve and @harels in #576
- Fix using
GoogleCloudServiceAccountDictProfileMapping
together withLoadMethod.DBT_LS
by @joppevos in #587 - Fix using the
full_refresh
argument in projects that contain tests by @EgorSemenov and @tatiana in #590 - Stop creating symbolic links for
dbt_packages
(solvesLocalExecutor
concurrency issue) by @tatiana in #600
Others
- Docs: add reference to original Jaffle Shop project by @erdos2n in #583
- Docs: retries & note about DagBag error by @TJaniF in #592
- pre-commit updates in #575 and #585
Bug fixes
- Only create task group and test task only if the model has a test by @raffifu in #543
- Fix parsing test nodes when using the custom load method (LoadMethod.CUSTOM) by @raffifu in #563
- Fix
DbtTestOperator
when test does not havetest_metadata
by @javihernovoa and @tatiana in #565 - Support dbt 1.6 and apache-airflow-providers-cncf-kubernetes 7.3.0 by @tatiana in #564
Bug fixes
- Fix using
ExecutionMode.KUBERNETES
by @pgoslatara and @tatiana in #554 - Add support to
apache-airflow-providers-cncf-kubernetes < 7.4.0
by @tatiana in #553 - Fix
on_warning_callback
behaviour onDbtTestLocalOperator
by @edgga, @marco9663 and @tatiana in #558 - Use
returncode
instead ofstderr
to determine dbt graph loading errors by @cliff-lau-cloverhealth in #547 - Improve error message in
config.py
by @meyobagero in #532 - Fix
DbtTestOperator
when test does not havetest_metadata
by @tatiana in #558 - Fix
target-path
not specified issue indbt-project.yml
by @tatiana in #533
Others
- Docs: add reference links to dbt and Airflow columns by @TJaniF in #542
- pre-commit updates #552 and #546
Bug fixes
- Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526
- Fix Rust dependency for Windows users by @tatiana in #526
- Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
- Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529
- Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524
- Fix rendering vars in
DbtModel
when usingLoadMode.CUSTOM
by @dojinkimm in #502
Others
- Docs: add documentation comparing Airflow and dbt concepts by @tatiana in #523.
- Update PyPI project links by @tatiana in #528
- pre-commit updates
Features
- Support dbt global flags (via dbt_cmd_global_flags in operator_args) by @tatiana in #469
- Support parsing DAGs when there are no connections by @jlaneve in #489
Enhancements
- Hide sensitive field when using BigQuery keyfile_dict profile mapping by @jbandoro in #471
- Consistent Airflow Dataset URIs, inlets and outlets with Openlineage package by @tatiana in #485. Read more.
- Refactor
LoadMethod.DBT_LS
to run from a temporary directory with symbolic links by @tatiana in #488 - Run
dbt deps
when usingLoadMethod.DBT_LS
by @DanMawdsleyBA in #481 - Update Cosmos log color to purple by @harels in #494
- Change operators to log
dbt
commands output as opposed to recording to XCom by @tatiana in #513
Bug fixes
- Fix bug on select node add exclude selector subset ids logic by @jensenity in #463
- Refactor dbt ls to run from a temporary directory, to avoid Read-only file system errors during DAG parsing, by @tatiana in #414
- Fix profile_config arg in DbtKubernetesBaseOperator by @david-mag in #505
- Fix SnowflakePrivateKeyPemProfileMapping private_key reference by @nacpacheco in #501
- Fix incorrect temporary directory creation in VirtualenvOperator init by @tatiana in #500
- Fix log propagation issue by @tatiana in #498
- Fix PostgresUserPasswordProfileMapping to retrieve port from connection by @jlneve in #511
Others
- Docs: Fix RenderConfig load argument by @jbandoro in #466
- Enable CI integration tests from external forks by @tatiana in #458
- Improve CI tests runtime by @tatiana in #457
- Change CI to run coverage after tests pass by @tatiana in #461
- Fix forks code revision in code coverage by @tatiana in #472
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #467
- Drop support to Python 3.7 in the CI test matrix by @harels in #490
- Add Airflow 2.7 to the CI test matrix by @tatiana in #487
- Add MyPy type checks to CI since we exceeded pre-commit disk quota usage by @tatiana in #510
Enhancements
- Improve logs to include astornomer-cosmos identifier by @tatiana in #450
- Support OAuth authentication for Big Query by @MonideepDe in #431
Bug fixes
- Fix selector for config tags by @javihernovoa in #441
- Fix BigQuery keyfile_dict mapping for connection created from webserver UI by @jbandoro in #449
Others
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #446
- Resolve MyPy errors when adding Airflow pre-commit dependency by @abhi12mohan in #434
- Initial release, with the following 6 workflow Operators/Parsers:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
DBTTestOperator |
from cosmos.providers.dbt.core.operators import DBTBaseOperator |
N/A |
DBTSeedOperator |
from cosmos.providers.dbt.core.operators import DBTSeedOperator |
Example DAG |
DBTRunOperator |
from cosmos.providers.dbt.core.operators import DBTRunOperator |
N/A |
DBTTestOperator |
from cosmos.providers.dbt.core.operators import DBTTestOperator |
N/A |
DbtDag |
from cosmos.providers.dbt.core.dag import DbtDag |
Example DAG |
DbtTaskGroup |
from cosmos.providers.dbt.core.dag import DbtTaskGroup |
Example DAG |