Add Deptry, split dev into dev and docs recipes, address bandit…

… security issues (#1806)

### What kind of change does this PR introduce?

* Adds the `deptry` package to the linters for performing dependency
analyses (unused or missing)
* Pinned a few linting dependencies
* Splits the `dev` dependencies into `dev` and `docs` (both can be
installed with `all`)
* Enables `flake8-bandit` like checks in `ruff`, fixes a few
security-related checks
* `xclim.testing.utils` now have more secure URL auditing checks

### Does this PR introduce a breaking change?

Yes, a new dependency has been added (`deptry`).

Also, `xclim` no longer bundles the documentation dependencies with the
`dev` recipe. Splitting this reduces the number of dependencies needed
to run the tests and also helps us know which dependencies are needed
and where (determined with the aid of `deptry`).

### Other information:

https://deptry.com/
https://docs.astral.sh/ruff/rules/#flake8-bandit-s

.pre-commit-config.yaml

@@ -3,7 +3,7 @@ default_language_version:
 
 repos:
   - repo: https://github.com/asottile/pyupgrade
-    rev: v3.15.2
+    rev: v3.16.0
     hooks:
       - id: pyupgrade
         args: ['--py39-plus']
@@ -41,16 +41,16 @@ repos:
     hooks:
       - id: isort
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.4.7
+    rev: v0.4.10
     hooks:
       - id: ruff
   - repo: https://github.com/pylint-dev/pylint
-    rev: v3.2.2
+    rev: v3.2.4
     hooks:
       - id: pylint
         args: [ '--rcfile=.pylintrc.toml', '--errors-only', '--jobs=0', '--disable=import-error' ]
   - repo: https://github.com/pycqa/flake8
-    rev: 7.0.0
+    rev: 7.1.0
     hooks:
       - id: flake8
         additional_dependencies: [ 'flake8-rst-docstrings ']
@@ -64,7 +64,7 @@ repos:
     hooks:
       - id: nbqa-pyupgrade
         args: [ '--py39-plus' ]
-        additional_dependencies: [ 'pyupgrade==3.15.2' ]
+        additional_dependencies: [ 'pyupgrade==3.16.0' ]
       - id: nbqa-black
         additional_dependencies: [ 'black==24.4.2' ]
       - id: nbqa-isort
@@ -87,11 +87,11 @@ repos:
       - id: codespell
         additional_dependencies: [ 'tomli' ]
   - repo: https://github.com/gitleaks/gitleaks
-    rev: v8.18.3
+    rev: v8.18.4
     hooks:
       - id: gitleaks
   - repo: https://github.com/python-jsonschema/check-jsonschema
-    rev: 0.28.4
+    rev: 0.28.6
     hooks:
       - id: check-github-workflows
       - id: check-readthedocs

CHANGES.rst

@@ -9,18 +9,26 @@ Contributors to this version: Trevor James Smith (:user:`Zeitsperre`).
 New features and enhancements
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 * Added the `op` keyword to the `growing_season_{start|end}` indices and indicators, allowing for customizable threshold operators using `indices.generic.compare()`. (:issue:`1794`, :pull:`1796`).
+* `xclim` now separates the optional dependencies into `dev` and `docs` recipes. Both can be installed with the `all` option (`$ python -m pip install xclim[all]`). (:pull:`1806`).
 
 Bug fixes
 ^^^^^^^^^
 * Clarified a typo in the docstring formula for `xclim.indices.growing_season_length`. (:pull:`1796`).
 
 Internal changes
 ^^^^^^^^^^^^^^^^
+* `netcdf4` has been pinned below v1.7 for test stability reasons. (:pull:`1791`).
+* `flake8-bandit`-like checks have been enabled via `ruff`, with fixes for a few security-related issues. (:pull:`1806`).
+* ``xclim.testing.utils`` now employs more secure URL auditing checks. (:pull:`1806`).
+
+CI changes
+^^^^^^^^^^
 * GitHub repository now uses Rulesets for branch protection. (:pull:`1790`).
 * Version bumping and project triage is now handled by the Ouranos Helper GitHub App. (:pull:`1790`).
-* `netcdf4` has been pinned below v1.7 for test stability reasons. (:pull:`1791`).
 * `bump-my-version` has been updated to v0.23.0. (:pull:`1790`).
 * The Ouranos Helper GitHub App now provides verified commits. (:issue:`1811`, :pull:`1812`).
+* Added the `deptry <https://github.com/fpgmaas/deptry>`_ package to the `dev` linter tools and linting workflows for performing dependency analyses. (:pull:`1806`).
+* Several linting tools have been updated to the latest versions and pinned. (:pull:`1806`).
 
 v0.50.0 (2024-06-17)
 --------------------

CI/requirements_ci.txt

@@ -1,5 +1,6 @@
 bump-my-version==0.24.1
 coveralls==4.0.1
+deptry==0.16.1
 flit==3.9.0
 pip==24.1.1
 pylint==3.2.4

CONTRIBUTING.rst

@@ -122,7 +122,7 @@ Ready to contribute? Here's how to set up `xclim` for local development.
 #. Create a development environment. We recommend using ``conda``::
 
     $ conda create -n xclim python=3.10 --file=environment.yml
-    $ python -m pip install -e ".[dev]"
+    $ python -m pip install -e --no-deps .
 
 #. Create a branch for local development::
 
@@ -174,7 +174,7 @@ Ready to contribute? Here's how to set up `xclim` for local development.
    .. warning::
     Starting from `xclim` v0.46.0, when running tests with `tox`, any `pytest` markers passed to `pyXX` builds (e.g. `-m "not slow"`) must be passed to `tox` directly. This can be done as follows::
 
-        $ tox -e py38 -- -m "not slow"
+        $ tox -e py310 -- -m "not slow"
 
     The exceptions to this rule are:
       `notebooks_doctests`: this configuration does not pass test  markers to its `pytest` call.

Makefile

@@ -62,6 +62,7 @@ lint: ## check style with flake8 and black
 	blackdoc --check --exclude=xclim/indices/__init__.py xclim
 	blackdoc --check docs
 	codespell xclim tests docs
+	deptry .
 	yamllint --config-file=.yamllint.yaml xclim
 
 test: ## run tests quickly with the default Python
@@ -115,7 +116,7 @@ install: clean ## install the package to the active Python's site-packages
 	python -m pip install --no-user .
 
 develop: clean ## install the package and development dependencies in editable mode to the active Python's site-packages
-	python -m pip install --no-user --editable ".[dev]"
+	python -m pip install --no-user --editable ".[dev,docs]"
 
 upstream: clean develop ## install the GitHub-based development branches of dependencies in editable mode to the active Python's site-packages
 	python -m pip install --no-user --requirement requirements_upstream.txt

docs/installation.rst

@@ -110,7 +110,7 @@ Afterwards, `SBCK` can be installed from PyPI using `pip`:
 .. _SBCK: https://github.com/yrobink/SBCK
 .. _Eigen3: https://eigen.tuxfamily.org/index.php
 
-From sources
+From Sources
 ------------
 
 .. warning::
@@ -132,11 +132,11 @@ Or download the `tarball`_:
 
     $ curl -OL https://github.com/Ouranosinc/xclim/tarball/main
 
-Once you have extracted a copy of the source, you can install it with pip:
+Once you have extracted a copy of the source, you can install it with `pip`_:
 
 .. code-block:: shell
 
-    $ python -m pip install -e ".[dev]"
+    $ python -m pip install -e ".[all]"
 
 Alternatively, you can also install a local development copy via `flit`_:
 
@@ -148,13 +148,13 @@ Alternatively, you can also install a local development copy via `flit`_:
 .. _tarball: https://github.com/Ouranosinc/xclim/tarball/main
 .. _flit: https://flit.pypa.io/en/stable
 
-Creating a Conda environment
+Creating a Conda Environment
 ----------------------------
 
 To create a conda environment including `xclim`'s dependencies and several optional libraries (notably: `clisops`, `eigen`, `sbck`, and `flox`) and development dependencies, run the following command from within your cloned repo:
 
 .. code-block:: console
 
-    $ conda env create -n my_xclim_env python=3.8 --file=environment.yml
+    $ conda env create -n my_xclim_env python=3.10 --file=environment.yml
     $ conda activate my_xclim_env
-    (my_xclim_env) $ python -m pip install -e .
+    (my_xclim_env) $ python -m pip install -e --no-deps .

environment.yml

@@ -35,13 +35,14 @@ dependencies:
   - codespell ==2.3.0
   - coverage >=7.5.0
   - coveralls >=4.0.0
+  - deptry ==0.16.1
   - distributed >=2.0
   - filelock
   - flake8 >=7.0.0
   - flake8-rst-docstrings
   - flit >=3.9.0
   - furo >=2023.9.10
-  - h5netcdf
+  - h5netcdf >=1.3.0
   - ipykernel
   - ipython
   - isort ==5.13.2
@@ -50,21 +51,21 @@ dependencies:
   - nbconvert <7.14  # Pinned due to directive errors in sphinx. See: https://github.com/jupyter/nbconvert/issues/2092
   - nbqa
   - nbsphinx
-  - nbval
+  - nbval >=0.11.0
   - nc-time-axis
   - netCDF4 >=1.4,<1.7
   - notebook
   - pandas-stubs
   - platformdirs
   - pooch
   - pre-commit >=3.7
-  - pybtex
+  - pybtex >=0.24.0
   - pylint >=3.1
   - pytest <8.0  # Pinned due to breakage with xdoctest. See: https://github.com/Erotemic/xdoctest/issues/151
   - pytest-cov
   - pytest-socket
   - pytest-xdist >=3.2
-  - ruff >=0.4.0
+  - ruff >=0.4.10
   - sphinx >=7.0.0
   - sphinx-autobuild >=2024.4.16
   - sphinx-autodoc-typehints

pyproject.toml

@@ -43,6 +43,7 @@ dependencies = [
   "numpy>=1.20.0,<2.0.0",
   "pandas>=2.2",
   "pint>=0.10,<0.24",
+  "platformdirs >=3.2",
   "pyarrow", # Strongly encouraged for pandas v2.2.0+
   "pyyaml",
   "scikit-learn>=0.21.3",
@@ -57,37 +58,40 @@ dev = [
   # Dev tools and testing
   "black[jupyter] ==24.4.2",
   "blackdoc ==0.3.9",
-  "bump-my-version >=0.23.0",
+  "bump-my-version ==0.23.0",
   "codespell ==2.3.0",
   "coverage[toml] >=7.5.0",
   "coveralls >=4.0.0",
-  "flake8 >=7.0.0",
+  "deptry ==0.16.1",
+  "flake8 >=7.1.0",
   "flake8-rst-docstrings",
-  "h5netcdf",
+  "h5netcdf>=1.3.0",
   "ipython",
   "isort ==5.13.2",
   "mypy",
   "nbconvert <7.14", # Pinned due to directive errors in sphinx. See: https://github.com/jupyter/nbconvert/issues/2092
   "nbqa >=1.8.2",
-  "nbval",
+  "nbval >=0.11.0",
   "netCDF4 >=1.4,<1.7",
   "pandas-stubs >=2.2",
   "platformdirs >=3.2",
+  "pooch",
   "pre-commit >=3.7",
-  "pybtex",
-  "pylint >=3.1",
+  "pylint >=3.2.4",
   "pytest <8.0", # Pinned due to breakage with xdoctest. See: https://github.com/Erotemic/xdoctest/issues/151
   "pytest-cov",
   "pytest-socket",
   "pytest-xdist[psutil] >=3.2",
-  "ruff >=0.4.0",
+  "ruff >=0.4.10",
   "tokenize-rt",
   "tox >=4.15.1",
   # "tox-conda",  # Will be added when a tox@v4.0+ compatible plugin is released.
   "tox-gh >=1.3.1",
   "vulture ==2.11",
   "xdoctest",
-  "yamllint ==1.35.1",
+  "yamllint ==1.35.1"
+]
+docs = [
   # Documentation and examples
   "distributed >=2.0",
   "furo >=2023.9.10",
@@ -96,6 +100,7 @@ dev = [
   "nbsphinx",
   "nc-time-axis",
   "pooch",
+  "pybtex >=0.24.0",
   "sphinx >=7.0.0",
   "sphinx-autobuild >=2024.4.16",
   "sphinx-autodoc-typehints",
@@ -105,6 +110,7 @@ dev = [
   "sphinxcontrib-bibtex",
   "sphinxcontrib-svg2pdfconverter[Cairosvg]"
 ]
+all = ["xclim[dev]", "xclim[docs]"]
 
 [project.scripts]
 xclim = "xclim.cli:cli"
@@ -161,6 +167,20 @@ ignore-words-list = "absolue,bloc,bui,callendar,degreee,environnement,hanel,infe
 relative_files = true
 omit = ["tests/*.py"]
 
+[tool.deptry]
+extend_exclude = ["docs"]
+ignore_notebooks = true
+pep621_dev_dependency_groups = ["all", "dev", "docs"]
+
+[tool.deptry.package_module_name_map]
+"scikit-learn" = "sklearn"
+"pyyaml" = "yaml"
+
+[tool.deptry.per_rule_ignores]
+DEP001 = ["SBCK"]
+DEP002 = ["bottleneck", "pyarrow"]
+DEP004 = ["matplotlib", "pytest_socket"]
+
 [tool.flit.sdist]
 include = [
   "AUTHORS.rst",
@@ -253,10 +273,12 @@ line-length = 150
 target-version = "py39"
 exclude = [
   ".git",
-  "docs",
   "build",
   ".eggs"
 ]
+extend-include = [
+  "*.ipynb" # Include notebooks
+]
 
 [tool.ruff.format]
 line-ending = "auto"
@@ -277,6 +299,7 @@ select = [
   "E", # pycodestyle errors
   "F", # pyflakes
   "N802", # invalid-function-name
+  "S", # bandit
   "W" # pycodestyle warnings
 ]
 
@@ -303,7 +326,7 @@ max-complexity = 20
 
 [tool.ruff.lint.per-file-ignores]
 "docs/*.py" = ["D100", "D101", "D102", "D103"]
-"tests/**/*test*.py" = ["D100", "D101", "D102", "D103", "N802"]
+"tests/**/*test*.py" = ["D100", "D101", "D102", "D103", "N802", "S101"]
 "xclim/**/__init__.py" = ["F401", "F403"]
 "xclim/core/indicator.py" = ["D214", "D405", "D406", "D407", "D411"]
 "xclim/core/locales.py" = ["E501", "W505"]

tests/test_sdba/test_base.py

@@ -29,7 +29,7 @@ def test_param_class():
     )
 
     s = jsonpickle.encode(obj)
-    obj2 = jsonpickle.decode(s)
+    obj2 = jsonpickle.decode(s)  # noqa: S301
     assert obj.parameters == obj2.parameters
 
 

tests/test_testing_utils.py

@@ -3,6 +3,7 @@
 import platform
 import sys
 from pathlib import Path
+from urllib.error import URLError
 
 import numpy as np
 import pytest
@@ -139,24 +140,26 @@ def test_release_notes_file_not_implemented(self, tmp_path):
 class TestTestingFileAccessors:
     def test_unsafe_urls(self):
         with pytest.raises(
-            ValueError, match="GitHub URL not safe: 'ftp://domain.does.not.exist/'."
+            ValueError, match="GitHub URL not secure: 'ftp://domain.does.not.exist/'."
         ):
             utilities.open_dataset(
                 "doesnt_exist.nc", github_url="ftp://domain.does.not.exist/"
             )
 
         with pytest.raises(
-            ValueError, match="OPeNDAP URL not safe: 'ftp://domain.does.not.exist/'."
+            OSError,
+            match="OPeNDAP file not read. Verify that the service is available: "
+            "'https://seemingly.trustworthy.com/doesnt_exist.nc'",
         ):
             utilities.open_dataset(
-                "doesnt_exist.nc", dap_url="ftp://domain.does.not.exist/"
+                "doesnt_exist.nc", dap_url="https://seemingly.trustworthy.com/"
             )
 
-    def test_bad_opendap_url(self):
+    def test_malicious_urls(self):
         with pytest.raises(
-            OSError,
-            match="OPeNDAP file not read. Verify that the service is available.",
+            URLError,
+            match="urlopen error OPeNDAP URL is not well-formed: 'doesnt_exist.nc'",
         ):
             utilities.open_dataset(
-                "doesnt_exist.nc", dap_url="https://dap.service.does.not.exist/"
+                "doesnt_exist.nc", dap_url="Robert'); DROP TABLE STUDENTS; --"
             )