spin test does not work with an editable install

[oscarbenjamin]

After an editable install spin install running spin test fails:

$ spin test
Invoking `build` prior to running tests:
Editable install of same source detected; skipping build
Editable install of same source directory detected; not setting PYTHONPATH
 $ /home/oscar/.pyenv/versions/3.11.3/envs/cython_coverage_demo-3.11.git/bin/python3.11 -P -c 'import stuff'
 $ /home/oscar/.pyenv/versions/3.11.3/envs/cython_coverage_demo-3.11.git/bin/python3.11 -P -m pytest stuff
======================================== test session starts ========================================
platform linux -- Python 3.11.3, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/oscar/current/active/cython_coverage_demo
configfile: pyproject.toml
plugins: cov-5.0.0
collected 0 items                                                                                   

======================================= no tests ran in 0.01s =======================================
ERROR: file or directory not found: stuff

It is trying to run pytest stuff when the command should be pytest --pyargs stuff:

$ pytest --pyargs stuff
======================================== test session starts ========================================
platform linux -- Python 3.11.3, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/oscar/current/active/cython_coverage_demo
configfile: pyproject.toml
plugins: cov-5.0.0
collected 1 item                                                                                    

src/stuff/test_thing.py .                                                                     [100%]

========================================= 1 passed in 0.11s =========================================
$ spin run pytest --pyargs stuff
======================================== test session starts ========================================
platform linux -- Python 3.11.3, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/oscar/current/active/cython_coverage_demo
configfile: pyproject.toml
plugins: cov-5.0.0
collected 1 item                                                                                    

src/stuff/test_thing.py .                                                                     [100%]

========================================= 1 passed in 0.11s =========================================

[stefanv]

I am not sure this is right. The tests work with the example_pkg, and it works with skimage. If I change it to using pyargs, then the tests are no longer picked up for skimage.

[oscarbenjamin]

The repo I used for demonstration is this one:
https://github.com/oscarbenjamin/cython_coverage_demo

In the demo the tests are in-package but the package uses src-layout (unlike skimage).

[stefanv]

Ah, yes, that would explain it. Curious why --pyargs modulename manages to find the tests, though.

[oscarbenjamin]

I think that for in-package tests --pyargs is the proper way to find them via the module search path (like -m mod.test). Then spin's PYTHONPATH manipulation is what makes them discoverable. Usual pytest discovery rules apply:
https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#conventions-for-python-test-discovery

If you are using spin run pytest skimage then I think that means that pytest is doing all kinds of weirdness to combine the test modules in ./skimage with the modules in ./build-install:
https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#test-package-name

[stefanv]

Yeah, so it needs to find the test files in the ./skimage directory, but the library from the editable install.

It sounds like we may need a setting to determine where pytest should search for tests: ./src vs ./packagename?

[oscarbenjamin]

Why can't it find the test files from the editable install?

[oscarbenjamin]

Why can't it find the test files from the editable install?

The reason that I ask this is because if the in-package tests are found via sys.path then there is no need to distinguish between a src vs non-src layout: if the module is importable then so are its tests. There is then no need to provide configuration options for the different schemes because it is already taken care of.

There is however a need to distinguish between in-package vs out-of-package tests. The out-of-package tests always need to be picked up by file system path rather than Python's import paths.

[stefanv]

I was referring to out-of-package tests, or tests that are not installed alongside the package (which is not that atypical).

[oscarbenjamin]

So which one does skimage use? It looks to me like it has in-package tests that are installed:
https://github.com/scikit-image/scikit-image/blob/4cc5b3dc0e3929c5ab9014d1a6031868eecd97eb/skimage/color/tests/meson.build#L1-L13

[stefanv]

Yes, AFAIK we do install the tests, but for me pytest --pyargs skimage does not pick them up.

[oscarbenjamin]

Yes, AFAIK we do install the tests, but for me pytest --pyargs skimage does not pick them up.

Maybe that is something that needs to be fixed in pytest.

What happens with:

pytest --pyargs skimage
pytest --pyargs skimage.color
pytest --pyargs skimage.color.tests
pytest --pyargs skimage.color.tests.test_adapt_rgb

[stefanv]

Only the last command manages to find any tests.

[oscarbenjamin]

If I try this:

pip install sympy pytest hypothesis
pytest --pyargs sympy

then as far as I can tell that recursively finds all tests in all test_* files in all tests directories.

[stefanv]

Yes, that works for me too.

[lesteve]

Can you make sure you are using meson-python latest release i.e. 0.16 to build?

This may partially be related to a collection issue with pytest --pyargs and editable install seen in mesonbuild/meson-python#568 and fixed in mesonbuild/meson-python#569.

[oscarbenjamin]

After pip install scikit-image it seems that pytest --pyargs skimage works. I haven't tried in an editable install...

[stefanv]

Latest meson-python seems to have done the trick for me!

[lesteve]

Latest meson-python seems to have done the trick for me!

Glad to hear, it was a bit of a wild guess but it worked 🎉

[stefanv]

This should be closed by #191

[oscarbenjamin]

I just tested and it seems to work.

[oscarbenjamin]

Thanks!

[stefanv]

This may have caused a regression on Windows (#203), so I'll need to figure out what to do on that platform.