Support pytest plugins

[dcavanagh]

Environment data

• VS Code version: 1.59.0-insider
• Extension version (available under the Extensions sidebar): v2021.7.1060902895
• OS and version: macOS 10.15.7
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.3
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv
• Value of the python.languageServer setting: Default

Expected behaviour

It discovers tests

Actual behaviour

The Python extension has run into an unexpected situation
while processing a pytest node during test discovery. Please
Please open an issue at:
https://github.com/microsoft/vscode-python/issues
and paste the following output there.

nodeid: test/views/test_stations.py::describe_get_stations_with_count::describe_loggernet_data_platform::data_platform::it_queries_station_count
kind: ('function', False)
class: Function
name: data_platform::it_queries_station_count
fspath: /Users/dan/groundwork/GroundNetEngine/test/views/test_stations.py
location: ('test/views/test_stations.py', 42, 'describe_get_stations_with_count.describe_loggernet_data_platform.data_platform::it_queries_station_count')
function: <function describe_get_stations_with_count..data_platform..it_queries_station_count at 0x106ad3d30>
markers: []
user_properties: []
attrnames: ['_ALLOW_MARKERS', 'annotations', 'class', 'delattr', 'dict', 'dir', 'doc', 'eq', 'format', 'ge', 'getattribute', 'gt', 'hash', 'init', 'init_subclass', 'le', 'lt', 'module', 'ne', 'new', 'reduce', 'reduce_ex', 'repr', 'setattr', 'sizeof', 'slots', 'str', 'subclasshook', 'weakref', '_fixtureinfo', '_getobj', '_initrequest', '_nodeid', '_obj', '_prunetraceback', '_pyfuncitem', '_report_sections', '_repr_failure_py', '_request', '_store', 'add_marker', 'add_report_section', 'addfinalizer', 'cls', 'config', 'extra_keyword_matches', 'fixturenames', 'from_parent', 'fspath', 'funcargs', 'function', 'get_closest_marker', 'getmodpath', 'getparent', 'ihook', 'instance', 'iter_markers', 'iter_markers_with_node', 'keywords', 'listchain', 'listextrakeywords', 'listnames', 'location', 'module', 'name', 'nextitem', 'nodeid', 'obj', 'originalname', 'own_markers', 'parent', 'reportinfo', 'repr_failure', 'runtest', 'session', 'setup', 'teardown', 'user_properties', 'warn']

extra info:
fullname: describe_get_stations_with_count.describe_loggernet_data_platform.data_platform::it_queries_station_count
testfunc: describe_get_stations_with_count.describe_loggernet_data_platform.data_platform.it_queries_station_count
parameterized:

traceback:
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/run_adapter.py", line 22, in
main(tool, cmd, subargs, toolargs)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/main.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 30, in discover
ec = _pytest_main(pytestargs, [_plugin])
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/config/init.py", line 162, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in call
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
return wrap_session(config, _main)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
config.hook.pytest_collection(session=session)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in call
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
session.perform_collect()
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 641, in perform_collect
hook.pytest_collection_finish(session=self)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in call
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 107, in pytest_collection_finish
test, parents = self.parse_item(item)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 77, in parse_item
return parse_item(item)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_pytest_item.py", line 172, in parse_item
raise should_never_reach_here(
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_pytest_item.py", line 132, in should_never_reach_here
traceback.print_stack()
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 637, in perform_collect
INTERNALERROR> hook.pytest_collection_modifyitems(
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in call
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 92, in pytest_collection_modifyitems
INTERNALERROR> test, parents = self.parse_item(item)
INTERNALERROR> File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 77, in parse_item
INTERNALERROR> return parse_item(item)
INTERNALERROR> File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_pytest_item.py", line 172, in parse_item
INTERNALERROR> raise should_never_reach_here(
INTERNALERROR> NotImplementedError: Unexpected pytest node (see printed output).
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in call
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR> session.perform_collect()
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/_pytest/main.py", line 641, in perform_collect
INTERNALERROR> hook.pytest_collection_finish(session=self)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in call
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/dan/.local/share/virtualenvs/GroundNetEngine-WLpcZVa5/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 107, in pytest_collection_finish
INTERNALERROR> test, parents = self.parse_item(item)
INTERNALERROR> File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 77, in parse_item
INTERNALERROR> return parse_item(item)
INTERNALERROR> File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_pytest_item.py", line 172, in parse_item
INTERNALERROR> raise should_never_reach_here(
INTERNALERROR> NotImplementedError: Unexpected pytest node (see printed output).

========================= 22 tests collected in 4.56s ==========================

Traceback (most recent call last):
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/run_adapter.py", line 22, in
main(tool, cmd, subargs, toolargs)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/main.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "/Users/dan/.vscode-insiders/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 3)

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

[dcavanagh]

It fails to discover tests even if I delete all the test files except for one.

[karthiknadig]

@dcavanagh Can you share the function and its params? also are you using any pytest plugins?

[dcavanagh]

I've been able to get some basic tests working. It looks like the error occurs when using the `behavies_like` decorator from pytest_describe. I will keep looking for other cases Here are the plugins pytest = "*" pytest-cov = "*" pytest-describe = "*" pytest-freezegun = "*" pytest-mock = "*" pytest-notifier = "*"

…

On Mon, Aug 2, 2021 at 10:01 PM Karthik Nadig ***@***.***> wrote: @dcavanagh <https://github.com/dcavanagh> Can you share the function and its params? also are you using any pytest plugins? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16852 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACV47WBNTTB63N7PTBERO7LT25EW7ANCNFSM5BNYMOIA> .

[karthiknadig]

Our current discovery code does not support plugins like pytest-describe. We have plans to improve our discovery to handle plugins.

[dcavanagh]

@karthiknadig This particular plugin had been working and then stopped recently.

[roeap]

Plugins like pytest-describe really help us in managing tests in complex code bases. At the same time some of our devs would like to use the testing UIs in vscode. @karthiknadig can you share any info if and when support for plugins might be worked on?

[karthiknadig]

@roeap We are currently working on this. #17242 an objective with the re-write of the test adapter is to allow it to work with other plugins.

[roeap]

Thanks for the update and this great extension. Looking forward for this to land.

[kr-hansen]

Will note that in the meantime, this same problem can be solved by pinning pytest-describe==2.0.0 in the environment for testing. For some reason VSCode only breaks with pytest-describe==2.0.1

[illia-karpenkov]

Possibly related: #19668

[Mirandatz]

Thanks to @kr-hansen comment, I found that uninstalling pytest-pspec "fixes" the issue.
Thank you :D

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To try it yourself, you need to be on the most recent release of the Python extension and add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check for Experiment 'pythonTestAdapter' is active in your python logs.

Let me know if the rewrite fixes your issue. Thanks!

[eleanorjboyd]

Generally plugins should now work on the rewrite, please open issues with specific plugins if this is not the case for you. Thanks

Trying the rewrite:
To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.