ERROR: Test not found; the issue occurs with a test that uses random data in pytest.mark.parametrize.

[a-kachurin]

Type: Bug

Behaviour

Expected vs. Actual

Expected:
Clicking on test in TEST EXPLORER bar, test successfully started

Actual:
Clicking on test in TEST EXPLORER bar, getting: ERROR: not found: test_example.py::TestExample::test_1[sTlq]
(no name in any of [])

Steps to reproduce:

1. Add test:

import random
import string

import pytest


def gen_data():
    return ''.join(random.choices(string.ascii_letters, k=4))


@pytest.fixture()
def return_data(request):
    return {"test": request.param[0]}


class TestExample:
    @pytest.mark.parametrize(
        "return_data",
        [
            (
                (gen_data())
            )
        ],
        indirect=["return_data"])
    def test_1(self, return_data):
        print(return_data)

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.6
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2023-11-25 07:35:07.435 [info] Experiment 'All' is active
2023-11-25 07:35:07.435 [info] Default formatter is set to null for workspace c:\Users\alexk\Documents\example
2023-11-25 07:35:07.435 [info] Test server listening.
2023-11-25 07:35:07.435 [info] Python interpreter path: c:\program files\python311\python.exe
2023-11-25 07:35:08.532 [info] Starting Pylance language server.
2023-11-25 07:35:16.525 [info] Discover tests for workspace name: example - uri: c:\Users\alexk\Documents\example
2023-11-25 07:35:16.525 [info] Running discovery for pytest using the new test adapter.
2023-11-25 07:35:16.525 [info] All environment variables set for pytest discovery for workspace c:\Users\alexk\Documents\example: {"ALLUSERSPROFILE":"C:\\ProgramData","APPDATA":"C:\\Users\\alexk\\AppData\\Roaming","APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL":"1","CHROME_CRASHPAD_PIPE_NAME":"\\\\.\\pipe\\crashpad_16532_QWCKZXEBYYBPXLLL","COMPUTERNAME":"K_PC","CONAN_USER_HOME":"F:\\","ELECTRON_RUN_AS_NODE":"1","HOMEDRIVE":"C:","HOMEPATH":"\\Users\\alexk","IGCCSVC_DB":"AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAA0NINSWi79ke+RjQoUv4UmAQAAAACAAAAAAAQZgAAAAEAACAAAAC4OdZGKVMbGx1YgwYLf+BVD0eKGtLp1BEMLWXbhYhUHgAAAAAOgAAAAAIAACAAAAC7qigYDqbvJeuNHgKIZqW6mEb0qXC16EymRy4lr3SzQGAAAABLERGwxq7GhZuqlNzuQcqx1SpCPxJ8uPNQ3CZKy9RRJThZX/+q/Xh/8VwGbyD4UlPBPDxQ8Ou79yg7EQt53g2BV/jcVaOJjvr4EnhyT/8SyacARIOOX8EaQsA343gujsZAAAAAvWWVBsM9o3tTp1ZUw+OqJFIS9H9mcffXs8mP/bYthWWeUm9vyj2nBwPyWNTI94EYm4T0EQdgij3lbYC4vvX0nQ==","LOCALAPPDATA":"C:\\Users\\alexk\\AppData\\Local","LOGONSERVER":"\\\\K_PC","NUMBER_OF_PROCESSORS":"24","ORIGINAL_XDG_CURRENT_DESKTOP":"undefined","OS":"Windows_NT","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 183 Stepping 1, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"b701","PUBLIC":"C:\\Users\\Public","TEMP":"C:\\Users\\alexk\\AppData\\Local\\Temp","TMP":"C:\\Users\\alexk\\AppData\\Local\\Temp","USERDOMAIN":"K_PC","USERDOMAIN_ROAMINGPROFILE":"K_PC","USERNAME":"alexk","USERPROFILE":"C:\\Users\\alexk","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_CODE_CACHE_PATH":"C:\\Users\\alexk\\AppData\\Roaming\\Code\\CachedData\\1a5daa3a0231a0fbba4f14db7ec463cf99d7768e","VSCODE_CRASH_REPORTER_PROCESS_TYPE":"extensionHost","VSCODE_CWD":"C:\\Program Files\\Microsoft VS Code","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_IPC_HOOK":"\\\\.\\pipe\\b1993ca3248d0b964d73e5c114b33e56-1.84.2-main-sock","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"ru\",\"availableLanguages\":{},\"_languagePackSupport\":true}","VSCODE_PID":"16532","VSLANG":"1033","ZES_ENABLE_SYSMAN":"1","ComSpec":"C:\\Windows\\system32\\cmd.exe","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","DriverData":"C:\\Windows\\System32\\Drivers\\DriverData","OneDrive":"C:\\Users\\alexk\\OneDrive","PSModulePath":"C:\\Program Files (x86)\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\AutoIt3\\AutoItX","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","SystemDrive":"C:","SystemRoot":"C:\\Windows","__PSLockDownPolicy":"0","windir":"C:\\Windows","Path":"c:\\program files\\python311\\scripts\\;c:\\program files\\python311\\;C:\\Program Files\\Python310\\Scripts\\;C:\\Program Files\\Python310\\;c:\\program files\\conan\\conan;c:\\program files\\python27;c:\\windows\\system32;c:\\windows;c:\\windows\\system32\\wbem;c:\\windows\\system32\\windowspowershell\\v1.0\\;c:\\windows\\system32\\openssh\\;c:\\program files\\git\\cmd;c:\\program files\\dotnet\\;c:\\program files\\google\\chrome\\application;c:\\program files\\nvidia corporation\\nvidia nvdlisr;c:\\program files\\nodejs\\;c:\\program files\\microsoft vs code\\bin;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\CMake\\bin;C:\\Users\\alexk\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\alexk\\AppData\\Roaming\\npm;C:\\Users\\alexk\\.dotnet\\tools;","PYTHONPATH":"c:\\Users\\alexk\\.vscode\\extensions\\ms-python.python-2023.21.13261010\\pythonFiles","TEST_UUID":"e83b228a-21e8-4e17-8956-4bfc19f961b8","TEST_PORT":"63397"} 

2023-11-25 07:35:16.539 [info] > "c:\program files\python311\python.exe" -m pytest -p vscode_pytest --collect-only
2023-11-25 07:35:16.539 [info] cwd: .
2023-11-25 07:35:16.984 [info] ============================= test session starts =============================
platform win32 -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0

2023-11-25 07:35:16.986 [info] rootdir: c:\Users\alexk\Documents\example
plugins: Faker-20.0.3, cov-4.1.0, repeat-0.9.3, xdist-3.4.0, testit-adapter-pytest-2.4.1

2023-11-25 07:35:16.990 [info] collected 1 item

<Module test_example.py>
  <Class TestExample>
    <Function test_1[zWpK]>

2023-11-25 07:35:16.992 [info] Test server connected to a client.
2023-11-25 07:35:16.993 [info] ResultResolver EOT received for discovery.
2023-11-25 07:35:16.994 [info] 
========================== 1 test collected in 0.01s ==========================

2023-11-25 07:35:17.022 [info] Disposing data receiver for c:\Users\alexk\Documents\example and deleting UUID; pytest discovery.
2023-11-25 07:35:20.782 [info] Server listening on port 63468
2023-11-25 07:35:20.785 [info] All environment variables set for pytest execution in c:\Users\alexk\Documents\example workspace: 
 {"ALLUSERSPROFILE":"C:\\ProgramData","APPDATA":"C:\\Users\\alexk\\AppData\\Roaming","APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL":"1","CHROME_CRASHPAD_PIPE_NAME":"\\\\.\\pipe\\crashpad_16532_QWCKZXEBYYBPXLLL","COMPUTERNAME":"K_PC","CONAN_USER_HOME":"F:\\","ELECTRON_RUN_AS_NODE":"1","HOMEDRIVE":"C:","HOMEPATH":"\\Users\\alexk","IGCCSVC_DB":"AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAA0NINSWi79ke+RjQoUv4UmAQAAAACAAAAAAAQZgAAAAEAACAAAAC4OdZGKVMbGx1YgwYLf+BVD0eKGtLp1BEMLWXbhYhUHgAAAAAOgAAAAAIAACAAAAC7qigYDqbvJeuNHgKIZqW6mEb0qXC16EymRy4lr3SzQGAAAABLERGwxq7GhZuqlNzuQcqx1SpCPxJ8uPNQ3CZKy9RRJThZX/+q/Xh/8VwGbyD4UlPBPDxQ8Ou79yg7EQt53g2BV/jcVaOJjvr4EnhyT/8SyacARIOOX8EaQsA343gujsZAAAAAvWWVBsM9o3tTp1ZUw+OqJFIS9H9mcffXs8mP/bYthWWeUm9vyj2nBwPyWNTI94EYm4T0EQdgij3lbYC4vvX0nQ==","LOCALAPPDATA":"C:\\Users\\alexk\\AppData\\Local","LOGONSERVER":"\\\\K_PC","NUMBER_OF_PROCESSORS":"24","ORIGINAL_XDG_CURRENT_DESKTOP":"undefined","OS":"Windows_NT","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 183 Stepping 1, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"b701","PUBLIC":"C:\\Users\\Public","TEMP":"C:\\Users\\alexk\\AppData\\Local\\Temp","TMP":"C:\\Users\\alexk\\AppData\\Local\\Temp","USERDOMAIN":"K_PC","USERDOMAIN_ROAMINGPROFILE":"K_PC","USERNAME":"alexk","USERPROFILE":"C:\\Users\\alexk","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_CODE_CACHE_PATH":"C:\\Users\\alexk\\AppData\\Roaming\\Code\\CachedData\\1a5daa3a0231a0fbba4f14db7ec463cf99d7768e","VSCODE_CRASH_REPORTER_PROCESS_TYPE":"extensionHost","VSCODE_CWD":"C:\\Program Files\\Microsoft VS Code","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_IPC_HOOK":"\\\\.\\pipe\\b1993ca3248d0b964d73e5c114b33e56-1.84.2-main-sock","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"ru\",\"availableLanguages\":{},\"_languagePackSupport\":true}","VSCODE_PID":"16532","VSLANG":"1033","ZES_ENABLE_SYSMAN":"1","ComSpec":"C:\\Windows\\system32\\cmd.exe","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","DriverData":"C:\\Windows\\System32\\Drivers\\DriverData","OneDrive":"C:\\Users\\alexk\\OneDrive","PSModulePath":"C:\\Program Files (x86)\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\AutoIt3\\AutoItX","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","SystemDrive":"C:","SystemRoot":"C:\\Windows","__PSLockDownPolicy":"0","windir":"C:\\Windows","Path":"c:\\program files\\python311\\scripts\\;c:\\program files\\python311\\;C:\\Program Files\\Python310\\Scripts\\;C:\\Program Files\\Python310\\;c:\\program files\\conan\\conan;c:\\program files\\python27;c:\\windows\\system32;c:\\windows;c:\\windows\\system32\\wbem;c:\\windows\\system32\\windowspowershell\\v1.0\\;c:\\windows\\system32\\openssh\\;c:\\program files\\git\\cmd;c:\\program files\\dotnet\\;c:\\program files\\google\\chrome\\application;c:\\program files\\nvidia corporation\\nvidia nvdlisr;c:\\program files\\nodejs\\;c:\\program files\\microsoft vs code\\bin;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\CMake\\bin;C:\\Users\\alexk\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\alexk\\AppData\\Roaming\\npm;C:\\Users\\alexk\\.dotnet\\tools;","PYTHONPATH":"c:\\Users\\alexk\\.vscode\\extensions\\ms-python.python-2023.21.13261010\\pythonFiles","TEST_UUID":"a30fe1c6-53e7-47d5-8838-d08951d79742","TEST_PORT":"63397","RUN_TEST_IDS_PORT":"63468"}
2023-11-25 07:35:20.785 [info] Running pytest with arguments: c:\Users\alexk\.vscode\extensions\ms-python.python-2023.21.13261010\pythonFiles\vscode_pytest\run_pytest_script.py --rootdir c:\Users\alexk\Documents\example for workspace c:\Users\alexk\Documents\example 

2023-11-25 07:35:20.785 [info] > "c:\program files\python311\python.exe" ~\.vscode\extensions\ms-python.python-2023.21.13261010\pythonFiles\vscode_pytest\run_pytest_script.py --rootdir .
2023-11-25 07:35:20.785 [info] cwd: .
2023-11-25 07:35:21.234 [info] Test server connected to a client.
2023-11-25 07:35:21.235 [info] ResultResolver EOT received for execution.
2023-11-25 07:35:21.242 [info] Client disconnected
2023-11-25 07:35:21.264 [info] Disposing data receiver for c:\Users\alexk\Documents\example and deleting UUID; pytest execution.

User Settings

languageServer: "Pylance"

testing
• pytestEnabled: true

experiments
• optInto: ["All"]

Extension version: 2023.21.13261010
VS Code version: Code 1.84.2 (1a5daa3a0231a0fbba4f14db7ec463cf99d7768e, 2023-11-09T10:51:52.184Z)
OS version: Windows_NT x64 10.0.22621
Modes:

System Info

Item	Value
CPUs	13th Gen Intel(R) Core(TM) i7-13700K (24 x 3418)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	63.76GB (51.87GB free)
Process Argv	--crash-reporter-id 2b5a0ed1-4981-4290-911b-a9374f02c7db
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
py29gd2263:30880072
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
282f8724:30602487
f6dab269:30613381
2i9eh265:30646982
showlangstatbar:30737416
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
welcomedialog:30887143
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixtcf:30871695
pythonregdiag2:30871582
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
h48ei257:30885898
pythontbext0:30879054
dsvsc016cf:30886111
dsvsc017cf:30886113
dsvsc018cf:30886115
aa_t_chat:30882232

[a-kachurin]

Also reproducing with:
Add test:

def gen_data():
    return ''.join(random.choices(string.ascii_letters, k=4))


@pytest.fixture()
def return_data(request):
    return {
        "test": request.param[0],
        "some_data": request.param[1]
    }


class TestExample:
    @pytest.mark.parametrize(
        "return_data, x",
        [
            (
                (gen_data(), "some data"),
                "x"
            )
        ],
        indirect=["return_data"])
    def test_1(self, return_data, x):
        print(x, return_data)

Add to conftest.py:

@pytest.hookimpl()
def pytest_make_parametrize_id(config, val):
    return repr(val)

[eleanorjboyd]

Hi! Yes the seems related to a few issues we have where generated names are not working on the rewrite. I will circle back when I start work on this or with another issue to close against. Thanks

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[eleanorjboyd]

Hi @akachurin93, following up on this.

On investigation this seems like a more complex issue since these tests have no stable id. Pytest has no single way of identifying them and therefore there is not a way I create a ID that will stay constant through out the discovery/run cycle.

A potential fix could be that when you click the parent node, like the file node which contains these tests than the request string it sends includes only the file name and not test_ids which would work to run since the file ID is more general. Is this the type of fix you would be looking for?

[a-kachurin]

Hi @eleanorjboyd !
Yes, But i would like to have the ability to run both the file containing such tests and the tests individually from the testing plugin menu in the test tree. That would be great. Is there a possibility to transform test identifiers into something more stable using something like hook pytest_make_parametrize_id?

[a-kachurin]

I mean to collect test identifiers as it was in the case of this code:
def pytest_make_parametrize_id(config, val, argname): return f"{argname}_{val}"

[eleanorjboyd]

Im not sure I would want the extension to be making changes on the level of renaming tests as this feels like overreach into a users project.

What would running the tests individually look like? Would this be running each parameterize test instance or the run the single parent parameterize test object?

ie test_ex would show up and not the random parameterize tests like test_ex[aeea], test_ex[fewra], test_ex[fdsare]

[a-kachurin]

[a-kachurin]

What would running the tests individually look like? Would this be running each parameterize test instance or the run the single parent parameterize test object?
I didn't quite understand what you mean.

[eleanorjboyd]

using the same example:
ie test_ex would show up and not the random parameterize tests like test_ex[aeea], test_ex[fewra], test_ex[fdsare]

do you want to be able to click to run test_ex[aeea] or just test_ex? If a user tries to run test_ex[aeea] what would you expect to run given it is randomly generated so you are going to be running a different test node with a different random var which could be confusing?

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[a-kachurin]

Ideally, the user should see test_ex[return_data0], test_ex[return_data1], but it's also acceptable to display random information in the title, as long as it still triggers test_ex[return_data0], test_ex[return_data1].

[eleanorjboyd]

Hello! Reviewing this issue again I think it is still out of scope and too challenging to implement. It would require renaming user test case names and somehow matching those to the pytest run results. If anyone feels strongly about this issue and wants to write a PR I will definitely take a look but I am going to close this issue as I do not see it as very possible. Thanks