unittest runner thinks failing test is not run

[programmerjake]

Environment data

• VS Code version: 1.60.2
• Extension version (available under the Extensions sidebar): v2021.9.1246542782
• OS and version:
  Ubuntu 20.04.3 LTS
  Linux x86_64 5.4.0-88-generic
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.10
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python packages and their versions: none?
• Relevant/affected Python-related VS Code extensions and their versions: none?
• Value of the python.languageServer setting: pylance

Expected behaviour

Reports failed test

Actual behaviour

Reports passing test/thinks test is still not run

Steps to reproduce:

Run below test using vscode:

import unittest


class Tests(unittest.TestCase):
    def test(self):
        with self.subTest():
            assert False

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/.virtualenvs/libre-riscv-venv/bin/python ~/.vscode/extensions/ms-python.python-2021.9.1246542782/pythonFiles/visualstudio_py_testlauncher.py --us=. --up=*test*.py --uvInt=2 --result-port=35131 -ttest_reproducer.Tests.test --testFile=/home/jacob/projects/libreriscv/simd_signal/test_reproducer.py
cwd: ~/projects/libreriscv/simd_signal

Output for Test Output in the Terminal panel

Running tests (unittest): /home/jacob/projects/libreriscv/simd_signal/test_reproducer.py
Running tests: [object Object]
Total number of tests expected to run: 1
Total number of tests run: 1
Total number of tests passed: 0
Total number of tests failed: 0
Total number of tests failed with errors: 0
Total number of tests skipped: 0
Finished running tests!


> Test run finished at 9/28/2021, 5:56:12 PM <

Output for Python Test Log in the Output panel

test (test_reproducer.Tests) ... 
======================================================================
FAIL: test (test_reproducer.Tests) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jacob/projects/libreriscv/simd_signal/test_reproducer.py", line 7, in test
    assert False
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

[coppinri]

This seems to be the behaviour particular to subtests.

Worse though, if the test passed previously, but now the test assertion fails, the UI still shows it as passed.

[karthiknadig]

Our test adapter for unittest, when it was written was not designed to handle unittest subtests. This is primarily due to unittest discovery cannot detect them before running the test. when subtests are used, the results are generated slightly differently from when they are not used. So we currently don't have a way to match the before subtest state.

We are in the process of re-writing the test adapter. With the redesign we will take unittest sub tests into consideration.

[karthiknadig]

Related #17242

[karthiknadig]

@programmerjake I made a PR (#17578) that addresses this issue in some form:

But it won't make it into up coming release. I will share the build from this PR when it is done here so you can try it out. For the code that yo shared the result should look like this:

[coppinri]

Wow, I was only expecting it to update the UI on a failed run, but if it's able to delve into- and report on- each subtest that would be great.

[karthiknadig]

@coppinri There are limitations to this. We can only detect sub-test at run time, unittest discovery does not know about sub-tests. The subtest nodes you see above are only reporting nodes, you can specifically select them and run.

if you want to give this a try, you will need insiders version of VS Code (v 1.61.0-insider) and this build of python extensions https://github.com/microsoft/vscode-python/suites/3925208299/artifacts/98213954