how can I test #13301 ?

[LeGEC]

issue #13301 is marked as fixed (btw: the comments on that issue is closed, which is why I can't ask on it ...) but I haven't found a release note mentioning it, and my current vscode doesn't have the feature.

Has it been released already ? (if yes: what version ?)
if not released, how can I test a version with the fix ?

thank you

[LeGEC]

In my context: in order to avoid having "one failed module load makes my entire 'Testing' tree empty", I manually add --continue-on-collection-errors to pytest arguments (python.testing.pytestArgs setting).

While this correctly allows me to run the other tests, it also silently completely hide the errors on failed imports.

[eleanorjboyd]

Hi! Can you send an outline of your project and youre logs from the "python" output channel when you set "developer tools: log level" to "info" via that command palette command? This feature is released and should work. And to confirm one module failing is the in a different file then the other ones you expect to find? We have it work on a per file basis right now.

[LeGEC]

https://github.com/LeGEC/pytest-collect-errros

directions:

• clone the repository
• open vscode in the pytest-collect-errors directory
• make sure you have the pylance extension installed

information:

• -> the code in src/tests/foo/test_foo_failed.py has a failing import which makes that particular file error on test collection

• -> version-info.txt contains information about the versions of vscode, python extension, ... here are a few of these:
  code: 1.87.2
  [email protected]
  python 3.11.6
  pytest 7.4.0

• point 1: the "Testing" tab initially shows an empty test tree and an error message

• edit the .vscode/settings.json file, and un-comment the --continue-on-collection-errors line

• point 2: the "Testing" tab now shows entries for all tests that are succesfully discovered, but completely hides the fact that one import failed (nothing appears in red)

running pytest from the command line, on the other hand, behaves as I expect (you can run the script runtests.sh):

• pytest alone mentions the import error and stops without running any of the tests,
• pytest --continue-on-collection-errors does run all the other tests, and ends with an error, reporting ERROR src/tests/foo/test_foo_fail.py (and more details about the import error in its logs)

[LeGEC]

I don't know if the expectations of the pytest extension is to have point 1 above work without further configuration, or to have point 2 work,
my expectations is to have at least one state (I would be ok with 2) allowing me to browse and launch the tests that can be run, while still reporting reporting failure on test files that fail at discovery time.

[LeGEC]

Logs seen in Output > Python:

without the --continue-on-collection-errors option, I see the error at collection stopping pytest

the successfully discovered modules and test functions are printed in the output, the failing test module is mentioned only in the error message

(click to view logs)

2024-03-20 17:33:12.535 [info] Discover tests for workspace name: pytest-collect-errros - uri: /home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros/.vscode/settings.json
2024-03-20 17:33:12.558 [info] > /usr/bin/python3 ~/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
2024-03-20 17:33:12.558 [info] cwd: .
2024-03-20 17:33:12.771 [error] Error discovering pytest tests:
 [n [Error]: ============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros
configfile: pytest.ini
collected 2 items / 1 error

<Module src/tests/bar/test_bar.py>
  <Function test_something>
<Module src/tests/foo/test_foo_ok.py>
  <Function test_something_ok>

==================================== ERRORS ====================================
_______________ ERROR collecting src/tests/foo/test_foo_fail.py ________________
ImportError while importing test module '/home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros/src/tests/foo/test_foo_fail.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/tests/foo/test_foo_fail.py:1: in <module>
    from fooo import say_foo # <- wrong import, fails at import time
E   ModuleNotFoundError: No module named 'fooo'
=========================== short test summary info ============================
ERROR src/tests/foo/test_foo_fail.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== 2 tests collected, 1 error in 0.04s ======================

Traceback (most recent call last):
  File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/adapter/__main__.py", line 99, in main
    parents, result = run(toolargs, **subargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 47, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/out/client/extension.js:2:226040)
	at Object.onceWrapper (node:events:629:26)
	at ChildProcess.emit (node:events:514:28)
	at maybeClose (node:internal/child_process:1091:16)
	at Socket.<anonymous> (node:internal/child_process:449:11)
	at Socket.emit (node:events:514:28)
	at Pipe.<anonymous> (node:net:323:12)]
2024-03-20 17:33:13.227 [info] Discover tests for workspace name: pytest-collect-errros - uri: /home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros
2024-03-20 17:33:13.243 [info] > /usr/bin/python3 ~/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
2024-03-20 17:33:13.243 [info] cwd: .
2024-03-20 17:33:13.416 [error] Error discovering pytest tests:
 [n [Error]: ============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros
configfile: pytest.ini
collected 2 items / 1 error

<Module src/tests/bar/test_bar.py>
  <Function test_something>
<Module src/tests/foo/test_foo_ok.py>
  <Function test_something_ok>

==================================== ERRORS ====================================
_______________ ERROR collecting src/tests/foo/test_foo_fail.py ________________
ImportError while importing test module '/home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros/src/tests/foo/test_foo_fail.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/tests/foo/test_foo_fail.py:1: in <module>
    from fooo import say_foo # <- wrong import, fails at import time
E   ModuleNotFoundError: No module named 'fooo'
=========================== short test summary info ============================
ERROR src/tests/foo/test_foo_fail.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== 2 tests collected, 1 error in 0.03s ======================

Traceback (most recent call last):
  File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/adapter/__main__.py", line 99, in main
    parents, result = run(toolargs, **subargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 47, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/out/client/extension.js:2:226040)
	at Object.onceWrapper (node:events:629:26)
	at ChildProcess.emit (node:events:514:28)
	at maybeClose (node:internal/child_process:1091:16)
	at Socket.<anonymous> (node:internal/child_process:449:11)
	at Socket.emit (node:events:514:28)
	at Pipe.<anonymous> (node:net:323:12)]

---

with the --continue-on-collection-errors option, the logs show no error whatsoever:

2024-03-20 18:01:10.797 [info] Discover tests for workspace name: pytest-collect-errros - uri: /home/gvarouchas/code/github.com/LeGEC/pytest-collect-errros
2024-03-20 18:01:10.814 [info] > /usr/bin/python3 ~/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --continue-on-collection-errors .
2024-03-20 18:01:10.814 [info] cwd: .

[eleanorjboyd]

Hi! Sorry for the delay, look at your logs here File "/home/gvarouchas/.vscode/extensions/ms-python.python-2024.2.1/pythonFiles/testing_tools/adapter/__main__.py", line 99, in main it looks like you are using the old testing infrastructure.
Can you add this to your user settings.json: "python.experiments.optInto": ["pythonTestAdapter"]. You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check for Experiment 'pythonTestAdapter' is active in your python logs.

Let me know if the rewrite fixes your issue. Thanks!

[LeGEC]

Hi, thank you for the feedback.

Since I opened that discussion, there were several updates (to both vscode and the python plugin), and test discovery now works as you describe without me having to manually add --continue-on-collection-errors

For clarification: I now get both:

• an error message shown when one (or more) of the test modules can't be imported,
• a runnable list of tests for the other test modules

I will now close this discussion, as resolved.