Why does ms-python test execution discover all tests?

[tomghc]

I recently noticed that in ms-python v2023.20.0 running a specific test, test module, test directory etc triggers a discovery. I traced this back to unittestadapter, which seemingly uses my unittest settings (see below) and this results in all tests being discovered. Ultimately, the plugin seems to discard the outputs immediately though.

In my case, this actually causes problems as I am using abseil to manage CLI flags in a distributed manner. When test execution triggers discovery, this results in all CLI flags for all modules being loaded, even though they are not being tested. If any tests or tested code trigger abseil's flag parsing, I get exceptions as I won't have passed in values for mandatory flags values from unrelated modules.

I tried commenting out the line above, which indeed solved my issue. I then opted out of pythonTestAdapter experiment as a better solution.
This brings me to the ultimate question of what this line is trying to achieve. I first thought that this might be in case I triggered tests from the top level directory, but I can see this still results in all test ids being explicitly passed in. Am I missing a use case for this behaviour and/or some implicit side effects of running discovery? Or could this be removed/made optional?

FWIW, my python unittest args are as follows to enable all tests to be discovered. Perhaps I'm missing a trick there.

    "python.testing.unittestEnabled": true,
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        ".",
        "-p",
        "*test*.py"
    ],

[eleanorjboyd]

Hi @tomghc, thanks for reaching out! First which line do you mean by "I tried commenting out the line above"? I might be missing something but I am not sure what line you are referring to.

[tomghc]

Hey, I'm referring to the link to the unittestadapter code I had higher up:

vscode-python/pythonFiles/unittestadapter/execution.py

Line 194 in caa50dc

suite = loader.discover(start_dir, pattern, top_level_dir) # noqa: F841

edit: If only I'd known that pasting the link without using the editor's link formatting would insert a preview

[eleanorjboyd]

Thank you for clarifying yes. So to answer your most specific question, that line does discovery before then running execution to ensure that we have gathered all necessary tests we then need to execute based on the parameters you have set in your settings. Think of the case where you didn't run discovery and instead just ran execution but want all your tests to be picked up.

When you say " all test ids being explicitly passed in", we are passing in the test-ids for your run request. If you click "run" on only a single test in the test explorer we will only run that specific test id. We do run discovery for the whole directory or subdirectory you defined and then pick that test id out of that suite to specifically run- is this what you are asking about? I think what you are saying is why are we running discovery on the whole defined directory if we know exactly which test to run- does this sound right? If so, I think thats a great point, I would love to investigate work-arounds as this feels like a valid concern / use case.

[tomghc]

I think what you are saying is why are we running discovery on the whole defined directory if we know exactly which test to run- does this sound right?

I'd go one step further, if I know which test(s) to run why do I need to run any discovery? Unittest does not require discovery (i.e. python -m unittest test_module.TestClass.test_method is valid as per the python docs), which is also backed up by the fact we are not using the suite returned by discovery. The line below shows a new suite being created after discovery, so it doesn't seem like it serves any purpose (unless unittest is propagating something at a module level?) https://github.com/microsoft/vscode-python/blob/caa50dcf146219e9233f7d17a5d859d345baca91/pythonFiles/unittestadapter/execution.py#L209C3-L209C3

Think of the case where you didn't run discovery and instead just ran execution but want all your tests to be picked up.

Yes, in that case discovery is needed, so perhaps it's just a case of skipping discovery when you have the test ids. However, I think I'm missing out on a test execution journey via VSCode. I'm not able to run any tests via VSCode features (other than via a terminal command) unless VSCode performs discovery beforehand and populates the discovered tests in the "Testing" panel and highlights tests in the gutters for relevant test files. Is there a different way to run tests that doesn't require discovery having already run?

[eleanorjboyd]

Hi valid case here "Unittest does not require discovery (i.e. python -m unittest test_module.TestClass.test_method is valid as per the python docs)". We use the suite returned by discovery to populate the test result panel, so that is what discovery is used for (and to create those green run arrows in the gutter). So yeah, I think we could probably get rid of the redundant discovery on run since we have the ids?

There is no other way to run tests that doesn't require discovery to be done. We populate the UI with discovery then the user can click to run some of all of the tests. What is the use case for running without first having discovery?

[tomghc]

Cool, sounds like we're on the same page then (discovery not needed as part of execution itself, since it's already been done before). I'm not aware of any other use case.

[tomghc]

Hello @eleanorjboyd,

edit: it turns out I can revert to the ms-python.python 2024.22.2 which temporarily addresses my issue, but I'm not sure how long that'll be sustainable for.

I seem to be running into the same issue again, since an auto-update on our company's side (i.e. I can't back out to an older version). In all cases I can see test discovery running prior to test execution, using the new python test adapter, which triggers the aforementioned issues. Unfortunately disabling the experiment no longer changes anything, and a quick browse of the VScode repo seems to suggest it has switched over to the new test running approach altogether. Do you have any suggestions as-to how I can avoid this pattern of discovery as part of test execution?

Thanks,