Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test execution timeout it not properly detected #1496

Open
Marcono1234 opened this issue Sep 5, 2022 · 2 comments
Open

Test execution timeout it not properly detected #1496

Marcono1234 opened this issue Sep 5, 2022 · 2 comments
Labels
bug Something isn't working good first issue Good for newcomers VSCode

Comments

@Marcono1234
Copy link
Contributor

Describe the bug
When test execution times out (after 5 minutes by default), the extension does not seem to notice that properly:

Version
Extension:

CodeQL extension version: 1.6.12 
CodeQL CLI version: 2.10.4 
Platform: win32 x64

VS Code:

Version: 1.71.0 (user setup)
Commit: 784b0177c56c607789f9638da7b6bf3230d47a8c
Date: 2022-09-01T07:36:10.600Z
Electron: 19.0.12
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.19043
Sandboxed: No

To reproduce

  1. Open the extension settings
  2. Edit "Additional Test Arguments" and add "--timeout=1"
  3. Run a test
    ❌ The extension does not properly handle the test failure

(remember to revert your settings changes)

Expected behavior

  • The extension should properly detect that the test failed
  • The timeout error message should appear in the "CodeQL Tests" output and not in the .actual file

Additional context
The .actual file contains:

Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff

(with api-visibility-issues being the query name)

Running the test with codeql test run --format=json yields the following response (might helpful to adjust the parsing logic in the extension?):

{
  "test" : "<test-dir>/api-visibility-issues.qlref",
  "pass" : false,
  "failureStage" : "EXECUTION",
  "messages" : [ ],
  "evaluationMs" : 305698,
  "expected" : "<test-dir>/api-visibility-issues.expected",
  "actual" : "<test-dir>/api-visibility-issues.actual",
  "diff" : [
    "--- expected",
    "+++ actual",
    "@@ -1,30 +1,1 @@",
    "-| ...",
    "+Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff"
  ]
}

(slightly minified)

@Marcono1234 Marcono1234 added the bug Something isn't working label Sep 5, 2022
@github-actions github-actions bot added the VSCode label Sep 5, 2022
@aeisenberg aeisenberg added the good first issue Good for newcomers label Sep 6, 2022
@aeisenberg
Copy link
Contributor

Thanks for raising this issue. I have put this on our backlog and we will look at this issue shortly.

@aeisenberg
Copy link
Contributor

@stonish Can your team track this task? I'm removing it from our board.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers VSCode
Projects
None yet
Development

No branches or pull requests

2 participants