Slow test running, with plugin dependencies slow-down further

[deblaci]

Expected Behavior

All the Unit test run within e.g. 1 minutes.

Actual Behavior

I have only few unit test, approximately 37 but they are running up to 3 minutes.
Most of them are class test from src folder, one test usually run in 10 sec. I have one pipeline test from vars it takes 30sec.
Usually the dependences (other own class, jenkins stuff) are mocked or stubed. I tested only the logic and function calls.

Other interesting:
When I added example podTemplate dependency to pom.xml because of the framework complain about no mock for podTeplate, the test run was more slower(+15s) than I didn't add to dependency but added explicitlyMockPipelineStep and I stubed it with getPipelineMock.

Steps to Reproduce

Well, I think it is not easy with simple code.

[abe-frickelbude]

Hi :-) I'd like to upvote this if possible - the test initialization is quite slow :-(

@deblaci Re: "steps to reproduce" - I'm not sure what your project setup is.. I'm using a Gradle-based configuration based on heavily on the examples developed by corporategadfly, found in this fork here:

https://github.com/corporate-gadfly/jenkins-spock

I'm using the official Jenkins plugin BOM and apart from jenkins-core itself I have relatively few plugin dependecies, namely:

testImplementation("org.jenkins-ci.plugins.workflow:workflow-cps")
testImplementation("org.jenkins-ci.plugins.workflow:workflow-durable-task-step")
testImplementation("org.jenkins-ci.plugins.workflow:workflow-scm-step")

I've made a very simple "smoke test" that extends the JenkinsPipelineSpecification and has one test method that simply "expects true", so essentially a no-op. This alone already takes over 30 seconds to execute :-(

I've managed to narrow it down somewhat by debugging into the Spock lifecycle inside JenkinsPipelineSpecification. Most of the time appears to be spent during the classpath scanning phase that determines the available pipeline extensions, global variables etc. I cannot provide specific timings as I would need to instrument the code to take accurate measurements. But taken "at face value" I'd say that improving the performance of these sections would require improvements to the said classpath scan, if such are indeed possible at all...

Hope this provides a little more context

[ghost]

@deblaci I can confirm @abe-frickelbude's suspicion. I tested these assertions by grabbing the JenkinsPipelineSpecification class, removing anything related to pipeline extensions and classpath scanning (at the time of writing this post that includes removal of the setupSpec() method and this block in setup() that interacts with the WholeClasspathPipelineExtensionDetector class). I tried overriding these two methods in a separate class but continued seeing invocations of the WholeClasspathPipelineExtensionDetector. It appears this is the preferred methodology of changing the test suite's setup according to comments applied to the sets and maps of the JenkinsPipelineSpecification class, but I'm likely doing something incorrectly.

Executing ~40 tests with these adjustments resulted in a major decrease in time; originally 3 minutes and 30 seconds vs. ~15 seconds. Note that this is without parallel test execution. Please let me know if you're interested in additional information regarding the testing procedure.

As far as recommendations are concerned... I'm not sure (perhaps a cached list of these mocked extensions would help?). The intended result of auto-mocking these extensions is a really nice feature as it trims down the amount of steps/variables that need explicit mocking, but honestly I'm also fine with doing just that.

[awittha]

I am fairly certain that the classpath only needs to be scanned once, at "JVM-that's-going-to-run-tests" initialization-time; not per test suite class. This would cause jenkins-spock to ship with ... most of the time-savings you found, without giving up any functionality.

However, I would want to test that fairly rigorously before deploying such a change, as there is a lot of classpath/classloader trickery going into making jenkins-spock work the way it does.

The actual work item, then would be

Enhancement: Scan classpaths for extensions only once, at time of jenkins-spock classload

[deblaci]

@mdeitrick could you please check this #109 ?

[ghost]

@deblaci Those changes look good to me! Are there any additional actions that you need me to take to move this change forward?

[deblaci]

I think no, we have to wait @awittha.