Handler for TestRunStarted event gets triggered after all the tests finish in case of parallel execution of tests

[MazurK239]

👓 What did you see?

I registered a plugin that catches TestRunStarted and TestRunFinished events to do some set-up / tear-down logic.
This plugin extends EventListener and overrides method setEventPublisher.

When the tests are run sequentially (both via CLI and JUnit runners), the TestRunStarted handler is invoked, then tests are executed, then TestRunFinished handler is invoked, everything is fine
CLI runner executed in IDEA

JUnit runner executed via Gradle

But when I run the tests in parallel, the handler of TestRunStarted gets invoked right after all tests have been executed. Even though the actual event time stored in event object is correct (see the timestamps in log on the screenshots)
CLI runner executed in IDEA

JUnit runner executed via Gradle

In case of CLI runner I specified --threads 2 in the program args, in case of JUnit runner I specified cucumber.execution.parallel.enabled=true inside the junit-platform.properties file. The threads count doesn't affect anything in the latter case, it is reproduced even with one thread****

✅ What did you expect to see?

I expected that during the parallel test execution the TestRunStarted handler will be executed once the event happens, before the tests are started. So that I could run some set-up logic inside it. Currently It seems to not be practical

📦 Which tool/library version are you using?

org.junit:junit-bom:5.11.0
io.cucumber:cucumber-bom:7.18.1

🔬 How could we reproduce it?

Minimal reproducible example forked from the cucumber-java skeleton: cucumber-java-issue-events

Run feature file via IntelliJ IDEA (CLI runner will be used) or via Gradle

📚 Any additional context?

No response

[mpkorstanje]

That is intentional. Most plugins can't handle parallel execution. You'll want to use the ConcurrentEventListener.

cucumber-jvm/cucumber-plugin/src/main/java/io/cucumber/plugin/EventListener.java

Lines 9 to 15 in 0cdf507

	* When cucumber executes test in parallel or in a framework that supports
	* parallel execution (e.g. JUnit or TestNG)
	* {@link io.cucumber.plugin.event.Event}s are stored and published in canonical
	* order after the test run has completed.
	*
	* @see io.cucumber.plugin.event.Event
	* @see ConcurrentEventListener

[mpkorstanje]

Btw, for setup you might want to look at the @BeforeAll and @AfterAll hooks from Cucumber or depending on what you need, those recently added to the JUnit 5 Suite Engine.

[MazurK239]

Thank you @mpkorstanje. Works like a charm! sorry that I missed this in the documentation.
Didn't know about @BeforeAll and @AfterAll either, will definitely give them a try

[MazurK239]

@mpkorstanje, since there is yet no documentation regarding the use of BeforeAll/AfterAll could you, please, advise, how to correctly use it in Kotlin? As I see, internally, there is a check that this method is static, but with Kotlin it seems impossible to make this check pass

[mpkorstanje]

Docs are here:

https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-java#beforeall--afterall

And for Java to Kotlin interop you'll want to read:

https://kotlinlang.org/docs/java-to-kotlin-interop.html#static-methods

[MazurK239]

Many thanks!

[ejbartelds]

I saw @MazurK239 's question and @mpkorstanje 's answer today, concerning the use of BeforeAll/AfterAll with Cucumber in Kotlin, but I ran into a problem when trying to use those. See screenshots.

It appears Cucumber's method scanning in io.cucumber.java.MethodScanner::scan sees both the original companion object method (which is not static) AND the synthesized one (due to @JvmStatic) (which has been made static). I am using cucumber-java 7.20.1

As a result, I cannot get this to work in Kotlin...

Any thoughts?

[mpkorstanje]

@ejbartelds for bugs it's usually better to start a new issue.

[ejbartelds]

See #2934 @mpkorstanje