[BUG] OpenSearch 2.19.0 with installed repository-s3 plugin cannot be stopped

[olfuerniss]

Describe the bug

When the repository-s3 plugin is installed, it is no longer possible to stop a running OpenSearch instance. Happens since 2.19.0.

Related component

Plugins

To Reproduce

1. Start OpenSearch with installed 'repository-s3' plugin

curl -O https://artifacts.opensearch.org/releases/bundle/opensearch/2.19.0/opensearch-2.19.0-linux-x64.tar.gz
tar xvfz opensearch-2.19.0-linux-x64.tar.gz
cd opensearch-2.19.0/
bin/opensearch-plugin install repository-s3
echo 'plugins.security.disabled: true' >> config/opensearch.yml
bin/opensearch

2. now wait until OpenSearch has started ... and try to stop it by pressing CONTROL/STRG+c

=> It does not stop, shows the following exception and hangs (must be killed)

[2025-02-14T11:02:16,876][ERROR][i.n.u.c.D.rejectedExecution] [Mac.lan] Failed to submit a listener notification task. Event loop shut down?
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
	at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2060) ~[?:?]
	at java.base/java.lang.Thread.getContextClassLoader(Thread.java:1502) ~[?:?]
	at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:239) ~[netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:232) ~[netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:226) ~[netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1059) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

Expected behavior

OpenSearch should not hang on stop when the repository-s3 plugin is installed.

Additional Details

Plugins

• all plugins of 2.19.0
• repository-s3

Host/Environment (please complete the following information):

• OS: macOS Sequoia
• Version 15.3.1

[spapadop]

faced exactly the same just after the upgrade (running on AlmaLinux9 and installing using the RPM)

[HenryTheSir]

Can confirm on RHEL8 with tarball installation and added s3 plugin

[cwperks]

Looks like the same stacktrace from netty/netty#14743

@reta will the netty upgrade to 4.1.118.Final resolve this issue?

[reta]

Thanks @cwperks

Looks like the same stacktrace from netty/netty#14743

Yes

@reta will the https://github.com/opensearch-project/OpenSearch/pull/17320resolve this issue?

Yes

@spapadop @HenryTheSir @olfuerniss could you folks please confirm if the issue is not happening when repository-s3 plugin is not installed?

[olfuerniss]

@reta Yes

[spapadop]

yeap I double-confirm, everything seems happy without repository-s3.

[reta]

The 2.19.0 didn't get Netty 4.1.118 (sadly), listing possible options here:

• 2.19.1 patch release (best option)
• build and publish s3-repository plugin locally
• in place patching (replace Netty 4.1.117 with 4.1.118 JARs)

@andrross any other options I am missing? thank you