Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load testing MQTT QOS 0,1,2 with Empty messages in the retained topic and Zilla crashing #1164

Closed
vordimous opened this issue Jul 22, 2024 · 1 comment
Closed

Load testing MQTT QOS 0,1,2 with Empty messages in the retained topic and Zilla crashing #1164

vordimous opened this issue Jul 22, 2024 · 1 comment
Assignees
@bmaidics
Labels
bug Something isn't working

Comments

@vordimous
Copy link
Contributor

vordimous commented Jul 22, 2024 •
edited
Loading

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. run an MQTT kafka broker
  2. run repeat mosquito command
mosquitto_pub --url mqtt://localhost:7183/zilla -q 1 --message "Hello, world $(date)" --repeat 10000 -d -i sim-1;

  1. see empty key and empty messages in the mqtt-retained topic

  2. More load testing using 100 clients that produce 500 msgs each for an expected total of 50k messages:

QOS 2 has empty messages in the retained topic and doesn't deliver all 50k messages. ~500 msg/sec

for i in {1..100} ; do
mosquitto_pub --url mqtt://localhost:7183/qos2 -q 2 --message "QOS 2" --repeat 500 -d -i "sim-$i" >> "sim2-$i.out" &
done

QOS 1 has empty messages in the retained topic and does deliver all 50k messages. ~1k msg/sec

for i in {1..100} ; do
mosquitto_pub --url mqtt://localhost:7183/qos1 -q 1 --message "QOS 1" --repeat 500 -d -i "sim-$i" >> "sim1-$i.out" &
done

QOS 0 crashes before being able to finish

for i in {1..100} ; do
mosquitto_pub --url mqtt://localhost:7183/qos0 -q 0 --message "QOS 0" --repeat 500 -d -i "sim-$i" >> "sim0-$i.out" &
done

Exceptions encoutered during load testing QOS 0:

org.agrona.concurrent.AgentTerminationException: java.lang.IllegalArgumentException: hostname can't be null
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:841)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.doWork(AgentRunner.java:304)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.workLoop(AgentRunner.java:296)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.run(AgentRunner.java:162)
    at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.IllegalArgumentException: hostname can't be null
    at java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:159)
    at java.base/java.net.InetSocketAddress.<init>(InetSocketAddress.java:226)
    at [email protected]/io.aklivity.zilla.runtime.binding.tcp.internal.stream.TcpClientRouter.resolveInetSocketAddress(TcpClientRouter.java:288)
    at [email protected]/io.aklivity.zilla.runtime.binding.tcp.internal.stream.TcpClientRouter.resolve(TcpClientRouter.java:96)
    at [email protected]/io.aklivity.zilla.runtime.binding.tcp.internal.stream.TcpClientFactory.newStream(TcpClientFactory.java:155)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleBeginInitial(EngineWorker.java:1576)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleDefaultReadInitial(EngineWorker.java:1340)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleReadInitial(EngineWorker.java:1280)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleRead(EngineWorker.java:1227)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.concurent.ManyToOneRingBuffer.read(ManyToOneRingBuffer.java:229)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:835)
    ... 4 more
    Suppressed: java.lang.Exception: [engine/data#3]        [0x0303000000001115] streams=[consumeAt=0x0025e978 (0x000000000025e978), produceAt=0x0025f4d8 (0x000000000025f4d8)]
            at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:839)
            ... 4 more

org.agrona.concurrent.AgentTerminationException: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:841)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.doWork(AgentRunner.java:304)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.workLoop(AgentRunner.java:296)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.run(AgentRunner.java:162)
    at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
    at java.base/java.util.Objects.checkIndex(Objects.java:365)
    at java.base/java.util.ArrayList.get(ArrayList.java:428)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaCacheServerProduceFactory$KafkaCacheServerProduceFan.doServerFanInitialDataIfNecessary(KafkaCacheServerProduceFactory.java:650)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaCacheServerProduceFactory$KafkaCacheServerProduceFan.onServerFanInitialWindow(KafkaCacheServerProduceFactory.java:827)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaCacheServerProduceFactory$KafkaCacheServerProduceFan.onServerFanMessage(KafkaCacheServerProduceFactory.java:874)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleReadInitial(EngineWorker.java:1292)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleRead(EngineWorker.java:1227)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.concurent.ManyToOneRingBuffer.read(ManyToOneRingBuffer.java:229)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:835)
    ... 4 more
    Suppressed: java.lang.Exception: [engine/data#3]        [0x0303000000001a31] streams=[consumeAt=0x003681c8 (0x0000000000b681c8), produceAt=0x0036e948 (0x0000000000b6e948)]
            at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:839)
            ... 4 more
org.agrona.concurrent.AgentTerminationException: java.lang.IllegalStateException: Unable to write to streams buffer
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:841)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.doWork(AgentRunner.java:304)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.workLoop(AgentRunner.java:296)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.run(AgentRunner.java:162)
    at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.IllegalStateException: Unable to write to streams buffer
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.stream.Target.handleWrite(Target.java:159)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory.doFlush(KafkaMergedFactory.java:910)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaUnmergedProduceStream.doProduceInitialFlush(KafkaMergedFactory.java:4059)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaMergedStream.onMergedFetchFlush(KafkaMergedFactory.java:1463)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaMergedStream.onMergedInitialFlush(KafkaMergedFactory.java:1389)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaMergedStream.onMergedMessage(KafkaMergedFactory.java:1142)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleReadInitial(EngineWorker.java:1271)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleRead(EngineWorker.java:1227)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.concurent.ManyToOneRingBuffer.read(ManyToOneRingBuffer.java:229)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:835)
    ... 4 more
    Suppressed: java.lang.Exception: [engine/data#0]        [0x0000000000000a8d] streams=[consumeAt=0x001f49a0 (0x00000000005f49a0), produceAt=0x003ffff0 (0x00000000007ffff0)]
            at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:839)
            ... 4 more
org.agrona.concurrent.AgentTerminationException: java.lang.IllegalStateException: Unable to write to streams buffer
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:841)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.doWork(AgentRunner.java:304)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.workLoop(AgentRunner.java:296)
    at org.agrona.core/org.agrona.concurrent.AgentRunner.run(AgentRunner.java:162)
    at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.IllegalStateException: Unable to write to streams buffer
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.stream.Target.handleWrite(Target.java:159)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory.doFlush(KafkaMergedFactory.java:910)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaUnmergedProduceStream.doProduceInitialFlush(KafkaMergedFactory.java:4059)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaMergedStream.onMergedFetchFlush(KafkaMergedFactory.java:1463)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaMergedStream.onMergedInitialFlush(KafkaMergedFactory.java:1389)
    at [email protected]/io.aklivity.zilla.runtime.binding.kafka.internal.stream.KafkaMergedFactory$KafkaMergedStream.onMergedMessage(KafkaMergedFactory.java:1142)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleReadInitial(EngineWorker.java:1271)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.handleRead(EngineWorker.java:1227)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.concurent.ManyToOneRingBuffer.read(ManyToOneRingBuffer.java:229)
    at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:835)
    ... 4 more
    Suppressed: java.lang.Exception: [engine/data#2]        [0x0202000000000367] streams=[consumeAt=0x001da128 (0x00000000001da128), produceAt=0x0008a2c8 (0x000000000048a2c8)]
            at [email protected]/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker.doWork(EngineWorker.java:839)
            ... 4 more
@vordimous vordimous added the bug Something isn't working label Jul 22, 2024
@vordimous vordimous changed the title Empty messages in the mqtt retained topic Load testing MQTT QOS 0,1,2 with Empty messages in the retained topic and Zilla crashing Jul 23, 2024
@bmaidics
Copy link
Contributor

MQTT fixes related to this ticket:

Related issue regarding QoS 2 performance testing:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bmaidics bmaidics

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jfallows @vordimous @bmaidics