fork-aware-tx-pool: add heavy load tests based on zombienet

[iulianbarbu]

Description

Builds up towards addressing #5497 by creating some zombienet-sdk code infra that can be used to spin regular networks, as described in the fork aware transaction pool testing setup added here #7100. It will be used for developing tests against such networks, and to also spawn them on demand locally through tooling that will be developed in follow ups.

Integration

Node/runtime developers can run tests based on the zombienet-sdk infra that spins frequently used networks which can be used for analyzing behavior of various node related components, like fork aware transaction pool.

Review Notes

• Uses ttxt API implemented here: https://github.com/michalkucharczyk/tx-test-tool/pull/22/files
• currently only a test scenario is considered: 10k future & 10k ready txs are sent in parallel to a fatp based collator, and at the end we assert on two 10k batches of txs have been finalized.

[michalkucharczyk]

So far looks good.

I was thinking about CLI. Maybe we don't need it, after all? Instead, we could use following:

$ cargo test --test stand_alone -- --exact run_single_collator_network

which would just:

• spawn the network,
• print out the location of the executed binaries (this one is important to me, to be absolutely sure that I don't run test on old binaries),
• print out the location of logs file,
• ... or maybe just print zombienet summary - with ports, params, etc...
• wait forever

Then one could use any tooling to just send transcations to this network.
We could start with this and see how it goes. We could do next iteration from here, and not over-complicate it from the beginning. In that way we would use the same config for manual testing, and for pre-defined test suits.

One more idea to control parameters would be using environment variables (not needed in the first step). Provides flexibility, less convenient to use comparing to CLI args, but much easier to implement.

export TXPOOLTESTS_POOL_LIMIT=1000
$ cargo test --test stand_alone -- --exact run_single_collator_network

Still we can have all the integration tests in different test module, reusing the same network configurations as those spawned in stand_alone mod, for example:

$ cargo test --test integration -- --exact single_collator_network__single_account_1M_txs

stand_alone tests would be excluded from cargo test command (as they never terminated on their own).

Any thoughts on this?

[pepoviola]

Hi, sorry to chime in. This is already checked by zombienet-sdk internally (for each cmd to execute and the workers).

[michalkucharczyk]

Is zombienet-sdk capable of printing full executable paths? (I know, I am a bit paranoid on this 😅)

[iulianbarbu]

@pepoviola can you point me to where we're doing these checks in zombienet-sdk?

[iulianbarbu]

So far the logs zombienet emits aren't showing the full binary paths for the node binaries it executes. @michalkucharczyk does it help if the logs contain the dump of the $PATH variable? Thinking we can make a feature request for zn-sdk logs to show that when configuring the network.

At the same time, @pepoviola , is there a way to tell a log file path to zn-sdk so that the log records emitted by zn-sdk could be tailed and easily analyzed later on, instead of showing it directly to stdout (which depending on the terminal settings, can be limited and polluted with other outputs?

[pepoviola]

Hi @iulianbarbu / @michalkucharczyk, I'm working on an small cli to spawn from toml and we already can load tomls. Did you think that could be handy here?
Thanks!

[michalkucharczyk]

Hi @iulianbarbu / @michalkucharczyk, I'm working on an small cli to spawn from toml and we already can load tomls. Did you think that could be handy here? Thanks!

My 3 cents:
Our goal here is to have some abstraction that allows to run some testsuit against predefined network, and also run exactly the same predefined network to conduct manual tests.

We actually want to spawn network programatically. So I am not sure that cli will be helpful here. But having some API in zombienet that would accept toml and spawn the network could be potentially helpful. Especially when it comes to customization - instead of playing with CLI args or enviroment variables as I proposed in my previous comment we could just edit toml file.

On the other hand, it seems that using zn-sdk is not that difficult.

@iulianbarbu what is your opinion?

[iulianbarbu]

Responding to the last messages where @pepoviola chimed in:

having some API in zombienet that would accept toml and spawn the network could be potentially helpful. Especially when it comes to customization - instead of playing with CLI args or enviroment variables as I proposed in my previous comment we could just edit toml file.

+1 to this idea @michalkucharczyk . I personally prefer Rust and zn-sdk for the testsuite, while for manual runs, if we'd be able to import the tomls directly with zn-sdk, and have the option to also use them with a CLI, then we can have the best of both worlds. It would be just a preference for how we'd like to do the manual testing, because we can still run the testsuite locally, by changing things within the rust tests, but if we want just to run the network and then do other stuff against it, we'd have the CLI as well.

we already can load tomls.

yup, thanks @pepoviola for confirming this offline. For reference: https://docs.rs/zombienet-sdk/latest/zombienet_sdk/struct.NetworkConfig.html#method.load_from_toml.

I'm working on an small cli to spawn from toml

@pepoviola how different would be from the existing zombienet CLI and why do we need another one?

[alindima]

We don't want to commit the chainspecs to the repo, right? The better way is to generate them using build scripts, like we do here for example: https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/zombienet-sdk-tests/build.rs

[michalkucharczyk]

We don't want to commit the chainspecs to the repo, right? The better way is to generate them using build scripts, like we do here for example: https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/zombienet-sdk-tests/build.rs

Good point. Chain-spec can be avoided with this commit:
#6267

Shall be enough to define dev-accounts in the genesis-patch (which in turn can be given in zobienet toml file).

However, not sure if we need extra aka-build.rs step. My guess would be it is not needed.

[iulianbarbu]

I think we need to mention the path to the runtime as well if mentioning the patch. That's a variable path, but we can assume it is target/release/wbuild/..., which should be fine for 99% of the cases (?). When loading the network with zn-sdk from zombienet.toml we must mention the runtime path in the toml file explicitly since there is no API to achieve changes after obtaining the network config.

However, not sure if we need extra aka-build.rs step. My guess would be it is not needed.

Can't see either how a build.rs can help when using zombienet-sdk with tomls loading.

[michalkucharczyk]

I think we need to mention the path to the runtime as well if mentioning the patch.

Maybe we don't need runtime path?
For runtimes that are already embedded in polkadot-parachain binary it should be enought to just give the name of the runtime. It is already done in many tomls across the codebase.

We could skip yap which is kinda experimental. (or add it in 2nd phase / followup).

[iulianbarbu]

Oh yeah, I guess asset-hub-* is covered. Leaving out experimental runtimes sounds good.

[michalkucharczyk]

Maybe we should expose rpc port for bob too?

I think we can also enabled fatxpool on the relaychain. So we would need to copy some args to the relaychain nodes.

Some scenarios sending to the relay-chain would nice. As discussed in element they can also be quicker for development (no need to wait for 10 blocks).

[iulianbarbu]

Sending txs to relaychain nodes fail with \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")). I exposed the rpc port for bob, and I enabled fatxpool on the relaychain nodes. Will debug a bit more the issue, but lmk if you have any idea for why I am getting the error.

[michalkucharczyk]

I guess devAccounts missing in relay chain's genesis.

[iulianbarbu]

Yup. I tested locally with the devAccounts patch for the relaychain, but it didn't look like it was considered by zn-sdk, but I tried it once more time after our recent discussions and seems like it gets applied. Maybe I added the patch to another file initially, when it didn't work.

Anyways, I'll leave the test in a loop for tonight. The first time I observed the patch was applied had the test failing after finalizing a few thousands of ready txs, with a panic in ttxt:

---- send_future_and_ready_from_many_accounts_to_relaychain stdout ----
thread 'send_future_and_ready_from_many_accounts_to_relaychain' panicked at /home/ubuntu/.cargo/git/checkouts/tx-test-tool-6354a49bb28b4c59/d0a6ab2/src/runner.rs:140:17:
explicit panic
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[iulianbarbu]

I observed three failures in a few hours ran on repeat of the relaychain test (but not the panic above). Now I have doubts for the parachain too. I am running again the relaychain test in a loop and try to isolate better the failures to analyze them. I'll do the same for the parachain test.

[michalkucharczyk]

Looks good, almost there. Some final nits left.
Will do one more round of review.

[michalkucharczyk]

Spotted one more annoying issue - just made ls -al in the tests dir 🙈

I propose to add .with_log_file_dir, .with_log_file_name method to the builder, and place the file to the zombienet directory. If log file name is not set it could be automatically generated like this:

file_name = format!("ttxt_{log_prefix}")`

The log shall be stored in file_dir/file_name location.

This will allow to associate ttxt logs with zombienet run.