compute: persistent worker command channel

[teskje]

This PR makes the compute workers reuse their broadcast command channels across client reconnects, rather than creating a new ones for each connection. Doing so avoids bugs caused by workers rendering the channel dataflow with different operator IDs upon reconnecting multiple times.

The change only applies to the compute cluster, so the ClusterClient, which is shared between both compute and storage, still retains its previous behavior of using a new set of channels for each client connection. To interface between the ClusterClient and the compute workers, a "channel adapter" is introduced to bridge between the two.

Now that the command channel isn't recreated for each client connection anymore, channel disconnects can't be used to detect client disconnects anymore. Instead, the CreateTimely command is used as punctuation. The ClusterClient is modified to pass through CreateTimely commands to the workers. The compute worker is modified to monitor the command stream for CreateTimely commands and initiate a reconciliation when it observes one. Additionally, the epoch is also used to tag responses internally, so they can be discarded if they were not meant for the current connection.

Now that the command channel isn't recreated for each client connection anymore, channel disconnects can't be used to detect client disconnects anymore. Instead, the CreateTimely command is used as punctuation. The ClusterClient is modified to pass through CreateTimely commands to the workers. The channel adapter is modified to extract for each new connection its epoch from the CreateTimely command and use it to tag commands sent through the command channel. The compute worker learns to monitor for epoch changes and initiate a reconciliation when it observes one. Likewise, the compute worker is modified to tag responses with the current epoch and the channel adapter learns to filter out responses with outdated epochs, which were intended for previous client connections.

Motivation

• This PR fixes a recognized bug.

Fixes https://github.com/MaterializeInc/database-issues/issues/8964

Also connected to https://github.com/MaterializeInc/incidents-and-escalations/issues/174

Tips for reviewer

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[teskje]

I considered making this a wrapper around client_rx instead to transform the command/response stream without spawning additional threads, similar to how the GenericClients work. But the thread version ends up much simpler to implement, so I stuck with that.

If we are worried with the overhead of having one additional thread per worker (no clue if we should be), we could make this a tokio task instead.