Support for Multiple Schema Registries

[kjkondratuk]

Reposting here because I accidentally posted in the provectus repo initially!

I currently have a use case where it seems like I could benefit from support for schema registries as a separate entity from kafka clusters. While it's true that in many cases its admirable to strive for a single schema registry to centrally manage schemas, I find myself at a bit of an impasse where I have to support a Buf schema registry (with compatibility for the Confluent Schema Registry API) as well as a self-hosted Confluent Schema Registry.

As far as I'm aware, it is not currently possible to define a kafka cluster which can resolve schemas against multiple registries--you need to define a separate cluster for each registry with the same broker configuration. Duplicating the cluster is necessary to have a version of the cluster which points to each registry. This is necessary in my circumstance because Buf doesn't support hosting Avro schemas (it's primarily a protobuf tool) which we use in some legacy cases, and CSR doesn't support some of the advanced features of Buf.

Currently, what the config looks like is:

kafka:
  clusters:
    - name: cluster-a
      bootstrapServers: cluster-a:<port>
      schemaRegistry: http://registry-a
      schemaRegistryAuth:
        username: <user>
        password: <pass>

    - name: cluster-a-other-registry
      bootstrapServers: cluster-a:<port>
      schemaRegistry: http://registry-b
      schemaRegistryAuth:
        username: <user>
        password: <pass>

I'd think something like the following would accomplish this quite nicely:

kafka:
  clusters:
    - name: cluster-a
      bootstrapServers: cluster-a:<port>
      schemaRegistries:
        - registry-a
        - registry-b

  schemaRegistries:
    registry-a:
      url: http://registry-a
      auth:
       username: <user>
       password: <pass>
    registry-b:
      url: http://registry-b
      auth:
        username: <user>
        password: <pass>

Where the schema registry connection reference can be re-used, and multiple schema registries specified per cluster. There are some UI controls that would have to be changed, but I'd expect that the schema registry could be selected (from amongst the configured schema registries, and possibly a default configured) when browsing messages in a topic.

It looks like there were a couple similar issues (or with overlap) out there already:

provectus/kafka-ui#4120
provectus/kafka-ui#1076
I am likely going to go with the simple repetition for the time being, but would be willing to contribute even though my Java is about 3 years out of date, if the change is a welcome one :)