Kafka Transactional writer

[big-andy-coates]

Is your feature request related to a problem? Please describe.
If change events read from the database are being output to Kafka, with state written to a file on disk, there is the possibility to get duplicate events in Kafka, e.g. if OLR writes to Kafka, then crashes before writing to the file on disk, when it restarts it will resend change events to Kafka, causing duplicates in Kafka.

Describe the solution you'd like
Exactly once delivery semantics are possible using Kafka's transactional support. One solution would be to enhance OLR to support storing state in a Kafka topic, with writes to store change events and state contained in the same Kafka transaction. Then Kafka either contains both the change events and the updated state, or neither.

Describe alternatives you've considered
If storing change events and state in two different technologies, e.g. Kafka and file on disk, it's not possible to support exactly once delivery semantics. So the only solution I can think of is to enhance OLR to support storing state in Kafka (either in its own state topic, or potentially in the headers of the change events it writes to Kafka), and leverage Kafka's transaction support to ensure no duplicates are published in the presence of partial failures.

Additional context
Currently running a spike to investigate OLR. At this point, I'm looking to touch base and see if such a feature seems possible for OLR...? It's been a long time since I've worked in C++, but there may be scope for me to help with the feature. TBC.

[bersler]

Hi @big-andy-coates, thanks for creating a separate ticket for this task.
This is a known topic. The current implementation does not use transactional Kafka API.
This might be a good task to do if there is a business demand for this feature. If there is a business case then this feature might be implemented.

[big-andy-coates]

Good to hear @bersler.

Let's see which way the winds blow with my employer and if this gets resources assigned.

[bersler]

Hi @big-andy-coates , do you actually have a business case with a customer who would benefit from this improvement?

[big-andy-coates]

I'm working with a client that is currently investigating if OLR meets their needs.

Initially, eos is not a requirement, but eos is a requirement in the medium term.

[bersler]

I plan to make some code refactor of the Kafka writer anyway. So maybe it would get implemented.