output: allow records size 0 on flb event type logs processor

[niedbalski]

Description.

For full context, refer to #9553.

A chunk with no records is a valid scenario for logs (e.g., when all are dropped) and should either be added to the flush list or released immediately. However, the chunk should not attempt to flush.

This patch allows for the record size to be zero for FLB_EVENT_TYPE_LOGS.

I created a basic processor that calls flb_mp_chunk_cobj_record_destroy on each record, resulting in a chunk with no records (size = 0). When this happens, coroutine tasks do not run, and fluentbit_storage_mem_chunks continues to grow, causing a memory leak (refer to the issue report).

#include <fluent-bit/flb_processor_plugin.h>

static int cb_init(struct flb_processor_instance *ins,
                   void *source_plugin_instance,
                   int source_plugin_type,
                   struct flb_config *config)
{
    return FLB_PROCESSOR_SUCCESS;
}

static int cb_exit(struct flb_processor_instance *ins, void *data)
{
    return FLB_PROCESSOR_SUCCESS;
}

static int cb_process_logs(struct flb_processor_instance *ins,
                           void *chunk_data, const char *tag, int tag_len){
    struct flb_mp_chunk_cobj *chunk_cobj = (struct flb_mp_chunk_cobj *) chunk_data;
    struct flb_mp_chunk_record *record;
    while (flb_mp_chunk_cobj_record_next(chunk_cobj, &record) == FLB_MP_CHUNK_RECORD_OK) {
        flb_mp_chunk_cobj_record_destroy(chunk_cobj, record);  
    }
    return FLB_PROCESSOR_SUCCESS;
}


static struct flb_config_map config_map[] = {
    {0}
};

struct flb_processor_plugin processor_template_plugin = {
    .name               = "chunk_leaks",
    .description        = "chunk_leaks processor",
    .cb_init            = cb_init,
    .cb_process_logs    = cb_process_logs,
    .cb_process_metrics = NULL,
    .cb_process_traces  = NULL,
    .cb_exit            = cb_exit,
    .config_map         = NULL,
    .flags              = 0
};

With this configuration

service:
  http_server: on
pipeline:
  inputs:
    - name: dummy
      rate: 1000
      dummy: '{"message": "a message", "event_cluster": "raptor"}'
  outputs:
    - name: stdout
      match: '*'
      processors:
        logs:
            - name: chunk_leaks

Without this patch heap allocation graphs show continuous growth in heap memory allocations not being returned, with the chunk counter also incrementing in metrics.

With this patch enabled, chunks are released

Testing

• Example configuration file for the change
• Debug log output from testing the change
• Attached Valgrind output that shows no leaks or memory corruption was found

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[niedbalski]

@edsiper / @cosmo0920 seems stable / no chunk leaks for 15m or so. Appreciated suggestions on testing if required.

[cosmo0920]

Looks good and I have never imagined and though as not returned chunks cause memory leaks. This could be interesting behavior and we need to return when the size of chunks are zero. It won't be caused without processors but now we have processors. So, we need to change this behavior.