out_azure: improvement: add table prefix support for Azure Log Analytics plugin #7663

kforeverisback · 2023-07-07T00:11:12Z

Added Prefix support for Azure Log Analytics output plugin

This PR implements this issue: #7420

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change
Attached Valgrind output that shows no leaks or memory corruption was found

~~- [N/A] Run local packaging test showing all targets (including any new ones) build.~~
~~- [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).~~

Documentation
Documentation PR Link: fluent/fluent-bit-docs#1154

Input Configs

Config scenario 1: Original config

Will create a table named someapp

[INPUT]
    NAME   dummy
    dummy {"log":"simple log 1", "kubernetes":{"pod_name":"pod-695df8f557-pq22p", "namespace_name":"default", "labels":{"app":"sampletable", "pod-template-hash":"695df8f557", "version":"v1"}}}
    Tag    test_tag
[OUTPUT]
    Name         azure
    Match        *
    Log_Type     someapp
    Customer_ID  ras3ngan-398e-abcd-b84c-9xxxxxxxx19e
    Shared_Key   g3njutsu

Config scenario 2: Reading prefix from k8s label

[INPUT]
    NAME   dummy
    dummy {"log":"simple log 1", "kubernetes":{"pod_name":"pod-695df8f557-pq22p", "namespace_name":"default", "labels":{"app":"sampletable", "pod-template-hash":"695df8f557", "version":"v1"}}}
    Tag    test_tag
[OUTPUT]
    Name         azure
    Match        *
    log_type_key $kubernetes['labels']['app']
    Customer_ID  g3njutsu-398e-abcd-b84c-9xxxxxxxx19e
    Shared_Key   ras3ngan

Config scenario 3: non-existing k8s label

[INPUT]
    NAME   dummy
    dummy {"log":"simple log 1", "kubernetes":{"pod_name":"pod-695df8f557-pq22p", "namespace_name":"default", "labels":{"app":"sampletable", "pod-template-hash":"695df8f557", "version":"v1"}}}
    Tag    test_tag
[OUTPUT]
    Name         azure
    Match        *
    log_type_key $kubernetes['labels']['non-existing-label']
    Customer_ID  ras3ngan-398e-abcd-b84c-9xxxxxxxx19e
    Shared_Key   g3njutsu

This'll still work, making it backward compatible; but will print a warning message that the label was not found.

Valgrind Logs

Showed output for config scenario 3.
But all other configs produce same valgrind output without any mem leaks.

$ valgrind --leak-check=full bin/fluent-bit -c ./sample-config.conf
==2898== Memcheck, a memory error detector
==2898== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2898== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==2898== Command: bin/fluent-bit -c ../.vscode/sample-config.conf
==2898== 
Fluent Bit v2.1.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/03 09:50:57] [ info] Configuration:
[2023/07/03 09:50:57] [ info]  flush time     | 1.000000 seconds
[2023/07/03 09:50:57] [ info]  grace          | 5 seconds
[2023/07/03 09:50:57] [ info]  daemon         | 0
[2023/07/03 09:50:57] [ info] ___________
[2023/07/03 09:50:57] [ info]  inputs:
[2023/07/03 09:50:57] [ info]      dummy
[2023/07/03 09:50:57] [ info] ___________
[2023/07/03 09:50:57] [ info]  filters:
[2023/07/03 09:50:57] [ info] ___________
[2023/07/03 09:50:57] [ info]  outputs:
[2023/07/03 09:50:57] [ info]      azure.0
[2023/07/03 09:50:57] [ info] ___________
[2023/07/03 09:50:57] [ info]  collectors:
[2023/07/03 09:50:58] [ info] [fluent bit] version=2.1.5, commit=3102005552, pid=2898
[2023/07/03 09:50:58] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/03 09:50:58] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/03 09:50:58] [ info] [cmetrics] version=0.6.2
[2023/07/03 09:50:58] [ info] [ctraces ] version=0.3.1
[2023/07/03 09:50:58] [ info] [input:dummy:dummy.0] initializing
[2023/07/03 09:50:58] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/07/03 09:50:58] [debug] [dummy:dummy.0] created event channels: read=23 write=24
[2023/07/03 09:50:58] [debug] [azure:azure.0] created event channels: read=25 write=26
[2023/07/03 09:50:58] [ info] [output:azure:azure.0] customer_id='ras3ngan-398e-abcd-b84c-9xxxxxxxx19e' host='ras3ngan-398e-abcd-b84c-9xxxxxxxx19e.ods.opinsights.azure.com:443'
[2023/07/03 09:50:58] [ info] [sp] stream processor started
[2023/07/03 09:50:59] [debug] [input chunk] update output instances with new chunk size diff=166, records=1, input=dummy.0
[2023/07/03 09:51:00] [debug] [task] created task=0x53369c0 id=0 OK
[2023/07/03 09:51:00] [debug] [input chunk] update output instances with new chunk size diff=166, records=1, input=dummy.0
[2023/07/03 09:51:01] [debug] [task] created task=0x547fc50 id=1 OK
[2023/07/03 09:51:01] [debug] [input chunk] update output instances with new chunk size diff=166, records=1, input=dummy.0
[2023/07/03 09:51:01] [debug] [upstream] KA connection #33 to ras3ngan-398e-abcd-b84c-9xxxxxxxx19e.ods.opinsights.azure.com:443 is connected
[2023/07/03 09:51:01] [ warn] [output:azure:azure.0] Record accessor key not matched
[2023/07/03 09:51:01] [debug] [http_client] not using http_proxy for header
[2023/07/03 09:51:02] [ info] [output:azure:azure.0] customer_id=ras3ngan-398e-abcd-b84c-9xxxxxxxx19e, HTTP status=200
[2023/07/03 09:51:02] [debug] [upstream] KA connection #33 to ras3ngan-398e-abcd-b84c-9xxxxxxxx19e.ods.opinsights.azure.com:443 is now available
....
....
[2023/07/03 09:51:09] [debug] [out flush] cb_destroy coro_id=9
[2023/07/03 09:51:09] [debug] [task] destroy task=0x6838b20 (task_id=0)
^C[2023/07/03 09:51:09] [engine] caught signal (SIGINT)
[2023/07/03 09:51:09] [debug] [task] created task=0x6896c50 id=0 OK
[2023/07/03 09:51:09] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/03 09:51:09] [ info] [input] pausing dummy.0
[2023/07/03 09:51:09] [debug] [upstream] KA connection #33 to ras3ngan-398e-abcd-b84c-9xxxxxxxx19e.ods.opinsights.azure.com:443 has been assigned (recycled)
[2023/07/03 09:51:09] [ warn] [output:azure:azure.0] Record accessor key not matched
[2023/07/03 09:51:09] [debug] [http_client] not using http_proxy for header
[2023/07/03 09:51:09] [ info] [output:azure:azure.0] customer_id=ras3ngan-398e-abcd-b84c-9xxxxxxxx19e, HTTP status=200
[2023/07/03 09:51:09] [debug] [upstream] KA connection #33 to ras3ngan-398e-abcd-b84c-9xxxxxxxx19e.ods.opinsights.azure.com:443 is now available
[2023/07/03 09:51:09] [debug] [out flush] cb_destroy coro_id=10
[2023/07/03 09:51:09] [debug] [task] destroy task=0x6896c50 (task_id=0)
[2023/07/03 09:51:10] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/03 09:51:10] [ info] [input] pausing dummy.0
==2898== 
==2898== HEAP SUMMARY:
==2898==     in use at exit: 761,304 bytes in 5,812 blocks
==2898==   total heap usage: 22,235 allocs, 16,423 frees, 6,698,345 bytes allocated
==2898== 
==2898== LEAK SUMMARY:
==2898==    definitely lost: 0 bytes in 0 blocks
==2898==    indirectly lost: 0 bytes in 0 blocks
==2898==      possibly lost: 0 bytes in 0 blocks
==2898==    still reachable: 761,304 bytes in 5,812 blocks
==2898==         suppressed: 0 bytes in 0 blocks
==2898== Reachable blocks (those to which a pointer was found) are not shown.
==2898== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2898== 
==2898== For lists of detected and suppressed errors, rerun with: -s
==2898== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Kushal Azim Ekram <[email protected]>

patrick-stephens · 2023-07-26T17:10:50Z

For fuzzer issues I would check with @DavidKorczynski but this looks like an issue that was fixed in master recently.

DavidKorczynski · 2023-07-26T17:15:16Z

CIFuzz failed because of a detected leak:

2023-07-07T00:26:06.4838110Z =================================================================
2023-07-07T00:26:06.4838407Z ==104==ERROR: LeakSanitizer: detected memory leaks
2023-07-07T00:26:06.4838594Z 
2023-07-07T00:26:06.4838738Z Direct leak of 8 byte(s) in 1 object(s) allocated from:
2023-07-07T00:26:06.4839213Z     #0 0x52ef96 in malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
2023-07-07T00:26:06.4839762Z     #1 0x56f166 in ctr_attributes_create /src/fluent-bit/lib/ctraces/src/ctr_attributes.c:26:12
2023-07-07T00:26:06.4840358Z     #2 0x57483b in unpack_instrumentation_scope_attributes /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:111:22
2023-07-07T00:26:06.4841009Z     #3 0x57a73c in ctr_mpack_unpack_map /src/fluent-bit/lib/ctraces/src/ctr_mpack_utils.c:378:30
2023-07-07T00:26:06.4841595Z     #4 0x574567 in unpack_scope_span_instrumentation_scope /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:154:12
2023-07-07T00:26:06.4842215Z     #5 0x57a73c in ctr_mpack_unpack_map /src/fluent-bit/lib/ctraces/src/ctr_mpack_utils.c:378:30
2023-07-07T00:26:06.4842766Z     #6 0x57439a in unpack_scope_span /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:609:12
2023-07-07T00:26:06.4843289Z     #7 0x57a9bb in ctr_mpack_unpack_array /src/fluent-bit/lib/ctraces/src/ctr_mpack_utils.c:444:18
2023-07-07T00:26:06.4843867Z     #8 0x572b46 in unpack_resource_span_scope_spans /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:616:12
2023-07-07T00:26:06.4844431Z     #9 0x57a73c in ctr_mpack_unpack_map /src/fluent-bit/lib/ctraces/src/ctr_mpack_utils.c:378:30
2023-07-07T00:26:06.4844973Z     #10 0x57286b in unpack_resource_span /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:651:12
2023-07-07T00:26:06.4845508Z     #11 0x57a9bb in ctr_mpack_unpack_array /src/fluent-bit/lib/ctraces/src/ctr_mpack_utils.c:444:18
2023-07-07T00:26:06.4846055Z     #12 0x572706 in unpack_resource_spans /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:658:12
2023-07-07T00:26:06.4846596Z     #13 0x57a73c in ctr_mpack_unpack_map /src/fluent-bit/lib/ctraces/src/ctr_mpack_utils.c:378:30
2023-07-07T00:26:06.4847128Z     #14 0x5724f7 in unpack_context /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:669:12
2023-07-07T00:26:06.4847664Z     #15 0x5724f7 in ctr_decode_msgpack_create /src/fluent-bit/lib/ctraces/src/ctr_decode_msgpack.c:691:14
2023-07-07T00:26:06.4848245Z     #16 0x569f05 in LLVMFuzzerTestOneInput /src/fluent-bit/tests/internal/fuzzers/ctrace_fuzzer.c:16:5
2023-07-07T00:26:06.4849001Z     #17 0x43de03 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
2023-07-07T00:26:06.4849827Z     #18 0x43d5ea in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:514:3
2023-07-07T00:26:06.4850750Z     #19 0x43f454 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:826:7
2023-07-07T00:26:06.4851715Z     #20 0x43f689 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:857:3
2023-07-07T00:26:06.4852530Z     #21 0x42ecef in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:912:6
2023-07-07T00:26:06.4853197Z     #22 0x458342 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
2023-07-07T00:26:06.4853820Z     #23 0x7f3eeb38e082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)

The leak is unrelated to the PR, so the PR is good to go from that perspective.

kforeverisback · 2023-07-26T17:55:40Z

Thanks @DavidKorczynski and @patrick-stephens for the reply and help.

felfa01 · 2023-08-09T14:03:52Z

Thanks @DavidKorczynski and @patrick-stephens for the reply and help.

@patrick-stephens
Can this be merged anytime soon? It is a nice, sought-after feature 😀

patrick-stephens · 2023-08-09T14:15:37Z

I cannot review the implementation - I cover build automation really - but it will be in the review queue so I'm sure @leonardo-albertovich will get to it as soon as he can. Make sure everything is ready but I think it is.

kforeverisback · 2023-08-10T22:46:29Z

I'll do a main ~~rebase~~ merge next week, to make sure everything is latest (if its not reviewed by then already ⚡)

…alytics_prefix Signed-off-by: Kushal Azim Ekram <[email protected]>

…ics plugin (#7663) --------- Signed-off-by: Kushal Azim Ekram <[email protected]>

kforeverisback added 2 commits June 25, 2023 11:10

out_azure: Added log_type_key option to azure plugin

6bf6a67

Signed-off-by: Kushal Azim Ekram <[email protected]>

pipeline: output: azure: changed log level

3102005

Signed-off-by: Kushal Azim Ekram <[email protected]>

kforeverisback requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners July 7, 2023 00:11

github-actions bot added the docs-required label Jul 7, 2023

kforeverisback temporarily deployed to pr July 7, 2023 00:11 — with GitHub Actions Inactive

This was referenced Jul 7, 2023

out_azure: documentation for table prefix support fluent/fluent-bit-docs#1154

Merged

Dynamically set the logType in the Azure Log Analytics fluentbit plugin #7420

Closed

kforeverisback temporarily deployed to pr July 7, 2023 00:34 — with GitHub Actions Inactive

Merge branch 'fluent:master' into feature/azure_loganalytics_prefix

82c492b

kforeverisback temporarily deployed to pr July 26, 2023 17:14 — with GitHub Actions Inactive

patrick-stephens added the ok-package-test Run PR packaging tests label Jul 26, 2023

kforeverisback temporarily deployed to pr July 26, 2023 17:40 — with GitHub Actions Inactive

kforeverisback temporarily deployed to pr July 26, 2023 17:42 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/master' into feature/azure_logan…

75e7965

…alytics_prefix Signed-off-by: Kushal Azim Ekram <[email protected]>

kforeverisback temporarily deployed to pr August 21, 2023 19:44 — with GitHub Actions Inactive

kforeverisback temporarily deployed to pr August 21, 2023 20:14 — with GitHub Actions Inactive

edsiper merged commit 4759dcc into fluent:master Sep 23, 2023
71 of 73 checks passed

Porkepix mentioned this pull request Sep 28, 2023

fluent-bit 2.1.10 Homebrew/homebrew-core#146545

Merged

felfa01 mentioned this pull request Oct 4, 2023

add logTypeKey to azureLogAnalytics fluent/fluent-operator#945

Open

leonardo-albertovich pushed a commit that referenced this pull request Oct 5, 2023

out_azure: improvement: add table prefix support for Azure Log Analyt…

d2021f7

…ics plugin (#7663) --------- Signed-off-by: Kushal Azim Ekram <[email protected]>

kforeverisback mentioned this pull request Jul 16, 2024

out_azure_kusto : fix multiple files tail issue and timeout issue #8430

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_azure: improvement: add table prefix support for Azure Log Analytics plugin #7663

out_azure: improvement: add table prefix support for Azure Log Analytics plugin #7663

kforeverisback commented Jul 7, 2023 •

edited

Loading

patrick-stephens commented Jul 26, 2023

DavidKorczynski commented Jul 26, 2023

kforeverisback commented Jul 26, 2023

felfa01 commented Aug 9, 2023

patrick-stephens commented Aug 9, 2023

kforeverisback commented Aug 10, 2023 •

edited

Loading

out_azure: improvement: add table prefix support for Azure Log Analytics plugin #7663

out_azure: improvement: add table prefix support for Azure Log Analytics plugin #7663

Conversation

kforeverisback commented Jul 7, 2023 • edited Loading

Input Configs

Config scenario 1: Original config

Config scenario 2: Reading prefix from k8s label

Config scenario 3: non-existing k8s label

Valgrind Logs

patrick-stephens commented Jul 26, 2023

DavidKorczynski commented Jul 26, 2023

kforeverisback commented Jul 26, 2023

felfa01 commented Aug 9, 2023

patrick-stephens commented Aug 9, 2023

kforeverisback commented Aug 10, 2023 • edited Loading

kforeverisback commented Jul 7, 2023 •

edited

Loading

kforeverisback commented Aug 10, 2023 •

edited

Loading