Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_tcp: Add a capability to inject source IP #7672

Merged
merged 10 commits into from
Mar 14, 2024

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Jul 10, 2023

Currently, in_tcp plugin does not offer adding source address feature.
This PR provides this feature.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[INPUT]
    Name        tcp
    Listen      0.0.0.0
    Port        5170
    Chunk_Size  32
    Buffer_Size 64
    Format      json
    Source_Address_Key source_host

[OUTPUT]
    Name        stdout
    Match       *
  • Debug log output from testing the change
Fluent Bit v2.1.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/10 14:30:42] [ info] Configuration:
[2023/07/10 14:30:42] [ info]  flush time     | 1.000000 seconds
[2023/07/10 14:30:42] [ info]  grace          | 5 seconds
[2023/07/10 14:30:42] [ info]  daemon         | 0
[2023/07/10 14:30:42] [ info] ___________
[2023/07/10 14:30:42] [ info]  inputs:
[2023/07/10 14:30:42] [ info]      tcp
[2023/07/10 14:30:42] [ info] ___________
[2023/07/10 14:30:42] [ info]  filters:
[2023/07/10 14:30:42] [ info] ___________
[2023/07/10 14:30:42] [ info]  outputs:
[2023/07/10 14:30:42] [ info]      stdout.0
[2023/07/10 14:30:42] [ info] ___________
[2023/07/10 14:30:42] [ info]  collectors:
[2023/07/10 14:30:42] [ info] [fluent bit] version=2.1.7, commit=0714d10889, pid=372853
[2023/07/10 14:30:42] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/10 14:30:42] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/10 14:30:42] [ info] [cmetrics] version=0.6.3
[2023/07/10 14:30:42] [ info] [ctraces ] version=0.3.1
[2023/07/10 14:30:42] [ info] [input:tcp:tcp.0] initializing
[2023/07/10 14:30:42] [ info] [input:tcp:tcp.0] storage_strategy='memory' (memory only)
[2023/07/10 14:30:42] [debug] [tcp:tcp.0] created event channels: read=21 write=22
[2023/07/10 14:30:42] [debug] [downstream] listening on 0.0.0.0:5170
[2023/07/10 14:30:42] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2023/07/10 14:30:42] [ info] [sp] stream processor started
[2023/07/10 14:30:42] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/10 14:30:44] [debug] [input chunk] update output instances with new chunk size diff=86, records=1, input=tcp.0
[2023/07/10 14:30:44] [debug] [task] created task=0x7f5a4802a4b0 id=0 OK
[2023/07/10 14:30:44] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] tcp.0: [[1688967044.132801215, {}], {"key 1"=>123456789, "key 2"=>"abcdefg", "source_host"=>"tcp://127.0.0.1:46958"}]
[2023/07/10 14:30:44] [debug] [out flush] cb_destroy coro_id=0
[2023/07/10 14:30:44] [debug] [task] destroy task=0x7f5a4802a4b0 (task_id=0)
[2023/07/10 14:30:57] [debug] [input chunk] update output instances with new chunk size diff=95, records=1, input=tcp.0
[2023/07/10 14:30:58] [debug] [task] created task=0x7f5a4802bda0 id=0 OK
[2023/07/10 14:30:58] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] tcp.0: [[1688967057.809867670, {}], {"msg"=>[{"key 1"=>123456789, "key 2"=>"abcdefg"}], "source_host"=>"tcp://127.0.0.1:35472"}]
[2023/07/10 14:30:58] [debug] [out flush] cb_destroy coro_id=1
[2023/07/10 14:30:58] [debug] [task] destroy task=0x7f5a4802bda0 (task_id=0)
^C[2023/07/10 14:30:59] [engine] caught signal (SIGINT)
[2023/07/10 14:30:59] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/10 14:31:00] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/10 14:31:00] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/10 14:31:00] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==373310== 
==373310== HEAP SUMMARY:
==373310==     in use at exit: 95,170 bytes in 702 blocks
==373310==   total heap usage: 2,921 allocs, 2,219 frees, 1,306,526 bytes allocated
==373310== 
==373310== LEAK SUMMARY:
==373310==    definitely lost: 0 bytes in 0 blocks
==373310==    indirectly lost: 0 bytes in 0 blocks
==373310==      possibly lost: 0 bytes in 0 blocks
==373310==    still reachable: 95,170 bytes in 702 blocks
==373310==         suppressed: 0 bytes in 0 blocks
==373310== Reachable blocks (those to which a pointer was found) are not shown.
==373310== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==373310== 
==373310== For lists of detected and suppressed errors, rerun with: -s
==373310== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1155

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 force-pushed the cosmo0920-append-source-address-in_tcp branch from d1809fb to 1655169 Compare July 10, 2023 05:46
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:46 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:46 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:46 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 force-pushed the cosmo0920-append-source-address-in_tcp branch from 1655169 to ae3ecd8 Compare July 10, 2023 05:47
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:50 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:50 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 05:50 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 06:14 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 force-pushed the cosmo0920-append-source-address-in_tcp branch from ae3ecd8 to 973ab52 Compare July 10, 2023 07:11
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 07:11 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 07:11 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 07:11 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 07:34 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 16:00 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 16:00 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 16:00 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 16:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:59 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:59 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 marked this pull request as ready for review July 25, 2023 09:53
@cosmo0920 cosmo0920 temporarily deployed to pr September 1, 2023 02:34 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 1, 2023 03:07 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 1, 2023 03:14 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 1, 2023 03:14 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 1, 2023 03:14 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 1, 2023 03:47 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 force-pushed the cosmo0920-append-source-address-in_tcp branch from 36b16c9 to 50077bc Compare September 27, 2023 10:23
@cosmo0920 cosmo0920 temporarily deployed to pr September 27, 2023 10:24 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 27, 2023 10:24 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 27, 2023 10:24 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr September 27, 2023 10:49 — with GitHub Actions Inactive
@edsiper edsiper merged commit 71d00b7 into master Mar 14, 2024
41 of 42 checks passed
@edsiper edsiper deleted the cosmo0920-append-source-address-in_tcp branch March 14, 2024 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants