build: use the system provided packages if found #8930

ThomasDevoogdt · 2024-06-08T13:57:15Z

Some packages might already be present in the system, so use them if that is the case.
E.g. in buildroot, it doesn't make sense that e.g. luajit is compiled twice.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

[N/A] Example configuration file for the change
[N/A] Debug log output from testing the change

[N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

[N/A] Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

This overrules these PRs:

cosmo0920

I think it'll be reasonable. However, we shouldn't break the priority of building and including the bundled libraries. So, how about adding FLB_USE_SYSTEM_LIBRARIRS or similar flag to provide an option to prioritize the system installed libraries?
This is because using system installed libraries by default diversifies the version of dependent libraries.

ThomasDevoogdt · 2024-06-10T09:33:34Z

I think it'll be reasonable. However, we shouldn't break the priority of building and including the bundled libraries. So, how about adding FLB_USE_SYSTEM_LIBRARIRS or similar flag to provide an option to prioritize the system installed libraries? This is because using system installed libraries by default diversifies the version of dependent libraries.

Hi, I once did that in #7286, but that PR is somewhat ignored in the meantime. I kept it open for reference. Please consider that one first, if it gets merged, then I can rework this PR to do the same for some other packages.

The only difference in the other PR is that I've used FLB_PREFER_SYSTEM_LIBS iso FLB_USE_SYSTEM_LIBRARIRS.

cosmo0920

The direction of this PR is all good. But I found a concern to use system/bundled cares library case. Could you take a look?

CMakeLists.txt

ThomasDevoogdt · 2024-06-21T12:43:39Z

can you please add a simple workflow that builds Fluent Bit in the CI with all those libs in shared mode ?

I have no experience with GitHub workflows. So that would take me some time to get started with it. Also, not all packages are up-to-date in e.g. the ubuntu docker containers. I tested all of them by linking to /usr/local/lib installs. I know that the apple homebrew eco system is quite up to date, so perhaps, that can be a good start? But I would propose to not hamper this PR with this extra request.

ThomasDevoogdt · 2024-06-27T07:51:52Z

@edsiper, sorry that I don't find the time to have a look at these GitHub actions. But is there any reason to still block this PR for this? I would like to have these changes so that I can improve the support in Buildroot.

patrick-stephens · 2024-06-27T09:43:45Z

What exactly has to be invoked to build this? I can potentially update CI to do that

ThomasDevoogdt · 2024-06-27T12:35:38Z

What exactly has to be invoked to build this? I can potentially update CI to do that

@patrick-stephens A test job that compiles fluent-bit against a set of native system libraries, where we do enable the FLB_PREFER_SYSTEM_ flags.

e.g. for ubuntu 2022.04:

sudo apt-get install libc-ares-dev libluajit-5.1-dev libnghttp2-dev libjemalloc-dev librdkafka-dev

and then use

cmake -GNinja -DFLB_PREFER_SYSTEM_LIB_CARES=Yes -DFLB_PREFER_SYSTEM_LIB_LUAJIT=Yes -DFLB_PREFER_SYSTEM_LIB_NGHTTP2=Yes -DFLB_PREFER_SYSTEM_LIB_JEMALLOC=Yes -DFLB_PREFER_SYSTEM_LIB_KAFKA=Yes ../

The only problem is that Ubuntu doesn't always ship the latest libraries. A very similar test can be made, using the homebrew environment. Homebrew typically is more up-to-date, and if I'm right, homebrew does allow the installation of a fixed version.

patrick-stephens · 2024-06-27T12:54:56Z

What exactly has to be invoked to build this? I can potentially update CI to do that

@patrick-stephens A test job that compiles fluent-bit against a set of native system libraries, where we do enable the FLB_PREFER_SYSTEM_ flags.

e.g. for ubuntu 2022.04:
sudo apt-get install libc-ares-dev libluajit-5.1-dev libnghttp2-dev libjemalloc-dev librdkafka-dev
and then use
cmake -GNinja -DFLB_PREFER_SYSTEM_LIB_CARES=Yes -DFLB_PREFER_SYSTEM_LIB_LUAJIT=Yes -DFLB_PREFER_SYSTEM_LIB_NGHTTP2=Yes -DFLB_PREFER_SYSTEM_LIB_JEMALLOC=Yes -DFLB_PREFER_SYSTEM_LIB_KAFKA=Yes ../
The only problem is that Ubuntu doesn't always ship the latest libraries. A very similar test can be made, using the homebrew environment. Homebrew typically is more up-to-date, and if I'm right, homebrew does allow the installation of a fixed version.

Probably not homebrew - macOS runners cost a lot and we already have Linux based package tests in place for all targets via simple container builds.
My preference would be just updating ./packaging/build.sh to support the new option to make it easy to then invoke all in one go via the existing test workflow which then means we do it for all targets and stays up-to-date plus gives users a simple approach too using our actual release tooling.
I can probably do this after this PR is merged though, it likely just needs some extra packages adding to the container base images for builds then passing the options in.

cosmo0920 · 2024-06-27T14:27:40Z

Another option is like using ArchLinux via docker containers. ArchLinux's packaging is blazingly fast and typically no patches are applied. If we need to test against the latest packages, we'll be able to consider this option. However, that distribution is sometimes too cutting edge. So, I feel that testing with system packages on Ubuntu 22.04 is enough for most cases. ;)

cosmo0920 · 2024-07-05T07:05:14Z

can you please add a simple workflow that builds Fluent Bit in the CI with all those libs in shared mode ?

In #7286, we added simple workflow which uses system libluajit-5.1-dev and its associated libraries. So, once that PR is merged we can continue to add dependent libraries as apt packages.

ThomasDevoogdt · 2024-07-31T20:39:18Z

@edsiper @cosmo0920 kind remember to take this up again

ThomasDevoogdt · 2024-08-07T19:33:01Z

@edsiper thank you for merging #7286, this is a follow up pull request on top of the previous one. Can you also have a look at this one? I rebased the branch on top of the master branch.

e.g. buildroot has logic to build libnghttp2, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_NGHTTP2=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

e.g. buildroot has logic to build backtrace, so if find_path/find_library can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_BACKTRACE=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

e.g. buildroot has logic to build jemalloc, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_JEMALLOC=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

e.g. buildroot has logic to build c-ares, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_CARES=Yes. Note that the CheckIncludeFiles include for wasm is added, since that was done by c-ares otherwise. Signed-off-by: Thomas Devoogdt <[email protected]>

e.g. buildroot has logic to build librdkafka, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_KAFKA=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

Signed-off-by: Thomas Devoogdt <[email protected]>

ThomasDevoogdt · 2024-08-25T14:35:30Z

@edsiper @cosmo0920 @patrick-stephens @leonardo-albertovich Hi, I rebased this PR to the current master. Is it possible to consider this PR?

cosmo0920 · 2024-08-26T06:56:37Z

Hi, thanks for continuously rebasing on top of master. Currently, I added a milestone for v3.2.0 in this PR.

ThomasDevoogdt requested review from fujimotos, niedbalski, patrick-stephens, celalettin1286, edsiper, leonardo-albertovich and koleini as code owners June 8, 2024 13:57

github-actions bot added the docs-required label Jun 8, 2024

ThomasDevoogdt mentioned this pull request Jun 8, 2024

build: use the system provided LuaJIT if found #8929

Closed

2 tasks

ThomasDevoogdt changed the title ~~Feature/use system provided packages~~ build: use the system provided packages if found Jun 8, 2024

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch from 6093bda to 4c6da55 Compare June 8, 2024 14:02

ThomasDevoogdt requested a review from PettitWesley as a code owner June 8, 2024 14:02

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch 10 times, most recently from 95225eb to 169ee05 Compare June 9, 2024 14:10

cosmo0920 requested changes Jun 10, 2024

View reviewed changes

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch from 169ee05 to 95ccc2d Compare June 10, 2024 18:50

ThomasDevoogdt mentioned this pull request Jun 13, 2024

build: use the system provided LuaJIT if found #7286

Merged

2 tasks

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch from 95ccc2d to 70c5a0e Compare June 15, 2024 15:56

patrick-stephens added the ok-package-test Run PR packaging tests label Jun 17, 2024

ThomasDevoogdt requested a review from cosmo0920 June 18, 2024 11:20

cosmo0920 requested changes Jun 19, 2024

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

ThomasDevoogdt dismissed cosmo0920’s stale review via 266520e July 6, 2024 21:06

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch 3 times, most recently from 1cfe759 to beae70e Compare July 7, 2024 06:02

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch from beae70e to ef73d77 Compare August 7, 2024 17:28

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch from ef73d77 to 4bdf7a6 Compare August 16, 2024 12:03

ThomasDevoogdt added 8 commits August 25, 2024 15:21

build: use the system provided libnghttp2 if found

88bc70c

e.g. buildroot has logic to build libnghttp2, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_NGHTTP2=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

build: use the system provided backtrace if found

44e6d9f

e.g. buildroot has logic to build backtrace, so if find_path/find_library can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_BACKTRACE=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

build: use the system provided jemalloc if found

b1433eb

e.g. buildroot has logic to build jemalloc, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_JEMALLOC=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

build: use the system provided librdkafka if found

243f9c8

e.g. buildroot has logic to build librdkafka, so if pkg_check_modules can find a suitable version, then use that one if -DFLB_PREFER_SYSTEM_LIB_KAFKA=Yes. Signed-off-by: Thomas Devoogdt <[email protected]>

workflows: pr-compile-check.yaml: add libbacktrace system library test

0375a2f

Signed-off-by: Thomas Devoogdt <[email protected]>

workflows: pr-compile-check.yaml: add libnghttp2 system library test

881256c

Signed-off-by: Thomas Devoogdt <[email protected]>

workflows: pr-compile-check.yaml: add libjemalloc system library test

a24eac7

Signed-off-by: Thomas Devoogdt <[email protected]>

ThomasDevoogdt force-pushed the feature/use-system-provided-packages branch from 4bdf7a6 to a24eac7 Compare August 25, 2024 13:21

cosmo0920 added this to the Fluent Bit v3.2.0 milestone Aug 26, 2024

edsiper merged commit 231734b into fluent:master Nov 9, 2024
69 checks passed

ThomasDevoogdt mentioned this pull request Nov 9, 2024

build: use the system provided libmsgpack and libsqlite3 if found #9572

Open

ThomasDevoogdt deleted the feature/use-system-provided-packages branch November 9, 2024 19:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: use the system provided packages if found #8930

build: use the system provided packages if found #8930

ThomasDevoogdt commented Jun 8, 2024

cosmo0920 left a comment •

edited

Loading

ThomasDevoogdt commented Jun 10, 2024 •

edited

Loading

cosmo0920 left a comment

ThomasDevoogdt commented Jun 21, 2024 •

edited

Loading

ThomasDevoogdt commented Jun 27, 2024

patrick-stephens commented Jun 27, 2024

ThomasDevoogdt commented Jun 27, 2024

patrick-stephens commented Jun 27, 2024

cosmo0920 commented Jun 27, 2024

cosmo0920 commented Jul 5, 2024 •

edited

Loading

ThomasDevoogdt commented Jul 31, 2024

ThomasDevoogdt commented Aug 7, 2024 •

edited

Loading

ThomasDevoogdt commented Aug 25, 2024

cosmo0920 commented Aug 26, 2024

build: use the system provided packages if found #8930

build: use the system provided packages if found #8930

Conversation

ThomasDevoogdt commented Jun 8, 2024

cosmo0920 left a comment • edited Loading

Choose a reason for hiding this comment

ThomasDevoogdt commented Jun 10, 2024 • edited Loading

cosmo0920 left a comment

Choose a reason for hiding this comment

ThomasDevoogdt commented Jun 21, 2024 • edited Loading

ThomasDevoogdt commented Jun 27, 2024

patrick-stephens commented Jun 27, 2024

ThomasDevoogdt commented Jun 27, 2024

patrick-stephens commented Jun 27, 2024

cosmo0920 commented Jun 27, 2024

cosmo0920 commented Jul 5, 2024 • edited Loading

ThomasDevoogdt commented Jul 31, 2024

ThomasDevoogdt commented Aug 7, 2024 • edited Loading

ThomasDevoogdt commented Aug 25, 2024

cosmo0920 commented Aug 26, 2024

cosmo0920 left a comment •

edited

Loading

ThomasDevoogdt commented Jun 10, 2024 •

edited

Loading

ThomasDevoogdt commented Jun 21, 2024 •

edited

Loading

cosmo0920 commented Jul 5, 2024 •

edited

Loading

ThomasDevoogdt commented Aug 7, 2024 •

edited

Loading