From c3c7be904c7b98dc995cfc0cd99d4ca7d67599ee Mon Sep 17 00:00:00 2001 From: Jakub Dardzinski Date: Fri, 27 Sep 2024 14:55:31 +0200 Subject: [PATCH] Add blog post about configuration with dynamic env vars. (#3117) Signed-off-by: Jakub Dardzinski --- website/blog/authors.yml | 6 + website/blog/dynamic-env-variables/index.mdx | 138 +++++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100644 website/blog/dynamic-env-variables/index.mdx diff --git a/website/blog/authors.yml b/website/blog/authors.yml index b93e1592ce..c234ccb1df 100644 --- a/website/blog/authors.yml +++ b/website/blog/authors.yml @@ -8,6 +8,12 @@ Collado: url: https://www.github.com/collado-mike image_url: https://www.github.com/collado-mike.png +Dardzinski: + name: Jakub Dardziński + title: OpenLineage Committer + url: https://github.com/JDarDagran + image_url: https://www.github.com/JDarDagran.png + Johnson: name: Will Johnson title: Guest Blogger and OpenLineage Committer diff --git a/website/blog/dynamic-env-variables/index.mdx b/website/blog/dynamic-env-variables/index.mdx new file mode 100644 index 0000000000..68b855f7bf --- /dev/null +++ b/website/blog/dynamic-env-variables/index.mdx @@ -0,0 +1,138 @@ +--- +title: Simplify OpenLineage Configuration with Dynamic Environment Variables +date: 2024-09-25 +authors: [Dardzinski] +description: Simplify your OpenLineage setup with dynamic environment variables. This innovative approach offers flexibility, centralized management, and seamless integration. Easily configure the client without modifying code or configuration files. Ideal for development, testing, and production environments. +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +The OpenLineage community has frequently highlighted the need for a more flexible and scalable approach to configuration management, particularly through environment variables. +Many users reported challenges in maintaining separate configuration files across development, testing, and production environments. +In response, the 1.23.0 release introduces dynamic environment variables—addressing these concerns head-on by simplifying the configuration process, +reducing code changes, and improving security across different environments. + + +In this guide, we'll explore how to leverage dynamic environment variables to simplify OpenLineage configuration, enhance flexibility, and improve security. + + + +# What's New in 1.23.0 + +In the upcoming **1.23.0 release** of OpenLineage, we're excited to introduce dynamic environment variables, which will revolutionize the way you configure your OpenLineage client. With this feature, you'll be able to simplify your configuration, improve maintainability, and adapt to changing environments with ease. + +# Why Dynamic Environment Variables? + +### Flexibility on a Whole New Level +Stop worrying about reconfiguring code or files—dynamic environment variables let you adjust settings on the fly. This feature is perfect for developers, testers, and production environments that demand real-time adaptability. + +### Centralized Management at Your Fingertips +With environment variables, you can securely store configuration details alongside other essential application settings. No more scattering information across multiple files—it’s all in one place. + +### Cleaner Code, Happier Developers +Say goodbye to the clutter of parsing configuration files in your codebase. Dynamic environment variables lead to cleaner, more maintainable applications, saving you time and headaches. + +### Seamless Integration for Effortless Deployment +Dynamic environment variables easily sync with modern deployment tools and practices, ensuring smooth and stress-free configuration management every time. + +## When to Use Dynamic Environment Variables + +### Development & Testing +Need to experiment with new configurations quickly? Dynamic environment variables let you do just that—without ever touching your code. + +### Production Deployments +Keep your production environment secure and organized by managing all configuration settings with ease and efficiency. + +### Dynamic Environments +No more manual updates! Easily adapt to rapidly changing environments with this flexible configuration approach. + +## Key Features You’ll Love +* **Dynamic Updates**: Modify configurations in real-time without restarting your application. +* **Hierarchical Organization**: Keep your settings clear and manageable with nested sections. +* **Automatic Conversions**: OpenLineage automatically converts variable names to camelCase in Java, ensuring consistency. +* **JSON Support**: Got complex configurations? Embed them directly in environment variables with JSON. +* **Effortless Integration**: Tie your configuration process into deployment tools and scripting languages for smooth automation. + + +### Quick Setup Example: HTTP Transport Configuration + +Let’s see dynamic environment variables in action! Suppose you need to configure your client to send events to an HTTP endpoint with basic authentication. Here’s how simple it is using environment variables: + + + + +```sh +OPENLINEAGE__TRANSPORT__TYPE=http +OPENLINEAGE__TRANSPORT__URL=http://localhost:5050 +OPENLINEAGE__TRANSPORT__ENDPOINT=/api/v1/lineage +OPENLINEAGE__TRANSPORT__AUTH__TYPE=api_key +OPENLINEAGE__TRANSPORT__AUTH__API_KEY=your_api_key +OPENLINEAGE__TRANSPORT__COMPRESSION=gzip +``` + + + + +```sh +OPENLINEAGE__TRANSPORT__TYPE=http +OPENLINEAGE__TRANSPORT__URL=http://localhost:5050 +OPENLINEAGE__TRANSPORT__ENDPOINT=/api/v1/lineage +OPENLINEAGE__TRANSPORT__AUTH='{"type":"api_key", "apiKey":"your_api_key"}' +OPENLINEAGE__TRANSPORT__COMPRESSION=gzip +``` + + + + +This translates to the equivalent YAML configuration: + +```yaml +transport: + type: http + url: http://localhost:5050 + endpoint: /api/v1/lineage + auth: + type: api_key + apiKey: your_api_key + compression: gzip +``` + + +### Configuring CompositeTransport with Aliases for Ultimate Flexibility + +In Python, configuring CompositeTransport is a breeze with aliases for `OPENLINEAGE_URL`, `OPENLINEAGE_API_KEY`, and `OPENLINEAGE_ENDPOINT`. +It allows you to create a default configuration for the default HTTP transport and easily add extra targets. + +Here’s an example if you’ve already set the following environment variables: +```sh +OPENLINEAGE_URL=http://localhost:5050 +OPENLINEAGE_API_KEY=your_api_key +OPENLINEAGE_ENDPOINT=/api/v1/lineage +``` + +Now, configure the CompositeTransport: +```sh +OPENLINEAGE__TRANSPORT__TYPE=composite +OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__TYPE=http +OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__URL=http://another_host:5050 +OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__AUTH='{"type":"api_key", "apiKey":"random_token"}' +OPENLINEAGE__TRANSPORT__TRANSPORTS__ANOTHER_TARGET__COMPRESSION=gzip +``` + +You’ve just configured a CompositeTransport with two targets: `default_http` and `another_target`. The best part? You didn’t have to touch your config files! + +### Troubleshooting + +Got issues? Here’s how to troubleshoot common problems with OpenLineage configuration: + +* **Check Variable Names**: Ensure the environment variable names match what OpenLineage expects. +* **Validate Values**: Double-check that you’re assigning the correct values. +* **Client Permissions**: Ensure your OpenLineage client has access to read the environment variables, especially in containers like Docker. +* **Environment Variable Verification**: Confirm all necessary environment variables are set in the correct environment. + +## Wrap up: Embrace Dynamic Configuration + +Dynamic environment variables offer a powerful and flexible approach to configuring the OpenLineage client. By leveraging this feature, you can streamline your setup, improve maintainability, and adapt configurations seamlessly within your dynamic environments. Say goodbye to complex configuration files and hello to a more efficient and scalable OpenLineage setup! + +**Ready to learn more?** Check out the [OpenLineage documentation](https://openlineage.io/docs) for more information on configuring the CompositeTransport and other advanced features.