Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add production caveat docs #5

Merged
merged 1 commit into from
Nov 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/_static/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,9 @@
padding-left: 0;
}
}


.admonition.warning {
background: #FFF9CC;
border-color: #E1C80B;
}
84 changes: 84 additions & 0 deletions docs/caveats.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Caveats

This package works great for development and testing, but there are some caveats to be aware of before you can take it to production.

HashiCorp expects providers to be written in Go using Terraform Plugin Framework.
This compiles the provider down into a single binary (per os/arch) that can be distributed to users.
That's the accepted way to write a provider and their tooling and documentation is built around this.

This is very different from the Python ecosystem, where we expect users to have Python installed and to install packages from PyPI.
A lot of frustrations come from this difference in expectations.
_It's an uphill battle to get a Python provider into production._

If you want to write a provider in Python for production use, you will have to overcome these hurdles.
It's doable, but **outside of scope of this package and you're on your own.**

You should very strongly consider only using this package for development, testing, and proof-of-concept work.

#### What's the big problem?

The biggest problem is that Terraform **expects the provider to be a single binary**.
You need to get your entire artifact into one executable file that you can move to someone else's machine.
This is VERY at odds with the default Python experience.

To get around this, you can use something like [Nuitka](https://nuitka.net/) to compile your Python code into a single binary with `--onefile` and `--standalone`.
This is very fickle, and you need to consider:

* How many OS/Architecture combinations do you need to support?
* How old of a version of GLIBC are you are targeting? You might want to run nuitka in docker container for an older distro.
* Do any of your dependencies have C extensions? Or do dynamic linking?
This might be a dealbreaker -- you can't install these on the fly.

With a lot of project-specific configuration, you can get this working and building a single `main.bin`. But YMMV.

#### Development mode already works, why not use that?

Without single-binaries you need to use developer mode and a hack.

In TF, developer mode is essentially an override telling TF to look for the provider binary in a specific location.

```hcl
provider_installation {
dev_overrides {
"tf.mydomain.com/mypackage" = "/path/to/your/.venv/bin"
}

direct {}
}
```

This package abuses this by symlinking the "binary" to app's script's entrypoint in the package, which is just a shell script that fires up Python and points to your entrypoint function.
When TF executes the "binary", it's actually running bash, which runs the Python interpreter, which runs your code.

There are two drawbacks with this approach for production:

##### No Lockfiles

First, the most serious drawback is that this prevents generating a lockfile.
In practice, this means you cannot use other providers at the same time as your Python provider.
You will eventually want to mix-and-match providers, and you will need to have a lockfile to do that.
No lockfile means no mixing providers.

##### Versioning Mismatch

Without lockfiles, you are going to end up with a versioning mismatch.

Normally, each TF project consumes providers and each project has its own lockfile.
TF can vary which provider version it uses based on the lockfile for each project.

Without a lockfile and while using developer mode, all TF projects on your machine will use the same installed
version of the provider. This can lead to versioning mismatches and bugs that are hard to track down.

You'll need to checkout the correct version of your provider for each project in one directory or repo before consuming it in a different folder.
This is a huge pain and an accident waiting to happen.


#### Preparing for Registry Upload

Once you have single binaries ready for each OS/Arch, you need to generate hash sums, sign them, and generate a manifest.
In theory, the official HashiCorp registry will allow you to upload these binaries and metadata using the standard Github release workflow.

However, I would strongly consider using a private registry or a different distribution method.
HashiCorp will almost certainly not officially approve and verify your provider for [the integration program](https://developer.hashicorp.com/terraform/docs/partnerships).

You will have to set that up yourself.
9 changes: 9 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
.. mdinclude:: ../README.md

.. warning::

There are serious hurdles to taking a Python provider to production.
Read :doc:`caveats` before considering a production use case.

For development, testing, and proof-of-concept, continue on to :doc:`usage`.

.. toctree::
:maxdepth: 1
:caption: Contents:
Expand All @@ -7,3 +15,4 @@
elements.md
api.rst
example.md
caveats.md
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "tf"
version = "1.0.3"
version = "1.0.4"
description = "Python Terraform Provider framework"
authors = ["Hunter Fernandes <[email protected]>"]
license = "MIT"
Expand Down