Fix Airflow-Ext Dependency

[keefertaylor]

The airflow-ext package is specified to be main, which is a branch that has changed since plugins.meltano.yml was authored. When I build today, I get constraints errors:

To Reproduce:

• Tested on Machine: Apple M2, OSX 14.2, Python v3.9.6, pip3 v23.3.1
• Run the command that Meltano will run underneath the covers:

$ pip3 install git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.7.0 psycopg2-binary --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.0/constraints-3.11.txt
<snip>

ERROR: Cannot install airflow-ext and apache-airflow==2.7.0 because these package versions have conflicting dependencies.

To fix, I've simply pinned plugins.meltano.yml to the commit on airflow-ext's main branch at the time plugins.meltano.yml was last modified.

I note that the airflow-ext project has started to version their library with tags and releases, and using one of them is probably more appropriate. I'm unfortunately unable to figure out the best way to do this since I'm not very familiar with any of these packages, or python/pip in general.

[johntdyer]

Ran into this in #16, and I tried your fix but it didnt seem to make much of a difference

▓    ~/Pr/pixbyt  on   main !1 ?1 ▓▒░ docker compose build
[+] Building 24.9s (17/22)                                                          docker:desktop-linux
 => [pixbyt internal] load build definition from Dockerfile                                         0.0s
 => => transferring dockerfile: 1.58kB                                                              0.0s
 => [pixbyt internal] load metadata for docker.io/meltano/meltano:latest                            0.3s
 => [pixbyt internal] load .dockerignore                                                            0.0s
 => => transferring context: 110B                                                                   0.0s
 => [pixbyt base  1/12] FROM docker.io/meltano/meltano:latest@sha256:1729d935e12010461f19319bd6635  0.0s
 => [pixbyt internal] load build context                                                            0.0s
 => => transferring context: 3.66kB                                                                 0.0s
 => CACHED [pixbyt base  2/12] WORKDIR /project                                                     0.0s
 => CACHED [pixbyt base  3/12] RUN apt-get update                                                   0.0s
 => CACHED [pixbyt base  4/12] RUN apt-get install -y curl                                          0.0s
 => CACHED [pixbyt base  5/12] RUN echo 0.27.2                                                      0.0s
 => CACHED [pixbyt base  6/12] RUN curl -L -o pixlet.tar.gz https://github.com/tidbyt/pixlet/relea  0.0s
 => CACHED [pixbyt base  7/12] RUN tar -xvf pixlet.tar.gz                                           0.0s
 => CACHED [pixbyt base  8/12] RUN chmod +x ./pixlet                                                0.0s
 => CACHED [pixbyt base  9/12] RUN mv pixlet /usr/local/bin/pixlet                                  0.0s
 => CACHED [pixbyt base 10/12] COPY ./meltano.yml .                                                 0.0s
 => CACHED [pixbyt base 11/12] COPY ./plugins/utilities/ ./plugins/utilities/                       0.0s
 => [pixbyt base 12/12] COPY ./plugins/plugins.meltano.yml ./plugins/plugins.meltano.yml            0.0s
 => ERROR [pixbyt prod-preinstall 1/1] RUN meltano --log-level=debug install                       24.5s
------
 > [pixbyt prod-preinstall 1/1] RUN meltano --log-level=debug install:
1.264 2024-02-06T15:22:03.013759Z [debug    ] meltano 3.3.1, Linux
1.266 2024-02-06T15:22:03.016993Z [debug    ] /etc/timezone found, contents:
1.266  Etc/UTC
1.266
1.267 2024-02-06T15:22:03.017489Z [debug    ] /etc/localtime found
1.268 2024-02-06T15:22:03.018880Z [debug    ] 2 found:
1.268  {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
1.270 2024-02-06T15:22:03.021143Z [debug    ] Creating DB engine for project at '/project' with DB URI 'sqlite://project/.meltano/meltano.db'
1.425 2024-02-06T15:22:03.175766Z [debug    ] Found plugin parent            parent=airflow plugin=airflow source=<DefinitionSource.LOCKFILE: 4>
1.425 Installing 4 plugins...
1.428 Installing extractor 'tap-pixlet'...
1.458 2024-02-06T15:22:03.208806Z [debug    ] Packages for 'extractors/tap-pixlet' have changed so performing a clean install.
1.459 2024-02-06T15:22:03.209675Z [debug    ] Removed old virtual environment for 'extractors/tap-pixlet'
1.459 2024-02-06T15:22:03.209799Z [debug    ] Creating virtual environment for 'extractors/tap-pixlet'
1.468 Installing loader 'target-tidbyt'...
1.484 2024-02-06T15:22:03.234734Z [debug    ] Variable '$TIDBYT_TOKEN' is not set in the provided env dictionary.
1.485 2024-02-06T15:22:03.235427Z [debug    ] Variable '$TIDBYT_DEVICE_ID' is not set in the provided env dictionary.
1.485 2024-02-06T15:22:03.236062Z [debug    ] Variable '$TIDBYT_TOKEN' is not set in the provided env dictionary.
1.486 2024-02-06T15:22:03.236568Z [debug    ] Variable '$TIDBYT_DEVICE_ID' is not set in the provided env dictionary.
1.488 2024-02-06T15:22:03.238575Z [debug    ] Packages for 'loaders/target-tidbyt' have changed so performing a clean install.
1.488 2024-02-06T15:22:03.238805Z [debug    ] Removed old virtual environment for 'loaders/target-tidbyt'
1.488 2024-02-06T15:22:03.238957Z [debug    ] Creating virtual environment for 'loaders/target-tidbyt'
1.509 Installing loader 'target-webp'...
1.528 2024-02-06T15:22:03.278793Z [debug    ] Packages for 'loaders/target-webp' have changed so performing a clean install.
1.529 2024-02-06T15:22:03.279771Z [debug    ] Removed old virtual environment for 'loaders/target-webp'
1.530 2024-02-06T15:22:03.280051Z [debug    ] Creating virtual environment for 'loaders/target-webp'
1.536 Installing utility 'airflow'...
1.576 2024-02-06T15:22:03.326661Z [debug    ] Packages for 'utilities/airflow' have changed so performing a clean install.
1.577 2024-02-06T15:22:03.327244Z [debug    ] Removed old virtual environment for 'utilities/airflow'
1.577 2024-02-06T15:22:03.327374Z [debug    ] Creating virtual environment for 'utilities/airflow'
2.505 2024-02-06T15:22:04.255025Z [debug    ] Upgrading pip for 'extractors/tap-pixlet'
2.505 2024-02-06T15:22:04.255550Z [debug    ] Upgrading with args '--upgrade pip' in existing virtual environment for 'extractors/tap-pixlet'
2.530 2024-02-06T15:22:04.280599Z [debug    ] Upgrading pip for 'loaders/target-webp'
2.530 2024-02-06T15:22:04.280821Z [debug    ] Upgrading with args '--upgrade pip' in existing virtual environment for 'loaders/target-webp'
2.542 2024-02-06T15:22:04.292215Z [debug    ] Upgrading pip for 'loaders/target-tidbyt'
2.542 2024-02-06T15:22:04.292454Z [debug    ] Upgrading with args '--upgrade pip' in existing virtual environment for 'loaders/target-tidbyt'
2.567 2024-02-06T15:22:04.317798Z [debug    ] Upgrading pip for 'utilities/airflow'
2.568 2024-02-06T15:22:04.318796Z [debug    ] Upgrading with args '--upgrade pip' in existing virtual environment for 'utilities/airflow'
6.820 2024-02-06T15:22:08.569507Z [debug    ] Installing with args 'git+https://github.com/DouweM/target-webp.git' into virtual environment for 'loaders/target-webp'
6.899 2024-02-06T15:22:08.648245Z [debug    ] Installing with args 'git+https://github.com/DouweM/tap-pixlet.git' into virtual environment for 'extractors/tap-pixlet'
6.939 2024-02-06T15:22:08.689701Z [debug    ] Installing with args 'git+https://github.com/meltano/airflow-ext.git@811287a6c2e7dc74498c26ef5f830e9df78dc976 apache-airflow==2.7.0 psycopg2-binary --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.4/constraints-no-providers-3.10.txt' into virtual environment for 'utilities/airflow'
7.193 2024-02-06T15:22:08.943683Z [debug    ] Installing with args 'git+https://github.com/DouweM/target-tidbyt.git' into virtual environment for 'loaders/target-tidbyt'
12.80 2024-02-06T15:22:14.548413Z [info     ] Logged pip install output to /project/.meltano/logs/pip/utilities/airflow/pip.log
12.80 Utility 'airflow' could not be installed: Failed to install plugin 'airflow'.
12.80 Collecting git+https://github.com/meltano/airflow-ext.git@811287a6c2e7dc74498c26ef5f830e9df78dc976
12.80   Cloning https://github.com/meltano/airflow-ext.git (to revision 811287a6c2e7dc74498c26ef5f830e9df78dc976) to /tmp/pip-req-build-fl0zd8lh
12.80   Running command git clone --filter=blob:none --quiet https://github.com/meltano/airflow-ext.git /tmp/pip-req-build-fl0zd8lh
12.80   Running command git rev-parse -q --verify 'sha^811287a6c2e7dc74498c26ef5f830e9df78dc976'
12.80   Running command git fetch -q https://github.com/meltano/airflow-ext.git 811287a6c2e7dc74498c26ef5f830e9df78dc976
12.80   Running command git checkout -q 811287a6c2e7dc74498c26ef5f830e9df78dc976
12.80   Resolved https://github.com/meltano/airflow-ext.git to commit 811287a6c2e7dc74498c26ef5f830e9df78dc976
12.80   Installing build dependencies: started
12.80   Installing build dependencies: finished with status 'done'
12.80   Getting requirements to build wheel: started
12.80   Getting requirements to build wheel: finished with status 'done'
12.80   Preparing metadata (pyproject.toml): started
12.80   Preparing metadata (pyproject.toml): finished with status 'done'
12.80 ERROR: Cannot install apache-airflow==2.7.0 because these package versions have conflicting dependencies.
12.80
12.80 The conflict is caused by:
12.80     The user requested apache-airflow==2.7.0
12.80     The user requested (constraint) apache-airflow==2.3.4
12.80
12.80 To fix this you could try to:
12.80 1. loosen the range of package versions you've specified
12.80 2. remove package versions to allow pip attempt to solve the dependency conflict
12.80
12.80 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
12.80
22.62 Installed loader 'target-webp'
22.86 Installed extractor 'tap-pixlet'
23.32 Installed loader 'target-tidbyt'
23.32 Installed 3/4 plugins
24.00 2024-02-06T15:22:25.075773Z [debug    ] Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
24.00 join our friendly Slack community.
24.00
24.00 Failed to install plugin(s)
24.00 ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/__init__.py:102 in _run_cli    │
24.00 │                                                                              │
24.00 │    99 │   """                                                                │
24.00 │   100 │   try:                                                               │
24.00 │   101 │   │   try:  # noqa: WPS225, WPS505                                   │
24.00 │ ❱ 102 │   │   │   cli(obj={"project": None})                                 │
24.00 │   103 │   │   except ProjectReadonly as err:                                 │
24.00 │   104 │   │   │   raise CliError(                                            │
24.00 │   105 │   │   │   │   f"The requested action could not be completed: {err}", │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/click/core.py:1157 in __call__             │
24.00 │                                                                              │
24.00 │   1154 │                                                                     │
24.00 │   1155 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:       │
24.00 │   1156 │   │   """Alias for :meth:`main`."""                                 │
24.00 │ ❱ 1157 │   │   return self.main(*args, **kwargs)                             │
24.00 │   1158                                                                       │
24.00 │   1159                                                                       │
24.00 │   1160 class Command(BaseCommand):                                           │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/cli.py:46 in main              │
24.00 │                                                                              │
24.00 │    43 │   │   │   args: Positional arguments for the Click group.            │
24.00 │    44 │   │   │   kwargs: Keyword arguments for the Click group.             │
24.00 │    45 │   │   """                                                            │
24.00 │ ❱  46 │   │   return super().main(*args, windows_expand_args=False, **kwargs │
24.00 │    47                                                                        │
24.00 │    48                                                                        │
24.00 │    49 @click.group(                                                          │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/click/core.py:1078 in main                 │
24.00 │                                                                              │
24.00 │   1075 │   │   try:                                                          │
24.00 │   1076 │   │   │   try:                                                      │
24.00 │   1077 │   │   │   │   with self.make_context(prog_name, args, **extra) as c │
24.00 │ ❱ 1078 │   │   │   │   │   rv = self.invoke(ctx)                             │
24.00 │   1079 │   │   │   │   │   if not standalone_mode:                           │
24.00 │   1080 │   │   │   │   │   │   return rv                                     │
24.00 │   1081 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!           │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/utils.py:641 in invoke         │
24.00 │                                                                              │
24.00 │   638 │   │   enact_environment_behavior(self.environment_behavior, ctx)     │
24.00 │   639 │   │   if ctx.obj.get("tracker"):                                     │
24.00 │   640 │   │   │   ctx.obj["tracker"].add_contexts(CliContext.from_click_cont │
24.00 │ ❱ 641 │   │   super().invoke(ctx)                                            │
24.00 │   642                                                                        │
24.00 │   643                                                                        │
24.00 │   644 class InstrumentedDefaultGroup(InstrumentedGroupMixin, DefaultGroup, D │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/click/core.py:1688 in invoke               │
24.00 │                                                                              │
24.00 │   1685 │   │   │   │   super().invoke(ctx)                                   │
24.00 │   1686 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx │
24.00 │   1687 │   │   │   │   with sub_ctx:                                         │
24.00 │ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub │
24.00 │   1689 │   │                                                                 │
24.00 │   1690 │   │   # In chain mode we create the contexts step by step, but afte │
24.00 │   1691 │   │   # base command has been invoked.  Because at that point we do │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/utils.py:690 in invoke         │
24.00 │                                                                              │
24.00 │   687 │   │   if ctx.obj.get("tracker"):                                     │
24.00 │   688 │   │   │   ctx.obj["tracker"].add_contexts(CliContext.from_click_cont │
24.00 │   689 │   │   │   ctx.obj["tracker"].track_command_event(CliEvent.started)   │
24.00 │ ❱ 690 │   │   super().invoke(ctx)                                            │
24.00 │   691                                                                        │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/click/core.py:1434 in invoke               │
24.00 │                                                                              │
24.00 │   1431 │   │   │   echo(style(message, fg="red"), err=True)                  │
24.00 │   1432 │   │                                                                 │
24.00 │   1433 │   │   if self.callback is not None:                                 │
24.00 │ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx.params)            │
24.00 │   1435 │                                                                     │
24.00 │   1436 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List │
24.00 │   1437 │   │   """Return a list of completions for the incomplete value. Loo │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/click/core.py:783 in invoke                │
24.00 │                                                                              │
24.00 │    780 │   │                                                                 │
24.00 │    781 │   │   with augment_usage_errors(__self):                            │
24.00 │    782 │   │   │   with ctx:                                                 │
24.00 │ ❱  783 │   │   │   │   return __callback(*args, **kwargs)                    │
24.00 │    784 │                                                                     │
24.00 │    785 │   def forward(                                                      │
24.00 │    786 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # no │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/click/decorators.py:33 in new_func         │
24.00 │                                                                              │
24.00 │    30 │   """                                                                │
24.00 │    31 │                                                                      │
24.00 │    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":        │
24.00 │ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)               │
24.00 │    34 │                                                                      │
24.00 │    35 │   return update_wrapper(new_func, f)                                 │
24.00 │    36                                                                        │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/params.py:27 in decorate       │
24.00 │                                                                              │
24.00 │   24 │   │   if database_uri:                                                │
24.00 │   25 │   │   │   ProjectSettingsService.config_override["database_uri"] = da │
24.00 │   26 │   │                                                                   │
24.00 │ ❱ 27 │   │   return func(*args, **kwargs)                                    │
24.00 │   28 │                                                                       │
24.00 │   29 │   return functools.update_wrapper(decorate, func)                     │
24.00 │   30                                                                         │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/params.py:75 in decorate       │
24.00 │                                                                              │
24.00 │   72 │   │   │   │   except MigrationError as err:                           │
24.00 │   73 │   │   │   │   │   raise CliError(str(err)) from err                   │
24.00 │   74 │   │   │                                                               │
24.00 │ ❱ 75 │   │   │   func(project, *args, **kwargs)                              │
24.00 │   76 │   │                                                                   │
24.00 │   77 │   │   return functools.update_wrapper(decorate, func)                 │
24.00 │   78                                                                         │
24.00 │                                                                              │
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/install.py:110 in install      │
24.00 │                                                                              │
24.00 │   107 │   )                                                                  │
24.00 │   108 │   if not success:                                                    │
24.00 │   109 │   │   tracker.track_command_event(CliEvent.failed)                   │
24.00 │ ❱ 110 │   │   raise CliError("Failed to install plugin(s)")                  │
24.00 │   111 │   tracker.track_command_event(CliEvent.completed)                    │
24.00 │   112                                                                        │
24.00 │   113                                                                        │
24.00 ╰──────────────────────────────────────────────────────────────────────────────╯
24.00 CliError: Failed to install plugin(s)
24.00
24.00 The above exception was the direct cause of the following exception:
24.00
24.00 ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
24.00 │ /venv/lib/python3.9/site-packages/meltano/cli/__init__.py:112 in _run_cli    │
24.00 │                                                                              │
24.00 │   109 │   │   except MeltanoError as err:                                    │
24.00 │   110 │   │   │   handle_meltano_error(err)                                  │
24.00 │   111 │   │   except Exception as err:                                       │
24.00 │ ❱ 112 │   │   │   raise CliError(f"{troubleshooting_message}\n{err}") from e │
24.00 │   113 │   except CliError as cli_error:                                      │
24.00 │   114 │   │   cli_error.print()                                              │
24.00 │   115 │   │   sys.exit(1)                                                    │
24.00 ╰──────────────────────────────────────────────────────────────────────────────╯
24.00 CliError: Need help fixing this problem? Visit http://melta.no/ for
24.00 troubleshooting steps, or to
24.00 join our friendly Slack community.
24.00
24.00 Failed to install plugin(s)
24.00
24.00 Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
24.00 join our friendly Slack community.
24.00
24.00 Failed to install plugin(s)
------
failed to solve: process "/bin/sh -c meltano --log-level=debug install" did not complete successfully: exit code: 1
░▒▓    ~/Pr/pixbyt  on   main !1 ?1 ▓▒░

[DouweM]

Thanks @keefertaylor! I can't test this right now, but if you (or someone else, cc @johntdyer) can get this to work with a specific tag on the git+https://github.com/meltano/airflow-ext repo I'll gladly merge it!

[bboatwright]

@DouweM @keefertaylor @johntdyer confirmed just now that the fix works, at least when building locally, as does any commit of airflow-ext from ~mid-2023. FWIW, I was able to install Docker Compose and clone the Pixbyt repo on an AWS Cloud9 instance and run everything completely from the cloud.

[DouweM]

@bboatwright Looks like we were both looking into this at the same time! I just published #18 which fixes this issue by using the latest version of Airflow!