v6 fails due to collection path not aligning with ansible collection path

[lod]

Issue Type

• Bug report

I'm not entirely sure that this is a molecule bug, it may be a ansible-compat bug, I'm not familiar enough with the ecosystem to be certain.

However the failures started with molecule v6.0, so it should be documented here until it is fixed even if the fix is elsewhere.

Molecule and Ansible details

ansible --version && molecule --version
ansible [core 2.15.3]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
molecule 6.0.0 using python 3.11
    ansible:2.15.3
    azure:23.4.1 from molecule_plugins
    containers:23.4.1 from molecule_plugins requiring collections: ansible.posix>=1.3.0 community.docker>=1.9.1 containers.podman>=1.8.1
    default:6.0.0 from molecule
    docker:23.4.1 from molecule_plugins requiring collections: community.docker>=3.0.2 ansible.posix>=1.4.0
    ec2:23.4.1 from molecule_plugins
    gce:23.4.1 from molecule_plugins requiring collections: google.cloud>=1.0.2 community.crypto>=1.8.0
    podman:23.4.1 from molecule_plugins requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0
    vagrant:23.4.1 from molecule_plugins

Test environment is an official Debian bookworm-slim container, the following steps were run to create the environment

apt-get update
apt-get install --assume-yes --no-install-recommends ca-certificates python3 python3-pip python3-wheel python3-setuptools libxml2-utils locales git
python3 -m pip --no-cache-dir install --break-system-packages setuptools_rust
python3 -m pip --no-cache-dir install --break-system-packages ansible jmespath github3.py boto3 ansible-lint yamllint flake8 "molecule>5.0.0" "molecule-plugins[podman]<23.5.0"

Actual Behaviour

On running molecule test I get an exception, the tail of which is

  File "/usr/local/lib/python3.11/dist-packages/molecule/dependency/base.py", line 91, in execute
    self._config.runtime.require_collection(name, version)
  File "/usr/local/lib/python3.11/dist-packages/ansible_compat/runtime.py", line 721, in require_collection
    self.require_collection(name=name, version=version, install=False)
  File "/usr/local/lib/python3.11/dist-packages/ansible_compat/runtime.py", line 725, in require_collection
    raise InvalidPrerequisiteError(msg)
ansible_compat.errors.InvalidPrerequisiteError: Collection 'containers.podman' not found in '['/root/.ansible/collections', '/usr/share/ansible/collections']'

Details

The collection files are all installed in /usr/local/lib/python3.11/dist-packages/ansible_collections

Ansible galaxy uses the following search paths in execute_list_collection, cli/galaxy.py:1619

        collections_search_paths = (
            set(context.CLIARGS['collections_path'] or []) | default_collections_path | set(AnsibleCollectionConfig.collection_paths)
        )

The third element, AnsibleCollectionConfig.collection_paths, is set to ['/root/.ansible/collections', '/usr/local/lib/python3.11/dist-packages']

It isn't clear to me how collection_paths gets the dist-packages element, but it is there.
https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths says that the default doesn't include that path. There is no /etc/ansible/ or ~/.ansible.cfg

molecule uses ansible-compat, ansible-compat follows the documentation not the actual ansible behaviour and only searches ['/root/.ansible/collections', '/usr/share/ansible/collections']. Which is why it can't find any of my collections.

ansible-galaxy collection list shows all the collections

ansible-galaxy collection install containers.podman community.docker ansible.posix fails with "
Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using --force."

ansible -c local -i 127.0.0.1, -m containers.podman.podman_image_info all finds and executes the module (which fails, but it finds and tries it)

Workaround

ansible-galaxy collection install --collections-path /usr/share/ansible/collections containers.podman community.docker ansible.posix --force

This installs the specified collections in /usr/share/ansible/collections allowing molecule to run.

(I also tried symlinking but ansible-compat doesn't seem to follow the link)

[isuftin]

I don't know if my issue is similar, but since Molecule 6.0.0, Molecule has been installing roles and collections into the roles and collections top level dirs in my roles and playbooks. It used to install it into ~/.cache/molecule/....

But then the other issue is that when Molecule actually runs the prepare stage, Ansible can't find my role because Ansible is running from the scenario's directory. I also find that adding ansible.cfg in the root of my project doesn't seem to help. This only seems to work when I set ANSIBLE_ROLES_PATH: ../../roles under my scenario's provisioner parameter.

provisioner:
  env:
    ANSIBLE_ROLES_PATH: ../../roles

But this also forces me to:

• Do this in every scenario for every role or playbook I write
• Do some funky stuff with .gitignore since I don't want these external roles and collections making their way into my project

So what's the proper solution here and am I doing this wrong?

[zhan9san]

Thanks for reporting this issue.

I'll take a look.

[isuftin]

Guessing this is also related to #4017

[geerlingguy]

Same issue here, can reproduce locally if I update everything to the latest releases on my Mac:

    raise InvalidPrerequisiteError(msg)
ansible_compat.errors.InvalidPrerequisiteError: Collection 'community.docker' not found in '['/Users/jgeerling/.ansible/collections', '/usr/share/ansible/collections']'

[hulr]

I don't know if my issue is similar, but since Molecule 6.0.0, Molecule has been installing roles and collections into the roles and collections top level dirs in my roles and playbooks. It used to install it into ~/.cache/molecule/....

But then the other issue is that when Molecule actually runs the prepare stage, Ansible can't find my role because Ansible is running from the scenario's directory.

I experience exactly the same issue with Molecule 6.0.1. In the prepare stage, the roles cannot be found because they are installed in the top-level roles directory.

[isuftin]

Note that Updating to latest ansible-compat 4.1.7 did not fix this issue

[zhan9san]

@lod and @geerlingguy

Could you help verify whether the issue is fixed in ansible-compat 4.1.7?

@isuftin and @hulr

I believe the issue you describe is different. We are still working on it.

[hulr]

What I noticed today (with ansible-compat 4.1.7) is that when the folder <role-name>/roles exists, Molecule sets ANSIBLE_ROLES_PATH:

$ molecule create
INFO     default scenario test matrix: dependency, create, prepare
INFO     Performing prerun with role_name_check=2...
INFO     Set ANSIBLE_ROLES_PATH=roles:/home/user/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Using /home/user/.ansible/roles/<role name> symlink to current repository in order to enable Ansible to find the role using its expected full name.

I have to override it in the molecule.yml file, that roles installed from <role-name>/molecule/default/requirements.yml will be found:

provisioner:
  env:
    ANSIBLE_ROLES_PATH: ../../roles

If the top-level <role-name>/roles folder doesn't exist, Molecule doesn't set ANSIBLE_ROLES_PATH and it works fine without overriding it in the molecule.yml file:

$ molecule create
INFO     default scenario test matrix: dependency, create, prepare
INFO     Performing prerun with role_name_check=2...
INFO     Using /home/user/.ansible/roles/<role name> symlink to current repository in order to enable Ansible to find the role using its expected full name.

[33Fraise33]

I seem to have the same issue as @isuftin:

INFO     apple scenario test matrix: dependency, create, prepare, converge
INFO     Performing prerun with role_name_check=0...
INFO     Using /Users/gianni-cm/.config/ansible/roles/citymesh.openvpn symlink to current repository in order to enable Ansible to find the role using its expected full name.
INFO     Running apple > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running apple > create
INFO     Running apple > prepare
WARNING  Skipping, prepare playbook not configured.
INFO     Running apple > converge
ERROR! the role 'citymesh.openvpn' was not found in /Users/gianni-cm/git/Ansible/openvpn/molecule/apple/roles:/Users/gianni-cm/.cache/molecule/openvpn/apple/roles:/Users/gianni-cm/git/Ansible:/Users/gianni-cm/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/gianni-cm/git/Ansible/openvpn/molecule/apple

The error appears to be in '/Users/gianni-cm/git/Ansible/openvpn/molecule/apple/converge.yml': line 5, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  roles:
    - citymesh.openvpn
      ^ here

This was working before v6.
The weird thing is the symlink exists under ~/.config/ansible/roles but not under ~/.ansible/roles

[johrstrom]

Could you help verify whether the issue is fixed in ansible-compat 4.1.7?

Upgrading from ansible-compat 3.0.2 to 4.1.8 seems to have fixed the original issue around collections for me in github's action CI (I couldn't replicate locally).

I'm using molecule 6.0.1 and can provide links to the action if anyone needs.

[lod]

Thanks @zhan9san working with ansible_compat-4.1.9

[isuftin]

I think things are working as well on my end. Just updated to ansible-compat 4.1.0..

ansible==8.3.0
ansible-compat==4.1.10
ansible-core==2.15.3
molecule==6.0.2
molecule-plugins==23.5.0

[isuftin]

Spoke too soon. It seems we're getting tripped up in the prepare stage.

We're running this on a GitLab runner...

Here's the interesting bits..

Running: molecule --verbose test --scenario-name aws-marketplace-rhel9-ec2
INFO     Found config file /builds/ctek/automation/ansible/roles/docker/.config/molecule/config.yml
DEBUG    Validating schema /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/molecule.yml.
WARNING  Driver ec2 does not provide a schema.
INFO     aws-marketplace-rhel9-ec2 scenario test matrix: dependency, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO     Performing prerun with role_name_check=0...
INFO     Running ansible-galaxy role install -vr roles/requirements.yml
INFO     Running ansible-galaxy collection install -v -r collections/requirements.yml
INFO     Set ANSIBLE_ROLES_PATH=roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Using /root/.ansible/roles/wma.docker symlink to current repository in order to enable Ansible to find the role using its expected full name.
DEBUG    Inventory /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/../common/group_vars linked to /root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/inventory/group_vars
DEBUG    Inventory /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/../common/ec2/host_vars linked to /root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/inventory/host_vars
INFO     Running aws-marketplace-rhel9-ec2 > dependency
Molecule aws-marketplace-rhel9-ec2 > dependency
00:03
Starting galaxy role install process
- extracting wma.ssl_filtering to /builds/ctek/automation/ansible/roles/docker/roles/wma.ssl_filtering
- wma.ssl_filtering (1.7.0) was installed successfully
- adding dependency: geerlingguy.java (2.3.1)
- downloading role 'java', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-java/archive/2.3.1.tar.gz
- extracting geerlingguy.java to /builds/ctek/automation/ansible/roles/docker/roles/geerlingguy.java
- geerlingguy.java (2.3.1) was installed successfully
INFO     Dependency completed successfully.
Skipping install, no requirements found
INFO     Dependency completed successfully.


...


INFO     Running aws-marketplace-rhel9-ec2 > prepare
Molecule aws-marketplace-rhel9-ec2 > prepare
00:07
Using /root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/ansible.cfg as config file
PLAY [Prepare RHEL 9 system] ***************************************************
TASK [Gathering Facts ] ********************************************************
Wednesday 06 September 2023  17:40:02 +0000 (0:00:00.013)       0:00:00.013 ***
ok: [aws-marketplace-rhel9-ec2--8]
TASK [Add root certificates to cert store name=wma.ssl_filtering] **************
Wednesday 06 September 2023  17:40:08 +0000 (0:00:05.134)       0:00:05.147 ***
ERROR! the role 'wma.ssl_filtering' was not found in /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/roles:/root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/roles:/builds/ctek/automation/ansible/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2
The error appears to be in '/builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/prepare.yml': line 15, column 15, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
      include_role:
        name: wma.ssl_filtering
              ^ here

So while it installed my wma.ssl_filtering role to /builds/ctek/automation/ansible/roles/docker/roles (which also is not great because I don't want third party deps in my project), /builds/ctek/automation/ansible/roles/docker/roles does not seem to be on the path that the prepare step looks at.

Also note my config.yml for molecule includes:

dependency:
  name: galaxy
  options:
    role-file: molecule/common/role_requirements.yml
    requirements-file: molecule/common/collections_requirements.yml
    force: false

The role that can't be found is specified @ molecule/common/role_requirements.yml but NOT in meta/requirements.yml - Thats because it's a test-time only role. And it gets installed into my role's roles/ directory as a sibling to requirements.yml. However geerlingguy.java does not get installed there. That role is specified in meta/requirements.yml. And for scenarios without a prepare step, my role works fine because geerlingguy.java is found no problem.

Finally, the only other difference I can think of is that we install geerlingguy.java from ansible galaxy but we pull wma.ssl_filtering from our private gitlab instance.

[isuftin]

Has there been any movement on the roles portion of this? This still seems to be an issue in the latest Molecule install. Still working as expected in molecule<6.0.0

[33Fraise33]

I still have the same issue indeed:

molecule 24.9.0 using python 3.13
    ansible:2.17.5
    default:24.9.0 from molecule
    digitalocean:0.1.dev44+gd71a4f2 from molecule_digitalocean
    vagrant:2.0.0 from molecule_vagrant

running molecule<6 still works

running from ~/git/ansible/hyperglass

$ molecule converge -s apple
ERROR! the role 'citymesh.hyperglass' was not found in /Users/gianni.stubbe/git/ansible/hyperglass/molecule/apple/roles:/Users/gianni.stubbe/.cache/molecule/hyperglass/apple/roles:/Users/gianni.stubbe/git/ansible:/Users/gianni.stubbe/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/gianni.stubbe/git/ansible/hyperglass/molecule/apple

The error appears to be in '/Users/gianni.stubbe/git/ansible/hyperglass/molecule/apple/converge.yml': line 10, column 15, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

      ansible.builtin.include_role:
        name: citymesh.hyperglass
              ^ here
fatal: [debian12-hyperglass]: FAILED! => {"changed": false, "reason": "the role 'citymesh.hyperglass' was not found in /Users/gianni.stubbe/git/ansible/hyperglass/molecule/apple/roles:/Users/gianni.stubbe/.cache/molecule/hyperglass/apple/roles:/Users/gianni.stubbe/git/ansible:/Users/gianni.stubbe/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/gianni.stubbe/git/ansible/hyperglass/molecule/apple\n\nThe error appears to be in '/Users/gianni.stubbe/git/ansible/hyperglass/molecule/apple/converge.yml': line 10, column 15, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n      ansible.builtin.include_role:\n        name: citymesh.hyperglass\n              ^ here\n"}

[33Fraise33]

I think I found my issue:
my environment contains the following variables:

ANSIBLE_HOME=/Users/gianni.stubbe/.config/ansible
ANSIBLE_CONFIG=/Users/gianni.stubbe/.config/ansible.cfg
ANSIBLE_GALAXY_CACHE_DIR=/Users/gianni.stubbe/.cache/ansible/galaxy_cache

This is following the XDG Base directory spec and ansible is listed here:
https://wiki.archlinux.org/title/XDG_Base_Directory#Partial

if I add the directory to my env, this fixes it: (notice the $ANSIBLE_HOME at the end)

provisioner:
  name: ansible
  env:
    ANSIBLE_ROLES_PATH: $MOLECULE_EPHEMERAL_DIRECTORY/roles:${HOME}/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:$ANSIBLE_HOME/roles