[BUG] SetUpTools Auto Discovery - Won’t Disable

[ehkristen]

With setuptools <61.0.0, installation works fine and the imports described above work fine.

This is probably related to the specific method that you were using for the installation (i.e. editable install). Editables installs are "complicated"™️:. Inherently they have to deal with a compromise "emulating the behaviour of a package installed normally" vs "allowing users to do changes like adding new files"1, so not necessarily what you observe in an editable install is what the package configuration you have written is actually doing.

For example, if you try to build the package using Setuptools<60 "for real" (not in editable mode, see snippet below), you can see that the package configuration you are using results in an empty package with no .py file. This happens because in your configuration you do not specify packages=[...]2; and before v61.0.0 setuptools would not attempt to derive that configuration option automatically.

> docker run --rm -it python:3.11-bullseye /bin/bash

git clone https://github.com/radian-software/straight.el/
cd straight.el
git reset --hard 9b11112b2e7aedd994feb2d8f95bd66dbc5749a5
cd watcher

python -m venv .venv
.venv/bin/python -m pip install 'setuptools==60.0.0' wheel build
.venv/bin/python -m build --no-isolation
unzip -l dist/*.whl
# Archive:  dist/straight_watcher-1.0.dev0-py3-none-any.whl
#   Length      Date    Time    Name
# ---------  ---------- -----   ----
#       300  2023-08-14 18:53   straight_watcher-1.0.dev0.dist-info/METADATA
#        92  2023-08-14 18:53   straight_watcher-1.0.dev0.dist-info/WHEEL
#         1  2023-08-14 18:53   straight_watcher-1.0.dev0.dist-info/top_level.txt
#       344  2023-08-14 18:53   straight_watcher-1.0.dev0.dist-info/RECORD
# ---------                     -------
#       737                     4 files

Newer versions of setuptools will halt the build because this kind of wheel looks off (it is missing packages/modules), and it prefers to prompt the user to be explicit about it instead of creating a potentially problematic package.

However, as of yet I have been unable to find any combination of setup.py and pyproject.toml options that allows the package to be installed and both modules to be imported. Either installation fails with a different error, or the package is installed with the modules not being importable.

The error message says:

If you are trying to create a single distribution with multiple modules
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

You said option 2 is not possible for you, but have you tried options 1 or 3? What are the problems with those options?

For example, I gave it a try for option 3 (re-using the same project directory from the snippet above):

.venv/bin/python -m pip install 'setuptools==68'
sed -i 's/)/    py_modules=["straight_watch", "straight_watch_callback"])/' setup.py  # add py_modules configuration.
.venv/bin/python -m build --no-isolation
# ...
# Successfully built straight-watcher-1.0.dev0.tar.gz and straight_watcher-1.0.dev0-py3-none-any.whl
unzip -l dist/*.whl
#   Length      Date    Time    Name
# ---------  ---------- -----   ----
#      2611  2023-08-14 18:53   straight_watch.py
#      2031  2023-08-14 18:53   straight_watch_callback.py
#       273  2023-08-14 19:09   straight_watcher-1.0.dev0.dist-info/METADATA
#        92  2023-08-14 19:09   straight_watcher-1.0.dev0.dist-info/WHEEL
#        39  2023-08-14 19:09   straight_watcher-1.0.dev0.dist-info/top_level.txt
#       502  2023-08-14 19:09   straight_watcher-1.0.dev0.dist-info/RECORD
# ---------                     -------
#      5548                     6 files
.venv/bin/python -m pip install -e .
# ...
# Successfully installed psutil-5.6.6 straight-watcher-1.0.dev0
.venv/bin/python -c 'import straight_watch_callback; print(straight_watch_callback)'
# <module 'straight_watch_callback' from '/straight.el/watcher/straight_watch_callback.py'>
.venv/bin/python -c 'import straight_watch; print(straight_watch)'
# <module 'straight_watch' from '/straight.el/watcher/straight_watch.py'>

Or did I simply not find the right combination of configuration to make this work? (I tried different options for numerous hours and even looked through the setuptools source code.)

Not sure if I am understanding which would be the combinations that you are talking about. The error message lists 3 possible solutions. All of them should work for the reproducer you have presented. Could you please clarify and/or provide a minimal reproducer similar to the snippet above?

Footnotes

1. You can have an idea on how complicated this compromise is by reading the many threads in https://discuss.python.org/search?q=pep%20660. ↩
2. You can see more details of this configuration option in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html ↩

I’m experiencing an issue with setuptools, specifically that my directory layout is upsetting the auto discovery feature (for my packages and modules). The error I get is shown above, which instructs me to do one of three listed options… Option 2 is not an option for me. I tried setting up custom discovery which did not work (it still complained about the layout and ignored the exclude call). I also tried explicitly listing out the packages and modules, but it seemingly still tries to use auto discovery regardless….

I had my script working setuptools (using setup.py) with Python 3.8, but this issue is now occurring after upgrading to 3.11

Originally posted by @ehkristen in #4013 (comment)

[abravalheri]

Hi @ehkristen , could you please provide a mimum reproducible example so that we can investigate the issue?

[ehkristen]

setuptools version
Not sure -> using py2exe

Python version
Python 3.11

OS
Edition Windows 10 Enterprise
Version 22H2
OS build 19045.5011
Experience Windows Feature Experience Pack 1000.19060.1000.0

Additional environment information
No response

Description
I have an existing Python project that defines five packages, three modules (four if you count main.py), and a folder with files that get used at the top level,

The relevant directory structure is as follows:

.../src/backend:

├── package_a
|        ├──   __init__.py
|        └──   package_a.py
├── files 
|        └──   (csv files in this folder)
├── package_b
|        ├──   __init__.py
|        └──   package_b.py
├── package_c
|        ├──   __init__.py
|        └──   package_c.py
├── package_d
|        ├──   __init__.py
|        └──   package_d.py
├── package_e
|        ├──   __init__.py
|        └──   package_e.py
├── module_a.py
├── module_b.py
├── main.py
├── setup.py
└── module_c.py

I have a build.cmd file (with a _build.cmd file) which tries to install the modules/packages and then it is expected for it to import all the packages/ modules.

The setup.py is:

from setuptools import setup

import setuptools
import distutils.core import setup
import py2exe
import os

setuptool.setup(
    name="<Name>",
    author="<Name>",
    author_email="<name>@<company>.com",
    description="<description>", 
    long_description= long_description,
    long_description_content_type="text/markdown",
    packages= ['package_a', 'package_b', 'package_c', 'package_d', 'package_e'],
    package_dir={"package_a":"package_a", "package_b":"package_b", 
                          "package_c":"package_c", "package_d":"package_d", 
                          "package_e":"package_e"}
    py_modules=[ "module_a", "module_b", "module_c"],
    data_files=[ <os.walk of the file directory location, gathers all the file directories> ],
    python_requires=">=3.6",
) 

setup(console=['main.py'])

Expected behavior
With python <= 3.8, installation works fine and the imports described above work fine and the executable main.exe gets built (But note: I was using the auto discovery function, which worked perfectly fine. Auto discovery with python 3.11 with my flat layout errors and build gets terminated).

How to Reproduce
With python >=3.11 installation fails with the following error:

error: Multiple top-level modules discovered in a flat-layout: ['package_a', 'package_b', 'package_c', 'package_d', 'package_e', 'files'].

Documentation herecontains the note:

If you are using auto-discovery with flat-layout, setuptools will refuse to create distribution archives with multiple top-level packages or modules.

This is done to prevent common errors such as accidentally publishing code not meant for distribution (e.g. maintenance-related scripts).

Users that purposefully want to create multi-package distributions are advised to use Custom discovery or the src-layout.

However, as of yet I have been unable to find any combination of setup.py options that allows the packages to be installed and all modules to be imported with a successful build. Either installation fails with the above error about the flat-layout, or the main.exe can't locate the packages/modules when ran. Please note that I am not using a MANIFEST.in nor .egg-info files (did not need them before upgrading python from 3.8 to 3.11). I tried adding these files, but it made no difference in terms of outcomes.

When I try to change the structure of the project, which is a less than desirable option, I am able to have the build go through, but the resulting executable main.exe is not able to locate the modules/packages and crashes. I filed this as a bug because the existing directory structure worked fine with python 3.8, but there seems to be no support for it in python 3.11 (mainly that the 3 listed options did not work). Could you provide an example of a working setuptools setup.py that works with a flat-layout?

Did I simply just not find the right combination of configuration to make this work? (I tried different options for numerous hours and even looked through the setuptools source code.)

Output
See above

[abravalheri]

Umm... we probably need to further simplify this reproducer. As described in minimum reproducible example, we should remove anything that is not related to the investigation of the problem. This includes removing py2exe (since the suspicion is of a bug in setuptools not in py2exe). The provided setup.py script also contains some syntax errors, so I assume it was not tested before posting above and that it also needs some extra fixes.

---

There seems to be a conceptual error in your setup.py script: it is calling setup(...) twice.

The first time it includes packages=... and py_modules=..., and that is exactly what it needed to disable the aut-discovery. The second time however, it does not. That is the likely origin of the problem. Since the second call does not have any of the arguments, the auto-discovery is triggered.

In general it does not make sense to call the setup(...) function multiple times: it should be called only once per setup.py file. If you have multiple packages, then you somehow need to organise them to have multiple setup.py files. Anyway, if you want to disable auto-discovery you need to ensure that any call to setup(...) contains either packages=... or py_modules=... as indicated in the error message.

Also, please do not import distutils.core.setup instead use setuptools.setup.

---

Please find bellow a small example of how the example project would work once fixed the points discussed above and removed the unrelated py2exe. I am also adding a pyproject.toml file as recommended by the setuptools docs. This example is running in Linux, but it is very likely that an equivalent setup in Windows would work fine:

> docker run --rm -it python:3.11-bookworm /bin/bash
mkdir /tmp/proj && cd /tmp/proj
mkdir package_a package_b package_c package_d package_e
touch package_a/__init__.py package_a/package_a.py
touch package_b/__init__.py package_b/package_b.py
touch package_c/__init__.py package_c/package_c.py
touch package_d/__init__.py package_d/package_d.py
touch package_e/__init__.py package_e/package_e.py
touch module_a.py
touch module_b.py
touch module_c.py
touch main.py

cat <<EOF > setup.py
import setuptools

setuptools.setup(
    name="package",
    author="<Name>",
    author_email="<name>@<company>.com",
    description="<description>", 
    long_description="long_description",
    long_description_content_type="text/markdown",
    packages= ['package_a', 'package_b', 'package_c', 'package_d', 'package_e'],
    package_dir={"package_a":"package_a", "package_b":"package_b", 
                          "package_c":"package_c", "package_d":"package_d", 
                          "package_e":"package_e"},
    py_modules=[ "module_a", "module_b", "module_c"],
    # data_files=[ <os.walk of the file directory location, gathers all the file directories> ],
    python_requires=">=3.6",
)
EOF

cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools>=75.2"]
build-backend = "setuptools.build_meta"
EOF

apt update -y
apt install tree
tree
# .
# ├── main.py
# ├── module_a.py
# ├── module_b.py
# ├── module_c.py
# ├── package_a
# │   ├── __init__.py
# │   └── package_a.py
# ├── package_b
# │   ├── __init__.py
# │   └── package_b.py
# ├── package_c
# │   ├── __init__.py
# │   └── package_c.py
# ├── package_d
# │   ├── __init__.py
# │   └── package_d.py
# ├── package_e
# │   ├── __init__.py
# │   └── package_e.py
# ├── pyproject.toml
# └── setup.py


python -m pip install .
# ...
# Successfully built package
# Installing collected packages: package
# Successfully installed package-0.0.0

---

Could you please try fixing the points discussed above? Let me know how it goes and feel free to close the issue if the changes suggested above are enough to fix the problems.

Otherwise, if you want more help with the investigation, please make sure to provide an updated version of the reproducer, reduced to the minimum (without dependencies not maintained by setuptools such as py2exe), and that clearly showcases the error. Please also make sure to follow the tips in https://stackoverflow.com/help/minimal-reproducible-example.

[ehkristen]

Hi, sorry for the confusion! the second call at the bottom is part of the second portion of my setup call and is used to signify that I want the executable to be set to main.py (I did not include the parts below that line).

The second call is using the distutils.core.setup

the second call looks like this:
setup(console =['main.py']
followed by code that zips the library in a folder that gets saved in the /dist folder that's generated.

Having both calls was not an issue with python 3.8

[abravalheri]

Hi @ehkristen , many things might be happening since you last tried with 3.8, including things "accidentally working", changes in the version of setuptools that is used, or deprecated functionality being removed. So, I recommend paying attention to the points I highlighted above.

Please make sure that you include packages=... and of py_modules=... in any calls to setup(...), and please don't use distutils.core.setup. I also don't recommend executing setup(...) more than once in the same script, this is very likely error prone, and setuptools is not really designed to support that.

Moreover it is a good idea to have a look at any deprecation warnings and try to tackle them.

If you don't plan to submit a new version of the reproducer, let's close this issue because we have exhausted the investigation for the given example.

As I showed in my previous comment, once you fix the setup.py script and follow the suggestions in the error message, the problems seem to go away.

[ehkristen]

Hi @abravalheri , so I reworked everything, utilizing py2exe.freeze, but I'm experiencing an issue where the zipped library does not include the base python modules that are called in the script? (Example pandas) I tried explicitly writing out the module in the options={ "includes": "pandas"} but when I tell it to include pandas, the hook_pandas from hooks.py errors saying that there's no module named packaging . I checked inside of this function to see what was happening, and it appears to import from a module called packaging? The only packaging module that I see in Python 311 is under site-packages/pkg_resources/_vendor/packaging while hook.py is under py2exe.

[abravalheri]

Hi @ehkristen, I will not be able to assist you with py2exe related questions.
If you have a minimal reproducer that only uses setuptools and PyPA tools like build or pip (without py2exe), I can try to have a look, but I do need a proper minimal reproducible example for that (so if you are interested please make sure to share a reproducer that follow the steps in https://stackoverflow.com/help/minimal-reproducible-example).

packaging is used by setuptools, but because of the bootstrap limitations of the Python packaging ecosystem, setuptools ships its own vendored version of packaging. However, it will always try to first use the version that is already available in the environment, and just rely on the vendored copy as last resort.

[ehkristen]

Hi @abravalheri , is it possible to use setuptools to create an executable? From my understanding it does not make the executable?

[abravalheri]

Setuptools allows specifying executable scripts to be installed alongside packages via entry-points: https://setuptools.pypa.io/en/latest/userguide/entry_point.html .

Other than that, setuptools does not natively support creating single file executables.

Other tools may use setuptools as an underlying step in this process, but that logic is not implemented in the setuptools repository.