Bootstrapping IPM from artifact ignores requirements.txt

[isc-shuliu]

During recent meetings with Eric (from IPE) and @isc-jili , we discovered that when loading the installer artifact of IPM 0.10.0 (e.g., zpm-0.10.0-beta.3.xml) on a fresh iris instance, the python requirements.txt is ignored, causing all import oras calls to fail.

However, manually installation from source code works fine. Namely,

do $System.OBJ.Load("/<ipm-repo-root>/preload/cls/IPM/Installer.cls","ck")
do ##class(IPM.Installer).setup("/<ipm-repo-root>/", 3)

Something in the bootstrap process is missing.

[isc-shuliu]

The artifacts, e.g. zpm-0.10.0-beta.3.xml, do not contain any information from requirements.txt (as expected).

Unless there's a way to attach arbitrary files into the xml artifact, we may need to hard code the python dependency installation into the code if we want to keep using the .xml artifact for one-liner installation.

[isc-kiyer]

@isc-shuliu This got me thinking about what would happen in an environment that has no internet access. How would it install python dependencies? Could the oras python package be packaged up at time of publishing the zpm artifact? Also, unrelated note but if we are changing the artifact type from xml, perhaps also rename it to ipm?

cc @isc-tleavitt @isc-eneil @isc-jili @isc-jlechtne

[isc-kiyer]

Useful post for downloading python packages for packaging into IPM distributable: https://stackoverflow.com/questions/49333152/copy-installed-packages-using-pip-to-another-environment

[isc-eneil]

@isc-shuliu The work-around mentioned is not working for me (when trying with stakeholder devs). I was on a 0.10.0-beta.11 version and tried loading from the source code zip file on the Releases page. It fails when it hits the reload phase and tries to load requirements.txt with this error:

[USER|ZPM] Initialize SUCCESS
[USER|ZPM] Reload START (/Users/jlou/Downloads/ipm-0.10.0-beta.11/)
[USER|zpm] requirements.txt START
[USER|zpm] requirements.txt FAILURE
[ZPM] Reload FAILURE
ERROR! ObjectScript error: InstallPythonRequirements+25^%IPM.Lifecycle.Base.1 *Failed to Load Python: Check documentation and messages.log, Check CPF parameters:[PythonRuntimeLibrary,PythonRuntimeLibraryVersion], Check sys.path setup in: $INSTANCE/lib/python/iris_site.py2025-01-30 15:38:23 0 IPM.Installer: ERROR #5002: ObjectScript error: InstallPythonRequirements+25^%IPM.Lifecycle.Base.1 *Failed to Load Python: Check documentation and messages.log, Check CPF parameters:[PythonRuntimeLibrary,PythonRuntimeLibraryVersion], Check sys.path setup in: $INSTANCE/lib/python/iris_site.py
2025-01-30 15:38:23 0 IPM.Installer: Installation failed at 2025-01-30 15:38:23
2025-01-30 15:38:23 0 %Installer: Elapsed time 2.66866s

[isc-shuliu]

@isc-shuliu The work-around mentioned is not working for me (when trying with stakeholder devs). I was on a 0.10.0-beta.11 version and tried loading from the source code zip file on the Releases page. It fails when it hits the reload phase and tries to load requirements.txt with this error:

[USER|ZPM] Initialize SUCCESS [USER|ZPM] Reload START (/Users/jlou/Downloads/ipm-0.10.0-beta.11/) [USER|zpm] requirements.txt START [USER|zpm] requirements.txt FAILURE [ZPM] Reload FAILURE ERROR! ObjectScript error: InstallPythonRequirements+25^%IPM.Lifecycle.Base.1 *Failed to Load Python: Check documentation and messages.log, Check CPF parameters:[PythonRuntimeLibrary,PythonRuntimeLibraryVersion], Check sys.path setup in: $INSTANCE/lib/python/iris_site.py2025-01-30 15:38:23 0 IPM.Installer: ERROR #5002: ObjectScript error: InstallPythonRequirements+25^%IPM.Lifecycle.Base.1 *Failed to Load Python: Check documentation and messages.log, Check CPF parameters:[PythonRuntimeLibrary,PythonRuntimeLibraryVersion], Check sys.path setup in: $INSTANCE/lib/python/iris_site.py 2025-01-30 15:38:23 0 IPM.Installer: Installation failed at 2025-01-30 15:38:23 2025-01-30 15:38:23 0 %Installer: Elapsed time 2.66866s

Does your instance have embedded python enabled? It seems the error stems from this line

Set tSysModule = ##class(%SYS.Python).Import("sys")

which should work as long as embedded python is available.

As a sanity check, can you try running

USER>:py

Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] on linux
Type quit() or Ctrl-D to exit this shell.
>>> import sys
>>> sys.__loader__
<class '_frozen_importlib.BuiltinImporter'>

[isc-shuliu]

@isc-tleavitt
After discussion with HSOPS, @isc-kiyer suggested 2 possibles solutions.

1. Create a new StudioDocument class that represents the python wheel file (which is a zip under the hood) and attach it to the XML installer. Not sure how much work will be involved.

2. Publish zpm as a tarball instead, and modify the existing one-liner installation command to use the tarball instead of XML. It's easier but may break some backward compatibility for users relying on the one-liner as part of the workflow.

Which way do you recommend?

[isc-tleavitt]

@isc-shuliu neither - let's use the features in https://github.com/intersystems/ipm/tree/main/src/cls/IPM/Lifecycle/StudioProject to attach the file to the XML installer.

[isc-shuliu]

@isc-tleavitt @isc-kiyer

I have been overlooking something for a long time. While the oras package itself is pure Python, it has a translative dependency rpds-py (required by dependency jsonschema) which requires a Rust build-time dependency. The prebuilt wheel is also platform specific.

While there are talks to get rid of rpds-py in jsonschema, it has low priority and we probably won't see it any time soon.

This poses a dilemma for us if we want to ship oras along with IPM. We can't just include a wheel, because it's platform specific, unless we include a different wheel for every platform (and potentially python versions).

[isc-tleavitt]

Oof. Maybe this is an awful question to ask, but would oras function without that dependency?

[isc-shuliu]

Unfortunately no. I tried to import oras.client without rpds and it failed.

It may sound like (and probably is) over engineering to implement the HashTrieMap, HashTrieSet, and List classes in rpds, and place such an rpds adaptor toward the end of the PYTHTONPATH (such as lib/python as opposed to mgr/python. Because all rpds does is provide those data structures.