Kaleido:TNG failure using WSL ubuntu 24.04

[gvwilson]

Here is what I tried - I have not been able to make it work so far.

I use Windows 11 + Ubuntu on WSL
WSL ubuntu 20.04 - Does not support GUI applications
WSL ubuntu 22.04 - Supports GUI applications, but its quite buggy
WSL ubuntu 24.04 - Supports GUI applications and chrome

Due to this, I decided to test with WSL ubuntu 24.04 with chromium browser
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04 LTS
Release:        24.04
Codename:       noble
$ chromium-browser --version
Chromium 129.0.6668.100 snap

I get the following error with test.py
$ mkdir -p /tmp/kal
$ cd /tmp/kal
<downloaded the files here>
$ uv pip install -r requirements.txt
Resolved 11 packages in 1.00s
Prepared 4 packages in 3.32s
Installed 11 packages in 5.74s
 + choreographer==0.99.1
 + kaleido==0.4.0rc1
 + numpy==2.1.2
 + packaging==24.1
 + pandas==2.2.3
 + plotly==5.24.1
 + python-dateutil==2.9.0.post0
 + pytz==2024.2
 + six==1.16.0
 + tenacity==9.0.0
 + tzdata==2024.2
$ .venv/bin/python -V
Python 3.11.9
$ ls -lh .venv/bin/python
lrwxrwxrwx 1 ali ali 79 Oct 25 03:11 .venv/bin/python -> /home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/bin/python3.11
$ .venv/bin/python test.py
Traceback (most recent call last):
  File "/tmp/kal/.venv/lib/python3.11/site-packages/kaleido/__init__.py", line 82, in to_image
    js_response = json.loads(response.get("result").get("result").get("value"))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/kal/test.py", line 3, in <module>
    fig.write_image("figure.png", engine="kaleido")
  File "/tmp/kal/.venv/lib/python3.11/site-packages/plotly/basedatatypes.py", line 3835, in write_image
    return pio.write_image(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/kal/.venv/lib/python3.11/site-packages/plotly/io/_kaleido.py", line 266, in write_image
    img_data = to_image(
               ^^^^^^^^^
  File "/tmp/kal/.venv/lib/python3.11/site-packages/plotly/io/_kaleido.py", line 143, in to_image
    img_bytes = scope.transform(
                ^^^^^^^^^^^^^^^^
  File "/tmp/kal/.venv/lib/python3.11/site-packages/kaleido/scopes/plotly.py", line 197, in transform
    img = kaleido.to_image_block(spec, Path(self._[tempfile.name](http://tempfile.name/)).absolute(), self._topojson, self._mapbox_access_token)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/kal/.venv/lib/python3.11/site-packages/kaleido/__init__.py", line 27, in to_image_block
    return asyncio.run(to_image(spec, f, topojson, mapbox_token))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/tmp/kal/.venv/lib/python3.11/site-packages/kaleido/__init__.py", line 86, in to_image
    raise RuntimeError(response) from e
RuntimeError: {'id': 3, 'result': {'result': {'type': 'object', 'subtype': 'error', 'className': 'ReferenceError', 'description': 'ReferenceError: kaleido_scopes is not defined\n    at <anonymous>:1:73', 'objectId': '78168610404079178.2.2'}, 'exceptionDetails': {'exceptionId': 1, 'text': 'Uncaught', 'lineNumber': 0, 'columnNumber': 72, 'scriptId': '4', 'stackTrace': {'callFrames': [{'functionName': '', 'scriptId': '4', 'url': '', 'lineNumber': 0, 'columnNumber': 72}]}, 'exception': {'type': 'object', 'subtype': 'error', 'className': 'ReferenceError', 'description': 'ReferenceError: kaleido_scopes is not defined\n    at <anonymous>:1:73', 'objectId': '78168610404079178.2.3'}}}, 'sessionId': '069DAC5009E6A686F51191EFE53F4DBE'}

This is repeatable - and has never successfully worked for me.

I can see there is an error about kaleidoscopes not defined - so, I tried checking if my chromium works with:
$ chromium-browser
and it opened up fine.

[gvwilson]

$ .venv/bin/dtdoctor
**************************************************
*****Collecting information about the system:*****
Linux
5.15.146.1-microsoft-standard-WSL2
#1 SMP Thu Jan 11 04:09:03 UTC 2024
uname_result(system='Linux', node='DESKTOP-72DNRM8', release='5.15.146.1-microsoft-standard-WSL2', version='#1 SMP Thu Jan 11 04:09:03 UTC 2024', machine='x86_64')
***************Looking for browser:***************
looking for chrome
looking for None
looking for Chrome
looking for None
looking for google-chrome
looking for None
looking for google-chrome-stable
looking for None
looking for Chrome.app
looking for None
looking for Google Chrome
looking for None
looking for Google Chrome.app
looking for None
looking for chromium
looking for /snap/bin/chromium
/snap/bin/chromium
************Looking for version info:*************
/tmp/kal/.venv/bin/python3: No module named pip
*************Done with version info.**************
****************Sync test headless****************
DEBUG REPORT:
BROWSER_PATH: /snap/bin/chromium
USER_DATA_DIR: /tmp/tmpc1bsqfl0
Old Headless mode will be removed from the Chrome binary soon. Please use the new Headless mode (https://developer.chrome.com/docs/chromium/new-headless) or the chrome-headless-shell which is a standalone implementation of the old Headless mode (https://developer.chrome.com/blog/chrome-headless-shell).

[1025/051913.968305:WARNING:sandbox_linux.cc(430)] InitializeSandbox() called with multiple threads in process gpu-process.
[1025/051913.972788:WARNING:viz_main_impl.cc(85)] VizNullHypothesis is disabled (not a warning)
write_json:
write_json: {"id": 0, "method": "Browser.close"}
wrote_json.
[1025/051915.892264:ERROR:devtools_pipe_handler.cc(188)] Connection terminated while reading from pipe
[1025/051915.892865:ERROR:devtools_pipe_handler.cc(273)] Could not write into pipe
[1025/051915.892906:ERROR:devtools_pipe_handler.cc(273)] Could not write into pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
pipe.close() (or slow Browser.close) method closed browser
Tempfile still exists?: False
***********Done with sync test headless***********
***************Async Test headless****************
DEBUG REPORT:
BROWSER_PATH: /snap/bin/chromium
USER_DATA_DIR: /tmp/tmpyec2k4kv
read_jsons (blocking):
write_json:
write_json: {"id": 0, "method": "Target.getTargets"}
wrote_json.
Starting watchdog
Old Headless mode will be removed from the Chrome binary soon. Please use the new Headless mode (https://developer.chrome.com/docs/chromium/new-headless) or the chrome-headless-shell which is a standalone implementation of the old Headless mode (https://developer.chrome.com/blog/chrome-headless-shell).

[1025/051916.469567:WARNING:sandbox_linux.cc(430)] InitializeSandbox() called with multiple threads in process gpu-process.
[1025/051916.471829:WARNING:viz_main_impl.cc(85)] VizNullHypothesis is disabled (not a warning)
{"id":0,"result":{"targetInfos":[]}}
read_jsons: {'id': 0, 'result': {'targetInfos': []}}
run_read_loop() found future for key ('', 0)
read_jsons (blocking):
is_closed called with wait: 0
write_json:
write_json: {"id": 1, "method": "Browser.close"}
wrote_json.
{"id":1,"result":{}}
read_jsons: {'id': 1, 'result': {}}
run_read_loop() found future for key ('', 1)
is_closed called with wait: 0
read_jsons (blocking):
read_jsons pipe was closed, raising
caught OSError in read()
PipeClosedError caught
Browser is being closed because chrom* closed
Browser.close method closed browser
Tempfile still exists?: False
**********Done with async test headless***********

Error in: System Info
Traceback (most recent call last):
  File "/tmp/kal/.venv/lib/python3.11/site-packages/choreographer/browser.py", line 735, in diagnose
    print(subprocess.check_output([sys.executable, '-m', 'pip', 'freeze']))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/kal/.venv/bin/python3', '-m', 'pip', 'freeze']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/tmp/kal/.venv/bin/dtdoctor", line 8, in <module>
    sys.exit(diagnose())
             ^^^^^^^^^^
  File "/tmp/kal/.venv/lib/python3.11/site-packages/choreographer/browser.py", line 777, in diagnose
    raise BaseException("There was an exception, see above.")
BaseException: There was an exception, see above.

[ayjayt]

Compiling important parts here:

$ .venv/bin/python test.py
Traceback (most recent call last):
  File "/tmp/kal/.venv/lib/python3.11/site-packages/kaleido/__init__.py", line 82, in to_image
    js_response = json.loads(response.get("result").get("result").get("value"))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ali/.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib/python3.11/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType

RuntimeError: 
{'id': 3, 
'result': {
   'result': {
      'type': 'object', 
      'subtype': 'error', 
      'className': 'ReferenceError', 
      'description': 'ReferenceError: kaleido_scopes is not defined\n    at <anonymous>:1:73', 'objectId': '78168610404079178.2.2'}, 'exceptionDetails': {'exceptionId': 1, 'text': 'Uncaught', 'lineNumber': 0, 'columnNumber': 72, 'scriptId': '4', 'stackTrace': {'callFrames': [{'functionName': '', 'scriptId': '4', 'url': '', 'lineNumber': 0, 'columnNumber': 72}]}, 'exception': {'type': 'object', 'subtype': 'error', 'className': 'ReferenceError', 'description': 'ReferenceError: kaleido_scopes is not defined\n    at <anonymous>:1:73', 'objectId': '78168610404079178.2.3'}}}, 'sessionId': '069DAC5009E6A686F51191EFE53F4DBE'}

FYI this is chromium reporting a javascript error. I'm going to improve the debugging, install WSL, and debug this myself.

It's probably a path issue.

[ayjayt]

• Improve debugging options on kaleido, especially in regards to paths to vendored libraries
• Install WSL and see how it goes

[ayjayt]

So @gvwilson AFAIK this should work w/ the new release candidate.