Hanging Test on GitHub Action with windows-latest with Kaleido 0.21.0

[idanmoradarthas]

Description:

I'm encountering an issue where a unit test freezes on windows-latest machine in GitHub Actions when using the Kaleido library (version 0.21.0) to create a Plotly figure. The test runs successfully on both Ubuntu and macOS.

Expected Behavior:

The test should create a Plotly figure, write it to an image file, and then compare the generated image to a baseline.

Test Code:

@pytest.mark.parametrize("add_random_classifier_line", [True, False], ids=["default", "without_random_classifier"])
def test_plot_roc_curve_with_thresholds_annotations(mocker, add_random_classifier_line, plotly_models_dict,
                                                    baseline_path,
                                                    result_path):
    y_true = np.array(plotly_models_dict["y_true"])
    classifiers_names_and_scores_dict = {name: np.array(data["y_scores"]) for name, data in plotly_models_dict.items()
                                         if name != "y_true"}

    def _mock_roc_curve(y_true, y_score, **kwargs):
        for classifier, scores in classifiers_names_and_scores_dict.items():
            if np.array_equal(scores, y_score):
                data = plotly_models_dict[classifier]["roc_curve"]
                return np.array(data["fpr_array"]), np.array(data["tpr_array"]), np.array(data["thresholds"])

    mocker.patch("ds_utils.metrics.roc_curve", side_effect=_mock_roc_curve)

    def _mock_roc_auc_score(y_true, y_score, **kwargs):
        for classifier, scores in classifiers_names_and_scores_dict.items():
            if np.array_equal(scores, y_score):
                return np.float64(plotly_models_dict[classifier]["roc_auc_score"])

    mocker.patch("ds_utils.metrics.roc_auc_score", side_effect=_mock_roc_auc_score)

    fig = plot_roc_curve_with_thresholds_annotations(
        y_true,
        classifiers_names_and_scores_dict,
        add_random_classifier_line=add_random_classifier_line
    )

    fig.write_image(str(result_path)) # this is the problematic line

    compare_images_from_paths(str(baseline_path), str(result_path))

GitHub Actions Workflow:

name: Build

on:
  push:
  pull_request:
    types: [ opened, reopened ]

permissions:
  contents: read

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ ubuntu-latest, windows-latest, macos-latest ]
        python-version: [ "3.9", "3.10", "3.11", "3.12" ]
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
      - name: Install tox and coveralls
        run: pip install tox coveralls
      - name: Install graphviz binaries for Linux (${{ runner.arch }})
        if: runner.os == 'Linux'
        run: |
          sudo apt-get update
          sudo apt-get -y install graphviz
      - name: Install graphviz binaries for Windows (${{ runner.arch }})
        if: runner.os == 'Windows'
        run: choco install graphviz
      - name: Install graphviz binaries for macOS (${{ runner.arch }})
        if: runner.os == 'macOS'
        run: brew install graphviz
      - name: Run tox (install test dependencies)
        # Run tox using the version of Python in `PATH`
        run: tox -e py -p auto --parallel-live
      - name: Upload coverage to Coveralls
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          COVERALLS_FLAG_NAME: ${{ matrix.os }}-python-${{ matrix.python-version }}
          COVERALLS_PARALLEL: true
        run: coveralls --service=github

  coveralls:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Finish Coveralls
        uses: coverallsapp/github-action@v2
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          parallel-finished: true

Troubleshooting Steps:

I've tried the following without success:

• Isolating the issue to a minimal reproducible example.
• Checking for any conflicting dependencies or environment variables.
• Verifying that the Kaleido library is installed correctly on the Windows runner.
• Setting the DISPLAY environment variable on Windows:

- name: Set display for Kaleido on Windows
  if: runner.os == 'Windows'
  run: |
    set DISPLAY=:0

• Ensuring Chromium is installed and runs smoothly on Windows.
• Increasing the GitHub Actions timeout for the test.
• Using a Headless Browser:

import plotly.io as pio
# Set the default renderer to 'kaleido' for headless operation
pio.renderers.default = "kaleido"

• Specifying the kaleido Renderer at Export Time:

fig.write_image("figure.png", engine="kaleido")

Additional Notes:

• I'm open to providing more information or debugging steps if needed.

I'm hoping to resolve this issue so that my tests can run consistently across all platforms.

[ayjayt]

Hi, this should be fixed immanently!

[ayjayt]

A release candidate is in alpha testing internally!