datafusion-ray is not published to pypi.org

[sairamkrish]

Hi

We are trying to use datafusion-ray. To get started we tried to pip install datafusion-ray. However datafusion-ray is not published.

I tried installing directly from github like this :

poetry add git+https://github.com/apache/datafusion-ray.git

Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 1 install, 0 updates, 0 removals

  - Installing datafusion-ray (0.1.0 2783613): Failed

Failure reason is not displayed even in --verbose mode.
If someone wants to do datafusion + ray, should we continue using raysql till then ?

I would like to contribute here if you are open for it.

[robtandy]

👋 Hi @sairamkrish !

I've been working on a revised version of datafusion ray for the past month or so. I submitted a PR today in fact.
It replaces the disk based shuffle with ArrowFlight streaming shuffle, and is able to scale beyond one node.

It should be in a place where it can build and work for you. You can read the PR here

#60

And access the branch from here: https://github.com/robtandy/datafusion-ray/tree/main/src

If you have any trouble installing or using, you can open an issue here and I'll try to help.

[sairamkrish]

@robtandy Thanks for the response. Since datafusion-ray is not published to pypi.org, finding it difficult to install in a python project and experiment with it. Without a datafusion-ray package in pypi, building it from source and packaging it as a dependency on another project etc., becomes too complex.

Do you have any suggestion or work around ?