Merge pull request #29 from Bears-R-Us/sanbox-prod-partitioning

Partitioning Arkouda-NJIT into Arachne and Arachne-Development

.gitignore

@@ -1,4 +1,3 @@
-GraphServerModules.cfg.*
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -128,3 +127,9 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# MacBook Stuff
+.DS_Store
+
+# Old Arkouda-NJIT stuff
+GraphServerModules.cfg.*

README.md

@@ -1,64 +1,43 @@
-## Arachne: An Arkouda Package for Large-Scale Graph Analytics
+## Arkouda-NJIT
+This is an external repository to build functionality for [Arkouda](https://github.com/Bears-R-Us/Arkouda) with a focus on advanced graph processing. It is built with the same structure as [arkouda-contrib](https://github.com/Bears-R-Us/arkouda-contrib) to manage modules and easily swap between the production (`arachne`) and development (`arachne_development`) directories.
 
-## Purpose
+## Prerequisites
+1. Download and build [Chapel](https://chapel-lang.org/download.html).
+2. Download but **do not build** [Arkouda](https://github.com/Bears-R-Us/arkouda). **We recommend using the most recent release.**
+3. Follow instructions to activate the Arkouda environment and install all [prerequisites](https://github.com/Bears-R-Us/arkouda#prerequisites-toc). **We recommend using `Anaconda` to manage all dependencies.** 
 
-This is an external repo to build Graph related functionality for `Arkouda`
-(see https://github.com/Bears-R-Us/Arkouda)
-
-## Generate Server Code
-To generate the server code run the `server-config.py` file.
-It will generate commands which you can pipe to a shell for execution which
-will build the Arkouda server including the Graph server functions.
+## Installation
+Installation is performed through running the `module_configuration.py` file. The complete path to the location of `arkouda` must be specified through `ak_loc` and the complete path to the location of `arachne` should be specified through `pkg_path`.
 
 ```bash
-# Print usage instructions
-python server-config.py --help
-
-# Sample invocation
-python server-config.py --arkouda=$ArkoudaHomeDirectory
+# Print usage instructions.
+python module_configuration.py --help
 
-# Sample execution
-python server-config.py --arkouda=$ArkoudaHomeDirectory | bash
+# Command to execute to build the Arkouda server with Arachne.
+python module_configuration.py --ak_loc=/complete/path/to/arkouda/ --pkg_path=/complete/path/to/arkouda-njit/arachne/ | bash
 ```
 
-## Client code
-Now we have two directories
+The above command will pipe the following three commands to terminal that installs Arachne using pip, copies the Arkouda server modules to a temporary file, and combines them with the Arachne server modules to build the enhanced `arkouda_server`.
 ```bash
-arkouda_graph/
-All graph algorithm related functions
-
-suffix_array/
-All suffix array related python functions
-
+pip install -U /complete/path/to/arkouda-njit/arachne/client
+cp /complete/path/to/arkouda/ServerModules.cfg ~/TmpServerModules.cfg.1683320760
+ARKOUDA_SERVER_USER_MODULES=" /complete/path/to/arkouda-njit/arachne/server/BuildGraphMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/PropertyGraphMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/GraphInfoMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/BFSMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/TriCtrMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/TriCntMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/TrussMsg.chpl  /complete/path/to/arkouda-njit/arachne/server/CCMsg.chpl" ARKOUDA_CONFIG_FILE=~/TmpServerModules.cfg.1683320760 ARKOUDA_SKIP_CHECK_DEPS=true make -C /Users/alvaradoo/Research/arkouda
 ```
 
-## Server code
-Now we have the following chapel modules.
-```bash
-BFSMsg--for breadth first search
-TrussMsg--for truss analysis
-SuffixArrayMsg--for suffix array
-GraphMsg-- for the basic graph operation
-TriCntMsg--for triangle counting
-TriCtrMsg-- for triangle centrality
-JaccardMsg-- for Jaccard coefficients
-CCMsg-- for Connected Components
-```
+The server can be started as specified in the [Arkouda documentation](https://github.com/Bears-R-Us/arkouda#running-arkouda_server-toc). To run a simple test file as well as pytests please proceed to the [arachne](arachne/) folder for those instructions.
 
+### Installing Development Arachne
+If you are interested in installing the development version of Arachne, please follow the same instructions as above, but for `pkg_path` include `/complete/path/to/arkouda-njit/arachne_development/`.
 
-## Test  code
-```bash
-client: Python test code
-   benchmarks: different benchmark code
-server:
-   UnitTestCh: chaple unit testing code of the extended functions
+## Usage Notes
+```python
+import arkouda as ak
+import arachne as ar
+# code using arachne and arkouda below
 ```
 
-
-## Call extended functions 
-```bash
-(1) Under the master arkouda directory, copy the arkouda-njit directory to here and rename it as arkouda_njit or 
-create a arkouda_njit link to the arkouda-njit directory
-(2) import arkouda_njit as njit
-(3) call all the extended function  as njit.function
-```
+## Common Issues
+* **Issue**: Unrecognized HDF5, Apache Arrow, etc. installations. 
+  **Fix**: Ensure `Makefile.paths` was properly added to the base Arkouda directory. More information can be found in the [Arkouda build instructions](https://github.com/Bears-R-Us/arkouda#building-arkouda-toc).
+* **Issue**: Arkouda or Arachne functions are not recognized when executing scripts.
+  **Fix**: Make sure to run `pip3 install -e .` at both `/complete/path/to/arkouda-njit/arachne/client/.` and `/complete/path/to/arkouda/.` This issue should not pop up for Arachne if the commands generated by `module_configuration.py` are piped to the terminal.

arachne/README.md

@@ -4,49 +4,31 @@
 The Arachne project provides an extension for graph analysis that works with Arkouda. Arachne mimics the NetworkX API while providing high-performance implementations of graph kernels in Chapel. 
 
 ## Installation
-Installation is performed by running `module_configuration.py`. This file is copied from the `Bears-R-Us/arkouda-contrib` repository. When running `module_configuration.py`, the complete path to the location of the Arkouda repo must be specificed through the `ak_loc` flag along with the complete path to the Arkouda contrib repo. In our case, it would be the complete path to this directory housing Arachne. An example follows below:
+Installation is performed by running `module_configuration.py` in the main `arkouda-njit` repo. The `module_configuration.py` script should be run as below. Then, follow all the further instructions in the base `arkouda-njit` repo. 
 
-```
+```bash
 python3 module_configuration.py --ak_loc=/complete/path/to/arkouda/ --pkg_path=/complete/path/to/arkouda-njit/arachne/
 ```
 
-When the above command completes, the result will be a couple of commands along the lines of:
-```
-pip install -U /rhome/oaa9/Research/arkouda-njit/arachne/client
-cp /rhome/oaa9/Research/arkouda/ServerModules.cfg ~/TmpServerModules.cfg.1681487975
-ARKOUDA_SERVER_USER_MODULES=" /rhome/oaa9/Research/arkouda-njit/arachne/server/GraphMsg.chpl" ARKOUDA_CONFIG_FILE=~/TmpServerModules.cfg.1681487975 ARKOUDA_SKIP_CHECK_DEPS=true make -C /rhome/oaa9/Research/arkouda
-```
-
-The length of the last command is dependent on the number of modules you have listen inside of `arkouda-njit/arachne/server/ServerModules.cfg`. Within here you can comment in and out the lines of the modules you wish to compile and not compile. By default, all of them are left uncommented.
-
-Next, simply just copy and paste the outputted commands into terminal and your arkouda server should start to build with Arachne added onto it. 
-
-If you run into environment errors, you can run the commands below to ensure both Arachne and Arkouda have been pip installed into your Python environment. This will oftentimes fix errors where Arkouda or Arachne commands are unrecognized by Python.
-```
-pip3 install -e /path/to/arachne/client/.
-pip3 install -e /path/to/arkouda/.
-```
-
 ## Usage
-To ensure Arachne is property installed, you can use a `arkouda-njit/arachne/arachne-simple-tests.py` to build a small property graph. The file is executed as follows:
-```
+To ensure Arachne is property installed, you can use a `arkouda-njit/arachne/arachne-simple-tests.py` to build a small property graph, filer it, and run some graph kernels. The file is executed as follows:
+```bash
 python3 arachne-simple-tests.py node port
 ```
 Where the host name and port number change according to your configuration.
 
-For more involved benchmarking, functions currently available can be found in `arkouda-njit/arachne/client/README.md` with benchmark files that can be used as a sample found in the `arkouda-njit/arachne/benchmarks` folder. For benchmarking a single graph you can execute: 
-```
+For more involved benchmarking, we have included benchmark files that can be used as a sample found in the `arkouda-njit/arachne/benchmarks` folder. These are currently only available for breadth-first search (bfs). For benchmarking a single graph you can execute: 
+```bash
 python3 bfs.py node port -f /path/to/arkouda-njit/arachne/data/karate.mtx -t 10
 ```
 
 For benchmarking a directory of graphs you can execute: 
-```
+```bash
 python3 bfs.py node port -d /path/to/arkouda-njit/arachne/data -t 10
 ```
 
 ## Testing
-The Arachne tests are executed from the arkouda-njit/arachne directory as follows:
-```
+The Arachne tests are executed from the arkouda-njit/arachne directory as follows with pytest:
+```bash
 python3 -m pytest test/algorithm_test.py test/class_test.py test/prop_graph_test.py test/reading_test.py
 ```
-**Note**: Errors when executing using pytest from arachne directory. 

arachne/client/README.md

@@ -7,14 +7,12 @@ Due to the emergence of massive real-world graphs such as social networks, compu
 - Graph Double-Index Data Structuren Class(es) and Methods
     - `Graph` class for undirected (un)weighted graphs. 
     - `DiGraph` class for directed weighted graphs. 
-    - `PropGraph` class for property graphs that can hold node labels, edge relationships, and properties for both nodes and edges.
     - `__len__(self) -> int` returns the number of vertices of a Graph (DiGraph) G. Use `len(G)`.
     - `size(self) -> int` returns the number of edges of a Graph (DiGraph) G. Use `G.size()`.
     - `add_edges_from(self, akarray_src: pdarray, akarray_dst: pdarray, akarray_weight: Union[None, pdarray] = None) -> None` adds to a Graph (DiGraph) object edges from arkouda arrays. The arrays `akarray_src` and `akarray_dst` must be specificed. An optional extra array called `akarray_weight` could be used to make a weighted Graph (DiGraph).
-    - **More found in arachne.py**
 - Graph Building Methods
     - `read_edgelist(path: str, weighted: bool = False, directed: bool = False, comments: str = "#", filetype: str = "txt") -> Union[Graph, DiGraph]` returns a Graph or DiGraph object with read graph data. Removes self-loops and multiedges. Has option to enable reading in a matrix market file that is typically used to store graphs. 
     - `read_known_edgelist(ne: int, nv: int, path: str, weighted: bool = False, directed: bool = False, comments: str = "#", filetype: str = "txt") -> Union[Graph, DiGraph]` returns a Graph or DiGraph object with read graph data. Does not remove self-loops or multiedges. Intended for use when the number of vertices or edges are known for faster graph building times. 
 - Algorithms
     - `bfs_layers(graph: Graph, source: int) -> pdarray` returns a depth array with how far each vertex is from the source vertex. Currently, we only support single source BFS.
-    - **More found in arachne.py**
+    - **More to Come!**

arachne/client/setup.py

@@ -9,7 +9,7 @@
 
 setup(
     name="arachne",
-    version="0.0.1",
+    version="2023.04.30",
     description="Graph functionality in Chapel for use with Arkouda.",
     long_description=long_description,
     long_description_content_type="text/markdown",

arachne/data/prop_graph_arrays.txt

@@ -15,7 +15,7 @@
 5 RELATIONSHIPS
 [lives-with] [drives, owns] [lives-with] [drives] [drives]
 4 NODE_LABELS
-[Person] [Person] [Car, Vehicle] []
+[Person] [Person] [Vehicle, Car] []
 3 NODE_PROPS
 [(name, Dan), (born, 12-05-1975)] [(name, Ann), (born, 05-29-1990)] []
 5 EDGE_PROPS

arachne/server/BuildGraphMsg.chpl

@@ -340,7 +340,7 @@ module BuildGraphMsg {
                 }
             } else {
                 try  { 
-                    combine_sort(srcR, dstR, e_weightR, weighted, false);
+                    combine_sort(srcR, dstR, e_weightR, weighted, true);
                 } catch {
                     try!  smLogger.error(getModuleName(),getRoutineName(),getLineNumber(),
                                         "Combine sort error");
@@ -401,7 +401,7 @@ module BuildGraphMsg {
         var myedgeD = mysrc.domain;
 
         var myneighbor = makeDistArray(new_nv, int);
-        var myvertexD=myneighbor.domain;
+        var myvertexD = myneighbor.domain;
 
         // Arrays made from the edge domain. 
         var mydst, mysrcR, mydstR, myiv: [myedgeD] int;
@@ -656,7 +656,7 @@ module BuildGraphMsg {
                 }
             } else {
                 try  { 
-                    combine_sort(srcR, dstR, e_weightR, weighted, false);
+                    combine_sort(srcR, dstR, e_weightR, weighted, true);
                 } catch {
                     try!  smLogger.error(getModuleName(),getRoutineName(),getLineNumber(),
                                         "Combine sort error");