CONTRIBUTING.md

Welcome to nx-parallel!

Hi, Thanks for stopping by!

This project is part of the larger NetworkX project. If you're interested in contributing to nx-parallel, you can first go through the NetworkX's contributing guide for general guidelines on contributing, setting up the development environment, and adding tests/docs, etc.

Development workflow

To set the local development environment:

• Fork this repository.
• Clone the forked repository locally:

git clone git@github.com:<your_username>/nx-parallel.git

• Create a fresh conda/mamba virtualenv (learn more)

# Creating a virtual environment
python -m venv nxp-dev

# Activating the venv
source nxp-dev/bin/activate

# For Windows
.\nxp-dev\Scripts\Activate.ps1

• Install the dependencies using the following command

make install

make commands aren't supported on Windows-- so use the commands below instead:

pip install -e ".[test]"
pip install git+https://github.com/networkx/networkx.git@main
pip install git+https://github.com/joblib/joblib.git@main

• Create a new branch for your changes using

git checkout -b <branch_name>

• Make the changes in this new feature branch and run all tests before pushing them (learn more):

  make test

  • To only run only networkx's test suite with nx-parallel as the backend run:

    make run-networkx-tests

    • For Windows:

      $env:NETWORKX_TEST_BACKEND="parallel";
      $env:NETWORKX_FALLBACK_TO_NX="True";
      pytest --pyargs networkx/
      
      # linters
      pip install -e ".[developer]"
      pre-commit run --all-files

  • To only run nx-parallel specific tests run:

    make test-only-nx-parallel
    
    # With Windows:
    pytest nx_parallel

  • To run both nx-parallel specific test and networkx's test suite with the parallel backend run:

    make test-backend

• If all the tests run successfully, stage your changes, then commit and push and then create a PR

git add .
git commit -m"Your commit message"
git push origin <branch_name>

Adding tests in nx-parallel

• If an algorithm introduces an additional backend-specific argument make sure to add tests (using pytest) for it in a tests directory (in that algorithm's parent directory). For more, refer the networkx's tests.

• get_chunks is the additional backend-specific argument for almost all the algorithms in nx-parallel and it's tested for all algorithms together in nx_parallel/tests/test_get_chunks.py file.

Documentation syntax

For displaying a small note about nx-parallel's implementation at the end of the main NetworkX documentation, we use the backend_info entry_point (in the pyproject.toml file). The get_info function is used to parse the docstrings of all the algorithms in nx-parallel and display the nx-parallel specific documentation on the NetworkX's main docs, in the "Additional Backend implementations" box, as shown in the screenshot below.

nx-parallel follows sphinx docstring guidelines for writing docstrings. But, while extracting the docstring to display on the main networkx docs, only the first paragraph of the function's description and the first paragraph of each parameter's description is extracted and displayed. So, make sure to include all the necessary information in the first paragraphs itself. And you only need to include the additional backend parameters in the Parameters section and not all the parameters. Also, it is recommended to include a link to the networkx function's documentation page in the docstring, at the end of the function description.

Here is an example of how the docstrings should be formatted in nx-parallel:

def parallel_func(G, nx_arg, additional_backend_arg_1, additional_backend_arg_2=None):
    """The parallel computation is implemented by dividing the
    nodes into chunks and ..... [ONLY THIS PARAGRAPH WILL BE DISPLAYED ON THE MAIN NETWORKX DOCS]

    Some more additional information about the function.

    networkx.func : <link to the function's networkx docs page>

    Parameters
    ----------
    additional_backend_arg_1 : int or float
        [YOU CAN HAVE MULTIPLE PARAGRAPHS BUT ONLY THE FIRST PARAGRAPH WILL BE DISPLAYED ON THE MAIN NETWORKX DOCS]

    additional_backend_arg_2 : None or str (default=None)
        ....
    """

Chunking

In parallel computing, "chunking" refers to dividing a large task into smaller, more manageable chunks that can be processed simultaneously by multiple computing units, such as CPU cores or distributed computing nodes. It's like breaking down a big task into smaller pieces so that multiple workers can work on different pieces at the same time, and in the case of nx-parallel, this usually speeds up the overall process.

Chunking in nx-parallel defaults to slicing the input into n_jobs chunks (n_jobs=-1 means using all CPU cores; see chunk.py). To know how to change config options like n_jobs, see Config.md. A user can override chunking by passing a custom function to the get_chunks kwarg. When adding a new algorithm, you may modify this default chunking behavior if needed (e.g. PR#33).

General guidelines on adding a new algorithm

• To get started with adding a new algorithm, you can refer to the existing implementations in nx-parallel and also refer to the joblib's documentation on embarrassingly parallel for loops.
• The algorithm that you are considering to add to nx-parallel should be in the main networkx repository and it should have the _dispatchable decorator. If not, you can consider adding a sequential implementation in networkx first.
• check-list for adding a new function:
  • Add the parallel implementation(make sure API doesn't break), the file structure should be the same as that in networkx.
  • Include the get_chunks additional parameter. Currently, all algorithms in nx-parallel offer the user to pass their own custom chunks. Unless it is impossible to chunk, please do include this additional parameter.
  • add the function to the ALGORITHMS list in interface.py. Take care of the name parameter in _dispatchable for the algorithms with same name but different implementations. The name parameter is used distinguish such algorithms in a single namespace. (ref. docs)
  • update the __init__.py files accordingly
  • docstring following the above format
  • add additional test, if needed. The smoke tests for the additional parameter get_chunks are done here together for all the algorithms.
  • run the timing script to get the performance heatmap.
  • add benchmark(s) for the new function(ref. the README in benchmarks folder for more details)

Happy contributing! 🎉