ENH : Added get_chunks to betweenness_centrality and create_iterables in utils

[Schefflera-Arboricola]

Please do "Squash and merge" while merging this PR

1. Added get_chunks kwarg to betweenness_centrality :
   • description: A user-defined function to create node_chunks(instead of how we currently do it i.e. by creating partitions on the G.nodes; if get_chunks is "chunks"(default) then this is executed). To understand the motivation for this see the additional pytest added
   • also added additional tests(specific to nx-parallel) to test additional kwargs
2. added create_iterables(ref. Pr [WIP]: Refactor-- consolidate and simplify #7)

[dschult]

I like this approach. It is general (and works for nodes, edges and isolated_nodes) and allows the user to override the choice of node chunks. I'm not sure how much difference it makes in betweenness centrality to be able to split up the "hard" nodes, but in general it will make a difference if you can split them up. And betweenness centrality could have some nodes take a lot longer than others. So this is a good one to start with.

I don't know of a way to test that this is working. Maybe you should have a test that computes it using the built-in default chunking and also computes it with a user-defined method. They could be "smoke-tests" to see if using that option breaks it so badly that smoke starts coming out. If smoke doesn't come out, it might still be wrong. But you have at least tested that there are no fundamental problems. Have you thought about tests for this?

[dschult]

This PR has style issues. Next time you get to it, I think it is just one line with a space at the end of the line.

[dschult]

It might be assuming too much to assert that the time should be less when chunking is used.
Of course, asserting that could allow you to track down why it is slower, I suppose.

But should it always be slower? How might get_chunks slow it down?

[Schefflera-Arboricola]

It might be assuming too much to assert that the time should be less when chunking is used. Of course, asserting that could allow you to track down why it is slower, I suppose.

But should it always be slower? How might get_chunks slow it down?

here, I have compared the time for running the parallel implementation of betweenness_centrality with the default chunking and with a custom chunking(done based on the edge_betweenness_centrality values). And I locally ran it on 8 cores and in github workflow test it runs on 2 cores, and for some machines, this test is failing and for some it is passing.

This could be a nice example of why get_chunk might be useful as an option, that would let a user do custom chunking when they have a really big graph. But, I don't think we should put it as a test because it takes a lot of time and it also doesn't pass for all machines.

[dschult]

That makes good sense to me.
Timing is dependent on so many things... Hard to do well...
Nice to have parameters to tweak for each situation and get_chunk is a good one to make available. :)

[dschult]

I think this is ready to be merged. So I'm approving it.

Are there any more changes planned for this PR?

[Schefflera-Arboricola]

Are there any more changes planned for this PR?
No, I think it's ready

Thanks @dschult !