Replacing or updating distributed matrix on GPUs

[mawxcarroll]

Hello,

I have a calculation that is working very well using Ginkgo's distributed matrix to execute a Runge-Kutta algorithm across 4 GPUs. It's really just a bunch of SpMV operations, and it's incredibly fast and has extended the region of parameter space I can explore in my research a great deal -- so thanks for that!

My question is about updating or replacing the matrix on the GPU. The first time I fill the matrix, I do the following:

const auto num_rows = N;
auto partition = gko::share (part_type::build_from_global_size_uniform (exec->get_master (), comm.size (), static_cast<GlobalIndexType> (num_rows)));
gko::matrix_data<ValueType, GlobalIndexType> Hdata;
Hdata.size = { num_rows, num_rows };

I then add the nonzero values with Hdata.emplace_back(). Once the matrix is filled, I move it over to the GPU with:

auto Hhost = gko::share (dist_mtx::create (exec->get_master (), comm));
Hhost->read_distributed (Hdata, partition);
auto H = gko::share (dist_mtx::create (exec, comm));
H->copy_from (Hhost);

As I said, this is working beautifully. Now, to replace the matrix on the GPU with the new values, will the following work?

Hdata.nonzeros.clear()
/* calculate new nonzero elements and add them with Hdata.nonzeros.emplace_back() */
Hhost->read_distributed (Hdata, partition);
H->copy_from (Hhost);

Or do I need to somehow free the memory on the GPUs first?

Thanks for the help!

Cheers,
tom

[MarcelKoch]

Yes that approach will work.
I also want to add, that the intermediate host matrix is unnecessary. You can call read_distributed directly on the device matrix, and pass it the matrix_data. (It will get copied to the device automatically.)