Poor thread scaling in multi-group mode

[jtramm]

The multigroup mode for OpenMC currently does not scale very well when run using shared memory threading (via OpenMP) past 10 threads or so. For example, on this 2D C5G7 input deck, when I run it on a dual socket Intel Xeon platinum 8180M node with 56 cores and 112 threads, I get the following behavior:

MPI Ranks	OpenMP Threads per Rank	Inactive [particles/sec]	Active [particles/sec]
1	1	39,510	23,165
1	4	115,423	64,612
1	8	154,411	67,412
1	16	103,526	59,432
1	56	96,408	56,838
1	112	134,937	80,139
56	1	868,351	498,800
56	2	1,102,130	754,118
28	4	1,270,910	673,048
2	56	233,411	142,513

These results were generated with gcc 10.2.0 and OpenMPI 2.1.6. I also tried out some other compilers (namely, intel and llvm) and found the results followed the same trend. So, I don't think this can be chalked up to a poor compiler implementation, especially as continuous energy MC scales well with OpenMP on this node.

One initial problem that comes to mind is that the tally space is fairly small for this problem (a 51 x 51 fission rate mesh), which may lead to a lot of memory contention when doing tallies. However, the poor performance seems to affect inactive batches equally, so there must be a more fundamental problem.

My guess is that there is a false sharing issue specific to the multi-group MC mode. Hopefully it can be easily fixed once spotted. I may try to hunt this down at some point in the future but thought I would open the issue now in the interim.

[jtramm]

My current belief is that the poor scaling is caused by false sharing due to a thread-specific cacheing structure in the get_xs() function:

openmc/src/mgxs.cpp

Lines 428 to 439 in 33ead74

	double Mgxs::get_xs(
	MgxsType xstype, int gin, const int* gout, const double* mu, const int* dg)
	{
	// This method assumes that the temperature and angle indices are set
	#ifdef _OPENMP
	int tid = omp_get_thread_num();
	XsData* xs_t = &xs[cache[tid].t];
	int a = cache[tid].a;
	#else
	XsData* xs_t = &xs[cache[0].t];
	int a = cache[0].a;
	#endif

We should test if we would be better off without the cache entirely. If it is still advantageous to have this data structure, we can remedy the issue by padding the thread specific allocations or by making it a formal thread private data structure.

[jtramm]

Before attempting a more involved restructuring (or potential removal) of the cacheing logic, I did some experimentation by padding the cache structure:

struct CacheData {
  double sqrtkT; // last temperature corresponding to t
  int t;         // temperature index
  int a;         // angle index
  // last angle that corresponds to a
  double u;
  double v;
  double w;
  char padding[24]; // Padding to make struct 64 bytes wide
};

but this had no effect on the dual socket 8180M (56c/112t total) system I was running on, with scaling still limited to about 4x that of serial. I also did some performance analysis and found that vtune was reporting that pretty much all the major functions were just generally slow, even ones that didn't seem to be looking up cross sections. I also experimented with several compilers to no avail.

As a sanity check, I then ran on two other systems:

• 2X Xeon Gold 6336Y, with 48c/96t total
• 2X AMD EPYC 7742, with 128c/256t total

and found that scaling was great and there were no problems with MGMC performance at all. These are all big dual socket systems, and I used the same gcc 7.5.0 binary on all three, so nothing comes to mind as to what might be causing the weird scaling dropoff in the 8180M and not the others.

It would still be good to do more analysis here to figure out what is causing the problem on that system, but for now at least it is good to know the problem is not universal across architectures.