Performant OpenMP+MPI mini apps/arguments?

Hey all,

This might be a long shot, but my group is working on research to improve MVAPICH2's integration with OpenMP and we are having a ridiculously tough time finding any mini app that performs better with hybrid OpenMP+MPI over pure MPI.

Our system has 2 sockets per node and 14 cores per socket. Theoretically, a few MPI processes per node with OpenMP threads doing the work using shared memory should perform better than with just MPI processes, but that's far from what we've been seeing. Some runtimes using CloverLeaf are below.

1 MPI x 28 threads: 7.15 seconds

28 MPI x 1 thread: 5.74 seconds

4 MPI x 7 threads: 7.60 seconds

7 MPI x 4 threads: 6.32 seconds

This pattern persists for every mini app that we have used, although miniMD is the only other app that we have tested almost as thoroughly as CloverLeaf.

I have played around with KMP_AFFINITY but honestly it seems like the defaults are just as well as what I have ran into using granularity=core and compact.

Example run command:

mpiexec -ppn 1 -genv OMP_NUM_THREADS 28 ./clover_leaf  

Any suggestions on what else to try to find some decent hybrid times? I have been working on this all summer with very little progress, so any advice is greatly appreciated!