User's PMIx call fails after MPI_Init + UCX/HCOLL

[angainor]

@artpol84 FYI As discussed with @rhc54 in an issue reported in the PMIx repo, it seems something in the OpenMPI runtime 'breaks' the PMIx infrastructure so that it is not possible to distribute user's keys if a call to PMIx_Set + PMIx_Commit is made after the MPI_Init call. That is, PMIx_Get fails on the clients with error -46. If the user's code sets the custom key before MPI_Init, then the code works as expected.

What's puzzling is that I only observe this problem when the UCX pml and HCOLL are enabled. I compile the code attached at the end of this post against OMPI master + it's internal PMIx, but I see the same behavior for OMPI 4.0.1 + PMIx 2.1.4:

$ mpirun -map-by node ./pmixtest
PMIx initialized
PMIx_Put on test-key
PMIx_Put on test-key
Tue Sep 17 14:47:15 2019 ERROR: pmixtest.c:55  Client ns 39583745 rank 1: PMIx_Get test-key: -46
Tue Sep 17 14:47:15 2019 ERROR: pmixtest.c:55  Client ns 39583745 rank 0: PMIx_Get test-key: -46

If I turn off UCX and HCOLL, things work as expected:

$ mpirun -mca pml ^ucx -mca coll_hcoll_enable 0 -map-by node ./pmixtest
PMIx initialized
PMIx_Put on test-key
PMIx_Put on test-key
PMIx_get test-key returned 256 bytes
0: obtained data "rank 1"
PMIx_get test-key returned 256 bytes
1: obtained data "rank 0"
PMIx finalized

Here is the reproducing code. To compile one needs to pass include and link path to the PMIx installation used by OpenMPI. I'd appreciate any insight.

#include <stdio.h>
#include <pmix.h>
#include <mpi.h>

static pmix_proc_t allproc = {};
static pmix_proc_t myproc = {};

#define ERR(msg, ...)							\
    do {								\
	time_t tm = time(NULL);						\
	char *stm = ctime(&tm);						\
	stm[strlen(stm)-1] = 0;						\
	fprintf(stderr, "%s ERROR: %s:%d  " msg "\n", stm, __FILE__, __LINE__, ## __VA_ARGS__); \
	exit(1);							\
    } while(0);


int pmi_set_string(const char *key, void *data, size_t size)
{
    int rc;
    pmix_value_t value;

    PMIX_VALUE_CONSTRUCT(&value);
    value.type = PMIX_BYTE_OBJECT;
    value.data.bo.bytes = data;
    value.data.bo.size  = size;
    if (PMIX_SUCCESS != (rc = PMIx_Put(PMIX_GLOBAL, key, &value))) {
        ERR("Client ns %s rank %d: PMIx_Put failed: %d\n", myproc.nspace, myproc.rank, rc);
    }

    if (PMIX_SUCCESS != (rc = PMIx_Commit())) {
        ERR("Client ns %s rank %d: PMIx_Commit failed: %d\n", myproc.nspace, myproc.rank, rc);
    }

    /* protect the data */
    value.data.bo.bytes = NULL;
    value.data.bo.size  = 0;
    PMIX_VALUE_DESTRUCT(&value);
    printf("PMIx_Put on %s\n", key);


    return 0;
}

int pmi_get_string(uint32_t peer_rank, const char *key, void **data_out, size_t *data_size_out)
{
    int rc;
    pmix_proc_t proc;
    pmix_value_t *pvalue;

    PMIX_PROC_CONSTRUCT(&proc);
    (void)strncpy(proc.nspace, myproc.nspace, PMIX_MAX_NSLEN);
    proc.rank = peer_rank;
    if (PMIX_SUCCESS != (rc = PMIx_Get(&proc, key, NULL, 0, &pvalue))) {
        ERR("Client ns %s rank %d: PMIx_Get %s: %d\n", myproc.nspace, myproc.rank, key, rc);
    }
    if(pvalue->type != PMIX_BYTE_OBJECT){
        ERR("Client ns %s rank %d: PMIx_Get %s: got wrong data type\n", myproc.nspace, myproc.rank, key);
    }
    *data_out = pvalue->data.bo.bytes;
    *data_size_out = pvalue->data.bo.size;

    /* protect the data */
    pvalue->data.bo.bytes = NULL;
    pvalue->data.bo.size = 0;
    PMIX_VALUE_RELEASE(pvalue);
    PMIX_PROC_DESTRUCT(&proc);

    printf("PMIx_get %s returned %zi bytes\n", key, data_size_out[0]);

    return 0;
}

int main(int argc, char *argv[])
{
    char data[256] = {};
    char *data_out;
    size_t size_out;
    int rc;
    pmix_value_t *pvalue;

    // if MPI_Init is executed before PMIx_set, PMIx_Get fails with -46
    MPI_Init(&argc, &argv);

    if (PMIX_SUCCESS != (rc = PMIx_Init(&myproc, NULL, 0))) {
	ERR("PMIx_Init failed");
        exit(1);
    }
    if(myproc.rank == 0) printf("PMIx initialized\n");

    sprintf(data, "rank %d", myproc.rank);
    pmi_set_string("test-key", data, 256);

    // if MPI_Init is executed after PMIx_set, PMIx_Get works fine
    // MPI_Init(&argc, &argv);
    
    pmi_get_string((myproc.rank+1)%2, "test-key", (void**)&data_out, &size_out);
    printf("%d: obtained data \"%s\"\n", myproc.rank, data_out);

    if (PMIX_SUCCESS != (rc = PMIx_Finalize(NULL, 0))) {
        ERR("Client ns %s rank %d:PMIx_Finalize failed: %d\n", myproc.nspace, myproc.rank, rc);
    }
    if(myproc.rank == 0) printf("PMIx finalized\n");

    MPI_Finalize();
}

[artpol84]

@angainor thank you for the information.
@karasevb, please take a look.

[artpol84]

@jsquyres, for some reasons I can't assign @karasevb. Can you help please?

[artpol84]

Thanks @jsquyres !

[jsquyres]

@artpol84 @karasevb @jladd-mlnx Ping.

[bosilca]

I have noticed some weird issues with what I suppose is the memory allocator by the UCX PML. In my case the issue arise if I dlopen cuda (via cublas) after an MPI_Init where the UCX PML is loaded but not enabled (so it is dlclosed). The issue manifest as a segfault deep inside dlopen while allocating some string. If I prevent the UCX PML from loading, then the code works just fine.

[artpol84]

@jsquyres @angainor sorry for the delay.
We will try to take it ASAP.

@bosilca thank you for the note.

@karasevb please, remind me the UCX memory hooks setting that we can try.

[karasevb]

@angainor could you try to run with added -mca patcher ^overwrite?

[angainor]

@karasevb unfortunately, still the same:

$ mpirun -x LD_LIBRARY_PATH -np 2 -H c3-3,c3-4 -mca patcher ^overwrite ./out 
PMIx initialized
PMIx_Put on test-key
PMIx_Put on test-key
Fri Dec 13 08:05:37 2019 ERROR: pmitest.c:55  Client ns 2865364993 rank 1: PMIx_Get test-key: -46

Fri Dec 13 08:05:37 2019 ERROR: pmitest.c:55  Client ns 2865364993 rank 0: PMIx_Get test-key: -46

[artpol84]

@bosilca disabling patcher may help in your case.

[artpol84]

@angainor I'll try to repro next week.

[artpol84]

@angainor, I tried to reproduce with the following results:

First, I had to fix the return value, without the fix I was getting an error messages about procs exiting with non-zero status:

--- repro.c	2019-12-17 11:23:36.216701105 -0800
+++ repro_new.c	2019-12-17 11:34:33.000156999 -0800
@@ -96,5 +96,6 @@
     }
     if(myproc.rank == 0) printf("PMIx finalized\n");
 
+    return 0;
     // MPI_Finalize();
 }

I was using HPCX that is based on ompi-v4.0.x (af04a9d) and running on 2 nodes:

$ module load hpcx-gcc && mpirun --map-by ppr:1:node hostname                                                                                                                                                                                 
cn1
cn2

For me the patched reproducer is passing successfully (note that I'm explicitly requesting UCX PML):

$ module load hpcx-gcc && mpirun --map-by ppr:1:node --mca pml ucx ./repro                                                                                                                                                                    
PMIx_Put on ghex-rank-address
PMIx_Commit on ghex-rank-address
PMIx initialized
PMIx_Put on ghex-rank-address
PMIx_Commit on ghex-rank-address
PMIx_get ghex-rank-address returned 256 bytes
PMIx_get ghex-rank-address returned 256 bytes
PMIx finalized

Is there anything I am missing?

P.S. in order to build the repro I had to explicitly link it with OMPI's PMIx:

$ module load hpcx-gcc && mpicc -o repro repro.c  -L <HPCX_PATH>/ompi/lib/ -lpmix

otherwise, I was getting an error:

$ module load hpcx-gcc && mpicc -o repro repro.c                                                                                                                                                                                              
/tmp/cchpYK8Q.o: In function `pmi_set_string':
repro.c:(.text+0xfbe): undefined reference to `OPAL_MCA_PMIX3X_PMIx_Put'
repro.c:(.text+0x1082): undefined reference to `OPAL_MCA_PMIX3X_PMIx_Commit'
/tmp/cchpYK8Q.o: In function `pmi_get_string':
repro.c:(.text+0x11b7): undefined reference to `OPAL_MCA_PMIX3X_PMIx_Get'
/tmp/cchpYK8Q.o: In function `main':
repro.c:(.text+0x13bc): undefined reference to `OPAL_MCA_PMIX3X_PMIx_Init'
repro.c:(.text+0x14bc): undefined reference to `OPAL_MCA_PMIX3X_PMIx_Finalize'
collect2: error: ld returned 1 exit status

How are you building your repro? Can it be that a different PMIx version is used? It doesn't look like that, otherwise, the presence or absence of UCX wouldn't matter.

I noticed that you provide LD_LIBRARY_PATH in your command invocation, why do you need it?

$ mpirun -x LD_LIBRARY_PATH -np 2 -H c3-3,c3-4 -mca patcher ^overwrite ./out 

[angainor]

@artpol84 I'll be able to look at it in more detail tomorrow, but some thoughts: You have to run on different compute nodes. If you start the ranks on the same node, it works. I was building OpenMPI against an external PMIx version, which was also used to link the reproducing program. I did not use the internal PMIx, also for the reasons you've described.

LD_LIBRARY_PATH was needed on my system when I was running outside of slurm, simply specifying two compute nodes as a commandline arg to mpirun. If I do not do that, the environment is not preserved and the PMIx libs (and OpenMPI libs) are not available.

[artpol84]

Thanks, @angainor

If you start the ranks on the same node, it works.

Yeah, I noticed that and that's why above I explicitly said I was using 2 nodes:

I was using HPCX that is based on ompi-v4.0.x (af04a9d) and running on 2 nodes:

$ module load hpcx-gcc && mpirun --map-by ppr:1:node hostname                                                                                                                                                                                 
cn1
cn2

---

I was building OpenMPI against an external PMIx version, which was also used to link the reproducing program.

Ok, this could be a key difference. I'll try to build my own OMPI and link it against PMIx.

LD_LIBRARY_PATH was needed on my system when I was running outside of slurm, simply specifying two compute nodes as a commandline arg to mpirun. If I do not do that, the environment is not preserved and the PMIx libs (and OpenMPI libs) are not available.

I see, I probably should try to run without Slurm as well.