Question about mca parameter passing

[wckzhang]

Using 20ee752 for PRRTE.

I've been messing around with the ompi schizo component in order to add some warning/aborts around the old --mca mca_base_param_files parameters. However it looks like the arg is dropped before we get to the mca parameter parsing. See my garbage parameters:

[ec2-user@ip-10-0-0-28 runtime]$ ~/tmp/ompi/install/bin/mpirun --np 2 -N 1 --mca mca_garbage_param 1 --mca opal_mca_garbage_param 1 --mca garbage_param 1 --hostfile ~/hostfile /bin/true
mca p1 is: opal_mca_garbage_param
mca p1 is: garbage_param

It looks like parameters prefixed with "mca" are being dropped. Do you have any idea what's happening here or if this is intended behavior?

[rhc54]

Don't look at me - I haven't looked at the ompi schizo component in a very long time. Command line parsing starts with your parse_cli function.

[wckzhang]

It looks like it's because the options becomes --pmixmca mca_garbage_param 1
It looks like the ompi_schizo component does not look at this prefix (pmixmca), I can add another loop for it

[wckzhang]

So it turns out that --mca mca_base_param_files is not ignored, it was just that pmixmca prefixes were all ignored. The relevant code is all in place

[wckzhang]

Actually let me ask a clarifying question: Should the parameter be prefixed with pmixmca and should this be read by the ompischizo component? (This prefixing seems to happen somewhere before parse_cli is called)

[rhc54]

You have a problem. PRRTE and PMIx share a common MCA "base", and so any generic MCA param on the cmd line gets interpreted as belonging to PMIx. Those get converted at the very beginning of prte as we have to process the PMIx and PRRTE params prior to calling their init functions.

Problem you have is that this means all generic MCA params that begin with mca are going to be converted to PMIx. Only solution I can see is to mandate that PMIx MCA params that belong to the MCA "base" must be prefixed with pmix or prte on the cmd line. I'll have to think about that as this feels more like an OMPI problem for failing to prefix their base params, but I suspect your side of the house will strenuously object to having to change.

[rhc54]

Tentative proposal: we reject all generic --mca params. All params must be prefixed by their intended project. It's the only solution I can come up with that doesn't unfairly penalize one project over another.

Alternative option: apply that only to params that start with mca_ as they are ambiguous.

[rhc54]

@jsquyres @bwbarrett Any thoughts? You two generally care the most about such things.

[rhc54]

Alternative option: we reject all generic params that start with mca_ as they are ambiguous.

In the absence of any feedback, I'm going to implement this option. Please note that this applies to more than just the mca_base_param_files option.

[naughtont3]

@rhc54 There was discussion on this point during weekly ompi-dev call, but several key people were out due to travel/conflicts. So not sure if there was any consensus. I'll defer to @wckzhang & others , but just wanted to chime in just in case.

[wckzhang]

Yes, I've been trying to get @jsquyres or @bwbarrett to chime in but they've been busy. I've seen push back from George and Edgar about the change as it would break lots of user scripts. George proposed everyone just check every argument, and there was some debate about why pmix prefixed an --mca mca_ as --pmixmca, we could leave it generic and have each component check to see if they can handle it.

I didn't want to make a decision either way without more consensus from our community, but I have not forgotten about this issue.

[rhc54]

about why pmix prefixed an --mca mca_ as --pmixmca, we could leave it generic and have each component check to see if they can handle it.

I'm afraid that isn't possible. We only change it for the MCA base params, and you cannot have those generally apply as the various layers can't discern which layer is the intended target. It winds up generating a great deal of confusion.

I'm probably not available this coming Tues, but I could participate on the OMPI call the week after if it would help.

[jsquyres]

Sorry for the huge delay in replying to this; my bandwidth has been sucked up elsewhere. 😦

Hey @rhc54 Could we just add mca_base_param_files to the list of prefixes that are examined by OMPI schizo to know that this parameter belongs to OMPI? We already pass in a list of all of Open MPI's frameworks via the OMPI_MCA_PREFIXES env variable, which is then examined by

prrte/src/mca/schizo/ompi/schizo_ompi.c

Lines 1448 to 1481 in aa498b9

	static void setup_ompi_frameworks(void)
	{
	if (ompi_frameworks_setup) {
	return;
	}
	ompi_frameworks_setup = true;
	char *env = getenv("OMPI_MCA_PREFIXES");
	if (NULL == env) {
	return;
	}
	// If we found the env variable, it will be a comma-delimited list
	// of values. Split it into an argv-style array.
	char **tmp = PMIX_ARGV_SPLIT_COMPAT(env, ',');
	if (NULL != tmp) {
	ompi_frameworks = tmp;
	}
	}
	static bool check_generic(char *p1)
	{
	setup_ompi_frameworks();
	/* See if the parameter we were passed belongs to one of the OMPI
	frameworks or prefixes */
	for (int j = 0; NULL != ompi_frameworks[j]; j++) {
	if (0 == strncmp(p1, ompi_frameworks[j], strlen(ompi_frameworks[j]))) {
	return true;
	}
	}
	return false;
	}

Having OMPI add mca_base_param_files to the comma-delimited list would effectively disambiguate that name and make it "owned" by OMPI (we can/probably should also do the same thing to mca_base_env_list, which is already special-cased -- and potentially any other mca_-prefixed OMPI variable...?).

However, I notice that by the time OMPI schizo is invoked, any --mca CLI tokens have been converted to --pmixmca, making at least the current logic ineffective

prrte/src/mca/schizo/ompi/schizo_ompi.c

Line 1617 in aa498b9

if (NULL != (opt = pmix_cmd_line_get_param(results, "mca"))) {

[rhc54]

The problem is that PMIx has a duplicate of every MCA base parameter, so you cannot disambiguate them. Root problem is that the MCA base doesn't prefix its params by project, and we now have multiple projects that contain an MCA base.

We do look for frameworks and convert them - no ambiguity there as we already ensure that the params are properly prefixed with the framework/component name. It is just the MCA base that is the problem, and only when the user doesn't prefix the --mca cmd line option.

One possible solution is to say that non-prefixed --mca shall be reserved for OMPI. I can kinda wrap my head around it from the standpoint that direct PRRTE users are outnumbered by pre-existing OMPI users (I don't worry about PMIx users as we are talking about prterun, a PRRTE tool here). Just concerned that it creates more confusion as you can use a non-prefixed cmd line option for PMIx/PRRTE frameworks/components - but now we say you can't for their MCA base.

Probably no ideal solution 🤷‍♂️

[bosilca]

Depends on what perspective you are looking from. From a user perspective what is not ideal is to be forced to learn the internal naming scheme of a software I am using. I am passing an argument to OMPI, and everything behind shall obey it. If PRRTE wants to have their own parameter naming other than --mca their problem, but when I provide a file with the MCA parameters it shall be read and used by the entire software stack.