Interpret the hints from the server only?

[aldbr]

The client can just add them to the CWL if they are separated (pack)

---

• Also, in the SubmissionModels, I have the feeling it would be cleaner to read the hints just once, in the service. They would come either integrated in the workflow or as a separate file that would be merged within the workflow in the client code (pack).
• So the SubmissionModels would not carry the hints: they would be in the CWL. The advantage is that a given user can provide a CWL through the REST interface in a more "standard" way, not having to specify the hints separately in the JSON blob.
• And then they would be read by the services (as the clients are doing right now).

What's your opinion?

Originally posted by @aldbr in #23 (comment)

[Stellatsuu]

@aldbr If I understand correctly, you want to remove those from the TransformationSubmissionModel:
https://github.com/aldbr/dirac-cwl-proto/blob/e97b83d0d65a46042fa9b9fbb725b5bbed498665/src/dirac_cwl_proto/submission_models.py#L76-L77

And then, in the client, it would only be something like this:

 transformation = TransformationSubmissionModel(task=task)

And in the router, when asking for the hints, we'd call:
transformation_execution_hooks, transformation_scheduling = extract_dirac_hints(transformation.task)
Instead of:
transformation.execution_hooks and transformation.scheduling?

Also, I don't quite understand your first comment, mostly the pack part:

Also, in the SubmissionModels, I have the feeling it would be cleaner to read the hints just once, in the service. They would come either integrated in the workflow or as a separate file that would be merged within the workflow in the client code (pack).

[Stellatsuu]

Does the pack part means that we combine those hints data to the ones we already have in the CWL?:
https://github.com/aldbr/dirac-cwl-proto/blob/e97b83d0d65a46042fa9b9fbb725b5bbed498665/src/dirac_cwl_proto/transformation/__init__.py#L77-L85

[aldbr]

@aldbr If I understand correctly, you want to remove those from the TransformationSubmissionModel:

dirac-cwl-proto/src/dirac_cwl_proto/submission_models.py

Lines 76 to 77 in e97b83d

execution_hooks: TransformationExecutionHooksHint
scheduling: SchedulingHint
And then, in the client, it would only be something like this:

 transformation = TransformationSubmissionModel(task=task)

And in the router, when asking for the hints, we'd call: transformation_execution_hooks, transformation_scheduling = extract_dirac_hints(transformation.task) Instead of: transformation.execution_hooks and transformation.scheduling?

Yes exactly!
For context, before recent changes the Dirac-specific hints (like SchedulingHint and ExecutionHooksHint) were expressed outside of the CWL workflow, this is why they are part of the pydantic models and separated from the task.
Now, they are meant to be part of the CWL, and so we don't want to let the users expressing them both within the CWL workflow and withing the JSON blob sent to the server (outside of the CWL).
We just haven't had the time to remove them from the pydantic models.

Is it clear? 😅

Does the pack part means that we combine those hints data to the ones we already have in the CWL?:

Natively. CWL can combine multiple related files into a single document.

• We already use this principle to pack multiple subworkflows into a single large workflow as you can see here for instance: https://github.com/aldbr/dirac-cwl-proto/blob/e97b83d0d65a46042fa9b9fbb725b5bbed498665/test/workflows/mandelbrot/description.cwl#L74
• We don't do it for the hints, but we should probably use the same mechanism (e.g. hints can probably be referenced using $import https://www.commonwl.org/v1.0/SchemaSalad.html#Import).

In this way, the CLI would just take a CWL workflow in parameter, and the user would add the hints directly in the CWL workflow, or in another document referenced in the CWL workflow.

[Stellatsuu]

It is much clearer now, thank you ! I will take a look at that tomorrow :)