Hints serverside

[Stellatsuu]

Closes #31

Hints are now handled by the server.

How to create/use hints

In CWL only:

cwlVersion: v1.2
# What type of CWL process we have in this document: ComandLineTool or Workflow.
class: CommandLineTool

# The inputs for this process: none.
inputs: []
# The outputs for this process: none.
outputs: []

hints:
    - dirac:ExecutionHooks:
         ...
    - dirac:Scheduling:
         ...
baseCommand: ["echo", "Hello World"]

Using an external file:

In CWL:

cwlVersion: v1.2
# What type of CWL process we have in this document: ComandLineTool or Workflow.
class: CommandLineTool

# The inputs for this process: none.
inputs: []
# The outputs for this process: none.
outputs: []

hints:
      $import: "my_hints.yaml"

baseCommand: ["echo", "Hello World"]

In my_hints.yaml:

- class: dirac:ExecutionHooks
  hook_plugin: ...
  group_size:
    ...
  configuration:
    ...
...

[aldbr]

Thank you @Stellatsuu, I think you are on the right track!

Some production tests are broken from this (do not merge now), I can't see why, maybe the new $import in cwl worflows I'm not sure of.

I haven't checked carefully but it is likely coming from https://github.com/aldbr/dirac-cwl-proto/blob/e97b83d0d65a46042fa9b9fbb725b5bbed498665/src/dirac_cwl_proto/production/__init__.py#L194-L198

Can you try to adjust all the SubmissionModels? If you adjust them all, then may be the errors will go away.

Should we describe the execution-hooks like this in the *.yaml (with dirac: prefix)?

I think what you did is fine!

I'm not sure of the call to TransformationExecutionHooksHint.from_cwl(transformation.task), I had some struggle with Mypi because it takes it as an ExecutionHooksHint. I don't think we have to rewrite from_cwl but right now I don't see a workaround for that?
I also tried (in local) calling extract_dirac_hints and modifying it to match TransformationExecutionHooksHint class, same problem.

See my comment below about Self 🙂

Note: may be TransformationSubmissionModel and ProductionSubmissionModel will be very similar in the end, let's see if we can/should refactor them in the end.

[Stellatsuu]

Thanks @aldbr for the quick review ! I will check all that 😄

[Stellatsuu]

What does it mean to override here? It doesn't override anything anywhere, should it?

[aldbr]

Thanks @Stellatsuu, I just have a few comments 🙂