We need to catch tensorflow code's exception and do somethings.

[taeguk]

For example, in case of gpu's memory is lack to run,
we catch tensorflow.python.framework.errors.ResourceExhaustedError and do somethigs.
"Somethings" can be 'report resorce exhausted error to executor(maybe worker) through stderr'.
But, we need to discuss about it.

[ovekyc]

Thats terrible
But i just wonder that is that a 'research-ml's issue?
Im not sure cz task running is not considered in this part

Validation what you said is about the model? or data?
if model, we have to consider about the valudation but not in case of data,

[ovekyc]

Even though it is about model, Im not sure about ur question
In the ML side, it doent know running env

[taeguk]

I'm sorry. I described this issue in wrong way.
I modified the issue in correct way.
Please check one more time. You misunderstand my issue now.

[taeguk]

Not real implementation, but just draft below..

try:
    # tensorflow codes
except <Some Exceptions>:
    # do somethings (like reporting to worker through stderr)

[ovekyc]

No sorry bro lol
I just wondered about that
I still cant flow you Please give me more detail

[taeguk]

@ovekyc let's talk about this issue in membership soon.