Test failure in dataflow/gpu-workers

[tmatsuo]

Example build

Probably a flake, restarting the build to see.

[tmatsuo]

It seems like it always fails.

@davidcavazos Do you have any idea why it fails?

[tmatsuo]

I see a dlerror in the log:

2021-01-15 22:29:10.524176: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-01-15 22:29:10.524235: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

[tmatsuo]

I found a more important error:

Startup of the worker pool in zone us-central1-a failed to bring up any of the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone 'projects/python-docs-samples-tests/zones/us-central1-a' does not have enough resources available to fulfill the request.  '(resource type:compute)'.

[tmatsuo]

There were leaked GCE instances. I deleted them and I think we're good to go.

[tmatsuo]

Now it's constantly passing.

[davidcavazos]

Thanks. Sorry, I missed these notifications. The dynamic library warning is normal. It's trying to load GPUs, but the VM that launches the job doesn't have GPUs, but when the workers run the pipeline they should work as expected.

About the quota issue, Valentyn also mentioned that we could change to us-central1-f and that could solve that. I applied that in #5275