[CUDA] Make CudaEvent work with multi-device

[zcbenz]

Most APIs of CUDA event have requirements on being used on the same device with the device that the event was created on, this PR makes sure the CudaEvent class set current device before invoking the APIs.

There is also an issue that the CudaEvent could be destroyed in worker thread when used for waiting on a GPU work in CPU stream, which would cause race conditions when accessing event cache, this PR uses a simple strategy by skipping the cache when event is created or destroyed on worker threads, so we don't have to add thread-safety overhead for those rare cases.

Note that this PR does not guarantee multi-thread or multi-device safety, this is mostly a defensive change.

[angeloskath]

This looks great to me!

I am missing something here for sure, when will events be created and destroyed in a worker thread 🤔, I thought they would be created and destroyed in the eval loop. I guess that this is precisely your point, they will only be created and destroyed in the eval loop but in case they are created and destroyed somewhere else then skip the cache ...

[zcbenz]

when will events be created and destroyed in a worker thread

Waiting a CPU stream is implemented by copying the event to the CPU thread, so the event could be destroyed there:

mlx/mlx/backend/cuda/event.cu

Lines 91 to 97 in 7a6adda

	void wait(Stream s) {
	if (s.device == mlx::core::Device::cpu) {
	scheduler::enqueue(s, [*this]() mutable {
	check_recorded();
	event_->wait();
	});
	} else {

Another case is user may run eval() in workers, I did that to implement await eval() in node-mlx, and I also saw some users doing it: #2067