Add support for Intel Gaudi Backend

[jerome-habana]

Added support for intel gaudi backend based on new interfaces defined in #40286

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jjyao]

Lg

[jjyao]

Have you tested this on the machine with Gaudi?

[jjyao]

Last comment

[jjyao]

Lint failure:

Fri Oct 27 12:06:51 UTC 2023 Flake8....
--
  | python/ray/_private/utils.py:338:89: E501 line too long (108 > 88 characters)
  | python/ray/tests/accelerators/test_hpu.py:3:1: F401 'subprocess' imported but unused
  | python/ray/tests/accelerators/test_hpu.py:110:74: E711 comparison to None should be 'if cond is None:'

[jjyao]

Lint failure:

def test_get_current_process_visible_accelerator_ids():
--
  | os.environ[hpu.HABANA_VISIBLE_DEVICES_ENV_VAR] = "0,1,2"
  | -    assert HPUAcceleratorManager.get_current_process_visible_accelerator_ids() == ["0", "1", "2"]  # noqa: E501
  | +    assert HPUAcceleratorManager.get_current_process_visible_accelerator_ids() == [
  | +        "0",
  | +        "1",
  | +        "2",
  | +    ]  # noqa: E501

[jjyao]

Lint failure:

if HPUAcceleratorManager.is_initialized():
--
  | -        assert "Intel-GAUDI" in HPUAcceleratorManager.get_current_node_accelerator_type()
  | +        assert (
  | +            "Intel-GAUDI" in HPUAcceleratorManager.get_current_node_accelerator_type()
  | +        )
  | else:
  | assert HPUAcceleratorManager.get_current_node_accelerator_type() is None

[jerome-habana]

Lint failure:

if HPUAcceleratorManager.is_initialized():
--
  | -        assert "Intel-GAUDI" in HPUAcceleratorManager.get_current_node_accelerator_type()
  | +        assert (
  | +            "Intel-GAUDI" in HPUAcceleratorManager.get_current_node_accelerator_type()
  | +        )
  | else:
  | assert HPUAcceleratorManager.get_current_node_accelerator_type() is None

might be nice to have auto corrector

[jjyao]

Can you also add INTEL_GAUDI2 here?

[jerome-habana]

Sure. I've kept it generic for now. Lets update post closure of the right instance usage ?

[kira-lin]

Hi all, I'm working on LLM serving on Gaudi 2. Is Gaudi 2 not supported yet?