Updates for supporting multi-host CI

[akirby-TT]

Ticket

Link to Github Issue

Problem description

• We currently don't have any testing for multi-host systems on CI
• Distributed features are thus being tested in multi-process environments, which is fairly limited

What's changed

• Add multi-host-physical.yaml workflow file for capturing all CI workloads running on 2 Loudboxes
• These Loudboxes have been allocated/provisioned as dedicated multi-host CI machines
• Add dual T3K rank binding file
• Update Intermesh Routing tests to cover more traffic patterns
• Update tt-run to propagate TT_METAL_HOME, PYTHONPATH and LD_LIBRARY_PATH to child processes that may be called remotely

Checklist

• All post commit CI passes
• Blackhole Post commit CI with demo tests passes (if applicable)
• Model regression CI passes (if applicable)
• Device performance regression CI passes (if applicable)
• (For models and ops writers) Single-card demo tests CI passes (if applicable) See recommended dev flow.
• Galaxy quick CI passes (if applicable)
• TG demo tests, for Llama CI passes, if applicable, because of current Llama work
• (For runtime and ops writers) T3000 unit tests CI passes (if applicable, since this is run on push to main)
• (For models and ops writers) T3000 demo tests CI passes (if applicable, since this is required for release)
• (For models and ops writers) Single-card demo tests CI passes (if applicable, since this is required for release)
• New/Existing tests provide coverage for changes

[github-actions]

⚠️ Clang-Tidy found issue(s) with the introduced code (1/1)

[Copilot]

Pull Request Overview

This PR adds debug output and configuration updates to support multi-host CI testing. It introduces extensive logging throughout the intermesh ethernet link initialization and routing logic to help diagnose multi-host connectivity issues.

• Adds debug console output for intermesh link configuration and status
• Updates test cases to focus on specific multicast routing patterns
• Includes new configuration files and CI workflow for dual T3K testing

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 12 comments.

File	Description
tt_metal/fabric/control_plane.cpp	Adds debug logging throughout intermesh link initialization and routing
tests/tt_metal/multihost/fabric_tests/intermesh_routing.cpp	Simplifies test cases to focus on north/south multicast patterns
tests/tt_metal/distributed/config/dual_t3k_rank_bindings.yaml	New configuration file for dual T3K rank bindings
.github/workflows/multi-host-physical.yaml	New CI workflow for multi-host physical testing

[omilyutin-tt]

Is it possible to not override LD_LIBRARY_PATH but to extend it with .build/lib? Also ideally add a comment why this is here.

[tt-asaigal]

No longer capturing this in the rank binding file. Instead we can pass this in as a top level env var to the tt-run script.

This way the same binding file can be reused across setups and users can set the env vars they require in a custom manner.

Not sure why the multi-host Github Actions setup requires the path to be explicitly specified though we can look into this as a cleanup step.

[cfjchu]

Approving to unblock

[cfjchu]

should this be main?

[tt-asaigal]

This is just here temporarily, will be removed right before merging.

This allows us to trigger the workflow on the current PR, we won't be able to do so otherwise.

[cfjchu]

Path.home()? Can we leave a TODO comment for @akirby-TT to revisit why this was needed?

[tt-asaigal]

Latest commit removes this. Will remove this entirely if it passes. Will fix the path otherwise with a comment.

[tt-asaigal]

seems to be working without the path specified. Removing.