Skip to content

Add chakra_linter to validate Chakra execution traces against a rule set #254

Description

@ajbalogh

Problem

  • Chakra execution traces (ET) have no automated validation step.
  • Malformed or incomplete traces surface only as downstream failures in the simulator, replay, or visualizer.
  • Converter output (PyTorch ET, FlexFlow, and others) varies in attribute completeness, with no common quality gate.
  • CPU replay readiness cannot be verified before a trace reaches a consumer.
  • Contributors have no shared, documented definition of a "valid" trace.

Solution

  • Add a chakra_linter command-line tool under src/linter/.
  • Define checks declaratively in rules.toml, validated by rules.schema.json.
  • Cover three rule families:
    • Structural checks (node IDs, dependencies, graph integrity).
    • Converter attribute checks (required attributes per node type).
    • CPU replay checks (fields required for replay).
  • Emit findings with a severity and rule ID for each violation.
  • Generate HTML and markdown reports for human review and CI artifacts.
  • Return a non-zero exit code on rule violations to support CI gating.
  • Register the console script chakra_linter in pyproject.toml.

Purpose

  • Detect trace defects at the earliest point in the workflow.
  • Give converter authors and trace producers objective, repeatable feedback.
  • Reduce time spent diagnosing downstream failures caused by invalid traces.
  • Establish the rule set as the single documented definition of trace validity.

Goals

  • Provide a single command that lints one or more Chakra ET files.
  • Keep the rule set data-driven, so adding or tuning a rule requires no code change.
  • Provide a schema that rejects malformed rule files.
  • Produce reports readable by project managers and engineers without tooling.
  • Support integration into CI pipelines through exit codes and report artifacts.
  • Maintain automated test coverage for rule loading, rule inventory, converter attributes, and CPU replay checks.
  • Document usage and rule authoring in src/linter/README.md.

Acceptance Criteria

  • chakra_linter installs as a console script with pip install ..
  • The tool lints a valid trace and exits with code 0.
  • The tool lints a trace with violations and exits with a non-zero code.
  • Each finding lists the rule ID, severity, and affected node or file.
  • The tool writes an HTML report and a markdown report on request.
  • rules.toml validates against rules.schema.json, and a test fails on an invalid rule file.
  • Every rule in rules.toml has at least one test, enforced by the rule inventory test.
  • Unit tests under tests/linter/ pass in CI on the supported Python version (3.11).
  • src/linter/README.md documents installation, usage, report formats, and rule authoring.
  • Ruff lint and format checks pass on all new files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions