Broker 0.8 #447
Merged
Broker 0.8 #447
+5,393
−1,805
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactoring: - Migrated all codebase modules from `logzero` to the standard Python `logging` library. - Centralized logging configuration logic into a new `broker.logging` module. - Implemented a `RedactingFilter` to automatically redact sensitive information (e.g., passwords, tokens) from log records. - Reworked logging initialization within CLI commands (`broker.commands`) to ensure early setup and capture all messages, including those during module imports. Features: - Introduced support for structured JSON log files alongside existing text file logging, configurable via settings. - Added a custom `TRACE` log level (level 5) for highly verbose debugging. - Leveraged `rich` for enhanced, color-coded console output. - Included utility functions to disable `urllib3` warnings and patch `awxkit` to support trace-level logging of API calls. Configuration: - Extended `broker.settings` with new `LOGGING.LOG_PATH` and `LOGGING.STRUCTURED` options for fine-grained control over log file behavior. - Updated `pyproject.toml` to replace `logzero` with `python-json-logger` as a dependency. - Adjusted `pyproject.toml` and `tox.toml` pytest configurations, including new `ruff` per-file ignores to allow necessary early logging imports. Documentation: - Added a new "API Usage" section to `README.md`, detailing how to integrate Broker as a Python library and manage its logging via standard `logging` configuration. Tests: - Configured logging specifically for the test session within `conftest.py` to ensure proper log capture during testing.
- Split the monolithic helpers.py file into separate modules by functionality: - dict_utils.py: Dictionary manipulation utilities - file_utils.py: File handling utilities - inventory.py: Inventory management utilities - results.py: Result and MockStub classes - misc.py: CLI and miscellaneous utilities - Created broker/helpers/__init__.py to re-export all functions for backward compatibility. - Updated logging configuration to use standard python logging. - Addressed code review feedback regarding imports and type checking.
This adds a very basic "broker shell" command that enters a basic interactive shell. This can be expanded in the future.
This switch is pretty straight forward and continues the trend of beautifying the cli.
This marks uv.lock as a binary file to suppress diffs and configures a custom merge driver 'uv-lock' to automatically regenerate the lockfile on conflicts. Instructions for configuring the local git merge driver have been added to README.md.
Features: - Beaker provider help: - Display individual job XML with syntax highlighting using `rich.syntax.Syntax`. - Format lists of available jobs into `rich` tables. - Container provider help: - Present detailed image information in `rich` tables. - Display image configuration as syntax-highlighted YAML. - Convert lists of available host and app images into `rich` tables. - Add support for the `container_app` parameter to retrieve details for a specific container application. - Foreman provider help: - Render lists of hostgroups and individual hostgroup details in `rich` tables. Refactoring: - Replaced direct `logging.info` calls for user-facing output in `provider_help` methods with `rich.console.Console` for improved presentation. - Removed the `results_limit` parameter, as `rich` tables manage display and filtering more effectively. Configuration: - Integrated `_settings.less_colors` to control color output in the `rich` console, allowing users to disable colors.
Features: - Introduce the new 'scenarios' feature, enabling the definition and execution of complex workflows as chained Broker actions within YAML files. - Add a new `broker scenarios` CLI group to manage these workflows: - `list`: Displays all available scenario files found in the scenarios directory. - `execute <scenario>`: Runs a specified scenario, supporting command-line variable and configuration overrides, and background execution. - `info <scenario>`: Provides a summary of a scenario's configuration, variables, and steps. - `validate <scenario>`: Checks a scenario file against its defined JSON schema for structural correctness. - Scenario functionality includes: - Jinja2 templating for dynamic values within scenario definitions. - Conditional step execution using `when` clauses. - Iterative step execution via `loop` constructs over iterables or inventory filters. - Capture of step outputs into new scenario variables for subsequent steps. - Configurable error handling with `on_error` blocks and `exit_on_error` flags. - Management of a scenario-specific inventory for `checkout` and `checkin` actions. - Support for nesting and running other scenarios as steps. - Implement `ScenarioRunner` as the core class for loading, validating, and executing scenario definitions. - Add `ScenarioError` to handle exceptions specific to scenario execution. Configuration: - Introduce `broker/scenario_schema.json` to formally define and validate the structure of scenario YAML files. - Update `pyproject.toml` to include `paramiko` as a recognized SSH session and interactive shell entry point.
Features:
- Add `output` action for scenario steps to write content to stdout, stderr, or files, with automatic format detection based on file extension.
- Introduce `provider_info` action to query configured providers for available resources (e.g., Ansible Tower workflows, OpenStack images) and retrieve structured data for programmatic use.
- Significantly enhance scenario loop capabilities:
- Allow the `iterable` argument to be a Jinja2 expression, variable, or inventory filter (`@inv`, `@scenario_inv`).
- Support tuple unpacking for `iter_var` (e.g., `key, value` for iterating over dict items).
- Implement `capture.key` to dynamically set keys for loop output within the step memory.
- Evaluate `when` conditions on a per-iteration basis within loops.
- Allow `on_error` in scenario steps to be either a list of recovery steps or the string `"continue"` to proceed despite failures.
- Export the new `save_file` helper for general use.
- Add the `scenarios` command to the interactive Broker shell.
- Provider `provider_help` methods (e.g., AnsibleTower, Beaker, Container, Foreman, OpenStack) now return structured data in addition to printing it, enabling programmatic use within scenarios.
Fixes:
- Correct the `helpers.merge_dicts` behavior to ensure values from the second dictionary argument consistently take precedence over the first.
- Update all `merge_dicts` call sites (e.g., Broker initialization, inventory updates) to correctly reflect the desired precedence, ensuring user-provided arguments or new data override defaults or existing data.
- Ensure scenario configuration values passed via CLI are deep-merged into existing settings rather than overwriting top-level dictionaries.
- Resolve a definition order issue for `broker_shell` commands in `commands.py` by moving shell definition.
Refactoring:
- Improve scenario host resolution to support filtering against both the main Broker inventory (`@inv`) and the scenario-specific inventory (`@scenario_inv`), including proper host reconstruction.
- Refine `_action_ssh`, `_action_scp`, and `_action_sftp` to return a single `Result` object for single-host operations and a dictionary mapping hostnames to `Result` objects for multiple hosts, simplifying templating.
- Simplify `list_scenarios` implementation for clarity.
- Add `_clear_scenario_inventory` to ensure scenarios start with a fresh inventory, preventing carry-over from previous runs.
Documentation:
- Update `scenario_schema.json` to reflect new actions, enhanced loop features, and flexible `on_error` handling.
- Clarify the `merge_dicts` docstring regarding value precedence.
- Add a `.gitignore` file to the `scenarios` directory to exclude non-versioned files.
…mentation Features: - Implement custom scenario-specific logging, allowing users to configure a dedicated log file for each scenario with flexible path resolution rules. - Improve Jinja2 templating to preserve Python types (e.g., int, bool, list, dict) for simple variable references, preventing unnecessary string conversions. - Introduce a comprehensive JSON schema (`scenario_schema.json`) for validating scenario files, formalizing the structure of steps, actions, arguments, loops, conditionals, and error handling. This enables a richer set of scenario capabilities. Fixes: - Improve container hostname resolution in `container_info`, adding robust fallbacks to container name or short ID if the `Config` or `Hostname` attributes are missing. Refactoring: - Adjust the log level for scenario initialization messages from 'info' to 'debug' for cleaner default output. Documentation: - Add a new, in-depth `scenarios_tutorial.md` covering all aspects of scenario creation and execution, including available actions, configuration options, templating, loops, error handling, and examples. Tests: - Introduce new functional tests for scenarios, including a comprehensive test for the Container provider, and specific tests for deploying/checking in containers and querying provider details. - Add `test_scenarios.py` to integrate and run these new functional scenario tests.
Tests: - Introduce `test_scenarios.py` with extensive unit tests for the `broker.scenarios` module. - Cover scenario loading, schema validation, Jinja2 templating, expression evaluation, and conditional logic. - Verify support for loop iteration, output capture, and argument mapping within scenario steps. - Add specific sample scenario YAML files (`capture_scenario.yaml`, `conditions_scenario.yaml`, `loop_scenario.yaml`, `invalid_schema.yaml`, `valid_scenario.yaml`) to exercise various scenario features and validation rules. - Test `StepMemory` functionality and `ScenarioRunner` initialization, configuration, and built-in actions (`output`, `exit`). Configuration: - Add `jinja2` and `jsonschema` as core dependencies to `pyproject.toml` to enable scenario templating and schema validation. - Update `tox.toml` to include the new `test_scenarios.py` in the test suite.
JacobCallahan
added
enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a big one!
Release 0.8: Scenarios Engine, Logging Overhaul, and Interactive Shell
Summary
This PR merges the
0.8release branch into master. The centerpiece of this release is the new Scenarios Engine, which enables users to define and execute complex, chained workflows via YAML. Additionally, this release includes a complete overhaul of the logging subsystem (migrating to standard Python logging), the introduction of an interactive shell, and significant UI improvements using therichlibrary.Key Features
Scenarios Engine
Introduces a powerful workflow engine to chain Broker actions.
broker scenarioscommand group (list,execute,info,validate).when), loops (over iterables or inventory), and error handling (on_error,continue).StepMemory).output(stdout/file writing) andprovider_info(programmatic resource querying).scenarios_tutorial.mdand schema documentation.Logging Overhaul
Refactors the entire logging system for better library integration and debugging.
logzeroto standard Pythonlogging.RedactingFilterto scrub sensitive data (tokens/passwords) from logs.TRACElevel (5), and consolidated configuration inbroker.logging.Interactive Shell & UI
broker shellcommand for a basic interactive session.clickprogress bars withrichprogress bars.provider_helpoutput with formatted tables and syntax-highlighted XML/YAML.Refactoring & Fixes
helpers.pyinto specialized modules (dict_utils,file_utils,inventory,results,misc).merge_dictsto ensure correct precedence (second argument overrides first).uv.lockas a binary file with a custom merge driver and added a.gitignorefor the scenarios directory.Testing
broker.scenarioscovering schema validation, templating, and logic (test_scenarios.py).tox.tomlandpyproject.tomlto support new dependencies (jinja2,jsonschema,python-json-logger).