Skip to content

Add weekly agent examples test workflow #3902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Aug 26, 2025
Merged

Add weekly agent examples test workflow #3902

merged 3 commits into from
Aug 26, 2025
+389 −0

Conversation

strickvl
Copy link
Contributor

@strickvl strickvl commented Aug 25, 2025 •
edited
Loading

This PR adds a new GitHub Actions workflow to test all agent framework integration examples weekly, ensuring they continue to work with the latest ZenML code.

Changes

  • Added scripts/run-agent-examples.sh - Bash script that:

    • Iterates through each subfolder in examples/agent_framework_integrations
    • Creates isolated uv virtual environments (Python 3.11) for each example
    • Installs local ZenML via editable install to test against repository code
    • Runs each example and collects success/failure statistics
    • Generates detailed logs and summary reports

  • Added .github/workflows/weekly-agent-pipelines-test.yml - GitHub Actions workflow that:

    • Runs weekly on Mondays at 8:30 AM CET
    • Uses the OpenAI API key from repository secrets
    • Executes all agent examples sequentially
    • Uploads artifacts with logs and failure lists
    • Sends Discord notifications when examples fail

Key Features

  • Environment isolation: Each example runs in its own fresh virtual environment
  • Tests against local code: Uses editable install to validate examples against current branch
  • Comprehensive logging: Individual log files for debugging failures
  • Graceful failure handling: Continues testing all examples even if some fail
  • Rich reporting: Markdown summary with timing and pass/fail counts

Testing

The workflow can be manually triggered via workflow dispatch to test immediately.

Related Issues

N/A - Proactive addition for CI coverage of agent framework examples

@strickvl
Add weekly agent examples test workflow
f29899e 
- Create scripts/run-agent-examples.sh to test all agent framework integrations
- Each example runs in isolated uv virtual environment with Python 3.11
- Tests against local ZenML code via editable install
- Collects failures and generates detailed reports
- Update workflow to run weekly on Mondays at 8:30 AM CET
- Send Discord notifications with failure details when tests fail
@strickvl strickvl added the internal To filter out internal PRs and issues label Aug 25, 2025
@strickvl strickvl requested a review from htahir1 August 25, 2025 13:17
@coderabbitai coderabbitai
Copy link
Contributor

coderabbitai bot commented Aug 25, 2025

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch testing/weekly-agent-examples-smoke-test

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@strickvl strickvl added the CI label Aug 25, 2025
@strickvl strickvl requested a review from bcdurak August 25, 2025 13:17
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 25, 2025 •
edited
Loading

Documentation Link Check Results

❌ Absolute links check failed
There are broken absolute links in the documentation. See workflow logs for details
✅ Relative links check passed
Last checked: 2025-08-26 13:45:24 UTC

htahir1
htahir1 requested changes Aug 25, 2025
View reviewed changes
Copy link
Contributor

@htahir1 htahir1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to test this but overall looks good

@strickvl
Update weekly agent pipelines test workflow environment variables
3e0a25b 
- Adjust environment variables in the weekly agent pipelines test workflow YAML file.
- Set ZENML_LOGGING_VERBOSITY to INFO and ensure MLSTACKS_ANALYTICS_OPT_OUT is true.
- Maintain existing configurations for other environment variables.

This update enhances the logging configuration and analytics settings for better test management.
@strickvl
Remove ineffective UV cache from weekly agent test workflow
6439086 
The cache key included github.run_id which made it unique for every
run, defeating the purpose of caching. This would create cache pollution
without providing any performance benefits.

UV has its own internal caching and is already fast enough for this
weekly job without the GitHub Actions cache layer.
@strickvl strickvl requested a review from htahir1 August 26, 2025 10:08
@strickvl strickvl merged commit 49a32d8 into develop Aug 26, 2025
50 of 54 checks passed
@strickvl strickvl deleted the testing/weekly-agent-examples-smoke-test branch August 26, 2025 13:35
@strickvl strickvl restored the testing/weekly-agent-examples-smoke-test branch August 26, 2025 13:38
strickvl added a commit that referenced this pull request Aug 26, 2025
@strickvl
Revert "Add weekly agent examples test workflow (#3902)"
8ada696 
This reverts commit 49a32d8.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@htahir1 htahir1 htahir1 approved these changes

@bcdurak bcdurak Awaiting requested review from bcdurak

Assignees
No one assigned
Labels
CI internal To filter out internal PRs and issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@strickvl @htahir1