notify maintainers on evaluation errors #376

Activity

1. Mic92 commented on Aug 25, 2021

   

   Mic92

   on Aug 25, 2021

   Member

   More actions

   Yes that would be nice. Is there a github action for that?
2. milahu commented on Aug 25, 2021

   

   milahu

   on Aug 25, 2021

   Last edited by milahu

   ContributorAuthor

   More actions

   Is there a github action for that?

   i guess this belongs in nur/update.py

           logger.info(f"Evaluate repository {repo.name}")
           proc = subprocess.Popen(
               cmd, env=dict(PATH=os.environ["PATH"]), stdout=subprocess.DEVNULL
           )
           try:
               res = proc.wait(10)
           except subprocess.TimeoutExpired:
               raise EvalError(f"evaluation for {repo.name} timed out of after 10 seconds")
           if res != 0:
               raise EvalError(f"{repo.name} does not evaluate:\n$ {' '.join(cmd)}")

   so we need:

   • capture the output of the subprocess, correctly interleave stdout and stderr
   • a github api client in python, for example PyGithub to create issue
   • a github account for the bot -> @nur-bot?
   • hide the access token -> github repo config?
   • avoid duplicate issues -> get issue?
   • handle repos without issue trackers -> create issue in NUR/issues and mention the maintainer
   • allow maintainers to turn off the notifications -> repos.json? default on?

   concept:

   #!/usr/bin/env python3
   
   # https://github.com/nix-community/NUR/blob/master/nur/update.py
   
   import subprocess
   import os
   from github import Github # https://github.com/PyGithub/PyGithub
   
   bot_name = "nur-bot" # https://github.com/nur-bot
   bot_token = os.environ["GITHUB_BOT_TOKEN"] # access token
   # TODO set token in github config
   
   eval_timeout = 10 # seconds
   
   github = Github(bot_token)
   
   # TODO loop values from repos.json
   # sample: https://github.com/milahu/nur-packages
   repo_owner = "milahu"
   repo_name = "nur-packages"
   repo_version = "TODO"
   
   cmd = [
       "bash",
       "test.sh"
   ]
   
   proc = subprocess.Popen(
       cmd, env=dict(PATH=os.environ["PATH"]),
       stdout=subprocess.PIPE, stderr=subprocess.STDOUT
   )
   
   timeout = False
   nonzero = False
   
   try:
       proc.wait(eval_timeout)
   except subprocess.TimeoutExpired:
       timeout = True
   
   if proc.returncode != 0:
       nonzero = True
   
   if timeout or nonzero:
       output = proc.stdout.read().decode() # stdout and stderr are interleaved
   
       # send output to logfile
       print(output)
   
       # notify maintainer
       # TODO check if maintainer wants to receive notifications -> repos.json
       # check if we already sent a notification for this error
       repo = github.get_repo(f"{repo_owner}/{repo_name}")
       last_issue = repo.get_issues(creator=bot_name, sort="newest")[0]
       issue_title = f"evaluation failed for version {repo_version}"
       if issue_title != last_issue.title:
           # we have not yet notified the maintainer of this error
           # create a new issue on the maintainer's repo
           issue_body = f"evaluation failed.\n\ntimeout: {"yes" if timeout else "no"}\nreturn code: {proc.returncode}\n\noutput:\n\n```\n" + output.strip() + "\n```\n"
           repo.create_issue(title=issue_title, body=issue_body)
           # TODO handle "no issue tracker"
   
   if timeout:
       raise EvalError(f"evaluation for {{repo.name}} timed out of after {eval_timeout} seconds")
   
   if nonzero:
       raise EvalError(f"{{repo.name}} does not evaluate:\n$ {' '.join(cmd)}")
3. Mic92 commented on Aug 25, 2021

   

   Mic92

   on Aug 25, 2021

   Member

   More actions

   Btw. you can also check evaluation in your own repo using: https://github.com/nix-community/nur-packages-template/blob/cc8c7823f2280d6216c521efb3814312eeae483e/.github/workflows/build.yml#L57
4. milahu commented on Aug 27, 2021

   

   milahu

   on Aug 27, 2021

   Last edited by milahu

   ContributorAuthor

   More actions

   yepp:

   nix-env -f . -qa \* --meta --xml \
     --allowed-uris https://static.rust-lang.org \
     --option restrict-eval true \
     --option allow-import-from-derivation true \
     --drv-path --show-trace \
     -I nixpkgs=$(nix-instantiate --find-file nixpkgs) \
     -I ./ \
     >/dev/null
   

   edit: error: access to path '/nix/store/j777ivm3g8b3nq6vii1bccfbahsa9cl3-nixos-21.11pre309670.253aecf69ed/nixos' is forbidden in restricted mode is solved by replacing -I $PWD with -I ./
5. Rhys-T commented on Mar 3, 2025

   

   Rhys-T

   on Mar 3, 2025

   Contributor

   More actions

   Just a heads up that this 'Check evaluation' nix-env command doesn't seem to catch everything. In particular, I recently had a mistake in my repo where one package's meta.homepage was failing to evaluate, because it was being inherited from the wrong attrset. The nix-env step in my repo's workflow just silently left the homepage out of its XML output, but it caused an actual eval error on NUR's side, causing my repo to disappear from the site. Still helps with the more common cases, though.

   👀React with 👀1Reacted by Gavin John
6. Pandapip1

   added the

   Feature

   issue type on Jul 30, 2026