Skip to content

Require AI-generated demo artifacts to be verified against a real build - #3936

Open
Diolor wants to merge 2 commits into
OWASP:masterfrom
Diolor:demo-artifact-generation-rules
Open

Diolor wants to merge 2 commits into
OWASP:masterfrom
Diolor:demo-artifact-generation-rules

Conversation

@Diolor

@Diolor Diolor commented Aug 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

[AI generated]

An AI assistant can produce a convincing MastgTest_reversed.java, output.txt, or r2 disassembly that never came from a build. It reads correctly and looks verified, so it survives review — a fabricated artifact is worse than a missing one.

Demos written or modified with AI therefore have to be verified by actually building the MASTestApp and running the reverse-engineering or scanning pipeline against it. Nothing in the instructions asked for that, and mastg-demo.instructions.md applied only to demos/**/*.md, so editing output.txt or MastgTest_reversed.java loaded no demo guidance at all.

These artifacts also drift on their own: changing a demo's run.sh silently invalidates its committed output.txt, and one untouched demo's output no longer reproduced under a newer semgrep. Both are only visible if you run it.


AI Tool Disclosure

  • This contribution does not include AI-generated content.

  • This contribution includes AI-generated content.

  • AI tools used: Claude Code

  • Models and versions: Claude Opus 5

  • Prompt summary: Asked the assistant to review an in-progress branch for missing content and incorrect cross-references. It found the applyTo gap and the absence of any rule against hand-writing artifacts; I then asked it to draft both changes. Wording reviewed and adjusted before opening.


Contributor Checklist

  • I have read and understood the contributing guidelines.
  • I followed the project style guide.
  • I validated the technical correctness of my changes and understand the topic.
  • This PR adds clear value and is not spam or low-effort content.

Demo build outputs (decompiled sources, reversed manifests and plists,
app binaries, and output.txt) are only trustworthy if they came from an
actual build plus reverse engineering or scanning run. Nothing in the
instructions said so explicitly, and the demo instructions applied only
to `demos/**/*.md`, so an assistant editing output.txt or
MastgTest_reversed.java loaded no demo guidance at all.

- Widen applyTo from `demos/**/*.md` to `demos/**` so the rules load
  when any demo file is edited, not just the write-up.
- State that build artifacts must be generated and never written,
  edited, or reconstructed by hand or from memory, calling out AI
  assistants explicitly.
- Note the two consequences that bite in practice: r2 script addresses
  shift on every rebuild, and editing run.sh changes output.txt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Diolor Diolor changed the title Require demo build artifacts to be generated, never hand-written Require AI-generated demo artifacts to be verified against a real build Aug 16, 2026
@Diolor
Diolor requested a review from cpholguera August 16, 2026 22:00
@Diolor Diolor self-assigned this Aug 16, 2026
@Diolor
Diolor requested a review from jacobocasado October 20, 2026 09:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant