Skip to content

feat(search): tag query lexer + parser → AST - #5

Merged
deckyfx merged 2 commits into
mainfrom
feat/search-lexer
Jun 28, 2026
Merged

deckyfx merged 2 commits into
mainfrom
feat/search-lexer

Conversation

@deckyfx

@deckyfx deckyfx commented Jun 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

The booru tag query language in packages/search — Query → Lexer → Parser → AST. Pure and DB-free (no HTTP, no Drizzle); the AST is the canonical representation everything downstream consumes (per ARCHITECTURE.md's search rule). Built test-first.

What's included

  • tokenize() (token.ts) — splits a query into classified terms (tag | metatag), each carrying its - (negate) / ~ (or) prefix; lowercases tag names + metatag keys; drops blanks / bare -/~.
  • AST (ast.ts) — and / or / not / tag / metatag discriminated union + CompareOp.
  • parse() (parser.ts) — implicit AND of terms; -term → NOT; ~term ~term → one OR group AND'd with the rest; metatag operators >,>=,<,<= and a..b ranges.

Syntax (v1)

1girl solo (AND) · -monochrome (NOT) · ~blue_eyes ~green_eyes (OR group) · rating:safe, score:>10, width:100..200 (metatags)

Verification

  • 21 tests (lexer + parser), test-first · bun run test ✓ (43 total)
  • bun run typecheck ✓ · bun run lint:boundaries ✓ (search stays a leaf)

Out of scope (lands with the DB)

The optimizer and SQL builder — they consume this AST.

Summary by CodeRabbit

  • New Features

    • Added query tokenization and parsing into a structured, canonical query representation.
    • Search queries now support tags, negation, ~-based OR grouping, and metatag filters with comparison operators and range syntax.
    • Expanded the public search API by exposing the parser/tokenizer and related query/token types.
  • Tests

    • Added automated coverage for tokenization and parsing, including empty queries, implicit AND behavior, negation, OR grouping, and metatag comparison/range edge cases.

The booru query language in `packages/search`, pure and DB-free
(`Query → Lexer → Parser → AST`; the AST is the canonical representation
everything downstream consumes). Optimizer + SQL builder land with the DB.

- `tokenize()` → classified terms (tag | metatag) carrying `-`/`~` prefixes
- AST node types (and/or/not/tag/metatag, CompareOp)
- `parse()` → AST: implicit AND; `-term` negates; `~term ~term` form one OR
  group AND'd with the rest; metatag operators `>`,`>=`,`<`,`<=` and `a..b` ranges
- 21 tests (lexer + parser), built test-first

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@deckyfx

deckyfx commented Jun 28, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Jun 28, 2026 •

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9f747f7d-808e-4d60-8530-b6b826a3cd36

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Introduces a search query tokenizer, AST types, parser, tests, and public re-exports in packages/search. The parser builds typed AND/OR/NOT/tag/metatag trees from query strings, including prefix modifiers and metatag comparisons or ranges.

Changes

Search Query Pipeline

Layer / File(s) Summary
AST type definitions
packages/search/src/ast.ts
Defines CompareOp, AndNode, OrNode, NotNode, TagNode, MetatagNode, and the QueryNode union type.
Tokenizer: types and implementation
packages/search/src/token.ts, packages/search/test/token.test.ts
Defines Prefix, TagToken, MetatagToken, and Token, implements tokenize() and classify(), and adds tokenizer tests for prefixes, metatags, whitespace, and bare-prefix handling.
Parser: token-to-AST conversion
packages/search/src/parser.ts, packages/search/test/parser.test.ts
Defines metatag comparison constants, implements parse(), toNode(), and parseMetatag(), and adds parser tests for AND/OR/NOT, metatag operators, ranges, and combined queries.
Public re-exports
packages/search/src/index.ts
Expands the package public surface by re-exporting tokenize, parse, and token and AST type definitions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐇 I hopped through tags and metatags bright,
-not and ~or now parse just right,
The AST grew roots in forest green,
With tokens tidy and tests serene,
Crunch-crunch, this query garden sings! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding a search query lexer/parser that produces an AST.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/search-lexer

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/search/src/parser.ts`:
- Around line 48-57: The range detection in parseMetatag is too broad because
any raw value containing “..” is treated as a range before operator parsing,
which blocks valid literals and mixed comparison inputs. Update parseMetatag to
only classify true range syntax, using a stricter check around the value pattern
rather than a plain includes() test, and keep the existing operator handling for
OPERATORS and the default eq path so inputs like source:https://a..b and
score:>10..20 can parse correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f3dce521-167b-4628-b5a4-6fee7c886d11

📥 Commits

Reviewing files that changed from the base of the PR and between b8fbd5c and 55defe1.

📒 Files selected for processing (6)
  • packages/search/src/ast.ts
  • packages/search/src/index.ts
  • packages/search/src/parser.ts
  • packages/search/src/token.ts
  • packages/search/test/parser.test.ts
  • packages/search/test/token.test.ts

Comment thread packages/search/src/parser.ts Resolved
Operators are now matched before ranges, and a range requires the lo..hi
pattern with no scheme/path chars — so source:https://a..b stays eq and
score:>10..20 is a comparison, not a misparsed range. +2 tests.

Addresses CodeRabbit review on PR #5.
@deckyfx

deckyfx commented Jun 28, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@deckyfx
deckyfx merged commit 9e71084 into main Jun 28, 2026
2 checks passed
@deckyfx
deckyfx deleted the feat/search-lexer branch June 28, 2026 11:03
deckyfx added a commit that referenced this pull request Jun 29, 2026
Complete the search pipeline started in #5: the parsed query AST compiles to
a Drizzle WHERE and filters the assets listing.

- core: add the Search Engine (`compileAssetSearch`) — walks the query AST
  into a Drizzle condition. Metatags map to asset columns (`rating:` enum
  equality; `width`/`height`/`size` numeric eq/gt/gte/lt/lte and `lo..hi`
  ranges, with malformed multi-bound ranges like `1..2..3` ignored, not
  silently truncated); tag terms compile to an `asset_tags` EXISTS subquery
  (valid SQL now, matching once tagging lands). and/or/not via Drizzle;
  unknown keys and invalid values are dropped, not errored.
- db: `findMany`/`count` accept an optional prebuilt `where: SQL` so the page
  and its total share one filter (db never parses text).
- core: `AssetService.list` accepts `query`; trims then compiles it once.
- api: `GET /api/v1/assets?q=` (capped at 1024 chars) forwards to the service.
- tests: integration coverage for rating/numeric/range/negation/tag and
  pagination metadata under a filter; API q-forwarding unit test.

Test safety: the destructive integration suites (they TRUNCATE) read a
dedicated, opt-in `TEST_DATABASE_URL` instead of the app's `DATABASE_URL`, so
a misconfigured dev/prod env can never be wiped — unset → they skip. CI sets
it to its ephemeral Postgres; `.env.example` documents it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant