Skip to content

refactor: reorganize the Rust workspace (layers, newtype crates, grouped crates) - #10111

Open
404Wolf wants to merge 128 commits into
mainfrom
wolf/repo-reorg
Open

404Wolf wants to merge 128 commits into
mainfrom
wolf/repo-reorg

Conversation

@404Wolf

@404Wolf 404Wolf commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Why

Reorganize the Rust workspace into explicit layers and crate groups, reduce dependency coupling, and split existing hexagonal crates into separate core, inbound and outbound packages. MACRO-4953.

This PR is held for Wolf's review; do not merge. The full hexagonal split is still being integrated. Passing checks on an earlier head do not establish that this head is ready.

How it works

The refreshed move/rename series has 127 linear commits, preserving the original order and subjects, on main e22fa6483f5b. Mechanical move and rename commits stay separate from handwritten changes; adaptations for code added by main are recorded in their owning commits or the final follow-main commit.

The scripts and move maps are in tooling/scripts/crate_moves/. They update Cargo members and dependencies, source and fixture paths, generators, Nix selectors, install links, and directory guides. cargo x deps --check enforces layer assignment and rejects new violations, stale allows, and unmatched paths.

Split families use this layout (the crate names retain their suffixes):

crates/<group>/<name>/
  Cargo.toml + src/  # <name> facade
  core/             # <name>_core: domain models, ports, services
  inbound/          # <name>_inbound: entry points
  outbound/         # <name>_outbound: storage and external adapters

Only crates that already have domain and adapter modules are in the full-split scope. Proposed exceptions require explicit review; they are not silently omitted.

What

  • Group crates under crates/<group>/<crate> and introduce leaf ID/type crates and dependency ports.
  • Preserve native and web wire formats and runtime artifact names while renaming packages. Generated contracts and native feature graphs require fresh verification on the final integrated head.
  • Keep main's development-shell settings as the source of truth for debuginfo; this PR does not introduce the old branch's dev-profile change.
  • Preserve main's Outlook changes, provider imports and integration tests through the email adapter split; convert raw tool thread IDs at the new EmailThreadId boundary.
  • Preserve main's privacy-safe span fields and existing span names through the email source-port changes.
  • Preserve incoming rule discovery and fail-closed email reads while adapting the source port and new test callers; correct removed adapter imports and package-relative SQL resources without changing query text.
  • Refresh six production SQLx descriptors after the model_entity to entity_kind output-type alias rename, preserving SQL predicates, parameters and all existing metadata files.
  • Require verified owned database URLs after dev-shell setup in the database development guide, and remove ambiguous shared/default setup and reset examples.
  • Follow macrod's grouped path in installer URLs and Nix package selectors.
  • Preserve recorded data and validation literals during moves; interpret resource paths in their actual contexts. Retain historical reconstruction support by resolving packages and library aliases from the target checkout.
  • Resolve coverage source packages through the existing Cargo workspace graph, preserving coverage semantics across grouped paths and declared library targets.
  • Make the layer check blocking. The refreshed move/rename snapshot retains 24 documented existing allows and adds no new allows. The browser SQL engine dependency remains explicitly documented for a separate wasm-safe core follow-up.
  • Let the privacy scanner recognize large rename sets (diff.renameLimit=50000) without adding privacy exceptions.

The prior PR history is preserved at backup/repo-reorg-2026-11-12. Its audit and performance measurements are historical evidence, not results for this refreshed head.

Moving crate paths and names is expected to cause initial workspace-crate cache misses in sccache/kache; registry dependencies retain their registry paths. The reorg keeps main's shared build-cache and incremental-build setup rather than adding a parallel configuration (including the work from #10785).

Follow-ups

  • Current PR head: 95063282e5f033336098e2e5c808b7e3a6bcc27f. The 127-commit move/rename series ends at a4ece6f4e116b638a656009d5ccd6b20b3aaae2a, based on e22fa6483f5b0ac38bd5d04abb9f7ebf762b6a52; all original messages and order remain unchanged. That series adds only six reviewed SQLx metadata files to preserved predecessor ca2ef794f6d4ee2fe2a81cb3aacaeea1e4f957ac. One separate documentation follow-up clarifies owned database targets after Nix setup. The last recorded main comparison had 59 conflicts; this historical comparison has not been refreshed. This snapshot is not merge-ready.
  • Complete the full hexagonal split in this PR. Accepted foundation/tooling and group checkpoints remain on separate worker branches; they are not all integrated into this snapshot. SQLx-derived domain enums, provider types, search boundaries and other recorded decisions remain explicit work.
  • Complete fresh service, generated-code, full workspace, native and browser verification on the resulting head; get all required CI green. The predecessor's online five-package/AHS checks and 17 selected-thread tests passed, while its offline service build exposed the six missing metadata keys repaired here. Focused saved_views validation and scoped local checks passed for the repair. Corrected-head offline service/preview verification remains pending; stopped local services require coordinated restart. Full workspace, frontend, infra and native verification still require final integrated-head evidence.
  • Native version/store gates may be skipped only after the authorized replacement proof: generated contracts, wire formats, native feature comparison, host/Android Tauri checks, iOS simulator preview, and macOS desktop QA (suite=full). No store release or deploy is authorized by this PR description. Native run links will be added when available.
  • After this PR is green, open the separately authorized wasm-safe engine-core follow-up, with its own native verification, to remove the documented browser SQL engine layer allow.

@macro-application

Copy link
Copy Markdown

@404Wolf 404Wolf added the run-ci Run CI on the PR's head commit; removed when the run starts label Nov 7, 2026
@github-actions github-actions Bot removed the run-ci Run CI on the PR's head commit; removed when the run starts label Nov 7, 2026
@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

Warning

Review details and warnings were omitted to fit the comment limit.

This was referenced Nov 7, 2026
404Wolf and others added 30 commits November 19, 2026 21:28
Type crates are named <x>_types; models_forms was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_grouping was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_opensearch was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_pagination was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_permissions was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_properties was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_search_cursor was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_sfs was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_team was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_search was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type crates are named <x>_types; models_soup was the last naming scheme. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ent_sync and document_sync_router

sync_service_hex and sync_router are the document sync hexagon and its DSS router; next to services/sync-service (package sync_service) their names were confusing. Rename them document_sync and document_sync_router. Rename the crate and its
directory; types and wire names are unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: the rename script rewrites code, manifests and whole-word mentions of
distinctive names, but leaves common words (agent, model, chat, cal) in
prose alone, and two TS generators build the agent_tools path from
separate segments.

What: the crate group indexes name the renamed crates (rows re-sorted),
generate-dcs-tools.ts and generate-mcp-tool-pages.ts find
crates/ai/agent_tools, and the calcom_webhooks debug hint names its new
tracing target.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: the layer report only printed, so a new edge across layers, a stale
allow or a crate outside every layer could land unnoticed. With the
crates grouped and the edge cuts in, the baseline is down to 23 allows.

What: cargo x deps (and CI's cargo x deps --check) fails when the report
lists a new violation, a stale [[allow]], an unassigned crate or a layer
path that matches no crate. The 23 baselined allows keep their reasons
and only print. The Rust guide and .config/layers.toml say the allowlist
only shrinks.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: open branches need the renames' old and new directories and package
names to catch up, and rename_crate.py's --words pass treated Rust
attribute lines (`#[cfg(feature = "ai_tools")]`) as comments, so renaming
ai_tools also renamed a feature that kept its name.

What: maps/22-renames.txt lists every rename after the group moves, and
the README gives the loop that rewrites dependency keys and paths from it.
The --words pass only rewrites `//` and `/* */` lines in Rust files. (The
ai_tools rename commit already keeps its `feature = "ai_tools"` gates.)

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: main added routine-draft code naming model_entity while the renames
were in flight, and search_processing_service's absolute `::chat::`
import was left alone because the file also has a local `chat` module.

What: those paths name entity_kind and ai_chat. rename_crate.py now
rewrites `::old::` even in files with a local item of the old name.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The longer illustrator_engine and database_sql_engine file names pushed
two new URL(...) calls past the line width.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: apps/web/tauri builds predicate_index by path, which depends on
cowlike, so renaming it to cow_str rewrote apps/web/tauri/Cargo.lock.
Files under tauri/ feed the iOS and Android release gates, which are out
of scope for this PR (CI's Tauri lockfile check failed on it).

What: the mechanical inverse of the cow_str rename (rename_crate.py
cow_str cowlike); maps/22-renames.txt drops the pair.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: database_sql_runner's row-source tests read the transcripts through
a format!-built CARGO_MANIFEST_DIR path the rename script does not parse,
and the search service's OpenAPI description quotes a doc comment that
the models_soup rename changed, so the generated web client drifted.

What: the tests read ../database_sql_engine/fixtures; the generated
service-search openapi.json and calendarEventSearchTime.ts carry the
regenerated soup_types wording.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: the earlier commit moved the crate to crates/agents/macrod but left
its package named coding_agent_worker, so `cargo -p`, the release
scripts and the docs still used the name of an unrelated Bun service.

What: the package is macrod (its binary already was); release scripts,
justfile, the web app's setup hints and docs follow. Install docs point
at crates/agents/macrod/install; the forwarding scripts in
crates/coding_agent_worker/install keep the old public URLs working.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: cargo-deny's tracing-subscriber ban lists the workspace crates
allowed to use it by package name, so agent, initiative, projects and
coding_agent_worker stopped matching after their renames.

What: the list names agent_loop, project, folder and macrod.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent setup instructions linked to crates/coding_agent_worker, which
now only holds the forwarding install scripts.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…soup rename

The SDK's copy of the search OpenAPI spec quotes the same doc comment
as the web client's.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ql_types

Why: the TypeScript generator tells the ops' ColumnKind and EntityKind
apart from the catalog's by module path, which still matched
models_databases, so both exported as ColumnKind and generation failed.

What: it matches databases_types; the generated types.ts is unchanged.

MACRO-4953

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: cargo-deny now reports RUSTSEC-2026-0462 for rmcp 1.7 (Streamable
HTTP clients buffer whole error response bodies). The fix ships in rmcp
3.9, two major versions past the 1.x API the MCP crates use, so the
upgrade is a change of its own.

What: an ignore entry with that reason in deny.toml.

MACRO-4953
…ore the renames

Why: the renames changed what the web app asks for and what traces record.
The database SQL and Illustrator wasm loaders asked for
database_sql_engine.js and illustrator_engine.js, but wasm-pack still
writes database_sql.js and ai_engine.js (--out-name is unchanged), so both
engines would 404 at load. Two spans had been renamed by their functions'
new names, and five spans recorded the Debug form of the new ChannelId and
DocumentId types instead of the raw id, which breaks trace searches.

What:
- Restore the loader and justfile artifact names to database_sql and
  ai_engine.
- Pin span names dynamic_email_thread_cursor and release.
- Record channel_id and document_id as their raw values.
- Correct the agents group doc about where macrod lives.
Why: the engines now build through thin *_wasm wrapper crates, but the
ensure-*-wasm fast paths still only watched the engine crates, so an edit
to just a wrapper served a stale dev wasm.

What: ensure-database-sql-wasm walks database_sql_wasm's dependency tree
(which includes database_sql_engine); ensure-fig/psd/ai-engine-wasm also
watch their wrapper crate. The wasm loaders' comments name the crate
wasm-pack actually builds.
…paces

Why: main made cache-core depend on database_sql, which the Tauri
workspaces build. On this branch that crate is database_sql_engine and
uses the new id crates, so the Tauri lockfiles no longer resolved under
--locked.

What: regenerate apps/web/tauri and desktop-cef Cargo.lock. The only
third-party additions (convert_case 0.11, kinded 0.5.1, nutype 0.8.0)
are the versions the root lockfile already uses.
Why: code main added after the reorg branched still named the old crates
where the move and rename scripts cannot see it. Three new queries in
macro_db_client and project spell a type override with the crate's path
(`Json<model_entity::EntityAppearance>`). The rename turned it into
`entity_kind`, which changes the query text, so their offline sqlx cache
entries no longer matched and SQLX_OFFLINE builds failed. The macro-box
ACP lock script builds its path to macrod's pinned adapter version from
its own directory, so it still pointed at crates/coding_agent_worker.

What: the three .sqlx entries carry the renamed query text under its new
hash, and the lock script reads crates/agents/macrod.

MACRO-4953
Why:
Nix shells can replace database URLs with a shared local default.

What:
Require verified owned targets in final commands after environment setup
and preserve explicit approval for disposable database resets.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant