feat(tidy3d): FXC-5275 schema-versioned config migrations for tidy3d config

CHANGELOG.md

@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `GaussianPort` and `AstigmaticGaussianPort` for S-matrix calculations using Gaussian beam sources and overlap monitors.
 - Added `symmetric_pseudo` option for `s_param_def` in `TerminalComponentModeler` which applies a scaling factor that ensures the S-matrix is symmetric in reciprocal systems.
 - Added deprecation warning for ``TemperatureMonitor`` and ``SteadyPotentialMonitor`` when ``unstructured`` parameter is not explicitly set. The default value of ``unstructured`` will change from ``False`` to ``True`` after the 2.11 release.
+- Added config versioning with automatic backward migrations by default; forward‑compat is best‑effort unless strict mode is enabled.
 
 ### Breaking Changes
 - Added optional automatic extrusion of structures at the simulation boundaries into/through PML/Absorber layers via `extrude_structures` field in class `AbsorberSpec`.

docs/api/configuration.rst

@@ -39,3 +39,13 @@ Registration Utilities
    tidy3d.config.register_handler
    tidy3d.config.get_sections
    tidy3d.config.get_handlers
+
+Schema Versioning
+-----------------
+
+.. autosummary::
+   :toctree: _autosummary/
+   :template: module.rst
+
+   tidy3d.config.CURRENT_CONFIG_VERSION
+   tidy3d.config.register_migration

docs/configuration/index.rst

@@ -60,6 +60,10 @@ Files Inside the Directory
 - ``profiles/<name>.toml`` – optional overrides for custom profiles. Each file
   only contains the differences from the base settings.
 
+Each configuration file includes a root ``config_version`` key. When it is
+missing, Tidy3D treats the file as version 0 and migrates it in memory to the
+current schema on load.
+
 Priority Order
 --------------
 
@@ -140,6 +144,12 @@ into the canonical directory, leaving the originals untouched unless you pass
 ``--delete-legacy``. Use ``--overwrite`` if you have already started using the
 new location and want to replace those files with the legacy versions.
 
+Use ``tidy3d config upgrade`` to preview or apply schema migrations for
+``config.toml`` and profile files. Pass ``--dry-run`` to inspect diffs or
+``--check`` in CI. Automatic write-back can be disabled with
+``TIDY3D_CONFIG_AUTO_MIGRATE=0``; use ``TIDY3D_CONFIG_FORWARD_COMPAT=strict`` to
+error on newer schema versions instead of best-effort parsing.
+
 Legacy Access Points
 --------------------
 

docs/configuration/migration.rst

@@ -29,3 +29,15 @@ Legacy Attributes
   and ``config.use_local_subpixel`` still work and set the equivalent fields in
   ``config.logging`` or ``config.simulation``. Each call raises a warning so
   you can update scripts at your own pace.
+
+Schema Versioning
+-----------------
+
+- Config files now include a root ``config_version`` to support incremental
+  schema migrations. Missing versions are treated as ``0``.
+- The loader migrates configs in memory and writes back upgraded files by
+  default. Set ``TIDY3D_CONFIG_AUTO_MIGRATE=0`` to disable auto write-back.
+- Use ``tidy3d config upgrade --dry-run`` to inspect schema diffs or
+  ``tidy3d config upgrade --check`` in CI to verify files are current.
+- Set ``TIDY3D_CONFIG_FORWARD_COMPAT=strict`` to error on newer schema versions
+  instead of best-effort parsing.

docs/configuration/nexus.rst

@@ -119,6 +119,7 @@ The base configuration file (``~/.config/tidy3d/config.toml``) stores the API ke
 .. code-block:: toml
 
    # Base config: ~/.config/tidy3d/config.toml
+   config_version = 1
    default_profile = "nexus"  # Profile to use by default
 
    [web]
@@ -129,6 +130,7 @@ The base configuration file (``~/.config/tidy3d/config.toml``) stores the API ke
 .. code-block:: toml
 
    # Nexus profile: ~/.config/tidy3d/profiles/nexus.toml
+   config_version = 1
    [web]
    api_endpoint = "http://nexus.company.com/tidy3d-api"
    website_endpoint = "http://nexus.company.com/tidy3d"

tests/config/conftest.py

@@ -12,6 +12,8 @@
     "TIDY3D_PROFILE",
     "TIDY3D_CONFIG_PROFILE",
     "TIDY3D_ENV",
+    "TIDY3D_CONFIG_AUTO_MIGRATE",
+    "TIDY3D_CONFIG_FORWARD_COMPAT",
     "SIMCLOUD_APIKEY",
     "TIDY3D_AUTH__APIKEY",
     "TIDY3D_WEB__APIKEY",

tests/config/test_versioning.py

@@ -0,0 +1,258 @@
+"""Tests for configuration schema versioning and migrations."""
+
+from __future__ import annotations
+
+import pytest
+import toml
+from click.testing import CliRunner
+
+import tidy3d as td
+from tidy3d.config import ConfigManager
+from tidy3d.config.migrations import CURRENT_CONFIG_VERSION
+from tidy3d.web.cli.app import tidy3d_cli
+
+
+def test_config_version_written_on_save(config_manager, mock_config_dir):
+    assert td.config.profile == config_manager.profile
+    config_manager.save(include_defaults=True)
+
+    config_path = mock_config_dir / "config.toml"
+    data = toml.loads(config_path.read_text(encoding="utf-8"))
+    assert data["config_version"] == CURRENT_CONFIG_VERSION
+
+
+def test_auto_migrate_disabled_does_not_write_back(mock_config_dir, monkeypatch):
+    config_path = mock_config_dir / "config.toml"
+    config_path.parent.mkdir(parents=True, exist_ok=True)
+    config_path.write_text('[logging]\nlevel = "INFO"\n', encoding="utf-8")
+
+    monkeypatch.setenv("TIDY3D_CONFIG_AUTO_MIGRATE", "0")
+
+    manager = ConfigManager(config_dir=mock_config_dir)
+    assert manager.logging.level == "INFO"
+    content = config_path.read_text(encoding="utf-8")
+    assert "config_version" not in content
+
+
+def test_forward_compat_best_effort_drops_unknown_keys(mock_config_dir):
+    config_path = mock_config_dir / "config.toml"
+    config_path.parent.mkdir(parents=True, exist_ok=True)
+    config_path.write_text(
+        "\n".join(
+            [
+                f"config_version = {CURRENT_CONFIG_VERSION + 1}",
+                "[logging]",
+                'level = "INFO"',
+                'extra_key = "ignored"',
+                "",
+            ]
+        ),
+        encoding="utf-8",
+    )
+
+    manager = ConfigManager(config_dir=mock_config_dir)
+    assert manager.logging.level == "INFO"
+
+
+def test_forward_compat_best_effort_coerces_non_dict(mock_config_dir):
+    from tests.utils import AssertLogStr
+
+    config_path = mock_config_dir / "config.toml"
+    config_path.parent.mkdir(parents=True, exist_ok=True)
+    config_path.write_text(
+        "\n".join(
+            [
+                f"config_version = {CURRENT_CONFIG_VERSION + 1}",
+                'logging = "oops"',
+                "",
+            ]
+        ),
+        encoding="utf-8",
+    )
+
+    with AssertLogStr(
+        log_level_expected="WARNING",
+        contains_str="Configuration section 'logging' should be a table",
+    ):
+        manager = ConfigManager(config_dir=mock_config_dir)
+    assert manager.logging.level == "WARNING"
+
+
+def test_forward_compat_strict_raises(mock_config_dir, monkeypatch):
+    config_path = mock_config_dir / "config.toml"
+    config_path.parent.mkdir(parents=True, exist_ok=True)
+    config_path.write_text(
+        f'config_version = {CURRENT_CONFIG_VERSION + 1}\n[logging]\nlevel = "INFO"\n',
+        encoding="utf-8",
+    )
+
+    monkeypatch.setenv("TIDY3D_CONFIG_FORWARD_COMPAT", "strict")
+
+    with pytest.raises(ValueError, match="config_version"):
+        ConfigManager(config_dir=mock_config_dir)
+
+
+def test_config_upgrade_dry_run(mock_config_dir):
+    config_path = mock_config_dir / "config.toml"
+    config_path.parent.mkdir(parents=True, exist_ok=True)
+    config_path.write_text('[web]\napikey = "token"\n', encoding="utf-8")
+
+    runner = CliRunner()
+    result = runner.invoke(tidy3d_cli, ["config", "upgrade", "--dry-run"])
+    assert result.exit_code == 0, result.output
+    assert "config_version" in result.output
+    assert "config_version" not in config_path.read_text(encoding="utf-8")
+
+
+def test_config_upgrade_profile_only_targets_profiles(mock_config_dir):
+    config_path = mock_config_dir / "config.toml"
+    config_path.parent.mkdir(parents=True, exist_ok=True)
+    base_text = '[logging]\nlevel = "INFO"\n'
+    config_path.write_text(base_text, encoding="utf-8")
+
+    profile_path = mock_config_dir / "profiles" / "dev.toml"
+    profile_path.parent.mkdir(parents=True, exist_ok=True)
+    profile_path.write_text('[logging]\nlevel = "DEBUG"\n', encoding="utf-8")
+
+    runner = CliRunner()
+    result = runner.invoke(tidy3d_cli, ["config", "upgrade", "--profile", "dev"])
+    assert result.exit_code == 0, result.output
+
+    assert config_path.read_text(encoding="utf-8") == base_text
+    assert "config_version" in profile_path.read_text(encoding="utf-8")
+
+
+def test_migration_chain_applies_and_validates(tmp_path, monkeypatch):
+    import importlib
+
+    import tomlkit
+    from pydantic import Field
+
+    from tidy3d.config import migrations as config_migrations
+    from tidy3d.config import registry as config_registry
+    from tidy3d.config.sections import ConfigSection
+
+    original_manager = config_registry._MANAGER
+    config_registry._MANAGER = None
+    try:
+
+        @config_registry.register_section("example")
+        class ExampleConfig(ConfigSection):
+            token: str = Field("default", json_schema_extra={"persist": True})
+
+        calls: list[int] = []
+
+        @config_migrations.register_migration(1)
+        def _migrate_v1_to_v2(document: tomlkit.TOMLDocument) -> None:
+            calls.append(1)
+            table = document.get("example")
+            if not isinstance(table, tomlkit.items.Table):
+                table = tomlkit.table()
+                document["example"] = table
+            if "legacy_token" in table:
+                table["token"] = table["legacy_token"]
+                del table["legacy_token"]
+            if "token" not in table:
+                table["token"] = "migrated"
+
+        monkeypatch.setattr(config_migrations, "CURRENT_CONFIG_VERSION", 2)
+        config_loader = importlib.import_module("tidy3d.config.loader")
+
+        monkeypatch.setattr(config_loader, "CURRENT_CONFIG_VERSION", 2)
+
+        config_dir = tmp_path / "config"
+        config_dir.mkdir()
+        config_path = config_dir / "config.toml"
+        config_path.write_text('[example]\nlegacy_token = "from-v0"\n', encoding="utf-8")
+
+        manager = ConfigManager(config_dir=config_dir)
+        assert manager.get_section("example").token == "from-v0"
+        assert calls == [1]
+
+        migrated = toml.loads(config_path.read_text(encoding="utf-8"))
+        assert migrated["config_version"] == 2
+        assert migrated["example"]["token"] == "from-v0"
+        assert "legacy_token" not in migrated["example"]
+    finally:
+        config_registry._SECTIONS.pop("example", None)
+        config_registry._MANAGER = original_manager
+        migrations = config_migrations._MIGRATIONS.get(1, [])
+        if "_migrate_v1_to_v2" in locals() and _migrate_v1_to_v2 in migrations:
+            migrations.remove(_migrate_v1_to_v2)
+        if not migrations:
+            config_migrations._MIGRATIONS.pop(1, None)
+        config_migrations._MIGRATION_CHAIN_VALIDATED = False
+
+
+def test_deprecated_field_warns(tmp_path, monkeypatch):
+    from pydantic import Field
+
+    from tests.utils import AssertLogStr
+    from tidy3d.config import migrations as config_migrations
+    from tidy3d.config import registry as config_registry
+    from tidy3d.config.sections import ConfigSection
+
+    original_manager = config_registry._MANAGER
+    config_registry._MANAGER = None
+    try:
+
+        @config_registry.register_section("deprecated_example")
+        class DeprecatedExample(ConfigSection):
+            old: str = Field(
+                "default",
+                json_schema_extra={
+                    "persist": True,
+                    "deprecated_in": 1,
+                    "replaced_by": "deprecated_example.new",
+                },
+            )
+
+        monkeypatch.setattr(config_migrations, "CURRENT_CONFIG_VERSION", 1)
+
+        config_dir = tmp_path / "config_dir"
+        config_dir.mkdir()
+        (config_dir / "config.toml").write_text(
+            '[deprecated_example]\nold = "value"\n', encoding="utf-8"
+        )
+
+        with AssertLogStr(
+            log_level_expected="WARNING",
+            contains_str="deprecated_example.old",
+        ):
+            ConfigManager(config_dir=config_dir)
+    finally:
+        config_registry._SECTIONS.pop("deprecated_example", None)
+        config_registry._MANAGER = original_manager
+
+
+def test_removed_field_raises(tmp_path, monkeypatch):
+    from pydantic import Field
+
+    from tidy3d.config import migrations as config_migrations
+    from tidy3d.config import registry as config_registry
+    from tidy3d.config.sections import ConfigSection
+
+    original_manager = config_registry._MANAGER
+    config_registry._MANAGER = None
+    try:
+
+        @config_registry.register_section("removed_example")
+        class RemovedExample(ConfigSection):
+            old: str = Field(
+                "default",
+                json_schema_extra={"persist": True, "removed_in": 1},
+            )
+
+        monkeypatch.setattr(config_migrations, "CURRENT_CONFIG_VERSION", 1)
+
+        config_dir = tmp_path / "config_dir"
+        config_dir.mkdir()
+        (config_dir / "config.toml").write_text(
+            '[removed_example]\nold = "value"\n', encoding="utf-8"
+        )
+
+        with pytest.raises(ValueError, match="removed in config schema v1"):
+            ConfigManager(config_dir=config_dir)
+    finally:
+        config_registry._SECTIONS.pop("removed_example", None)
+        config_registry._MANAGER = original_manager

tidy3d/config/README.md

@@ -53,7 +53,8 @@ flowchart LR
 - `sections.py` - Pydantic models for built-in sections (logging, simulation, microwave, adjoint, web, local cache, plugin container) registered via `register_section`. The bundled models inherit from the internal `ConfigSection` helper, but external code can use plain `BaseModel` subclasses. Optional handlers perform side effects. Fields mark persistence with `json_schema_extra={"persist": True}`.
 - `registry.py` - Stores section and handler registries and notifies the attached manager so new entries appear immediately.
 - `manager.py` - `ConfigManager` caches validated models, tracks runtime overrides per profile, filters persisted fields, exposes helpers such as `plugins`, `profiles`, and `format`. `SectionAccessor` routes attribute access to `update_section`.
-- `loader.py` - Resolves the config directory, loads `config.toml` and `profiles/<name>.toml`, parses environment overrides, and writes atomically through `serializer.build_document`.
+- `loader.py` - Resolves the config directory, loads `config.toml` and `profiles/<name>.toml`, parses environment overrides, applies schema migrations, and writes atomically through `serializer.build_document`.
+- `migrations.py` - Schema versioning utilities and the `vN -> vN+1` migration registry.
 - `serializer.py` - Builds stable TOML documents with descriptive comments derived from section docstrings.
 - `profiles.py` - Supplies builtin profiles merged ahead of user overrides.
 - `legacy.py` - Implements backward-compatible wrappers and deprecation warnings around the manager.
@@ -74,6 +75,14 @@ flowchart LR
 - Only fields tagged with `persist` write by default. Call `config.save(include_defaults=True)` to emit the full tree.
 - `ConfigLoader` writes files atomically and leaves a `.bak` backup while swapping.
 
+## Schema Versioning
+
+- Persisted config files include a root `config_version` key. Missing versions are treated as `0`.
+- `tidy3d.config.migrations` defines `CURRENT_CONFIG_VERSION` and a contiguous `vN -> vN+1` migration chain.
+- Loads always migrate in-memory before validation. Write-back happens after validation unless disabled with `TIDY3D_CONFIG_AUTO_MIGRATE=0`.
+- If `config_version` is newer than the installed client, the loader warns and performs a best-effort parse. Set `TIDY3D_CONFIG_FORWARD_COMPAT=strict` to raise instead.
+- Use `tidy3d config upgrade` to inspect or apply schema migrations manually.
+
 ## Debugging
 
 - `config.format()` prints the composed tree - handy for verifying merges.

tidy3d/config/__init__.py

@@ -7,6 +7,7 @@
 from . import sections  # noqa: F401 - ensure builtin sections register
 from .legacy import LegacyConfigWrapper, LegacyEnvironment, LegacyEnvironmentConfig
 from .manager import ConfigManager
+from .migrations import CURRENT_CONFIG_VERSION, register_migration
 from .registry import (
     get_handlers,
     get_sections,
@@ -16,6 +17,7 @@
 )
 
 __all__ = [
+    "CURRENT_CONFIG_VERSION",
     "ConfigManager",
     "Env",
     "Environment",
@@ -24,6 +26,7 @@
     "get_handlers",
     "get_sections",
     "register_handler",
+    "register_migration",
     "register_plugin",
     "register_section",
 ]