[sql-46] trigger kvdb to sql migration using sqldb/v2#1114
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ViktorTigerstrom, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request represents a significant architectural shift, laying the groundwork for litd to transition its data persistence from BoltDB to a SQL backend. By integrating sqldb/v2 and implementing code migrations, it enables the seamless transfer of existing data for key components like accounts, sessions, and firewall rules within dev environments. This change facilitates future development on a more scalable and flexible data layer, while ensuring data integrity during the transition.
Highlights
- KVDB to SQL Migration: I've implemented the core logic to migrate existing BoltDB (KVDB) data for
accounts,sessions, andfirewalldbstores to a SQL database. This is a 'code migration' that runs as a post-migration step, ensuring existing data is transferred to the new schema. sqldb/v2Integration: This PR integrates the newsqldb/v2library for database management. This includes adopting its robust transaction execution pattern and migration framework across theaccounts,session, andfirewalldbmodules.- Separate Dev Migrations: I've introduced a mechanism for
devbuilds to have their own SQL migration streams. This allows for experimental or in-progress migrations, such as the KVDB to SQL data migration, to be tested and developed without affecting production builds. - Database Schema Adaptation: Various database interaction layers, including SQLC generated code, store interfaces, and test helpers, have been updated. This aligns them with the new
sqldb/v2patterns and introducessqlcmig6types specifically for handling the migration's intermediate schema.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This is a substantial pull request that migrates the database backend from kvdb to sqldb/v2 and introduces the necessary migration logic. The changes are well-structured, particularly the use of a separate sqlcmig6 package for migration-specific queries and the centralized migration stream handling. I've identified one critical issue that will prevent compilation, and a couple of medium-severity issues related to error message formatting. After addressing these points, the code should be in good shape.
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
78d4b40 to
01695d5
Compare
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
01695d5 to
a12d714
Compare
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
a12d714 to
ddce17e
Compare
ViktorT-11
added
the
no-changelog
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
ddce17e to
f8bfb97
Compare
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
f8bfb97 to
cae9938
Compare
In upcoming commits, other packages than the `db` package will need to be able to access the `SqlSchemas`. This commit exports it in preparation for those changes
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
cae9938 to
9aca63b
Compare
ViktorT-11
changed the base branch from
add-mig6-package
to
sql-migration-base
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
9aca63b to
e3f9794
Compare
|
@ViktorT-11, remember to re-request review from reviewers when ready |
In upcoming commits, we will introduce a new migration stream package that will need to reference the db package, as well as the accounts, session and firewalldb package in future commits. To avoid circular dependencies, we therefore introduce a new migration stream that unit tests can use, in order to avoid having to import the new migration stream package.
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
e3f9794 to
40d30ac
Compare
bitromortac
left a comment
bitromortac
left a comment
There was a problem hiding this comment.
Very cool work 🎉. Nice to see that we're getting closer to the migration.
| var ( | ||
| // LitdMigrationStream defines the SQL migration stream used to create | ||
| // MakeMigrationStreams creates the migration streams for production | ||
| // environments. | ||
| func MakeMigrationStreams(_ context.Context, _ lnrpc.LightningClient, _ string, | ||
| _ clock.Clock) []sqldb.MigrationStream { | ||
|
|
||
| // migStream defines the SQL migration stream used to create | ||
| // and upgrade LiT's SQL schema. | ||
| LitdMigrationStream = sqldb.MigrationStream{ |
There was a problem hiding this comment.
nit: you could have introduced MakeMigrationStreams before
There was a problem hiding this comment.
Kept this one specifically as is, as I personally think it makes more sense to only introduce it when it's actually needed, especially given the functions parameters that do not make sense before the commit where the function is introduced.
The upcoming kvdb to sql code migration will be added to as part of the `sqldb/v2` migration stream. However, since the kvdb to sql migration will need to use the migration functions present in the `accounts`, `firewalldb`, and `session` packages, the migration will need to referernce those packages. That would lead to a circular dependency though if the migration stream was defined in the `db` package, as those packages need to import the `db` package. To avoid this, we introduce a new `migstreams` package that contains the migration streams, and ensure that the `db` package doesn't import the `migstreams` package.
When the kvdb to sql migration is initially introduced, we will want to ensure that it is only run under dev builds during the testing phase. We therefore introduce the functionality to have separate dev migrations, which are only included in a separate migration stream that is used only in dev builds. Note that these dev migrations are currently not included in the `sqlc.yaml` file, to ensure that the main sqlc models doesn't include the dev migrations.
The next commit that adds the kvdb to sql code migration will need to initialize the accounts kvdb DB. This commit exports that struct so that it can be referenced outside of the accounts package.
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
40d30ac to
1467714
Compare
Add the necessary code to trigger the kvdb to sql code migration in dev builds.
ViktorT-11
force-pushed
the
origin/2025-07-trigger-kvdb-to-sql-migrations-in-dev-builds
branch
from
1467714 to
7bba957
Compare
ViktorT-11
merged commit 28629df
into
lightninglabs:sql-migration-base
4 participants
Based on and blocked by #1113
Implements step 3. of "Phase 3" in #917.
This PR implements the functionality to use
sqldb/v2to trigger the kvdb to sql migration under thedevbuild-taggedlitdbuilds, using code migrations whichsqldb/v2adds support for. This PR also adds support for having separatedevsql migrations, which are only run under build-taggedlitdbuilds.Note that this is only a draft PR for now, as the sqldb/v2 is not released, and we therefore need to add forked dependencies in order to run this. Once we have an official release of the sqldb/v2 package, I'll update this PR to use that dependency.