Skip to content

feat: support index based filename naming strategy#1443

Merged
Shinigami92 merged 12 commits into
salsita:mainfrom
adrenalin:feat-index-based-naming-strategy
Jul 3, 2025
Merged

feat: support index based filename naming strategy#1443
Shinigami92 merged 12 commits into
salsita:mainfrom
adrenalin:feat-index-based-naming-strategy

Conversation

@adrenalin

Copy link
Copy Markdown
Contributor

I added a new prefix option: incremental index (as the already existing test cases indicate it is often more convenient to use short numbers for filenames rather than 13+ numbers)

Prefix "index" cannot be used in conjunction with other types since it is a small integer that cannot be in practice compared to timestamp based prefixes, so there are safeguards set in place to prevent mixed usage

@adrenalin adrenalin requested a review from Shinigami92 as a code owner June 21, 2025 10:54
@Shinigami92 Shinigami92 changed the title Support index based filename naming strategy feat: support index based filename naming strategy Jun 21, 2025
@Shinigami92 Shinigami92 added this to the vAnytime milestone Jun 21, 2025
@github-actions

github-actions Bot commented Jun 21, 2025

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🟢 Lines 92.68% (🎯 90%)
⬆️ +0.05%
3258 / 3515
🟢 Statements 92.68% (🎯 90%)
⬆️ +0.05%
3258 / 3515
🟢 Functions 96.7% (🎯 90%)
⬆️ +0.01%
264 / 273
🟢 Branches 91.09% (🎯 85%)
⬇️ -0.10%
859 / 943
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/migration.ts 76.07%
⬆️ +2.97%
86.56%
⬇️ -0.71%
68.75%
⬆️ +2.09%
76.07%
⬆️ +2.97%
149-152, 173-189, 231-237, 296-337, 402-403, 429-431, 439-440, 457-460, 489-490
Unchanged Files
src/db.ts 83.18% 88.88% 87.5% 83.18% 90-92, 122, 125-137, 177-179
src/index.ts 100% 100% 100% 100%
src/logger.ts 100% 100% 100% 100%
src/migrationBuilder.ts 96.19% 93.33% 100% 96.19% 833-837, 976-981
src/migrationOptions.ts 100% 100% 100% 100%
src/pgType.ts 100% 100% 100% 100%
src/runner.ts 82.13% 67.79% 100% 82.13% 186, 206-207, 219-220, 272-273, 315-318, 327-331, 344, 356-359, 382, 393, 423-436, 439-442, 462-464, 473, 475-484
src/sqlMigration.ts 85.36% 100% 80% 85.36% 54-60
src/operations/sql.ts 100% 100% 100% 100%
src/operations/casts/createCast.ts 100% 100% 100% 100%
src/operations/casts/dropCast.ts 100% 100% 100% 100%
src/operations/casts/index.ts 100% 100% 100% 100%
src/operations/domains/alterDomain.ts 100% 100% 100% 100%
src/operations/domains/createDomain.ts 100% 100% 100% 100%
src/operations/domains/dropDomain.ts 100% 100% 100% 100%
src/operations/domains/index.ts 100% 100% 100% 100%
src/operations/domains/renameDomain.ts 100% 100% 100% 100%
src/operations/domains/shared.ts 100% 100% 100% 100%
src/operations/extensions/createExtension.ts 100% 100% 100% 100%
src/operations/extensions/dropExtension.ts 100% 100% 100% 100%
src/operations/extensions/index.ts 100% 100% 100% 100%
src/operations/extensions/shared.ts 100% 100% 100% 100%
src/operations/functions/createFunction.ts 95.52% 94.44% 100% 95.52% 71-73
src/operations/functions/dropFunction.ts 100% 100% 100% 100%
src/operations/functions/index.ts 100% 100% 100% 100%
src/operations/functions/renameFunction.ts 100% 100% 100% 100%
src/operations/functions/shared.ts 100% 100% 100% 100%
src/operations/grants/grantOnSchemas.ts 100% 100% 100% 100%
src/operations/grants/grantOnTables.ts 100% 100% 100% 100%
src/operations/grants/grantRoles.ts 100% 100% 100% 100%
src/operations/grants/index.ts 100% 100% 100% 100%
src/operations/grants/revokeOnSchemas.ts 100% 100% 100% 100%
src/operations/grants/revokeOnTables.ts 100% 100% 100% 100%
src/operations/grants/revokeRoles.ts 100% 100% 100% 100%
src/operations/grants/shared.ts 95.45% 85.71% 100% 95.45% 71
src/operations/indexes/createIndex.ts 100% 100% 100% 100%
src/operations/indexes/dropIndex.ts 100% 100% 100% 100%
src/operations/indexes/index.ts 100% 100% 100% 100%
src/operations/indexes/shared.ts 88% 86.95% 100% 88% 23, 33-36, 48
src/operations/materializedViews/alterMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/createMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/dropMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/index.ts 100% 100% 100% 100%
src/operations/materializedViews/refreshMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/renameMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/renameMaterializedViewColumn.ts 100% 100% 100% 100%
src/operations/materializedViews/shared.ts 100% 87.5% 100% 100%
src/operations/operators/addToOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/createOperator.ts 100% 100% 100% 100%
src/operations/operators/createOperatorClass.ts 100% 83.33% 100% 100%
src/operations/operators/createOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/dropOperator.ts 100% 100% 100% 100%
src/operations/operators/dropOperatorClass.ts 100% 100% 100% 100%
src/operations/operators/dropOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/index.ts 100% 100% 100% 100%
src/operations/operators/removeFromOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/renameOperatorClass.ts 100% 100% 100% 100%
src/operations/operators/renameOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/shared.ts 85% 75% 100% 85% 24-25, 38
src/operations/policies/alterPolicy.ts 100% 100% 100% 100%
src/operations/policies/createPolicy.ts 100% 100% 100% 100%
src/operations/policies/dropPolicy.ts 100% 100% 100% 100%
src/operations/policies/index.ts 100% 100% 100% 100%
src/operations/policies/renamePolicy.ts 100% 100% 100% 100%
src/operations/policies/shared.ts 100% 100% 100% 100%
src/operations/roles/alterRole.ts 100% 100% 100% 100%
src/operations/roles/createRole.ts 100% 75% 100% 100%
src/operations/roles/dropRole.ts 100% 100% 100% 100%
src/operations/roles/index.ts 100% 100% 100% 100%
src/operations/roles/renameRole.ts 100% 100% 100% 100%
src/operations/roles/shared.ts 98.07% 76.92% 100% 98.07% 78
src/operations/schemas/createSchema.ts 100% 100% 100% 100%
src/operations/schemas/dropSchema.ts 100% 100% 100% 100%
src/operations/schemas/index.ts 100% 100% 100% 100%
src/operations/schemas/renameSchema.ts 100% 100% 100% 100%
src/operations/sequences/alterSequence.ts 93.75% 75% 100% 93.75% 23
src/operations/sequences/createSequence.ts 100% 100% 100% 100%
src/operations/sequences/dropSequence.ts 100% 100% 100% 100%
src/operations/sequences/index.ts 100% 100% 100% 100%
src/operations/sequences/renameSequence.ts 100% 100% 100% 100%
src/operations/sequences/shared.ts 78.57% 68.75% 100% 78.57% 41, 43-44, 49-50, 63-64, 69-70
src/operations/tables/addColumns.ts 100% 80% 100% 100%
src/operations/tables/addConstraint.ts 100% 100% 100% 100%
src/operations/tables/alterColumn.ts 89.33% 76.47% 100% 89.33% 69, 76-83
src/operations/tables/alterTable.ts 100% 100% 100% 100%
src/operations/tables/createTable.ts 89.39% 77.27% 100% 89.39% 51-55, 92-93
src/operations/tables/dropColumns.ts 100% 100% 100% 100%
src/operations/tables/dropConstraint.ts 100% 100% 100% 100%
src/operations/tables/dropTable.ts 100% 100% 100% 100%
src/operations/tables/index.ts 100% 100% 100% 100%
src/operations/tables/renameColumn.ts 100% 100% 100% 100%
src/operations/tables/renameConstraint.ts 100% 100% 100% 100%
src/operations/tables/renameTable.ts 100% 100% 100% 100%
src/operations/tables/shared.ts 83.15% 81.81% 80% 83.15% 157-158, 211-215, 241-242, 261-262, 287-288, 291-298, 301-302, 439-464
src/operations/triggers/createTrigger.ts 86.25% 68.18% 100% 86.25% 53-54, 66-67, 70-71, 74-77, 113
src/operations/triggers/dropTrigger.ts 100% 100% 100% 100%
src/operations/triggers/index.ts 100% 100% 100% 100%
src/operations/triggers/renameTrigger.ts 100% 100% 100% 100%
src/operations/triggers/shared.ts 100% 100% 100% 100%
src/operations/types/addTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/addTypeValue.ts 100% 100% 100% 100%
src/operations/types/createType.ts 100% 100% 100% 100%
src/operations/types/dropType.ts 100% 100% 100% 100%
src/operations/types/dropTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/index.ts 100% 100% 100% 100%
src/operations/types/renameType.ts 100% 100% 100% 100%
src/operations/types/renameTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/renameTypeValue.ts 100% 100% 100% 100%
src/operations/types/setTypeAttribute.ts 100% 100% 100% 100%
src/operations/views/alterView.ts 100% 100% 100% 100%
src/operations/views/alterViewColumn.ts 100% 100% 100% 100%
src/operations/views/createView.ts 100% 100% 100% 100%
src/operations/views/dropView.ts 100% 100% 100% 100%
src/operations/views/index.ts 100% 100% 100% 100%
src/operations/views/renameView.ts 100% 100% 100% 100%
src/operations/views/shared.ts 100% 66.66% 100% 100%
src/utils/PgLiteral.ts 90.47% 100% 80% 90.47% 44-45
src/utils/StringIdGenerator.ts 100% 100% 100% 100%
src/utils/createSchemalize.ts 100% 100% 100% 100%
src/utils/createTransformer.ts 100% 100% 100% 100%
src/utils/decamelize.ts 100% 100% 100% 100%
src/utils/escapeValue.ts 100% 100% 100% 100%
src/utils/formatLines.ts 100% 100% 100% 100%
src/utils/formatParams.ts 100% 100% 100% 100%
src/utils/formatPartitionColumns.ts 100% 100% 100% 100%
src/utils/getMigrationTableSchema.ts 100% 100% 100% 100%
src/utils/getSchemas.ts 100% 100% 100% 100%
src/utils/identity.ts 100% 100% 100% 100%
src/utils/index.ts 100% 100% 100% 100%
src/utils/intersection.ts 100% 100% 100% 100%
src/utils/makeComment.ts 100% 100% 100% 100%
src/utils/quote.ts 100% 100% 100% 100%
src/utils/toArray.ts 100% 100% 100% 100%
src/utils/types.ts 100% 100% 100% 100%
Generated in workflow #2169 for commit 507fb3b by the Vitest Coverage Report Action

Comment thread src/migration.ts Outdated
Comment thread src/migration.ts Outdated
@Shinigami92 Shinigami92 self-requested a review June 23, 2025 08:50
@Shinigami92 Shinigami92 added the c: feature Request for new feature label Jun 23, 2025
@Shinigami92

Copy link
Copy Markdown
Collaborator

tipp: try to run pnpm run preflight locally, this will run all checks

@adrenalin

adrenalin commented Jun 24, 2025

Copy link
Copy Markdown
Contributor Author

To me it appears that the integration tests are broken. They fail with error message Cannot determine numeric prefix for "2-test-migration.ts" and the test case name does not conform to the standard style (as I have understood) ^(?<prefix>\d)+(?<separator>_)(?<description>.+)\.(ts|js|sql)$. There is a hard-coded separator (row 56, ./src/migration.ts, defines const SEPARATOR = '_';) but the integration test uses - instead of _.

I suggest either modifying the integration test file names or using a regular expression to get the number from the filename (I can modify it to the current pull request since this is a breaking change).

@Shinigami92

Copy link
Copy Markdown
Collaborator

To me it appears that the integration tests are broken. They fail with error message Cannot determine numeric prefix for "2-test-migration.ts" and the test case name does not conform to the standard style (as I have understood) ^(?<prefix>\d)+(?<separator>_)(?<description>.+)\.(ts|js|sql)$. There is a hard-coded separator (row 56, ./src/migration.ts, defines const SEPARATOR = '_';) but the integration test uses - instead of _.

I suggest either modifying the integration test file names or using a regular expression to get the number from the filename (I can modify it to the current pull request since this is a breaking change).

I fear touching existing migration tests, as that could break existing workflows in the user-base.
I also think that having such a filename could be a valid case, as the user can just create files manually and the get processed in order.

Creating files via CLI is "just" a UX. (At least I assumed that so far 👀)

@adrenalin

Copy link
Copy Markdown
Contributor Author

Then I suggest that I will use the first digits as the prefix aka revision identifier. There is nonetheless a conflict with hard-coded separator and current test cases. Waiting for your confirmation on this.

@adrenalin

Copy link
Copy Markdown
Contributor Author

And I do use cli for the revisions. It's a habit learned by using Alembic for a long time for migrations, since it uses short hashes for tree-like migration versioning kind of in the spirit of Git commit identifiers.

@Shinigami92

Copy link
Copy Markdown
Collaborator

Then I suggest that I will use the first digits as the prefix aka revision identifier. There is nonetheless a conflict with hard-coded separator and current test cases. Waiting for your confirmation on this.

So, I re-checked (also to check my sanity/assumption 🤣), but yes

'Prefix type of migration filename (Only valid with the create action)',

So, the point is, this parameter is only for creation, but for consuming, I think it is just taken as in order of the folders content. Nowadays even with sub-directories as far as I remember, as there were PRs for that recently.

So we cant add breaking stuff right now when adding the feature for index based filename. But you could potentially assume, that if users use that newly strategy, the use it consistently. (hopefully)

But we (you 😉) need to find a way, to implement the feature without breaking current behavior.

@adrenalin

Copy link
Copy Markdown
Contributor Author

So, the point is, this parameter is only for creation, but for consuming, I think it is just taken as in order of the folders content. Nowadays even with sub-directories as far as I remember, as there were PRs for that recently.

So we cant add breaking stuff right now when adding the feature for index based filename. But you could potentially assume, that if users use that newly strategy, the use it consistently. (hopefully)

How would it sound then that instead of index it would be called increment, which would add one to the last found prefix?

It would be backwards compatible with both UTC and epoch as well as supporting starting from 0001. There is already a natural sorting since sort is done against the numeric representation anyway, so padding to meet a length is also completely optional.

It should - to support the miscellaneous existing custom naming conventiona - also get any numeric prefix regardless of the separator, so that the current acceptance tests do not fail and users don't need to migrate to anything

@Shinigami92

Copy link
Copy Markdown
Collaborator

So, the point is, this parameter is only for creation, but for consuming, I think it is just taken as in order of the folders content. Nowadays even with sub-directories as far as I remember, as there were PRs for that recently.
So we cant add breaking stuff right now when adding the feature for index based filename. But you could potentially assume, that if users use that newly strategy, the use it consistently. (hopefully)

How would it sound then that instead of index it would be called increment, which would add one to the last found prefix?

It would be backwards compatible with both UTC and epoch as well as supporting starting from 0001. There is already a natural sorting since sort is done against the numeric representation anyway, so padding to meet a length is also completely optional.

It should - to support the miscellaneous existing custom naming conventiona - also get any numeric prefix regardless of the separator, so that the current acceptance tests do not fail and users don't need to migrate to anything

Sounds good to me 🙂 I wont even care if we name it increment or index 🤷
The outcome is slightly the same when you use it only, but smart if you switch to it

adrenalin added 3 commits July 1, 2025 11:18
So that the existing use cases do not break I changed now the strategy to just increment by one the last found migration prefix and to keep the original prefix length for cohesion. I added also a sane starting point (i.e. start from '0001' if no existing migrations were found).
Comment thread test/migration.spec.ts
@Shinigami92 Shinigami92 added this pull request to the merge queue Jul 3, 2025
@Shinigami92

Copy link
Copy Markdown
Collaborator

@adrenalin do you also want now directly to work on the iso mode?

Merged via the queue into salsita:main with commit bd49ad9 Jul 3, 2025
41 checks passed
@adrenalin

adrenalin commented Jul 3, 2025

Copy link
Copy Markdown
Contributor Author

@adrenalin do you also want now directly to work on the iso mode?

@Shinigami92, as you pointed out earlier this is another pull request and I think that it would require some discussion first. I will open either a thread or an issue so we can have - with luck - more people participating and at least it will leave a track.

I noticed that the code coverage could be improved, so I could in the meanwhile write some unit tests (looking in particular at migration.ts with coverage of 76% of the statements). I already saw that there would be at least some testable cases there.

Then there's the issue that it appears that it is not possible to define explicitly the column for foreign keys with an object. It appears that it is only possible to refer with a string (schema.table(column)) or otherwise the referred column will be implicitly the same as the referring column name, since reference is now a Name type that allows only keys schema and name for objects. I prefer to use explicit over implicit, so this is one case that I had thought of giving a glance.

How should I proceed with these in terms of communication, discussions or issues?

@Shinigami92

Copy link
Copy Markdown
Collaborator

@adrenalin do you also want now directly to work on the iso mode?

@Shinigami92, as you pointed out earlier this is another pull request and I think that it would require some discussion first. I will open either a thread or an issue so we can have - with luck - more people participating and at least it will leave a track.

I noticed that the code coverage could be improved, so I could in the meanwhile write some unit tests (looking in particular at migration.ts with coverage of 76% of the statements). I already saw that there would be at least some testable cases there.

Then there's the issue that it appears that it is not possible to define explicitly the column for foreign keys with an object. It appears that it is only possible to refer with a string (schema.table(column)) or otherwise the referred column will be implicitly the same as the referring column name, since reference is now a Name type that allows only keys schema and name for objects. I prefer to use explicit over implicit, so this is one case that I had thought of giving a glance.

How should I proceed with these in terms of communication, discussions or issues?

IMO issues are very good for more short-time paced point-target discussions, while discussions are more for general feedback and longer discussion-flows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: feature Request for new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants