Add default board to new projects, remove uncategorized pseudo-board

[denyskon]

On creation of an empty project (no template) a default board will be created instead of falling back to the uneditable pseudo-board.

Every project now has to have exactly one default boards. As a consequence, you cannot unset a board as default, instead you have to set another board as default. Existing projects will be modified using a cron job, additionally this check will run every midnight by default.

Deleting the default board is not allowed, you have to set another board as default to do it.

Fixes #29873
Fixes #14679 along the way
Fixes #29853

Co-authored-by: delvh dev.lh@web.de

[delvh]

Wait, the default board is a pseudo-board?
Where is it set?

[denyskon]

@delvh

gitea/models/project/board.go

Line 258 in 16e3600

// represents a board for issues not assigned to one

[denyskon]

@lng2020 The default logic will apply if all columns are deleted which in fact can easily happen. Also, we cannot possibly do this change retroactively for all existing projects which currently display this default pseudo-column.

[lunny]

Do we really need the configuration? Maybe we can hardcode it and make it an option when we think it's necessary.

[denyskon]

@go-gitea/technical-oversight-committee I would like to ask for a TOC vote on which of the two strategies should be continued with, as the discussion here doesn't seem to be going anywhere.

The two possible solutions are:

1. Keep the consistency check ("self-healing") on every project viewing -> don't display errors but try to fix them (suggested by @delvh and @wxiaoguang) -> current state of this PR
2. Only keep the migration, consider every inconsistency in DB an issue which should be reported instead of trying to fix it automatically (proposed by @lunny) -> basically a revert to the state of Add default board to new projects, remove uncategorized pseudo-board #29874 (comment)

If there is no decision made by the TOC I would like for the PR to be either merged as-it-is (it has two necessary approvals and no blockers) or to be abandoned if no consensus is found, however then we'll remain with a state nobody likes.

[delvh]

I'm fine either way.
This PR was already changed way too often.
So, from my side, let's merge it (after the small bugfix below).
I don't know what others think about it.

[denyskon]

I think with 3 approvals, we can merge as-it-is.

[silverwind]

Unit tests show some data race error, not sure if related. Restarted it once.

[wxiaoguang]

Unit tests show some data race error, not sure if related. Restarted it once.

Not related

[lunny]

@go-gitea/technical-oversight-committee I would like to ask for a TOC vote on which of the two strategies should be continued with, as the discussion here doesn't seem to be going anywhere.

The two possible solutions are:

1. Keep the consistency check ("self-healing") on every project viewing -> don't display errors but try to fix them (suggested by @delvh and @wxiaoguang) -> current state of this PR

2. Only keep the migration, consider every inconsistency in DB an issue which should be reported instead of trying to fix it automatically (proposed by @lunny) -> basically a revert to the state of [Add default board to new projects, remove uncategorized pseudo-board #29874 (comment)](https://github.com/go-gitea/gitea/pull/29874#issuecomment-2016526148)

If there is no decision made by the TOC I would like for the PR to be either merged as-it-is (it has two necessary approvals and no blockers) or to be abandoned if no consensus is found, however then we'll remain with a state nobody likes.

I will not block this.

[6543]

I'd like to have pagination in this migration!

[denyskon]

Honestly I have no idea what you mean or how to do it 🙈

[6543]

Itterate over it in batches ... just look at older migrations

[6543]

e.g.

gitea/models/migrations/v1_21/v276.go

Line 55 in 274bc00

limit := setting.Database.IterateBufferSize

[denyskon]

Oh so the thing I did before 😆

[6543]

Uh you did?

[delvh]

Yep, and then it was removed again.
Also, I don't quite see the benefit of batching here.
How does it help with anything?

[silverwind]

I guess the only reason to batch is when amount of project boards is really massive and the SQL operation would risk running into timeouts maybe.

On the other hand, batched also brings the problem that the operation is not atomic, but if it's idempotent, it wouldn't matter when it's interrupted in the middle.

[denyskon]

Done

[silverwind]

@6543 looking good to you?

[denyskon]

Finally 🎉

[wxiaoguang]

Sorry but this migration seems broken ....

If the "SELECT" results are stable and never change, this for loop is right.

BUT, by each SELECT and for-loop, the "project boards" are changed. The next SELECT, it should still process the first page results, but not skip start.

For example: 150 projects.

1. First loop: SELECT gets 1-50 projects and updates them.
2. Second loop: SELECT gets "the second page" from "51-150", then it gets 101-150 projects.
3. Then the 51-100 projects are never migrated.

[lunny]

I will send a PR to fix it.

[lunny]

#30153

[melroy89]

@lunny I believe I still have an issue with this migration. It says: "Ensure every project has exactly one default column". It tries to do some SQL queriy on the project and project_board tables. Migration seems to fail with error: pq: invalid input syntax for type bigint: "true". Help?? I'm stuck on migration. My docker container keeps restating in a loop now. This happens after I updated to v1.22. On version 1.21 everything worked fine.

I'm using Docker gitea/gitea:latest-rootless image with PostgreSQL v14.

2024/06/12 13:41:30 ...ations/migrations.go:642:Migrate() [I] [SQL] SELECT tablename FROM pg_tables WHERE schemaname = $1 [public] - 371.738µs
2024/06/12 13:41:30 .../xorm@v1.3.8/sync.go:30:Sync() [I] [SQL] SELECT column_name, column_default, is_nullable, data_type, character_maximum_length, description,
    CASE WHEN p.contype = 'p' THEN true ELSE false END AS primarykey,
    CASE WHEN p.contype = 'u' THEN true ELSE false END AS uniquekey
FROM pg_attribute f
    JOIN pg_class c ON c.oid = f.attrelid JOIN pg_type t ON t.oid = f.atttypid
    LEFT JOIN pg_attrdef d ON d.adrelid = c.oid AND d.adnum = f.attnum
    LEFT JOIN pg_description de ON f.attrelid=de.objoid AND f.attnum=de.objsubid
    LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
    LEFT JOIN pg_constraint p ON p.conrelid = c.oid AND f.attnum = ANY (p.conkey)
    LEFT JOIN pg_class AS g ON p.confrelid = g.oid
    LEFT JOIN INFORMATION_SCHEMA.COLUMNS s ON s.column_name=f.attname AND c.relname=s.table_name
WHERE n.nspname= s.table_schema AND c.relkind = 'r' AND c.relname = $1 AND s.table_schema = $2 AND f.attnum > 0 ORDER BY f.attnum; [version public] - 2.886482ms
2024/06/12 13:41:30 .../xorm@v1.3.8/sync.go:30:Sync() [I] [SQL] SELECT indexname, indexdef FROM pg_indexes WHERE tablename=$1 AND schemaname=$2 [version public] - 431.557µs
2024/06/12 13:41:30 routers/common/db.go:31:InitDBEngine() [W] Table version column id db type is BIGINT, struct type is BIGSERIAL
2024/06/12 13:41:30 ...orm@v1.3.8/engine.go:1252:Get() [I] [SQL] SELECT "id", "version" FROM "version" WHERE "id"=$1 LIMIT 1 [1] - 223.978µs
2024/06/12 13:41:30 ...ations/migrations.go:688:Migrate() [I] Migration[293]: Ensure every project has exactly one default column
2024/06/12 13:41:30 routers/common/db.go:46:migrateWithSetting() [I] [SQL] BEGIN TRANSACTION [] - 53.6µs
2024/06/12 13:41:30 ...ations/v1_22/v293.go:55:CheckProjectColumnsConsistency() [I] [SQL] SELECT project.id as id, project.creator_id, project_board.id as board_id FROM "project" LEFT JOIN "project_board" ON project_board.project_id = project.id AND project_board."default"=$1 WHERE (project_board.id is NULL OR project_board.id = 0) LIMIT 50 [true] - 210.268µs
2024/06/12 13:41:30 ...ations/migrations.go:691:Migrate() [I] [SQL] ROLLBACK [] - 54.09µs
2024/06/12 13:41:30 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #8/10 failed. Error: migrate: migration[293]: Ensure every project has exactly one default column failed: pq: invalid input syntax for type bigint: "true"
2024/06/12 13:41:30 routers/common/db.go:37:InitDBEngine() [I] Backing off for 3 seconds
2024/06/12 13:41:31 ...eful/manager_unix.go:144:handleSignals() [W] PID 7. Received SIGTERM. Shutting down...

When I currently dump the project & project_board schema, it looks like this (hope this helps):

Project table:

CREATE TABLE "public"."project" (
    "id" bigint DEFAULT GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    "title" text NOT NULL,
    "description" text,
    "repo_id" bigint,
    "creator_id" bigint NOT NULL,
    "is_closed" boolean,
    "board_type" bigint,
    "type" bigint,
    "created_unix" bigint,
    "updated_unix" bigint,
    "closed_date_unix" bigint,
    "card_type" integer DEFAULT '0' NOT NULL,
    "owner_id" bigint,
    CONSTRAINT "project_pkey" PRIMARY KEY ("id")
) WITH (oids = false);

Project board table:

CREATE TABLE "public"."project_board" (
    "id" bigint DEFAULT GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    "title" text,
    "default" bigint DEFAULT '0' NOT NULL,
    "project_id" bigint NOT NULL,
    "creator_id" bigint NOT NULL,
    "created_unix" bigint,
    "updated_unix" bigint,
    "sorting" integer DEFAULT '0' NOT NULL,
    "color" character varying(7),
    CONSTRAINT "project_board_pkey" PRIMARY KEY ("id")
) WITH (oids = false);