[WIP] using db-pool library to create a pool of databases by momentary-lapse · Pull Request #5846 · LemmyNet/lemmy

momentary-lapse · 2025-07-04T15:17:21Z

Addresses: #4979

momentary-lapse · 2025-07-04T15:35:57Z

+    .await;
+
+  // TODO make compatible with ActualDbPool
+  db_pool.pull_immutable().await


I created this WIP PR to share the progress and the issue I'm stuck with currently. The crate I use operates with its own structure wrapping connection pools: code
And we have our own ActualDbPool. They are kinda same, but it's not obvious for me how to correctly convert one to another.
I had an idea to make ActualDbPool a enum with two possible values: RegularPool and ReusablePool, but stuck on trying to adapt stuff like LemmyContext, which also requires pool struct to be clone-able (and ReusablePool is not). And it seems a lot of changes to the main codebase for purely test changes.
Do you folks have any ideas how to manage that? Or should I stick to the initial plan without using this library?

Our ActualDbPool is just a type alias for deadpool Pool<AsyncPgConnection>.

Their crate should be able to work with deadpool pools, but I'm not familiar with how to plug that into their crate... you'll have to ask them.

Yeah, I see. I returned to this issue today after a week of a break. I'm in contact with the db-pool author and they're helping to understand a lot of moments and really willing to collaborate, so i think we'll make this work.

I'd like to clarify one moment: do we want build_db_pool_for_tests to return still ActualDbPool? db-pool has its own wrapper ReusableConnectionPool which works like a deadpool Pool, but a bit different and needs adaptation. And it might be easier to adapt tests for working with ReusableConnectionPool than converting ReusableConnectionPool to ActualDbPool

The return type of build_db_pool_for_tests may be changed. Also, a DbPool variant may be added if needed.

momentary-lapse · 2025-07-14T23:01:45Z

 #[derive(Clone)]
 pub struct LemmyContext {
-  pool: ActualDbPool,
+  pool: ContextPool,


This is the moment which currently blocks me, and i think it's better to consult with you again. LemmyContext structure must be cloneable, therefore all the fields, therefore the pool. But unfortunately, reusable pool from db-pool crate is not, and i don't have access to its fields to implement the trait here.
But before asking db-pool developer, i'd like to be sure we really need this pool cloning stuff, especially for the tests. Cloning the pool seems a bit strange to me, but i may miss something. I'm looking at the code now, but maybe you folks already have some insights on this

Wrap it in Arc for now.

momentary-lapse · 2025-07-27T23:10:08Z

Update: I'm working on the topic; cannot devote much time for it, but it slowly going forward, and i keep the code in the branch up-to-date. I connected db-pool crate to our tests, and reworked most of them. Currently have a runtime error, planning to look at it and fix this week.
After this, what is left is to change a few tests which are using build_db_pool function.

momentary-lapse · 2026-03-10T02:31:00Z

Okay, for now i simply ignored cleanup errors in db-pool, and tests are passing now. Tried to disable migrations/2025-08-01-000017_forbid_diesel_cli/ as Nutomic said, and it defeated that diesel cli error, but another one appeared: it tried to truncate table deps_saved_ddl, but failed. It's in util schema, and maybe there's something with permissions. Anyway, planning to look into it more carefully, since now there's no guarantee that databases are even properly cleaned. But at least the global problem is located.

There's a small speed improvement comparing to main build (~4 min). But i suspect that's because one heavy test is still ignored (lemmy_diesel_utils schema_setup::tests::test_schema_setup) and it feels something is hindering tests from running truly in parallel. Even tho now they definitely interfere: i put serial back to worker tests in lemmy_apub_send, because without that directive they all tried to use the same port 8085, and failed

Nutomic · 2026-03-10T14:56:15Z

There's a small speed improvement comparing to main build (~4 min). But i suspect that's because one heavy test is still ignored (lemmy_diesel_utils schema_setup::tests::test_schema_setup) and it feels something is hindering tests from running truly in parallel.

How many tests are currently running at the same time? Not sure where that is defined, but try to change the number higher or lower and see if it helps.

Even tho now they definitely interfere: i put serial back to worker tests in lemmy_apub_send, because without that directive they all tried to use the same port 8085, and failed

It might be possible to pass the port number as parameter to make them parallel. But there are only a few tests in that crate so it can be changed later in another PR.

momentary-lapse · 2026-03-10T22:49:47Z

How many tests are currently running at the same time? Not sure where that is defined, but try to change the number higher or lower and see if it helps.

I hardcoded the pool size for tests to 30 for now. But both pool sizes (restricted and privileged) don't seem to actually affect the test speed at all.

It might be possible to pass the port number as parameter to make them parallel. But there are only a few tests in that crate so it can be changed later in another PR.

Honestly, I tried to set up each test to its own port (changed it here and here) and run them together, but they started failing and hanging. I spent some time, but then switched to other stuff.

…l-db-tests

dessalines · 2026-04-06T13:59:39Z

Failing CI: https://woodpecker.join-lemmy.org/repos/129/pipeline/19826/1

…l-db-tests

Nutomic · 2026-05-04T14:06:37Z

Tried it locally and the tests are passing now, good job! Still needs cargo +nightly fmt --all and maybe other fixes to make CI pass.

And you need to use a proper release version of the db-pool crate in crates/diesel_utils/Cargo.toml.

Edit: Locally tests complete in 104s with this change, compared to 203s for main branch. So only half the time!

Nutomic · 2026-05-04T14:07:32Z


-  #[tokio::test]
-  #[serial]
+  #[tokio_shared_rt::test(shared = true, flavor = "multi_thread")]


Suggested change

#[tokio_shared_rt::test(shared = true, flavor = "multi_thread")]

#[tokio_shared_rt::test(shared, flavor = "multi_thread")]

Can be shortened like this (with search and replace on all files).

https://docs.rs/tokio-shared-rt/latest/tokio_shared_rt/attr.test.html

Nutomic · 2026-05-04T14:10:11Z

+        },
+      )
+      .await
+      .expect("diesel postgres backend");


Dont use so many expect, instead simply use ? and return LemmyResult from this closure.

Nutomic · 2026-05-04T14:10:33Z

+
+      let backend = DieselAsyncPostgresBackend::new(
+        config,
+        |manager| Pool::builder(manager).max_size(30), //TODO use some env var


Leftover todo, anyway this is probably fine to keep hardcoded.

Nutomic · 2026-05-04T14:11:57Z


  #[test]
-  #[serial]
+  #[ignore]


Need to enable this test again (it doesnt have to run in parallel).

Nutomic · 2026-05-04T14:13:12Z

@@ -0,0 +1 @@
+


Delete this folder (it was removed from main branch but came back somehow).

Nutomic · 2026-05-04T14:14:31Z

+  let pool_arc = data.pool();
+  let pool_ref = &***pool_arc;
+  let pool = &mut pool_ref.into();


Suggested change

let pool_arc = data.pool();

let pool_ref = &***pool_arc;

let pool = &mut pool_ref.into();

let pool = &build_db_pool_for_tests().await;

let pool = &mut pool.into();

Simplify it like the other tests. In this file its all written in this overly complicated way.

…l-db-tests

momentary-lapse · 2026-05-04T23:43:55Z

Thanks for the review, I'll try to address remarks soon-ish and contact with db-pool owner to release my forked changes as a new version.

Nutomic · 2026-05-05T09:10:12Z

To fix the task which is currently failing in CI (lemmy_api_common_works_with_wasm) you need to change crates/diesel_utils/Cargo.toml. For db-pool set optional = true and then add db-pool under the full feature.

draft: using db-pool form a 2021 branch

1443731

momentary-lapse requested review from Nutomic, SleeplessOne1917, dessalines, dullbananas and phiresky as code owners July 4, 2025 15:17

momentary-lapse commented Jul 4, 2025

View reviewed changes

momentary-lapse marked this pull request as draft July 4, 2025 15:36

momentary-lapse mentioned this pull request Jul 4, 2025

Make database tests run in parallel and not fail because of other failed tests #4979

Open

momentary-lapse added 4 commits July 11, 2025 14:30

Merge branch 'main' of github.com:LemmyNet/lemmy into parallel-db-tests

55d010d

fix

91613fb

some renaming

618f635

reusable pool integrated except everywhere, some tests adapted

ef155c5

momentary-lapse commented Jul 14, 2025

View reviewed changes

momentary-lapse added 15 commits July 15, 2025 02:03

Merge branch 'main' of github.com:LemmyNet/lemmy into parallel-db-tests

f949aea

generic pool in context and stat modules, now it compiles

6b13d21

Merge branch 'main' into parallel-db-tests

8528a7f

fix taplo

e90c8be

clippy fix

5d6fe0c

Merge branch 'main' of github.com:LemmyNet/lemmy into parallel-db-tests

906c36a

clippy fix

418e91b

fmt fix

fe3305b

Merge branch 'main' of github.com:LemmyNet/lemmy into parallel-db-tests

a95ad45

shear fix

4377da8

clippy + shear fixes

e18c7f7

fmt fix

5264ab1

Merge branch 'main' of github.com:LemmyNet/lemmy into parallel-db-tests

9db404d

fix merge issues

6e795ea

config forrunning new tests locally

603efb3

Anton Boitsov added 4 commits March 10, 2026 03:23

use serial for workers which share the same port

0230818

removing ignore

73aad3e

removing ignore

3b187a8

clippy and fmt fixes

dc3c087

Anton Boitsov added 3 commits April 1, 2026 00:05

Merge branch 'main' of https://github.com/LemmyNet/lemmy into paralle…

9c00d0a

…l-db-tests

db-pool dependency library supports diesel-async 0.8.0

5dfa1df

test fixes

1bcd4ff

Anton Boitsov added 5 commits April 6, 2026 20:58

Merge branch 'main' of https://github.com/LemmyNet/lemmy into paralle…

28d869e

…l-db-tests

fmt fix

46f2d66

Merge branch 'main' of https://github.com/LemmyNet/lemmy into paralle…

fabeac1

…l-db-tests

Merge branch 'main' of https://github.com/LemmyNet/lemmy into paralle…

72aaa20

…l-db-tests

unused import removed

4fbb473

Nutomic reviewed May 4, 2026

View reviewed changes

Anton Boitsov added 2 commits May 5, 2026 02:33

fmt fix

55c3312

Merge branch 'main' of https://github.com/LemmyNet/lemmy into paralle…

7da1f07

…l-db-tests

Anton Boitsov added 3 commits May 5, 2026 03:06

fix shear warning

d1fc5ba

taplo fix

3d1ac1c

removed empty files

6a185b7

	#[tokio_shared_rt::test(shared = true, flavor = "multi_thread")]
	#[tokio_shared_rt::test(shared, flavor = "multi_thread")]

Uh oh!

Conversation

momentary-lapse commented Jul 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

momentary-lapse commented Jul 27, 2025

Uh oh!

momentary-lapse commented Mar 10, 2026

Uh oh!

Nutomic commented Mar 10, 2026

Uh oh!

momentary-lapse commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dessalines commented Apr 6, 2026

Uh oh!

Nutomic commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nutomic May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

momentary-lapse commented May 4, 2026

Uh oh!

Nutomic commented May 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

momentary-lapse commented Mar 10, 2026 •

edited

Loading

Nutomic commented May 4, 2026 •

edited

Loading

Nutomic May 4, 2026 •

edited

Loading