Skip to content

Migrate DB to PostgreSQL #710

Open
@ConnorNelson

Description

@ConnorNelson

Preliminary testing suggests massive (hopefully easy) performance wins.

For example, a query like this has gone from ~3mins to ~3s:

SELECT
  COUNT(solves.id)               AS total_solves,
  COUNT(DISTINCT solves.user_id) AS total_users
FROM submissions
JOIN solves
  ON submissions.id = solves.id

JOIN dojo_challenges
  ON dojo_challenges.challenge_id = solves.challenge_id

JOIN dojo_modules
  ON dojo_modules.dojo_id      = dojo_challenges.dojo_id
 AND dojo_modules.module_index = dojo_challenges.module_index

LEFT JOIN dojo_users
  ON dojo_users.user_id = solves.user_id
 AND dojo_users.dojo_id = dojo_challenges.dojo_id

JOIN dojos
  ON dojos.dojo_id = dojo_challenges.dojo_id
 AND (
       dojos.official
    OR (dojos.data::json->>'type') = 'public'
    OR dojo_users.user_id IS NOT NULL
  )

JOIN users
  ON users.id = solves.user_id

LEFT JOIN dojo_challenge_visibilities
  ON dojo_challenge_visibilities.dojo_id         = dojo_challenges.dojo_id
 AND dojo_challenge_visibilities.module_index    = dojo_challenges.module_index
 AND dojo_challenge_visibilities.challenge_index = dojo_challenges.challenge_index

WHERE
  (dojo_challenge_visibilities.start IS NULL
    OR submissions.date >= dojo_challenge_visibilities.start)
  AND (dojo_challenge_visibilities.stop IS NULL
    OR submissions.date <= dojo_challenge_visibilities.stop)
  AND users.hidden = FALSE
  AND (dojo_users.type IS NULL OR dojo_users.type <> 'admin')
  AND dojo_challenges.dojo_id = -263864626

LIMIT 1;

Unfortunately I don't think we can just literally swap out the db, our sqlalchemy layer will have to adapt a bit, but hopefully that isn't so bad.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions