Back-End Problem Research Documentation

[divine-comedian]

Research and report on current bottlenecks of giveth back-end infrastructure. Started a document in notion here:
https://www.notion.so/giveth/Back-End-Improvements-1913ab28d48c80ff8dd7d4d643d2960a?pvs=4

Topics so far discussed worth exploring:

• how can we prevent resource usage spikes for good
• prevent fixes or new features from bring down the whole back-end
• identify issues with adminBro
• reduce complexity of codebase making it easier to debug
• improve logging from jobs and functions to make debugging easier
• when a big feature arrives it seems to break everything
• reducing size of back-end, split into smaller modular repos?
• do more cacheing
• optimize queries
• more queries that each call smaller amounts of data
• raw queries vs. ORM queries
• views taking a long time to load
• reduce amount of information called on views
• admin functions getting stuck
• improve vs. re-make from scratch

Objective

• Validate key problem statements
• Provide associated data from logs, monitoring tools etc
• Perform root cause analysis
• document everything
• propose solutions and recommendations
• considerations on future features we plan to integrate

Outcome

• estimate the amount of work needed to complete overhaul
• decide if we focus on making major refactors or re-make it from scratch

[mhmdksh]

@divine-comedian @geleeroyale We had an old epic that might be useful to add to this research.

Here is the link: https://github.com/Giveth/devops/issues/236

I have updated it to include some of the practices that we can deploy based on our past experience with running qacc and other backends.