Commit 340f4ae
authored
perf(api): buffer asset_references writes via Redis to eliminate row-lock contention (#467)
Under high concurrency (50+ concurrent learn sessions), BatchIncrementAssetRefs
causes PostgreSQL row-level lock contention on the asset_references table,
resulting in slow queries (avg 316ms), S3 context canceled errors, and 504s.
This replaces the synchronous DB upsert in StoreMessage with an async
Redis-buffered writer that:
- Uses HINCRBY to atomically accumulate ref counts in Redis (µs latency)
- Flushes coalesced batches to DB every 1s via a background goroutine
- Falls back to synchronous DB write if Redis enqueue fails
- Restores counts to Redis on DB flush failure for automatic retry
- Drains remaining data on graceful shutdown
The writer is stateless — any pod can flush any project's pending data,
and pod crash/restart loses at most 1s of ref_count increments (which are
only used for storage analytics and GC, not hard constraints).1 parent 8720f45 commit 340f4ae
7 files changed
Lines changed: 721 additions & 24 deletions
File tree
- src/server/api/go
- cmd/server
- internal
- bootstrap
- config
- infra/assetrefwriter
- modules/service
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
138 | 139 | | |
139 | 140 | | |
140 | 141 | | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
141 | 150 | | |
142 | 151 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
215 | 216 | | |
216 | 217 | | |
217 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
218 | 240 | | |
219 | 241 | | |
220 | 242 | | |
| |||
268 | 290 | | |
269 | 291 | | |
270 | 292 | | |
| 293 | + | |
271 | 294 | | |
272 | 295 | | |
273 | 296 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
104 | 109 | | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
117 | 123 | | |
118 | 124 | | |
119 | 125 | | |
| |||
151 | 157 | | |
152 | 158 | | |
153 | 159 | | |
| 160 | + | |
| 161 | + | |
154 | 162 | | |
155 | 163 | | |
156 | 164 | | |
| |||
0 commit comments