Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0 #29964

onepack · 2020-09-09T15:57:11Z

Preconditions (*)

Magento 2.4.0
PHP 7.3 and 7.4
Redis enabled for FPC and backend cache
Elasticsearch 7.8.1
Database mariadb 10.4
8gb to 10gb of VPS ram with Magento having 2gb assigned.

Steps to reproduce (*)

Set the shop to production mode
Have around 2000+ products in multiple categories
Clear all caches and also redis-cli flushall
Open a dropdown with categories and click them open in a new tab one by one.
Search a couple of times and wait for the result.

Expected result (*)

The expected result is that all 15 categories would open one by one and look ok
The search result would look ok.

Actual result (*)

The actual result is that maybe 12 categories would open ok but around 2 or 3 would look completely broken with broken styling or even a 500 server error.
The search result would look broken.
Sometimes the OOM killer was triggered that would sacrifice the Elasticsearch etc.

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

Severity: S0 - Affects critical data or functionality and leaves users without workaround.
[X ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

We found out via SSH top that the RAM consumption was peaking during the category generation on the frontend (from uncachd to cached).
First we thought it was a Redis issue because 9 out of ten times the page would look broken like we have seen in the M1 days because of a Redis issue.
But actually "the broken page layout" was the result of PHP running out of memory.
The result would get cached and stay in the cache until the redis cache was cleared.

After a lot of digging I found the source of the issue in:
vendor/magento/module-csp/Model/BlockCache.php
Under the function: load($identifier)
This function uses a crazy amound of RAM.
Quick workaround is disabling the module to keep the shop RAM usage healthy.

The log errors show that this function is causing issues.

I copied the log errors from another user running into the same issue:

[Mon Aug 17 19:40:50.052935 2020] [php7:error] [pid 16932:tid 140669993522944] [client 127.0.0.1:40654] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 48238592 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 24
[Mon Aug 17 19:40:51.592030 2020] [php7:error] [pid 16933:tid 140670169769728] [client 127.0.0.1:40666] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:41:12.706529 2020] [php7:error] [pid 17146:tid 140670127806208] [client 127.0.0.1:40702] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:41:12.740424 2020] [php7:error] [pid 17146:tid 140670127806208] [client 127.0.0.1:40702] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 122880 bytes) in Unknown on line 0
[Mon Aug 17 19:41:24.887257 2020] [php7:error] [pid 16939:tid 140670111020800] [client 127.0.0.1:40798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 78
[Mon Aug 17 19:41:24.913106 2020] [php7:error] [pid 16939:tid 140670111020800] [client 127.0.0.1:40798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 65536 bytes) in Unknown on line 0
[Mon Aug 17 19:41:39.997127 2020] [php7:error] [pid 16933:tid 140670152984320] [client 127.0.0.1:40904] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108, referer: http://ip-of-my-ec2-instance:80/
[Mon Aug 17 19:41:40.030659 2020] [php7:error] [pid 16933:tid 140670152984320] [client 127.0.0.1:40904] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 40960 bytes) in Unknown on line 0, referer: http://ip-of-my-ec2-instance:80/
[Mon Aug 17 19:42:08.849538 2020] [php7:error] [pid 17146:tid 140670111020800] [client 127.0.0.1:41016] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 42
[Mon Aug 17 19:42:08.865389 2020] [php7:error] [pid 17146:tid 140670111020800] [client 127.0.0.1:41016] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 32768 bytes) in Unknown on line 0
[Mon Aug 17 19:44:23.245570 2020] [php7:error] [pid 17146:tid 140670027093760] [client 127.0.0.1:41798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 42, referer: https://www.google.com/
[Mon Aug 17 19:44:23.312177 2020] [php7:error] [pid 17146:tid 140670027093760] [client 127.0.0.1:41798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 45056 bytes) in Unknown on line 0, referer: https://www.google.com/
[Mon Aug 17 19:44:29.224653 2020] [php7:error] [pid 16939:tid 140670052271872] [client 127.0.0.1:41742] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:44:29.264725 2020] [php7:error] [pid 16939:tid 140670052271872] [client 127.0.0.1:41742] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 90112 bytes) in Unknown on line 0
[Mon Aug 17 19:44:34.254009 2020] [php7:error] [pid 17146:tid 140670043879168] [client 127.0.0.1:41688] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:44:34.294439 2020] [php7:error] [pid 17146:tid 140670043879168] [client 127.0.0.1:41688] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 36864 bytes) in Unknown on line 0
[Mon Aug 17 19:44:40.650547 2020] [php7:error] [pid 17146:tid 140670010308352] [client 127.0.0.1:41882] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 42
[Mon Aug 17 19:44:40.690603 2020] [php7:error] [pid 17146:tid 140670010308352] [client 127.0.0.1:41882] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 49152 bytes) in Unknown on line 0
[Mon Aug 17 19:44:49.990225 2020] [php7:error] [pid 16939:tid 140670144591616] [client 127.0.0.1:42006] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 33554440 bytes) in /var/www/html/vendor/magento/module-csp/Model/Collector/DynamicCollector.php on line 31

m2-assistant · 2020-09-09T15:57:13Z

Hi @onepack. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

Summary of the issue
Information on your environment
Steps to reproduce
Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

gwharton · 2020-09-18T11:30:52Z

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /home/ubuntu/www/vendor/magento/module-csp/Model/BlockCache.php on line 108
Allowed memory size of 1073741824 bytes exhausted (tried to allocate 67108872 bytes) in /home/ubuntu/www/vendor/magento/module-csp/Model/Collector/DynamicCollector.php on line 31
Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /home/ubuntu/www/vendor/magento/framework/Serialize/Serializer/Json.php on line 37
Allowed memory size of 792723456 bytes exhausted (tried to allocate 4096 bytes) in /home/ubuntu/www/vendor/magento/module-csp/Model/BlockCache.php on line 78

I see it trying to allocate a 67MB CSP policy!!!

I'm going to run with CSP module disabled for a while and see if that helps

onepack · 2020-09-18T12:32:17Z

@gwharton a quick workaround is to disable the CSP module.
This weekend I'm going what the source of this issue with the CSP load is.

gwharton · 2020-09-18T13:01:21Z

Seemed to be something to do with the BlockCache with the limited debugging I was able to do before throwing one's hands in the air and disabling CSP.

m2-assistant · 2020-09-21T19:45:55Z

Hi @AlexMaxHorkun. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details
  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.
1. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.
1. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
1. Verify that the issue is reproducible on 2.4-develop branch
  Details
  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

Join Magento Community Engineering Slack and ask your questions in #github channel.

AlexMaxHorkun · 2020-09-22T18:35:28Z

I was not able to reproduce this locally with 1200+ products and dozens of categories. Perhaps this issue can only occur on a heavily resource limited setup. However I can see that there is a room for improvement performance wise in CSP implementation and it will be addressed in the next 2.4.x release

gwharton · 2020-09-22T18:41:13Z

eeeeekkkkk :(

67MB of CSP headers being allocated is not something that should be happening on any machine, let alone a resource starved one.

There is something wrong in the block cache/CSP area it seems.

I know when mine was failing, none of the hashes from the secure inline scripts were making it into the CSP headers either.

I am using varnish, I don't know if that matters.

gwharton · 2020-09-22T18:43:11Z

Also, it only seems to occurr at random times. I can let my site run for days with no problem, then all of a sidden, something triggers it and I am getting out of memory errors and apache is reporting random 503's to varnish. If I then clear cache, it runs fine again for a few days, then bang! I have had no issues whatsoever since disabling CSP. I for one, don't think module-csp is ready for production use.

dvershinin · 2020-09-29T07:54:59Z

This just happened to me on a second store. Redis usage grows to 14 GB in less than 24 hrs in a store with moderate traffic.
The amazing huge CSP hashes is the data that is constantly being inserted:

1601365923.254337 [0 unix:/var/run/redis/redis.sock] "HMSET" "zc:k:421_BLOCK_D0D5D79F37A04A787F6AD70C34ADE8FB6C40636F_7886_FINAL_PRICE_LIST_CATEGORY_PAGE_EUR_20200929_8_0_" "d" "{\"policies\":[{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"BStHi22teqUu50swU5OZrPUANmifudttOq1xnvuHFOI=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"48sb4Je7XoTlJimO7pm\\/+fwXo5BBI6oU4Vci+QqK2\\/I=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"tb6kG2xiP3wJ8b8k3K5Y66s8DN2QrZZrxDpFtEhn4Ss=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"9N89WMndeXJQQmez3zcXupuWhb0jRtPuHYgRtBa1Cjo=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"nHk4LqtZ0zapwB57kWmduJUqeJ54I7FBPnysQqhg0Lw=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"cHMJ2AAJCHWGIW4nyTqYh9J2NP0DzQ3KiiYGGPKJLUo=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"1z3VYjQl8LtcQRR81EXqqznKWOrYJyJVqu93BkXzkkk=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"21LoTH2EQ0uy3yQzlyZSgENO9IzGST3xuv2szY+LU\\/M=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"8iWqAQpE8l0bw+D23oEEajCTYJEGUTYn96PKWRkjb4Y=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"4rTRTwKUgPAVOi9Zp5e\\/754G27kLnd9Rff\\/NKCEXNNI=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"4eIGKpGuhjU8Bsof\\/Mm\\/AHwTzCZUFvevER0FsWS20nw=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"UuGKBjKpVoC\\/8zfiy\\/m2jdeGhNqz8QQKQ1K7x0R+A6M=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"Xj4FDkARMdOwyPTU2EZ0oj6QTAMX1Ymb\\/op4G0\\/\\/q04=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"DYh0vPP8uKimhqykQwOxc4w+rulqsQJ79neQPYRi5Ns=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"Wbek+bBB9O15rdozQeUDPmvlHeVul8ZfHTtyXyoRi+U=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"QZ98FJG2XexdAsIsPDnEsBCYIbUPWudEt6suwuYiUxo=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"LFhxcVU6KRNm8juJ49EGPihbH6dDlRb6s1pQFtrlYD4=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"PBEQfFi\\/gVIe7Z8Lwqi\\/WgweTpYkJhki3YFJgZoCzg4=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"Q7BepgmxKUsRtpFX4fb3xA3ldLYiBqk0Rk6SVfS5MXQ=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"HbpHVw9fO+wZnZFAa5gbVfSwOP\\/oIm7fpZUuTxBYQh8=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"ntqeMFyu\\/2v5BpzJHkvA3ZZVlClLaQ3PeLC4IFze8zA=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"Jj98CdytrDSwFu+0KylWE\\/9kVZSPQaFIiUWC0kL5mes=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"P87jUc1c\\/3wSpzp+87yecrXCwp7hCrve3cLlqumTxYo=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"RaxtWHpccZ2t8wqLOChw++JbLmKchcOgMG\\/QTwsex4o=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"tSAIDiDO+W1Rcytp2oUj2tF1hlUao38\\/perMphXecKY=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"j4lWFssGViA52hejnmgAonncxa2o\\/6Lr4aLD+GuNVFU=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"Y1SHvnQlHBvpdz4wFwxiW\\/q6ZOha+TsJxapUF6t+Ru4=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"Totv2RYsjc0hUZs2KX8S6eI5YD2c29xu3Xe4PndToT0=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"xFweBEfAIzSq85VerbpcJ0CivZDjPrdP9LHyUl5AiB8=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"+IUpoW8rS3Fpep3SBtuijdqoGf7VUQTXsC84m4uteR0=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"LWixx43imgqMI4Mx49DbtnSoQyQlWCssHJ71KLFWdmQ=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"M2F6PKej2Neu4sKPvbFUdC2vPE6uXs9LW4aSZaJUzxY=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"ec+4oeXNkrIxRdpBaTzWgj1KpclQHCtgkoVtYn8Xoqo=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"dsY+ngYpEyGwSwoEHhiSHthgmVwfisWUDuX4z4lC2ZE=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"S753tbiluj\\/bwkw2dRd\\/LupU8waF6iKffWjlvTFSLoo=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"2Jsxye4hbfekcMJ\\/fq\\/HNd6Q51wMgNV8j5UQO1+XkX4=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"CkzJPKXZMkxd9P2i9VCb1GTiH30lqzZ\\/P1e3H80ePWk=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"OV3DTDxoYMgkjIQZn3RK0sSgJE58mn4274mhvvgafmQ=\":\"sha256\"}},{\"id\":\"script-src\",\"hosts\":[],\"hashes\":{\"My9XEIRO+UJtEuIdIHqaryoloiSZ7K6QsnCV\\/OmI5cM=\":\"sha256\"}},{\"id\":\"style-src\",\"hosts\":[],\"hashes\":{\"jaZS\\/zfcxb9B9kZM+bYmauKa2GWqbrgmJn30+6uSDo8 ...

And it's always huge.

Prior to M2.4.0, the stores could run just fine without maxmemory configured in Redis.

The real problem is not the size, the real problem is continues insertion of varying hash data(?). Because only that would explain
continious growth to these sizes. So if one to restort to maxmemory-policy allkeys-lru for cache efficiency, they will have opposite results - cache having mostly the hash data for CSP. It will be highly inefficient with cache hit ratio close to ZERO.

hostep · 2020-09-29T09:15:20Z

Maybe the following patch can help: 1767ab8 (which is being mentioned in magento/magento-cloud-patches#44)

We had some Redis OOM issues on at least two Magento 2.3.5 shops even with the CSP module disabled. This patch seems to fix the OOM issue (although we've only had it running for a couple of days, so not 100% sure yet)

It's probably a workaround for the issue described in here, there seems to be a deeper underlying issue if I understand the comments correctly.

dvershinin · 2020-09-29T16:30:56Z

Technically speaking, everyone should set up some memory limit in Redis configuration to avoid OOM altogether.
It is still possible to have OOM with or without CSP, but with will lead to OOM quite fast, so the CSP module is very problematic in its current state.

will-wright1 · 2020-10-02T12:17:04Z

I'm seeing this issue too, disabling CSP caused me issues, I've had to setup a cron job to clear the cache every 6 hours!!

Quazz · 2020-10-08T10:06:20Z

An incredibly high amount of keys is being inserted for normal cache on Magento 2.4

I have a redis instance where cache and full page cache are shared and the normal cache endlessly growing eventually makes the fullpage cache useless since those keys get evicted since it runs out of its maxmemory limit.

It also appears that cache:clean does not clean those keys up. Only flushing the redis db helps...

Morgy93 · 2020-10-16T13:56:53Z

I can confirm this issue without Redis on Magento 2.4.1
Actually we disabled Redis because it consumed more than 60 GB of RAM.

Now with file cache the hard drive gets eaten and we're left with:

PHP message: PHP Fatal error:  Allowed memory size of 8589934592 bytes exhausted (tried to allocate 20480 bytes) in magento/module-csp/Model/BlockCache.php on line 108
PHP message: PHP Fatal error:  Allowed memory size of 8589934592 bytes exhausted (tried to allocate 77824 bytes) in Unknown on line 0

Yes, even 8 GB of RAM for PHP is not enough to process.

I assume that the foreach loop is just huge for whatever reason or whatever calls the save() method is a big loop.

https://github.com/magento/magento2/blob/2.4.1/app/code/Magento/Csp/Model/BlockCache.php#L108

@AlexMaxHorkun Our server is not at all resource limited with dedicated 16 vCPU, 64 GB of RAM and 160 GB SSD by the way.

Morgy93 · 2020-10-16T14:26:07Z

@nathanjosiah 2.4.2? Did you mean 2.4.1 which is "generally available" since yesterday?
If yes, I am on the 2.4.1 from yesterday with the issue described above.

Else just enlighten me 😁

nathanjosiah · 2020-10-16T14:27:22Z

@Morgy93 Apologies, I deleted my comment. I meant to say 2.4.1 but then I realized the fixes will not be available until 2.4.2

nathanjosiah · 2020-10-16T14:30:44Z

However, somebody could potentially test using 2.4-develop since the performance fixes have been merged there already.

Morgy93 · 2020-10-16T14:31:21Z

All I can see right now is this one 047629a#diff-f82a4c282711708e05d2f1db5f2f2369e2533e5bfb184b3f71db21a5756eafaf from 21 days ago.
I could try some manual patching if we find all necessary files.

Current workaround is to magento clean:cache every 15 minutes via cronjob............

stevenculshaw · 2020-11-05T10:55:44Z

Hi,

Firstly thanks for all of this, it's a problem that's been driving me nuts.

Is there an easy workaround or easy way to clear the issue on 2.4.0?

I've been monitoring the php error log and then reindexing/flushing which seems to make it go away until next time, which could be an hour but it's getting more frequent.

Any help much appreciated.

gwharton · 2020-11-05T10:58:54Z

@stevenculshaw Have you tried applying the patch listed in comment #29964 (comment)

I have been running with that patch on 2.4.1 with no problems. I don't know if it will apply cleanly on 2.4.0, but theres only one way to find out!

stevenculshaw · 2020-11-05T11:02:54Z

Yeah tried applying to 2.4.0 and just got a mash of errors so must be dependent on something else in 2.4.1.

Guess the route is update to 2.4.1 and then try patching the code in from 2.4.2 again.

I just wondered if there was anything anyone had found to help with the symptoms while I work out how to do that on the busiest shopping weekend of the year.

stevenculshaw · 2020-11-05T11:10:16Z

someone mentioned disabling module-csp, is that a really bad idea?

gwharton · 2020-11-05T11:26:53Z

not at all. unless you have configured it otherwise, CSP only runs in report only mode, so it doesn't provide ANY extra security unless you set it up to. So if you havent configured any additional policies for your site, or changed it to enforcing, then the only thing it is giving you is the crashes. Go ahead and disable it. You won't see any negative effects from disabling it.

I did check that patch against 2.4.0 and it should apply cleanly. But like you say, for a busy shopping period......... get it disabled if its causing you issues :)

stevenculshaw · 2020-11-05T11:48:30Z

Thanks for that clarification Graham, much appreciated, will give it a go

nathanjosiah · 2020-11-05T14:45:48Z

For what it's worth, @gwharton your patch contains much more than the changes I mentioned above in #29964 (comment) which is probably why it isn't applying cleanly for @stevenculshaw

Morgy93 · 2020-11-05T14:50:56Z

Yes, the patch contains much more than @nathanjosiah mentioned and I was a little confused about it, but I'd like to quickly confirm that the patch works wunderfully on 2.4.1
The production server currently is below 6 GB of RAM for hours (using Redis) and I can browse around as much as I like (not mentioning all the other bots and users doing stuff 😁 )

Finally fixed, thanks all! 👏

gwharton · 2020-11-05T14:51:27Z

@nathanjosiah As far as I can see, everything in the patch I listed (which was the whole commit by the core team) is needed to support the few lines you posted in your comment. It adds support for the merger that is in your comment.

I also deployed 2.4.0, manually made the same changes as in the patch i posted, then diff'd against a vanilla 2.4.0 install, and compared the diffs to the original patch, and they are identical, so the patch SHOULD apply cleanly on 2.4.0.

stevenculshaw · 2020-11-05T14:54:35Z

Hey all, temporarily disabling the csp module has provided valuable breathing-space, not a single error logged since. I've made a scratch copy of the site and later I will try the patch again on that and see what happens. Thanks all for your help!

iphigenie · 2020-11-11T10:11:43Z

Just to add a bit of context to this:

I just had this happen, both the cache growing to huge sizes and the out of memory on front end pages.

On my production site using external server level full page cache (livemage, in my case), the issue happens.
On my test server using the "internal" full page cache, the issue does not seem to happen. Switching the test server to the external cache, it happens.

So either disabling CSP or switching to the (slower, granted) internal cache are options to mitigate until 2.4.2 is released.

NudeWeb · 2021-01-19T23:16:41Z

I have the same problem but am a bit confused how to patch. Have a 2.4.1. site
This is my file vendor/magento/module-csp/Model/Collector/DynamicCollector.php

Could someone post how this should be changed and if any other changes need to be made please.

`<?php
/**

See COPYING.txt for license details.
*/
declare(strict_types=1);

namespace Magento\Csp\Model\Collector;

use Magento\Csp\Api\Data\PolicyInterface;
use Magento\Csp\Api\PolicyCollectorInterface;

/**

CSPs dynamically added during the rendering of current page (from .phtml templates for instance).
/
class DynamicCollector implements PolicyCollectorInterface
{
/*
- @var PolicyInterface[]
  */
  private $added = [];
/**
- Add a policy for current page.
- @param PolicyInterface $policy
- @return void
  */
  public function add(PolicyInterface $policy): void
  {
  $this->added[] = $policy;
  }
/**
- @inheritdoc
  */
  public function collect(array $defaultPolicies = []): array
  {
  return array_merge($defaultPolicies, $this->added);
  }
  }
  `

caricell · 2021-02-02T18:49:56Z

Can anyone please verify that this issue is fixed in M 2.42 ?
We have 2.41 with this issue and have disabled CSP module, which seems to work so far.
However, I do not see in 2.42 release notes that this issue is permanently fixed.

Can anyone advise please?
Thanks!

hostep · 2021-02-02T19:50:34Z

@caricell: I have access to the pre-release of 2.4.2 and can confirm that the commit @gwharton linked to before is included in there, but indeed it (MC-37799) doesn't appear to be included in the release notes of 2.4.2 which is strange.
I've alerted the person in charge of the release notes, hopefully she'll be able to figure out why it's not included yet.

caricell · 2021-02-02T19:59:05Z

Hello Pieter and thanks so much for your response! This OOM issue has been plaguing myself and my developer since we upgraded to 2.41. If you could let me know what you find out about why it was not included in the release notes, I would certainly appreciate it!!! Thank you! Larry From: Pieter Hoste <[email protected]> Sent: Tuesday, February 2, 2021 2:51 PM To: magento/magento2 <[email protected]> Cc: caricell <[email protected]>; Mention <[email protected]> Subject: Re: [magento/magento2] Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0 (#29964) @caricell <https://github.com/caricell> : I have access to the pre-release of 2.4.2 and can confirm that the commit @gwharton <https://github.com/gwharton> linked to before <#29964 (comment)> is included in there, but indeed it (MC-37799) doesn't appear to be included in the release notes of 2.4.2 which is strange. I've alerted the person in charge of the release notes, hopefully she'll be able to figure out why it's not included yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#29964 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASWO2ZHNVTROQMFZ2K2EJ5DS5BJRXANCNFSM4RCTZGCA> . <https://github.com/notifications/beacon/ASWO2ZHNID2BQQU757LJXCLS5BJRXA5CNFSM4RCTZGCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFYBLU2I.gif>

caricell · 2021-02-02T19:59:15Z

Thank you so much!
I look forward to hear your findings!

lagunacellar · 2021-02-02T20:02:23Z

We've been running 2.4.2. Can confirm this issue is resolved.

caricell · 2021-02-02T20:09:02Z

Thank you as well, laguna. I guess they just missed it in the release notes.

hostep · 2021-02-02T20:12:29Z

I got confirmation that it will be included in the release notes soon 🙂 . Thanks for the heads-up!

caricell · 2021-02-02T20:20:07Z

Thanks again! Larry From: Pieter Hoste <[email protected]> Sent: Tuesday, February 2, 2021 3:13 PM To: magento/magento2 <[email protected]> Cc: caricell <[email protected]>; Mention <[email protected]> Subject: Re: [magento/magento2] Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0 (#29964) I got confirmation that it will be included in the release notes soon 🙂 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#29964 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASWO2ZARB33MSKCDL5XZ4DDS5BMD7ANCNFSM4RCTZGCA> . <https://github.com/notifications/beacon/ASWO2ZCHE5O6XR4BU24OX5DS5BMD7A5CNFSM4RCTZGCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFYBPSOY.gif>

pmonosolo · 2021-07-08T20:07:14Z

Can anyone please verify that this issue is fixed in M 2.42 ?
We have 2.41 with this issue and have disabled CSP module, which seems to work so far.
However, I do not see in 2.42 release notes that this issue is permanently fixed.

Can anyone advise please?
Thanks!

Nope. I just upgraded to 2.4.2-p1, still the same issue on multiple environments and cache types (redis and file storage cache)

gwharton · 2021-07-08T21:50:18Z

Really. It was fixed in 2.4.2. it was missed from the release notes though but the fix was there. I've been running since 2.4.2 with csp enforcing with no issues.

pmonosolo · 2021-07-08T22:12:49Z

Really. It was fixed in 2.4.2. it was missed from the release notes though but the fix was there. I've been running since 2.4.2 with csp enforcing with no issues.

Yup, I created 2 threads about this on stackoverflow. I'm not sure how yours is working, but mine is an upgrade from 2.3.3 without any other modifications.

https://magento.stackexchange.com/questions/340934/magento-2-4-2-p1-redis-cache-full
https://stackoverflow.com/questions/68278489/docker-compose-mnt-sdb-is-always-full

gwharton · 2021-07-09T09:11:30Z

I wonder if there is a separate issue. The commit 047629a#diff-f82a4c282711708e05d2f1db5f2f2369e2533e5bfb184b3f71db21a5756eafaf definately fixed my issue and has been present since 2.4.2

pmonosolo · 2021-07-09T16:02:13Z

I wonder if there is a separate issue. The commit 047629a#diff-f82a4c282711708e05d2f1db5f2f2369e2533e5bfb184b3f71db21a5756eafaf definately fixed my issue and has been present since 2.4.2

I definitely have this in my vendor folder and both files are exact copies from the commit.

As soon as I disabled the CSP plugin and flushed the cache it started to work right away. I ran Search Indexing multiple times that day and the only time it stopped filling the Redis container RAM was after disabling the CSP plugin.

magento-engcom-team added the Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed label Sep 9, 2020

ghost added the Progress: ready for QA label Sep 9, 2020

onepack mentioned this issue Sep 9, 2020

Layout corruption with php-fpm (a Magento 1 problem returns) #6942

Closed

kandy assigned AlexMaxHorkun Sep 21, 2020

ghost added Progress: dev in progress and removed Progress: ready for QA labels Sep 21, 2020

AlexMaxHorkun closed this as completed Sep 22, 2020

ghost added Progress: done and removed Progress: dev in progress labels Sep 22, 2020

nathanjosiah reopened this Oct 16, 2020

m2-community-project bot added Progress: dev in progress and removed Progress: done labels Oct 16, 2020

hostep mentioned this issue Nov 5, 2020

Magento 2.3.5 Csp Issue #30804

Closed

magento-engcom-team added the Reported on 2.4.0 Indicates original Magento version for the Issue report. label Nov 13, 2020

hostep mentioned this issue Feb 11, 2021

Magento 2.4.1 - HUGE Cache Sizes grows quickly #32118

Closed

ikevinsolomon mentioned this issue May 5, 2021

Magento fails to unserialize config value from Redis in HA setup #32237

Closed

5 tasks

ananth-iyer mentioned this issue Nov 8, 2023

Redis / AOF - Magento 2.4.6-p3 #38148

Open

5 tasks

Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0 #29964

Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0 #29964

Comments

onepack commented Sep 9, 2020 • edited Loading

Preconditions (*)

Steps to reproduce (*)

Expected result (*)

Actual result (*)

m2-assistant bot commented Sep 9, 2020

gwharton commented Sep 18, 2020 • edited Loading

onepack commented Sep 18, 2020

gwharton commented Sep 18, 2020

m2-assistant bot commented Sep 21, 2020

AlexMaxHorkun commented Sep 22, 2020

gwharton commented Sep 22, 2020

gwharton commented Sep 22, 2020

dvershinin commented Sep 29, 2020 • edited Loading

hostep commented Sep 29, 2020

dvershinin commented Sep 29, 2020

will-wright1 commented Oct 2, 2020

Quazz commented Oct 8, 2020 • edited Loading

Morgy93 commented Oct 16, 2020

Morgy93 commented Oct 16, 2020

nathanjosiah commented Oct 16, 2020

nathanjosiah commented Oct 16, 2020

Morgy93 commented Oct 16, 2020

stevenculshaw commented Nov 5, 2020

gwharton commented Nov 5, 2020

stevenculshaw commented Nov 5, 2020

stevenculshaw commented Nov 5, 2020

gwharton commented Nov 5, 2020 • edited Loading

stevenculshaw commented Nov 5, 2020

nathanjosiah commented Nov 5, 2020

Morgy93 commented Nov 5, 2020

gwharton commented Nov 5, 2020 • edited Loading

stevenculshaw commented Nov 5, 2020

iphigenie commented Nov 11, 2020

NudeWeb commented Jan 19, 2021

caricell commented Feb 2, 2021

hostep commented Feb 2, 2021

caricell commented Feb 2, 2021 via email

caricell commented Feb 2, 2021

lagunacellar commented Feb 2, 2021

caricell commented Feb 2, 2021

hostep commented Feb 2, 2021 • edited Loading

caricell commented Feb 2, 2021 via email

pmonosolo commented Jul 8, 2021

gwharton commented Jul 8, 2021

pmonosolo commented Jul 8, 2021 • edited Loading

gwharton commented Jul 9, 2021

pmonosolo commented Jul 9, 2021

onepack commented Sep 9, 2020 •

edited

Loading

gwharton commented Sep 18, 2020 •

edited

Loading

dvershinin commented Sep 29, 2020 •

edited

Loading

Quazz commented Oct 8, 2020 •

edited

Loading

gwharton commented Nov 5, 2020 •

edited

Loading

gwharton commented Nov 5, 2020 •

edited

Loading

hostep commented Feb 2, 2021 •

edited

Loading

pmonosolo commented Jul 8, 2021 •

edited

Loading