X-Magento-Tags header too large

[katef]

Preconditions

I've been reading about people using Magento2 and the size of the X-Magento-Tags header.

Steps to reproduce

For example http://www.maxbucknell.com/blog/2016/2/10/troubleshooting-varnish-with-magento-2 writes:

The rub here is that Magento adds a tag for every product in a category. And the tag is of the form "catalog_product_{{PRODUCT_ID}}".

Actual result

That's an integer and 20 bytes of string per product, which gives some users headers several kB in size, when many products are in one category. In that case, the author describes hitting Varnish's limit at 400 products.

Expected result

I work for a CDN where some of our users use Magento, and we cache their content with Varnish. Increasing Varnish's http_resp_hdr_len size may suit that one person's situation, but it is infeasible for us, because it would affect all of our users.

Proposed solution

I have a suggestion to offer, for how to use header space more efficiently:

Each product ID is an integer. The X-Magento-Tags header describes the presence or absence of products only. That information can be done with a single bit per product. The index of the bit would give the product ID.

That's usually known as a bit vector, or a bitfield, or a bitmap. My suggestion is to switch the header to use a bitmap, encoded in hex. That'd take one bit per product, instead of over 20 bytes.

More on bitmaps: https://en.wikipedia.org/wiki/Bit_array

[stamster]

Excellent point!
But I'm afraid such change is not going to happen anytime soon, maybe for Magento 3.0... and this is only one of the topics out there to discuss when it comes to optimization.

[katef]

Thank you - Although I'd request that you consider this a blocking item, rather than an optimisation, because this header is so large that it prevents certain users from using Varnish.

[melnikovi]

@katef thank you for reporting this issue, internal ticket created MAGETWO-57851

[stamster]

@katef : I'm not representative of Magento Commerce Inc. ... :) I completely agree with you, of course it is blocking for certain use cases out there.

[pynej]

Some more detail:
The catalog_category_product_ and catalog_category_ could easily be changed to 2 characters that would still be unique and cut down the length of this header by ~80%.

Of critical issue though is that in Magento 2.1 all versions of a configured product are incorrectly added to this header field. In 2.0 only the parent item was added. As a result the size of the field grows exponentially and makes varnish unusable. I am trying to find the code in question that is incorrectly adding all versions so I can comment it out and restore 2.0 functionality and and have an actual working site.

[pynej]

This workaround fixes but wastes lots or memory.

[pynej]

I did this workaround to get my site working again. Basically disabling the addition of the child products of configurable items to the tag. There are cons't defined that specify the catalog_product_ tag, so that should be pretty straightforward to shorten.

--- vendor/magento/module-configurable-product/Plugin/Model/Product.php.orig    2016-09-07 14:10:22.201010000 -0400
+++ vendor/magento/module-configurable-product/Plugin/Model/Product.php 2016-09-07 14:08:17.107250000 -0400
@@ -26,9 +26,9 @@
         /** @var Configurable $productType */
         $productType = $product->getTypeInstance();
         if ($productType instanceof Configurable) {
-            foreach ($productType->getChildrenIds($product->getId())[0] as $productId) {
-                $result[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $productId;
-            }
+            #foreach ($productType->getChildrenIds($product->getId())[0] as $productId) {
+            #    $result[] = \Magento\Catalog\Model\Product::CACHE_TAG . '_' . $productId;
+            #}
         }
         return $result;
     }

[pboisvert]

@choukalos can you ensure this item gets the magetwo ticket? pulled the improvement label

[choukalos]

Tracking internally - MAGETWO-58265 ; Thanks @pynej for diagnosing and the code diff.

[veloraven]

Internal ticket for 2.1 - MAGETWO-58362

[choukalos]

Closing; this was resolved in 2.1.3

[katef]

@choukalos What was the resolution?

[djfordz]

I'm still having issue with this in version 2.1.3

Error 503 Backend fetch failed

With Category Pages that have tons of categories.

[dcabrejas]

@choukalos I think this ticket should be reopen, I have encountered this issue on a Magento site running on 2.1.5. Category pages containing a large number of products return a X-Magento-Tag header which is bigger than the size allowed by the HTTP specification.

We have implemented this workaround for now http://devdocs.magento.com/guides/v2.0/config-guide/varnish/tshoot-varnish-503.html

But this is not a solution, just a workaround because the http header size is still larger than the limit set by the HTTp specification.

[redelschaap]

Any update on this? This is still present in 2.1.7.

[tedsalmon]

As previously stated, this is still an issue in 2.1.8. I have implemented a Plugin around Magento\Catalog\Model\Product like so:

<?php

namespace YourCompany\YourModule\Plugin;

class Product{

    protected $_request;

    public function __construct(\Magento\Framework\App\Request\Http $request){
       $this->_request = $request;
    }

    /*
     * Return no simple product tags when returning category pages
     */
    public function afterGetIdentities(\Magento\Catalog\Model\Product $subject, $tags){
        if(!$this->_request){
            return $tags;
        }
        $r = $this->_request;
        $fullRoute = sprintf(
            '%s_%s_%s', $r->getRouteName(), $r->getControllerName(), $r->getActionName()
        );
        if($fullRoute === 'catalog_category_view'){
            return [];
        }else{
            return $tags;
        }
    }
}

?>

I very much realize that this is a hack, but we simply have too many products (over 64kB of tags) for the tag shortening to work. This will still show any parent product tags, FWIW.

[satishgadhave]

Can you pls provide link to pull request?

[magento-engcom-team]

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.3-develop branch(es).
Related commit(s):

• 8280a9a6478d27131e7a12fa4b112baaf84ca86f
• 1729dcc4d9172e2dd51d061e7d5f560f244ed304

The fix will be available with the upcoming 2.3.5 release.

[katef]

@magento-engcom-team Wonderful, thank you!

The commits you linked to 404 though. Was that branch rebased and now they have different hashes now, perhaps?

[magento-engcom-team]

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.4-develop branch(es).
Related commit(s):

• bde57a3a0c018cfb9924f150f4d651600ed92406
• b0071aff2c0289d72431e05f22ac353236242107
• d2e252cf2696247715b927833aad7c22d8565ab6
• 219024d611ebeef5234901d2f3d784c9fc355721
• b6206fec5a4a1c88dd3b46c512c1e6314e7451c9
• e85903535b71e3e6fe534123d60a259099ce6aa5
• ca59b15fe42e18cd26ea8cfc093fdd5e4c6773a8
• 520590a2db348a1f8e6d4c1a8274a4f248fcdcb8

The fix will be available with the upcoming 2.4.0 release.

[katef]

Hi @magento-engcom-team! That's great, thank you!
All your links 404 for me, though. Are those private perhaps?

[bmitchell-ldg]

Hi @magento-engcom-team we would also like to see the related commits for 2.4.0 if you would be able to resolve those 404 links please?

[ecoprince]

Hi @katef, @o-iegorov.

Thank you for your report and collaboration!

The issue was fixed by Magento team. The fix was delivered into magento/magento2:2.4-develop branch(es).
Related commit(s):

• bde57a3a0c018cfb9924f150f4d651600ed92406
• b0071aff2c0289d72431e05f22ac353236242107
• d2e252cf2696247715b927833aad7c22d8565ab6
• 219024d611ebeef5234901d2f3d784c9fc355721
• b6206fec5a4a1c88dd3b46c512c1e6314e7451c9
• e85903535b71e3e6fe534123d60a259099ce6aa5
• ca59b15fe42e18cd26ea8cfc093fdd5e4c6773a8
• 520590a2db348a1f8e6d4c1a8274a4f248fcdcb8

The fix will be available with the upcoming 2.4.0 release.

@magento-engcom-team All links goes to 404. Please update it.

[ihor-sviziev]

@ecoprince,
Now these commits are available. Here is merge commit that contains all these changes: 3b611a7

[katef]

Thank you!

[websecureNL]

Any way to get this fixed on 2.3.5? I understand the fix is made for 2.4 develop but for our production store we are using 2.3.5-p1 and having this problem.

[ihor-sviziev]

@websecureNL you could try to create patch and apply on top of 2.3.5.
You can use https://devdocs.magento.com/guides/v2.4/comp-mgr/patching.html or https://github.com/vaimo/composer-patches

[websecureNL]

@ihor-sviziev, I tried this and it seems to work properly. We will do some further testing. Since making patches was new to me a quick hint for other users. We used this file (based on merge commit 3b611a7) to patch using instructions https://devdocs.magento.com/guides/v2.4/comp-mgr/patching.html

[websecureNL]

The problem also occurs when using catalogsearch. No idea if this also impacts 2.4 but since the patch did not fix this it could be.

[ihor-sviziev]

Could you check on empty 2.4-develop installation, and if issue there - report it with exact steps to reproduce?

…

On Thu, 2 Jul 2020 at 00:29, websecureNL ***@***.***> wrote: The problem also occurs when using catalogsearch. No idea if this also impacts 2.4 but since the patch did not fix this it could be. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6401 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOJOUJWTKQRJSTOTHLVGO3RZOTCXANCNFSM4COCM5GA> .

[websecureNL]

@ihor-sviziev. Of course I can do this but isn't this originated from Magento 2.3? We are having problems with 2.3.5 and would like to get it fixed there. I will check 2.4-develop asap.

[elvinristi]

if you compare commits then actually nothing was done. was reverted probably something what didn't exist yet in M2.1 and therefor no matter of the version, issue continues.

temporary fix can be currently #6401 (comment)

but longer run should be similar what was done here but closed and ignored due inactivity #12831 which splits headers.

[ihor-sviziev]

if you compare commits then actually nothing was done. was reverted probably something what didn't exist yet in M2.1 and therefor no matter of the version, issue continues.

temporary fix can be currently #6401 (comment)

but longer run should be similar what was done here but closed and ignored due inactivity #12831 which splits headers.

@elevinskii #6401 (comment) is really hack and not a correct solution, it breaks automatic cache flushing for category pages when any product was changed in the admin. definitely don't recommend it.

#12831 this solution is quite good, but it's not finalized in order to be accepted. You could create new PR based on this PR and apply requested changes. Only one concern - such change might break some services like Fastly (for instance used in Magento Cloud), so probably it should be double checked additionally

Also would like to add that there was a fix for configurable products that mostly caused this issue, this fix you can see in #6401 (comment)

BTW if you have steps to reproduce of this issue on 2.4-develop - please report it separately

[toonvd]

I just landed on this issue, a more complete solution would be to implement #6401 (comment). And create an observer on product save that gets parents on simple products and invalidates those too.

[amenk]

Still happens on some circumstances on 2.4.2. Is this header only needed if varnish is used? Can it be safely removed, if not?

[hostep]

@amenk: as far as I'm aware: yes! The X-Magento-Tags header is only used by Varnish to indicate what tags are associated to a certain url. So when Magento tells Varnish to purge some tags later on, it will purge all url's from its cache where that particular tag got associated to it.
So I believe you should be able to safely remove the header if you aren't using Varnish (or a similar reverse proxy that works with this header).

[amenk]

@hostep Thanks - we have a patch on StackOverflow: https://magento.stackexchange.com/questions/287621/ah01070-error-parsing-script-headers-in-php-fpm-when-accessing-certain-product .

What should be noted is that the clearing should happen after the tags are extracted from the Header and used for the built-in cache as well.

[atty31]

This happens on 2.4.3-p1 . Any updates or patches ? Thanks

[mlaurense]

After having raised the http_resp_hdr_len to 1MB (derived by calculating the largest number of products in a category multiplied by 21, as advised by Magento: https://support.magento.com/hc/en-us/articles/360034631211-Troubleshooting-503-error-caused-by-necessity-to-change-default-Varnish-settings), we were still experiencing intermittent 503 errors. These would happen on static files, e.g. javascripts and images.

To further debug this, I opened a varnishlog using the following command:
varnishlog -g request -q "RespStatus >= 500"
This returns all requests with a response code of 500 or higher. Now we just had to wait for an error to appear.

Soon after, the errors appeared:

--  FetchError     Workspace overflow
--  FetchError     overflow
[...]
--  BerespStatus   503
--  BerespReason   Service Unavailable
--  BerespReason   Backend fetch failed
--  Error          out of workspace (bo)

This is indicating a clear error, and pointed me in the right direction to further raise the workspace parameters as well. I am now using these parameters and the errors haven't reappeared:

    -p workspace_backend=2097152
    -p http_resp_size=1082768
    -p http_resp_hdr_len=1050000
    -p workspace_client=2097152
    -p http_req_size=65536
    -p http_req_hdr_len=32768

Of course, this shouldn't be necessary, but given the fact that Magento uses these very large tag headers, I'm afraid we have no choice other than to tune these varnish parameters.

[hostep]

Potential fix: #33468