Multisite installation, default website slow (X-Magento-Vary)

[Flowlance]

Hi. We have a multisite installation. The default website is fast, but the non-default website is really slow. When we switch the default website to the slow one, that ones becomes fast. What I've found out so far is that the slow site always get a cookie named X-Magento-Vary, while the fast one get a PHPSESSID cookie.

Preconditions

Ubuntu 16.04
PHP 7.1.16
Aurora MySQL 5.6.10a (Aws RDS)
Magento 2.2.2
Nginx 1.14.0

Running on multiple AWS instances in OpsWorks with continuous integration (Jenkins).
Media folder is shared on EFS.

Steps to reproduce

The website codes are "base" and "exampleone". In the tests below we have "exampleone" set as the default website.

Nginx configs:

upstream fastcgi_backend {
  server   unix:/run/php/php7.1-fpm.sock;
}

fastcgi_cache_path /var/cache/nginx levels=1:2 keys_zone=magento:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

map $http_host $MAGE_RUN_CODE {
    default base;
    www.example1.com exampleone;
    example1.com exampleone;
}

server {
    listen 80 default_server;
    server_name _;
    set $MAGE_ROOT /srv/www/production/current/;
    set $MAGE_MODE production; # or developer
    index  index.php index.html index.htm;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    include /srv/www/production/current/nginx.conf;
}

Relevant part of /srv/www/production/current/nginx.conf:

# PHP entry point for main application
location ~ (index|get|static|report|404|503|autocomplete|health_check)\.php$ {
    try_files $uri =404;
    fastcgi_pass   fastcgi_backend;
    fastcgi_buffers 1024 4k;
    fastcgi_param  MAGE_RUN_CODE $MAGE_RUN_CODE;
    fastcgi_param  MAGE_RUN_TYPE $MAGE_RUN_TYPE;
    fastcgi_param https on;
    fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
    fastcgi_param  PHP_VALUE "memory_limit=1024M \n max_execution_time=60";
    fastcgi_read_timeout 60s;
    fastcgi_connect_timeout 60s;

    fastcgi_param REMOTE_ADDR $http_x_forwarded_for;
    fastcgi_param Host $http_host;
    fastcgi_param X-Forwarded-Proto https;
    fastcgi_param X-Forwarded-Port 443;
    fastcgi_param X-Forwarded-Host $http_host;
    fastcgi_param X-Real-IP $remote_addr;
    fastcgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
    add_header mage-run-code $MAGE_RUN_CODE;
    add_header mage-run-type $MAGE_RUN_TYPE;

    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include        fastcgi_params;
}

1. Just by visiting the website, we can clearly tell that one website is much slower than the other one. TTFB is really high (8+ seconds).

2. By CURL'ing the websites, we can see the X-Magento-Vary and PHPSESSID cookies

curl -I https://www.example1.com/

HTTP/2 200
date: Tue, 05 Jun 2018 08:16:59 GMT
content-type: text/html; charset=UTF-8
server: nginx/1.14.0
vary: Accept-Encoding
set-cookie: PHPSESSID=kn4lppbu32dgfrd6ejnit2b307; expires=Wed, 06-Jun-2018 08:16:59 GMT; Max-Age=86400; path=/; domain=www.example1.com; secure; HttpOnly
pragma: no-cache
cache-control: max-age=0, must-revalidate, no-cache, no-store
expires: Mon, 05 Jun 2017 08:09:10 GMT
x-cache-warmer: HIT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
mage-run-code: exampleone
mage-run-type: website

curl -I https://www.example2.com/

HTTP/2 200
date: Tue, 05 Jun 2018 08:17:16 GMT
content-type: text/html; charset=UTF-8
server: nginx/1.14.0
vary: Accept-Encoding
set-cookie: X-Magento-Vary=4dc58702c807c010b15fff2913b3c86682a31e1e; path=/; secure; HttpOnly
pragma: no-cache
cache-control: max-age=0, must-revalidate, no-cache, no-store
expires: Mon, 05 Jun 2017 08:17:16 GMT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN

Expected result

Both websites should be fast. I'm not sure what X-Magento-Vary does, but every time the website is slow, this cookie is set. I was thinking it could be related to user specific caches, but we are testing on the index page as a non-logged in user so the same content should be delivered every time.

Actual result

Non-default website is slow.
As a note, magento does set the correct website and everything works, it's just slow.

[oefterdal]

I have the same issue, and if I flip what site should be the default the secondary site is slow. So It's nothing with the website.

[Flowlance]

@oefterdal This is exactly what happens here too. And it's holding back the launch of our second website.

[phoenix128]

Are you using Varnish or the Built-in Full page cache?

[Flowlance]

@phoenix128 The built in FPC. We do this because of the multiple servers / load balancing setup. FPC with Redis can share cache between servers. Having one Varnish per server would cause different caches to show when reloading the page. We can probably set up a separate Varnish server that all the servers communicate with, but it's a bit of a hassle and i doubt it would do anything with the x-magento-vary problem.

As a side note, we already experienced having different caches on every page reload using a shared Redis server. I digged in to the code and found the cache key prefix to be the issue.
The prefix was generated from the path to the web root (/srv/www/production/releases/20180528123754), where the version number changes between every deployment. So unless all the servers finishes deploying on the exact same second, they will have different cache. Even if the root is set to /srv/www/production/current which is a symlink into the current release, it looks like PHP sees the real path with version number. The solution here was to override the id_prefix in env.php. However, I'm not sure if it's related to the issue or not.

'page_cache' => array(
    'id_prefix' => 'PROD_',
    'backend' => 'Cm_Cache_Backend_Redis',
    'backend_options' => array(
        'server' => 'example.com',
        'port' => '6379',
        'database' => '1',
        'compress_data' => '1',
    ),
),

[phoenix128]

Thank you @Flowlance , I am investigating on it trying to reproduce the issue.

[phoenix128]

@Flowlance ,
how did you configure the multiwebsite? Did you configure your NGINX or you created multiple path endpoints?

Actually it is quite weird that you see X-Magento-Vary in your cookies while not running Magento with Varnish. As far as I know it shouldn't be there.

Are you sure your settings are correct and you configured Magento to use the built-in cache from Configuration > System > Full Page Cache ?

I am asking you because I was not able to reproduce it in any way.

[Flowlance]

@phoenix128 thanks for your reply.

I also find it weird that this cookie is being set. See the attached screenshot of our current cache configuration:

We have set specific url's per store. Also note in my nginx config we have this:

map $http_host $MAGE_RUN_CODE {
    default base;
    www.example1.com exampleone;
    example1.com exampleone;
}

Unless it matches one of those two url's, it defaults to the base website.

As a side note, it also happens in our development environment. Is there anything else I can provide you with to help you reproduce the issue?

[phoenix128]

Thank you @Flowlance ,
I think it should be enough. I am still trying to reproduce it locally.

[glwstor]

We have the same issue with Magento 2.2.4 with two store views.
The default store view has no X-Magento-Vary set and the page load is around 52ms.
The non-default store view has X-Magento-Vary set and the page load is around 352ms.
We do not use Varnish, only the build in FPC with Redis.

Tested on Ubuntu18.04
Apache2.4
MySQL 5.6
PHP7.1
Redis

and
Ubuntu 18.04
Nginx
MariaDB
PHP7.1
Redis

Is there any solution?

Best regards

[phoenix128]

Not yet, till now I was unable to reproduce it. I am setting up a new clean environment to give another chance.

[Hansmueller88]

@phoenix128 Hello, we have also the same problem.

1. web browser: Google Chrome; page A: one page of a non-default store view; page B: another page of the same non-default store view

2.1. clear the browsing data of the web browser, open page A, it is slow
2.2. reload the page A, the speed is fast
2.3. open page B, it is slow
2.4. reload the page B, the speed is fast
until now, page A and page B of this non-default store view should be cached

3.1. clear the browsing data of the web browser, open page A, it is slow
3.2. open page B, it is fast

4.1. clear the browsing data of the web browser, open page B, it is slow
4.2. open page A, it is fast

5.1. using PageSpeed Insights (or Pingdom) to test the speed of page A and page B, all these 2 pages get a very bad score

In my opinion, Magento cache system is not returning page from cache if the page is a non-default store view page. But this problem occurs only during the first page open after clearing the browsing data (see point 3.1 - 4.2).

It is not a big problem for the customers, because of only the first visit is slow, if the pages are cached (see point 3.1 - 4.2).

But it is a big problem for SEO(see point 5.1)

[phoenix128]

@Hansmueller88 ,
I tried again with no success. The pages are loading at the same speed.
Can you please try to see what happens calling the pages in the same order you gave before but using curl --head ?

[Hansmueller88]

@phoenix128
using curl --head to get the following:

HTTP/1.1 200 OK
Server: nginx
Date: Thu, 27 Sep 2018 11:37:32 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Pragma: no-cache
Cache-Control: max-age=0, must-revalidate, no-cache, no-store
Expires: Wed, 27 Sep 2017 11:37:32 GMT
Strict-Transport-Security: max-age=31536000
Content-Security-Policy: upgrade-insecure-requests
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: X-Magento-Vary=b8017c883354ce24bb00e72d165aaf417ec8bf32; expires=Thu, 27-Sep-2018 12:37:32 GMT; Max-Age=3600; path=/; secure; HttpOnly
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
X-UA-Compatible: IE=edge

If it has "X-Magento-Vary", then it is very slow while the first visit one page of the non-default store view after cleared the browsing data. The speed of the other pages of this store view will be OK after opened one page(whatever which page is)

Actually, we have fixed this issue if the default currency of the non-default store view is the base currency(still need more time to confirm it). we have fixed this issue if:

1. Base Currency is A, if the default currency of the non-default store view is also A

but It still has this problem if:
2. Base Currency is A, if the default currency of the non-default store view is B

If you want, I can give you the SSH and Magento Admin of our test website to test it.

[schmengler]

As a side note, we already experienced having different caches on every page reload using a shared Redis server. I digged in to the code and found the cache key prefix to be the issue.
The prefix was generated from the path to the web root (/srv/www/production/releases/20180528123754), where the version number changes between every deployment. So unless all the servers finishes deploying on the exact same second, they will have different cache. Even if the root is set to /srv/www/production/current which is a symlink into the current release, it looks like PHP sees the real path with version number. The solution here was to override the id_prefix in env.php. However, I'm not sure if it's related to the issue or not.

I can confirm that. It is especially nasty if the backend web server has a different cache prefix than the frontend web servers, and cache invalidation from backend does not work.

A solution on Magento side would be to automatically set id_prefix when installing Magento, so that it does not change anymore and is the same for all web servers.

[carylewis]

I am experiencing the same issue - the time to first byte is almost 1500 ms slower on the non-default site.

I have opened a case with magento, as we have the EE version, but the issue has not been resolved yet.

Does anyone know exactly where in the code magento issues the x-magento-vary cookie?

[sidolov]

Hi @Flowlance. Thank you for your report.
The issue has been fixed in #18641 by @schmengler in 2.3-develop branch
Related commit(s):

• 12b7e08c2f262d0683e5b61d5fd2487aac19c27d
• 6a68a187e972b1e241e55153b0ba2a1483255f92
• 3ccd55e308823f8342c0503ab90af11b91c26f61
• f14949c840286e75199d9696377f65995fd69278

The fix will be available with the upcoming 2.3.1 release.

[bobmats]

Same problem on magento 2.2.5 when fpc is activated some pages have 1 tot 1.5 seconds to first byte.
How can this be solved

[carylewis]

Magento has a patch, but it broke PayPal. We had to patch their patch.

…

On Nov 29, 2018, at 9:09 AM, bobmats ***@***.***> wrote: Same problem on magento 2.2.5 when fpc is activated some pages have 1 tot 1.5 seconds to first byte. How can this be solved — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[bobmats]

@carylewis where can I find the patch and patch to fix the patch ?

[bobmats]

Have done some further testing magento 2.2.5 CE with FPC on.
The main domain used in installation is fast. time to first byte 100-200 ms
We also have multistore, several other websites/shops.
All of these have time to first byte between 1.2 and 1.5 seconds.

[bobmats]

What I found so far is only the domain used during installation gets the PHPSESSID cookie set.
All other domains within this installation get the X-Magento-Vary which causes an extra delay between 1.2 and 1.5 seconds.

So far I have been unable to find a fix for this, if there is any fix. I read som e links in this thread but as far as I understand this is for a new installation ? and only in magento 2.3 or 2.3.1

Is there any fix for this in 2.2.5 o5 2.2.6/2.27

[carylewis]

The fix has to do with the store id not being set correctly, and therefore magento can’t use the cache.

…

On Dec 3, 2018, at 7:48 AM, bobmats ***@***.***> wrote: What I found so far is only the domain used during installation gets the PHPSESSID cookie set. All other domains within this installation get the X-Magento-Vary which causes an extra delay between 1.2 and 1.5 seconds. So far I have been unable to find a fix for this, if there is any fix. I read som e links in this thread but as far as I understand this is for a new installation ? and only in magento 2.3 or 2.3.1 Is there any fix for this in 2.2.5 o5 2.2.6/2.27 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[bobmats]

Ok, question still remains how this can be fixed. Know you were on EE I am on CE. Have not been able to find something.

[bobmats]

Found this in 2.2.6 update: Pagecache

Full-page cache now works as expected in multistore deployments. Previously, when you opened the URL of a non-default store in a multistore deployment, full-page cache did not return the URL.

so i hope update to 2.2.6/2.2.7 will resolve this

[bobmats]

update to 2.2.7 did not resolve the problem.
nginx was disabled only appace no solution still x-magento-vary cookie is set on all domains except the installtion domain.

Even when fpc is turned of there is a problem.
Main domain will only show phpsesid and other domains phpsesid and x-magento-vary.

when pfc on also x-magento-cach show miss in combination with x-magento-vacy
on main domain cache is hit.

This drives me crazy and no idea where to find the solution.

[lokeshvnaik]

Facing the same issue in a multi-store setup 2.2.7, X-Magento-Cache-Debug shows MISS when the non-default store is opened in a new browser.

Is there any patch to make it work on 2.2.7 or someone refer the related commit of the issue fixed in 2.3.1?

Tried adding changes from below commits but still not working

#15828 (comment)

[magento-engcom-team]

Hi @Flowlance. Thank you for your report.
The issue has been fixed in #22439 by @Ctucker9233 in 2.2-develop branch
Related commit(s):

• 4b60fad08f07d39e052ef69746b9233a3270f618
• 9d26084636bdd7f0b1ca428300afcf614f94e123

The fix will be available with the upcoming 2.2.9 release.

[carylewis]

We are seeing this exact same issue on a 2.3.5p2 EE installation.

Non default web sites appear to not be using the FPC.

[p24-max]

As this issue still occurs with 2.4.4-p3, I hotfixed this with this code in pub/index.php
Replace storename with your store code and put into the condition where you set the store code.

    if (!array_key_exists('X-Magento-Vary', $_COOKIE)) {
        $vary = [
            "store" => "storename_store_view",
        ];
        ksort($vary);
        $varyString = sha1(json_encode($vary));
        $_COOKIE['X-Magento-Vary'] = $varyString;
    }