Encountered an issue wherein certain product listing pages with 60 products per page loads properly when using M2's built-in FPC but returns Error 500 when using Varnish.
After a lot of hours spent in investigating the issue and even trying to tune Varnish by configuring certain parameters, I noticed this recently submitted issue here in the M2 repository and tried to look into the X-Magento-Tags header.
I didn't find any pages that has spaces in the said header, nor blocks with block ID containing spaces, but I noticed something: perhaps due to the sheer amount of products that is being displayed in our product listing pages, some of them have X-Magento-Tags header lengths that reach up to more than 16k characters. I commented-out the "unset" statements in the default.vcl Varnish configuration file to check the contents of the said header and one of them are as follows:
store,cms_block,cms_block_header_mini_banner,catalog_category_42,catalog_category,catalog_category_product_42,catalog_product_13897,catalog_product,catalog_product_8512,catalog_product_8498,catalog_product_8499,catalog_product_8500,catalog_product_8501,catalog_product_8502,catalog_product_8503,catalog_product_8504,catalog_product_8513,catalog_product_8506,catalog_product_8507,catalog_product_8508,catalog_product_8509,catalog_product_8510,catalog_product_8511,catalog_product_13898,catalog_product_8514,catalog_product_8515,catalog_product_8516,catalog_product_8517,catalog_product_8518,catalog_product_8519,catalog_product_8520,catalog_product_8521,catalog_product_8522,catalog_product_8523,catalog_product_8524,catalog_product_8525,catalog_product_8526,catalog_product_8527,catalog_product_8528,catalog_product_13932,catalog_product_8590,catalog_product_8591,catalog_product_8592,catalog_product_8593,catalog_product_8594,catalog_product_8595,catalog_product_8603,catalog_product_8596,catalog_product_8597,catalog_product_8598,catalog_product_8599,catalog_product_8600,catalog_product_8601,catalog_product_8602,catalog_product_8632,catalog_product_8633,catalog_product_8634,catalog_product_8635,catalog_product_8636,catalog_product_8637,catalog_product_16189,catalog_product_16190,catalog_product_16191,catalog_product_16192,catalog_product_16193,catalog_product_16194,catalog_product_16195,catalog_product_16196,catalog_product_13959,catalog_product_8721,catalog_product_8722,catalog_product_8723,catalog_product_8724,catalog_product_8725,catalog_product_8726,catalog_product_8727,catalog_product_8728,catalog_product_8729,catalog_product_8730,catalog_product_15754,catalog_product_15755,catalog_product_15756,catalog_product_15757,catalog_product_15758,catalog_product_15759,catalog_product_15760,catalog_product_15761,catalog_product_15762,catalog_product_18991,catalog_product_18992,catalog_product_18993,catalog_product_18994,catalog_product_18995,catalog_product_13971,catalog_product_8793,catalog_product_8788,catalog_product_8789,catalog_product_8790,catalog_product_8791,catalog_product_8792,catalog_product_8794,catalog_product_8795,catalog_product_8796,catalog_product_8797,catalog_product_8798,catalog_product_8799,catalog_product_8812,catalog_product_8813,catalog_product_8814,catalog_product_8815,catalog_product_8816,catalog_product_8817,catalog_product_16026,catalog_product_16027,catalog_product_16028,catalog_product_16029,catalog_product_16030,catalog_product_16031,catalog_product_16032,catalog_product_16033,catalog_product_16034,catalog_product_16035,catalog_product_16036,catalog_product_16038,catalog_product_16039,catalog_product_16040,catalog_product_16041,catalog_product_16042,catalog_product_16043,catalog_product_17363,catalog_product_17364,catalog_product_17365,catalog_product_17366,catalog_product_17367,catalog_product_17368,catalog_product_13977,catalog_product_8837,catalog_product_8838,catalog_product_8839,catalog_product_8840,catalog_product_8841,catalog_product_8842,catalog_product_8845,catalog_product_8846,catalog_product_8847,catalog_product_8848,catalog_product_8849,catalog_product_8850,catalog_product_8851,catalog_product_8852,catalog_product_14004,catalog_product_8999,catalog_product_9000,catalog_product_9001,catalog_product_9002,catalog_product_9003,catalog_product_16175,catalog_product_9004,catalog_product_9005,catalog_product_9006,catalog_product_9007,catalog_product_9008,catalog_product_9009,catalog_product_9012,catalog_product_9013,catalog_product_9014,catalog_product_9015,catalog_product_9016,catalog_product_16176,catalog_product_16183,catalog_product_16184,catalog_product_16185,catalog_product_16186,catalog_product_16187,catalog_product_16188,catalog_product_16177,catalog_product_16178,catalog_product_16179,catalog_product_16180,catalog_product_16181,catalog_product_16182,catalog_product_17357,catalog_product_17358,catalog_product_17359,catalog_product_17360,catalog_product_17361,catalog_product_17362,catalog_product_14014,catalog_product_17411,catalog_product_9055,catalog_product_9056,catalog_product_9057,catalog_product_9058,catalog_product_9059,catalog_product_17412,catalog_product_9060,catalog_product_9061,catalog_product_9062,catalog_product_9063,catalog_product_9064,catalog_product_18526,catalog_product_18527,catalog_product_18528,catalog_product_18529,catalog_product_18530,catalog_product_18531,catalog_product_17413,catalog_product_9071,catalog_product_9072,catalog_product_9073,catalog_product_9074,catalog_product_9075,catalog_product_15866,catalog_product_15867,catalog_product_15868,catalog_product_15870,catalog_product_18532,catalog_product_18533,catalog_product_18534,catalog_product_18535,catalog_product_18536,catalog_product_18537,catalog_product_17415,catalog_product_17406,catalog_product_17407,catalog_product_18821,catalog_product_17410,catalog_product_14228,catalog_product_10288,catalog_product_10289,catalog_product_10290,catalog_product_10291,catalog_product_10292,catalog_product_10295,catalog_product_10296,catalog_product_10297,catalog_product_10298,catalog_product_10299,catalog_product_10300,catalog_product_10302,catalog_product_10303,catalog_product_10304,catalog_product_10305,catalog_product_10306,catalog_product_10307,catalog_product_10310,catalog_product_10311,catalog_product_10313,catalog_product_10314,catalog_product_15961,catalog_product_15962,catalog_product_15963,catalog_product_15964,catalog_product_15965,catalog_product_15966,catalog_product_15967,catalog_product_15968,catalog_product_15969,catalog_product_15971,catalog_product_15972,catalog_product_15974,catalog_product_14231,catalog_product_10318,catalog_product_10319,catalog_product_10320,catalog_product_10321,catalog_product_10322,catalog_product_10323,catalog_product_10324,catalog_product_14240,catalog_product_10406,catalog_product_10407,catalog_product_10408,catalog_product_10409,catalog_product_10410,catalog_product_10411,catalog_product_15860,catalog_product_15861,catalog_product_15862,catalog_product_15863,catalog_product_17530,catalog_product_17531,catalog_product_17532,catalog_product_17533,catalog_product_17534,catalog_product_14272,catalog_product_10660,catalog_product_10661,catalog_product_10662,catalog_product_10663,catalog_product_10664,catalog_product_10665,catalog_product_10666,catalog_product_10667,catalog_product_10668,catalog_product_10669,catalog_product_10670,catalog_product_10671,catalog_product_10672,catalog_product_10673,catalog_product_15883,catalog_product_15884,catalog_product_15885,catalog_product_15886,catalog_product_15887,catalog_product_15888,catalog_product_15889,catalog_product_14277,catalog_product_10701,catalog_product_10702,catalog_product_10703,catalog_product_10704,catalog_product_10705,catalog_product_10711,catalog_product_10712,catalog_product_10713,catalog_product_10714,catalog_product_10715,catalog_product_10706,catalog_product_10707,catalog_product_10708,catalog_product_10709,catalog_product_10710,catalog_product_15692,catalog_product_15693,catalog_product_15694,catalog_product_15695,catalog_product_15696,catalog_product_15697,catalog_product_15698,catalog_product_15699,catalog_product_15700,catalog_product_15701,catalog_product_17834,catalog_product_17835,catalog_product_17836,catalog_product_17837,catalog_product_17838,catalog_product_14343,catalog_product_11360,catalog_product_11361,catalog_product_11362,catalog_product_11363,catalog_product_11364,catalog_product_11365,catalog_product_11354,catalog_product_11355,catalog_product_11356,catalog_product_11357,catalog_product_11358,catalog_product_16012,catalog_product_16013,catalog_product_16014,catalog_product_16015,catalog_product_16016,catalog_product_16017,catalog_product_16019,catalog_product_16020,catalog_product_16021,catalog_product_16022,catalog_product_16023,catalog_product_17304,catalog_product_17305,catalog_product_17306,catalog_product_17307,catalog_product_17308,catalog_product_17309,catalog_product_17310,catalog_product_18996,catalog_product_18997,catalog_product_18998,catalog_product_18999,catalog_product_19000,catalog_product_19001,catalog_product_19002,catalog_product_19003,catalog_product_14357,catalog_product_11543,catalog_product_11544,catalog_product_11545,catalog_product_11546,catalog_product_11547,catalog_product_11548,catalog_product_15891,catalog_product_15893,catalog_product_15894,catalog_product_15895,catalog_product_15897,catalog_product_15898,catalog_product_15899,catalog_product_15901,catalog_product_18234,catalog_product_18235,catalog_product_18236,catalog_product_18237,catalog_product_18238,catalog_product_14359,catalog_product_11572,catalog_product_11573,catalog_product_11574,catalog_product_11575,catalog_product_11576,catalog_product_11577,catalog_product_11595,catalog_product_11596,catalog_product_11597,catalog_product_11598,catalog_product_11599,catalog_product_11600,catalog_product_11589,catalog_product_11590,catalog_product_11591,catalog_product_11592,catalog_product_11593,catalog_product_11594,catalog_product_16171,catalog_product_17313,catalog_product_17314,catalog_product_17315,catalog_product_17316,catalog_product_17317,catalog_product_17318,catalog_product_17344,catalog_product_17345,catalog_product_17346,catalog_product_17347,catalog_product_17348,catalog_product_17349,catalog_product_17828,catalog_product_17829,catalog_product_17830,catalog_product_17831,catalog_product_17832,catalog_product_14366,catalog_product_11653,catalog_product_11654,catalog_product_11655,catalog_product_11656,catalog_product_11657,catalog_product_11658,catalog_product_11659,catalog_product_11660,catalog_product_11661,catalog_product_11662,catalog_product_11663,catalog_product_11664,catalog_product_11680,catalog_product_11681,catalog_product_11682,catalog_product_11683,catalog_product_11684,catalog_product_11685,catalog_product_16071,catalog_product_16072,catalog_product_16073,catalog_product_16074,catalog_product_16075,catalog_product_18208,catalog_product_18209,catalog_product_18210,catalog_product_18211,catalog_product_18212,catalog_product_18213,catalog_product_18214,catalog_product_18215,catalog_product_18216,catalog_product_18217,catalog_product_14423,catalog_product_12004,catalog_product_12005,catalog_product_12006,catalog_product_12007,catalog_product_12008,catalog_product_12009,catalog_product_12010,catalog_product_12011,catalog_product_12012,catalog_product_12013,catalog_product_12014,catalog_product_12016,catalog_product_12017,catalog_product_12018,catalog_product_12019,catalog_product_12020,catalog_product_12021,catalog_product_12022,catalog_product_12023,catalog_product_16077,catalog_product_16078,catalog_product_16079,catalog_product_16080,catalog_product_16081,catalog_product_16082,catalog_product_16083,catalog_product_16084,catalog_product_16085,catalog_product_16086,catalog_product_16087,catalog_product_16088,catalog_product_18353,catalog_product_14436,catalog_product_12214,catalog_product_12215,catalog_product_12216,catalog_product_12217,catalog_product_12218,catalog_product_12219,catalog_product_14441,catalog_product_12293,catalog_product_12294,catalog_product_12295,catalog_product_12296,catalog_product_12297,catalog_product_12298,catalog_product_12305,catalog_product_12306,catalog_product_12307,catalog_product_12308,catalog_product_12309,catalog_product_12310,catalog_product_12311,catalog_product_12312,catalog_product_12313,catalog_product_12314,catalog_product_12315,catalog_product_12316,catalog_product_16094,catalog_product_16095,catalog_product_16096,catalog_product_16097,catalog_product_16098,catalog_product_16099,catalog_product_16101,catalog_product_16102,catalog_product_16103,catalog_product_16104,catalog_product_16105,catalog_product_16106,catalog_product_14456,catalog_product_12450,catalog_product_12451,catalog_product_12452,catalog_product_12453,catalog_product_12454,catalog_product_12455,catalog_product_15930,catalog_product_15924,catalog_product_15925,catalog_product_15926,catalog_product_15927,catalog_product_15928,catalog_product_15929,catalog_product_15931,catalog_product_15918,catalog_product_15919,catalog_product_15920,catalog_product_15921,catalog_product_15922,catalog_product_15923,catalog_product_15932,catalog_product_14513,catalog_product_12692,catalog_product_12693,catalog_product_12694,catalog_product_12695,catalog_product_12696,catalog_product_12697,catalog_product_15878,catalog_product_15879,catalog_product_15880,catalog_product_15881,catalog_product_15882,catalog_product_14515,catalog_product_12707,catalog_product_12708,catalog_product_12709,catalog_product_12710,catalog_product_12711,catalog_product_12712,catalog_product_12713,catalog_product_15910,catalog_product_15911,catalog_product_15912,catalog_product_15913,catalog_product_15914,catalog_product_15915,catalog_product_15916,catalog_product_15903,catalog_product_15904,catalog_product_15905,catalog_product_15906,catalog_product_15907,catalog_product_15908,catalog_product_15909,catalog_product_18411,catalog_product_18412,catalog_product_18413,catalog_product_18414,catalog_product_18415,catalog_product_18416,catalog_product_18405,catalog_product_18406,catalog_product_18407,catalog_product_18408,catalog_product_18409,catalog_product_18410,catalog_product_14516,catalog_product_12714,catalog_product_12715,catalog_product_12716,catalog_product_12717,catalog_product_12718,catalog_product_12719,catalog_product_15934,catalog_product_15935,catalog_product_15936,catalog_product_15937,catalog_product_15938,catalog_product_15939,catalog_product_15940,catalog_product_15941,catalog_product_15942,catalog_product_15943,catalog_product_15944,catalog_product_15945,catalog_product_15946,catalog_product_15947,catalog_product_15948,catalog_product_15949,catalog_product_15950,catalog_product_15951,catalog_product_15952,catalog_product_15953,catalog_product_15954,catalog_product_15955,catalog_product_17443,catalog_product_17444,catalog_product_17445,catalog_product_17446,catalog_product_17447,catalog_product_17448,catalog_product_17449,catalog_product_15460,catalog_product_15436,catalog_product_15437,catalog_product_15438,catalog_product_15439,catalog_product_15440,catalog_product_15441,catalog_product_15443,catalog_product_15444,catalog_product_15445,catalog_product_15446,catalog_product_15448,catalog_product_15449,catalog_product_15450,catalog_product_15452,catalog_product_15455,catalog_product_15456,catalog_product_15457,catalog_product_15458,catalog_product_15461,catalog_product_15462,catalog_product_15463,catalog_product_15464,catalog_product_17394,catalog_product_17395,catalog_product_17396,catalog_product_17397,catalog_product_17398,catalog_product_17399,catalog_product_16859,catalog_product_16838,catalog_product_16839,catalog_product_16840,catalog_product_16841,catalog_product_16842,catalog_product_16843,catalog_product_16844,catalog_product_16845,catalog_product_16846,catalog_product_16847,catalog_product_16848,catalog_product_16849,catalog_product_16850,catalog_product_16851,catalog_product_16852,catalog_product_16853,catalog_product_16854,catalog_product_16855,catalog_product_16856,catalog_product_16857,catalog_product_16858,catalog_product_18591,catalog_product_18592,catalog_product_18593,catalog_product_18594,catalog_product_18595,catalog_product_18596,catalog_product_18597,catalog_product_16964,catalog_product_16958,catalog_product_16959,catalog_product_16960,catalog_product_16961,catalog_product_16962,catalog_product_16963,catalog_product_16971,catalog_product_17043,catalog_product_17037,catalog_product_17038,catalog_product_17039,catalog_product_17040,catalog_product_17041,catalog_product_17042,catalog_product_17044,catalog_product_17051,catalog_product_17045,catalog_product_17046,catalog_product_17047,catalog_product_17048,catalog_product_17049,catalog_product_17050,catalog_product_18691,catalog_product_18488,catalog_product_18489,catalog_product_18490,catalog_product_18491,catalog_product_18692,catalog_product_18693,catalog_product_17053,catalog_product_17054,catalog_product_17055,catalog_product_17056,catalog_product_17057,catalog_product_17058,catalog_product_17060,catalog_product_18694,catalog_product_18493,catalog_product_18494,catalog_product_18495,catalog_product_18496,catalog_product_18497,catalog_product_18695,catalog_product_18696,catalog_product_16965,catalog_product_16967,catalog_product_16968,catalog_product_16969,catalog_product_18119,catalog_product_18120,catalog_product_18121,catalog_product_18122,catalog_product_18123,catalog_product_18124,catalog_product_19004,catalog_product_19005,catalog_product_19006,catalog_product_19007,catalog_product_19008,catalog_product_19009,cms_block_customer-register,cms_block_,cms_block_newsletter_popup
I'm just not sure of the implications of this solution, like when cleaning or flushing the cache or whatever. I'll just leave it to you guys. Thanks!
Encountered an issue wherein certain product listing pages with 60 products per page loads properly when using M2's built-in FPC but returns Error 500 when using Varnish.
Preconditions
Steps to reproduce
Expected result
The product listing page should display properly
Actual result
Page returns Error 500
After a lot of hours spent in investigating the issue and even trying to tune Varnish by configuring certain parameters, I noticed this recently submitted issue here in the M2 repository and tried to look into the X-Magento-Tags header.
#7640 X-Magento-Tags header containing whitespaces causes exception
I didn't find any pages that has spaces in the said header, nor blocks with block ID containing spaces, but I noticed something: perhaps due to the sheer amount of products that is being displayed in our product listing pages, some of them have X-Magento-Tags header lengths that reach up to more than 16k characters. I commented-out the "unset" statements in the default.vcl Varnish configuration file to check the contents of the said header and one of them are as follows:
I paginated through the category and found a page that loads properly even with X-Magento-Tags header containing almost 5000 characters.
So what I did was to try modifying the setHeader() function of vendor/magento/framework/HTTP/PhpEnvironment/Response.php to create a sha1 hash of the header's value and then the pages started loading properly.
So I guess a possible fix for this is to either hash the value of the X-Magento-Tags or perhaps compress it if we want a reversible string, e.g.:
$value = base64_encode(gzencode($value));I'm just not sure of the implications of this solution, like when cleaning or flushing the cache or whatever. I'll just leave it to you guys. Thanks!