Skip to content

Magento fails to unserialize config value from Redis in HA setup #32237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 of 5 tasks
peimanja opened this issue Feb 22, 2021 · 4 comments
Closed
1 of 5 tasks

Magento fails to unserialize config value from Redis in HA setup #32237

peimanja opened this issue Feb 22, 2021 · 4 comments
Labels
Issue: ready for confirmation

Comments

@peimanja
Copy link

Preconditions (*)

  1. Magento 2.3.5
  2. Database MySQL 5.7.28
  3. Default Cache: Redis 6.0.5
  4. Magento Deployed on EKS (3 Pods) with persistent storage

Steps to reproduce (*)

  1. Deploy Magento on Kubernetes using MySQL as Database and Redis as default cache

Expected result (*)

  1. all Pods can use Redis as cache without any issue

Actual result (*)

Magento works fine without setting redis as the default cache.

When deploy Magento and configure to use redis as cache, only the first pod that comes up and writes to redis works properly but if a request get routed to rest of the pods it fails:

curl --request GET \
  --url 'https://store.mycompany.com/rest/V1/products?searchCriteria=' \
  --header 'Authorization: Bearer $TOKEN'
An error has happened during application run. See exception log for details.

Here are logs from problematic pods:

/opt/bitnami/magento/htdocs/var/log/exception.log 
[2021-02-22 07:23:12] main.CRITICAL: Unable to unserialize value, string is corrupted.
{"exception":"[object] (InvalidArgumentException(code: 0): 
Unable to unserialize value, string is corrupted. at /bitnami/magento/htdocs/vendor/magento/framework/Serialize/Serializer/Serialize.php:38)"} []

Same in the system.log

/opt/bitnami/magento/htdocs/var/log/system.log 
[2021-02-22 07:58:03] main.ERROR: Unable to unserialize value, string is corrupted.
#0 [internal function]: Magento\Framework\Serialize\Serializer\Serialize->Magento\Framework\Serialize\Serializer\{closure}(8, 'unserialize(): ...', '/bitnami/magent...', 42, Array)
#1 /bitnami/magento/htdocs/vendor/magento/framework/Serialize/Serializer/Serialize.php(42): unserialize('\xDB\xED\xE8&t$%\xED\xCA\x84\xBC&N\xEB\xCB...', Array)
#2 /bitnami/magento/htdocs/vendor/magento/module-config/App/Config/Type/System.php(252): Magento\Framework\Serialize\Serializer\Serialize->unserialize('\xDB\xED\xE8&t$%\xED\xCA\x84\xBC&N\xEB\xCB...')
#3 /bitnami/magento/htdocs/vendor/magento/framework/Cache/LockGuardedCacheLoader.php(84): Magento\Config\App\Config\Type\System->Magento\Config\App\Config\Type\{closure}()
#4 /bitnami/magento/htdocs/vendor/magento/module-config/App/Config/Type/System.php(261): Magento\Framework\Cache\LockGuardedCacheLoader->lockedLoadData('SYSTEM_CONFIG', Object(Closure), Object(Closure), Object(Closure))
#5 /bitnami/magento/htdocs/vendor/magento/module-config/App/Config/Type/System.php(195): Magento\Config\App\Config\Type\System->loadDefaultScopeData('default')
#6 /bitnami/magento/htdocs/vendor/magento/module-config/App/Config/Type/System.php(169): Magento\Config\App\Config\Type\System->getWithParts('default/newreli...')
#7 /bitnami/magento/htdocs/vendor/magento/framework/App/Config.php(132): Magento\Config\App\Config\Type\System->get('default/newreli...')
#8 /bitnami/magento/htdocs/vendor/magento/framework/App/Config.php(80): Magento\Framework\App\Config->get('system', 'default/newreli...')
#9 /bitnami/magento/htdocs/vendor/magento/module-new-relic-reporting/Model/Config.php(174): Magento\Framework\App\Config->getValue('newrelicreporti...')
#10 /bitnami/magento/htdocs/vendor/magento/module-new-relic-reporting/Plugin/StatePlugin.php(97): Magento\NewRelicReporting\Model\Config->isSeparateApps()
#11 /bitnami/magento/htdocs/vendor/magento/module-new-relic-reporting/Plugin/StatePlugin.php(60): Magento\NewRelicReporting\Plugin\StatePlugin->shouldSetAppName()
#12 /bitnami/magento/htdocs/vendor/magento/framework/Interception/Interceptor.php(146): Magento\NewRelicReporting\Plugin\StatePlugin->afterSetAreaCode(Object(Magento\Framework\App\State\Interceptor), NULL, 'crontab')
#13 /bitnami/magento/htdocs/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Framework\App\State\Interceptor->Magento\Framework\Interception\{closure}('crontab')
#14 /bitnami/magento/htdocs/generated/code/Magento/Framework/App/State/Interceptor.php(26): Magento\Framework\App\State\Interceptor->___callPlugins('setAreaCode', Array, Array)
#15 /bitnami/magento/htdocs/vendor/magento/framework/App/Cron.php(78): Magento\Framework\App\State\Interceptor->setAreaCode('crontab')
#16 /bitnami/magento/htdocs/vendor/magento/module-cron/Console/Command/CronCommand.php(117): Magento\Framework\App\Cron->launch()
#17 /bitnami/magento/htdocs/vendor/symfony/console/Command/Command.php(255): Magento\Cron\Console\Command\CronCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /bitnami/magento/htdocs/vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /bitnami/magento/htdocs/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Cron\Console\Command\CronCommand\Interceptor->___callParent('run', Array)
#20 /bitnami/magento/htdocs/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Cron\Console\Command\CronCommand\Interceptor->Magento\Framework\Interception\{closure}(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#21 /bitnami/magento/htdocs/generated/code/Magento/Cron/Console/Command/CronCommand/Interceptor.php(26): Magento\Cron\Console\Command\CronCommand\Interceptor->___callPlugins('run', Array, Array)
#22 /bitnami/magento/htdocs/vendor/symfony/console/Application.php(1001): Magento\Cron\Console\Command\CronCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /bitnami/magento/htdocs/vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(Magento\Cron\Console\Command\CronCommand\Interceptor), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /bitnami/magento/htdocs/vendor/magento/framework/Console/Cli.php(115): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /bitnami/magento/htdocs/vendor/symfony/console/Application.php(147): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /bitnami/magento/htdocs/bin/magento(23): Symfony\Component\Console\Application->run()
#27 {main} [] []

From my observation, looks like when cache is enabled, Magento reads some of the config from Redis on each request:

1613981738.746761 [0 10.19.202.148:50418] "hget" "zc:k:f73_SYSTEM_DEFAULT" "d"
1613981738.851295 [0 10.19.202.148:50418] "hget" "zc:k:f73_GLOBAL_PLUGIN_LIST" "d"
1613981738.857723 [0 10.19.202.148:50418] "hget" "zc:k:f73_SYSTEM_DEFAULT" "d

By running magento config:show on pods with issue, I noticed they are failing at getting one of the configs:

# magento config:show

yotpo/module_info/yotpo_installation_date - 2021-02-22
yotpo/sync_settings/orders_sync_start_date - 2021-02-22
web/unsecure/base_url - http://store.mycompany.com/
web/secure/base_url - https://store.mycompany.com/
web/secure/use_in_frontend - 1
web/secure/use_in_adminhtml - 1
web/cookie/cookie_domain - store-mle-magento.internal.dev.cloud.skillz.com
general/locale/code - en_US
general/locale/timezone - America/Los_Angeles
general/region/display_all - 1
general/region/state_required - AU,BR,CA,CH,CN,CO,EE,ES,HR,IN,LT,LV,MX,PL,RO,US
currency/options/base - USD
currency/options/default - USD
Unable to unserialize value, string is corrupted.

The problematic path looks like to be payment/authorizenet_acceptjs/currency.

I am pretty much stuck as this does not makes sense to me where the first pod can read and unserialize all values from redis without any issue while other pods fail at this specific path.

on the the first pod it shows the config on this path without any issue:

# magento config:show payment/authorizenet_acceptjs/currency
USD

I have tried flushing the redis cache and multiple fresh installs and no luck passing this issue.

  • Severity: S0 - Affects critical data or functionality and leaves users without workaround.
  • Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
  • Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
  • Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
  • Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.
@m2-assistant
Copy link

m2-assistant bot commented Feb 22, 2021

Hi @peimanja. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

  • Summary of the issue
  • Information on your environment
  • Steps to reproduce
  • Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

@peimanja
Copy link
Author

I think I figured this out as soon as I post this.
By looking at /opt/bitnami/magento/htdocs/app/etc/env.php, looks like the crypt key was different between pods.

I notice there is a --key flag that can be passed during the installation to set that. Will try that and if that works, we can close this issue

magento setup:install --key="YOURKEYHERE" ... OTHER ARGS ...

@peimanja
Copy link
Author

It worked!

@ikevinsolomon
Copy link

I'm facing a similar issue on EKS with redis, even with all the pods having the same app/etc/env.php with same crypt key.
I also reckon this might also have something to do something with #29964

Can you please share more of your finding or your env.php

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Issue: ready for confirmation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ikevinsolomon @peimanja