Skip to content

After 2.2.6 upgrade reports generates enormous .tar file in tmp/analytics folder in root #18243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Jilco opened this issue Sep 25, 2018 · 38 comments
Closed

After 2.2.6 upgrade reports generates enormous .tar file in tmp/analytics folder in root #18243

Jilco opened this issue Sep 25, 2018 · 38 comments
Labels
Fixed in 2.3.x The issue has been fixed in 2.3 release line Issue: Clear Description Gate 2 Passed. Manual verification of the issue description passed Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed Reproduced on 2.2.x The issue has been reproduced on latest 2.2 release

Comments

@Jilco
Copy link

Jilco commented Sep 25, 2018

Preconditions

  1. Magento 2.2.6
  2. Ngingx
  3. Redis

Steps to reproduce

  1. Upgrade to 2.2.6
  2. Switch on reports (or don't switch of if you already switched it on in earlier version)

Expected result

  1. I don't realy know, i think the files are needed for advance reports

Actual result

  1. In the /tmp/analytics folder in de root of the webserver an huge (1.7 Gb) .tar (~tmp-1537867209.3073tar.tar) file is generated every 24 hours (happend 2 times now after upgrade and exact the same time). Because it is in the tmp folder of the root/webserver Nginx cant make new sessions and the site goes down.

When i look in the .tar (i downloaded it) i see a lot of the same files and the same size. Maybe generating the file goes in a loop somewhere?
@magento-engcom-team
Copy link
Contributor

Hi @Jilco. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

  • Summary of the issue
  • Information on your environment
  • Steps to reproduce
  • Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@Jilco do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

  • yes
  • no

@magento-engcom-team magento-engcom-team added the Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed label Sep 25, 2018
@Jilco
Copy link
Author

Jilco commented Sep 25, 2018

@magento-engcom-team give me 2.2.6 instance

@magento-engcom-team
Copy link
Contributor

Hi @Jilco. Thank you for your request. I'm working on Magento 2.2.6 instance for you

@magento-engcom-team
Copy link
Contributor

Hi @Jilco, here is your Magento instance.
Admin access: https://i-18243-2-2-6.engcom.dev.magento.com/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

@Jilco
Copy link
Author

Jilco commented Sep 25, 2018

Settings in the profided instance are the same as in my situation, but i can't see what happens when generation the report files

@ghost ghost self-assigned this Sep 26, 2018
@magento-engcom-team
Copy link
Contributor

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

  • 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

    DetailsIf the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

  • 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

  • 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 4. Verify that the issue is reproducible on 2.3-develop branch

    Details- Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 5. Verify that the issue is reproducible on 2.2-develop branch.

    Details- Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

  • 6. Add label Issue: Confirmed once verification is complete.

  • 7. Make sure that automatic system confirms that report has been added to the backlog.

@ghost ghost added the Issue: Clear Description Gate 2 Passed. Manual verification of the issue description passed label Sep 26, 2018
@ghost
Copy link

ghost commented Sep 26, 2018

HI @Jilco seems like this bug is acknowledged -> #11543

@Jilco
Copy link
Author

Jilco commented Sep 26, 2018 via email

@snyderm
Copy link

snyderm commented Sep 26, 2018

This issue is 2.2.6 specific, and appears to be generated by magento advanced analytics. This issue did not occur in 2.3.

@ghost
Copy link

ghost commented Sep 26, 2018

@Jilco As a temporary solution try disable advanced reporting.

@ghost ghost removed the Progress: needs update label Sep 26, 2018
@Jilco
Copy link
Author

Jilco commented Sep 26, 2018 via email

@Jilco
Copy link
Author

Jilco commented Sep 26, 2018 via email

@ghost
Copy link

ghost commented Sep 26, 2018

@Jilco Problem in this file
selection_138
But i can't understand why this method uses in loop.

@Jilco
Copy link
Author

Jilco commented Sep 26, 2018 via email

@ghost
Copy link

ghost commented Sep 26, 2018 •
edited by ghost
Loading

@Jilco Problem in Cron for Analytics module this cron runs report to write file, if you turn off reports, they not be generate files.

@ghost ghost added Reproduced on 2.2.x The issue has been reproduced on latest 2.2 release Fixed in 2.3.x The issue has been fixed in 2.3 release line labels Sep 26, 2018
@ghost ghost closed this as completed Sep 26, 2018
@ghost
Copy link

ghost commented Oct 1, 2018

Hi @Jilco This may be fix your issue -> #18322

@ghost
Copy link

ghost commented Oct 4, 2018

Hi @Jilco This issue was fixed by this commit -> 8e1a5d3

@Flowlance
Copy link

Same problem. A 83GB tar file in /tmp/analytics...

screen shot 2018-10-08 at 10 22 57

@ghost
Copy link

ghost commented Oct 8, 2018

Hi @Flowlance This issue has been fixed by this commit -> 8e1a5d3

@maderlock
Copy link

Is this fixed in 2.2.7 then?

@Jilco
Copy link
Author

Jilco commented Dec 4, 2018 via email

@maderlock
Copy link

Sadly, saving the settings again has not helped. Two days later the disk filled up with 200Gb of rubbish. Going to have to turn this off in 2.2.7.

Would this fix be hard to back-port to 2.2?

@maderlock
Copy link

Apparently there is a back-port to 2.2 that got merged in October, but this has oddly not ended up in 2.2.7.

I've tried to override \Magento\Framework\Archive\Tar to patch this myself (pulling in everything by composer means I cannot just change the file), but sadly this has not worked because this class is pull in via a low-level "new" statement that bypasses DI.

Any advice on how this can be fixed in 2.2.7 when using composer would be gratefully received. Currently my only option looks to be to tell composer to skip the entire framework so I can copy in the framework locally and patch these few line :(

@DanielRuf
Copy link
Contributor

@engcom-backlog-nazar not sure why you linked my clone and commit but this is clearly not the cause. Also this was not what we've merged in the end.

@DanielRuf
Copy link
Contributor

@maderlock you can use composer-patches and / or the patch file of the commit.

@DanielRuf
Copy link
Contributor

https://github.com/magento/magento2/commit/45f22aca2e455f2df3fdb0f7ae91563cf388e407.patch

@stu177
Copy link

stu177 commented Jan 18, 2019

I just had this occur on 2.3.0 on Ubuntu 18.04.1.

@Ctucker9233 Ctucker9233 reopened this Jan 22, 2019
@Ctucker9233
Copy link

Not fixed as of 2.2.7.

@ghost
Copy link

ghost commented Jan 22, 2019

@Ctucker9233 this fix will be available on 2.2.8 release and 2.3.1 release

@ghost ghost closed this as completed Jan 22, 2019
@ghost
Copy link

ghost commented Jan 22, 2019

@Ctucker9233 #18322

@smadasam
Copy link

I am having the same issue on my 2.3.0 site. How do I patch it until the next release comes out?

:/tmp/analytics# ll -ah
total 62G
drwxrwxr-x 2 sw_temp psacln 4.0K Jan 25 10:02 .
drwxrwxrwt 11 root root 20K Jan 26 00:41 ..
-rw-rw-r-- 1 sw_temp psacln 101K Jan 22 10:00 ~tmp-1548151203.3161tar.tar
-rw-rw-r-- 1 sw_temp psacln 71M Jan 23 10:00 ~tmp-1548237603.0528tar.tar
-rw-rw-r-- 1 sw_temp psacln 41G Jan 24 10:02 ~tmp-1548324002.9252tar.tar
-rw-rw-r-- 1 sw_temp psacln 22G Jan 25 10:02 ~tmp-1548410402.865tar.tar

@DanielRuf
Copy link
Contributor

DanielRuf commented Jan 26, 2019 •
edited
Loading

#18243 (comment)

https://patch-diff.githubusercontent.com/raw/magento/magento2/pull/18322.patch

https://support.magento.com/hc/en-us/articles/360005484154-Create-a-patch-for-a-Magento-2-Composer-installation-from-a-GitHub-commit

@GreenGinkgo
Copy link

Just as a note, storing data in /tmp is misguided.

Shared servers that have a single /tmp would be co-mingling different user data together. I suggest that this data be stored within the Magento installation itself. Maybe in (magento)/var/analytics or similar.

@smadasam
Copy link

smadasam commented Feb 1, 2019 via email

@magento magento deleted a comment Feb 6, 2019
@srenon
Copy link
Member

srenon commented Feb 8, 2019

Just upgraded to 2.2.7 and run into this issue with over 50gb of diskspace disappear in 2 days.

If you are not using this feature, you can disable "Advanced REporting Service" in admin and flush cache.

Stores > Configuration > General > Advanced Reporting

image

@DanielRuf
Copy link
Contributor

@srenon see #18243 (comment) ;-)

@DanielRuf
Copy link
Contributor

I'm locking this issue as it is already fixed in the next releases and we have linked the patches and all needed steps to either disable Advanced Reporting or apply the patches.

@magento magento locked as resolved and limited conversation to collaborators Feb 8, 2019
@DanielRuf
Copy link
Contributor

Shared servers that have a single /tmp would be co-mingling different user data together.

They have a much more critical issue - a security issue.

This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Fixed in 2.3.x The issue has been fixed in 2.3 release line Issue: Clear Description Gate 2 Passed. Manual verification of the issue description passed Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed Reproduced on 2.2.x The issue has been reproduced on latest 2.2 release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests