Added hash based check to prevent image duplication on product import

[erfanimani]

Description

I've added a MD5 based hash check to prevent image duplication upon successive product CSV imports.

Note that this is a proof of concept, even though it works — sort of.

The problem is that the Add/Update product import method should be declarative, and thus idempotent — it should not cause side effects when importing the same CSV multiple times.

As it stands now, this is not the case (confirmed), running the import multiple times causes images to be added. The "Replace" import method is beyond silly for production systems as it trashes the old IDs (and takes quotes, reports, wish lists, comparison lists, and so on with it).

We also can't replace just the images as it causes the different filenames, thus realistically a filename + hash comparison is the way to go. But filename checks are hard because we're not keeping track of the file metadata upon import (and as the file is renamed when Magento moves it into media/catalog/product, any reference to the old filename is lost) — so the easiest solution that kinda works is a hash based check.

I want to know whether this is the way to implement it or not. It's already a messy, mostly undocumented, class and no straight forward way to implement it properly, but any guidance would be helpful.

• I haven't fully tested this yet
• I'm not happy with code quality yet
• Doesn't delete images, only skips addition of existing images
• Doesn't rename the files — only compares file hash (to do this, I believe, would require Magento to store file metadata)
• Does not work when images are referenced using HTTP
• Crashes if image was not found (Warning: md5_file: failed to open stream..)

Fixed Issues

• Images Not Replaced via CSV Import #14398 : Images Not Replaced via CSV Import

Manual testing scenarios (*)

1. Import product CSV using Add/Update method multiple times
2. Check if image duplication happens or not

There's many testing scenario's actually. Some of them being:

• Use HTTP references in images
• Same filename different file
• Different filename same file
• Same file path for multiple products
• Change image labels/titles, ensure they're updated
• Manipulate thumbnail, small image and base image tags, and ensure it's updated properly after import
• etc.

Please let me know how to proceed and I can amend my commit message and implement it properly.

[magento-engcom-team]

Hi @erfanimani. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento-engcom-team give me test instance - deploy test instance based on PR changes
• @magento-engcom-team give me 2.3-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Assistant documentation

[raivisdejus]

@erfanimani I have used your PR and added image deletion to it #21855

Thanks for the PR, saved me quite a lot of time

[sidolov]

Please, make a method private and add strict types ti the method

[sidolov]

Please, make a method private and add strict types ti the method

[sidolov]

@erfanimani , I am closing this PR now due to inactivity.
Please reopen and update if you wish to continue.
Thank you for the collaboration!

[m2-assistant]

Hi @erfanimani, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

[cptX]

As I just realized that magento uses different copies of images for the same images when uploaded manually during new product creation I want to ask if the attempt here to solve the issue would work also when creating a new product and reupload the same images. Would this also use the hash check or the check runs only during csv import?

[erfanimani]

@cptX it's been a while, but I believe it just affects the CSV import.