Why is the newer 2.3.5 Step/Eav/Data.php so much smaller

[ChameleonDevil]

Magento 1 1.7.0.2 to Magento 2.3.4 / Magento 2.3.5 Migration

I have so many issues with Migration from 1.7.0.2 to 2.3.4 / 2.3.5

Previously, I managed to get migration to work (1.7.0.2 - to 2.2x); but newer versions give more issues. I merged everything to map.xml and deltalog.xml from older versions that were required for the custom tables / data fields.

The recent version 2.3.4 Data Migration Tool
composer require magento/data-migration-tool:2.3.4 failed at another section compared to this 2.3.5 version.

So decided to update:
composer require magento/product-community-edition:2.3.5
composer require magento/data-migration-tool:2.3.5

Now the error moved to this:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '9-meta-information' for key 'EAV_ATTRIBUTE_GROUP_ATTRIBUTE_
SET_ID_ATTRIBUTE_GROUP_CODE'

So then I investigated the code of this commit and a previous commit

Two questions:

1. The first one that made me query this is that the older commit had this method:

    public function getIterationsCount()
    {
        return count($this->readerGroups->getGroup('documents'));
    }

and was called like this inside perform()
$this->progress->start($this->getIterationsCount());

The newer commit doesnt have this method anymore:
$this->progress->start(7);

2. The newer commit has 672 lines of code, the older commit 1309 lines.
   Ever since that commit where $this->progress->start(7);, is called, newer code was added and the latest commit is on 719 lines.

• Why is the $this->progress->start(7); now hardcoded, and the dynamic count calculation method removed, isn't there alot of code missing that were helping with processing from the older commits?

• Since this version has much less processing code, what are my options now for Migrating 2.3.5?

• Obviously if Magento 2.3.4 / data-migration-tool:2.3.4 was working, I would have done it that way... but it doesn't work either.

• Should I revert data-migration-tool to 2.3.4 and then try and migrate directly on Magento 2.3.5?

• The questions are basically :
  1 - $this->progress->start(7); made me question why its not dynamic when it previously were, so didn't alot of processing code get lost somewhere?
  2 - OR is the difference between 2.3.4 so vast to 2.3.5 that less EAV processing code is required?

[ChameleonDevil]

Just want to add: this happens on a clean install of Magento 2.3.5.

[KZNcode]

Think you narrowed down the culprit. Now we need a fixed version of data-migration-tool:2.3.5 , or try a workaround like downgrading to data-migration-tool 2.3.4 and trying that on M2.3.5... or at worse, downgrade both the tool and magento...ugh.

Added my error to your other issue
#813

[ChameleonDevil]

I have tried 2.3.4 migration-tool with M2.3.4 which had other errors; before deciding to upgrade to 2.3.5 since its a very recent version.

I can confirm 2.3.4 does not work either. As for older versions, that most definitely skips alot of mapped columns/tables, so you will have to merge all the recently added mapped items. I also do not know how this will go since the issue regardless is to test delta migration (migrate:delta), which I can't get to now since these issues exist in migrate:data.

[drinkingsouls]

I was running regular delta from a Magento 1.9.0.1 to Magento 2.3.4 for a period absolutely fine.
I updated to Magento 2.3.5-p1 yesterday and I am now unable to delta. I edited the map files, including configs but I can no longer migrate data fully.
I get parts of data across, incomplete product. I don't get Price, Visibility, Status, Websites or Categories.

[ChameleonDevil]

Unfortunately with my data, especially products and categories have so many issues that I need to apply changes manually; MySQL queries etc, so I am still not even able to get to the stage where I'm ready to run migrate:delta.

For instance, my Magento 1.7.x data seems to not have page_layout correctly set - but they work perfectly on the frontend - for years now.

SELECT * FROM catalog_product_entity_varchar evc
WHERE evc.attribute_id IN (SELECT attribute_id FROM eav_attribute WHERE attribute_code = 'page_layout'); 

or

SELECT evc.* FROM catalog_product_entity_varchar AS evc
INNER JOIN eav_attribute AS eatt
ON evc.attribute_id = eatt.attribute_id
WHERE eatt.attribute_code = 'page_layout';

1553783 4 107 0 16911 No layout updates
1553850 4 107 0 16912 No layout updates

When indexing these products, they just return blank page (if you inspect the HTML source you see the BODY has a attribute indicate 'no-layout-updates').

I had to run

UPDATE catalog_product_entity_varchar AS evc SET evc.value = '1column' WHERE evc.attribute_id IN (SELECT attribute_id FROM eav_attribute WHERE attribute_code = 'page_layout');

in order for the products to apply.

Clearly Magento 2 ignores No layout updates or NULL as values, or at least doesn't process it the same as Magento 1 used to.

These are changes that I had to apply manually, 100s of them. Unfortunately the data-migration-tool is not in a state to correctly migrate 1.7x to 2.3.x.

I used to have a data-migration-tool version that migrated better, but this is not an option since my main concern is that I need to create steps to reproduce an import so that I can do migrate:delta eventually.

[yogeshkhasturi]

@ChameleonDevil You are right even migration form 1.8 is not successful. "catalog_product_entity_varchar" table has much value as NULL and i changed pagelayout manually but still many other need attribute value is NULL do we need to add value to all manually or what needs to be done?

Migration is stuck at this table https://prnt.sc/spacyo

Thanks,

[ChameleonDevil]

@yogeshkhasturi

You have uncovered something that I never noticed, but it might be a big issue. Also if you have any more details of which important attributes are NULL, it will be useful for @victor-v-rad .

To start off this discussion:

• Running this on a Magento 1 database (which has 262 attributes): returns 109,718 rows
• Running this on a migrated Magento 2 database: (which has 271 attributes - 9 of which are the default M2 fresh attributes): returns 30,670 rows only.

SELECT COUNT(*) FROM catalog_product_entity_varchar cpe INNER JOIN eav_attribute eAtt ON cpe.attribute_id = eAtt.attribute_id WHERE cpe.value = "" OR cpe.value IS NULL;

Same for catalog_product_entity_text:

• M1 : 40,895 rows
• M2 : 22,244 rows

It would be useful if you have more information of which important attributes now don't have values. With 262 attributes for me it is really hard to see.

This is concerning though: which things do we not see imported?

[yogeshkhasturi]

@ChameleonDevil Can we do like this.

If the Attribute value is NULL system should add "String" there just to avoid the Process to be hanged. ?
I don't know why @victor-v-rad is not available and not helping.

[victor-v-rad]

Please take a look at our Wiki.

Reported issues in this project are supported mainly by Magento community. Please submit a support ticket here https://magento.com/contact-us if you migrate to Magento 2 Commerce

Also please consider making your contribution into the project through pull requests

[yogeshkhasturi]

Hi @victor-v-rad I will do contribution, can you please answer below question so that i will do that in my end and contribute through pull request once that issue is fixed.

Thanks

[danmentzer]

I noticed a similar issue with the varchar table for products.
When running the migration tool with 2.3.5 I don't get any custom attributes from the varchar table carried over to m2 from m1.

For example i have an attribute station_sku its a special sku we use with shipstation but it is a different value then the regular magento sku.

And none of that attribute were carried over.

Let me know any thoughts on this issue.
There were no errors with the migration.

[ChameleonDevil]

I have not as of yet uncovered which missing values for custom attributes I mentioned above (inconsistent counts) except for my configurable products, the simple products do not have their catalog_product_entity_int to catalog_product_super_link values defined.

It seems migration decided to INNER JOIN() my source data and didnt import any simple products that do not have the attribute defined in attribute set --- even though Magento 1 worked perfectly with simple products when configurable using it this way.

• I now have configurables that absolutely do not work - they just show out of stock. Nobody on StackExchange even has any assistance for me.

As for which other attributes miss data - just the count above tells me.... I do not yet have any need for missing attributes if Configurables do not even work :(

[danmentzer]

I have managed to track my issue down to the PostProcessing Step of the migration one of the clean up function calls removes my attributes from the attribute set they are assigned to then removes the data of those attributes.

Prior to this step I can see in the db that the data did import prior to the step.
So I am currently trying to narrow down the clean up method that could be removing my data.

[ChameleonDevil]

I have managed to track my issue down to the PostProcessing Step of the migration one of the clean up function calls removes my attributes from the attribute set they are assigned to then removes the data of those attributes.

Prior to this step I can see in the db that the data did import prior to the step.
So I am currently trying to narrow down the clean up method that could be removing my data.

Wow that is just unfortunate that clean up removes attributes from attribute sets.... this does not seem like expected behaviour. I wonder if this clean up also removes catalog_product_entity_int entries and their catalog_product_super_link and super_attribute for those attribute sets that do not currently have the attribute item in the attribute set...

Is there any starting point you can point me to where you found these cleanup code, and see if I can figure out based on my product information (configurables) if it is similar behaviour.

[danmentzer]

@ChameleonDevil The actual code can be found in

/vendor/magento/data-migration-tool/src/Migration/Step/PostProcessing/Data.php

within the perform() function
I haven't been able to determine which exact function within that is the issue yet.

[danmentzer]

@ChameleonDevil
From what I can see the clean functions are just a symptom of another issue.
I went back through my m1 db and found that the data was first removed on the m1 side and then clean just removes it on the m2 side.

If you have any idea where in the migration process it removes data on the m1 side that would be helpful to know.

[ChameleonDevil]

@ChameleonDevil The actual code can be found in

/vendor/magento/data-migration-tool/src/Migration/Step/PostProcessing/Data.php
within the perform() function
I haven't been able to determine which exact function within that is the issue yet.

Thank you @danmentzer, I was going to start debugging there as well ... but then you mentioned:

@ChameleonDevil
From what I can see the clean functions are just a symptom of another issue.
I went back through my m1 db and found that the data was first removed on the m1 side and then clean just removes it on the m2 side.

If you have any idea where in the migration process it removes data on the m1 side that would be helpful to know.

I am also not sure where it happens right at the moment. When I see anything I will update. Do you mean the migration tool removes things from M1 first?? This is dangerous, as we run live websites and one of the _features of this tool is delta migration which depends on triggers on a M1 livesite. I hope that is not true; I have not noticed anything myself missing.

[awby]

It appears that the call, in perform() to
$this->attributeSetLeftoverDataCleaner->clean();
results in the removal of product values for attributes which aren't in the attribute set

[ChameleonDevil]

It appears that the call, in perform() to
$this->attributeSetLeftoverDataCleaner->clean();
results in the removal of product values for attributes which aren't in the attribute set

|Thank you :)

I will then try a new magento 2 migration when I disable / investigate the clean() function further. Maybe this will be sufficient to have products' values in catalog_product_super_link / catalog_product_entity_int for those attributes.

[ChameleonDevil]

Edit 2021:

I just want to reiterate this:

Summary: Looking at the current commits (2021) for Step/Eav/Data.php it still has less code (alot old code removed.... and hard-coded quantity $this->progress->start(7); vs OLDER commits: $this->progress->start($this->getIterationsCount()); ) as previous commits I mentioned in that topic; I wondered then, and still do: is this not the reason why Step/Eav/Data.php processing is giving us so much issues? Why were so much attribute processing removed, why are the dynamic getIterationsCount() not used anymore?