Migration from 1.8.1.0 stuck at map step step: table: catalog_product_entity_varchar]:

[yogeshkhasturi]

Preconditions

1. Migration from Magento 1.8.1.0 to Magento 2.3.3 and 2.3.5(both have same error)
2. tried Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812 it was related to that. but m1 table "catalog_product_entity_varchar " has thousands of more Value as NULL

Steps to reproduce

1. Data Migration -> Map Step

Expected result

1. it should avoid all issues and complete the migration

Actual result

1. [2020-05-28 08:38:58][INFO][mode: data][stage: data migration][step: Map Step]: started [2020-05-28 08:39:01][DEBUG][mode: data][stage: data migration][step: Map Step][table: adminnotification_inbox]: migrating [2020-05-28 08:39:13][DEBUG][mode: data][stage: data migration][step: Map Step][table: captcha_log]: migrating [2020-05-28 08:39:16][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_entity]: migrating [2020-05-28 08:39:18][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_entity_datetime]: migrating [2020-05-28 08:39:20][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_entity_decimal]: migrating [2020-05-28 08:39:23][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_entity_int]: migrating [2020-05-28 08:39:25][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_entity_text]: migrating [2020-05-28 08:39:34][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_product]: migrating [2020-05-28 08:39:40][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_category_product_index]: migrating [2020-05-28 08:40:08][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_compare_item]: migrating [2020-05-28 08:40:11][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_bundle_option]: migrating [2020-05-28 08:40:13][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_bundle_option_value]: migrating [2020-05-28 08:40:15][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_bundle_selection]: migrating [2020-05-28 08:40:17][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_bundle_selection_price]: migrating [2020-05-28 08:40:19][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity]: migrating [2020-05-28 08:40:28][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_datetime]: migrating [2020-05-28 08:40:37][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_decimal]: migrating [2020-05-28 08:40:53][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_gallery]: migrating [2020-05-28 08:40:55][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_int]: migrating [2020-05-28 08:41:55][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_media_gallery]: migrating [2020-05-28 08:42:13][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_text]: migrating [2020-05-28 08:42:24][DEBUG][mode: data][stage: data migration][step: Map Step][table: catalog_product_entity_varchar]: migrating

Additional notes

this is not completing migration at all.

Hope @victor-v-rad can help

Thanks

[m2-assistant]

Hi @yogeshkhasturi. Thank you for your report.
To help us process this issue please make sure that you provided sufficient information.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[yogeshkhasturi]

@victor-v-rad hopefully you can check this ASAP,

[ChameleonDevil]

@yogeshkhasturi

(Also posted in #812)

You have uncovered something that I never noticed, but it might be a big issue. Also if you have any more details of which important attributes are NULL, it will be useful for @victor-v-rad .

To start off this discussion:

• Running this on a Magento 1 database (which has 262 attributes): returns 109,718 rows
• Running this on a migrated Magento 2 database: (which has 271 attributes - 9 of which are the default M2 fresh attributes): returns 30,670 rows only.

SELECT COUNT(*) FROM catalog_product_entity_varchar cpe INNER JOIN eav_attribute eAtt ON cpe.attribute_id = eAtt.attribute_id WHERE cpe.value = "" OR cpe.value IS NULL;

Same for catalog_product_entity_text:

• M1 : 40,895 rows
• M2 : 22,244 rows

It would be useful if you have more information of which important attributes now don't have values. With 262 attributes for me it is really hard to see.

This is concerning though: which things do we not see imported?

[yogeshkhasturi]

@ChameleonDevil How can we know that which attribute in m2 can have NULL value so that we can add dummy value to other attributes. ?

Thanks

[ChameleonDevil]

Hi @yogeshkhasturi

I am not sure if you found a solution, please update if you have more information.
I guess something like this:

1. is_required ? yes/no
2. default_value ? [NULL]
3. entity_type_id ?

---

SELECT * FROM eav_entity_type;
Returns catalog_product as entity_type_id = 4

---

SELECT * FROM eav_attribute AS ea WHERE ea.is_required = 0 AND ea.default_value IS NULL AND ea.entity_type_id = 4;

I think the above will give you non-required fields of catalog products that allow NULLs, for all others you would need to investigate more!

---

SELECT * FROM eav_attribute AS eaNot WHERE eaNot.entity_type_id = 4 AND eaNot.attribute_id NOT IN (SELECT attribute_id FROM eav_attribute AS ea WHERE ea.is_required = 0 AND ea.default_value IS NULL);

So the above gives us sort of the inverse - everything except the filtered attributes.

Please inform me if you found a better solution.

[yogeshkhasturi]

@ChameleonDevil Thanks for your reply. i was actually thinking to update 1.8 to 1.9 and then 1.9 to 2.3. but after your reply i will give it one more try. we have 54 attributes with required value.

https://prnt.sc/sruhbh

[ChameleonDevil]

Just to make sure you understood your results:

**Those are your ** is_required = 0 attributes :) You need to take the opposite of that as I had explained in previous post.

Good luck with the upgrade to 1.9x, I never successfully managed to upgrade one site without the whole theme breaking completely.

[yogeshkhasturi]

@ChameleonDevil

**Those are your ** is_required = 0 attributes :) You need to take the opposite of that as I had explained in previous post.
-- I guess I didn't understand completely, because i am not familiar with magento database too much, can you please help me to find the solution?

Good luck with the upgrade to 1.9x, I never successfully managed to upgrade one site without the whole theme breaking completely.
-- as you said theme will broken completely then we use not use that. thanks for your advice.

[yogeshkhasturi]

@ChameleonDevil I was thinking the attributes which have nulled value and are is_required = 1, should we ignore those attributes from EAV steps. will those attributes still create an issue in the Map step? what do you think?

Thanks,

[ChameleonDevil]

The suggestion is to just see what the results of these are:

SELECT * FROM eav_attribute AS ea WHERE ea.entity_type_id = 4 AND ea.is_required = 1 AND (ea.default_value IS NULL or ea.default_value = "");

This gives you all the products (entity_type_id = 4) that has (is_required = 1) and where default_value is NULL or empty string .

Then you also need to see which attributes you want to see filled with NON-NULL values (but aren't required). (is_required = 0).

Then you will need to fill those attributes (required values) in your database with values, for at least catalogproduct_entity_varchar, catalogproduct_entity_text, catalogproduct_entity_* tables etc.

I am not sure about which other tables though.

[ChameleonDevil]

@ChameleonDevil I was thinking the attributes which have nulled value and are is_required = 1, should we ignore those attributes from EAV steps. will those attributes still create an issue in the Map step? what do you think?

Thanks,

If they allow NULL or empty strings, but are required... I don't think they should give any errors. (Can't confirm atm though)

[yogeshkhasturi]

Then you will need to fill those attributes (required values) in your database with values, for at least catalogproduct_entity_varchar, catalogproduct_entity_text, catalogproduct_entity_* tables etc.

I see. i will check and i will update you how it goes,

[yogeshkhasturi]

@ChameleonDevil its not working by ignoring Product attributes in the EAV step. also, i tried to add values in those null values but after migration completed in Magento 2 data is incomplete like, the product name is missing and all.

we need to find solutions where we can ignore NULL values or bypass the values.

ANY IDEA?

T

[ChameleonDevil]

@yogeshkhasturi

I do not understand how name attribute can be empty after you added values :)

SELECT * FROM eav_attribute AS ea WHERE ea.entity_type_id = 4 AND ea.is_required = 1 AND (ea.default_value IS NULL or ea.default_value = "");

This clearly shows that attributes like name, price, sku, etc requires values even though their default values are NULL. At the very least I thought these particular ones should be given values if they somehow do not. Technically; in a proper M1 database, they should all have non-NULL values already.

SELECT DISTINCT(evar.value) FROM catalog_product_entity AS cpe LEFT JOIN catalog_product_entity_varchar AS evar ON cpe.entity_id = evar.entity_id WHERE evar.attribute_id IN (SELECT attribute_id FROM eav_attribute AS ea WHERE ea.attribute_code = 'name') ORDER BY evar.value ASC;

I checked this for instance, finding DISTINCT values from _varchar table for my products (name attribute only). I do not have any NULL or empty string values returned in this query. (EVEN RIGHT JOIN doesn't return any NULL values).

Or do you mean your name attribute is now all of a sudden gone from the database??

Either way, I do not think I ever mentioned to ignore attributes, the discussion at least was just to attempt to find a solution that we can use to 'bypass values/make non-NULL values where you need them'.

I'm out of ideas then.

@victor-v-rad Is the maintainer of this tool, he may just shed some light on the inner workings better for the EAV step.

[yogeshkhasturi]

@ChameleonDevil Actaully "name" values are not NULL but after migration Product names are not available in Products.

[ChameleonDevil]

@yogeshkhasturi Then it is probably possible that the migrated attribute sets (or attribute set groups) you use for your products have not migrated.

Investigate attribute_set table and see if the attribute 'name' is in the attribute sets that your products use. (You could quickly confirm from M2 backend as well I guess). If it is not attached to attribute groups (Those seperate sections in M2 attribute set backend, then the attribute also don't appear).