Undefined index: 1-Default in Migration\Step\Eav\Data on line 624

[p810]

Preconditions

1. Migration is from 1.14.4.5 to a clean installation of 2.3.5-p1
2. DMT is 2.3.5
3. PHP is 7.2.22
4. MySQL is 5.7.25
5. A custom module Foo_Migration is used to map custom attributes, tables, etc.

The contents of eav_attribute_set for the source database are as follows:

+------------------+----------------+--------------------+------------+
| attribute_set_id | entity_type_id | attribute_set_name | sort_order |
+------------------+----------------+--------------------+------------+
|                1 |              1 | Default            |          1 |
|                2 |              2 | Default            |          1 |
|                3 |              3 | Default            |          1 |
|                4 |              4 | Default            |          2 |
|                5 |              5 | Default            |          1 |
|                6 |              6 | Default            |          1 |
|                7 |              7 | Default            |          1 |
|                8 |              8 | Default            |          1 |
|                9 |              9 | Default            |          1 |
+------------------+----------------+--------------------+------------+

And the contents of the same table in our destination database are:

+------------------+----------------+--------------------+------------+
| attribute_set_id | entity_type_id | attribute_set_name | sort_order |
+------------------+----------------+--------------------+------------+
|                1 |              1 | Default            |          2 |
|                2 |              2 | Default            |          2 |
|                3 |              3 | Default            |          1 |
|                4 |              4 | Default            |          1 |
|                5 |              5 | Default            |          1 |
|                6 |              6 | Default            |          1 |
|                7 |              7 | Default            |          1 |
|                8 |              8 | Default            |          1 |
|                9 |              9 | Default            |          1 |
+------------------+----------------+--------------------+------------+

Steps to reproduce

Run the data migration with the following:

./bin/magento migrate:settings --reset -vvv ./app/code/Foo/Migration/etc/commerce-to-commerce/1.14.4.5/config.xml
./bin/magento migrate:data --reset -vvv ./app/code/Foo/Migration/etc/commerce-to-commerce/1.14.4.5/config.xml

Expected result

1. Data should be migrated from the source database to the destination

Actual result

An exception is thrown from Migration\Step\Eav\Data::createMapAttributeSetIds() on line 624, complaining about an undefined index:

[2020-06-12 20:51:24][INFO][mode: data][stage: data migration][step: EAV Step]: started
28% [========>-------------------] Remaining Time: 3 secs
In ErrorHandler.php line 61:
                                                                                                                                                        
  [Exception]                                                                                                                                           
  Notice: Undefined index: 1-Default in /Users/pbice/dev/m2_ebscomags_2/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 624  
                                                                                                                                                        

Additional notes

From stepping through the debugger, it looks like this method is called from Migration\Step\Eav\Data::migrateAttributeSets(), which is responsible for:

1. Creating a backup of eav_attribute_set in the destination database
2. Transforming the attribute set names from Default to Migration_Default for each entity type
3. Saving the newly named default sets, and any sets for custom entity types
4. Calling out to createMapAttributeSetIds() to do some mapping before perform() advances to the next task

By the time that createMapAttributeSetIds() is called, the update has already taken place and the attribute sets have been renamed. Because this latest version of the migration tool made it so only the Migration_Default sets are saved in the database (unlike 2.3.4 and lower which kept the original attribute set names as well) the call to Migration\Step\Eav\Helper::getDestinationRecords() is going to return an array that lacks the necessary keys for the mapping, e.g. 1-Default. It will only have keys like 1-Migration_Default.

The error occurs between lines 623 and 624, when an attempt is made to get an array with the row's data by using a key that's assembled by concatenating the entity type ID and attribute set name (e.g., 1-Default):

$entityTypeId = $this->mapEntityTypeIdsDestOldNew[$record['entity_type_id']];
$newAttributeSet = $this->newAttributeSet[$entityTypeId . '-' . $record['attribute_set_name']];

$newAttributeSet will then be null and the mapping on subsequent lines can't be done.

Potential solution

I'm looking into a solution whereby an additional mapping is done, which points the dynamic source key to the corresponding row in Migration\Step\Eav\Data::$newAttributeSets. I haven't tested this yet so I don't know if this will be the right data that Migration\Step\Eav\Data::$mapAttributeSetIdsDestOldNew and Migration\Step\Eav\Data::$defaultAttributeSetIds, but it looks like it is. If this works I'll submit a PR.

[m2-assistant]

Hi @p810. Thank you for your report.
To help us process this issue please make sure that you provided sufficient information.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[p810]

Fixing the mapping seems to solve the problem, but then it goes onto encounter the undefined offset issue that's been discussed in #813. Applying the patch from ea9cc9f however has fully resolved the issue. PR incoming...

@magento I am working on this

[m2-assistant]

Hi @p810! 👋
Thank you for collaboration. Only members of Community Contributors Team are allowed to be assigned to the issue. Please use @magento add to contributors team command to join Contributors team.

[p810]

I'm welcome to feedback for the fix I just pushed at #826. I personally haven't had any trouble with the migration since applying this patch, but I opted to fallback to the destination field key in case there are scenarios where the source key should be preferred. I think that maybe the case for custom attribute sets. Assigning a new key to the map is done when an attribute set is known to be one of the default sets - I think I put that in the right conditional but let me know if not!

[ChameleonDevil]

@p810, I wish I knew, but I am so confused with this migration-tool's results at the moment.

Later on when I started investigating my products, they started showing out of stock, more specifically configurable products.

In this issue I asked why the older versions of Step\Eav\Data are so vastly different to older versions, see (#812) .

I never really got any answers, especially concerning was $this->progress->start($this->getIterationsCount()); in older versions are now `$this->progress->start(7);. I just think alot of processing code was removed (around ~500 lines if I remember), but then again; I have no idea why v2.3.5 and v.2.3.4 are required to be so different....

I am still having issues with migration, at the moment trying to solve 'out of stock issue for configurables'... and investigating it just gives so many confusing results.

For instance
SELECT * FROM eav_attribute_set attset INNER JOIN eav_entity_attribute eent ON attset.attribute_set_id = eent.attribute_set_id INNER JOIN eav_attribute_group attgroup ON attgroup.attribute_group_id = eent.attribute_group_id INNER JOIN eav_attribute att ON att.attribute_id = eent.attribute_id WHERE att.attribute_code LIKE ('%config%') ORDER BY eent.entity_attribute_id DESC;

In source M1 database : 55 results - 55 combinations of attribute sets to configurable attributes.
M2 database : 33 results

I just wish the processor did not ignore attribute sets and their details to think it is for the best. I now have configurable products and their associated simples that can not be matched when Magento runs this query on product page of configurable:

SELECT entity.sku, product_entity.entity_idASproduct_id, attribute.attribute_code, entity_value.valueASvalue_index, attribute_label.valueASsuper_attribute_labelFROMcatalog_product_super_attributeASsuper_attributeINNER JOINcatalog_product_entityASproduct_entityON product_entity.entity_id = super_attribute.product_id INNER JOINcatalog_product_super_linkASproduct_linkON product_link.parent_id = super_attribute.product_id INNER JOINeav_attributeASattributeON attribute.attribute_id = super_attribute.attribute_id INNER JOINcatalog_product_entityASentityON entity.entity_id = product_link.product_id INNER JOINcatalog_product_entity_intASentity_valueON entity_value.attribute_id = super_attribute.attribute_id AND entity_value.store_id = 0 AND entity_value.entity_id = entity.entity_id LEFT JOINcatalog_product_super_attribute_labelASattribute_labelON super_attribute.product_super_attribute_id = attribute_label.product_super_attribute_id AND attribute_label.store_id = 0 LEFT JOINeav_attribute_optionASattribute_optionON attribute_option.option_id = entity_value.value WHERE (super_attribute.product_id IN (419541)) AND (attribute.attribute_id IN (238)) ORDER BYattribute_option.sort_order ASC;

It only returns the configurable row, and not the simples/virtuals associated to it. Magento 1 had a different way of looking up configs and their simples - they didnt need to be in the same attribute set; you could have the master configurable set that had the attribute defining the information for the configurable, and the simples configurable set that didnt contain the same attribute set.

Now it seems I have 100000's of attribute_group to catalog_product_entity_int values that do not match inside catalog_product_super_attribute/link.

To be honest, I am at a loss with how M1-to-M2 EAV attribute migration will ever be successful in my case.

[p810]

@magento add to contributors team

[m2-assistant]

Hi @p810! 👋
Thank you for joining. Please accept team invitation 👉 here 👈 and add your comment one more time.

[p810]

See the initial comment above. Just adding my name to the contributor's team in case it's necessary as I noticed I missed that before.

[victor-v-rad]

Hi @p810

Sorry for the late reply. Since the recently, reported issues in this project are supported mainly by Magento community. But your PR contribution is very welcome! Thank you! When migrate using DMT 2.3.5, Migrated_ attribute sets should not exist because this #627 functionality was implemented. Please take a look at dfd8a6e handler \Migration\Handler\AddPrefix was removed from map file. Check that your map-eav.xml does not contain this handler. Your Magento 1 attribute sets should not have Migration_ prefix after migration

[p810]

@victor-v-rad No problem, thanks for the reply! It looks like that took care of it - we must've missed that when we updated the config initially. I will close the pull request since this isn't a bug. Thanks again!

[schnere]

I have got the problem in another function:
`Notice: Undefined index: 1-Default in /var/www/html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 637

Preconditions:

• Source Magento 1.9.2.2
• Target: Magento 2.4.3-p1
• PHP 7.4
• A custom module Foo_Migration is used to map custom attributes, tables, etc.

It works with the following patch, but I did not check if it causes further issues.

--- a/src/Migration/Step/Eav/Data.php
+++ b/src/Migration/Step/Eav/Data.php
@@ -634,6 +634,9 @@ class Data implements StageInterface, RollbackInterface
         );
         foreach ($this->initialData->getAttributeSets(ModelData::TYPE_DEST) as $attributeSetId => $record) {
             $entityTypeId = $this->mapEntityTypeIdsDestOldNew[$record['entity_type_id']];
+            if (!isset($this->newAttributeSets[$entityTypeId . '-' . $record['attribute_set_name']])) {
+                continue;
+            }
             $newAttributeSet = $this->newAttributeSets[$entityTypeId . '-' . $record['attribute_set_name']];
             $this->mapAttributeSetIdsDestOldNew[$attributeSetId] = $newAttributeSet['attribute_set_id'];
             $this->defaultAttributeSetIds[$newAttributeSet['entity_type_id']] = $newAttributeSet['attribute_set_id'];