Huge memory usage when importing large quantities of products

[diwipl]

When importing huge amounts of products Magento 2 is loading contents of importexport_importdata to memory and then iterate it in memory which results in huge memory usage

Preconditions

1. Magento 2.1.7
2. CSV with 1 500 000 products

Steps to reproduce

1. Validate 1 500 000 products
2. Run import

Expected result

1. Memory usage for iterating trough import data should be constant

Actual result

1. Whole table contents is loaded to memory

[magento-engcom-team]

@diwipl, thank you for your report.
We've created internal ticket(s) MAGETWO-84592 to track progress on the issue.