[BUG] Slow convert processor in opensearch 2.11.0

[shikeli]

Describe the bug

We are migrating to opensearch 2.11.0 from elasticsearch 6.8.1. We faced ingestion throughput 70% drop. After investigation, we found the convert processor is very slow compared to elasticsearch 6.8.1. We removed the processor as work around. I know most of team doesn't run processor out of indexing. We are doing this first then we do indexing.

Related component

Indexing:Performance

To Reproduce

Write one plugin to call the processor.

Expected behavior

We expect the same performance or better.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[andrross]

Thanks @shikeli. Can you share more detail? i.e. what kind of documents are you sending, what does your ingest pipeline configuration look like, any profiling information you captured, etc. If you have a basic reproduction you can share that would be very helpful too.

[shikeli]

We have one pipeline added that is using convert processor

processors:
  - convert: { field: 'foo', target_field: 'bar', type: 'long', ignore_failure: true }

In our ingestion plugin, we have one class PipelineExecutor, this will execute all pipeline for each document before we do bulk indexing. Below is my profiling data.

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[gaobinlong]

In order to compare the convert processor performance, I used the benchmark tool opensearch-benchmark for OpenSearch 2.11.1 and esrally for elasticsearch 6.8.1, geonames dataset with a default ingest pipeline containing convert processor, the result shows that OpenSearch outperforms elasticsearch, see below:

OpenSearch 2.11.1:

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=geonames --target-host=http://dev-dsk-gbinlong-2b-3cdd5cb8.us-west-2.amazon.com:9200/ --include-tasks=delete-index,create-index,index-append

Min Throughput | index-append |   37554.9 | docs/s |
|                                                Mean Throughput | index-append |   38968.2 | docs/s |
|                                              Median Throughput | index-append |   38871.3 | docs/s |
|                                                 Max Throughput | index-append |   39999.8 | docs/s |
|                                        50th percentile latency | index-append |   828.356 |     ms |
|                                        90th percentile latency | index-append |   1189.44 |     ms |
|                                        99th percentile latency | index-append |   3253.41 |     ms |
|                                      99.9th percentile latency | index-append |   4270.27 |     ms |
|                                       100th percentile latency | index-append |   4816.58 |     ms |
|                                   50th percentile service time | index-append |   828.356 |     ms |
|                                   90th percentile service time | index-append |   1189.44 |     ms |
|                                   99th percentile service time | index-append |   3253.41 |     ms |
|                                 99.9th percentile service time | index-append |   4270.27 |     ms |
|                                  100th percentile service time | index-append |   4816.58 |     ms |
|                                                     error rate | index-append |         0 |      % |

elasticsearch 6.8.1:

esrally race --pipeline=benchmark-only --track=geonames --target-host=http://dev-dsk-gbinlong-2b-3cdd5cb8.us-west-2.amazon.com:9200 --challenge=append-no-conflicts-index-only

Min Throughput | index-append | 34682.4         | docs/s |
|                                                Mean Throughput | index-append | 35659.7         | docs/s |
|                                              Median Throughput | index-append | 35682.3         | docs/s |
|                                                 Max Throughput | index-append | 36271.1         | docs/s |
|                                        50th percentile latency | index-append |   974.415       |     ms |
|                                        90th percentile latency | index-append |  1505.35        |     ms |
|                                        99th percentile latency | index-append |  2116.63        |     ms |
|                                      99.9th percentile latency | index-append |  2798.38        |     ms |
|                                       100th percentile latency | index-append |  2960.17        |     ms |
|                                   50th percentile service time | index-append |   974.415       |     ms |
|                                   90th percentile service time | index-append |  1505.35        |     ms |
|                                   99th percentile service time | index-append |  2116.63        |     ms |
|                                 99.9th percentile service time | index-append |  2798.38        |     ms |
|                                  100th percentile service time | index-append |  2960.17        |     ms |

, but the result may vary in different dataset, @shikeli could you give some sample documents in your case and show the main code in your ingestion plugin to make sure we can get consistent result? Thanks!