Skip to content

[META] IngestService needs xContentRegistry for parametized models. #16865

Open
@brianf-aws

Description

@brianf-aws

Please describe the end goal of this project

Currently search pipelines is able to pass xContentRegistry

final SearchPipelineService searchPipelineService = new SearchPipelineService(
clusterService,
threadPool,
this.environment,
scriptService,
analysisModule.getAnalysisRegistry(),
xContentRegistry,
namedWriteableRegistry,
pluginsService.filterPlugins(SearchPipelinePlugin.class),
client

This allows the search response/request processors to handle customized parametrized local models.

However Ingest processors have no way of having this field meaning you can not ingest documents using a model that requires parametrized arguments such as local asymmetric embedding model . The fix would be to have a contentRegistry within the ingestService class

final IngestService ingestService = new IngestService(
clusterService,
threadPool,
this.environment,
scriptService,
analysisModule.getAnalysisRegistry(),
pluginsService.filterPlugins(IngestPlugin.class),
client,
indicesService
);

as well as configuring the Ingest processor Parameters class to have a xcontentRegistry just how the search response/request processors have the field.

Supporting References

Taken from my comment

opensearch-project/ml-commons#3276 (comment)

Action Items in this change

opensearch-project/ml-commons#3276 (comment)

Issues

Related component

Plugins

Metadata

Metadata

Assignees

No one assigned

    Labels

    MetaMeta issue, not directly linked to a PRingest-pipeline

    Type

    No type

    Projects

    Status

    New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions