Skip to content

Add copy_from option to the Append processor #132003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

chrisberkhout
Copy link
Contributor

Summary

A copy_from option for the Append processor, matching the option in the Set processor.

This makes it possible to refer to existing non-String values.

Discussion

The alternative is to use a Script processor, which requires Painless code to manually initialize the destination field and to avoid duplicates if necessary.

One variation on this use case is here and in the following 7 script processors.

I'd be happy to add an ignore_empty_value option as well, which would make it easier to collect values that may exist in multiple locations into one destination.


  • Have you signed the contributor license agreement? ✅
  • Have you followed the contributor guidelines? ✅
  • If submitting code, have you built your formula locally prior to submission with gradle check? ✅
  • If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed. ✅
  • If submitting code, have you checked that your submission is for an OS and architecture that we support? ✅
  • If you are submitting this code for a class then read our policy for that.

@chrisberkhout chrisberkhout added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team labels Jul 28, 2025
@elasticsearchmachine elasticsearchmachine added v9.2.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 28, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@elasticsearchmachine
Copy link
Collaborator

Hi @chrisberkhout, I've created a changelog YAML for you.

Copy link
Contributor

github-actions bot commented Jul 28, 2025

🔍 Preview links for changed docs

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to have this feature to avoid templating.

This change will also need an accompanying modification to the elasticsearch-specification to add the new parameter at https://github.com/elastic/elasticsearch-specification/blob/e585438d116b00ff34643179e6286e402c0bcaaf/specification/ingest/_types/Processors.ts#L329-L344

@chrisberkhout
Copy link
Contributor Author

chrisberkhout commented Jul 29, 2025

This change will also need an accompanying modification to the elasticsearch-specification to add the new parameter at https://github.com/elastic/elasticsearch-specification/blob/e585438d116b00ff34643179e6286e402c0bcaaf/specification/ingest/_types/Processors.ts#L329-L344

Thanks. I opened a draft PR for it: elastic/elasticsearch-specification#5056

It mentioned 6 types, the full list is 15:

Map, List, Set, byte[], double[][], double[], null, String, Integer,
Long, Float, Double, Boolean, ZonedDateTime, Date.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement external-contributor Pull request authored by a developer outside the Elasticsearch team Team:Data Management Meta label for data/management team v9.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants