Skip to content

[Feature Request]: enable setting max_writer_per_bundle for avroIO and other IO #29729

@patelprateek

Description

@patelprateek

What would you like to happen?

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L166
Not able to customize max_writer_per_bundle for avroIO .
When we have multiple pcollection or partitioned pcollection , its not possible to open avro sink for every partition (specially when paritions are large , for example > 32 ) . It leads to OOM

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions