From an internal chat:
i'm curious how we can make a good UX around non-schema-matching batches. if you're doing a large workflow, it'd be really nice to "peel off" all the non-compliant items, put them into that initial schema-inference batch, and then re-run
This should be doable:
- Down in Rust, the writer can accumulate all failed batches ... this could lead to a big memory blow-up, so maybe instead we should just return the failed batch right away?
- The user could them use add failed batch to the initial "schema inference" batch and re-run