feat - add crlf support for csv exports

[7amza79]

Add CRLF Support for CSV Exports

Description

This PR adds support for Windows-style line endings (CRLF) in CSV exports, while maintaining the default UNIX-style line endings (LF). This enhancement improves compatibility with Windows-based systems and tools that expect CRLF line endings.

Changes

• Added line_ending configuration option to CSV writers with two possible values:
  • lf (default): Uses UNIX-style line endings
  • crlf: Uses Windows-style line endings
• Updated both CsvWriter and ArrowToCsvWriter to support line ending configuration
• Added comprehensive test coverage for both line ending types
• Updated documentation to reflect the new configuration option

Technical Details

• Added CsvLineEnding type to configuration specs
• Implemented custom CSV dialect for Python's csv writer to handle CRLF
• Added version-aware implementation for PyArrow CSV writer (supports line endings from version 14.0)
• Added configuration options in both TOML and environment variables

Testing

• Added unit tests for both CsvWriter and ArrowToCsvWriter
• Verified correct line ending generation for both LF and CRLF
• Ensured no mixed line endings in output files
• Added tests for both direct data writing and Arrow table conversion

Documentation

• Updated CSV format documentation to include the new line_ending option
• Added examples for both TOML and environment variable configurations

Configuration Example

[NORMALIZE]
data_writer__line_ending="crlf"

Or via environment variable:

NORMALIZE__DATA_WRITER__LINE_ENDING=crlf

Checklist

• Added new configuration option
• Implemented support in both CSV writers
• Added comprehensive tests
• Updated documentation
• Verified backward compatibility
• Added examples for configuration

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	3409386
🔍 Latest deploy log	https://app.netlify.com/projects/dlt-hub-docs/deploys/687a24db5b204b0008ec8dc6
😎 Deploy Preview	https://deploy-preview-2783--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[rudolfix]

this looks pretty good! pls see my comments. thx for the tests!

[rudolfix]

I do not see this used. do you need it for testing? We pass csv options via config injection (you do it correctly below)

[rudolfix]

@7amza79 please also fix linter: make format and make lint

[burnash]

Hey @7amza79, would you like to continue working on this PR or you'd prefer that we take it from here? Thanks again for your effort!

[7amza79]

@burnash I 've noticed that pyArrow didn't support CRLF writing neither so I've "abandonned" working on this implementation and used pandas instead. So I think you can take it on from here if this "feature" is interesting for the product (which I think it is :D )

[rudolfix]

LGTM but maybe we could make lineterminator more flexible? see my comments. if you think it is not worth the effort you can also merge