feat(import,csv,psv): Add support for importing CSV and PSV files without header rows #9204

codeaucafe · 2025-05-12T02:26:38Z

Summary

Add --no-header flag to treat the first row in CSV/PSV files as data instead of column names
Add --columns option to specify column names when importing files without headers
Fix nil pointer panic when importing from stdin with --create-table

In short, this feature makes Dolt more compatible with MySQL/SQLite workflows and provides users with more flexibility when importing data.

Problem

Previously, Dolt always expected the first row of CSV/PSV files to contain column names. This differs from MySQL and SQLite which support importing files where the first row contains data. Users migrating from these systems or working with headerless data files couldn't import them without modifying their files.

Additionally, when users attempted to import data from stdin using --create-table, they would encounter a nil pointer panic instead of receiving a error message.

Solution

The implementation adds:

A new --no-header flag that treats the first row as data instead of column headers
A complementary --columns option to specify column names when headers aren't present
Proper validation to ensure correct flag combinations
Comprehensive error handling for stdin imports with clear error messages
Integration tests for both CSV and PSV files

Testing

Added integration tests for both CSV and PSV files that verify:
Importing files with --no-header and --columns options
Error cases when required options are missing
Original behavior is maintained when not using --no-header
Behavior of --columns with and without --no-header
Edge cases like stdin imports

timsehn · 2025-05-12T19:01:52Z

running workflows

nicktobey

Thank you, David! This is a very clean fix.

It seems like the newly added tests aren't currently passing though, and I left a couple of comments pertaining to documentation, tests, and code duplication. After that, this should be good to merge.

go/cmd/dolt/commands/tblcmds/import.go

go/libraries/doltcore/mvdata/file_data_loc.go

go/cmd/dolt/commands/tblcmds/import.go

integration-tests/bats/import-no-header.bats

nicktobey · 2025-05-12T23:22:02Z

Don't worry about the "Benchmark SQL Correctness" check failing: that's a known issue for PRs that come from forks.

If the other checks are green, we'll be good.

codeaucafe · 2025-05-13T06:42:56Z

hi @nicktobey I think I got it in a better position, so its "ready for review". FYI I realized trying to use no-headers and columns while streaming in the csv was panicking. So, I added error handling where // todo: capture stream data to file so we can use schema inference was. Not sure if that's right or not. I made some comments about it, please review them.

thanks for reviewing.

codeaucafe · 2025-05-13T07:07:10Z

go/libraries/doltcore/mvdata/csv_helper.go

+// CreateCSVInfo creates a CSVInfo object based on the provided options.
+// This is a helper function that extracts and processes CSV-related options
+// from the generic options interface.
+func CreateCSVInfo(opts interface{}, defaultDelim string) *csv.CSVFileInfo {


not sure how y'all feel about this file name. Wasn't sure if I should name it after helper func, or keep it a generic helper file. 🤷‍♂️

go/cmd/dolt/commands/tblcmds/import.go

codeaucafe · 2025-05-13T07:09:32Z

go/cmd/dolt/commands/tblcmds/import.go

@@ -325,6 +330,14 @@ func validateImportArgs(apr *argparser.ArgParseResults) errhand.VerboseError {
 		return errhand.BuildDError("parameters %s and %s are mutually exclusive", allTextParam, schemaParam).Build()
 	}

+	if apr.Contains(noHeaderParam) && !apr.Contains(columnsParam) {


similar handling in the issues I had testing with std streaming of csv

@nicktobey FYI, this ^ comment is somewhat confusing. I meant in that I was similarly handling configs based on table import options, not panic handling from stdin streaming of csv

go/cmd/dolt/commands/tblcmds/import.go

This feature allows users to import CSV files where the first row contains data rather than column names, matching the behavior found in MySQL and SQLite: - Add new flag --no-header to treat first row as data instead of column headers - Add new option --columns to specify column names when no header is present - Require --columns when creating new tables with --no-header - Provide helpful error messages with usage guidance - Update validation to ensure proper combination of options - Add comprehensive tests for the new functionality Closes: dolthub#7831

nicktobey · 2025-05-14T00:17:15Z

Good catch noticing the panic, and thanks for including the fix for that in your PR.

I left one comment about some confusion on my part, but otherwise I like this. I'm running the CI workflows now.

This commit fixes an issue where importing CSV data from stdin using --create-table would result in a nil pointer panic. It adds proper error handling for this case by: 1. Improving error checking in validateImportArgs to verify schema file is provided when using stdin with --create-table 2. Adding a nil check for rdSchema in newImportSqlEngineMover before attempting to access it 3. Updating the parameter description for --columns to clarify that it overrides header names when used without --no-header 4. Adding tests to verify error messages and behavior with stdin imports The improved error handling provides clear, helpful error messages instead of panicking, and documents the workaround of creating the table first and then using -u (update) mode for importing. Closes: dolthub#7831

nicktobey · 2025-05-14T23:30:50Z

This looks good!

I forgot to mention: since the newly added bats test files are testing functionality that only works when run against a local database, we need to make sure that they don't get run when testing remote functionality. (That's why the "Test Bats Unix Remote" test is failing.)

We can address that by adding entries to the SKIP_SERVER_TESTS variable defined in integration-tests/bats/helper/local-remote.bash.

After that, I think this PR is good to go!

(As a side note, I noticed that you've been rebasing your branch in response to PR feedback. While rebasing creates a cleaner commit history, it makes the PR process more confusing because it makes it harder to see what changed, and comments left on the old diff may not be visible on the new diff. Once a PR has comments on it, we try to avoid rebasing.)

codeaucafe · 2025-05-14T23:47:03Z

Thank you. I'll change that tonight.

Sorry about the rebases. Also, I'm sorry if I missed that in the Contributing markdown.

Thanks again

This update ensures that the new import-no-header-csv.bats and import-no-header-psv.bats tests are skipped when running remote tests, as they only work against a local database.

codeaucafe · 2025-05-15T06:09:04Z

Added, thanks again.

nicktobey · 2025-05-16T06:10:07Z

I've merged the PR.

Thank you for implementing this. I've had to work around this in the past, so I'm relieved that it's a built-in feature now.

codeaucafe · 2025-05-16T06:40:35Z

Awesome! Glad to hear that.

Thank you for letting me contribute and reviewing my first Dolt contribution. I really appreciate it.

cheers

coffeegoddd added the contribution label May 12, 2025

codeaucafe mentioned this pull request May 12, 2025

Allow importing CSVs without column names. #7831

Closed

codeaucafe force-pushed the codeaucafe/feat/7831/allow-import-csv-without-header-row branch from fda0260 to bd40600 Compare May 12, 2025 02:32

codeaucafe changed the title ~~Add support for importing CSV files without header rows~~ feat(import,csv,psv): Add support for importing CSV and PSV files without header rows May 12, 2025

nicktobey requested changes May 12, 2025

View reviewed changes

nicktobey reviewed May 12, 2025

View reviewed changes

integration-tests/bats/import-no-header.bats Outdated Show resolved Hide resolved

codeaucafe force-pushed the codeaucafe/feat/7831/allow-import-csv-without-header-row branch 3 times, most recently from 4c86051 to 700a067 Compare May 13, 2025 06:40

codeaucafe marked this pull request as ready for review May 13, 2025 06:42

codeaucafe force-pushed the codeaucafe/feat/7831/allow-import-csv-without-header-row branch from 700a067 to 597b19a Compare May 13, 2025 07:05

codeaucafe commented May 13, 2025

View reviewed changes

go/cmd/dolt/commands/tblcmds/import.go Outdated Show resolved Hide resolved

codeaucafe commented May 13, 2025

View reviewed changes

go/cmd/dolt/commands/tblcmds/import.go Show resolved Hide resolved

codeaucafe force-pushed the codeaucafe/feat/7831/allow-import-csv-without-header-row branch 2 times, most recently from 86b031d to ceb25a7 Compare May 13, 2025 07:59

codeaucafe force-pushed the codeaucafe/feat/7831/allow-import-csv-without-header-row branch from ceb25a7 to 4a1c1f6 Compare May 14, 2025 06:45

fix(import,csv,psv): add no-header test files to SKIP_SERVER_TESTS

23d1ea9

This update ensures that the new import-no-header-csv.bats and import-no-header-psv.bats tests are skipped when running remote tests, as they only work against a local database.

nicktobey merged commit 23eea58 into dolthub:main May 16, 2025
19 of 21 checks passed

BrewTestBot mentioned this pull request May 16, 2025

dolt 1.53.5 Homebrew/homebrew-core#223734

Merged

Uh oh!

feat(import,csv,psv): Add support for importing CSV and PSV files without header rows #9204

feat(import,csv,psv): Add support for importing CSV and PSV files without header rows #9204

Uh oh!

Conversation

codeaucafe commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Testing

Uh oh!

timsehn commented May 12, 2025

Uh oh!

nicktobey left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicktobey commented May 12, 2025

Uh oh!

codeaucafe commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codeaucafe May 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codeaucafe May 13, 2025

Choose a reason for hiding this comment

Uh oh!

codeaucafe May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nicktobey commented May 14, 2025

Uh oh!

nicktobey commented May 14, 2025

Uh oh!

codeaucafe commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codeaucafe commented May 15, 2025

Uh oh!

Uh oh!

nicktobey commented May 16, 2025

Uh oh!

codeaucafe commented May 16, 2025

Uh oh!

Uh oh!

codeaucafe commented May 12, 2025 •

edited

Loading

nicktobey left a comment •

edited

Loading

codeaucafe commented May 13, 2025 •

edited

Loading

codeaucafe May 14, 2025 •

edited

Loading

codeaucafe commented May 14, 2025 •

edited

Loading