encoding/csv: "LazyQuotes" not lazy enough, need an "IgnoreQuotes" mode for highly messy csv input

[gopherbot]

by philipp.schumann:

---What steps will reproduce the problem?---

1. Download and extract http://download.geonames.org/export/dump/allCountries.zip --
contains a single tab-separated file about 940MB

2. Set up a TSV reader in Go:

        tsv = csv.NewReader(txtFile)
        tsv.Comma = '\t'
        tsv.Comment = '#'
        tsv.LazyQuotes = true
        tsv.TrailingComma = true // retain rather than remove empty slots
        tsv.TrimLeadingSpace = false // retain rather than remove empty slots

3. Iterate through the records returned by tsv.Read() (after each read, set
tsv.FieldsPerRecord = 0) until the file's line 2293755 which begins like this:

3376027 ”S” Falls   "S" Falls     4.533......

---What is the expected result?---

With LazyQuotes set, the reader should return a string array containing the
tab-separated items of only this line.

---What do you see instead?---

The reader packs all fields of the current line, starting from field 2 (if counting
0-based), PLUS all consecutive lines until line 3043730 (record 6489131 B&B "a
Casa di Griffi"    B&B "a Casa di Griffi"    ... into a single 91MB big
string value.

6g, weekly.2012-02-22 under openSuSE 12.1 64bit.

NOTES: guessing this is due to quote character mismatches of some sort or another. So
this behaviour might very well be due to non-sanitized input data. However, such is the
nature of 99.9% of real-world CSV files out there. If I have to run custom code to scan
and sanitize this 950MB file myself prior to feeding it to encoding/csv, I can just
parse manually in the first place. Ideally, the csv package would offer an
"IgnoreQuotes" mode -- for use-cases where I *know* that there are no
multi-line records and where I *know* all newlines and commas (or tabs in this case)
absolutely *need* to take the strictest precedence over any quotes that may or may not
be appearing in the data and should ideally be taken as they are since nobody ever
bothered to properly escape those quotes that are 100% part of the data, not record or
field delimiters...

[robpike]

Comment 1:

Labels changed: added priority-later, removed priority-triage.

Status changed to Accepted.

[rsc]

Comment 2:

[The time for maybe has passed.]

[rsc]

Comment 3:

Labels changed: added go1.3maybe.

[rsc]

Comment 4:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 5:

Labels changed: added repo-main.

[gopherbot]

Comment 6:

CL https://golang.org/cl/13659043 references this issue.

[adg]

Comment 7:

We need to make a policy decision about how liberal we want the encoding/csv package to
be.

[gopherbot]

Comment 8 by Tom.Maiaroto:

In the meantime, I found this to be helpful: https://code.google.com/p/gocsv/  ... It's
tolerant of the quotes. You also don't need to specify the separator and number of
fields. I tested with Geonames cities data set which is tab separated.

[nikharris0]

I'm for addressing this issue as well, as I've run into it a couple of times now and have put messy workarounds in to address it.

One thought is that we could allow setting the quote rune (just like the Comma rune), whereas zero would be no quote rune, i.e. ignored and considered value data.

[gopherbot]

CL https://golang.org/cl/23281 mentions this issue.

[gopherbot]

CL https://golang.org/cl/23401 mentions this issue.

[marceloboeira]

any updates for this?

[ianlancetaylor]

Since this issue was filed, we've decided that the standard library's encoding/csv package is going to focus only on RFC 4180, while retaining backward compatibility with what is already there. We aren't going to add new features to handle formats not described in RFC 4180. Instead, we encourage people to make their own copy of the library and modify it for their needs.

I'm sorry this doesn't help you, but the reasons for this decision are 1) there are many many many variants of CSV files, and writing one package that can handle all of them will give us a package so complex that it will be hard to use and impossible to test; 2) the code that reads a CSV file is less than 400 lines of Go code anyhow, and it's easy to tweak for whatever odd formats one encounters.

I'm going to close this old issue, since we aren't going to be implementing it.