Skip to content

Commit 33ea0f4

Browse files
rsdoielrsdoiel
committed
Quick Save
1 parent 24bb860 commit 33ea0f4

15 files changed

+408
-108
lines changed

‎README.md

Lines changed: 22 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -8,20 +8,20 @@ Posix shell scripting (e.g. writing scripts that run under Bash). The tools are
88

99
Command line utilities for simplifying work with CSV, JSON, Excel Workbooks and plain text files or content.
1010

11-
+ [csv2json](docs/csv2json.html) - a tool to take a CSV file and convert it into a JSON array or a list of JSON blobs one per line
12-
+ [csv2mdtable](docs/csv2mdtable.html) - a tool to render CSV as a Github Flavored Markdown table
13-
+ [csv2xlsx](docs/csv2xlsx.html) - a tool to take a CSV file and add it as a sheet to a Excel Workbook
14-
+ [csvcleaner](docs/csvcleaner.html) - normalize a CSV file by column and row including trimming spaces and removing comments
15-
+ [csvcols](docs/csvcols.html) - a tool for formatting command line arguments into CSV row of columns or filtering CSV rows for specific columns
16-
+ [csvfind](docs/csvfind.html) - a tool for filtering a CSV file rows by column
17-
+ [csvjoin](docs/csvjoin.html) - a tool to join two CSV files on common values in designated columns, writes combined CSV rows
18-
+ [csvrows](docs/csvrows.html) - a tool for formatting command line arguments into CSV columns of rows or filtering CSV columns for specific rows
19-
+ [jsoncols](docs/jsoncols.html) - a tool for exploring and extracting JSON values into columns
20-
+ [jsonjoin](docs/jsonjoin.html) - a tool for joining JSON object documents
21-
+ [jsonmunge](docs/jsonmunge.html) - a tool to transform JSON documents into something else
22-
+ [jsonrange](docs/jsonrange.html) - a tool for iterating over JSON objects and arrays (return keys or values)
23-
+ [xlsx2csv](docs/xlsx2csv.html) - a tool for converting Excel Workbooks sheets to CSV files
24-
+ [xlsx2json](docs/xlsx2json.html) - a tool for converting Excel Workbooks to JSON files
11+
+ [csv2json](docs/csv2json/) - a tool to take a CSV file and convert it into a JSON array or a list of JSON blobs one per line
12+
+ [csv2mdtable](docs/csv2mdtable/) - a tool to render CSV as a Github Flavored Markdown table
13+
+ [csv2xlsx](docs/csv2xlsx/) - a tool to take a CSV file and add it as a sheet to a Excel Workbook
14+
+ [csvcleaner](docs/csvcleaner/) - normalize a CSV file by column and row including trimming spaces and removing comments
15+
+ [csvcols](docs/csvcols/) - a tool for formatting command line arguments into CSV row of columns or filtering CSV rows for specific columns
16+
+ [csvfind](docs/csvfind/) - a tool for filtering a CSV file rows by column
17+
+ [csvjoin](docs/csvjoin/) - a tool to join two CSV files on common values in designated columns, writes combined CSV rows
18+
+ [csvrows](docs/csvrows/) - a tool for formatting command line arguments into CSV columns of rows or filtering CSV columns for specific rows
19+
+ [jsoncols](docs/jsoncols/) - a tool for exploring and extracting JSON values into columns
20+
+ [jsonjoin](docs/jsonjoin/) - a tool for joining JSON object documents
21+
+ [jsonmunge](docs/jsonmunge/) - a tool to transform JSON documents into something else
22+
+ [jsonrange](docs/jsonrange/) - a tool for iterating over JSON objects and arrays (return keys or values)
23+
+ [xlsx2csv](docs/xlsx2csv/) - a tool for converting Excel Workbooks sheets to CSV files
24+
+ [xlsx2json](docs/xlsx2json/) - a tool for converting Excel Workbooks to JSON files
2525

2626

2727
Compiled versions are provided for Linux (amd64), Mac OS X (amd64),
@@ -58,13 +58,13 @@ See [string](docs/string/) for full details
5858

5959
Various utilities for simplifying work on the command line.
6060

61-
+ [findfile](docs/findfile.html) - find files based on prefix, suffix or contained string
62-
+ [finddir](docs/finddir.html) - find directories based on prefix, suffix or contained string
63-
+ [mergepath](docs/mergepath.html) - prefix, append, clip path variables
64-
+ [range](docs/range.html) - emit a range of integers (useful for numbered loops in Bash)
65-
+ [reldate](docs/reldate.html) - display a relative date in YYYY-MM-DD format
66-
+ [timefmt](docs/timefmt.html) - format a time value based on Golang's time format language
67-
+ [urlparse](docs/urlparse.html) - split a URL into parts
61+
+ [findfile](docs/findfile/) - find files based on prefix, suffix or contained string
62+
+ [finddir](docs/finddir/) - find directories based on prefix, suffix or contained string
63+
+ [mergepath](docs/mergepath/) - prefix, append, clip path variables
64+
+ [range](docs/range/) - emit a range of integers (useful for numbered loops in Bash)
65+
+ [reldate](docs/reldate/) - display a relative date in YYYY-MM-DD format
66+
+ [timefmt](docs/timefmt/) - format a time value based on Golang's time format language
67+
+ [urlparse](docs/urlparse/) - split a URL into parts
6868

6969
Compiled versions are provided for Linux (amd64), Mac OS X (amd64),
7070
Windows 10 (amd64) and Raspbian (ARM7). See https://github.com/caltechlibrary/datatools/releases.
@@ -74,7 +74,7 @@ Use the utilities try "-help" option for a full list of options.
7474

7575
## Installation
7676

77-
See [INSTALL.md](install.html) for details for installing pre-compiled versions of the programs.
77+
See [INSTALL.md](install/) for details for installing pre-compiled versions of the programs.
7878

7979
_datatools_ are go get-able. If you have go v1.8 (or newer) you can install with the command below.
8080

‎cmds/string/string.go

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,26 @@ string is a command line tool for transforming strings in common ways.
4545
`
4646

4747
examples = `
48+
Convert text to upper case
49+
50+
string toupper "one"
51+
52+
Convert text to lower case
53+
54+
string tolower "ONE"
55+
56+
Captialize an English phrase
57+
58+
string englishtitle "one more thing to know"
59+
60+
Split a space newline delimited list of words into a JSON array
61+
62+
string -i wordlist.txt split "\n"
63+
64+
Join a JSON array of strings into a newline delimited list
65+
66+
string join '\n' '["one","two","three","four","five"]'
67+
4868
`
4969

5070
// Standard Options
@@ -60,6 +80,8 @@ string is a command line tool for transforming strings in common ways.
6080
eol string
6181

6282
// App Options
83+
delimiter string
84+
outputDelimiter string
6385
)
6486

6587
//
@@ -161,7 +183,7 @@ func doSplit(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
161183
fmt.Fprintln(eout, "first parameter is the delimiting string")
162184
return 1
163185
}
164-
delimiter := args[0]
186+
delimiter := datatools.NormalizeDelimiter(args[0])
165187
args = args[1:]
166188
// Handle the case where out input is piped in or read from a file.
167189
if inputFName != "" {
@@ -185,7 +207,7 @@ func doSplitN(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
185207
fmt.Fprintln(eout, "first parameter is the delimiting string, second is the count")
186208
return 1
187209
}
188-
delimiter := args[0]
210+
delimiter := datatools.NormalizeDelimiter(args[0])
189211
// Now convert to cnt an integer
190212
cnt, err := strconv.Atoi(args[1])
191213
if err != nil {
@@ -216,7 +238,7 @@ func doJoin(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
216238
fmt.Fprintln(eout, "first parameter is the delimiter to join with")
217239
return 1
218240
}
219-
delimiter := args[0]
241+
delimiter := datatools.NormalizeDelimiter(args[0])
220242
args = args[1:]
221243

222244
// Handle the case where out input is piped in or read from a file.
@@ -388,6 +410,20 @@ func doTrimRight(in io.Reader, out io.Writer, eout io.Writer, args []string) int
388410
return 0
389411
}
390412

413+
func doTrimSpace(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
414+
// Handle content coming from file
415+
if inputFName != "" {
416+
src, err := ioutil.ReadAll(in)
417+
exitOnError(eout, err, quiet)
418+
args = append(args, string(src))
419+
}
420+
// Handle content common from args
421+
for _, arg := range args {
422+
fmt.Fprintf(out, "%s%s", strings.TrimSpace(arg), eol)
423+
}
424+
return 0
425+
}
426+
391427
func doContains(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
392428
if len(args) < 1 {
393429
fmt.Fprintf(eout, "first parameter is the target string\n")
@@ -585,6 +621,8 @@ func main() {
585621
app.BoolVar(&generateMarkdownDocs, "generate-markdown-docs", false, "output documentation in Markdown")
586622

587623
// App Options
624+
app.StringVar(&delimiter, "d,delimiter", "", "set the delimiter")
625+
app.StringVar(&outputDelimiter, "do,output-delimiter", "", "set the output delimiter")
588626

589627
// Add verbs and functions
590628
app.AddAction("toupper", doToUpper, "to upper case: [STRING]")
@@ -601,6 +639,7 @@ func main() {
601639
app.AddAction("trim", doTrim, "trim (beginning and end), CUTSET [STRING]")
602640
app.AddAction("trimleft", doTrimLeft, "left trim: CUTSET [STRING]")
603641
app.AddAction("trimright", doTrimRight, "right trim: CUTSET [STRING]")
642+
app.AddAction("trimspace", doTrimSpace, "trim leading and trailing spaces: [STRING]")
604643
app.AddAction("count", doCount, "count substrings: SUBSTRING [STRING]")
605644
app.AddAction("contains", doContains, "has substrings: SUBSTRING [STRING]")
606645
app.AddAction("length", doLength, "length of string: [STRING]")
@@ -623,7 +662,7 @@ func main() {
623662
cli.ExitOnError(app.Eout, err, quiet)
624663
defer cli.CloseFile(inputFName, app.In)
625664

626-
app.Out, err = cli.Create(inputFName, os.Stdout)
665+
app.Out, err = cli.Create(outputFName, os.Stdout)
627666
cli.ExitOnError(app.Eout, err, quiet)
628667
defer cli.CloseFile(outputFName, app.Out)
629668

‎docs/csvcols/index.html

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -35,19 +35,19 @@ <h2>SYNOPSIS</h2>
3535

3636
<h2>OPTIONS</h2>
3737

38-
<pre><code> -col, -cols output specified columns (e.g. -col 1,12:14,2,4))
39-
-d, -delimiter set the input delimiter character
40-
-examples display example
41-
-generate-markdown-docs generate markdown documentation
42-
-h, -help display help
43-
-i, -input input filename
44-
-l, -license display license
45-
-o, -output output filename
46-
-od, -output-delimiter set the output delimiter character
47-
-quiet suppress error messages
48-
-skip-header-row skip the header row
49-
-uuid add a prefix row with generated UUID cell
50-
-v, -version display version
38+
<pre><code> -col, -cols output specified columns (e.g. -col 1,12:14,2,4))
39+
-d, -delimiter set the input delimiter character
40+
-examples display example
41+
-generate-markdown-docs generate markdown documentation
42+
-h, -help display help
43+
-i, -input input filename
44+
-l, -license display license
45+
-o, -output output filename
46+
-od, -output-delimiter set the output delimiter character
47+
-quiet suppress error messages
48+
-skip-header-row skip the header row
49+
-uuid add a prefix row with generated UUID cell
50+
-v, -version display version
5151
</code></pre>
5252

5353
<h2>EXAMPLES</h2>

‎docs/csvcols/index.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,19 +14,19 @@ listed on the commandline (first column is 1 not 0).
1414
## OPTIONS
1515

1616
```
17-
-col, -cols output specified columns (e.g. -col 1,12:14,2,4))
18-
-d, -delimiter set the input delimiter character
19-
-examples display example
20-
-generate-markdown-docs generate markdown documentation
21-
-h, -help display help
22-
-i, -input input filename
23-
-l, -license display license
24-
-o, -output output filename
25-
-od, -output-delimiter set the output delimiter character
26-
-quiet suppress error messages
27-
-skip-header-row skip the header row
28-
-uuid add a prefix row with generated UUID cell
29-
-v, -version display version
17+
-col, -cols output specified columns (e.g. -col 1,12:14,2,4))
18+
-d, -delimiter set the input delimiter character
19+
-examples display example
20+
-generate-markdown-docs generate markdown documentation
21+
-h, -help display help
22+
-i, -input input filename
23+
-l, -license display license
24+
-o, -output output filename
25+
-od, -output-delimiter set the output delimiter character
26+
-quiet suppress error messages
27+
-skip-header-row skip the header row
28+
-uuid add a prefix row with generated UUID cell
29+
-v, -version display version
3030
```
3131

3232

‎docs/string/index.html

Lines changed: 38 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -44,15 +44,17 @@ <h2>OPTIONS</h2>
4444

4545
<p>Options are shared between all actions and must precede the action on the command line.</p>
4646

47-
<pre><code> -e, -examples display examples
48-
-generate-markdown-docs output documentation in Markdown
49-
-h, -help display help
50-
-i, -input input file name
51-
-l, -license display license
52-
-nl, -newline if true add a trailing newline
53-
-o, -output output file name
54-
-quiet suppress error messages
55-
-v, -version display version
47+
<pre><code> -d, -delimiter set the delimiter
48+
-do, -output-delimiter set the output delimiter
49+
-e, -examples display examples
50+
-generate-markdown-docs output documentation in Markdown
51+
-h, -help display help
52+
-i, -input input file name
53+
-l, -license display license
54+
-nl, -newline if true add a trailing newline
55+
-o, -output output file name
56+
-quiet suppress error messages
57+
-v, -version display version
5658
</code></pre>
5759

5860
<h2>ACTIONS</h2>
@@ -79,12 +81,38 @@ <h2>ACTIONS</h2>
7981
trimleft left trim: CUTSET [STRING]
8082
trimprefix trims prefix: PREFIX [STRING]
8183
trimright right trim: CUTSET [STRING]
84+
trimspace trim leading and trailing spaces: [STRING]
8285
trimsuffix trim suffix: SUFFIX [STRING]
8386
</code></pre>
8487

8588
<h2>EXAMPLES</h2>
8689

87-
<p>Related: <a href="contains.html">contains</a>, <a href="count.html">count</a>, <a href="englishtitle.html">englishtitle</a>, <a href="hasprefix.html">hasprefix</a>, <a href="hassuffix.html">hassuffix</a>, <a href="join.html">join</a>, <a href="length.html">length</a>, <a href="padleft.html">padleft</a>, <a href="padright.html">padright</a>, <a href="position.html">position</a>, <a href="replace.html">replace</a>, <a href="replacen.html">replacen</a>, <a href="slice.html">slice</a>, <a href="split.html">split</a>, <a href="splitn.html">splitn</a>, <a href="tolower.html">tolower</a>, <a href="totitle.html">totitle</a>, <a href="toupper.html">toupper</a>, <a href="trim.html">trim</a>, <a href="trimleft.html">trimleft</a>, <a href="trimprefix.html">trimprefix</a>, <a href="trimright.html">trimright</a>, <a href="trimsuffix.html">trimsuffix</a></p>
90+
<p>Convert text to upper case</p>
91+
92+
<pre><code>string toupper &quot;one&quot;
93+
</code></pre>
94+
95+
<p>Convert text to lower case</p>
96+
97+
<pre><code>string tolower &quot;ONE&quot;
98+
</code></pre>
99+
100+
<p>Captialize an English phrase</p>
101+
102+
<pre><code>string englishtitle &quot;one more thing to know&quot;
103+
</code></pre>
104+
105+
<p>Split a space newline delimited list of words into a JSON array</p>
106+
107+
<pre><code>string -i wordlist.txt split &quot;\n&quot;
108+
</code></pre>
109+
110+
<p>Join a JSON array of strings into a newline delimited list</p>
111+
112+
<pre><code>string join '\n' '[&quot;one&quot;,&quot;two&quot;,&quot;three&quot;,&quot;four&quot;,&quot;five&quot;]'
113+
</code></pre>
114+
115+
<p>Related: <a href="contains.html">contains</a>, <a href="count.html">count</a>, <a href="englishtitle.html">englishtitle</a>, <a href="hasprefix.html">hasprefix</a>, <a href="hassuffix.html">hassuffix</a>, <a href="join.html">join</a>, <a href="length.html">length</a>, <a href="padleft.html">padleft</a>, <a href="padright.html">padright</a>, <a href="position.html">position</a>, <a href="replace.html">replace</a>, <a href="replacen.html">replacen</a>, <a href="slice.html">slice</a>, <a href="split.html">split</a>, <a href="splitn.html">splitn</a>, <a href="tolower.html">tolower</a>, <a href="totitle.html">totitle</a>, <a href="toupper.html">toupper</a>, <a href="trim.html">trim</a>, <a href="trimleft.html">trimleft</a>, <a href="trimprefix.html">trimprefix</a>, <a href="trimright.html">trimright</a>, <a href="trimspace.html">trimspace</a>, <a href="trimsuffix.html">trimsuffix</a></p>
88116

89117
<p>string v0.0.20-pre</p>
90118

‎docs/string/index.md

Lines changed: 33 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,15 +21,17 @@ string is a command line tool for transforming strings in common ways.
2121
Options are shared between all actions and must precede the action on the command line.
2222

2323
```
24-
-e, -examples display examples
25-
-generate-markdown-docs output documentation in Markdown
26-
-h, -help display help
27-
-i, -input input file name
28-
-l, -license display license
29-
-nl, -newline if true add a trailing newline
30-
-o, -output output file name
31-
-quiet suppress error messages
32-
-v, -version display version
24+
-d, -delimiter set the delimiter
25+
-do, -output-delimiter set the output delimiter
26+
-e, -examples display examples
27+
-generate-markdown-docs output documentation in Markdown
28+
-h, -help display help
29+
-i, -input input file name
30+
-l, -license display license
31+
-nl, -newline if true add a trailing newline
32+
-o, -output output file name
33+
-quiet suppress error messages
34+
-v, -version display version
3335
```
3436

3537

@@ -58,15 +60,36 @@ Options are shared between all actions and must precede the action on the comman
5860
trimleft left trim: CUTSET [STRING]
5961
trimprefix trims prefix: PREFIX [STRING]
6062
trimright right trim: CUTSET [STRING]
63+
trimspace trim leading and trailing spaces: [STRING]
6164
trimsuffix trim suffix: SUFFIX [STRING]
6265
```
6366

6467

6568
## EXAMPLES
6669

6770

71+
Convert text to upper case
6872

73+
string toupper "one"
6974

70-
Related: [contains](contains.html), [count](count.html), [englishtitle](englishtitle.html), [hasprefix](hasprefix.html), [hassuffix](hassuffix.html), [join](join.html), [length](length.html), [padleft](padleft.html), [padright](padright.html), [position](position.html), [replace](replace.html), [replacen](replacen.html), [slice](slice.html), [split](split.html), [splitn](splitn.html), [tolower](tolower.html), [totitle](totitle.html), [toupper](toupper.html), [trim](trim.html), [trimleft](trimleft.html), [trimprefix](trimprefix.html), [trimright](trimright.html), [trimsuffix](trimsuffix.html)
75+
Convert text to lower case
76+
77+
string tolower "ONE"
78+
79+
Captialize an English phrase
80+
81+
string englishtitle "one more thing to know"
82+
83+
Split a space newline delimited list of words into a JSON array
84+
85+
string -i wordlist.txt split "\n"
86+
87+
Join a JSON array of strings into a newline delimited list
88+
89+
string join '\n' '["one","two","three","four","five"]'
90+
91+
92+
93+
Related: [contains](contains.html), [count](count.html), [englishtitle](englishtitle.html), [hasprefix](hasprefix.html), [hassuffix](hassuffix.html), [join](join.html), [length](length.html), [padleft](padleft.html), [padright](padright.html), [position](position.html), [replace](replace.html), [replacen](replacen.html), [slice](slice.html), [split](split.html), [splitn](splitn.html), [tolower](tolower.html), [totitle](totitle.html), [toupper](toupper.html), [trim](trim.html), [trimleft](trimleft.html), [trimprefix](trimprefix.html), [trimright](trimright.html), [trimspace](trimspace.html), [trimsuffix](trimsuffix.html)
7194

7295
string v0.0.20-pre

0 commit comments

Comments
 (0)