|
| 1 | +<!-- Always newline after period so diffs are easier to read. --> |
| 2 | +# netstring - Robust encoding and decoding of netstrings across network connections |
| 3 | + |
| 4 | +## Introduction |
| 5 | + |
| 6 | +The `netstring` package is a go implementation of |
| 7 | +[netstring](https://cr.yp.to/proto/netstrings.txt) serialization as originally specified |
| 8 | +by D. J. Bernstein [djb](https://cr.yp.to/cv.html). |
| 9 | + |
| 10 | +`netstring` provides an Encoder and a Decoder. |
| 11 | + |
| 12 | +A netstring.Encoder writes go types as encoded netstrings to a supplied io.Writer. Encoder |
| 13 | +has helper functions to encode basic go types such as bool, ints, floats, strings and |
| 14 | +bytes to netstrings. Structs, maps and other complex data structures are not supported. A |
| 15 | +netstring.Decoder accepts a byte-stream via its io.Reader and presents successfully parsed |
| 16 | +netstrings via the Decode.Decode() and Decoder.DecodeKeyed() functions. |
| 17 | + |
| 18 | +Alternatively applications can use the message level Encoder.Marshal() and |
| 19 | +Decoder.Unmarshal() functions to encode and decode a simple struct containing "keyed" |
| 20 | +netstrings with an end-of-message sentinel. |
| 21 | + |
| 22 | +The overall goal of this package is to make it easy to attach netstring Encoders and |
| 23 | +Decoders to network connections or other reliable transports so that applications can |
| 24 | +exchange messages with either the lower level Encode*()/Decode*() functions or the higher |
| 25 | +level Marshal() and Unmarshal() functions. |
| 26 | + |
| 27 | +### Project Status |
| 28 | + |
| 29 | +[](https://github.com/markdingo/netstring/actions/workflows/go.yml) |
| 30 | +[](https://codecov.io/gh/markdingo/netstring) |
| 31 | +[](https://github.com/markdingo/netstring/actions/workflows/codeql-analysis.yml) |
| 32 | +[](https://goreportcard.com/report/github.com/markdingo/netstring) |
| 33 | +[](https://pkg.go.dev/github.com/markdingo/netstring) |
| 34 | + |
| 35 | +## Background |
| 36 | + |
| 37 | +A netstring is a simple serialization technique where each value is expressed in the form |
| 38 | +[length] ":" [value] "," where [value] is the payload, [length] is the length of [value] |
| 39 | +in decimal bytes and the colon and comma are leading and trailing delimiters respectively. |
| 40 | +An example of the value "The Hitchhiker's Guide to the Galaxy - D.A." encoded as a |
| 41 | +netstring is: |
| 42 | + |
| 43 | + "42:The Hitchhiker's Guide to the Galaxy - DA.," |
| 44 | + |
| 45 | +Storing binary values in netstrings, while possible, is not recommended for obvious |
| 46 | +reasons of incompatible CPU architectures. |
| 47 | +Best practise is to convert all binary values to strings prior to encoding. |
| 48 | +To assist in this best practice, helper functions are available to encode basic go-types |
| 49 | +such as ints and floats to netstrings. |
| 50 | +E.g. the function Encoder.EncodeInt() converts int(2^16) to the netstring: |
| 51 | + |
| 52 | + "5:65536," |
| 53 | + |
| 54 | +## "Keyed" netstrings |
| 55 | + |
| 56 | +In addition to standard netstrings, this package supports "keyed" netstrings which are |
| 57 | +nothing more than netstrings where the first byte signifies a "type" which describes the |
| 58 | +value in some useful way to the application. |
| 59 | + |
| 60 | +The benefit of "keyed" netstrings is that they create a simple self-describing typing system |
| 61 | +which allows applications to encode multiple netstrings in a message without having to |
| 62 | +rely on positional order or nested netstrings to convey semantics or encapsulate a message. |
| 63 | + |
| 64 | +To demonstrate the benefits of "keyed" netstrings, say you want to encode Name, Age and |
| 65 | +Country as netstrings to be transmitted to a remote service? With standard netstrings |
| 66 | +you'd have to agree on the positional order for each value, say netstring #0 for Name, #1 |
| 67 | +for Age and so on. |
| 68 | +Here is what that series of netstrings looks like: |
| 69 | + |
| 70 | + "4:Name,3:Age,7:Country," |
| 71 | + |
| 72 | +If any of these values are optional, you'd still have to provide an empty netstring to |
| 73 | +ensure positional order is maintained which creates the dilemma as to whether an empty |
| 74 | +netstring is a value or a place-holder. |
| 75 | + |
| 76 | +With "keyed" netstrings the values can be presented in any order and optional values are |
| 77 | +simply omitted. |
| 78 | +In the above example if we assign the "key" of 'f' to First Name, 'l' to Last Name, 'a' to |
| 79 | +Age and 'h' to Height the series of "keyed" netstrings looks like: |
| 80 | + |
| 81 | + "11:fFirst Name,10:lLast Name,4:aAge,7:hHeight," |
| 82 | + |
| 83 | +But if Age is optional the series of netstrings simply becomes: |
| 84 | + |
| 85 | + "7:hHeight,11:fFirst Name,10:lLast Name," |
| 86 | + |
| 87 | +Note the change of order as well as the missing 'a' netstring? |
| 88 | + |
| 89 | +## Installation |
| 90 | + |
| 91 | +`netstring` is available using the standard `go get` command or it should automatically be |
| 92 | +installed by `go tidy` or `go build` if your project uses go modules. |
| 93 | + |
| 94 | +Install with: |
| 95 | + |
| 96 | +``` |
| 97 | + go get github.com/markdingo/netstring |
| 98 | +``` |
| 99 | + |
| 100 | +and run tests with: |
| 101 | + |
| 102 | +``` |
| 103 | + go test github.com/markdingo/netstring |
| 104 | +``` |
| 105 | + |
| 106 | +and of course once installed, package documentation is available via: |
| 107 | + |
| 108 | +``` |
| 109 | + go doc github.com/markdingo/netstring |
| 110 | +``` |
| 111 | + |
| 112 | +## Usage |
| 113 | + |
| 114 | +``` go |
| 115 | +import "github.com/markdingo/netstring" |
| 116 | +``` |
| 117 | + |
| 118 | +To create a message of netstrings, call NewEncoder() then call the various Encode*() |
| 119 | +functions to encode the basic go types. This code fragment encodes a message into a |
| 120 | +bytes.Buffer. |
| 121 | + |
| 122 | +``` |
| 123 | + var buf bytes.Buffer |
| 124 | + enc := netstring.NewEncoder(&buf) |
| 125 | + enc.EncodeInt('a', 21) // Age |
| 126 | + enc.EncodeString('C', "Iceland") // Country |
| 127 | + enc.EncodeString('n', "Bjorn") // Name |
| 128 | + enc.EncodeBytes('z') // End-of-message sentinel |
| 129 | + fmt.Println(buf.String()) // "3:a21,8:CIceland,6:nBjorn,1:z," |
| 130 | +``` |
| 131 | + |
| 132 | +And this fragment decodes the same message. |
| 133 | + |
| 134 | +``` |
| 135 | + dec := netstring.NewDecoder(&buf) |
| 136 | + k, v, e := dec.DecodeKeyed() // k=a, v=21 |
| 137 | + k, v, e = dec.DecodeKeyed() // k=C, v=Iceland |
| 138 | + k, v, e = dec.DecodeKeyed() // k=n, v=Bjorn |
| 139 | + k, v, e = dec.DecodeKeyed() // k=z End-Of-Message |
| 140 | +``` |
| 141 | + |
| 142 | +The message can more easily be encoded with Marshal() as this fragment shows: |
| 143 | + |
| 144 | +``` |
| 145 | + type record struct { |
| 146 | + Age int `netstring:"a"` |
| 147 | + Country string `netstring:"c"` |
| 148 | + Name string `netstring:"n"` |
| 149 | + } |
| 150 | +
|
| 151 | +var buf bytes.Buffer |
| 152 | + enc := netstring.NewEncoder(&buf) |
| 153 | + out := &record{21, "Iceland", "Bjorn"} |
| 154 | + enc.Marshal('z', out) |
| 155 | +``` |
| 156 | + |
| 157 | +and more easily decoded with Unmarshal() as this fragment shows: |
| 158 | + |
| 159 | +``` |
| 160 | + dec := netstring.NewDecoder(&buf) |
| 161 | + in := &record{} |
| 162 | + dec.Unmarshal('z', in) |
| 163 | +``` |
| 164 | + |
| 165 | +Full working programs can be found in the _examples directory. |
| 166 | + |
| 167 | +### Community |
| 168 | + |
| 169 | +If you have any problems using `netstring` or suggestions on how it can do a better job, |
| 170 | +don't hesitate to create an [issue](https://github.com/markdingo/netstring/issues) on the |
| 171 | +project home page. This package can only improve with your feedback. |
| 172 | + |
| 173 | +### Copyright and License |
| 174 | + |
| 175 | +`netstring` is Copyright :copyright: 2023 Mark Delany and is licensed under the BSD |
| 176 | +2-Clause "Simplified" License. |
0 commit comments