Skip to content

Suggest Prefix encoding instead of a version number #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vancem wants to merge 2 commits into from
Closed

Suggest Prefix encoding instead of a version number #15

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d760c7
Suggest Prefix encoding instead of a version number
vancem Sep 20, 2017
793c401
Move the hierarchical (structured) IDs to their own document.
vancem Oct 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 114 additions & 44 deletions HTTP_HEADER_FORMAT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,49 +10,48 @@ balancers, etc.)

## Header name

`Trace-Context`
`Trace-Context` is the name of the [Http header field](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields) being defined here.

## Field value

`base16(<version>)-<version_format>`
The format of the value a list of data values, each value is identified by a
unique prefix character the defines the meaning of the value as well as its
format.

The value will be US-ASCII encoded (which is UTF-8 compliant). Character `-` is
used as a delimiter between fields.
values of a particular kind need not necessarily be present, but they must be
in the order defined below (e.g. if the Trace-id is present it must be first)

### Version
The prefix values currently defined are

Is a 1-byte representing a 8-bit unsigned integer. Version 255 reserved.
### Prefix '*' The Trace-id

### Version = 0
The '*' character indicates that the next 32 characters are hexadecimal digits (lower case)
that represent a 16 byte identifier. This identifier is meant to uniquely identify
new 'trace'. The intent is that all requests that are caused directly or
Copy link
Member

@yurishkuro yurishkuro Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if traceID is always the first element in the header string, what's the role of the prefix? If it specifically indicates "hexadecimal string", then the current Spec doesn't allow anything else anyway (#16).

Copy link
Author

@vancem vancem Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct that we may wish to simply drop the '*' prefix (I suggested that as an option in the original submission comment). The value it does have is that it allows us to have Trace-Contexts that encode a Trace-id (or differently (e.g. Base64), some other string etc).

It is a worthy point of discussion.

indirection from this initial request will share this Trace-id. It is expected that
this identifier will be generated by selecting a 16byte random number.

#### Format

`base16(<trace-id>)-base16(<span-id>)-base16(<trace-options>)`

All fields are required. Character `-` is used as a delimiter between fields.

#### Trace-id

Is the ID of the whole trace forest. It is represented as a 16-bytes array,
e.g., `4bf92f3577b34da6a3ce929d0e0e4736`. All bytes 0 is considered invalid.

Implementation may decide to completely ignore the trace-context if the trace-id
is invalid.
Example
```
*4bf92f3577b34da6a3ce929d0e0e4736
```

#### Span-id
### Prefix '-' The Span-id

Is the ID of the caller span (parent). It is represented as a 8-bytes array,
e.g., `00f067aa0ba902b7`. All bytes 0 is considered invalid.
The '-' prefix indicates that the next 16 characters are hexadecimal digits (lower case)
Copy link
Member

@yurishkuro yurishkuro Sep 22, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this - happens in the middle of the string, it's not a prefix

Copy link
Author

@vancem vancem Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I say prefix, I mean for the value (chunk). Thus the 'Trace-Context' is logically a list of chunks, each value (chunk) has a prefix character that define the format and meaning of that value (chunk).

that represent an 8 byte identifier. This identifier is meant to , in combination with
the Trace-id, uniquely identify the request being serviced.

Implementation may decide to completely ignore the trace-context if the span-id
is invalid.
Example
```
-00f067aa0ba902b7
```

#### Trace-options
### Prefix '~' The Trace Recommendation Flag

Controls tracing options such as sampling, trace level etc. It is a 1-byte
representing a 8-bit unsigned integer. The least significant bit provides
recommendation whether the request should be traced or not (1 recommends the
request should be traced, 0 means the caller does not make a decision to trace
The '~' prefix is not followed by any additional characters. If present, it provides
a recommendation whether the request should be traced or not. If present, the
caller recommends that the request should be traced, otherwise caller does not make a decision to trace
and the decision might be deferred). The flags are recommendations given by the
caller rather than strict rules to follow for 3 reasons:

Expand All @@ -61,26 +60,97 @@ caller rather than strict rules to follow for 3 reasons:
3. Different load between caller service and callee service might force callee
to down sample.

The behavior of other bits is currently undefined.
### Prefix '.' The Hierarchical-id
Copy link

@jacpull jacpull Sep 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is minor issue. What is unique about the Hierarchical-id is that it is a vector of logical clocks, i.e. unlike the other identifiers, its values have a sort. So consider renaming this to be just Vector or something similar. Yes, Hierarchical-Ids also work, but the structure generated by Span-Ids is also hierarchical, and does not quite reflect its uniqueness.

Copy link
Author

@vancem vancem Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am OK with that. We should do as a follow-on PR.


#### Examples of HTTP headers
The '.' character indicates that a variable number of characters follow and represent
a hierarchical (path based) id. It has the form of a list of hexadecimal numbers
(lower case) separated by '.' characters. The list is optionally terminated by
a '!' character. The id ends if either the string ends or a character that
is not a hexadecimal character or a '.' or '!' is encountered.

*Valid sampled Trace-Context:*
Examples

```
Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
base16(<Version>) = 00
base16(<TraceId>) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(<SpanId>) = 00f067aa0ba902b7
base16(<TraceOptions>) = 01 // sampled
.1
.3.b.34.ca25.56
.334.ba34.93a4.1.1!
```
The hierarchical ID is meant to represent series of causal requests within the scope
of a given Trace-id. Take has a whole, like the Span-ID, combined with the Trace-id
it uniquely identifies the request. However in addition the components also represent
other requests with that same trace that are causality connects. For example the Trace-Context
value of
```
*4bf92f3577b34da6a3ce929d0e0e4736.3.b.34
```
Represents a request with ID
```
*4bf92f3577b34da6a3ce929d0e0e4736.3.b.34
```
that was caused by a request with ID
```
*4bf92f3577b34da6a3ce929d0e0e4736.3.b
Copy link
Member

@yurishkuro yurishkuro Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requestID == spanID, is it not? Instead it looks like the hierarchical parts are appended to the traceID, which should be stable across all hops

Copy link
Author

@vancem vancem Sep 22, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what your question is. However in its simplest use you would only have a Trace-ID value and a hierarchical-ID value in in a system that employs hierarchical IDs.

What may be causing confusion here is that we are not trying to describe a single system. but a framework by which DIFFERENT systems can coexists to the extent possible. Thus some systems might use Trace-id Span-id pairs to represent a request ID and other systems will use a Trace-id - hierarchical-id to do so.

It is still an open question how much interoperability is feasible in a system compost of both types of logging conventions, but a assumption of this specification is that making it possible to share the concept of TraceId, is huge step forward and is likely to be required for any interesting interoperability.

Instead it looks like the hierarchical parts are appended to the traceID, which should be stable across all hops

Yes the hierarchical-id , combined with the trace-id and makes a particular request unique.

Copy link
Member

@yurishkuro yurishkuro Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding of Microsoft spec that mentioned hierarchical-id was that it still had a stable "trace id" (that was suggested to be passed in the "baggage" header), which was different from unique ID assigned to each request (span-id). Even if it's not correct and the hierarchical id contains the stable part, why can't that stable part be separated into a "trace-id" field understood by this spec?

Copy link
Author

@vancem vancem Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding of Microsoft spec that mentioned hierarchical-id was that it still had a stable "trace id"

No, in this spec, hierarchical-id does not include Trace-id, but it expected to be used in conjunction with it. Thus the Trace-Context value

*4bf92f3577b34da6a3ce929d0e0e4736.3.b

Has a trace ID in it 4bf92f3577b34da6a3ce929d0e0e4736 (which is table for everything caused by this), and 3.b which is the hierarchical-id (which indicates a particular request within the scope of the Trace-id.

Now of course the full request ID in Microsoft's hierarchical system would include the Trace-id, but in this spec a least 'hierarchical-id' refers only to the part that excluded the Trace-id.

Copy link
Member

@yurishkuro yurishkuro Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great, so why not *4bf92f3577b34da6a3ce929d0e0e4736-3.b, thus clearly separating the stable portion into the trace-id field? This way the receiving instrumentation doesn't need to even understand hierarchical IDs as long as it can record 3.b as correlation and is able to reuse 4bf92f3577b34da6a3ce929d0e0e4736 as the trace id.

Copy link
Author

@vancem vancem Sep 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great, so why not *4bf92f3577b34da6a3ce929d0e0e4736-3.b

Because the - prefix is already used to represent a 16 character span id. (the whole idea behind the prefixes, is to allow you to 'mix and match' parts of the value in this Header field.

```
which in turn was caused by request with
```
*4bf92f3577b34da6a3ce929d0e0e4736.3
```
Hierarchical ID are conceptually unbounded in length, but as a practical matter must be
limited, if only to avoid issues when infinite request loops form. The '!' character
is meant to indicate the ID has been truncated. The ID as a whole should still be unique
within the Trace-id, but the fine grained causality information will not be present.

### Examples of HTTP headers

*Valid not-sampled Trace-Context:*
Note that not all the allowed value prefixes are expected to be used simultaneously. __In particular it is expected that only one of the Span-id or Hierarchical-id will be used.__
Copy link

@jacpull jacpull Sep 21, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While generally this is true, we will use the span-id in conjunction with the hierarchical-id when we reset the hierarchical-id, to handle bad callers. When reset, a new span-id is generated and the hierarchical-id could be generated with subsequent extensions and increments. So would suggest removing the parts in bold or updating the text suitably.


It is however expected that the Trace-id will always be present.

Parsers are free to reject any prefixes they do not wish to support, but implementations
are encouraged to support in some way any prefixes they can do something useful with, and
to simply ignore things that they don't understand. This maximizes interoperability.

Example 1
```
Trace-context:*4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7~
```
This represents
```
base16(<Trace-id>) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(<Span-id>) = 00f067aa0ba902b7
<TraceRecommendation> = true
```

Example 2
```
Trace-context:*4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7
```
This represents
```
Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00
base16(<Version>) = 00
base16(<TraceId>) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(<SpanId>) = 00f067aa0ba902b7
base16(<TraceOptions>) = 00 // not-sampled
base16(<Trace-id>) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(<Span-id>) = 00f067aa0ba902b7
<TraceRecommendation> = false
```

Example 3
```
Trace-context:*4bf92f3577b34da6a3ce929d0e0e.4736.34.22.23~
```
This represents
```
base16(<Trace-id>) = 4bf92f3577b34da6a3ce929d0e0e
base16(<Hierarchicial-id>) = 4736.34.22.23
<TraceRecommendation> = true
```

# Versioning

There is no version number in the Trace-Context field. The intent is that new prefixes will
be defined if changes are needed. These new prefixes would be put at the end of the id
and existing parsers would encounter them and give up, ignoring the data. Thus existing
parsers will parse what they can (most critically the Trace-id) to maximize interoperability.

In the case that the parser encounters prefixes that it does not understand, or other syntax
errors, ideally the logging system would original text of the Trace-Context field, so that the
back end of the logging system has all the information possible with which to reconstruct
the causality of the requests.