Codec Freedom -- Specifications -- Escaping in packet mode

Codecs send frames in bursts, which must be sent as fast as possible. This means that a data flow can be seen as a background process that may be interrupted by codec frames. This, as well as line resets and escapes for values that cannot be sent through the underlying connection profile, calls for a generic escaping layer over all packet modes.

Special byte values

The connection profile over which a packet mode runs indicates the following pieces of information to the generic escaping layer:

• Esc, a byte value to use as an escape character
• Bad, a list of characters that must never be sent
• MaxCode, maximum byte value to interpret as the maximum frame length
• Fillup, a boolean value that indicates whether the line must always be supplied with data to send

All byte values are considered unsigned integers in the range [0..255].

Note that Esc must not be contained in Bad, as it would be silly to pick an escape character that may not be sent. Similarly, MaxCode is not in Bad and Reset is not in Bad.

The byte sent after Esc is vital for distinguishing the meaning of the escape:

• bytes up to and including MaxCode encode a codec frame bytes
• the value MaxCode+1 encodes RESET
• the value MaxCode+2 encodes the literal byte value matching Esc
• values over MaxCode+2 encode the values of Bad

Note that MaxCode+1 and MaxCode+2 are in fact names, defined below.

Escaping special byte values

A line is normally in "unescaped" mode, and sending Esc then turns it into a line in "escaped" mode. The next byte then returns it to unescaped mode, regardless of its value.

An unescaped line may be in "data" mode or in "codec" mode. The distinction between these modes is made through a combination of escape codes and frame length counters.

We must be cautious not to use values from Bad in escaped mode, so the following procedure is used:

• The value Zero is the lowest byte value not in Bad;
• The value MaxCode+1 is the lowest byte value over MaxCode that is not in Bad;
• The value MaxCode+2 is the lowest byte value over MaxCode+1 that is not in Bad;
• The value list Good contains all byte values over MaxCode+2 that are not in Bad, sorted on ascending value.
• The counter PipeLen is the number of bytes that may be underway in the back-and-forth pipelines between two escaping layers.

The values in Bad are always sent through an escaped-mode construction, both in data mode and in codec mode. To send a value in Bad, we enter escaped mode by sending Esc, and then send the byte value from the Good list that appears at the same index as the value of Bad that we would have liked to send.

This assigns special value to Esc, so it too must be sent in a special manner, namely by going to escaped mode with Esc and then sending the value MaxCode+2. This is done in both data mode and codec mode.

Filling the line with idle data

When the connection is in data mode but the upper data layer provides nothing to send, we may need to idle the connection. This is the case if the Fillup flag is set in the connection profile.

Filling the line is done by sending empty frames, namely Esc followed by the Zero byte value. These empty frames do not normally occur on the line, so there is no risk of confusion.

Sending and processing RESET

The upper data layer protocol may get confused, and feel a need to return to a stable state, and reset the two endpoints to achieve that. This is only done in data mode.

When the data layer locally detects a desire to reset to the stable state, it will send a RESET down. When in data mode, this leads to entering escaped mode through Esc, and sending the byte value MaxCode+1.

Once the reset signalling is acknowledged to have beensent over the connection and has been acknowledged, the fact that this has been done is relayed back up with a RESDONE notification; until this is received, the upper layer ignores all incoming data bytes. Meanwhile at the remote end, the RESET signal is sent up to return the remote end to the same stable state.

The connection profile defines a value PipeLen that indicates how many bytes may be underway in the back-and-forth pipelines. This should be dvided by two and rounded up, to come to a minimum number of Esc and Zero sequences after the Esc and MaxCode+1 to send first to delay sending up the RESDONE notifiction.

Conceptually, the data mode transmits a single byte at a time; more optimal implementations are likely to pass buffers. The RESET signal should be seen as an asynchronous signal that can bypass the buffer. It will in fact invalidate what has been sent and received up to that point.

Sending and processing ESC

When the connection profile is capable of switching to bit-insertion mode, it defines an ESC code, to be sent in escaped mode, so after Esc. Note that the two symbols represent quite different forms of escaping!

Since bit-insertion mode never deals with distortion detection, there is no distinction between these two procedures.

The actual switch of the line should be deferred until both sides have sent the ESC code. Switching between bit-insertion mode and packet mode is likely to disrupt the audio flow of the line, leading to a possible tick on the line.

Interrupting data with codec frames

Codec frames are non-empty sequences of bytes that should be sent as quickly as possible.

At any point, the flow of data may be interrupted to insert a codec frame. Basically, this is done by entering escaped mode by sending Esc, followed by the frame length encoding.

Frame lengths may coincide with a value in Bad, so there is a need to translate between frame lengths and the transmitted frame length encoding. To that end, we define:

• LenCode is the list of byte values up to and including MaxCode that are not in Bad, sorted in ascending order.

To send a certain frame length, the frame length is used as an index in this list, and the value at that position is sent as the frame length encoding.

Frames may exceed the length of this list, in which case it is broken down into chunks; all chunks have the last length available in LenCode (with frame length encoding MaxCode), and there always is a last chunk that has a shorter length, possibly 0 bytes (with frame length encoding Zero).

The first chunk starts in escaped mode, and provides a frame lenth encoding which now acts as a chunk length encoding. Then, the bytes of the chunk follow, possibly escaped in they are in Bad or if they equal Esc. After the bytes of the chunk have been sent, the line returns to escaped mode if the last-ent chunk length encoding was equal to MaxCode; that is, no Esc is now sent but the next chunk length encoding follows and the process repeats. However, if the last-sent chunk length encoding was not equal to MaxCode, then the connection will not automatically return to escaped mode, but continue in unescaped mode. When another codec frame is waiting to be sent, then it will be sent; otherwise, data mode is assumed.

Upon arrival, only non-zero frame lengths are accepted and zero-length frames (those sent as Esc followed by Zero) are ignored and neither delivered as a frame nore as data.

Note that a peephole optimisation is possible to expedite sending of codec frames in some cases; namely, when in data mode and an Esc has just been sent, then it is possible to immediately take over and insert the frame length encoding. This saves waiting for the data layer to free the line and then sending the codec's Esc code. In return for the earlier submission, the codec frame must be followed with an Esc that allows the data mode to continue as it was planning to.

A protocol state diagram

A state diagram reflecting most of the above is included below. Its start state is Data mode and it does not have a termination state.

A few remarks:

• This diagram describes the protocol, not an upward or downward translation
• RESET processing is therefore not shown explicitly
• The Esc-stealing peephole optimisation for faster switches to codec mode cannot be seen in the protocol itself

Escaped packet mode is 8-bit clean

All packet modes use this escape mode. As a result, each provides the following services to the upper layers:

• Barely any slow-down for sending codec frames, so as to pass around a burst from a codec as soon as it is produced, with the least amount of delay and jitter.

• An 8-bit clean protocol, so binary data can be carried. In practice, this means that transfer-encoding such as base64 is needed when carrying binary data over the byte-level protocol.

• A data resynchronisation facility through RESET signaling to the other side.

• Data transmission at an unpredictable rate, but one that will usually continue reasonably consistently. Average transmission rates depend on the size of the codec frames and their rate of production, and to a lesser degree, what portion of it must be escaped.

The same facilities will be provided by the bit-insertion mode protocols, so the upper layer protocols can rely on these properties without knowing if the underlying transport is a bit-insertion or packet mode.