CCNx Content Object Chunking

3.1. Protocol Specification

Chunking, as used in this specification, means serializing user data into one or more chunks, each encapsulated in a CCNx Content Object. A chunk is a contiguous byte range within the user data. One segment in the Name of each Content Object represents the chunk number. A field in the Content Object that a chunk is the last chunk of the object Chunks are denoted by a serial counter, beginning at 0 and incrementing by 1 for each contiguous chunk. Any Interest issued with a chunk number higher than the last chunk number MUST be dropped with no response.¶

This document introduces the new CCNx Name Segment TLV, ChunkNumber, for conveying chunk numbers. The ChunkNumber is the serial order of the chunks. It MUST begin at 0 and MUST be incremented by 1. The value of the ChunkNumber TLV is a network byte order compact encoding of the chunk number.¶

This document also introduces the new CCNx Message Type TLV, EndChunkNumber. The EndChunkNumber indicates the last chunk number of the chunked user data. It MUST be included in the Content Object which is the last chunk of user data, where the ChunkNumber TLV value and the EndChunkNumber TLV value MUST be the same. The value of the EndChunkNumber value is the same format as the ChunkNumber name segment. For example, if 3000 bytes of user data is split with a 1200 byte block size, there will be 3 chunks: 0, 1, and 2. The EndChunkNumber is 2.¶

The EndChunkNumber field MAY be present in any of the object’s chunks, such as to assist in pipelining Interests. It MAY be placed repeatedly across multiple chunks to maximize the fetching flexibility of a consumer.¶

The value of the EndChunkNumber MAY be increased by subsequent EndChunkNumber fields. However, it SHOULD NOT be decreased. If a publisher wishes to close terminate creation of chunks before reaching the last chunk, it should publish empty Content Objects to fill out to the maximum EndChunkNumber ever published. These padding chunks MUST contain the real last chunk number in the EndChunkNumber with an empty Payload TLV. This mechanism for early termination is to accommodate circumstances where a data creation by the producer ended before expected.¶

It may occur that a publisher never includes an EndChunkNumber, and thus has never published the last chunk. This may happen due to an error or network conditions that do not allow finding the last chunk. If a consumer times out trying to retrieve chunks, it SHOULD report an error to the user and terminate. This is similar to a TCP client not hearing a FIN.¶

If the user data fits within one Content Object and the publisher uses the Chunking protocol, the publisher names the content with "Chunk=0" and includes "EndChunkNumber=0" (assuming the Payload is in the first chunk).¶

In addition, this document introduces the new CCNx Message Type TLV, ChunkSize, for specifying a constant payload size. Chunking MAY use a fixed block size, where only the last chunk MAY use a smaller block size. This is required to allow a reader to seek to a specific byte offset once it knows the block size. If the first (Chunk 0) object has the field ChunkSize, then the series of chunked objects uses a fixed payload size. ChunkSize MUST only appear in Chunk 0, if it is used. The first chunk of user data (a chunk with a Payload TLV) may not be chunk 0.¶

Any mismatch between the value of the length field of the payload TLV and the the ChunkSize TLV is considered an error of chunk creation by the producer. Such mismatched Data messages returned from a matching Interest MUST be discarded by the consumer.¶

Because of the possible use of a fixed block size, the inclusion of certain cryptographic fields in the same Content Objects as user data would throw off the ability to seek. Therefore, it is RECOMMENDED that all required cryptographic data, such as public keys or key name links, be included in the leading chunks before the first byte of user data. User data SHOULD then run continuously. If the publisher included ChunkSize in Chunk 0, then all Payload sizes must be of ChunkSize except the last, which may be smaller.¶

Data produced using the chunking protocol may also be represented via a Manifest (e.g., FLIC [I-D.irtf-icnrg-flic]). An advantage of using a Manifest is that all cryptographic data is in the manifest, which then links to each chunk via a full hash name. This means that the chunked user data can devote all available Content Object bytes after the name to the user data.¶

To summarize:¶

1. Every chunk MUST have a ChunkNumber name segment, beginning at 0 and incrementing by 1.¶
2. The leading chunks MAY have missing or empty Payload TLVs and convey only cryptographic or other information.¶
3. The last chunk MUST have an EndChunkNumber TLV and the value MUST be equal to the ChunkNumber TLV value.¶
4. Content Objects before the last chunk MAY have an EndChunkNumber TLV with the expected last chunk number. These hints MAY be updated in subsequent Content Objects but SHOULD NOT decrease.¶
5. If the last chunk has a ChunkNumber less than a previously published EndChunkNumber, the publisher SHOULD pad out the chunks with empty Content Objects that have the real last chunk number in the EndChunkNumber.¶
6. An optional ChunkSize TLV MAY be in, and only in, Chunk 0. If it is present, it indicates that all Payloads are the same number of bytes, except the last chunk which may be smaller.¶

3.2. Cryptographic Material

Chunk 0 MAY include the public key or key name link used to verify the chunked data. It is RECOMMENDED to use the same key for the whole set of chunked data. If a publisher uses multiple keys, then the public key or key name link for all keys SHOULD be in the leading chunks before any user data. Each subsequent chunk only needs to include the KeyId and signature.¶

As noted above, using a Manifest eliminates the need for cryptographic material in the user-data Content Objects.¶

3.3. Examples

Here are some examples of chunked Names using the Labeled Content Identifier URI scheme in human readable form (ccnx:).¶

In this example, the content producer publishes a JPG that takes 4 Chunks. The EndChunkNumber is missing in the first content object (Chunk 0), but is known and included when Chunk 1 is published. It is omitted in Chunk 2, then appears in Chunk 3, where it is mandatory.¶

ccnx:/Name=example.com/Name=picture.jpg/Chunk=0  --
ccnx:/Name=example.com/Name=picture.jpg/Chunk=1  EndChunkNumber=3
ccnx:/Name=example.com/Name=picture.jpg/Chunk=2  --
ccnx:/Name=example.com/Name=picture.jpg/Chunk=3  EndChunkNumber=3

In this example, the publisher is publishing an audio stream that ends before expected so the publisher fills empty Content Objects out to the maximum ChunkNumber, stating the correct EndChunkNumber. Chunks 4, 5, and 6 do not contain any new user data.¶

ccnx:/Name=example.com/Name=talk.wav/Chunk=0  --
ccnx:/Name=example.com/Name=talk.wav/Chunk=1  EndChunkNumber=6
ccnx:/Name=example.com/Name=talk.wav/Chunk=2  --
ccnx:/Name=example.com/Name=talk.wav/Chunk=3  EndChunkNumber=3
ccnx:/Name=example.com/Name=talk.wav/Chunk=4  EndChunkNumber=3
ccnx:/Name=example.com/Name=talk.wav/Chunk=5  EndChunkNumber=3
ccnx:/Name=example.com/Name=talk.wav/Chunk=6  EndChunkNumber=3

4.1. ChunkNumber TLV

CCNx chunking defines one new CCNx Name Segment type in the IANA Registry: ChunkNumber.¶

In ccnx: URI form, it is denoted as "Chunk", like the example shown in Section 3.3.¶

                     1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+--------------+
|           T_CHUNK             |             Length           |
+---------------+---------------+---------------+--------------+
|     variable length integer   /
+---------------+---------------+

4.2. EndChunkNumber and ChunkSize TLVs

CCNx chunking defines two new CCNx Message types in the IANA Registry: EndChunkNumber and ChunkSize.¶

                     1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+--------------+
|          T_ENDCHUNK           |             Length           |
+---------------+---------------+---------------+--------------+
|     variable length integer   /
+---------------+---------------+

                     1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+--------------+
|         T_CHUNK_SIZE          |             Length           |
+---------------+---------------+---------------+--------------+
|     variable length integer   /
+---------------+---------------+

6.1. CCNx Name Segment Type Registry

This document defines one message type, T_CHUNK, whose suggested value is %x0005.¶

6.2. CCNx Message Type Registry

This document defines two message types, T_ENDCHUNK and T_CHUNK_SIZE, whose suggested values are %x0008 and %0x0009, respectively.¶