
                        Description of the zisofs2 Format
                              Revision 2.0-dev

                           as of zisofs2-tools by
                   Valentín KIVACHUK BURDÁ and Thomas SCHMITT

                                1 Oct 2020


The zisofs2 format was invented by Valentín KIVACHUK BURDÁ and 
Thomas SCHMITT (as extension of zisofs by H. Peter Anvin). It compresses
data file content, marks it by a header and provides a pointer array for 
coarse random access. Within a RRIP enhanced ISO 9660 image the format 
is additionally marked by a System Use entry with signature "ZF" or "Z2".

The uncompressed size of a single zisofs2 compressed file is restricted
to 2^64 - 1 bytes. Larger files shall not be compressed. 

The format of version 1 of zisofs is supported by this specification.
Using it for files with uncompressed size smaller than 4 GiB is friendly
towards software which does not know about zisofs2.
See section **LEGACY** for a summary of version 1 of zisofs.



                                Data Types

ISO 9660:7.3.1 - little endian 4-byte words
ISO 9660:7.1.1 - unsigned single bytes
ISO 9660:7.3.3 - 8-bytes value, first in little endian, then big endian.
#uint64        - 8-bytes unsigned value in little endian

                                Supported compressors

The file header has this layout:
    @alg_id       @alg_char                Description
     1            'PZ' (50)(5A)            Zlib
     2            'XZ' (78)(7A)            XZ
     3            'L4' (6C)(34)            LZ4
     4            'ZD' (7A)(64)            Zstandard
     5            'B2' (62)(32)            Bzip2

@alg_id is a 7.1.1 value. @alg_char is 2 ASCII characters stored as 2 bytes
Values of @alg_id = 0 and @alg_char = 'pz'(70)(7A) are reserved and 
must not be used. Other compressors are allowed and may be added to this 
list in the future

                                Compressor strategy

The default strategy for a compressor is to compress each input data block 
independently. The zisofs2 spec may define in the future other strategies,
which will have a new @alg_id, @alg_char and a description in this section.


                                File Header

The file header has this layout:
Offset     Type         Identifier    Contents
--------------------------------------------------------------------------
  0        (8 bytes)    @hdr_magic    Magic num (EF 22 55 A1 BC 1B 95 A0)
  8        7.1.1        @hdr_version  File header version (0)
  9        7.1.1        @hdr_size     header_size >> 2 (6)
 10        7.1.1        @alg_id       Algorithm Type (>=1)
 11        7.1.1        @hdr_bsize    log2(block_size) (15, 16, or 17)
 12        #uint64      @size         Uncompressed file size
 20        (4 bytes)    -             Padding. Ignored

So its size is 24.

Readers shall be able to handle log2(block_size) values 15, 16 and 17
i.e. block sizes 32 kB, 64 kB, and 128 kB. Writers must not use 
other sizes.

                               Block Pointers

There are ceil(input_size / block_size) input resp. output blocks.
Each input block is of fixed size whereas the output blocks have varying
size (down to 0). For each output block there is an offset pointer giving
its byte address in the overall file content. The next block pointer in the
array tells the start of the next block which begins immediately after the
end of its predecessor. A final pointer (*eob*) gives the first invalid 
byte address and thus marks the end of the last block.

So there are ceil(input_size / block_size) + 1 block pointers.
They are stored directly after the file header, i.e. beginning at byte 24, 
as an array of values in #uint64 format (8 bytes).

Legacy format (zisofs) may be used, which is described in section *LEGACY*



                                 Data Part

The data part begins immediately after the pointer array (*eob*). In 
principle it consists of the variable length output blocks as delivered by 
different compression algorithms when fed with the fixed size input blocks.

A special case of input and output block is defined:
Zero-length blocks represent a block full of 0-bytes.
Such input blocks do not get processed by compress2() but shall be mapped 
to 0-sized output directly. Vice versa 0-sized blocks have to bypass 
uncompress() when being read.


                         ZF System Use Entry Format

The ZF entry follows the general layout of SUSP resp. RRIP.
Its fields are:

  [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").

  [2] "BP 3 - Length" shall specify as an 8-bit number the length in 
      bytes of the ZF entry recorded according to ISO 9660:7.1.1.
      This length is 16 decimal.
      Refer to **LEGACY**

  [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.
      Refer to **LEGACY**

  [4] "BP 5 to BP 6 - Algorithm"  shall be two chars to indicate the
      compression algorithm. For example, (50)(5A) ("PZ")
      (This is a copy of @alg_char). Refer to **LEGACY**

  [5] "BP 7 - Header Size Div 4" shall specify as an 8-bit number the 
      number of 4-byte words in the header part of the file data recorded 
      according to ISO 9660:7.1.1.
      (This is a copy of @hdr_size).

  [6] "BP 8 - Log2 of Block Size" shall specify as an 8-bit number the 
      binary logarithm of the compression block size recorded according to
      ISO 9660:7.1.1.
      (This is a copy of header byte 13 (@hdr_bsize), resp. header BP 14.
       The value has to be 15, 16 or 17 i.e. 32 kiB, 64 kiB, or 128 kiB.)

  [7] "BP 9 to BP 16 - Virtual Uncompressed File Size" shall contain
      as a 64-bit unsigned little endian number the uncompressed 
      file size represented by the given extent. Refer to **LEGACY**
       

  | 'Z' | 'F' | LENGTH | 2 | 'P' | 'Z' | HEADER SIZE DIV 4 | 
  | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |

Example (block size 128 kiB, uncompressed file size = 40 TB):
  { 'Z',  'F',  16,   2,  'P',  'Z',    8,   17,
   0x00, 0x80, 0xCA, 0x39, 0x61, 0x24, 0x00, 0x00 }


                         Z2 System Use Entry Format

Linux kernels which are configured by CONFIG_ZISOFS to recognize zisofs
but are not aware of zisofs2 will complain about ZF entries which announce
algorithms other than "pz". The system log will show for each zisofs2
compressed file at first stat(2) or open(2) a line like this:

  isofs: Unknown ZF compression algorithm: PZ

To avoid these complaints, it is possible to use 

  [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(32) ("Z2").

instead of "ZF" with the System Use Entry Format specified above.
Everything else shall be like in ZF format version 2, including the version
number itself:

  [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.


                                 **LEGACY**

zisofs2 supports old readers by respecting the zisofs format. This section
describes which definitions from zisofs2 must change to be compatible 
with zisofs.

- General behaviour
  The uncompressed size of a single zisofs compressed file is restricted
  to 4 GiB - 1. Larger files shall not be compressed.

- Supported algorithms
  Only algorithm Zlib with default strategy is supported.

- The file header must follow this structure:

  Offset   Type        Identifier    Contents
    0      (8 bytes)   @hdr_magic    Magic number (37 E4 53 96 C9 DB D6 07)
    8      7.3.1       @size         Uncompressed file size
   12      7.1.1       @hdr_size     header_size >> 2 (4)
   13      7.1.1       @hdr_bsize    log2(block_size) (15, 16, or 17)
   14      (2 bytes)   -             Reserved, must be zero
    
  So its size is 16.

- Block pointers
  The array must use ISO 9660:7.3.1 (4 bytes) values.

- ZF entry

Its fields are:

  [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").

  [2] "BP 3 - Length" must be 16 decimal.

  [3] "BP 4 - System Use Entry Version" must be 1.

  [4] "BP 5 to BP 6 - Algorithm" must be (70)(7A) ("pz").

  [5] "BP 7 - Header Size Div 4" - same as zisofs2.

  [6] "BP 8 - Log2 of Block Size" - same as zisofs2.

  [7] "BP 9 to BP 16 - Uncompressed Size" This field shall be recorded
      according to ISO 9660:7.3.3.
      (This number is the same as @size )

  | 'Z' | 'F' | LENGTH | 1 | 'p' | 'z' | HEADER SIZE DIV 4 | 
  | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |

Example (block size 32 kiB, uncompressed file size = 1,234,567 bytes):
  { 'Z',  'F',   16,    1,  'p',  'z',    4,   15,
   0x87, 0xD6, 0x12, 0x00, 0x00, 0x12, 0xD6, 0x87 }


References:

zisofs2-tools
  https://github.com/vk496/zisofs2-tools

zisofs-tools
  http://freshmeat.net/projects/zisofs-tools/

zlib:
  /usr/include/zlib.h

cdrtools with mkisofs
  ftp://ftp.berlios.de/pub/cdrecord/alpha

ECMA-119 aka ISO 9660
  http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf

SUSP 1.12
  ftp://ftp.ymi.com/pub/rockridge/susp112.ps

RRIP 1.12
  ftp://ftp.ymi.com/pub/rockridge/rrip112.ps

zisofs version 1
  libisofs-*/doc/zisofs_format.txt
  https://dev.lovelyhq.com/libburnia/libisofs/raw/branch/master/doc/zisofs_format.txt

---------------------------------------------------------------------------

This text is under
Copyright (c) 2009 - 2010, 2020 Thomas SCHMITT <scdbackup@gmx.net>
Copyright (c) 2020 - Valentín KIVACHUK BURDÁ <vk18496@gmail.com>
It shall reflect the effective technical specifications as implemented in
zisofs2-tools and the Linux kernel. So please contact mailing list
<bug-xorriso@gnu.org> or to the copyright holders in private, if you
want to make changes.
Only if you cannot reach the copyright holder for at least one month it is
permissible to modify and distribute this text under the license "GPLv3".

