pkg(5): image packaging system

This information is Copyright (c) 2010, Oracle and/or its affiliates.

All rights reserved.

ON-DISK FORMAT PROPOSAL

1.  Introduction

    1.1. Date of This Document:

        06/02/2010

    1.2. Name of Document Author/Supplier:

        Shawn Walker, Oracle,

          on behalf of the pkg(5) project team

    1.3. Acknowledgements:

        This document is largely based on comments from the following

        individuals to whom the author is exceedingly indebted to:

        - Danek Duvall

        - Mike Gerdts

        - Stephen Hahn

        - Krister Johansen

        - Dan Price

        - Brock Pytlik

        - Bart Smaalders

        - Peter Tribble

2. Project Summary

    2.1. Project Description:

        "...the repository can be archived up, put on a CD, memory

        stick, 2D barcode, and protected by the Black Knight, fire

        moats, komodo dragons, etc." - Danek Duvall

        pkg(5) is primarily a network-oriented binary packaging system.

        Although some of the tools it provides support filesystem-based

        operations for publication, the primary expected use for package

        operations (such as install, update, search, etc.) is between an

        intelligent client and one or more servers that provide access

        to a package repository and/or other interactive services.

        This project seeks to define and establish an on-disk format

        (and corresponding container format), for the pkg(5) system,

        with the intent that it can enable the ubiquitous, transparent

        use of package data from filesystem-based resources.

        The changes proposed by this project are evolutionary, not

        revolutionary, in nature.  In particular, this project seeks

        to refine and adopt the existing repository format used by the

        pkg(5) depot server as the on-disk format.  Supplementary to

        that, it also seeks the addition of a container format to ease

        provisioning of the on-disk format, and the unification of the

        scheme used by the client and server to store package data.

    2.2. Problem Area:

        For some deployments, network-based package data access is not

        possible or is undesirable.  Concerns often cited in this area

        include:

        - lack of access control or ability to easily integrate with

          existing access control systems,

        - inability to rely on alternative (or existing) provisioning

          arrangements (such as NFS-based file servers),

        - environmental or procedural requirements that prohibit the

          ability to or use of a network-based service,

        - characteristics of network protocols (such as HTTP, etc.) that

          artificially limit functionality or performance (as opposed to

          iSCSI or other alternatives),

        - ease of administration of filesystem-based resources, and

        - ease of transferring package data.

3. Project Technical Description:

    3.1. Details:

        This project defines an on-disk format (and corresponding con-

        tainer format) that is intended for the supplemental or complete

        provisioning of package data at all stages of the package life-

        cycle.  That is, when package data is published, stored by the

        client or server, or otherwise used during package operations.

        The on-disk format (defined in detail later in this document)

        is intended to be distributable in its raw form (a pre-defined

        structure of directories and files) or within a container format

        (such as a zip file, etc.).

        Out of necessity, the use of filesystem-based resources (such as

        those provided by the on-disk format) will sometimes limit the

        operations that can be performed to a subset of those normally

        available when interacting with a network-based repository.  For

        example, search and publisher configuration may not be possible,

        and purely interactive services such as the BUI (Browser UI)

        offered by the depot server for a repository, RSS feeds, and

        others will not be available.

        Because of the wide-ranging impact of the changes required to

        implement this functionality, it is intended that the project

        be implemented in the following sequence:

        - Client Support for filesystem-based Repository Access

        - Depot Storage, Client Transport and Publication Tool Update

        - Client Storage and Image Format Update

        - Client and Depot Support for On-Disk Archive Format

    3.2. Bug/RFE Number(s):

        As an example of the kinds of defects and RFEs intended to be

        resolved by this project, see the following selection of

        defect.opensolaris.org bug IDs:

2152 standalone package support needed (on-disk format)

166 depot doesn't set directory mode when creating directories

2086 validate that a repository is really a repository in pkg.depotd

6335 publisher repo with invalid certificate information shouldn't

    prevent querying other repos

6576 pkg install/update support for temporary publisher origins desired

6940 depot support for file:// URI desired

7213 ability to remove published packages

7273 manifests should be arranged in a hierarchy by publisher

7276 /var/pkg metadata needs reorg (looks busy)

8433 client and pull need to refer to refer to "repository" instead of

    "server"

8722 advanced repository metadata store needed

8725 versioning information for depot and repository metadata needed

9571 CachedManifest should be named FactoredManifest

9572 CachedManifest should allow consumers to specify cache location

9872 publication api should use new transport subsystem

9933 ability to control repository creation behaviour or removal of it

10244 caching dictionaries as a class variable prevents multi-image and

    repo search

11362 Image update dying when trying to talk to a disabled and offline

    publisher

11740 publishers with installed packages should not be removable

12814 publisher prefixes should be forcibly lower-cased or case

    insensitive

14802 ability to have separate read / write download caches

15320 pkgsend will traceback if unable to parse server error response

15371 repository property defaults opensolaris.org-specific

    3.3. In Scope:

        Filesystem-based data resourcing for package operations.

    3.4. Out of Scope:

        Package signing and fine-grained access control for package

        repositories.

4. On-Disk Format Technical Description:

    4.1. Overview:

        The on-disk format is intended to exist both in a raw format as

        a pre-defined structure of directories and files, and in an

        archive format which is primarily a simple container for

        the raw format.

    4.2. Raw Format:

        4.2.1. Goals:

            The goals for the raw on-disk format include:

            - unification of client and server package data storage

              for data common to both,

            - transparent usage of package data regardless of operation

              or use by client or server,

            - ease in composition and decomposition of package data

              stored within by publisher or package,

            - re-use of existing publication tools for on-disk format,

            - enablement of future publication tools to automatically

              be able to manipulate or use on-disk format, and

            - ease of provisioning.

        4.2.2. Raw Format specification:

            The pkg(5) repository format is a set of directories and

            files that conform to a pre-defined structure.

            For a version 3 repository (the current format), the

            structure is as follows:

            <REPO_ROOT>/

                catalog/

                    <catalog v1 files>

                index/

                    <index files>

                file/

                    <first two letters of file hash>/

                        <file-named-by-hash>

                pkg/

                    <stem>/

                        <manifest-file>

                trans/

                    <in-flight transaction files>

                cfg_cache (optional repository configuration file)

            Version 4 of the repository format eliminates the potential

            for unintended collisions between package metadata from

            different publishers and simplifies composition and decomp-

            osition of repository content.  The top-level is an optional

            shared storage space for data common to all publishers in

            the repository, while the publisher subdirectory contains

            data specific to a publisher.  It is essentially a nested

            repository format, and can be defined as follows:

            <REPO_ROOT>/

                file/ (optional)

                publisher/ (optional)

                    <prefix>/ (optional)

                        catalog/ (optional)

                            <catalog v1 files>

                        file/ (optional)

                            <first two letters of file hash>/

                                <file-named-by-hash>

                        index/ (optional)

                        pkg/ (optional)

                            <stem>/

                                <manifest-file-for-pkg-version>

                        trans/ (optional)

                            <in-flight transaction files>

                        pub.p5i (optional)

                pkg5.repository (required)

            By default, repository operations will store data in the

            publisher-specific location found under publisher/<prefix>

            for new repositories.

            In the case that the top-level file/ directory is used,

            automatic decomposition of contents into its publisher-

            specific components will not be possible unless

            corresponding package manifests are also available. 

            To support easy composition, filtering, and creation of

            package archives, directories above marked with the text

            '(optional)' must not be required.  The behaviour of

            consumers accessing the contents of the repository should

            be as follows based on the directory accessed:

            - file/

                This optional directory serves as a place to store file

                data for more than one publisher.  Package files are

                stored in gzip format using a sha1sum of the file as the

                filename, and then the first two letters of the filename

                as the parent directory's name.

            - publisher/<prefix>/catalog/

                If absent, consumers should determine the list of

                packages available based on the manifest files present

                in the publisher/ subdirectory.  If present, consumers

                should expect v1 (or newer) catalog files, or none at

                all, to be contained within.

            - publisher/<prefix>/file/

                Consumers should always check this subdirectory first

                (if present) when retrieving package file data if the

                publisher is known.  Package files are stored in gzip

                format using a sha1sum of the file as the filename, and

                then the first two letters of the filename as the parent

                directory's name.

            - publisher/<prefix>/index/

                If absent, search functionality should be disabled for

                this publisher, or a fallback to 'slow manifest-based

                search' performed.  If present, consumers should expect

                v1 (or newer) search files, or none at all, to be con-

                tained within.

            - publisher/<prefix>/pkg/

                If absent, search must be disabled for this publisher

                even if index is present.  If present, manifests are

                stored in pkg(5) manifest format using the uri-encoded

                version of the package FMRI as the filename, and using

                the uri-encoded package FMRI stem (name) as the parent

                directory's name.

            - publisher/<prefix>/trans/

                If absent, this directory will be created during

                publication operations.  If present, in progress

                transaction data is stored in a directory named

                by the open time of the transaction as a UTC UNIX

                timestamp plus an '_' and the URI-encoded package

                FRMI.  As an example:

                  1245176111_pkg%3A%2FBRCMbnx%400.5.11%2C5.11-0.116

                  %3A20090616T181511Z

            - publisher/<prefix>/pub.p5i

                This pkg(5) information (p5i) file should contain

                suggested configuration information for clients such as

                origins, mirrors, alias, etc.  Consumers can use this to

                provide clients with initial or suggested configuration

                information for a given publisher.  If not present, the

                publisher's identity should be assumed based on the

                directory structure, while the refresh interval should

                be assumed to be 4 hours.

            - pkg5.repository

                This file serves as an identifier and a place to store

                configuration information specific to the repository.

                It *is not* an equivalent to the existing cfg_cache

                file which will no longer be used.  Its format and

                structure are as follows:

                [repository]

                version = <integer>

            Any information found in the cfg_cache used in the previous

            repository format related to a publisher is now stored in

            the pub.p5i file for the related publisher.  (Examples of

            information include origins, mirrors, maintainer info,

            etc.)  As a result, the cfg_cache file is no longer used.

            Any depot-specific properties, such as the feed icon, logo,

            etc. are now completely managed using SMF or a user-provided

            configuration file.  This change was made not only to sim-

            plify configuration, but to separate depot configuration

            from repsitory configuration.

            An example version 4 repository might be structured as

            follows:

            <REPO_ROOT>/

              publisher/

                example.com/

                  catalog/

                    catalog.attrs

                    catalog.base.C

                  file/

                    ff/

                      fffff277f5a8fb63e57670afc178415c2c5e706d

                  index/

                    __at_depend

                    ...

                  pkg/

                    package%2Fpkg/

                      0.5.11%2C5.11-0.136%3A20100327T063139Z

                  trans/

                    1245176111_pkg%3A%2FBRCMbnx%400.5.11%2C5.11-0.116

                    %3A20090616T181511Z

                  pub.p5i

                example.net/

                  catalog/

                    catalog.attrs

                    catalog.base.C

                  file/

                    af/

                      affff277f5a8fb63e57670afc178415c2c5e706d

                  index/

                    __at_depend

                    ...

                  pkg/

                    package%2Fpkg/

                      0.5.11%2C5.11-0.133%3A20090327T062137Z

                  trans/

                    1245176111_pkg%3A%2FFAAMbnx%400.5.11%2C5.11-0.139

                    %3A20100616T181511Z

                  pub.p5i

              pkg5.repository:

                [repository]

                version = 4

    4.3. Archive Format:

        4.3.1. Requirements:

            The requirements for the on-disk archive format include:

            - support for archives greater than 8GB in size,

            - support for files in archive greater than 4GB in size,

            - support for efficient storage of hard links,

            - support for pathnames sigificantly greater than > 255

              characters in length,

            - core Python bindings exist or can be easily created using

              an existing library,

            - can be a container of compressed files, as opposed to a

              compressed container of uncompressed files,

            - open, royalty-free, well-documented format with wide

              platform support and acceptance,

            - multi-threaded decompression and compression possible,

            - creation and basic manipulation of package archives

              possible using widely-available tools,

            - simple composition and filtering of its content should be

              possible, and

            - random access to the archive contents must be possible

              without reading the entire archive file.

        4.3.2. Candidates:

            A number of potential archive formats have been considered

            for use, including:

            - 7z (7-Zip)

            - cpio

            - pax (portable archive exchange format)

            - ZIP

            The evaluations provided for each format here are not in-

            tended to be exhaustive; rather they focus on the specific

            requirements of this project.  For more information about

            these formats, and the documents used to evaluate them,

            please refer to section 6 of this proposal.

        4.3.3. 7z Evaluation:

            The 7z format was rejected for the following reasons:

            - Does not permit random access to archive contents or

              requires the entire archive file to access the contents

              and adding this would require a custom variation of 7z.

            - Although the 7z format supports compression methods other

              than LZMA, a primary motivator for using 7z would be the

              ability to use LZMA natively as part of the conatiner

              format.  However, the tradeoffs in terms of CPU and memory

              footprint currently make LZMA unsuitable for pkg(5) when

              compared to other compression algorithms such as those

              used by gzip(1).

            - Use of the 7z format would require integration of the LZMA

              SDK (which also provides a basic 7z API in C) and the cre-

              ation of python bindings or the integration of a third

              party's (such as pylzma).

            - No native support for extended attributes or UNIX owner/

              group permissions.

        4.3.4. cpio Evaluation:

            The cpio format doesn't natively support random access to

            archive contents, but the format itself doesn't prevent

            this.  An index could be added first file in the archive

            with the information needed to provide fast, random access

            to the archive contents.

            The cpio format was rejected for the following reasons:

            - The length of pathnames in cpio archives is limited to

              256 characters for the portable format.

            - Available tools vary significantly in maximum archive size

              support.

            - The portable cpio format stores a copy of the file data

              with every hard link in an archive instead of simply

              storing a pointer to the source file in the archive.

        4.3.4. PAX Evaluation:

            The PAX format meets all of the requirements except that of

            random access to archive contents.  However, the format

            itself doesn't prevent this.  A table of contents file could

            be supplied as the first file in the archive with the info-

            rmation needed to provide fast, random access to the con-

            tainer contents.

        4.3.5. ZIP Evaluation:

            The ZIP format meets all of the requirements listed above

            (assuming that ZIP64 extensions are used), with the ex-

            ceptions listed below for which it was rejected:

            - The use or implementation of some of the functionality

              documented in the .ZIP file format requires a license from

              PKWARE.

            - While random archive content access is possible, the ZIP

              file format stores the index for the archive at the end of

              the archive (as opposed to the beginning).  This increases

              the number of round trips that would be required for

              potential remote random content access.  It also means

              that extraction requires multiple seeks to the end of the

              file before any content can be extracted from the archive,

              which can be detrimental to performance for some media

              types (optical, etc.).

        4.3.6. Evaluation Conclusion:

            Based on the requirements set forth in section 4.3.1, the

            PAX format was selected as the on-disk archive format

            for pkg(5) packages.  However, to enable efficient access

            to the archive contents, an index file needs to be present

            as the first file in the archive.

            Early evaluations of an unoptimised prototype were performed

            using a repository containing all packages for build 136 and

            unbundleds.  The on-disk size of the repository was appox-

            imately 4.98G.  The resulting archive was 5.0G in size, with

            an archive index file 9.7M in size (when the index was comp-

            ressed using gzip).

            First time access to the prototype archive for extraction of

            a single file after creation yielded a total time of approx-

            imately 5 seconds compared to approximately 36-42 seconds

            for utilities such as pax(1), tar(1), or gtar(1).

            Creation of the archive took 7 minutes, 35 seconds on a

            custom-built Intel Core 2 DUO E8400, with 8GB Memory,

            and a 1TB 10000 RPM SATA Drive w/ 64MB Cache.