pkg(5): image packaging system

CATALOG FORMAT AND CACHING PROPOSAL

1.  Overview

    The pkg(5) server and client catalogs currently provides a summary

    view of the packages provided by a repository: the FMRIs of each

    package, the last time the set of available packages changed, and

    the total number of packages.  The server uses this information

    for publication checks, to fulfill client requests, for search

    indexing and analysis, and to enable browser-based access to the

    repository via the BUI (Browser User Interface).  pkg(5) clients

    use this information to determine what packages are available, to

    validate user input, and to fulfill packaging operation requests.

1.1  History

    As development of the Image Packaging System has progressed, both

    the server and client have increasingly required access to more

    packaged metadata as fixes and various improvements have been

    implemented.  This has resulted in increased demand on server and

    client system resources when analyzing package metadata, and

    increased processing times as well.

    To address catalog performance issues, a client-side unified

    catalog cache was implemented, and initially contained all known

    package stems from the set of publishers defined within the image

    configuration.  The caching mechanism was then replaced, using a

    Python dict structure designed for efficient lookups of package

    information by stem or FMRI and providing an ordered list of

    versions, that was then serialized to disk.

    Recently, the caching was revised to use a custom, delta-encoded

    text format that avoided object serialization as that created an

    implicit dependency on object versions and definitions, as well as

    significant overhead in the on-disk footprint.  To improve package

    metadata performance, a new cache format was created that factored

    package manifests by the types of actions contained within, and

    then stored each type of action in a separate file for each

    manifest.

1.2  Challenges

    Despite past improvements, significant performance improvements

    are still needed for both the server and client when processing

    and analyzing package metadata.  The work done so far has also

    only benefited the client, leaving server performance behind.

    Specifically, the underlying catalog data, caching mecahnisms,

    and catalog retrieval operations suffer from the following

    issues:

    - the catalog format used for the server and client is not

      consistent and the server uses local time instead of UTC

    - the client does not maintain a 1:1 copy of the server's catalog

      and attributes making it difficult to verify its integrity and

      complicates the logic needed to obtain updates

    - the caching mechanisms implemented are not granular enough,

      causing some operations to run slower than necessary as more

      information than is needed is loaded and processed

    - no efficient lookup mechanism exists for some of the metadata,

      causing operations such as dependency calculation to require a

      linear scan and retrieval of manifests

    - the existing caching mechanisms require clients to retrieve

      manifests for all known packages to be able to perform summary

      listings of available packages (at least 65 MiB for a new build

      of OpenSolaris) -- which is especially harmful to GUI clients

      such as packagemanager(1)

    - the existing caching mechanisms do not provide the information

      needed to determine (ahead of time) what package manifests need

      to be retrieved during packaging operations, which leaves pkg(5)

      clients unable to provide sufficient feedback to the user during

      plan creation such as number of bytes to be transferred, time

      estimates, etc.

    - the catalog operation and caching mechanisms offered by the

      depot server are not extensible, and cannot accommodate new

      metadata that may be needed to perform client operations

      without a client and server revision

    - the catalog and caching mechanisms do not account for

      future localization needs

1.3  Goals

    So then, the changes proposed within this document have the

    following goals:

    - unification of the server and client catalog format and code

    - simplification of catalog update and retrieval mechanisms

    - improved granularity and transparency in caching mechanisms

      allowing operations to only retrieve the information they need

    - reduction of resource requirements and processing time forserver

      and client

    - increase of available metadata before long-running package

      operations to enable improved progress and user feedback

    - improved extensibility of the catalog depot operation and the

      caching mechanisms used by the client

    - unification and implementation of caching mechanisms and code

      for client and server

2.  Proposed Changes

    The changes needed to accomplish the goals listed in section 1.3

    are grouped below by the type of change.  It should be noted that

    what is described in this document is dependent on an upcoming image

    and repository format versioning proposal since these changes will

    require a change to the structure of both images and repositories.

2.1 Catalog Format Changes

2.1.1  Current Catalog Format

    To better understand the proposed changes, it may first be helpful

    to understand the current catalog format and how it is composed.

    Currently, the catalog could be viewd as being composed of three

    files:

    - attrs

        The attrs file contains metadata about the catalog.  The

        server and client attrs file are text/plain, and currently

        have the following content:

         S Last-Modified: 2009-06-23T07:58:35.686485

         S prefix: CRSV

         S npkgs: 40802

        The client adds this content:

         S origin: <repository_uri>

        The Last-Modified value is an ISO-8601 date and time in server

        local time (not UTC).

    - catalog

        The server catalog file currently contains entries of this

        format:

        <type> <fmri><newline>

        Where type can be 'V' (version), 'C' (critical; not used), or

        'R' (renamed).

        As a special exception, the format of 'R' entries is:

        R <src_stem> <src_version> <dest_stem> <dest_version><newline>

        If a destination package is not provided for 'R', then 'NULL'

        is used for the destination values.

        Examples:

        C pkg:/[email protected],5.11-0.111:20090507T161015Z

        V pkg:/[email protected],5.11-0.111:20090508T161015Z

        R foo 1.0:20090508T161015Z bar 1.0:20090509T161015Z

        R baz 1.0:20090508T161015Z NULL NULL

        The client catalog file contains entries of this format:

        <type> pkg <fmri_stem> <fmri_version><newline>

        As a special exception, the format of 'R' entries is:

        R <src_stem> <src_version> <dest_stem> <dest_version><newline>

        If a destination package is not provided for 'R', then 'NULL'

        is used for the destination values.

        Example:

        V pkg foo 0.5.11,5.11-0.111:20090508T161015Z

    - update log

        While not strictly a part of the catalog, the update logs serve

        as a record of changes to the catalog allowing clients to otbain

        incremental updates to a catalog instead of retrieving the

        entire catalog each time.

        It only exists on the server, and contains entries of this

        format:

        <update_type><type><space><fmri><newline>

        Where 'update_type' can be '+' (there were comments at one

        time referring to a '-' operation, but they were removed and

        the code for it was never implemented).

        Where 'type' can be 'V' (version), 'C' (critical; not used),

        or 'R' (renamed).

        As a special exception, the format of 'R' entries is:

        R <src_stem> <src_version> <dest_stem> <dest_version><newline>

2.1.2  Proposed Catalog Format

    To accomplish the goals listed in section 2.1, a new catalog

    format will be adopted.  This format will be used by the client

    to store catalog data locally, regardless of the format used by

    the repository (e.g. the repository only provides older catalog

    format).  All data is assumed to be encodable using Unicode as

    the JSON format specification requires this.

    The new catalog format splits the contents of the catalog into

    multiple parts, per-locale, but treats them as a unified set.

    That is, all of the parts have a common base, but can easily be

    merged at load time if access to multiple parts is needed.

    The catalog will be composed of the following files:

    - catalog.attrs

        This file will contain a python dict structure serialized in

        JSON (JavaScript Object Notation) format.  The metadata within

        is used to describe the catalog and its contents using the

        following attributes:

        _SIGNATURE:

            An optional dict of digest and/or cryptograhic values which

            can be used by clients to verify the integrity of the

            catalog.attrs data.  Each key should represent the name of

            the signature or digest used, and each value the signature

            itself.

        created:

            The value is an ISO-8601 'basic format' date in UTC time

            indicating when the catalog was created.  This value is

            provided by the server.

        last-modified:

            The value is an ISO-8601 'basic format' date in UTC time

            indicating when the catalog was last updated.  This value

            is provided by the server.

        package-count:

            An integer value indicating the total number of unique

            FMRI stems in the catalog.

        package-version-count:

            An integer value indicating the total number of unique

            FMRI versions in the catalog.

        parts:

            A dict of available catalog parts.  This is to enable

            clients to quickly determine what locale-specific catalog

            data might be available to them.  Each entry contains the

            date and time a part was created and last modified.  It

            may also contain digest signature entries for the part (if

            available) so that clients can verify parts after applying

            incremental updates.

        updates:

            A dict of available catalog updates.  Each entry corresponds

            to the filename of an update log named after the time the

            update occurred using an ISO-8601 'reduced accuracy basic

            format'.  Each entry also contains a last-modified date in

            ISO-8601 basic format to allow clients to determine when an

            update log was last changed without checking the repository.

        version:

            An integer value representing the version of the structure

            used within the attrs, update log, and catalog part files.

        Example:

        {

            "_SIGNATURE": {

              "sha-1": "8f5c22fd8218f7a0982d3e3fdd01e40671cb9cab"

            },

            "created": "20050614T080000.234231Z",

            "last-modified": "20090508T161025.686485Z",

            "package-count": 40802,

            "package-version-count": 1706,

            "parts": {

              "catalog.base.C": {

                "last-modified": "20090508T161025.686485Z",

                "signature-sha-1": "9b37ef267ae6aa8a31b878aad4e9baa234470d45",

              },

              "catalog.dependency.C": {

                "last-modified": "20090508T161025.686485Z",

                "signature-sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8",

              },

              "catalog.summary.C": {

                "last-modified": "20090508T161025.686485Z",

                "signature-sha-1": "b3a6ab53677c7b5f94c9bd551a484d57b54ed6f7",

              },

              "catalog.summary.FR": {

                "last-modified": "20081002T080903.235212Z",

                "signature-sha-1": "d2b6cb03677c725f94c9ba551a454d56b54ea2f8",

              },

            },

            "updates": {

              "update.20081002T08Z.C": {

                "last-modified": "20081002T080903.235212Z",

                "signature-sha-1": "a2b6cb03277c725a94c9ba551a454d56b54ea2f8",

              },

              "update.20090508T16Z.C": {

                "last-modified": "20090508T161025.686485Z",

                "signature-sha-1": "c2b6ca03473c725f94c8ba201a454d56b54ea2f8",

              },

            },

            "version": 1,

        }

    - catalog.<part_name>.<locale_name>

        Each part of the catalog will contain a python dict structure

        serialized in JSON format.  <locale_name> is an OpenSolaris

        system locale name, and should be 'C' if the file applies to

        all locales.  Version entries for each package stem are kept

        in ascending version order to allow fast lookups by the client

        and avoid sort overhead on load.  Also note that any top-level

        entry key in the structure starting with '_' will be treated

        as metadata related to the catalog or version entry and must

        be ignored unless the client can use it.

        Finally, each catalog entry can also contain an optional set

        of digest and signature key/value pairs that can be used to

        verify the content of the related package manifest.  Clients

        must ignore any key/value pairs that are unknown to them within

        the structure.  The catalog structure can be described as

        follows:

        {

          "<optional-signature-dict": {

            "<signature-name>": "<signature-value>",

          },

          "<publisher-prefix>": {

            "<FMRI package stem>": [

              {

                "version": <FMRI version string>,

                "<optional-actions>": <optional-actions-data>

                "<optional-signature-name>": "<signature-value>",

              }

            ]

          }

        }

    Initially, the server will offer the following catalog 'parts'.

    Each has its content based on a tradeoff between memory usage,

    load times, and bandwidth needs which depend on the client being

    used to perform packaging operations or the operation being

    performed.

    - catalog.base.C

        This catalog part contains the FMRIs of the packages that the

        repository contains, and an optional digest value that can be

        used for verifying the contents of retrieved manifests.  Loading

        just this content is useful when performing basic listing opera-

        tions using the cli, or when simply checking to see if a given

        package FMRI is valid.  Note that since this information is com-

        mon to all locales, this part of the catalog is only offered for

        the 'C' locale.  An example of its contents is shown below:

        {

          "_SIGNATURE": {

            "sha-1": "9b37ef267ae6aa8a31b878aad4e9baa234470d45",

          },

          "opensolaris.org":{

            "SUNWipkg":[

              {

                "version":"0.5.11,5.11-0.117:20090623T135937Z",

                "signature-sha-1": "596f26c4fc725b486faba089071d2b3b35482114"

              },

              {

                "version":"0.5.11,5.11-0.118:20090707T220625Z",

                "signature-sha-1": "ab6f26c4fc725b386faca089071d2b3d35482114"

              }

            ],

            "SUNWsolnm":[

              {

                "version":"0.5.11,5.11-0.117:20090623T144046Z",

                "signature-sha-1": "fe6f26c4fc124b386faca089071d2b3a35482114"

              },

              {

                "version":"0.5.11,5.11-0.118:20090707T224654Z",

                "signature-sha-1": "696f26c4fc124b386facb089071d2b3f35482114"

              }

            ]

          }

        }

    - catalog.dependency.C

        This catalog part contains the FMRIs of the packages that the

        repository contains, any 'depend' actions, and any 'set' actions

        for facets or variants.  This information is intended to be used

        during dependency calculation by install, uninstall, etc.  It is

        anticipated that package size summary information, and actions

        for set pkg.renamed and pkg.obsolete will be stored in this part

        as well when they become available.  Note that since this infor-

        mation is common to all locales, this part of the catalog is

        only offered for the 'C' locale.  An example of its contents is

        shown below:

        {

          "_SIGNATURE": {

            "sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8",

          },

          "opensolaris.org":{

            "SUNWdvdrw":[

              {

                "version":"5.21.4.10.8,5.11-0.108:20090218T042840Z",

                "actions":[

                  "set name=variant.zone value=global value=nonglobal",

                  "set name=variant.arch value=sparc value=i386",

                  "depend [email protected] type=require",

                  "depend [email protected] type=require",

                  "depend [email protected] type=require"

                ]

              }

            ],

            "SUNWthunderbirdl10n-extra":[

              {

                "version":"0.5.11,5.11-0.75:20071114T205327Z",

              }

            ]

          }

        }

    - catalog.summary.<locale_name>

        This catalog part contains the FMRIs of the packages that the

        repository contains and any 'set' actions (excluding those for

        facets or variants).  This information is intended to be used

        primarily by GUI clients such as packagemanager(1), or the BUI

        (Browser UI) provided by pkg.depotd(1m) for quick, efficient

        access to package metadata for listing.  An example is shown

        below:

        {

          "_SIGNATURE": {

            "catalog-sha-1": "b3a6ab53677c7b5f94c9bd551a484d57b54ed6f7",

          },

          "opensolaris.org":{

            "SUNWdvdrw":[

              {

                "version":"5.21.4.10.8,5.11-0.108:20090218T042840Z",

                "actions":[

                  "set name=description value=\"DVD creation utilities\"",

                  "set name=info.classification value=org.opensolaris.category.2008:System/Media",

                ]

              }

            ],

            "SUNWthunderbirdl10n-extra":[

              {

                "version":"0.5.11,5.11-0.75:20071114T205327Z",

                "actions":[

                  "set name=description value=\"Thunderbird localization - other 15 lang\"",

                  "set name=srcpkgs value=SUNWthunderbirdl10n-extra"

                ]

              }

            ]

          }

        }

    - update.<logdate>.<locale_name>

        This file will contain a python dict structure serialized in

        JSON (JavaScript Object Notation) format.  Where <logdate> is

        a UTC date and time in  ISO-8601 'reduced accuracy basic

        format'. <locale_name> is an OpenSolaris system locale name,

        and should be 'C' if the update log applies to all locales.

        The structure of catalog update files is similar to that of

        of catalog files, with a few exceptions.  First, each version

        entry contains additional elements indicating the catalog

        operation and the time of the operation.  Second, each entry

        also contains a set of dicts keyed by catalog part name

        indicating which catalog parts the package was added to

        contents of each of these dicts is the exact contents of the

        package's catalog part entry (excluding version).

        The supported types (<op-type> in the example below) of catalog

        operations are:

        'add'       Indicates that the corresponding FMRI and metadata

                    (if present) has been added to the catalog.

        'remove'    Indicates that the corresponding FMRI has been

                    removed from the catalog.

        The structure can be described as follows:

        {

          <optional-signature-or-signature-dict>: {

            <signature-or-signature-name>: <signature-or-signature-value>,

          },

          <publisher-prefix>: {

            <FMRI package stem>: [

              {

                "op-type": <type-of-operation>

                "op-time": <ISO-8601 Date and Time>

                "version": <FMRI version string>,

                "<catalog.part.name>": <catalog-type-metadata>,

              }

            ]

          }

        }

        An example update log might consist of the following:

        {

          "_SIGNATURE": {

            "sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8",

          },

          "opensolaris.org":{

            "SUNWthunderbirdl10n-extra":[

              {

                "op-type": "remove",

                "op-time": "20090218T042838Z"

                "version":"0.5.11,5.11-0.75:20071114T205327Z",

              }

            ],

            "SUNWdvdrw":[

              {

                "op-type": "add",

                "op-time": "20090524T042841Z",