pkg

DEPOT

1.  Summary

    This document describes the architecture of a pkg(5) depot server as

    implemented in pkg.depotd(1m).  This includes: an overview of the depot

    server's filesystem layout, operations provided by the depot server,

    the interfaces provided by the depot server, and a description of how

    clients communicate with the depot server for each operation.

    NOTE: The image packaging system is under development.  Changes to

    interfaces may occur as part of the architectural review process, as

    shortcomings are identified, and as new features are introduced.

    Questions about planned or possible change should be asked on

    pkg-discuss.

2.  Discussion

    A pkg(5) depot server provides a way for clients to interact with package

    content and metadata contained within a pkg(5) repository, and with the

    depot server itself.  It accomplishes this by providing an HTTP-based

    interface suitable for usage by pkg(1) clients and web user agents.

2.1.  Filesystem Layout

    The types of information that the depot server stores and/or retrieves can

    be categorized as follows:

    - depot data

        This includes: configuration data, presentation content (such as

        web page templates), publishing data (e.g. in-flight transactions),

        and temporary data (e.g. the feed cache).

    - repository data

        This includes: catalog information, package content (files), package

        metadata (manifests), and search data.

2.1.1.  Layout

    The depot server uses the following 'root' directory structures for the

    storage and retrieval of depot and repository data:

    - repo_dir (depot and repository data)

        cfg_cache (depot data)

            A file containing the cached configuration information for the

            depot server.

        catalog/ (repository data)

            This directory contains the repository catalog and its related

            metadata.

        file/ (repository data)

            This directory contains the file content of packages in the

            repository.

            Files are stored using a two-level path fragment, derived from the

            SHA1-hash of a file's content, assumed to have at least 8 distinct

            characters.

            Example:

                00/

                    0023bb/

                        000023bb53fdc7bcf35e62b7b0b353a56d36a504

        index/ (repository data)

            This directory contains the search indices for the repository.

        pkg/ (repository data)

            This directory contains the metadata (manifests) for the

            repository's packages.

            The manifests for each package are stored in a directory with the

            same name as the package stem using a URL-encoded filename.

            Example:

                entire/

                    0.5.11%2C5.11-0.86%3A20080422T234219Z

        trans/ (depot data)

            This directory contains in-flight transactions for packages that

            are waiting for the publication process to complete so that they

            can be added to the repository's catalog.

            Each transaction is stored in a directory named after the pending

            transaction id and contains the manifest waiting for publication

            to finish stored with the filename of 'manifest'.

            Example:

                1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/

                    manifest

        updatelog/ (repository data)

            This directory contains metadata detailing changes to the repository

            by publishing operations.

    - content_root (depot data)

        web/

            This directory contains all of the web presentation content for the

            depot.

2.2.  Operations

    When communicating with the depot server via HTTP, operations are presented

    via a URL-based mechanism that allows each to be versioned so that changes

    in protocol can be hidden from older clients as the interfaces to operations

    evolve.

    Operations made available by a pkg.depotd(5) server can be accessed via GET

    or POST, as appropriate for each operation, via a URL such as the following:

        http://pkg.opensolaris.org/release/manifest/0/SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z

    The above example can be broken down into four basic components:

        publisher_origin_url    - http://pkg.opensolaris.org/release/

        operation_name          - manifest

        protocol_version        - 0

        operation_arguments     - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z

    Each of these components can be described as follows:

        publisher_origin_url    - A URL that can be used to access a depot

                                  server's repository.

        operation_name          - The name of the operation that the client is

                                  wanting to initiate.

        protocol_version        - An integer value representing the version of

                                  the operation's protocol spoken by the client.

        operation_arguments     - String data (such as a package FMRI) that is

                                  parsed and then used to determine what

                                  resource(s) will be used to perform an

                                  operation.  Some operations expect arguments

                                  or data to be passed via POST-based form data,

                                  headers, or the request body instead.

2.2.1.  Operation Types

    Each operation that the depot server provides is either designed to interact

    with a pkg(5) repository, or with the depot server itself.  These operations

    can be categorized as follows:

    - content

        These operations are read-only, and retrieve file data that comprises

        the content of a package in a repository.

    - depot

        These operations are read-only, and permit retrieval of: the list of

        operations that the depot server currently provides (including protocol

        version and pkg(5) software version), statistics information, and other

        depot information.

    - metadata

        These operations are read-only, and retrieve metadata related to a

        package FMRI, such as its name, version, etc. stored in a repository's

        catalog.

    - publishing

        These operations alter a repository's catalog, package metadata, and

        allow storage of package content.

2.2.2. Modes

    Which types of operations are available is dependent on which mode the depot

    server is currently operating in:

        - default

            In default mode, the depot server allows content, depot, metadata,

            and publishing operations.

        - readonly

            In readonly mode, the depot server allows content, depot, and

            metadata operations.

        - mirror

            In mirror mode, the depot server allows content and depot

            operations.

2.2.3.  Content Operations

    The pkg.depotd(5) server provides the following operations for retrieving

    package content:

    - file

        Version 0:

            A GET operation that retrieves the contents of a file, belonging to a

            package, using a SHA-1 hash of the file's content.

            Example:

                URL:

                http://pkg.opensolaris.org/release/file/0/

                a00030db8b91f85d0b7144d0d4ef241a3f1ae28f

            Expects:

                A SHA-1 hash of the file's content belonging to a package in the

                request path.

            Returns:

                The contents of the file, compressed using the gzip compression

                algorithm.

2.2.2.  Depot Operations

    - versions

        Version 0:

            A GET operation that retrieves text data representing what operations

            are supported and version information about the depot server.

            Example:

                URL:

                http://pkg.opensolaris.org/versions/0/

            Expects:

                Nothing

            Returns:

                text/plain data containing the version of the pkg(5) software that

                the depot is based upon, a list of the operations currently

                supported, and the protocol version supported for each

                operation.

            Sample Output:

                pkg-server bfc04991436e

                info 0

                search 0

                versions 0

                catalog 0

                manifest 0

                add 0

                file 0

                abandon 0

                close 0

                open 0

2.2.2.  Meta-data Operations

    - catalog

        Version 0:

            A GET operation that retrieves a text/plain datastream

            representing a complete catalog or an incremental update to an

            existing one as requested.

            Example:

                URL:

                http://pkg.opensolaris.org/catalog/0/

            Expects:

                Nothing or the following headers:

                    If-Modified-Since: {ISO 8601 formatted date and time in UTC}

            Returns:

                Either the contents of a pkg(5) catalog file, or the entries

                that were added since the specified date as they are found

                in the catalog file, separated by newlines.

    - info

        Version 0:

            A GET operation that retrieves a text/plain description of a

            package and its licensing information specified by the provided

            FMRI.

            Example:

                URL:

                http://pkg.opensolaris.org/info/0/[email protected],5.11-0.101:20081119T235706Z

            Expects:

                A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix

                and publisher information, and including the full version

                information.

            Returns:

                A text/plain representation of the specified package and its

                licensing information.

            Sample Output:

                Name: entire

                Summary: entire incorporation

                Publisher: Unknown

                Version: 0.5.11

                Build Release: 5.11

                Branch: 0.101

                Packaging Date: Wed Nov 19 23:57:06 2008

                Size: 0.00 B

                FMRI: pkg:/[email protected],5.11-0.101:20081119T235706Z

                License:

    - manifest

        Version 0:

            A GET operation that retrieves the contents of the manifest file for

            a package specified by the provided FMRI.

            Example:

                URL:

                http://pkg.opensolaris.org/manifest/0/[email protected],5.11-0.101:20081119T235706Z

            Expects:

                A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix

                and publisher information and including the full version

                information.

            Returns:

                The contents of the package's manifest file.

    - p5i

        Version 0:

                A GET operation that retrieves an application/vnd.pkg5.info

                datastream representing publisher and package information.

                This is intended for consumption by clients for the purposes

                of auto-configuration, metadata management policy determination,

                and triggering packaging operations such as installation.

            Example:

                URL:

                http://pkg.opensolaris.org/release/p5i/0/SUNWcs

            Expects:

                A full or partial URL-encoded pkg(5) FMRI, excluding the

                publisher prefix.  If the partial or full FMRI is valid, it will

                be added to the datastream as is.  If it includes the wildcard

                character '*', a search of the repository's catalog for matching

                entries will be performed and the unique set of resulting

                package stems will be added to the datastream.  If no match is

                found, a 404 error will be raised.

            Returns:

                Returns a pkg(5) information datastream based on the repository

                configuration's publisher information and the provided full or

                partial FMRI or matching entries.  The Content-Type of the

                response is 'application/vnd.pkg5.info'.

    - publisher

        Version 0:

                A GET operation that retrieves an application/vnd.pkg5.info

                datastream representing publisher information.  This is intended

                for consumption by clients for auto-configuration and metadata

                management policy determination.

            Example:

                URL:

                http://pkg.opensolaris.org/release/publisher/0

            Expects:

                Nothing

            Returns:

                Returns a pkg(5) information datastream based on the repository

                configuration's publisher information.  The Content-Type of the

                response is 'application/vnd.pkg5.info'.

    - search

        Version 0:

            A GET operation that retrieves a text/plain list of packages with

            metadata that matches the specified criteria.

            Example:

                URL:

                http://pkg.opensolaris.org/release/search/0/vim

            Expects:

                A URL-encoded token representing the search criteria.

            Returns:

                A text/plain list of matching entries, separated by newlines.

                Each entry consists of a set of four space-separated values:

                    index   - what search index the entry was found in

                    action  - what package action the entry is related to

                    value   - the value that the matched the search criteria

                    package - the fmri of the package that contains the match

                Results are streamed to the client as they are found.

            Sample Output:

                basename pkg:/[email protected],5.11-0.101:20081119T230659Z dir usr/share/vim

                basename pkg:/[email protected],5.11-0.93:20080708T171331Z file usr/bin/vim

2.2.3.  Publishing Operations

    - add

        Version 0:

            A POST operation that adds content to an in-flight transaction for

            the Transaction ID specified.  This could either be file content

            for the package or metadata about the package.

            This data is not added to the repository for retrieval until a close

            operation for the specified Transaction ID is executed.

            Example:

                URL:

                http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z

                HEADERS:

                X-IPkg-SetAttr1: description=Package Name

                REQUEST BODY:

            Expects:

                A Transaction ID as output by pkgsend(1) in the request path.

                The file content (if applicable), to be added, in the request

                body.  Any attributes to be set in the headers in the pattern

                of:

                    X-IPkg-SetAttr{integer}: attr=value

            Returns:

                Response status of 200 on success; any other status indicates

                failure.

    - abandon

        Version 0:

            A GET operation that aborts an in-flight transaction for the

            Transaction ID specified.  This will discard any data related to

            the transaction.

            Example:

                URL:

                http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z

            Expects:

                A Transaction ID as output by pkgsend(1) in the request path.

            Returns:

                Response status of 200 on success; any other status indicates

                failure.

    - close

        Version 0:

            A GET operation that ends an in-flight transaction for the

            Transaction ID specified.  If successful, the corresponding package

            is added to the repository catalog and is immediately available to

            repository users.

            Example:

                URL:

                http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z

            Expects:

                A Transaction ID as output by pkgsend(1) in the request path.

            Returns:

                Response status of 200 on success; any other status indicates

                failure.

    - open

        Version 0:

            A GET operation that starts an in-flight transaction for the

            package FMRI specified.

            Example:

                URL:

                http://pkg.opensolaris.org/open/0/system%[email protected]

            Expects:

                A URL-encoded pkg(5) FMRI (excluding timestamp).

            Returns:

                Response status of 200 on success and an identifier for the new

                transaction in the 'Transaction-ID' response header; any other

                status indicates failure.

3.  References