Userland Consolidation Packaging Guidelines.

	Each component that integrates into the Userland consolidation must have at

least one package manifest that describes the content to be delivered.  In some

cases components *may* deliver through multiple packages.  Canonical component

package manifests must be placed in the component's build directory.  They also

must be named *.p5m.

    In order to understand what must go in the content of a package manifest,

it's useful to have an understanding of how a canonical manifest is transformed

into a final manifest used for package publication.  Manifest transformation

takes the following basic path:

    canonical manifest

    (.../{component}/{component}.p5m)

            |

            v

    mogrified manifest

    (.../{component}/{build-dir}/manifest-$(MACH)-{component}.mogrified)

            |

            v

    mangled manifest file contents

    (.../{component}/{build-dir}/manifest-$(ARCH)-{component}.mangled)

            |

            v

    dependencies generated

    (.../{component}/{build-dir}/manifest-$(MACH)-{component}.depend)

            |

            v

    dependencies resolved

    (.../{component}/{build-dir}/manifest-$(MACH)-{component}.depend.res)

            |

            v

    manifest validation

    (.../{component}/{build-dir}/.linted-$(MACH))

            |

            v

    publication manifest

    (.../{component}/{build-dir}/manifest-$(MACH)-{component}.published)

            |

            v

    publication

Canonical Manifest

    The canonical manifest contains actions that can't otherwise be generated

    automatically from the data encapsulated in the component Makefile, gate

    transformations, build tree, and packaging tools.  This includes actions

    for license information, some path related attributes, legacy actions, 

    non-discoverable dependencies, users, groups, drivers, and others.

    Actions that are associated with objects that are specific to a single

    architecture should be tagged with a 'variant.arch' attribute specific to

    the architecture that applied to the action.  Ex:

        file path=/usr/lib/$(MACH64)/libx86onlybits.so variant.arch=i386

    Actions for editable files must include an appropriate 'preserve' attribute:

        file path=etc/gnu/a2ps.cfg preserve=true mode=0644

    license actions should be placed in the canonical manifest, as should a

    handful of other special actions noted in the "Manually generated actions"

    section just below.

Manually generated actions

    * com.oracle.info.description is a terse description of what utilities,

      libraries and/or services the package provides.  This should be short,

      specific, concise text, identifying the technology covered by the

      associated license(s).  It should fit naturally in the sentence "This

      package may contain XXX."  For example, "XXX" might be "the tar command"

      or "bzip2 compression software."  When appropriate, this may begin with

      "portions of" or another, more specific qualifying clause.

    * com.oracle.info.tpno is the Oracle 3rd party license number.

    * info.classification is "org.opensolaris.category.2008:FOO" where FOO

      varies according to the sorts of utilities, libraries and/or services

      that the package provides.  Existing packages contain most useful

      values; check them out to find the closest match.  For a complete

      list of allowed values, refer to the Solaris system file

      /usr/share/lib/pkg/opensolaris.org.sections .

    * org.opensolaris.arc-caseid is typically "PSARC/YYYY/###" and multiple

      different values are allowed.

    * pkg.summary is a short synopsis of what the package provides.

    * org.opensolaris.consolidation is the name of the consolidation delivering

      the package.  In Userland, this is $(CONSOLIDATION) (which expands to

      "Userland" during the build).  Manifests in the Userland gate can also

      decorate this package attribute with an 'incorporate={incorporation-name}'

      decoration to specify where the package should be incorporated at the end

      of the userland build.  A special value of 'none' will cause the package

      to be unincorporated and float freely from the rest of the rest of the

      packages.  Note that unincorporated packages don't automatically get

      updated with the rest of the system when 'pkg update' is run unless the

      unincorporated package(s) are specified on the command line.

Mogrified Manifest

    The canonical manifest is combined with a set of the transforms

    in $(WS_TOP)/transforms, and a set of macros to more complete

    package manifest using pkgmogrify(1).  The transforms apply default

    attributes to the various actions in the canonical manifest(s).  More

    detail about the attributes can be found in the transform file themselves.

    The macros applied at the time of mogrification are as follows:

        $(MACH)

        $(MACH32)

        $(MACH64)

        $(PUBLISHER)

        $(CONSOLIDATION)

        $(BUILD_VERSION)

        $(SOLARIS_VERSION)

        $(OS_VERSION)

        $(IPS_COMPONENT_VERSION)

        $(COMPONENT_VERSION)

        $(COMPONENT_PROJECT_URL)

        $(COMPONENT_ARCHIVE_URL)

Dependencies Generated

    The mogrified manifest and the prototype install tree are passed through

    pkgdepend(1) to generate a set of dependencies for the package content.

    These dependencies are only those that "pkgdepend generate" can determine

    on its own.  Additional dependencies that cannot be automatically

    determined by pkgdepend(1) should be placed in the canonical manifest.

    Statically defined dependencies should be described in a canonical manifest

    in an unresolved form (ie. the form generated by "pkgdepend generate").

    Ex:

	    depend fmri=__TBD pkg.debug.depend.file=etc/passwd \

		        pkg.debug.reason=usr/bin/vipw type=require

        depend fmri=__TBD pkg.debug.depend.file=sh \

                pkg.debug.depend.path=usr/bin \

                pkg.debug.depend.reason=usr/bin/psmandup \

                pkg.debug.depend.type=script type=require

    This will allow the next step to resolve all dependencies to their proper

    package(s).

Dependencies Resolved

    The manifest with unresolved dependencies is passed through pkgdepend(1)

    again to resolve dependencies against the package repositories.  The result

    is a manifest that is suitable for publication.  All these manifests are

    processed together in a single step, which is more efficient than resolving

    dependencies in each manifest separately.  While each manifest ends up with

    a .depend.res copy in the build directory, the umbrella dependency

    resolution target is {build-dir}/.resolved-$(MACH).

    The resolution step is also set up to use the -e flag to pkgdepend resolve,

    which limits the set of packages it looks at to resolve the dependencies it

    generated in the previous step.  This makes the resolution step a great deal

    faster, but requires that you include a static list of these packages in

    your component Makefile REQUIRED_PACKAGES macro.  You can automatically

    add REQURIED_PACKAGES settings for the packages used in dependency

    resolution by running "gmake REQUIRED_PACKAGES".  Once you have done so,

    you should manually verify the Makefile additions.

Manifest Validation

    The resolved manifest(s) and prototype install tree are passed through

    a set of validations.  This includes running pkglint(1), comparing the

    manifest content to the prototype install tree, and validation of the file

    content of the prototype install tree.  Any anomalies are reported.

    Content validation is performed by extension to pkglint(1) in

    $(WS_TOP)/tools/python/userland-lint

    Note that when integrating new packages, and one or more of them depends

    on or more of the others, then this may result:

        WARNING pkglint.action005.1      obsolete dependency check skipped:

            unable to find dependency (target pkg) for (source pkg)

    This means that the target package was not found in the reference repo,

    which was the source of the pkglint cache that was created when 'gmake

    setup' was run after the workspace was created; thus the warning is

    harmless and can be ignored in this circumstance.

Publication.

    Once manifest validation has occurred, the package(s) is/are finally

    published to the workspace package repository.

Facets

    Many facets are automatically set by default via transforms 

    in $(WS_TOPS)/transforms, such as facet.doc.*, facet.devel, etc.

    optional.test

    In most cases, we do not include files in the package manifest

    if they are used exclusively for testing that component.

    If we do include component test files, they should be tagged

    with the facet optional.test=true.  There is no default transform

    for such files, since they do not follow a naming pattern.

    See the components python/python-3{4,5}, mailman, or 

    python/astroid for examples.

Renames

    Renames in IPS are tricky.  We will use a case study to illustrate how

    this needs to work.

    All of the library/python-2/FOO modules that we had in Userland at the

    time were renamed to library/python/FOO in s12_41.  This involved:

    * changing the FMRI to drop the "-2"

    * adding an optional dependency on the old name at the version and build

      in which the rename occurred; this forces the rename.

    * depend type=optional fmri=library/python-2/FOO-$(PYV)@VERSION,BUILD

      where FOO is the component name (e.g., "alembic", "amqp", etc.),

      "$(PYV)" is that literal string, VERSION was the expanded value of

      COMPONENT_VERSION from each component's Makefile (e.g., "0.6.0" for

      alembic, "1.0.12" for amqp, etc.), and BUILD was set to

      "5.12-5.12.0.0.0.41.0".  Note that BUILD needs to be set to the value of

      the build you are integrating into.

    * creation of the file "history" (or adding to it if it already exists)

      in each affected component directory (see NOTE 2 below); one or more

      new lines should be added (in the case of python or other language

      component, one for the versionless package plus one for each versioned

      instance of the package; for an example of a less complicated

      component that does not have a 'versionless' package like python, see

      components/subversion)

      with syntax:

	  renames:      [email protected],BUILD NEW-NAME

	  obsoletions:  [email protected],BUILD

      where e.g. OLD-NAME was library/python-2/FOO, NEW-NAME was

      library/python/FOO, VERSION and BUILD are all as above.  Most Python

      modules, including the two mentioned above, have such files which can

      serve as examples.

      By default, packages created from the history files will be part of

      the userland-incorporation.  If a package needs to be included in a

      different incorporation, then incorporate=INCORPORATION may be

      placed on the end of either the rename or obsoletion line.

      Examples with incorporate=consolidation/desktop/gnome-incorporation

      may be found in components/meta-packages/history/history.

    So far, not too bad.  This gets slightly more complicated when back-

    porting in that BUILD needs to be set to the back-port build (e.g.,

    "5.11-0.175.3.0.0.8.0", as was the case for tkinter-27) rather than

    the S12 build.

    If the component is at the same version as when the rename was done, then

    that's it.  But if the component has since been upgraded, then care must

    be taken to set VERSION in both places above to the value that it was in

    s12_41 when the rename was originally done.  In particular, the latest

    version of a package name in an earlier release branch must not be greater

    than the latest version of that same package name in the newer release

    branch.  Otherwise the audits from Release Engineering will complain and

    our gatekeeper will make you do a follow-up push to fix this.

    In addition, during the transition period before all CBE machines have

    the new package, the REQUIRED_PACKAGES in the Makefiles of any components

    that depend on your package must include both your old and new package

    names.  This is because those names are actually the limit set, to speed

    up processing, rather than a true required list.

    If you tell pkgdepend it can look for the the component dependencies in

    either the old package name or the new name, then it will find them and

    be happy.  If only one of your package names is listed and it doesn't

    happen to be the one installed on the build system, the dependencies will

    be unsatisifed and the build will break.

    Until the new package name is available for installation, "gmake

    system-check" will fail because it attempts to install all the packages

    listed in REQUIRED_PACKAGES.  This can be ignored, but should be noted

    in a flag-day so that people are aware they can ignore it.  Once the new

    package name can be installed (even simply after running "gmake publish"

    for the missing component and having your workspace publisher enabled on

    the test system), this is no longer an issue, so there is no urgency in

    removing the old name from the list.

    NOTE:  The package version for the old name must be in the format shown

    on in pkg contents -m output (before the colon) and not in the format

    seen in the output of pkg list or pkg info, e.g.:

    So, if contents -m library/python-2/subversion showed this:

    set name=pkg.fmri value=pkg://solaris/library/python-2/[email protected],5.12-5.12.0.0.0.41.0:20140130T085214Z

    they used this:

        library/python-2/[email protected],5.12-5.12.0.0.0.41.0

    not this:

        library/python-2/[email protected]

    NOTE 2: The generic package renaming directions in Solaris documentation,

    which call for preserving a 'rename' package (containing only metadata and

    dependencies on the new package names) is not applicable when using the

    Userland history file method.  The 'rename' package is created by

    the Userland consolidation build, using the data in the history file.

    So simply rename and edit the manifest (p5m file) to the new name instead

    of creating an additional file.

=============================

Component EOF (End Of Feature) 

=============================

This is a brief description on how to remove a component from Userland for

an EOF.

Deleting the Component

----------------------

At some point, you will need to do a "hg remove" of the component's directory.

This can be done before or after you make the necessary entry in the

components/meta-packages/history/history file.  In this file you will see plenty

of examples like this:

backup/[email protected]

You will need to make sure you have the correct package name, version, and

build you are targeting for the EOF for your new entry.  Also check for an old

SUNW entry at the top of the file.  If there is one you will need to change

this to match the targeted build.  So something like this:

[email protected],5.11-0.133 backup/[email protected]

Should be changed to:

[email protected]

Remember that the package may have had numerous names in its history.

Generally the way to find this is by using:

    pkg search -o search.match_type,pkg.name depend::<package name>

First with the package to be EOF'ed, and then examining each of the dependent

packages. If there are require dependencies on any of the dependents, then the

latter should be examined to see if it need to be either EOF'ed at the same

time, or at least updated to remove the dependency you're trying to EOF.

Check to see if the package you are removing is referenced by a group package

(pkg list -n 'group/*').  See the 'EOFs and Removals' and 'Group Packages'

sections in /usr/src/pkg/README.pkg from the on-gate for further details. 

Testing

-------

To test your changes the first thing you will need to do is to invoke

'gmake publish' from the components/meta-packages/history directory or at the

top level (a full Userland build) if you are removing multiple components.  Once

this is complete you will need to setup a client with the EOF package(s)

installed.

Then perform the following steps:

1. # pkg set-publisher \

     -P -p file:///net/strax/builds/your_name/your_dir/i386/repo

If you are removing just a single component:

2. pkg change-facet 'facet.version-lock.<eof_package_name_goes_here>=false'

3. pkg update pkg://nightly/<eof_package_name_goes_here>

After the update the component package should be removed from the test system.

If you are removing multiple components:

2. pkg change-facet \

     version-lock.consolidation/userland/userland-incorporation=false

3. pkg update pkg://nightly/consolidation/userland/userland-incorporation

After the update the EOF component packages should be removed from the test

system.

For languages and anything else that has modules or is otherwise part of the

Userland build environment you will need to do a full build, of both ISAs, on

the test machines you just updated.

# vi:set fdm=marker expandtab ts=4: