16992741 New publication tool, which reverts pkgs in repo if content hasn't changed
17295321 FactoredManifest stores incomplete manifest to disk
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/man/pkgsurf.1 Fri Aug 09 18:02:11 2013 -0700
@@ -0,0 +1,257 @@
+'\" te
+.\" Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+.TH pkgsurf 1 "09 Aug 2013" "SunOS 5.12" "User Commands"
+.SH NAME
+pkgsurf \- Image Packaging System repository re-surfacing utility
+.SH SYNOPSIS
+.LP
+.nf
+/usr/bin/pkgsurf -s \fItarget_repo\fR -r \fIreference_repo\fR [-n]
+ [-p \fIpublisher_prefix\fR]... [-i \fIname\fR]... [-c \fIpattern\fR]...
+.fi
+
+.SH DESCRIPTION
+.sp
+.LP
+\fBpkgsurf\fR is a package publication tool for replacing packages in a target repository that have not changed since the latest published version in the reference repository. The result is a new version surface of all packages in the target repository.
+.sp
+.LP
+\fBpkgsurf\fR operations are irreversible; the target repository should be stored in its own ZFS dataset and a snapshot of the dataset should be taken before running \fBpkgsurf\fR in case the operation must be reverted.
+.sp
+.LP
+Packages in the target repository are compared to a given reference repository and analyzed for content changes. If no content change can be determined, the package manifest will be removed from the target repository and replaced with that of the reference repository. Afterwards, the dependencies of all packages in the repository are adjusted to reflect the version changes and keep the integrity of the repository intact.
+.sp
+.LP
+The target repository must be filesystem-based and should only contain one version of each package. If the target repository contains a package which is newer than the latest version in the reference repository and older than the latest version in the target repository, no package version replacement will occur for that package. For optimal performance, the reference repository should also be filesystem-based.
+.sp
+.LP
+The reference repository may contain one or more versions of each package, however, only the latest version will be used for comparison.
+.SH OPTIONS
+.sp
+.LP
+The following options are supported:
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-c\fR \fIpattern\fR\fR
+.ad
+.sp .6
+.RS 4n
+Treat every package whose FMRI matches 'pattern' as changed and do not reversion it, even if there is no content change. Can be specified multiple times.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-n\fR\fR
+.ad
+.sp .6
+.RS 4n
+Perform a trial run with no changes made to the target repository.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-i\fR \fIname\fR\fR
+.ad
+.sp .6
+.RS 4n
+Ignore set actions with the name field set to \fIname\fR for determination of content change. Package will be reversioned even if this action differs between target and reference version. Can be specified multiple times.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-p\fR \fIpublisher_prefix\fR\fR
+.ad
+.sp .6
+.RS 4n
+Specify the name of the publisher to be re-surfaced. This option can be specified multiple times.
+.sp
+By default, packages from all publishers found in target and reference repositories are re-surfaced.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-r\fR \fIreference_repo\fR\fR
+.ad
+.sp .6
+.RS 4n
+Specify the URI of the reference repository to be used for manifest comparison. Only the latest version of each package is considered.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-s\fR \fItarget_repo\fR\fR
+.ad
+.sp .6
+.RS 4n
+Path to target repository. Packages in this repository get reversioned to the versions present in the reference repository. Repository should only contain one version of each package. Must be a filesystem-based repository.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB-?\fR\fR
+.ad
+.br
+.na
+\fB\fB--help\fR\fR
+.ad
+.sp .6
+.RS 4n
+Display a usage message.
+.RE
+
+.SH ENVIRONMENT VARIABLES
+.sp
+.LP
+The following environment variable is supported:
+.sp
+.ne 2
+.mk
+.na
+\fB\fBPKG_REPO\fR\fR
+.ad
+.RS 10n
+.rt
+The absolute path of the target repository.
+.RE
+
+.SH EXAMPLES
+.LP
+\fBExample 1 \fRRe-surface repository
+.sp
+.LP
+Reversion each package in the target repository which did not have any content change from the same package in the reference repository.
+
+.sp
+.in +2
+.nf
+$ \fBpkgsurf -s /path/to/target \e\fR
+\fB-r http://reference.example.com\fR
+.fi
+.in -2
+.sp
+
+.sp
+.LP
+Sample package in target:
+
+.sp
+.in +2
+.nf
+set name=pkg.fmri value=pkg://example.com/[email protected]:20381001T163427Z
+dir group=sys mode=0755 owner=root path=usr
+.fi
+.in -2
+
+.sp
+.LP
+Sample package in reference:
+
+.sp
+.in +2
+.nf
+set name=pkg.fmri value=pkg://example.com/[email protected]:20381001T163427Z
+dir group=sys mode=0755 owner=root path=usr
+.fi
+.in -2
+
+.sp
+.LP
+Sample package in target after operation:
+
+.sp
+.in +2
+.nf
+set name=pkg.fmri value=pkg://example.com/[email protected]:20381001T163427Z
+dir group=sys mode=0755 owner=root path=usr
+.fi
+.in -2
+
+
+.SH EXIT STATUS
+.sp
+.LP
+The following exit values are returned:
+.sp
+.ne 2
+.mk
+.na
+\fB\fB0\fR\fR
+.ad
+.RS 6n
+.rt
+Command succeeded.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB1\fR\fR
+.ad
+.RS 6n
+.rt
+An error occurred.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB2\fR\fR
+.ad
+.RS 6n
+.rt
+Invalid command line options were specified.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
+\fB\fB99\fR\fR
+.ad
+.RS 6n
+.rt
+An unanticipated exception occurred.
+.RE
+
+.SH ATTRIBUTES
+.sp
+.LP
+See \fBattributes\fR(5) for descriptions of the following attributes:
+.sp
+
+.sp
+.TS
+tab() box;
+cw(2.75i) |cw(2.75i)
+lw(2.75i) |lw(2.75i)
+.
+ATTRIBUTE TYPEATTRIBUTE VALUE
+_
+Availability\fBpackage/pkg\fR
+_
+Interface StabilityUncommitted
+.TE
+
+.SH SEE ALSO
+.sp
+.LP
+\fBpkgrepo\fR(1), \fBpkg\fR(5)
+.sp
+.LP
+\fBhttps://java.net/projects/ips\fR
--- a/src/modules/client/progress.py Thu Aug 08 12:08:13 2013 +0530
+++ b/src/modules/client/progress.py Fri Aug 09 18:02:11 2013 -0700
@@ -634,6 +634,9 @@
@pt_abstract
def _li_recurse_progress_output(self, lin): pass
+ @pt_abstract
+ def _reversion(self, pfmri, outspec): pass
+
class ProgressTrackerFrontend(object):
"""This essentially abstract class forms the interface that other
modules in the system use to record progress against various goals."""
@@ -1002,6 +1005,18 @@
"""Call to indicate that the named child made progress."""
pass
+ @pt_abstract
+ def reversion_start(self, goal_pkgs, goal_revs): pass
+
+ @pt_abstract
+ def reversion_add_progress(self, pfmri, pkgs=0, reversioned=0,
+ adjusted=0):
+ pass
+
+ @pt_abstract
+ def reversion_done(self): pass
+
+
class ProgressTracker(ProgressTrackerFrontend, ProgressTrackerBackend):
"""This class is used by the client to render and track progress
towards the completion of various tasks, such as download,
@@ -1071,6 +1086,11 @@
self.archive_items = GoalTrackerItem(_("Archived items"))
self.archive_bytes = GoalTrackerItem(_("Archived bytes"))
+ # reversioning support
+ self.reversion_pkgs = GoalTrackerItem(_("Processed Packages"))
+ self.reversion_revs = GoalTrackerItem(_("Reversioned Packages"))
+ self.reversion_adjs = GoalTrackerItem(_("Adjusted Packages"))
+
# Used to measure elapsed time of entire planning; not otherwise
# rendered to the user.
self.plan_generic = TrackerItem("")
@@ -1663,6 +1683,33 @@
"""Call to indicate that the named child made progress."""
self._li_recurse_progress_output(lin)
+ def reversion_start(self, goal_pkgs, goal_revs):
+ self.reversion_adjs.reset()
+ self.reversion_revs.reset()
+ self.reversion_pkgs.reset()
+ self.reversion_revs.goalitems = goal_revs
+ self.reversion_pkgs.goalitems = goal_pkgs
+ self.reversion_adjs.goalitems = -1
+
+ def reversion_add_progress(self, pfmri, pkgs=0, reversioned=0,
+ adjusted=0):
+ outspec = OutSpec()
+ if not self.reversion_pkgs.printed:
+ self.reversion_pkgs.printed = True
+ outspec.first = True
+
+ self.reversion_revs.items += reversioned
+ self.reversion_adjs.items += adjusted
+ self.reversion_pkgs.items += pkgs
+ self._reversion(pfmri, outspec)
+
+ def reversion_done(self):
+ self.reversion_pkgs.done()
+ self.reversion_revs.done()
+ self.reversion_adjs.done(goalcheck=False)
+ if self.reversion_pkgs.printed:
+ self._reversion("Done", OutSpec(last=True))
+
class MultiProgressTracker(ProgressTrackerFrontend):
"""This class is a proxy, dispatching incoming progress tracking calls
@@ -2213,6 +2260,30 @@
if self.linked_pkg_op == pkgdefs.PKG_OP_PUBCHECK:
return
+ def _reversion(self, pfmri, outspec):
+ if not self._ptimer.time_to_print() and not outspec:
+ return
+
+ if outspec.first:
+ # tell ptimer that we just printed.
+ self._ptimer.reset_now()
+
+ if outspec.last:
+ self.__generic_done(
+ msg=_("Reversioned %(revs)s of %(pkgs)s packages "
+ "and adjusted %(adjs)s packages.") %
+ {"revs": self.reversion_revs.items,
+ "pkgs": self.reversion_pkgs.items,
+ "adjs": self.reversion_adjs.items})
+ return
+
+ self._pe.cprint(
+ _("Reversioning: %(pkgs)s processed, %(revs)s reversioned, "
+ "%(adjs)s adjusted") %
+ {"pkgs": self.reversion_pkgs.pair(),
+ "revs": self.reversion_revs.pair(),
+ "adjs": self.reversion_adjs.items})
+
class LinkedChildProgressTracker(CommandLineProgressTracker):
"""This tracker is used for recursion with linked children.
@@ -2418,6 +2489,30 @@
if outspec.last:
self.__generic_done()
+ def _reversion(self, pfmri, outspec):
+
+ if not self._ptimer.time_to_print() and not outspec:
+ return
+
+ if isinstance(pfmri, pkg.fmri.PkgFmri):
+ stem = pfmri.get_pkg_stem(anarchy=True)
+ else:
+ stem = pfmri
+
+ # The first time, emit header.
+ if outspec.first:
+ self._pe.cprint("%-38s %13s %13s %11s" %
+ (_("PKG"), _("Processed"), _("Reversioned"),
+ _("Adjusted")))
+
+ s = "%-40.40s %11s %13s %11s" % \
+ (stem, self.reversion_pkgs.pair(),
+ self.reversion_revs.pair(), self.reversion_adjs.items)
+ self._pe.cprint(s, end='', erase=True)
+
+ if outspec.last:
+ self.__generic_done_newline()
+
def _mfst_commit(self, outspec):
if self.purpose == self.PURPOSE_PKG_UPDATE_CHK:
self._up2date()
--- a/src/modules/fmri.py Thu Aug 08 12:08:13 2013 +0530
+++ b/src/modules/fmri.py Fri Aug 09 18:02:11 2013 -0700
@@ -21,7 +21,7 @@
#
#
-# Copyright (c) 2007, 2012, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2007, 2013, Oracle and/or its affiliates. All rights reserved.
#
import fnmatch
@@ -317,7 +317,7 @@
pkg_str = "pkg://"
return "%s%s/%s" % (pkg_str, self.publisher, self.pkg_name)
- def get_short_fmri(self, default_publisher = None):
+ def get_short_fmri(self, default_publisher=None, anarchy=False):
"""Return a string representation of the FMRI without a specific
version."""
publisher = self.publisher
@@ -329,7 +329,8 @@
else:
version = "@" + self.version.get_short_version()
- if not publisher or publisher.startswith(PREF_PUB_PFX):
+ if not publisher or publisher.startswith(PREF_PUB_PFX) \
+ or anarchy:
return "pkg:/%s%s" % (self.pkg_name, version)
return "pkg://%s/%s%s" % (publisher, self.pkg_name, version)
--- a/src/modules/manifest.py Thu Aug 08 12:08:13 2013 +0530
+++ b/src/modules/manifest.py Fri Aug 09 18:02:11 2013 -0700
@@ -1828,6 +1828,12 @@
origin_exclude=origin_exclude,
self_exclude=self_exclude)
+ def store(self, mfst_path):
+ """Store the manifest contents to disk."""
+ if not self.loaded:
+ self.__load()
+ super(FactoredManifest, self).store(mfst_path)
+
@property
def pathname(self):
"""The absolute pathname of the file containing the manifest."""
--- a/src/pkg/manifests/package:pkg.p5m Thu Aug 08 12:08:13 2013 +0530
+++ b/src/pkg/manifests/package:pkg.p5m Fri Aug 09 18:02:11 2013 -0700
@@ -235,6 +235,7 @@
file path=usr/bin/pkgrepo
file path=usr/bin/pkgsend
file path=usr/bin/pkgsign
+file path=usr/bin/pkgsurf
dir path=usr/lib
file path=usr/lib/pkg.depotd mode=0755
dir path=usr/share
@@ -401,6 +402,7 @@
file path=usr/share/man/man1/pkgrepo.1
file path=usr/share/man/man1/pkgsend.1
file path=usr/share/man/man1/pkgsign.1
+file path=usr/share/man/man1/pkgsurf.1
dir path=usr/share/man/man1m
file path=usr/share/man/man1m/pkg.depotd.1m
dir path=usr/share/man/man5
--- a/src/po/POTFILES.in Thu Aug 08 12:08:13 2013 +0530
+++ b/src/po/POTFILES.in Fri Aug 09 18:02:11 2013 -0700
@@ -107,3 +107,4 @@
util/publish/pkglint.py
util/publish/pkgmerge.py
util/publish/pkgmogrify.py
+util/publish/pkgsurf.py
--- a/src/setup.py Thu Aug 08 12:08:13 2013 +0530
+++ b/src/setup.py Fri Aug 09 18:02:11 2013 -0700
@@ -158,6 +158,7 @@
['util/publish/pkglint.py', 'pkglint'],
['util/publish/pkgmerge.py', 'pkgmerge'],
['util/publish/pkgmogrify.py', 'pkgmogrify'],
+ ['util/publish/pkgsurf.py', 'pkgsurf'],
['publish.py', 'pkgsend'],
['pull.py', 'pkgrecv'],
['sign.py', 'pkgsign'],
@@ -246,6 +247,7 @@
'man/pkgmogrify.1',
'man/pkgsend.1',
'man/pkgsign.1',
+ 'man/pkgsurf.1',
'man/pkgrecv.1',
'man/pkgrepo.1',
'man/pm-updatemanager.1',
--- a/src/tests/api/t_manifest.py Thu Aug 08 12:08:13 2013 +0530
+++ b/src/tests/api/t_manifest.py Fri Aug 09 18:02:11 2013 -0700
@@ -33,6 +33,7 @@
import pkg as pkg
import pkg.client.api_errors as api_errors
import pkg.manifest as manifest
+import pkg.misc as misc
import pkg.actions as actions
import pkg.fmri as fmri
import pkg.variant as variant
@@ -411,6 +412,19 @@
m1.exclude_content([v.allow_action, lambda x: True])
self.assertEqual(len(list(m1.gen_actions_by_type("dir"))), 1)
+ def test_store_to_disk(self):
+ """Verfies that a FactoredManifest gets force-loaded before it
+ gets stored to disk."""
+
+ m1 = manifest.FactoredManifest("[email protected]", self.cache_dir,
+ pathname=self.foo_content_p5m)
+
+ tmpdir = tempfile.mkdtemp(dir=self.test_root)
+ path = os.path.join(tmpdir, "manifest.p5m")
+ m1.store(path)
+ self.assertEqual(misc.get_data_digest(path),
+ misc.get_data_digest(self.foo_content_p5m))
+
def test_get_directories(self):
"""Verifies that get_directories() works as expected."""
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/tests/cli/t_pkgsurf.py Fri Aug 09 18:02:11 2013 -0700
@@ -0,0 +1,723 @@
+#!/usr/bin/python
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+#
+
+import testutils
+if __name__ == "__main__":
+ testutils.setup_environment("../../../proto")
+import pkg5unittest
+
+import os
+import pkg.catalog as catalog
+import pkg.config as cfg
+import pkg.client.pkgdefs as pkgdefs
+import pkg.fmri as fmri
+import pkg.manifest as manifest
+import pkg.misc as misc
+import pkg.p5p as p5p
+import pkg.portable as portable
+import pkg.server.repository as repo
+import shutil
+import subprocess
+import tempfile
+import time
+import urllib
+import urlparse
+import unittest
+import zlib
+
+class TestPkgsurf(pkg5unittest.ManyDepotTestCase):
+ # Cleanup after every test.
+ persistent_setup = True
+
+ # The 1.0 version of each package will be in the reference repo,
+ # the 2.0 version in the target.
+ # Since we publish the expected package to an additional repo, we have
+ # to set the timestamps to make sure the target and expected packages
+ # are equal.
+
+ # The test cases are mainly in the different types of packages we
+ # have in the repo.
+
+ # Test cases:
+
+ # Pkg with no content change, should be reversioned.
+ # Pkg has all sorts of actions to make sure everything gets moved
+ # correctly.
+ tiger_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/bat
+ add dir mode=0755 owner=root group=bin path=/usr/tiger
+ add file tmp/sting mode=0444 owner=root group=bin path=/usr/tiger/sting
+ add link path=/usr/tiger/sting target=./stinger
+ add hardlink path=/etc/bat target=/usr/tiger/bat
+ add license tmp/copyright license=copyright
+ add user username=Tiger group=galeocerdones home-dir=/export/home/Tiger
+ add group groupname=galeocerdones gid=123
+ close
+ """
+
+ tiger_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/bat
+ add dir mode=0755 owner=root group=bin path=/usr/tiger
+ add file tmp/sting mode=0444 owner=root group=bin path=/usr/tiger/sting
+ add link path=/usr/tiger/sting target=./stinger
+ add hardlink path=/etc/bat target=/usr/tiger/bat
+ add license tmp/copyright license=copyright
+ add user username=Tiger group=galeocerdones home-dir=/export/home/Tiger
+ add group groupname=galeocerdones gid=123
+ close
+ """
+
+ tiger_exp = tiger_ref
+
+ # Another basic pkg which gets reversioned
+ sandtiger_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/sandtiger
+ """
+
+ sandtiger_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/sandtiger
+ """
+
+ sandtiger_exp = sandtiger_ref
+
+ # Basic package with content change, should not be reversioned.
+ hammerhead_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/hammerhead
+ close
+ """
+
+ hammerhead_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/hammerhead
+ close
+ """
+
+ hammerhead_exp = hammerhead_targ
+
+ # Package has only dep change but dependency package changed,
+ # should not be reversioned.
+ blue_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ blue_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ blue_exp = blue_targ
+
+ # Same as above but let's try an additional level in the dep chain.
+ bull_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ bull_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ bull_exp = bull_targ
+
+ # Package has only dep change and dependency package didn't change,
+ # should be reversioned.
+ mako_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ mako_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ mako_exp = mako_ref
+
+ # Same as above but let's try an additional level in the dep chain.
+ white_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ white_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ white_exp = white_ref
+
+ # Package has content change but depends on package which got reversioned,
+ # dependencies should be fixed.
+ # Pkg has all sorts of actions to make sure everything gets moved
+ # correctly.
+
+ angel_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/angel
+ add dir mode=0755 owner=root group=bin path=/usr/angel
+ add file tmp/sting mode=0444 owner=root group=bin path=/usr/angel/sting
+ add link path=/usr/angel/sting target=./stinger
+ add hardlink path=/etc/bat target=/usr/angel/bat
+ add license tmp/copyright license=copyright
+ add user username=Angel group=squatinae home-dir=/export/home/Angel
+ add group groupname=squatinae gid=123
+ add depend fmri=pkg:/[email protected] type=require
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ angel_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/angel
+ add dir mode=0755 owner=root group=bin path=/usr/angel
+ add file tmp/sting mode=0444 owner=root group=bin path=/usr/angel/sting
+ add link path=/usr/angel/sting target=./stinger
+ add hardlink path=/etc/bat target=/usr/angel/bat
+ add license tmp/copyright license=copyright
+ add user username=Angel group=squatinae home-dir=/export/home/Angel
+ add group groupname=squatinae gid=123
+ add depend fmri=pkg:/[email protected] type=require
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ angel_exp = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/angel
+ add dir mode=0755 owner=root group=bin path=/usr/angel
+ add file tmp/sting mode=0444 owner=root group=bin path=/usr/angel/sting
+ add link path=/usr/angel/sting target=./stinger
+ add hardlink path=/etc/bat target=/usr/angel/bat
+ add license tmp/copyright license=copyright
+ add user username=Angel group=squatinae home-dir=/export/home/Angel
+ add group groupname=squatinae gid=123
+ add depend fmri=pkg:/[email protected] type=require
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ # Package has content change and depends on package which didn't get
+ # reversioned, shouldn't be touched.
+
+ horn_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/horn
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ horn_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/horn
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ horn_exp = horn_targ
+
+
+ # Package has content change but has require-any dep on package which
+ # got reversioned, dependencies should be fixed.
+
+ lemon_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/lemon
+ add depend fmri=pkg:/[email protected] fmri=pkg:/[email protected] type=require-any
+ close
+ """
+
+ lemon_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/lemon
+ add depend fmri=pkg:/[email protected] fmri=pkg:/[email protected] type=require-any
+ close
+ """
+
+ lemon_exp = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/lemon
+ add depend fmri=pkg:/[email protected] fmri=pkg:/[email protected] type=require-any
+ close
+ """
+
+ # Package has content change but has require-any dep on package which
+ # got reversioned, however, the require-any dependency wasn't in the old
+ # version. The version of the pkg in the ref repo should be substituted
+ # for tiger but not for sandtiger (since dep pkg is still successor of
+ # dep FMRI).
+
+ leopard_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/leopard
+ add depend fmri=pkg:/[email protected] fmri=pkg:/[email protected] type=require-any
+ close
+ """
+
+ leopard_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/leopard
+ add depend fmri=pkg:/[email protected] fmri=pkg:/[email protected] fmri=pkg:/[email protected] fmri=pkg:/[email protected] type=require-any
+ close
+ """
+
+ leopard_exp = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/leopard
+ add depend fmri=pkg:/[email protected] fmri=pkg:/[email protected] fmri=pkg:/[email protected] fmri=pkg:/[email protected] type=require-any
+ close
+ """
+
+ # Package has no content change but dependency stem changed, should
+ # always be treated as content change.
+ blacktip_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ blacktip_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ blacktip_exp = blacktip_targ
+
+ # Package has no content change but dependency got added, should
+ # always be treated as content change, other dependencies should be
+ # adjusted.
+ whitetip_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ whitetip_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ whitetip_exp = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/[email protected] type=require
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ # Package has no content change but a change in an attribute,
+ # should be treated as content change by default but reversioned if
+ # proper CLI options are given (goblin_exp is just for the default
+ # behavior, gets modified in actual test case)
+
+ goblin_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add set name=info.home value="deep sea"
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/goblin
+ close
+ """
+
+ goblin_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add set name=info.home value="deeper sea"
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/goblin
+ close
+ """
+
+ goblin_exp = goblin_targ
+
+ # Package only found in target, not in ref, with dependency on
+ # reversioned package. Dependency should be fixed.
+ reef_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/reef
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ reef_exp = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/reef
+ add depend fmri=pkg:/[email protected] type=require
+ close
+ """
+
+ # Package is exactly the same as in ref repo, shouldn't be touched
+ sandbar_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/sandbar
+ close
+ """
+
+ sandbar_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/sandbar
+ close
+ """
+
+ sandbar_exp = sandbar_ref
+
+ # Packages with circular dependency and no change in dep chain.
+ greenland_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend [email protected] type=require
+ close
+ """
+
+ greenland_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend [email protected] type=require
+ close
+ """
+
+ greenland_exp = greenland_ref
+
+ sleeper_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend [email protected] type=require
+ close
+ """
+ sleeper_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend [email protected] type=require
+ close
+ """
+
+ sleeper_exp = sleeper_ref
+
+
+ # Check for correct handling of Varcets. Pkg contains same dep FMRI stem
+ # twice. It also covers the case where a facet changes and the tool has
+ # to sustitute the version in the ref repo (sandtiger case).
+ whale_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0444 owner=root group=bin path=/etc/whale
+ add depend fmri=pkg:/[email protected] facet.version-lock=True type=require
+ add depend fmri=pkg:/tiger type=require
+ add depend fmri=pkg:/[email protected] facet.version-lock=False type=require
+ close
+ """
+
+ whale_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/whale
+ add depend fmri=pkg:/[email protected] facet.version-lock=True type=require
+ add depend fmri=pkg:/tiger type=require
+ add depend fmri=pkg:/[email protected] facet.version-lock=True type=require
+ close
+ """
+
+ whale_exp = """
+ open [email protected],5.11-0:20000101T000000Z
+ add file tmp/bat mode=0644 owner=root group=bin path=/etc/whale
+ add depend fmri=pkg:/[email protected] facet.version-lock=True type=require
+ add depend fmri=pkg:/tiger type=require
+ add depend fmri=pkg:/[email protected] facet.version-lock=True type=require
+ close
+ """
+
+ # Pkg in ref repo is newer than the one in target.
+ # Should not be reversioned.
+ thresher_ref = """
+ open [email protected],5.11-0:20000101T000000Z
+ close
+ """
+
+ thresher_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ close
+ """
+
+ thresher_exp = thresher_targ
+
+ # Package only found in target, not in ref.
+ # Package has a dep on a reversioned pkg, but the reversioned pkg is
+ # still a successor of the dep FMRI.
+ # The dep should not be changed.
+ bamboo_targ = """
+ open [email protected],5.11-0:20000101T000000Z
+ add depend fmri=pkg:/tiger@1 type=require
+ close
+ """
+
+ bamboo_exp = bamboo_targ
+
+
+ # Create some packages for an additional publisher
+ humpback_targ = """
+ open pkg://cetacea/[email protected],5.11-0:20000101T000000Z
+ close
+ """
+
+ humpback_ref = """
+ open pkg://cetacea/[email protected],5.11-0:20000101T000000Z
+ close
+ """
+
+ humpback_exp = humpback_targ
+
+
+ misc_files = [ "tmp/bat", "tmp/sting", "tmp/copyright" ]
+
+ pkgs = ["tiger", "sandtiger", "hammerhead", "blue", "bull", "mako",
+ "white", "angel", "horn", "lemon", "leopard", "blacktip",
+ "whitetip", "goblin", "reef", "sandbar", "greenland", "sleeper",
+ "whale", "thresher", "bamboo"]
+
+ def setUp(self):
+ """Start 3 depots, 1 for reference repo, 1 for the target and
+ 1 which should be equal to the reversioned target.
+ """
+
+ self.ref_pkgs = []
+ self.targ_pkgs = []
+ self.exp_pkgs = []
+ for s in self.pkgs:
+ ref = s + "_ref"
+ targ = s + "_targ"
+ exp = s + "_exp"
+ try:
+ self.ref_pkgs.append(getattr(self, ref))
+ except AttributeError:
+ # reef_ref, bamboo_ref don't exist intentionally
+ pass
+ self.targ_pkgs.append(getattr(self, targ))
+ self.exp_pkgs.append(getattr(self, exp))
+
+ pkg5unittest.ManyDepotTestCase.setUp(self, ["selachii",
+ "selachii", "selachii", "selachii"], start_depots=True)
+
+ self.make_misc_files(self.misc_files)
+
+ self.dpath1 = self.dcs[1].get_repodir()
+ self.durl1 = self.dcs[1].get_depot_url()
+ self.published_ref = self.pkgsend_bulk(self.dpath1,
+ self.ref_pkgs)
+
+ self.dpath2 = self.dcs[2].get_repodir()
+ self.durl2 = self.dcs[2].get_depot_url()
+ self.published_targ = self.pkgsend_bulk(self.dpath2,
+ self.targ_pkgs)
+
+ self.dpath3 = self.dcs[3].get_repodir()
+ self.durl3 = self.dcs[3].get_depot_url()
+ self.published_exp = self.pkgsend_bulk(self.dpath3,
+ self.exp_pkgs)
+
+ # keep a tmp repo to copy the target into for each new test
+ self.dpath_tmp = self.dcs[4].get_repodir()
+
+ def test_0_options(self):
+ """Check for correct input handling."""
+ self.pkgsurf("-x", exit=2)
+ self.pkgsurf("-s pacific", exit=2)
+ self.pkgsurf("-s pacific -r atlantic arctic antarctic", exit=2)
+ # invalid patterns for -c
+ self.pkgsurf("-n -s %s -r %s -c [email protected]" % (self.dpath2,
+ self.dpath1), exit=1)
+ self.pkgsurf("-n -s %s -r %s -c tig" % (self.dpath2,
+ self.dpath1), exit=1)
+
+ # Check that -n doesn't modify repo.
+ tmpdir = tempfile.mkdtemp(dir=self.test_root)
+ path = os.path.join(tmpdir, "repo")
+ shutil.copytree(self.dpath2, path)
+
+ self.pkgsurf("-s %s -r %s -n" % (self.dpath2, self.dpath1))
+
+ ret = subprocess.call(["/usr/bin/gdiff", "-Naur", path,
+ self.dpath2])
+ self.assertTrue(ret==0)
+
+ def test_1_basics(self):
+ """Test basic resurfacing operation."""
+
+ # Copy target repo to tmp repo
+ self.copy_repository(self.dpath2, self.dpath_tmp,
+ { "selachii": "selachii" })
+ # The new repository won't have a catalog, so rebuild it.
+ self.dcs[4].get_repo(auto_create=True).rebuild()
+ #self.assertTrue(False)
+
+ # Check that empty repos get handled correctly
+ tempdir = tempfile.mkdtemp(dir=self.test_root)
+ # No repo at all
+ self.pkgsurf("-s %s -r %s" % (tempdir, self.dpath1), exit=1)
+ self.pkgsurf("-s %s -r %s" % (self.dpath1, tempdir), exit=1)
+
+ # Repo empty
+ self.pkgrepo("create -s %s" % tempdir)
+ self.pkgsurf("-s %s -r %s" % (tempdir, self.dpath1), exit=1)
+ self.pkgsurf("-s %s -r %s" % (self.dpath1, tempdir), exit=1)
+
+ # No packages
+ self.pkgrepo("add-publisher -s %s selachii" % tempdir)
+ self.pkgsurf("-s %s -r %s" % (tempdir, self.dpath1))
+ self.assertTrue("No packages to reversion." in self.output)
+ self.pkgsurf("-s %s -r %s" % (self.dpath1, tempdir))
+ self.assertTrue("No packages to reversion." in self.output)
+ shutil.rmtree(tempdir)
+
+ # Now check if it actually works.
+ self.pkgsurf("-s %s -r %s" % (self.dpath_tmp, self.dpath1))
+
+ ref_repo = self.get_repo(self.dpath1)
+ targ_repo = self.get_repo(self.dpath_tmp)
+ exp_repo = self.get_repo(self.dpath3)
+ for s in self.published_exp:
+ f = fmri.PkgFmri(s, None)
+ targ = targ_repo.manifest(f)
+
+ # Load target manifest
+ targm = manifest.Manifest()
+ targm.set_content(pathname=targ)
+
+ # Load expected manifest
+ exp = exp_repo.manifest(f)
+ expm = manifest.Manifest()
+ expm.set_content(pathname=exp)
+
+ ta, ra, ca = manifest.Manifest.comm([targm, expm])
+ self.debug("%s: %d %d" %(str(s), len(ta), len(ra)))
+
+ self.assertEqual(0, len(ta), "%s had unexpected actions:"
+ " \n%s" % (s, "\n".join([str(x) for x in ta])))
+ self.assertEqual(0, len(ra), "%s had missing actions: "
+ "\n%s" % (s, "\n".join([str(x) for x in ra])))
+
+ # Check that pkgsurf informed the user that there is a newer
+ # version of a pkg in the ref repo.
+ self.assertTrue("Packages with successors" in self.output)
+
+ # Check that ignore option works.
+ # Just run again and see if goblin pkg now gets reversioned.
+ self.pkgsurf("-s %s -r %s -i info.home" % (self.dpath_tmp,
+ self.dpath1))
+
+ # Find goblin package
+ for s in self.published_ref:
+ if "goblin" in s:
+ break
+ f = fmri.PkgFmri(s, None)
+ targ = targ_repo.manifest(f)
+ ref = ref_repo.manifest(f)
+ self.assertEqual(misc.get_data_digest(targ),
+ misc.get_data_digest(ref))
+
+ # Check that running the tool again doesn't find any pkgs
+ # to reversion. Use http for accessing reference repo this time.
+ self.pkgsurf("-s %s -r %s" % (self.dpath_tmp, self.durl1))
+ self.assertTrue("No packages to reversion." in self.output)
+
+ def test_2_publishers(self):
+ """Tests for correct publisher handling."""
+
+ # Copy target repo to tmp repo
+ self.copy_repository(self.dpath2, self.dpath_tmp,
+ { "selachii": "selachii" })
+ # The new repository won't have a catalog, so rebuild it.
+ self.dcs[4].get_repo(auto_create=True).rebuild()
+
+ # Add a package from a different publisher to target
+ self.pkgsend_bulk(self.dpath_tmp, [self.humpback_targ])
+
+ # Test that unknown publisher in ref repo gets skipped without
+ # issue.
+ self.pkgsurf("-s %s -r %s -n" % (self.dpath_tmp,
+ self.dpath1))
+ # Test that we also print a skipping notice
+ self.assertTrue("Skipping" in self.output)
+
+ # Test that we fail if we specify a publisher which is not in
+ # reference repo.
+ self.pkgsurf("-s %s -r %s -p cetacea -n" % (self.dpath_tmp,
+ self.dpath1), exit=1)
+
+ # Now add packages from the 2nd pub to the ref repo
+ self.pkgsend_bulk(self.durl1, [self.humpback_ref])
+
+ # Test that only specified publisher is processed
+ self.pkgsurf("-s %s -r %s -p cetacea -n" % (self.dpath_tmp,
+ self.dpath1))
+ self.assertFalse("selachii" in self.output)
+ self.pkgsurf("-s %s -r %s -p selachii -n" % (self.dpath_tmp,
+ self.dpath1))
+ self.assertFalse("cetacea" in self.output)
+
+ # Now do an actual resurfacing of just one publisher
+ self.pkgsurf("-s %s -r %s -p selachii" % (self.dpath_tmp,
+ self.dpath1))
+ # Check if we see anything about the unspecified publisher
+ # in the output.
+ self.assertFalse("cetacea" in self.output)
+ # Check if we didn't reversion packages of the unspecified
+ # publisher.
+ self.pkgrepo("-s %s list humpback" % (self.dpath_tmp))
+ self.assertTrue("2.0" in self.output)
+
+ def test_3_override_pkgs(self):
+ """Test for correct handling of user specified packages which
+ should not get reversioned."""
+
+ # Copy target repo to tmp repo
+ self.copy_repository(self.dpath2, self.dpath_tmp,
+ { "selachii": "selachii" })
+ # The new repository won't have a catalog, so rebuild it.
+ self.dcs[4].get_repo(auto_create=True).rebuild()
+
+ # Check multiple patterns with globbing
+ self.pkgsurf("-s %s -r %s -c *iger" % (self.dpath_tmp,
+ self.dpath1))
+ self.pkgrepo("-s %s list tiger" % (self.dpath_tmp))
+ self.assertTrue("2.0" in self.output)
+ self.pkgrepo("-s %s list sandtiger" % (self.dpath_tmp))
+ self.assertTrue("2.0" in self.output)
+
+ # Check specific name.
+ self.pkgsurf("-s %s -r %s -c tiger" % (self.dpath_tmp,
+ self.dpath1))
+ self.pkgrepo("-s %s list tiger" % (self.dpath_tmp))
+ self.assertTrue("2.0" in self.output)
+
+
+if __name__ == "__main__":
+ unittest.main()
--- a/src/tests/pkg5unittest.py Thu Aug 08 12:08:13 2013 +0530
+++ b/src/tests/pkg5unittest.py Fri Aug 09 18:02:11 2013 -0700
@@ -2452,6 +2452,12 @@
return self.cmdline_run(cmdline, comment=comment, exit=exit,
su_wrap=su_wrap, env_arg=env_arg, out=out, stderr=stderr)
+ def pkgsurf(self, command, comment="", exit=0, su_wrap=False,
+ env_arg=None, stderr=False, out=False):
+ cmdline = "%s/usr/bin/pkgsurf %s" % (g_proto_area, command)
+ return self.cmdline_run(cmdline, comment=comment, exit=exit,
+ su_wrap=su_wrap, env_arg=env_arg, out=out, stderr=stderr)
+
def pkgsign(self, depot_url, command, exit=0, comment="",
env_arg=None):
args = []
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/util/publish/pkgsurf.py Fri Aug 09 18:02:11 2013 -0700
@@ -0,0 +1,809 @@
+#!/usr/bin/python2.6
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
+#
+
+#
+# pkgsurf - Detailed operation description:
+#
+# After determining the packages present in the target repo, pkgsurf tries to
+# find the associated version of each package in the reference repo. Packages
+# which can't be found in the reference are ignored (not reversioned). Only the
+# latest version of the reference packages are considered.
+# We then compare the target and ref manifest for content changes. Any
+# difference in the manifests' actions is considered a content change unless
+# they differ in:
+# - the pkg FMRI (since this is what we'll adjust anyway)
+# - a set action whose name attribute is specified with -i/--ignore
+# - a signature action (signature will change when FMRI changes)
+# - a depend action (see below)
+#
+# Changes in depend actions are not considered a content change, however,
+# further analysis is required since the package can only be reversioned if the
+# dependency package didn't have a content change and its dependencies didn't
+# have a content change either.
+#
+# For the depend actions it is therefore required to recurse through the whole
+# dependency chain to determine a content change. Only if no package in the
+# chain had a content change the can the manifest be reversioned.
+#
+# Reversioning certain packages will break the inter-dependency integrity of the
+# target repo since certain package versions might not be available any longer.
+# Therefore pkgsurf will go through all depend actions in the repo and, if they
+# point to a reversioned package, adjust them to the correct version.
+# This requires that signature actions in these adjusted packages need to be
+# dropped since the manifest data changed.
+#
+# In the regular case, the new dependency FMRI of a certain package is taken
+# from the associated manifest of the reference version of this package.
+# However, if dependencies got added/removed it might not be found. In this case
+# pkgsurf uses the FMRI of the actual package which got reversioned as the new
+# dependency FMRI.
+#
+# pkgsurf deletes and inserts manifests in place for the target repo. File data
+# does not need to be modified since we only operate on packages with no content
+# change. It runs a catalog rebuild as the last step to regain catalog integrity
+# within the repo.
+
+import getopt
+import gettext
+import locale
+import os
+import shutil
+import sys
+import tempfile
+import traceback
+
+from itertools import repeat
+
+import pkg.actions as actions
+import pkg.client.api_errors as api_errors
+import pkg.client.pkgdefs as pkgdefs
+import pkg.client.progress as progress
+import pkg.client.publisher as publisher
+import pkg.client.transport.transport as transport
+import pkg.fmri as fmri
+import pkg.manifest as manifest
+import pkg.misc as misc
+import pkg.portable as portable
+import pkg.server.repository as sr
+
+from pkg.client import global_settings
+from pkg.misc import emsg, msg, PipeError
+
+PKG_CLIENT_NAME = "pkgsurf"
+
+temp_root = None
+repo_modified = False
+repo_finished = False
+repo_uri = None
+
+def error(text, cmd=None):
+ """Emit an error message prefixed by the command name """
+
+ if cmd:
+ text = "\n%s: %s" % (cmd, text)
+
+ else:
+ text = "\n%s: %s" % (PKG_CLIENT_NAME, text)
+
+
+ # If the message starts with whitespace, assume that it should come
+ # *before* the command-name prefix.
+ text_nows = text.lstrip()
+ ws = text[:len(text) - len(text_nows)]
+
+ # This has to be a constant value as we can't reliably get our actual
+ # program name on all platforms.
+ emsg(ws + text_nows)
+
+def cleanup(no_msg=False):
+ """Remove temporary directories. Print error msg in case operation
+ was not finished."""
+
+ global temp_root
+
+ if repo_modified and not repo_finished and not no_msg:
+ error(_("""
+The target repository has been modified but the operation did not finish
+successfully. It is now in an inconsistent state.
+
+To re-try the operation, run the following commands:
+ /usr/bin/pkgrepo rebuild -s %(repo)s --no-index
+ %(argv)s
+""") % {"repo": repo_uri, "argv": " ".join(sys.argv)})
+
+ if temp_root:
+ shutil.rmtree(temp_root)
+ temp_root = None
+
+def usage(usage_error=None, cmd=None, retcode=pkgdefs.EXIT_BADOPT):
+ """Emit a usage message and optionally prefix it with a more specific
+ error message. Causes program to exit."""
+
+ if usage_error:
+ error(usage_error, cmd=cmd)
+
+ emsg(_("""\
+Usage:
+ pkgsurf -s target_path -r ref_uri [-n] [-p publisher ...] [-i name ...]
+ [-c pattern ...]
+
+Options:
+ -c pattern Treat every package whose FMRI matches 'pattern' as
+ changed and do not reversion it. Can be specified
+ multiple times.
+
+ -i name Ignore set actions with the name field set to 'name' for
+ determination of content change. Can be specified
+ multiple times.
+
+ -n Perform a trial run with no changes made.
+
+ -p publisher Only operate on given publisher. Can be specified
+ multiple times.
+
+ -r ref_uri URI of reference repository.
+
+ -s target_path Path to target repository. Repository should only
+ contain one version of each package. Must be a
+ filesystem-based repository.
+
+ -?/--help Print this message.
+"""))
+
+
+ sys.exit(retcode)
+
+def abort(err=None, retcode=pkgdefs.EXIT_OOPS):
+ """To be called when a fatal error is encountered."""
+
+ if err:
+ # Clear any possible output first.
+ msg("")
+ error(err)
+
+ cleanup()
+ sys.exit(retcode)
+
+def fetch_catalog(src_pub, xport, temp_root):
+ """Fetch the catalog from src_uri."""
+
+ if not src_pub.meta_root:
+ # Create a temporary directory for catalog.
+ cat_dir = tempfile.mkdtemp(dir=temp_root)
+ src_pub.meta_root = cat_dir
+
+ src_pub.transport = xport
+ src_pub.refresh(full_refresh=True, immediate=True)
+
+ return src_pub.catalog
+
+def get_latest(cat):
+ """ Get latest packages (surface) from given catalog.
+ Returns a dict of the form:
+ { pkg-name: pkg-fmri, ... }
+ """
+ matching, ref, unmatched = cat.get_matching_fmris(["*@latest"])
+
+ del ref
+
+ matches = {}
+ for m in matching:
+ matches[m] = matching[m][0]
+
+ return matches
+
+def get_matching_pkgs(cat, patterns):
+ """Get the matching pkg FMRIs from catalog 'cat' based on the input
+ patterns 'patterns'."""
+
+ versions = set()
+ for p in patterns:
+ if "@" in p:
+ versions.add(p)
+
+ if versions:
+ msg = _("Packages specified to not be reversioned cannot "
+ "contain versions:\n\t")
+ msg += "\n\t".join(versions)
+ abort(msg)
+
+ matching, ref, unmatched = cat.get_matching_fmris(patterns)
+
+ if unmatched:
+ msg = _("The specified packages were not found in the "
+ "repository:\n\t")
+ msg += "\n\t".join(unmatched)
+ abort(msg)
+
+ return matching.keys()
+
+def get_manifest(repo, pub, pfmri):
+ """ Retrieve a manifest with FMRI 'pfmri' of publisher 'pub' from
+ repository object 'repo'. """
+
+ path = repo.manifest(pfmri, pub)
+ mani = manifest.Manifest(pfmri)
+ try:
+ mani.set_content(pathname=path)
+ except Exception, e:
+ abort(err=_("Can not open manifest file %(file)s: %(err)s\n"
+ "Please run 'pkgrepo verify -s %(rroot)s' to check the "
+ "integrity of the repository.") \
+ % {"file": path, "err": str(e), "rroot": repo.root})
+ return mani
+
+def get_tracker():
+ try:
+ progresstracker = \
+ progress.FancyUNIXProgressTracker()
+ except progress.ProgressTrackerException:
+ progresstracker = progress.CommandLineProgressTracker()
+ progresstracker.set_major_phase(progresstracker.PHASE_UTILITY)
+ return progresstracker
+
+def subs_undef_fmri_str(fmri_str, latest_ref_pkgs):
+ """ Substitute correct dependency FMRI if no counterpart can be found in
+ the reference manifest. Use the original FMRI in case the current
+ version of dependency pkg in the repo is still a successor of the
+ specified dependency FMRI, otherwise substitute the complete version of
+ the pkg currently present in the repo."""
+
+ dpfmri = fmri.PkgFmri(fmri_str, "5.11")
+ ndpfmri = latest_ref_pkgs[dpfmri.get_name()]
+
+ if ndpfmri.is_successor(dpfmri):
+ return fmri_str
+
+ return ndpfmri.get_short_fmri(anarchy=True)
+
+def get_dep_fmri_str(fmri_str, pkg, act, latest_ref_pkgs, reversioned_pkgs,
+ ref_xport):
+ """Get the adjusted dependency FMRI of package 'pkg' specified in
+ action 'act' based on if the FMRI belongs to a reversioned package or
+ not. 'fmri_str' contains the original FMRI string from the manifest to
+ be adjusted. This has to be passed in separately since in case of
+ require-any dependencies an action can contain multiple FMRIs. """
+
+ dpfmri = fmri.PkgFmri(fmri_str, "5.11")
+
+ # Versionless dependencies don't need to be changed.
+ if not dpfmri.version:
+ return fmri_str
+
+ # Dep package hasn't been changed, no adjustment necessary.
+ if dpfmri.get_pkg_stem() not in reversioned_pkgs:
+ return fmri_str
+
+ # Find the dependency action of the reference package
+ # and replace the current version with it.
+ try:
+ ref_mani = ref_xport.get_manifest(latest_ref_pkgs[pkg])
+ except KeyError:
+ # This package is not in the ref repo so we just substitute the
+ # dependency.
+ return subs_undef_fmri_str(fmri_str, latest_ref_pkgs)
+
+ for ra in ref_mani.gen_actions_by_type("depend"):
+ # Any difference other than the FMRI means we
+ # can't use this action as a reference.
+ diffs = act.differences(ra)
+ if "fmri" in diffs:
+ diffs.remove("fmri")
+ if diffs:
+ continue
+
+ fmris = ra.attrlist("fmri")
+
+ for rf in fmris:
+ rpfmri = fmri.PkgFmri(rf, "5.11")
+ if rpfmri.get_pkg_stem() != dpfmri.get_pkg_stem():
+ continue
+
+ # Only substitute dependency if it actually
+ # changed.
+ if not rpfmri.version \
+ or rpfmri.get_version() != dpfmri.get_version():
+ return rf
+
+ return fmri_str
+
+ # If a varcet changed we might not find the matching action.
+ return subs_undef_fmri_str(fmri_str, latest_ref_pkgs)
+
+def adjust_dep_action(pkg, act, latest_ref_pkgs, reversioned_pkgs, ref_xport):
+ """Adjust dependency FMRIs of action 'act' if it is of type depend.
+ The adjusted action will reference only FMRIs which are present in the
+ reversioned repo. """
+
+ modified = False
+
+ # Drop signatures (changed dependency will void signature value).
+ if act.name == "signature":
+ return
+ # Ignore anything other than depend actions.
+ elif act.name != "depend":
+ return act
+
+ # Require-any deps are list so convert every dep FMRI into a list.
+ fmris = act.attrlist("fmri")
+
+ new_dep = []
+ for f in fmris:
+ new_f = get_dep_fmri_str(f, pkg, act, latest_ref_pkgs,
+ reversioned_pkgs, ref_xport)
+ if not modified and f != new_f:
+ modified = True
+ new_dep.append(new_f)
+
+ if not modified:
+ return act
+
+ if len(new_dep) == 1:
+ new_dep = new_dep[0]
+
+ nact = actions.fromstr(str(act))
+ nact.attrs["fmri"] = new_dep
+
+ return nact
+
+def use_ref(a, deps, ignores):
+ """Determine if the given action indicates that the pkg can be
+ reversioned."""
+
+ if a.name == "set" and "name" in a.attrs:
+ if a.attrs["name"] in ignores:
+ return True
+ # We ignore the pkg FMRI because this is what
+ # will always change.
+ if a.attrs["name"] == "pkg.fmri":
+ return True
+
+ # Signature will always change.
+ if a.name == "signature":
+ return True
+
+ if a.name == "depend":
+ # TODO: support dependency lists
+ # For now, treat as content change.
+ if not isinstance(a.attrs["fmri"], basestring):
+ return False
+ dpfmri = fmri.PkgFmri(a.attrs["fmri"], "5.11")
+ deps.add(dpfmri.get_pkg_stem())
+ return True
+
+ return False
+
+def do_reversion(pub, ref_pub, target_repo, ref_xport, changes, ignores):
+ """Do the repo reversion.
+ Return 'True' if repo got modified, 'False' otherwise."""
+
+ global temp_root, tracker, dry_run, repo_finished, repo_modified
+
+ target_cat = target_repo.get_catalog(pub=pub)
+ ref_cat = fetch_catalog(ref_pub, ref_xport, temp_root)
+
+ latest_pkgs = get_latest(target_cat)
+ latest_ref_pkgs = get_latest(ref_cat)
+
+ no_revs = get_matching_pkgs(target_cat, changes)
+
+ # We use bulk prefetching for faster transport of the manifests.
+ # Prefetch requires an intent which it sends to the server. Here
+ # we just use operation=reversion for all FMRIs.
+ intent = "operation=reversion;"
+ ref_pkgs = zip(latest_ref_pkgs.values(), repeat(intent))
+
+ # Retrieve reference manifests.
+ # Try prefetching manifests in bulk first for faster, parallel
+ # transport. Retryable errors during prefetch are ignored and
+ # manifests are retrieved again during the "Reading" phase.
+ ref_xport.prefetch_manifests(ref_pkgs, progtrack=tracker)
+
+ # Need to change the output of mfst_fetch since otherwise we
+ # would see "Download Manifests x/y" twice, once from the
+ # prefetch and once from the actual manifest analysis.
+ tracker.mfst_fetch = progress.GoalTrackerItem(_("Analyzing Manifests"))
+
+ tracker.manifest_fetch_start(len(latest_pkgs))
+
+ reversioned_pkgs = set()
+ depend_changes = {}
+ dups = 0 # target pkg has equal version to ref pkg
+ new_p = 0 # target pkg not in ref
+ sucs = 0 # ref pkg is successor to pkg in targ
+ nrevs = 0 # pkgs requested to not be reversioned by user
+
+ for p in latest_pkgs:
+ # First check if the package is in the list of FMRIs the user
+ # doesn't want to reversion.
+ if p in no_revs:
+ nrevs += 1
+ tracker.manifest_fetch_progress(completion=True)
+ continue
+
+ # Check if the package is in the ref repo, if not: ignore.
+ if not p in latest_ref_pkgs:
+ new_p += 1
+ tracker.manifest_fetch_progress(completion=True)
+ continue
+
+ pfmri = latest_pkgs[p]
+ # Ignore if latest package is the same in targ and ref.
+ if pfmri == latest_ref_pkgs[p]:
+ dups += 1
+ tracker.manifest_fetch_progress(completion=True)
+ continue
+
+ # Ignore packages where ref version is higher.
+ if latest_ref_pkgs[p].is_successor(pfmri):
+ sucs += 1
+ tracker.manifest_fetch_progress(completion=True)
+ continue
+
+ # Pull the manifests for target and ref repo.
+ dm = get_manifest(target_repo, pub, pfmri)
+ rm = ref_xport.get_manifest(latest_ref_pkgs[p])
+ tracker.manifest_fetch_progress(completion=True)
+
+ tdeps = set()
+ rdeps = set()
+
+ # Diff target and ref manifest.
+ # action only in targ, action only in ref, common action
+ ta, ra, ca = manifest.Manifest.comm([dm, rm])
+
+ # Check for manifest changes.
+ if not all(use_ref(a, tdeps, ignores) for a in ta) \
+ or not all(use_ref(a, rdeps, ignores) for a in ra):
+ continue
+
+ # Both dep lists should be equally long in case deps have just
+ # changed. If not, it means a dep has been added or removed and
+ # that means content change.
+ if len(tdeps) != len(rdeps):
+ continue
+
+ # If len is not different we still have to make sure that
+ # entries have the same pkg stem. The test above just saves time
+ # in some cases.
+ if not all(td in rdeps for td in tdeps):
+ continue
+
+ # Pkg only contains dependency change. Keep for further
+ # analysis.
+ if tdeps:
+ depend_changes[pfmri.get_pkg_stem(
+ anarchy=True)] = tdeps
+ continue
+
+ # Pkg passed all checks and can be reversioned.
+ reversioned_pkgs.add(pfmri.get_pkg_stem(anarchy=True))
+
+ tracker.manifest_fetch_done()
+
+ def has_changed(pstem, seen=None, depth=0):
+ """Determine if a package or any of its dependencies has
+ changed.
+ Function will check if a dependency had a content change. If it
+ only had a dependency change, analyze its dependencies
+ recursively. Only if the whole dependency chain didn't have any
+ content change it is safe to reversion the package.
+
+ Note about circular dependencies: The function keeps track of
+ pkgs it already processed by stuffing them into the set 'seen'.
+ However, 'seen' gets updated before the child dependencies of
+ the current pkg are examined. This works if 'seen' is only used
+ for one dependency chain since the function immediately comes
+ back with a True result if a pkg has changed further down the
+ tree. However, if 'seen' is re-used between runs, it will
+ return prematurely, likely returning wrong results. """
+
+ MAX_DEPTH = 100
+
+ if not seen:
+ seen = set()
+
+ if pstem in seen:
+ return False
+
+ depth += 1
+ if depth > MAX_DEPTH:
+ # Let's make sure we don't run into any
+ # recursion limits. If the dep chain is too deep
+ # just treat as changed pkg.
+ error(_("Dependency chain depth of >%(md)d detected for"
+ " %(p)s." % {"md": MAX_DEPTH, "p": p}))
+ return True
+
+ # Pkg has no change at all.
+ if pstem in reversioned_pkgs:
+ return False
+
+ # Pkg must have content change, if it had no change it would be
+ # in reversioned_pkgs, and if it had just a dep change it would
+ # be in depend_changes.
+ if pstem not in depend_changes:
+ return True
+
+ # We need to update 'seen' here, otherwise we won't find this
+ # entry in case of a circular dependency.
+ seen.add(pstem)
+
+ return any(
+ has_changed(d, seen, depth)
+ for d in depend_changes[pstem]
+ )
+
+ # Check if packages which just have a dep change can be reversioned by
+ # checking if child dependencies also have no content change.
+ dep_revs = 0
+ for p in depend_changes:
+ if not has_changed(p):
+ dep_revs += 1
+ reversioned_pkgs.add(p)
+
+ status = []
+ status.append((_("Packages to process:"), str(len(latest_pkgs))))
+ status.append((_("New packages:"), str(new_p)))
+ status.append((_("Unmodified packages:"), str(dups)))
+ if sucs:
+ # This only happens if reference repo is ahead of target repo,
+ # so only show if it actually happened.
+ status.append((_("Packages with successors in "
+ "reference repo:"), str(sucs)))
+ if nrevs:
+ # This only happens if user specified pkgs to not revert,
+ # so only show if it actually happened.
+ status.append((_("Packages not to be reversioned by user "
+ "request:"), str(nrevs)))
+ status.append((_("Packages with no content change:"),
+ str(len(reversioned_pkgs) - dep_revs)))
+ status.append((_("Packages which only have dependency change:"),
+ str(len(depend_changes))))
+ status.append((_("Packages with unchanged dependency chain:"),
+ str(dep_revs)))
+ status.append((_("Packages to be reversioned:"),
+ str(len(reversioned_pkgs))))
+
+ rjust_status = max(len(s[0]) for s in status)
+ rjust_value = max(len(s[1]) for s in status)
+ for s in status:
+ msg("%s %s" % (s[0].rjust(rjust_status),
+ s[1].rjust(rjust_value)))
+
+ if not reversioned_pkgs:
+ msg(_("\nNo packages to reversion."))
+ return False
+
+ if dry_run:
+ msg(_("\nReversioning packages (dry-run)."))
+ else:
+ msg(_("\nReversioning packages."))
+
+ # Start the main pass. Reversion packages from reversioned_pkgs to the
+ # version in the ref repo. For packages which don't get reversioned,
+ # check if the dependency versions are still correct, fix if necessary.
+ tracker.reversion_start(len(latest_pkgs), len(reversioned_pkgs))
+
+ for p in latest_pkgs:
+ tracker.reversion_add_progress(pfmri, pkgs=1)
+ modified = False
+
+ # Get the pkg fmri (pfmri) of the latest version based on if it
+ # has been reversioned or not.
+ stem = latest_pkgs[p].get_pkg_stem(anarchy=True)
+ if stem in reversioned_pkgs:
+ tracker.reversion_add_progress(pfmri, reversioned=1)
+ if dry_run:
+ continue
+ pfmri = latest_ref_pkgs[p]
+ # Retrieve manifest from ref repo and replace the one in
+ # the target repo. We don't have to adjust depndencies
+ # for these packages because they will not depend on
+ # anything we'll reversion.
+ rmani = ref_xport.get_manifest(pfmri)
+ opath = target_repo.manifest(latest_pkgs[p], pub)
+ os.remove(opath)
+ path = target_repo.manifest(pfmri, pub)
+ try:
+ repo_modified = True
+ repo_finished = False
+ portable.rename(rmani.pathname, path)
+ except OSError, e:
+ abort(err=_("Could not reversion manifest "
+ "%(path)s: %(err)s") % (path, str(e)))
+ continue
+
+ # For packages we don't reversion we have to check if they
+ # depend on a reversioned package.
+ # Since the version of this dependency might be removed from the
+ # repo, we have to adjust the dep version to the one of the
+ # reversioned pkg.
+ pfmri = latest_pkgs[p]
+ omani = get_manifest(target_repo, pub, pfmri)
+ mani = manifest.Manifest(pfmri)
+ for act in omani.gen_actions():
+ nact = adjust_dep_action(p, act, latest_ref_pkgs,
+ reversioned_pkgs, ref_xport)
+ if nact:
+ mani.add_action(nact, misc.EmptyI)
+ if nact is not act:
+ modified = True
+
+ # Only touch manifest if something actually changed.
+ if modified:
+ tracker.reversion_add_progress(pfmri, adjusted=1)
+ if not dry_run:
+ path = target_repo.manifest(pfmri, pub)
+ repo_modified = True
+ repo_finished = False
+ mani.store(path)
+ tracker.reversion_done()
+
+ return True
+
+def main_func():
+
+ global temp_root, repo_modified, repo_finished, repo_uri, tracker
+ global dry_run
+
+ misc.setlocale(locale.LC_ALL, "", error)
+ gettext.install("pkg", "/usr/share/locale",
+ codeset=locale.getpreferredencoding())
+ global_settings.client_name = PKG_CLIENT_NAME
+
+ try:
+ opts, pargs = getopt.getopt(sys.argv[1:], "?c:i:np:r:s:",
+ ["help"])
+ except getopt.GetoptError, e:
+ usage(_("illegal option -- %s") % e.opt)
+
+ dry_run = False
+ ref_repo_uri = None
+ repo_uri = os.getenv("PKG_REPO", None)
+ changes = set()
+ ignores = set()
+ publishers = set()
+
+ processed_pubs = 0
+
+ for opt, arg in opts:
+ if opt == "-c":
+ changes.add(arg)
+ elif opt == "-i":
+ ignores.add(arg)
+ elif opt == "-n":
+ dry_run = True
+ elif opt == "-p":
+ publishers.add(arg)
+ elif opt == "-r":
+ ref_repo_uri = misc.parse_uri(arg)
+ elif opt == "-s":
+ repo_uri = misc.parse_uri(arg)
+ elif opt == "-?" or opt == "--help":
+ usage(retcode=pkgdefs.EXIT_OK)
+
+ if pargs:
+ usage(_("Unexpected argument(s): %s") % " ".join(pargs))
+
+ if not repo_uri:
+ usage(_("A target repository must be provided."))
+
+ if not ref_repo_uri:
+ usage(_("A reference repository must be provided."))
+
+ t = misc.config_temp_root()
+ temp_root = tempfile.mkdtemp(dir=t,
+ prefix=global_settings.client_name + "-")
+
+ ref_incoming_dir = tempfile.mkdtemp(dir=temp_root)
+ ref_pkg_root = tempfile.mkdtemp(dir=temp_root)
+
+ ref_xport, ref_xport_cfg = transport.setup_transport()
+ ref_xport_cfg.incoming_root = ref_incoming_dir
+ ref_xport_cfg.pkg_root = ref_pkg_root
+ transport.setup_publisher(ref_repo_uri, "ref", ref_xport,
+ ref_xport_cfg, remote_prefix=True)
+
+ target = publisher.RepositoryURI(misc.parse_uri(repo_uri))
+ if target.scheme != "file":
+ abort(err=_("Target repository must be filesystem-based."))
+ try:
+ target_repo = sr.Repository(read_only=dry_run,
+ root=target.get_pathname())
+ except sr.RepositoryError, e:
+ abort(str(e))
+
+ tracker = get_tracker()
+
+ for pub in target_repo.publishers:
+ if publishers and pub not in publishers \
+ and '*' not in publishers:
+ continue
+
+ msg(_("Processing packages for publisher %s ...") % pub)
+ # Find the matching pub in the ref repo.
+ for ref_pub in ref_xport_cfg.gen_publishers():
+ if ref_pub.prefix == pub:
+ found = True
+ break
+ else:
+ txt = _("Publisher %s not found in reference "
+ "repository.") % pub
+ if publishers:
+ abort(err=txt)
+ else:
+ txt += _(" Skipping.")
+ msg(txt)
+ continue
+
+ processed_pubs += 1
+
+ rev = do_reversion(pub, ref_pub, target_repo, ref_xport,
+ changes, ignores)
+
+ # Only rebuild catalog if anything got actually reversioned.
+ if rev and not dry_run:
+ msg(_("Rebuilding repository catalog."))
+ target_repo.rebuild(pub=pub)
+ repo_finished = True
+
+ ret = pkgdefs.EXIT_OK
+ if processed_pubs == 0:
+ msg(_("No matching publishers could be found."))
+ ret = pkgdefs.EXIT_OOPS
+ cleanup()
+ return ret
+
+
+#
+# Establish a specific exit status which means: "python barfed an exception"
+# so that we can more easily detect these in testing of the CLI commands.
+#
+if __name__ == "__main__":
+ try:
+ __ret = main_func()
+ except PipeError:
+ # We don't want to display any messages here to prevent
+ # possible further broken pipe (EPIPE) errors.
+ cleanup(no_msg =True)
+ __ret = pkgdefs.EXIT_OOPS
+ except (KeyboardInterrupt, api_errors.CanceledException):
+ cleanup()
+ __ret = pkgdefs.EXIT_OOPS
+ except (actions.ActionError, RuntimeError,
+ api_errors.ApiException), _e:
+ error(_e)
+ cleanup()
+ __ret = pkgdefs.EXIT_OOPS
+ except SystemExit, _e:
+ cleanup()
+ raise _e
+ except:
+ traceback.print_exc()
+ error(misc.get_traceback_message())
+ __ret = 99
+ sys.exit(__ret)