tools/userland-fetch
author Mike Sullivan <Mike.Sullivan@Oracle.COM>
Thu, 16 Jun 2016 20:28:32 -0700
changeset 6235 309c116f1e44
parent 5953 de8c30ca3eaa
child 6950 b14572a6af9b
permissions -rwxr-xr-x
Close of build 102.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
3770
ca450a806cc1 20440888 Userland python tools should migrate to 2.7
John Beck <John.Beck@Oracle.COM>
parents: 3533
diff changeset
     1
#!/usr/bin/python2.7
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     2
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     3
# CDDL HEADER START
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     4
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     5
# The contents of this file are subject to the terms of the
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     6
# Common Development and Distribution License (the "License").
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     7
# You may not use this file except in compliance with the License.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     8
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     9
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    10
# or http://www.opensolaris.org/os/licensing.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    11
# See the License for the specific language governing permissions
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    12
# and limitations under the License.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    13
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    14
# When distributing Covered Code, include this CDDL HEADER in each
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    15
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    16
# If applicable, add the following below this CDDL HEADER, with the
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    17
# fields enclosed by brackets "[]" replaced with your own identifying
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    18
# information: Portions Copyright [yyyy] [name of copyright owner]
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    19
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    20
# CDDL HEADER END
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    21
#
5682
94c0ca64c022 15558602 TCL_LD_SEARCH_FLAGS is wrongly defined in tclConfig.sh
Shawn Walker-Salas <shawn.walker@oracle.com>
parents: 5244
diff changeset
    22
94c0ca64c022 15558602 TCL_LD_SEARCH_FLAGS is wrongly defined in tclConfig.sh
Shawn Walker-Salas <shawn.walker@oracle.com>
parents: 5244
diff changeset
    23
#
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
    24
# Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved.
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    25
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    26
#
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    27
# userland-fetch - a file download utility
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    28
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    29
#  A simple program similiar to wget(1), but handles local file copy, ignores
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    30
#  directories, and verifies file hashes.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    31
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    32
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    33
import errno
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    34
import os
5862
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
    35
import re
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    36
import sys
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    37
import shutil
5862
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
    38
import json
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    39
import subprocess
5862
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
    40
from urllib import splittype, splithost
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
    41
from urllib2 import urlopen, HTTPError
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    42
import hashlib
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    43
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    44
def printIOError(e, txt):
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    45
	""" Function to decode and print IOError type exception """
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    46
	print "I/O Error: " + txt + ": "
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    47
	try:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    48
		(code, message) = e
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    49
		print str(message) + " (" + str(code) + ")"
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    50
	except:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    51
		print str(e)
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    52
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    53
def validate_signature(path, signature):
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    54
	"""Given paths to a file and a detached PGP signature, verify that
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    55
	the signature is valid for the file.  Current configuration allows for
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    56
	unrecognized keys to be downloaded as necessary."""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    57
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    58
	# Find the root of the repo so that we can point GnuPG at the right
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    59
	# configuration and keyring.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    60
	proc = subprocess.Popen(["hg", "root"], stdout=subprocess.PIPE)
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    61
	proc.wait()
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    62
	if proc.returncode != 0:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    63
		return False
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    64
	out, err = proc.communicate()
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    65
	gpgdir = os.path.join(out.strip(), "tools", ".gnupg")
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    66
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    67
        # Skip the permissions warning: none of the information here is private,
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    68
        # so not having to worry about getting mercurial keeping the directory
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    69
        # unreadable is just simplest.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    70
	try:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    71
		proc = subprocess.Popen(["gpg2", "--verify",
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    72
		    "--no-permission-warning", "--homedir", gpgdir, signature,
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    73
		    path], stdin=open("/dev/null"),
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    74
		    stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    75
	except OSError as e:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    76
		# If the executable simply couldn't be found, just skip the
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    77
		# validation.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    78
		if e.errno == errno.ENOENT:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    79
			return False
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    80
		raise
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    81
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    82
        proc.wait()
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    83
        if proc.returncode != 0:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    84
		# Only print GnuPG's output when there was a problem.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    85
                print proc.stdout.read()
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    86
                return False
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    87
        return True
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
    88
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    89
def validate(file, hash):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    90
	"""Given a file-like object and a hash string, verify that the hash
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    91
	matches the file contents."""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    92
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    93
	try:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    94
		algorithm, hashvalue = hash.split(':')
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    95
	except:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
    96
		algorithm = "sha256"
800
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
    97
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
    98
	# force migration away from sha1
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
    99
	if algorithm == "sha1":
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
   100
		algorithm = "sha256"
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   101
	try:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   102
		m = hashlib.new(algorithm)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   103
	except ValueError:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   104
		return False
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   105
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   106
	while True:
137
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   107
		try:
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   108
			block = file.read()
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   109
		except IOError, err:
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   110
			print str(err),
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   111
			break
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   112
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   113
		m.update(block)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   114
		if block == '':
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   115
			break
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   116
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   117
	return "%s:%s" % (algorithm, m.hexdigest())
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   118
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   119
def validate_container(filename, hash):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   120
	"""Given a file path and a hash string, verify that the hash matches the
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   121
	file contents."""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   122
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   123
	try:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   124
		file = open(filename, 'r')
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   125
	except IOError as e:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   126
		printIOError(e, "Can't open file " + filename)
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   127
		return False
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   128
	return validate(file, hash)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   129
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   130
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   131
def validate_payload(filename, hash):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   132
	"""Given a file path and a hash string, verify that the hash matches the
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   133
	payload (uncompressed content) of the file."""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   134
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   135
	import gzip
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   136
	import bz2
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   137
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   138
	expr_bz = re.compile('.+\.bz2$', re.IGNORECASE)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   139
	expr_gz = re.compile('.+\.gz$', re.IGNORECASE)
800
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
   140
	expr_tgz = re.compile('.+\.tgz$', re.IGNORECASE)
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   141
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   142
	try:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   143
		if expr_bz.match(filename):
137
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
   144
			file = bz2.BZ2File(filename, 'r')
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   145
		elif expr_gz.match(filename):
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   146
			file = gzip.GzipFile(filename, 'r')
800
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
   147
		elif expr_tgz.match(filename):
2ad056ed89ec 7163771 sha1 should not be used in userland-fetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents: 212
diff changeset
   148
			file = gzip.GzipFile(filename, 'r')
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   149
		else:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   150
			return False
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   151
	except IOError as e:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   152
		printIOError(e, "Can't open archive " + filename)
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   153
		return False
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   154
	return validate(file, hash)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   155
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   156
def download(url, timeout, filename=None, quiet=False):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   157
	"""Download the content at the given URL to the given filename
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   158
	(defaulting to the basename of the URL if not given.  If 'quiet' is
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   159
	True, throw away any error messages.  Returns the name of the file to
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   160
	which the content was donloaded."""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   161
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   162
	src = None
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   163
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   164
	try:
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   165
		src = urlopen(url=url, timeout=timeout)
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   166
	except IOError as e:
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   167
		if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   168
			printIOError(e, "Can't open url " + url)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   169
		return None
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   170
212
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   171
	# 3xx, 4xx and 5xx (f|ht)tp codes designate unsuccessfull action
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   172
	if 3 <= int(src.getcode()/100) <= 5:
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   173
		if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   174
			print "Error code: " + str(src.getcode())
212
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   175
		return None
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   176
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   177
	if filename == None:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   178
		filename = src.geturl().split('/')[-1]
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   179
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   180
	try:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   181
		dst = open(filename, 'wb');
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   182
	except IOError as e:
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   183
		if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   184
			printIOError(e, "Can't open file " + filename + " for writing")
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   185
		src.close()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   186
		return None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   187
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   188
	while True:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   189
		block = src.read()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   190
		if block == '':
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   191
			break;
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   192
		dst.write(block)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   193
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   194
	src.close()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   195
	dst.close()
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   196
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   197
	# return the name of the file that we downloaded the data to.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   198
	return filename
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   199
5862
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   200
def pypi_url(url, filename):
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   201
	"""Given a pypi: URL, return the real URL for that component/version.
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   202
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   203
	The pypi scheme has a host (with an empty host defaulting to
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   204
	pypi.python.org), and a path that should be of the form
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   205
	"component==version".  Other specs could be supported, but == is the
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   206
	only thing that makes sense in this context.
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   207
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   208
	The filename argument is the name of the expected file to download, so
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   209
	that when pypi gives us multiple archives to choose from, we can pick
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   210
	the right one.
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   211
	"""
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   212
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   213
	host, path = splithost(splittype(url)[1])
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   214
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   215
	# We have to use ==; anything fancier would require pkg_resources, but
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   216
	# really that's the only thing that makes sense in this context.
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   217
	try:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   218
		name, version = re.match("/(.*)==(.*)$", path).groups()
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   219
	except AttributeError:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   220
		print "PyPI URLs must be of the form 'pypi:///component==version'"
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   221
		return None
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   222
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   223
	if not host:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   224
		jsurl = "http://pypi.python.org/pypi/%s/json" % name
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   225
	else:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   226
		jsurl = "http://%s/pypi/%s/json" % (host, name)
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   227
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   228
	try:
5953
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   229
		# Don't wait very long for the connection
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   230
		f = urlopen(jsurl, None, 2)
5862
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   231
	except HTTPError as e:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   232
		if e.getcode() == 404:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   233
			print "Unknown component '%s'" % name
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   234
		else:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   235
			printIOError(e, "Can't open PyPI JSON url %s" % url)
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   236
		return None
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   237
	except IOError as e:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   238
		printIOError(e, "Can't open PyPI JSON url %s" % url)
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   239
		return None
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   240
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   241
	js = json.load(f)
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   242
	try:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   243
		verblock = js["releases"][version]
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   244
	except KeyError:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   245
		print "Unknown version '%s'" % version
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   246
		return None
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   247
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   248
	urls = [ d["url"] for d in verblock ]
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   249
	for archiveurl in urls:
5952
0ff1d1b8de45 23254586 source-download job doesn't work on pypi things anymore
Danek Duvall <danek.duvall@oracle.com>
parents: 5862
diff changeset
   250
		if archiveurl.endswith("/%s" % os.path.basename(filename)):
5862
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   251
			return archiveurl
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   252
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   253
	if urls:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   254
		print "None of the following URLs delivers '%s':" % filename
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   255
		print "  " + "\n  ".join(urls)
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   256
	else:
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   257
		print "Couldn't find any suitable URLs"
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   258
	return None
ce31ce66da02 23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents: 5682
diff changeset
   259
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   260
def download_paths(search, filename, url):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   261
	"""Returns a list of URLs where the file 'filename' might be found,
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   262
	using 'url', 'search', and $DOWNLOAD_SEARCH_PATH as places to look.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   263
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   264
	If 'filename' is None, then the list will simply contain 'url'.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   265
	"""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   266
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   267
	urls = list()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   268
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   269
	if filename != None:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   270
		tmp = os.getenv('DOWNLOAD_SEARCH_PATH')
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   271
		if tmp:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   272
			search += tmp.split(' ')
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   273
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   274
		file = os.path.basename(filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   275
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   276
		urls = [ base + '/' + file for base in search ]
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   277
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   278
		# filename should always be first
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   279
		if filename in urls:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   280
			urls.remove(filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   281
		urls.insert(0, filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   282
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   283
	# command line url is a fallback, so it's last
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   284
	if url != None and url not in urls:
5953
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   285
		urls.append(url)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   286
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   287
	return urls
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   288
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   289
def download_from_paths(search_list, file_arg, url, timeout_arg, link_arg, quiet=False):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   290
	"""Attempts to download a file from a number of possible locations.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   291
	Generates a list of paths where the file ends up on the local
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   292
	filesystem.  This is a generator because while a download might be
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   293
	successful, the signature or hash may not validate, and the caller may
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   294
	want to try again from the next location.  The 'link_arg' argument is a
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   295
	boolean which, when True, specifies that if the source is not a remote
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   296
	URL and not already found where it should be, to make a symlink to the
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   297
	source rather than copying it.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   298
	"""
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   299
	for url in download_paths(search_list, file_arg, url):
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   300
		if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   301
			print "Source %s..." % url,
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   302
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   303
		scheme, path = splittype(url)
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   304
		name = file_arg
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   305
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   306
		if scheme in [ None, 'file' ]:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   307
			if os.path.exists(path) == False:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   308
				if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   309
					print "not found, skipping file copy"
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   310
				continue
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   311
			elif name and name != path:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   312
				if link_arg == False:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   313
					if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   314
						print "\n    copying..."
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   315
					shutil.copy2(path, name)
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   316
				else:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   317
					if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   318
						print "\n    linking..."
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   319
					os.symlink(path, name)
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   320
		elif scheme in [ 'http', 'https', 'ftp' ]:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   321
			if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   322
				print "\n    downloading...",
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   323
			name = download(url, timeout_arg, file_arg, quiet)
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   324
			if name == None:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   325
				if not quiet:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   326
					print "failed"
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   327
				continue
5953
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   328
		elif scheme == "pypi":
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   329
			nurl = pypi_url(url, file_arg)
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   330
			if nurl:
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   331
				if not quiet:
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   332
					print "\n    translated %s to %s..." % (
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   333
					    url, nurl),
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   334
					print "\n    downloading...",
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   335
			else:
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   336
				if not quiet:
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   337
					print "\n    unable to contact PyPI",
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   338
				continue
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   339
			name = download(nurl, timeout_arg, file_arg, quiet)
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   340
			if name is None:
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   341
				if not quiet:
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   342
					print "failed"
de8c30ca3eaa 23261178 userland-fetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents: 5952
diff changeset
   343
				continue
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   344
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   345
		yield name
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   346
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   347
def usage():
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   348
	print "Usage: %s [-f|--file (file)] [-l|--link] [-h|--hash (hash)] " \
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   349
          "[-s|--search (search-dir)] [-S|--sigurl (signature-url)] " \
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   350
	  "[-t|--timeout (timeout)] --url (url)" % \
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   351
          (sys.argv[0].split('/')[-1])
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   352
	sys.exit(1)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   353
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   354
def main():
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   355
	import getopt
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   356
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   357
	# FLUSH STDOUT
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   358
	sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   359
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   360
	file_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   361
	link_arg = False
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   362
	hash_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   363
	url_arg = None
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   364
	sig_arg = None
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   365
	timeout_arg = 300
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   366
	search_list = list()
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   367
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   368
	try:
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   369
                opts, args = getopt.getopt(sys.argv[1:], "f:h:ls:S:t:u:",
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   370
			["file=", "link", "hash=", "search=", "sigurl=",
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   371
			"timeout=", "url="])
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   372
	except getopt.GetoptError, err:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   373
		print str(err)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   374
		usage()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   375
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   376
	for opt, arg in opts:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   377
		if opt in [ "-f", "--file" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   378
			file_arg = arg
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   379
		elif opt in [ "-l", "--link" ]:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   380
			link_arg = True
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   381
		elif opt in [ "-h", "--hash" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   382
			hash_arg = arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   383
		elif opt in [ "-s", "--search" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   384
			search_list.append(arg)
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   385
		elif opt in [ "-S", "--sigurl" ]:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   386
			sig_arg = arg
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   387
		elif opt in [ "-t", "--timeout" ]:
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   388
			try:
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   389
				timeout_arg = int(arg)
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   390
			except ValueError:
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   391
				print "Invalid argument for %s, should be a " \
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   392
				    "number, but is %s" % (opt, arg)
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   393
				sys.exit(1)
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   394
			if timeout_arg < 0:
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   395
				print "Invalid argument for %s, should be a " \
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   396
				    "positive number, but is %s" % (opt, arg)
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   397
				sys.exit(1)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   398
		elif opt in [ "-u", "--url" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   399
			url_arg = arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   400
		else:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   401
			assert False, "unknown option"
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   402
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   403
	for name in download_from_paths(search_list, file_arg, url_arg,
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   404
	    timeout_arg, link_arg):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   405
		print "\n    validating signature...",
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   406
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   407
		sig_valid = False
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   408
		if not sig_arg:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   409
			print "skipping (no signature URL)"
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   410
		else:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   411
			# Put the signature file in the same directory as the
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   412
			# file we're downloading.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   413
			sig_file = os.path.join(
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   414
			    os.path.dirname(file_arg),
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   415
			    os.path.basename(sig_arg))
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   416
			# Validate with the first signature we find.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   417
			for sig_file in download_from_paths(search_list, sig_file,
5244
c63fce595432 21888443 userland-fetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents: 4429
diff changeset
   418
			    sig_arg, timeout_arg, link_arg, True):
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   419
				if sig_file:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   420
					if validate_signature(name, sig_file):
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   421
						print "ok"
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   422
						sig_valid = True
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   423
					else:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   424
						print "failed"
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   425
					break
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   426
				else:
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   427
					continue
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   428
			else:
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   429
				print "failed (couldn't fetch signature)"
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   430
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   431
		print "    validating hash...",
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   432
		realhash = validate_container(name, hash_arg)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   433
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   434
		if not hash_arg:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   435
			print "skipping (no hash)"
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   436
			print "hash is: %s" % realhash
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   437
		elif realhash == hash_arg:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   438
			print "ok"
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   439
		else:
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   440
			payloadhash = validate_payload(name, hash_arg)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   441
			if payloadhash == hash_arg:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   442
				print "ok"
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   443
			else:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   444
				# If the signature validated, then we assume
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   445
				# that the expected hash is just a typo, but we
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   446
				# warn just in case.
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   447
				if sig_valid:
4429
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   448
					print "invalid hash! Did you forget " \
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   449
					    "to update it?"
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   450
				else:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   451
					print "corruption detected"
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   452
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   453
				print "    expected: %s" % hash_arg
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   454
				print "    actual:   %s" % realhash
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   455
				print "    payload:  %s" % payloadhash
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   456
4429
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   457
				# If the hash is invalid, but the signature
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   458
				# validation succeeded, rename the archive (so
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   459
				# the user doesn't have to re-download it) and
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   460
				# fail.  Otherwise, try to remove the file and
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   461
				# try again.
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   462
				if sig_valid:
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   463
					newname = name + ".invalid-hash"
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   464
					try:
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   465
						os.rename(name, newname)
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   466
					except OSError:
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   467
						pass
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   468
					else:
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   469
						print "archive saved as %s; " \
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   470
						    "if it isn't corrupt, " \
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   471
						    "rename to %s" % (newname,
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   472
						    name)
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   473
					sys.exit(1)
a6c5fd1cbbc9 21132783 userland-fetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents: 4196
diff changeset
   474
				else:
3533
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   475
					try:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   476
						os.remove(name)
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   477
					except OSError:
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   478
						pass
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   479
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   480
					continue
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   481
0b8107a40da7 20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents: 832
diff changeset
   482
		sys.exit(0)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   483
	sys.exit(1)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   484
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   485
if __name__ == "__main__":
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   486
	main()