tools/userland-fetch
author mmhunter<melynda.hunter@oracle.com>
Mon, 09 Apr 2012 15:13:23 -0700
branchs11-sru
changeset 2266 8b5c3a141a7c
parent 212 8a66c0c0e8dc
child 800 2ad056ed89ec
permissions -rwxr-xr-x
Added tag S11SRU4_05 for changeset 4307245757f6
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     1
#!/usr/bin/python2.6
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     2
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     3
# CDDL HEADER START
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     4
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     5
# The contents of this file are subject to the terms of the
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     6
# Common Development and Distribution License (the "License").
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     7
# You may not use this file except in compliance with the License.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     8
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     9
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    10
# or http://www.opensolaris.org/os/licensing.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    11
# See the License for the specific language governing permissions
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    12
# and limitations under the License.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    13
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    14
# When distributing Covered Code, include this CDDL HEADER in each
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    15
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    16
# If applicable, add the following below this CDDL HEADER, with the
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    17
# fields enclosed by brackets "[]" replaced with your own identifying
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    18
# information: Portions Copyright [yyyy] [name of copyright owner]
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    19
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    20
# CDDL HEADER END
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    21
#
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
    22
# Copyright (c) 2010, 2011, Oracle and/or its affiliates. All rights reserved.
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    23
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    24
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    25
# fetch.py - a file download utility
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    26
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    27
#  A simple program similiar to wget(1), but handles local file copy, ignores
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    28
#  directories, and verifies file hashes.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    29
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    30
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    31
import os
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    32
import sys
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    33
import shutil
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    34
from urllib import splittype, urlopen
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    35
import hashlib
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    36
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    37
def printIOError(e, txt):
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    38
	""" Function to decode and print IOError type exception """
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    39
	print "I/O Error: " + txt + ": "
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    40
	try:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    41
		(code, message) = e
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    42
		print str(message) + " (" + str(code) + ")"
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    43
	except:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    44
		print str(e)
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    45
	
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    46
def validate(file, hash):
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
    47
	algorithm, hashvalue = hash.split(':')
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    48
	try:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    49
		m = hashlib.new(algorithm)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    50
	except ValueError:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    51
		return False
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    52
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    53
	while True:
137
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    54
		try:
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    55
			block = file.read()
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    56
		except IOError, err:
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    57
			print str(err),
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    58
			break
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    59
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    60
		m.update(block)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    61
		if block == '':
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    62
			break
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    63
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
    64
	return "%s:%s" % (algorithm, m.hexdigest())
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    65
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    66
def validate_container(filename, hash):
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    67
	try:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    68
		file = open(filename, 'r')
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    69
	except IOError as e:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    70
		printIOError(e, "Can't open file " + filename)
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    71
		return False
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    72
	return validate(file, hash)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    73
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    74
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    75
def validate_payload(filename, hash):
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    76
	import re
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    77
	import gzip
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    78
	import bz2
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    79
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    80
	expr_bz = re.compile('.+\.bz2$', re.IGNORECASE)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    81
	expr_gz = re.compile('.+\.gz$', re.IGNORECASE)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    82
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    83
	try:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    84
		if expr_bz.match(filename):
137
6fb20db3eee4 7028115 userland-fetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 136
diff changeset
    85
			file = bz2.BZ2File(filename, 'r')
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    86
		elif expr_gz.match(filename):
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    87
			file = gzip.GzipFile(filename, 'r')
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    88
		else:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    89
			return False
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    90
	except IOError as e:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
    91
		printIOError(e, "Can't open archive " + filename)
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    92
		return False
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    93
	return validate(file, hash)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    94
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
    95
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    96
def download(url, filename = None):
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    97
	src = None
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    98
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    99
	try:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   100
		src = urlopen(url)
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   101
	except IOError as e:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   102
		printIOError(e, "Can't open url " + url)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   103
		return None
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   104
212
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   105
	# 3xx, 4xx and 5xx (f|ht)tp codes designate unsuccessfull action
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   106
	if 3 <= int(src.getcode()/100) <= 5:
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   107
		print "Error code: " + str(src.getcode())
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   108
		return None
8a66c0c0e8dc 7041047 userland-fetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 137
diff changeset
   109
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   110
	if filename == None:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   111
		filename = src.geturl().split('/')[-1]
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   112
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   113
	try:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   114
		dst = open(filename, 'wb');
136
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   115
	except IOError as e:
6a7dd3a5aaf9 7027131 userland-fetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents: 135
diff changeset
   116
		printIOError(e, "Can't open file " + filename + " for writing")
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   117
		src.close()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   118
		return None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   119
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   120
	while True:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   121
		block = src.read()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   122
		if block == '':
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   123
			break;
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   124
		dst.write(block)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   125
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   126
	src.close()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   127
	dst.close()
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   128
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   129
	# return the name of the file that we downloaded the data to.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   130
	return filename
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   131
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   132
def download_paths(search, filename, url):
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   133
	urls = list()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   134
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   135
	if filename != None:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   136
		tmp = os.getenv('DOWNLOAD_SEARCH_PATH')
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   137
		if tmp:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   138
			search += tmp.split(' ')
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   139
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   140
		file = os.path.basename(filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   141
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   142
		urls = [ base + '/' + file for base in search ]
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   143
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   144
		# filename should always be first
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   145
		if filename in urls:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   146
			urls.remove(filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   147
		urls.insert(0, filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   148
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   149
	# command line url is a fallback, so it's last
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   150
	if url != None and url not in urls:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   151
		urls.append(url)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   152
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   153
	return urls
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   154
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   155
def usage():
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   156
	print "Usage: %s [-f|--file (file)] [-l|--link] [-h|--hash (hash)] [-s|--search (search-dir)] --url (url)" % (sys.argv[0].split('/')[-1])
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   157
	sys.exit(1)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   158
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   159
def main():
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   160
	import getopt
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   161
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   162
	# FLUSH STDOUT 
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   163
	sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   164
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   165
	file_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   166
	link_arg = False
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   167
	hash_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   168
	url_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   169
	search_list = list()
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   170
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   171
	try:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   172
		opts, args = getopt.getopt(sys.argv[1:], "f:h:ls:u:",
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   173
			["file=", "link", "hash=", "search=", "url="])
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   174
	except getopt.GetoptError, err:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   175
		print str(err)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   176
		usage()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   177
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   178
	for opt, arg in opts:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   179
		if opt in [ "-f", "--file" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   180
			file_arg = arg
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   181
		elif opt in [ "-l", "--link" ]:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   182
			link_arg = True
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   183
		elif opt in [ "-h", "--hash" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   184
			hash_arg = arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   185
		elif opt in [ "-s", "--search" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   186
			search_list.append(arg)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   187
		elif opt in [ "-u", "--url" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   188
			url_arg = arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   189
		else:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   190
			assert False, "unknown option"
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   191
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   192
	if url_arg == None:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   193
		usage()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   194
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   195
	for url in download_paths(search_list, file_arg, url_arg):
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   196
		print "Source %s..." % url,
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   197
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   198
		scheme, path = splittype(url)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   199
		name = file_arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   200
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   201
		if scheme in [ None, 'file' ]:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   202
			if os.path.exists(path) == False:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   203
				print "not found, skipping file copy"
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   204
				continue
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   205
			elif name != path:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   206
				if link_arg == False:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   207
					print "\n    copying..."
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   208
					shutil.copy2(path, name)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   209
				else:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   210
					print "\n    linking..."
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   211
					os.symlink(path, name)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   212
			else:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   213
				pass
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   214
		elif scheme in [ 'http', 'https', 'ftp' ]:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   215
			print "\n    downloading...",
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   216
			name = download(url, file_arg)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   217
			if name == None:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   218
				print "failed"
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   219
				continue
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   220
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   221
		print "\n    validating...",
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   222
		if hash_arg == None:
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   223
			print "skipping (no hash)"
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   224
			sys.exit(0)
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   225
			
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   226
		realhash = validate_container(name, hash_arg)
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   227
		if realhash == hash_arg:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   228
			print "ok"
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   229
			sys.exit(0)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   230
		else:
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   231
			payloadhash = validate_payload(name, hash_arg)
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   232
			if payloadhash == hash_arg:
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   233
				print "ok"
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   234
				sys.exit(0)
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   235
			print "corruption detected"
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   236
			print "    expected: %s" % hash_arg
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   237
			print "    actual:   %s" % realhash
135
5c6feb0253fa 7028115 userland-fetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents: 58
diff changeset
   238
			print "    payload:  %s" % payloadhash
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   239
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   240
		try:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   241
			os.remove(name)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   242
		except OSError:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   243
			pass
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   244
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   245
	sys.exit(1)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   246
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   247
if __name__ == "__main__":
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   248
	main()