tools/userland-fetch
author Norm Jacobs <Norm.Jacobs@Oracle.COM>
Wed, 23 Feb 2011 10:37:11 -0800
changeset 99 c15c9099bb44
parent 58 d8024c042a00
child 135 5c6feb0253fa
permissions -rwxr-xr-x
6841644 OpenSolaris Python should support gdbm 6991811 Migrate Mercurial (Hg) to Python 2.6 7009592 PyOpenSSL should be updated from 0.8 to 0.11 7016735 move mercurial from sfw to userland 7018363 Userland LD_OPTIONS should include non-executable data mapfiles as well 7019410 python 2.6 should move to userland 7020772 userland build should provide more macros for common flags 7021204 userland runpath checking should relax slightly.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     1
#!/usr/bin/python2.6
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     2
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     3
# CDDL HEADER START
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     4
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     5
# The contents of this file are subject to the terms of the
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     6
# Common Development and Distribution License (the "License").
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     7
# You may not use this file except in compliance with the License.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     8
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
     9
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    10
# or http://www.opensolaris.org/os/licensing.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    11
# See the License for the specific language governing permissions
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    12
# and limitations under the License.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    13
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    14
# When distributing Covered Code, include this CDDL HEADER in each
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    15
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    16
# If applicable, add the following below this CDDL HEADER, with the
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    17
# fields enclosed by brackets "[]" replaced with your own identifying
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    18
# information: Portions Copyright [yyyy] [name of copyright owner]
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    19
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    20
# CDDL HEADER END
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    21
#
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
    22
# Copyright (c) 2010, 2011, Oracle and/or its affiliates. All rights reserved.
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    23
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    24
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    25
# fetch.py - a file download utility
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    26
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    27
#  A simple program similiar to wget(1), but handles local file copy, ignores
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    28
#  directories, and verifies file hashes.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    29
#
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    30
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    31
import os
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    32
import sys
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    33
import shutil
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    34
from urllib import splittype, urlopen
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    35
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    36
def validate(filename, hash):
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    37
	import hashlib
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    38
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    39
	try:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    40
		file = open(filename, 'r')
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    41
	except IOError:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    42
		return False
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    43
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
    44
	algorithm, hashvalue = hash.split(':')
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    45
	try:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    46
		m = hashlib.new(algorithm)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    47
	except ValueError:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    48
		return False
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    49
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    50
	while True:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    51
		block = file.read()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    52
		m.update(block)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    53
		if block == '':
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    54
			break
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    55
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
    56
	return "%s:%s" % (algorithm, m.hexdigest())
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    57
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    58
def download(url, filename = None):
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    59
	src = None
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    60
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    61
	try:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    62
		src = urlopen(url)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    63
	except IOError:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    64
		return None
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    65
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    66
	if filename == None:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    67
		filename = src.geturl().split('/')[-1]
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    68
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    69
	try:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    70
		dst = open(filename, 'wb');
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    71
	except IOError:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    72
		src.close()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    73
		return None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    74
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    75
	while True:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    76
		block = src.read()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    77
		if block == '':
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    78
			break;
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    79
		dst.write(block)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    80
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    81
	src.close()
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    82
	dst.close()
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    83
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    84
	# return the name of the file that we downloaded the data to.
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    85
	return filename
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    86
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    87
def download_paths(search, filename, url):
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    88
	urls = list()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
    89
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    90
	if filename != None:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    91
		tmp = os.getenv('DOWNLOAD_SEARCH_PATH')
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    92
		if tmp:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    93
			search += tmp.split(' ')
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    94
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    95
		file = os.path.basename(filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    96
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    97
		urls = [ base + '/' + file for base in search ]
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    98
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
    99
		# filename should always be first
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   100
		if filename in urls:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   101
			urls.remove(filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   102
		urls.insert(0, filename)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   103
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   104
	# command line url is a fallback, so it's last
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   105
	if url != None and url not in urls:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   106
		urls.append(url)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   107
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   108
	return urls
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   109
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   110
def usage():
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   111
	print "Usage: %s [-f|--file (file)] [-l|--link] [-h|--hash (hash)] [-s|--search (search-dir)] --url (url)" % (sys.argv[0].split('/')[-1])
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   112
	sys.exit(1)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   113
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   114
def main():
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   115
	import getopt
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   116
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   117
	# FLUSH STDOUT 
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   118
	sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   119
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   120
	file_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   121
	link_arg = False
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   122
	hash_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   123
	url_arg = None
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   124
	search_list = list()
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   125
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   126
	try:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   127
		opts, args = getopt.getopt(sys.argv[1:], "f:h:ls:u:",
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   128
			["file=", "link", "hash=", "search=", "url="])
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   129
	except getopt.GetoptError, err:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   130
		print str(err)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   131
		usage()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   132
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   133
	for opt, arg in opts:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   134
		if opt in [ "-f", "--file" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   135
			file_arg = arg
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   136
		elif opt in [ "-l", "--link" ]:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   137
			link_arg = True
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   138
		elif opt in [ "-h", "--hash" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   139
			hash_arg = arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   140
		elif opt in [ "-s", "--search" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   141
			search_list.append(arg)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   142
		elif opt in [ "-u", "--url" ]:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   143
			url_arg = arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   144
		else:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   145
			assert False, "unknown option"
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   146
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   147
	if url_arg == None:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   148
		usage()
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   149
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   150
	for url in download_paths(search_list, file_arg, url_arg):
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   151
		print "Source %s..." % url,
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   152
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   153
		scheme, path = splittype(url)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   154
		name = file_arg
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   155
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   156
		if scheme in [ None, 'file' ]:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   157
			if os.path.exists(path) == False:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   158
				print "not found, skipping file copy"
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   159
				continue
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   160
			elif name != path:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   161
				if link_arg == False:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   162
					print "\n    copying..."
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   163
					shutil.copy2(path, name)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   164
				else:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   165
					print "\n    linking..."
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   166
					os.symlink(path, name)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   167
			else:
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   168
				pass
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   169
		elif scheme in [ 'http', 'https', 'ftp' ]:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   170
			print "\n    downloading...",
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   171
			name = download(url, file_arg)
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   172
			if name == None:
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   173
				print "failed"
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   174
				continue
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   175
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   176
		print "\n    validating...",
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   177
		if hash_arg == None:
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   178
			print "skipping (no hash)"
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   179
			sys.exit(0)
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   180
			
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   181
		realhash = validate(name, hash_arg)
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   182
		if realhash == hash_arg:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   183
			print "ok"
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   184
			sys.exit(0)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   185
		else:
58
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   186
			print "corruption detected"
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   187
			print "    expected: %s" % hash_arg
d8024c042a00 7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 42
diff changeset
   188
			print "    actual:   %s" % realhash
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   189
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   190
		try:
42
566ce4d2ff99 fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents: 34
diff changeset
   191
			os.remove(name)
18
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   192
		except OSError:
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   193
			pass
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   194
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   195
	sys.exit(1)
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   196
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   197
if __name__ == "__main__":
6da6bd662127 Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff changeset
   198
	main()