author | Norm Jacobs <Norm.Jacobs@Sun.COM> |
Sun, 19 Sep 2010 15:50:53 -0500 | |
changeset 34 | d20b10eba317 |
parent 18 | tools/fetch.py@6da6bd662127 |
child 42 | 566ce4d2ff99 |
permissions | -rwxr-xr-x |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
1 |
#!/usr/bin/python2.6 |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
2 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
3 |
# CDDL HEADER START |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
4 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
5 |
# The contents of this file are subject to the terms of the |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
6 |
# Common Development and Distribution License (the "License"). |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
7 |
# You may not use this file except in compliance with the License. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
8 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
9 |
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
10 |
# or http://www.opensolaris.org/os/licensing. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
11 |
# See the License for the specific language governing permissions |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
12 |
# and limitations under the License. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
13 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
14 |
# When distributing Covered Code, include this CDDL HEADER in each |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
15 |
# file and include the License file at usr/src/OPENSOLARIS.LICENSE. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
16 |
# If applicable, add the following below this CDDL HEADER, with the |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
17 |
# fields enclosed by brackets "[]" replaced with your own identifying |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
18 |
# information: Portions Copyright [yyyy] [name of copyright owner] |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
19 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
20 |
# CDDL HEADER END |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
21 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
22 |
# Copyright (c) 2010, Oracle and/or it's affiliates. All rights reserved. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
23 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
24 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
25 |
# fetch.py - a file download utility |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
26 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
27 |
# A simple program similiar to wget(1), but handles local file copy, ignores |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
28 |
# directories, and verifies file hashes. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
29 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
30 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
31 |
import os |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
32 |
import sys |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
33 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
34 |
def validate(filename, hash): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
35 |
import hashlib |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
36 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
37 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
38 |
file = open(filename, 'r') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
39 |
except IOError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
40 |
return False |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
41 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
42 |
if (hash == None): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
43 |
return True |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
44 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
45 |
algorithm, value = hash.split(':') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
46 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
47 |
m = hashlib.new(algorithm) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
48 |
except ValueError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
49 |
return False |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
50 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
51 |
while True: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
52 |
block = file.read() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
53 |
m.update(block) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
54 |
if block == '': |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
55 |
break |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
56 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
57 |
return m.hexdigest() == value |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
58 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
59 |
def download(url, filename = None, hash = None, verbose = False): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
60 |
from urllib import splittype, urlopen |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
61 |
import hashlib |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
62 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
63 |
src = None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
64 |
filename = None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
65 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
66 |
#print "Downloading %s from %s" % (filename, url) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
67 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
68 |
# open the download source |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
69 |
if splittype(url)[0]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
70 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
71 |
src = urlopen(url) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
72 |
except IOError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
73 |
return None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
74 |
remote_name = src.geturl().split('/')[-1] |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
75 |
elif os.path.isfile(url): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
76 |
os.symlink(url, filename) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
77 |
return filename, filehash(filename) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
78 |
src = open(url, 'r') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
79 |
remote_name = url.split('/')[-1] |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
80 |
else: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
81 |
return None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
82 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
83 |
if filename == None: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
84 |
filename = remote_name |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
85 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
86 |
# if we have a source to download, open a destination and copy the data |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
87 |
if src: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
88 |
dst = open(filename, 'wb'); |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
89 |
if verbose: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
90 |
print "Downloading %s from %s" % (filename, url) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
91 |
while True: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
92 |
block = src.read() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
93 |
if block == '': |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
94 |
break; |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
95 |
dst.write(block) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
96 |
dst.close() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
97 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
98 |
# return the name of the file that we downloaded the data to. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
99 |
return filename |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
100 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
101 |
def download_paths(search, filename): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
102 |
tmp = os.getenv('DOWNLOAD_SEARCH_PATH') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
103 |
if tmp: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
104 |
search += tmp.split(' ') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
105 |
base = os.path.basename(filename) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
106 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
107 |
urls = list() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
108 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
109 |
for path in search: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
110 |
urls.append(path+'/'+base) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
111 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
112 |
return urls |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
113 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
114 |
def usage(): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
115 |
print "Usage: %s [-v|--verbose] [-f|--file (file)] [-h|--hash (hash)] --url (url)" % (sys.argv[0].split('/')[-1]) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
116 |
sys.exit(1) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
117 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
118 |
def main(): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
119 |
from urllib import splittype, urlopen |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
120 |
import getopt |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
121 |
import sys |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
122 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
123 |
# FLUSH STDOUT |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
124 |
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
125 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
126 |
verbose = False |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
127 |
filename = None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
128 |
url = None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
129 |
hash = None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
130 |
search = list() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
131 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
132 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
133 |
opts, args = getopt.getopt(sys.argv[1:], "f:h:s:u:v", |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
134 |
["file=", "hash=", "search=", "url=", "verbose"]) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
135 |
except getopt.GetoptError, err: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
136 |
print str(err) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
137 |
usage() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
138 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
139 |
for opt, arg in opts: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
140 |
if opt in [ "-f", "--file" ]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
141 |
filename = arg |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
142 |
elif opt in [ "-h", "--hash" ]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
143 |
hash = arg |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
144 |
elif opt in [ "-s", "--search" ]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
145 |
search.append(arg) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
146 |
elif opt in [ "-u", "--url" ]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
147 |
url = arg |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
148 |
elif opt in [ "-v", "--verbose" ]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
149 |
verbose = True |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
150 |
else: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
151 |
assert False, "unknown option" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
152 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
153 |
if url == None: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
154 |
usage() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
155 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
156 |
print "Fetching %s" % filename |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
157 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
158 |
if validate(filename, hash): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
159 |
print "\tcurrent copy is ok, nothing to do" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
160 |
sys.exit(0) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
161 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
162 |
# generate a list of URLS to search for a suitable download |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
163 |
urls = download_paths(search, filename) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
164 |
urls.append(url) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
165 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
166 |
for url in urls: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
167 |
print "\tTrying %s..." % url, |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
168 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
169 |
if os.path.isfile(url): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
170 |
os.symlink(url, filename) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
171 |
elif splittype(url)[0]: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
172 |
if download(url, filename) == None: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
173 |
print "failed" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
174 |
continue |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
175 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
176 |
print "retrieved...", |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
177 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
178 |
if validate(filename, hash): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
179 |
print "validated" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
180 |
sys.exit(0) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
181 |
else: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
182 |
print "corrupt" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
183 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
184 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
185 |
os.remove(filename) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
186 |
except OSError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
187 |
pass |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
188 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
189 |
sys.exit(1) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
190 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
191 |
if __name__ == "__main__": |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
192 |
main() |