author | Norm Jacobs <Norm.Jacobs@Sun.COM> |
Wed, 05 Jan 2011 09:51:40 -0800 | |
changeset 56 | 2b7bb5d1216b |
parent 42 | 566ce4d2ff99 |
child 58 | d8024c042a00 |
permissions | -rwxr-xr-x |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
1 |
#!/usr/bin/python2.6 |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
2 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
3 |
# CDDL HEADER START |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
4 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
5 |
# The contents of this file are subject to the terms of the |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
6 |
# Common Development and Distribution License (the "License"). |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
7 |
# You may not use this file except in compliance with the License. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
8 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
9 |
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
10 |
# or http://www.opensolaris.org/os/licensing. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
11 |
# See the License for the specific language governing permissions |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
12 |
# and limitations under the License. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
13 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
14 |
# When distributing Covered Code, include this CDDL HEADER in each |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
15 |
# file and include the License file at usr/src/OPENSOLARIS.LICENSE. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
16 |
# If applicable, add the following below this CDDL HEADER, with the |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
17 |
# fields enclosed by brackets "[]" replaced with your own identifying |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
18 |
# information: Portions Copyright [yyyy] [name of copyright owner] |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
19 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
20 |
# CDDL HEADER END |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
21 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
22 |
# Copyright (c) 2010, Oracle and/or it's affiliates. All rights reserved. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
23 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
24 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
25 |
# fetch.py - a file download utility |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
26 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
27 |
# A simple program similiar to wget(1), but handles local file copy, ignores |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
28 |
# directories, and verifies file hashes. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
29 |
# |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
30 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
31 |
import os |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
32 |
import sys |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
33 |
import shutil |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
34 |
from urllib import splittype, urlopen |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
35 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
36 |
def validate(filename, hash): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
37 |
import hashlib |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
38 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
39 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
40 |
file = open(filename, 'r') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
41 |
except IOError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
42 |
return False |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
43 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
44 |
if (hash == None): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
45 |
return True |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
46 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
47 |
algorithm, value = hash.split(':') |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
48 |
try: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
49 |
m = hashlib.new(algorithm) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
50 |
except ValueError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
51 |
return False |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
52 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
53 |
while True: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
54 |
block = file.read() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
55 |
m.update(block) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
56 |
if block == '': |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
57 |
break |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
58 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
59 |
return m.hexdigest() == value |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
60 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
61 |
def download(url, filename = None): |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
62 |
src = None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
63 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
64 |
try: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
65 |
src = urlopen(url) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
66 |
except IOError: |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
67 |
return None |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
68 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
69 |
if filename == None: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
70 |
filename = src.geturl().split('/')[-1] |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
71 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
72 |
try: |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
73 |
dst = open(filename, 'wb'); |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
74 |
except IOError: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
75 |
src.close() |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
76 |
return None |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
77 |
|
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
78 |
while True: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
79 |
block = src.read() |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
80 |
if block == '': |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
81 |
break; |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
82 |
dst.write(block) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
83 |
|
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
84 |
src.close() |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
85 |
dst.close() |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
86 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
87 |
# return the name of the file that we downloaded the data to. |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
88 |
return filename |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
89 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
90 |
def download_paths(search, filename, url): |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
91 |
urls = list() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
92 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
93 |
if filename != None: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
94 |
tmp = os.getenv('DOWNLOAD_SEARCH_PATH') |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
95 |
if tmp: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
96 |
search += tmp.split(' ') |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
97 |
|
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
98 |
file = os.path.basename(filename) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
99 |
|
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
100 |
urls = [ base + '/' + file for base in search ] |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
101 |
|
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
102 |
# filename should always be first |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
103 |
if filename in urls: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
104 |
urls.remove(filename) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
105 |
urls.insert(0, filename) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
106 |
|
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
107 |
# command line url is a fallback, so it's last |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
108 |
if url != None and url not in urls: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
109 |
urls.append(url) |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
110 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
111 |
return urls |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
112 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
113 |
def usage(): |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
114 |
print "Usage: %s [-f|--file (file)] [-l|--link] [-h|--hash (hash)] [-s|--search (search-dir)] --url (url)" % (sys.argv[0].split('/')[-1]) |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
115 |
sys.exit(1) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
116 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
117 |
def main(): |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
118 |
import getopt |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
119 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
120 |
# FLUSH STDOUT |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
121 |
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
122 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
123 |
file_arg = None |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
124 |
link_arg = False |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
125 |
hash_arg = None |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
126 |
url_arg = None |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
127 |
search_list = list() |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
128 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
129 |
try: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
130 |
opts, args = getopt.getopt(sys.argv[1:], "f:h:ls:u:", |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
131 |
["file=", "link", "hash=", "search=", "url="]) |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
132 |
except getopt.GetoptError, err: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
133 |
print str(err) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
134 |
usage() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
135 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
136 |
for opt, arg in opts: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
137 |
if opt in [ "-f", "--file" ]: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
138 |
file_arg = arg |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
139 |
elif opt in [ "-l", "--link" ]: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
140 |
link_arg = True |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
141 |
elif opt in [ "-h", "--hash" ]: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
142 |
hash_arg = arg |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
143 |
elif opt in [ "-s", "--search" ]: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
144 |
search_list.append(arg) |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
145 |
elif opt in [ "-u", "--url" ]: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
146 |
url_arg = arg |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
147 |
else: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
148 |
assert False, "unknown option" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
149 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
150 |
if url_arg == None: |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
151 |
usage() |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
152 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
153 |
for url in download_paths(search_list, file_arg, url_arg): |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
154 |
print "Source %s..." % url, |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
155 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
156 |
scheme, path = splittype(url) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
157 |
name = file_arg |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
158 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
159 |
if scheme in [ None, 'file' ]: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
160 |
if os.path.exists(path) == False: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
161 |
print "not found, skipping" |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
162 |
continue |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
163 |
elif name != path: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
164 |
if link_arg == False: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
165 |
print "copying...", |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
166 |
shutil.copy2(path, name) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
167 |
else: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
168 |
print "linking...", |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
169 |
os.symlink(path, name) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
170 |
else: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
171 |
pass |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
172 |
elif scheme in [ 'http', 'https', 'ftp' ]: |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
173 |
print "downloading...", |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
174 |
name = download(url, file_arg) |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
175 |
if name == None: |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
176 |
print "failed" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
177 |
continue |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
178 |
|
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
179 |
print "validating...", |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
180 |
if validate(name, hash_arg): |
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
181 |
print "ok" |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
182 |
sys.exit(0) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
183 |
else: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
184 |
print "corrupt" |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
185 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
186 |
try: |
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset
|
187 |
os.remove(name) |
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
188 |
except OSError: |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
189 |
pass |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
190 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
191 |
sys.exit(1) |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
192 |
|
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
193 |
if __name__ == "__main__": |
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset
|
194 |
main() |