author  Mike Sullivan <Mike.Sullivan@Oracle.COM> 
Fri, 07 Apr 2017 17:56:36 0700  
changeset 7851  f82a18f72950 
parent 6950  b14572a6af9b 
permissions  rwxrxrx 
3770
ca450a806cc1
20440888 Userland python tools should migrate to 2.7
John Beck <John.Beck@Oracle.COM>
parents:
3533
diff
changeset

1 
#!/usr/bin/python2.7 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

2 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

3 
# CDDL HEADER START 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

4 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

5 
# The contents of this file are subject to the terms of the 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

6 
# Common Development and Distribution License (the "License"). 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

7 
# You may not use this file except in compliance with the License. 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

8 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

9 
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

10 
# or http://www.opensolaris.org/os/licensing. 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

11 
# See the License for the specific language governing permissions 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

12 
# and limitations under the License. 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

13 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

14 
# When distributing Covered Code, include this CDDL HEADER in each 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

15 
# file and include the License file at usr/src/OPENSOLARIS.LICENSE. 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

16 
# If applicable, add the following below this CDDL HEADER, with the 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

17 
# fields enclosed by brackets "[]" replaced with your own identifying 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

18 
# information: Portions Copyright [yyyy] [name of copyright owner] 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

19 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

20 
# CDDL HEADER END 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

21 
# 
5682
94c0ca64c022
15558602 TCL_LD_SEARCH_FLAGS is wrongly defined in tclConfig.sh
Shawn WalkerSalas <shawn.walker@oracle.com>
parents:
5244
diff
changeset

22 

94c0ca64c022
15558602 TCL_LD_SEARCH_FLAGS is wrongly defined in tclConfig.sh
Shawn WalkerSalas <shawn.walker@oracle.com>
parents:
5244
diff
changeset

23 
# 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

24 
# Copyright (c) 2010, 2016, Oracle and/or its affiliates. All rights reserved. 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

25 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

26 
# 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

27 
# userlandfetch  a file download utility 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

28 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

29 
# A simple program similiar to wget(1), but handles local file copy, ignores 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

30 
# directories, and verifies file hashes. 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

31 
# 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

32 

3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

33 
import errno 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

34 
import os 
5862
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

35 
import re 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

36 
import sys 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

37 
import shutil 
5862
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

38 
import json 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

39 
import subprocess 
5862
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

40 
from urllib import splittype, splithost 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

41 
from urllib2 import urlopen, HTTPError 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

42 
import hashlib 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

43 

136
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

44 
def printIOError(e, txt): 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

45 
""" Function to decode and print IOError type exception """ 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

46 
print "I/O Error: " + txt + ": " 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

47 
try: 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

48 
(code, message) = e 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

49 
print str(message) + " (" + str(code) + ")" 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

50 
except: 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

51 
print str(e) 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

52 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

53 
def validate_signature(path, signature): 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

54 
"""Given paths to a file and a detached PGP signature, verify that 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

55 
the signature is valid for the file. Current configuration allows for 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

56 
unrecognized keys to be downloaded as necessary.""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

57 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

58 
# Find the root of the repo so that we can point GnuPG at the right 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

59 
# configuration and keyring. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

60 
proc = subprocess.Popen(["hg", "root"], stdout=subprocess.PIPE) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

61 
proc.wait() 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

62 
if proc.returncode != 0: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

63 
return False 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

64 
out, err = proc.communicate() 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

65 
gpgdir = os.path.join(out.strip(), "tools", ".gnupg") 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

66 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

67 
# Skip the permissions warning: none of the information here is private, 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

68 
# so not having to worry about getting mercurial keeping the directory 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

69 
# unreadable is just simplest. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

70 
try: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

71 
proc = subprocess.Popen(["gpg2", "verify", 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

72 
"nopermissionwarning", "homedir", gpgdir, signature, 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

73 
path], stdin=open("/dev/null"), 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

74 
stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

75 
except OSError as e: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

76 
# If the executable simply couldn't be found, just skip the 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

77 
# validation. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

78 
if e.errno == errno.ENOENT: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

79 
return False 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

80 
raise 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

81 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

82 
proc.wait() 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

83 
if proc.returncode != 0: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

84 
# Only print GnuPG's output when there was a problem. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

85 
print proc.stdout.read() 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

86 
return False 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

87 
return True 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

88 

135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

89 
def validate(file, hash): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

90 
"""Given a filelike object and a hash string, verify that the hash 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

91 
matches the file contents.""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

92 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

93 
try: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

94 
algorithm, hashvalue = hash.split(':') 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

95 
except: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

96 
algorithm = "sha256" 
800
2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

97 

2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

98 
# force migration away from sha1 
2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

99 
if algorithm == "sha1": 
2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

100 
algorithm = "sha256" 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

101 
try: 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

102 
m = hashlib.new(algorithm) 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

103 
except ValueError: 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

104 
return False 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

105 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

106 
while True: 
137
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

107 
try: 
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

108 
block = file.read() 
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

109 
except IOError, err: 
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

110 
print str(err), 
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

111 
break 
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

112 

18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

113 
m.update(block) 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

114 
if block == '': 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

115 
break 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

116 

58
d8024c042a00
7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
42
diff
changeset

117 
return "%s:%s" % (algorithm, m.hexdigest()) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

118 

135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

119 
def validate_container(filename, hash): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

120 
"""Given a file path and a hash string, verify that the hash matches the 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

121 
file contents.""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

122 

135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

123 
try: 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

124 
file = open(filename, 'r') 
136
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

125 
except IOError as e: 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

126 
printIOError(e, "Can't open file " + filename) 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

127 
return False 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

128 
return validate(file, hash) 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

129 

5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

130 

5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

131 
def validate_payload(filename, hash): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

132 
"""Given a file path and a hash string, verify that the hash matches the 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

133 
payload (uncompressed content) of the file.""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

134 

135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

135 
import gzip 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

136 
import bz2 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

137 

5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

138 
expr_bz = re.compile('.+\.bz2$', re.IGNORECASE) 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

139 
expr_gz = re.compile('.+\.gz$', re.IGNORECASE) 
800
2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

140 
expr_tgz = re.compile('.+\.tgz$', re.IGNORECASE) 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

141 

5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

142 
try: 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

143 
if expr_bz.match(filename): 
137
6fb20db3eee4
7028115 userlandfetch fails to validate gd2 archive (fix bzip)
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
136
diff
changeset

144 
file = bz2.BZ2File(filename, 'r') 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

145 
elif expr_gz.match(filename): 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

146 
file = gzip.GzipFile(filename, 'r') 
800
2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

147 
elif expr_tgz.match(filename): 
2ad056ed89ec
7163771 sha1 should not be used in userlandfetch and userland component/*/Makefile
Mike Sullivan <Mike.Sullivan@Oracle.COM>
parents:
212
diff
changeset

148 
file = gzip.GzipFile(filename, 'r') 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

149 
else: 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

150 
return False 
136
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

151 
except IOError as e: 
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

152 
printIOError(e, "Can't open archive " + filename) 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

153 
return False 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

154 
return validate(file, hash) 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

155 

5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

156 
def download(url, timeout, filename=None, quiet=False): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

157 
"""Download the content at the given URL to the given filename 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

158 
(defaulting to the basename of the URL if not given. If 'quiet' is 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

159 
True, throw away any error messages. Returns the name of the file to 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

160 
which the content was donloaded.""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

161 

18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

162 
src = None 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

163 

42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

164 
try: 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

165 
src = urlopen(url=url, timeout=timeout) 
136
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

166 
except IOError as e: 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

167 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

168 
printIOError(e, "Can't open url " + url) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

169 
return None 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

170 

212
8a66c0c0e8dc
7041047 userlandfetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
137
diff
changeset

171 
# 3xx, 4xx and 5xx (fht)tp codes designate unsuccessfull action 
8a66c0c0e8dc
7041047 userlandfetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
137
diff
changeset

172 
if 3 <= int(src.getcode()/100) <= 5: 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

173 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

174 
print "Error code: " + str(src.getcode()) 
212
8a66c0c0e8dc
7041047 userlandfetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
137
diff
changeset

175 
return None 
8a66c0c0e8dc
7041047 userlandfetch should handle http errors more gracefully
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
137
diff
changeset

176 

18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

177 
if filename == None: 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

178 
filename = src.geturl().split('/')[1] 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

179 

42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

180 
try: 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

181 
dst = open(filename, 'wb'); 
136
6a7dd3a5aaf9
7027131 userlandfetch should be more verbose when printing IOError exception
Vladimir Marek <Vladimir.Marek@oracle.com>
parents:
135
diff
changeset

182 
except IOError as e: 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

183 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

184 
printIOError(e, "Can't open file " + filename + " for writing") 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

185 
src.close() 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

186 
return None 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

187 

566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

188 
while True: 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

189 
block = src.read() 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

190 
if block == '': 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

191 
break; 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

192 
dst.write(block) 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

193 

566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

194 
src.close() 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

195 
dst.close() 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

196 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

197 
# return the name of the file that we downloaded the data to. 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

198 
return filename 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

199 

5862
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

200 
def pypi_url(url, filename): 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

201 
"""Given a pypi: URL, return the real URL for that component/version. 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

202 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

203 
The pypi scheme has a host (with an empty host defaulting to 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

204 
pypi.python.org), and a path that should be of the form 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

205 
"component==version". Other specs could be supported, but == is the 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

206 
only thing that makes sense in this context. 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

207 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

208 
The filename argument is the name of the expected file to download, so 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

209 
that when pypi gives us multiple archives to choose from, we can pick 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

210 
the right one. 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

211 
""" 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

212 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

213 
host, path = splithost(splittype(url)[1]) 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

214 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

215 
# We have to use ==; anything fancier would require pkg_resources, but 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

216 
# really that's the only thing that makes sense in this context. 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

217 
try: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

218 
name, version = re.match("/(.*)==(.*)$", path).groups() 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

219 
except AttributeError: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

220 
print "PyPI URLs must be of the form 'pypi:///component==version'" 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

221 
return None 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

222 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

223 
if not host: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

224 
jsurl = "http://pypi.python.org/pypi/%s/json" % name 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

225 
else: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

226 
jsurl = "http://%s/pypi/%s/json" % (host, name) 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

227 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

228 
try: 
5953
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

229 
# Don't wait very long for the connection 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

230 
f = urlopen(jsurl, None, 2) 
5862
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

231 
except HTTPError as e: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

232 
if e.getcode() == 404: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

233 
print "Unknown component '%s'" % name 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

234 
else: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

235 
printIOError(e, "Can't open PyPI JSON url %s" % url) 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

236 
return None 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

237 
except IOError as e: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

238 
printIOError(e, "Can't open PyPI JSON url %s" % url) 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

239 
return None 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

240 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

241 
js = json.load(f) 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

242 
try: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

243 
verblock = js["releases"][version] 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

244 
except KeyError: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

245 
print "Unknown version '%s'" % version 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

246 
return None 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

247 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

248 
urls = [ d["url"] for d in verblock ] 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

249 
for archiveurl in urls: 
5952
0ff1d1b8de45
23254586 sourcedownload job doesn't work on pypi things anymore
Danek Duvall <danek.duvall@oracle.com>
parents:
5862
diff
changeset

250 
if archiveurl.endswith("/%s" % os.path.basename(filename)): 
5862
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

251 
return archiveurl 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

252 

ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

253 
if urls: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

254 
print "None of the following URLs delivers '%s':" % filename 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

255 
print " " + "\n ".join(urls) 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

256 
else: 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

257 
print "Couldn't find any suitable URLs" 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

258 
return None 
ce31ce66da02
23183311 pypi archive url schemes have changed incompatibly
Danek Duvall <danek.duvall@oracle.com>
parents:
5682
diff
changeset

259 

42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

260 
def download_paths(search, filename, url): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

261 
"""Returns a list of URLs where the file 'filename' might be found, 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

262 
using 'url', 'search', and $DOWNLOAD_SEARCH_PATH as places to look. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

263 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

264 
If 'filename' is None, then the list will simply contain 'url'. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

265 
""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

266 

18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

267 
urls = list() 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

268 

42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

269 
if filename != None: 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

270 
tmp = os.getenv('DOWNLOAD_SEARCH_PATH') 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

271 
if tmp: 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

272 
search += tmp.split(' ') 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

273 

566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

274 
file = os.path.basename(filename) 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

275 

566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

276 
urls = [ base + '/' + file for base in search ] 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

277 

566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

278 
# filename should always be first 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

279 
if filename in urls: 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

280 
urls.remove(filename) 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

281 
urls.insert(0, filename) 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

282 

566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

283 
# command line url is a fallback, so it's last 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

284 
if url != None and url not in urls: 
5953
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

285 
urls.append(url) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

286 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

287 
return urls 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

288 

5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

289 
def download_from_paths(search_list, file_arg, url, timeout_arg, link_arg, quiet=False): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

290 
"""Attempts to download a file from a number of possible locations. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

291 
Generates a list of paths where the file ends up on the local 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

292 
filesystem. This is a generator because while a download might be 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

293 
successful, the signature or hash may not validate, and the caller may 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

294 
want to try again from the next location. The 'link_arg' argument is a 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

295 
boolean which, when True, specifies that if the source is not a remote 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

296 
URL and not already found where it should be, to make a symlink to the 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

297 
source rather than copying it. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

298 
""" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

299 
for url in download_paths(search_list, file_arg, url): 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

300 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

301 
print "Source %s..." % url, 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

302 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

303 
scheme, path = splittype(url) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

304 
name = file_arg 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

305 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

306 
if scheme in [ None, 'file' ]: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

307 
if os.path.exists(path) == False: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

308 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

309 
print "not found, skipping file copy" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

310 
continue 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

311 
elif name and name != path: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

312 
if link_arg == False: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

313 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

314 
print "\n copying..." 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

315 
shutil.copy2(path, name) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

316 
else: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

317 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

318 
print "\n linking..." 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

319 
os.symlink(path, name) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

320 
elif scheme in [ 'http', 'https', 'ftp' ]: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

321 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

322 
print "\n downloading...", 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

323 
name = download(url, timeout_arg, file_arg, quiet) 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

324 
if name == None: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

325 
if not quiet: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

326 
print "failed" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

327 
continue 
5953
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

328 
elif scheme == "pypi": 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

329 
nurl = pypi_url(url, file_arg) 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

330 
if nurl: 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

331 
if not quiet: 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

332 
print "\n translated %s to %s..." % ( 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

333 
url, nurl), 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

334 
print "\n downloading...", 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

335 
else: 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

336 
if not quiet: 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

337 
print "\n unable to contact PyPI", 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

338 
continue 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

339 
name = download(nurl, timeout_arg, file_arg, quiet) 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

340 
if name is None: 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

341 
if not quiet: 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

342 
print "failed" 
de8c30ca3eaa
23261178 userlandfetch relies on pypi too much if the archive is in the mirror
Danek Duvall <danek.duvall@oracle.com>
parents:
5952
diff
changeset

343 
continue 
6950
b14572a6af9b
24675631 userlandfetch fails with success for unknown scheme
Danek Duvall <danek.duvall@oracle.com>
parents:
5953
diff
changeset

344 
else: 
b14572a6af9b
24675631 userlandfetch fails with success for unknown scheme
Danek Duvall <danek.duvall@oracle.com>
parents:
5953
diff
changeset

345 
print " unknown scheme '%s'" % scheme 
b14572a6af9b
24675631 userlandfetch fails with success for unknown scheme
Danek Duvall <danek.duvall@oracle.com>
parents:
5953
diff
changeset

346 
return 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

347 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

348 
yield name 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

349 

18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

350 
def usage(): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

351 
print "Usage: %s [ffile (file)] [llink] [hhash (hash)] " \ 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

352 
"[ssearch (searchdir)] [Ssigurl (signatureurl)] " \ 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

353 
"[ttimeout (timeout)] url (url)" % \ 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

354 
(sys.argv[0].split('/')[1]) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

355 
sys.exit(1) 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

356 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

357 
def main(): 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

358 
import getopt 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

359 

5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

360 
# FLUSH STDOUT 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

361 
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

362 

42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

363 
file_arg = None 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

364 
link_arg = False 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

365 
hash_arg = None 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

366 
url_arg = None 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

367 
sig_arg = None 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

368 
timeout_arg = 300 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

369 
search_list = list() 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

370 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

371 
try: 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

372 
opts, args = getopt.getopt(sys.argv[1:], "f:h:ls:S:t:u:", 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

373 
["file=", "link", "hash=", "search=", "sigurl=", 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

374 
"timeout=", "url="]) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

375 
except getopt.GetoptError, err: 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

376 
print str(err) 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

377 
usage() 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

378 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

379 
for opt, arg in opts: 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

380 
if opt in [ "f", "file" ]: 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

381 
file_arg = arg 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

382 
elif opt in [ "l", "link" ]: 
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

383 
link_arg = True 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

384 
elif opt in [ "h", "hash" ]: 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

385 
hash_arg = arg 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

386 
elif opt in [ "s", "search" ]: 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

387 
search_list.append(arg) 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

388 
elif opt in [ "S", "sigurl" ]: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

389 
sig_arg = arg 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

390 
elif opt in [ "t", "timeout" ]: 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

391 
try: 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

392 
timeout_arg = int(arg) 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

393 
except ValueError: 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

394 
print "Invalid argument for %s, should be a " \ 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

395 
"number, but is %s" % (opt, arg) 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

396 
sys.exit(1) 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

397 
if timeout_arg < 0: 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

398 
print "Invalid argument for %s, should be a " \ 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

399 
"positive number, but is %s" % (opt, arg) 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

400 
sys.exit(1) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

401 
elif opt in [ "u", "url" ]: 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

402 
url_arg = arg 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

403 
else: 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

404 
assert False, "unknown option" 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

405 

5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

406 
for name in download_from_paths(search_list, file_arg, url_arg, 
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

407 
timeout_arg, link_arg): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

408 
print "\n validating signature...", 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

409 

3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

410 
sig_valid = False 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

411 
if not sig_arg: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

412 
print "skipping (no signature URL)" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

413 
else: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

414 
# Put the signature file in the same directory as the 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

415 
# file we're downloading. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

416 
sig_file = os.path.join( 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

417 
os.path.dirname(file_arg), 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

418 
os.path.basename(sig_arg)) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

419 
# Validate with the first signature we find. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

420 
for sig_file in download_from_paths(search_list, sig_file, 
5244
c63fce595432
21888443 userlandfetch should have a time out for reading from network
Vladimir Kotal <Vladimir.Kotal@oracle.com>
parents:
4429
diff
changeset

421 
sig_arg, timeout_arg, link_arg, True): 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

422 
if sig_file: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

423 
if validate_signature(name, sig_file): 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

424 
print "ok" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

425 
sig_valid = True 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

426 
else: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

427 
print "failed" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

428 
break 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

429 
else: 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

430 
continue 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

431 
else: 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

432 
print "failed (couldn't fetch signature)" 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

433 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

434 
print " validating hash...", 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

435 
realhash = validate_container(name, hash_arg) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

436 

3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

437 
if not hash_arg: 
58
d8024c042a00
7011369 gmake complains about fresh userland bits
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
42
diff
changeset

438 
print "skipping (no hash)" 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

439 
print "hash is: %s" % realhash 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

440 
elif realhash == hash_arg: 
42
566ce4d2ff99
fix using/downloading shared archives
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
34
diff
changeset

441 
print "ok" 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

442 
else: 
135
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

443 
payloadhash = validate_payload(name, hash_arg) 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

444 
if payloadhash == hash_arg: 
5c6feb0253fa
7028115 userlandfetch fails to validate gd2 archive
Norm Jacobs <Norm.Jacobs@Oracle.COM>
parents:
58
diff
changeset

445 
print "ok" 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

446 
else: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

447 
# If the signature validated, then we assume 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

448 
# that the expected hash is just a typo, but we 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

449 
# warn just in case. 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

450 
if sig_valid: 
4429
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

451 
print "invalid hash! Did you forget " \ 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

452 
"to update it?" 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

453 
else: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

454 
print "corruption detected" 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

455 

3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

456 
print " expected: %s" % hash_arg 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

457 
print " actual: %s" % realhash 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

458 
print " payload: %s" % payloadhash 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

459 

4429
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

460 
# If the hash is invalid, but the signature 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

461 
# validation succeeded, rename the archive (so 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

462 
# the user doesn't have to redownload it) and 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

463 
# fail. Otherwise, try to remove the file and 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

464 
# try again. 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

465 
if sig_valid: 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

466 
newname = name + ".invalidhash" 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

467 
try: 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

468 
os.rename(name, newname) 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

469 
except OSError: 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

470 
pass 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

471 
else: 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

472 
print "archive saved as %s; " \ 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

473 
"if it isn't corrupt, " \ 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

474 
"rename to %s" % (newname, 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

475 
name) 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

476 
sys.exit(1) 
a6c5fd1cbbc9
21132783 userlandfetch should fail if the hash fails, even if the signature doesn't
Danek Duvall <danek.duvall@oracle.com>
parents:
4196
diff
changeset

477 
else: 
3533
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

478 
try: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

479 
os.remove(name) 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

480 
except OSError: 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

481 
pass 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

482 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

483 
continue 
0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

484 

0b8107a40da7
20183619 userland should support PGP signatures
Danek Duvall <danek.duvall@oracle.com>
parents:
832
diff
changeset

485 
sys.exit(0) 
18
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

486 
sys.exit(1) 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

487 

6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

488 
if __name__ == "__main__": 
6da6bd662127
Add a fetch utility to do more than just download like wget does
Norm Jacobs <Norm.Jacobs@Sun.COM>
parents:
diff
changeset

489 
main() 