annotate sources/download_functions.sh @ 1816:fb05f99ffcce draft

Collate site-local variables under "MY_XXX" prefix: MY_PATCH_DIR, MY_ROOT_OVERLAY, MY_CROSS_PATH and MY_CROSS_PREFIX.
author Rob Landley <rob@landley.net>
date Sat, 21 Nov 2015 16:44:39 -0600
parents 5ff5fea0ad7a
children 40c2fe006fcf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
1 #!/bin/echo "This file is sourced, not run"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
2
1622
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
3 # Set of functions to download, extract, and patch source tarballs.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
4 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
5 # Tools to populate and verify a directory of package source tarballs
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
6 # (saved in $SRCDIR which defaults to $TOP/packages). Used by download.sh.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
7 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
8 # You can supply your own tarball in $SRCDIR to avoid downloading it.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
9 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
10 # You can also provide an expanded directory (same filename as $URL
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
11 # but no version or extension) to be used instead of a tarball. This is
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
12 # usually a source control checkout.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
13 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
14 # $IGNORE_REPOS - comma separated list of package names (or "all") to
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
15 # download tarballs for anyway, ignoring the directory version if present.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
16 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
17 # Functions to call from here:
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
18 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
19 # download
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
20 # - fetch a file (with wget) if it doesn't already exist, or doesn't match
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
21 # checksum.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
22 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
23 # It expects you to set:
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
24 # $URL - Default location of file, including filename
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
25 # $SHA1 - sha1sum of good file. (Blank means accept any file.)
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
26 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
27 # You can also set these (which use filename from $URL):
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
28 # $PREFERRED_MIRROR - Check here first (appending filename from $URL)
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
29 # $MIRROR_LIST - Space separated list of fallback locations (appending
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
30 # filename from $URL) to check if default location didn't have it.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
31 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
32 # Does not re-download existing tarballs if the $SHA1 matches/is blank.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
33 # Does not download tarball if expanded directory present.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
34 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
35 # cleanup_oldfiles
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
36 # - remove stale files from $SRCDIR
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
37 #
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
38 # Stale means not referenced by a download call since start of script.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
39 # Only affects files, not subdirectories.
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
40
ca0cf2dc37c2 Add more documentation.
Rob Landley <rob@landley.net>
parents: 1599
diff changeset
41
1257
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
42 # Remove version information and extension tarball name "$1".
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
43 # If "$2", add that version number back, keeping original extension.
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
44
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
45 noversion()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
46 {
1257
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
47 LOGRUS='s/-*\(\([0-9\.]\)*\([_-]rc\)*\(-pre\)*\([0-9][a-zA-Z]\)*\)*\(\.tar\(\..z2*\)*\)$'
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
48 [ -z "$2" ] && LOGRUS="$LOGRUS//" || LOGRUS="$LOGRUS/$2\\6/"
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
49
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
50 echo "$1" | sed -e "$LOGRUS"
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
51 }
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
52
1814
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
53 gather_patches()
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
54 {
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
55 ls "$PATCHDIR/${PACKAGE}"-*.patch 2> /dev/null | sort | while read i
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
56 do
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
57 if [ -f "$i" ]
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
58 then
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
59 echo "$i"
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
60 fi
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
61 done
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
62
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
63 # gather external package patches sorted by filename
1816
fb05f99ffcce Collate site-local variables under "MY_XXX" prefix: MY_PATCH_DIR,
Rob Landley <rob@landley.net>
parents: 1814
diff changeset
64 if [ ! -z "$MY_PATCH_DIR" ] && [ -d "${MY_PATCH_DIR}/${PACKAGE}" ]
1814
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
65 then
1816
fb05f99ffcce Collate site-local variables under "MY_XXX" prefix: MY_PATCH_DIR,
Rob Landley <rob@landley.net>
parents: 1814
diff changeset
66 for i in "${MY_PATCH_DIR}/${PACKAGE}/"*.patch
1814
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
67 do
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
68 if [ -f "$i" ]
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
69 then
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
70 echo "$i"
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
71 fi
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
72 done
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
73 fi
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
74 }
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
75
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
76 # Apply any patches to this package
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
77 patch_package()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
78 {
1814
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
79 gather_patches | while read i
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
80 do
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
81 if [ -f "$i" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
82 then
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
83 echo "Applying $i"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
84 (cd "${SRCTREE}/${PACKAGE}" &&
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
85 patch -p1 -i "$i" &&
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
86 sha1file "$i" >> "$SHA1FILE") ||
1418
f2f1238d25e2 Actually stop after the first failed patch.
Rob Landley <rob@landley.net>
parents: 1416
diff changeset
87 if [ -z "$ALLOW_PATCH_FAILURE" ]
f2f1238d25e2 Actually stop after the first failed patch.
Rob Landley <rob@landley.net>
parents: 1416
diff changeset
88 then
f2f1238d25e2 Actually stop after the first failed patch.
Rob Landley <rob@landley.net>
parents: 1416
diff changeset
89 dienow
f2f1238d25e2 Actually stop after the first failed patch.
Rob Landley <rob@landley.net>
parents: 1416
diff changeset
90 fi
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
91 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
92 done
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
93 }
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
94
1300
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
95 # Get the tarball for this package
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
96
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
97 find_package_tarball()
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
98 {
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
99 # If there are multiple similar files we want the newest timestamp, in case
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
100 # the URL just got upgraded but cleanup_oldfiles hasn't run yet. Be able to
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
101 # distinguish "package-123.tar.bz2" from "package-tests-123.tar.bz2" and
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
102 # return the shorter one reliably.
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
103
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
104 ls -tc "$SRCDIR/$1-"*.tar* 2>/dev/null | while read i
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
105 do
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
106 if [ "$(noversion "${i/*\//}")" == "$1" ]
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
107 then
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
108 echo "$i"
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
109 break
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
110 fi
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
111 done
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
112 }
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
113
1582
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
114 package_cache()
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
115 {
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
116 SNAPFROM="$SRCDIR/$1"
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
117 (is_in_list "$1" "$IGNORE_REPOS" || [ ! -d "$SNAPFROM" ]) &&
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
118 SNAPFROM="$SRCTREE/$1"
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
119
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
120 if [ ! -d "$SNAPFROM" ]
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
121 then
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
122 echo "$1 not found. Did you run download.sh?" >&2
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
123 dienow
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
124 fi
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
125
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
126 echo "$SNAPFROM"
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
127 }
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
128
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
129 # Extract tarball named in $1 and apply all relevant patches into
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
130 # "$BUILD/packages/$1". Record sha1sum of tarball and patch files in
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
131 # sha1-for-source.txt. Re-extract if tarball or patches change.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
132
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
133 extract_package()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
134 {
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
135 mkdir -p "$SRCTREE" || dienow
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
136
1642
73651ad7ad13 Move package announcement earlier so extracted (non-tarball) packages announce themselves.
Rob Landley <rob@landley.net>
parents: 1622
diff changeset
137 # Announce to the world that we're cracking open a new package
73651ad7ad13 Move package announcement earlier so extracted (non-tarball) packages announce themselves.
Rob Landley <rob@landley.net>
parents: 1622
diff changeset
138
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
139 PACKAGE="$1"
1642
73651ad7ad13 Move package announcement earlier so extracted (non-tarball) packages announce themselves.
Rob Landley <rob@landley.net>
parents: 1622
diff changeset
140 announce "$PACKAGE"
73651ad7ad13 Move package announcement earlier so extracted (non-tarball) packages announce themselves.
Rob Landley <rob@landley.net>
parents: 1622
diff changeset
141
1553
da21ff27331e Add ability to check out repositories in packages, IGNORE_REPOS to build from tarballs anyway, a FAQ entry on it, and more/repo.sh to set up/update some of the common ones.
Rob Landley <rob@landley.net>
parents: 1552
diff changeset
142 ! is_in_list "PACKAGE" "$IGNORE_REPOS" && [ -d "$SRCDIR/$PACKAGE" ] &&
da21ff27331e Add ability to check out repositories in packages, IGNORE_REPOS to build from tarballs anyway, a FAQ entry on it, and more/repo.sh to set up/update some of the common ones.
Rob Landley <rob@landley.net>
parents: 1552
diff changeset
143 return 0
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
144
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
145 # Find tarball, and determine type
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
146
1300
a80d7f256ff9 Handle package tarballs with overlapping names (thingy and thingy-test for example).
Rob Landley <rob@landley.net>
parents: 1257
diff changeset
147 FILENAME="$(find_package_tarball "$PACKAGE")"
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
148 DECOMPRESS=""
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
149 [ "$FILENAME" != "${FILENAME/%\.tar\.bz2/}" ] && DECOMPRESS="j"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
150 [ "$FILENAME" != "${FILENAME/%\.tar\.gz/}" ] && DECOMPRESS="z"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
151
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
152 # If the source tarball doesn't exist, but the extracted directory is there,
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
153 # assume everything's ok.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
154
1515
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
155 SHA1NAME="sha1-for-source.txt"
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
156 SHA1FILE="$SRCTREE/$PACKAGE/$SHA1NAME"
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
157 if [ -z "$FILENAME" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
158 then
1257
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
159 if [ ! -e "$SRCTREE/$PACKAGE" ]
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
160 then
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
161 echo "No tarball for $PACKAGE" >&2
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
162 dienow
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
163 fi
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
164
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
165 # If the sha1sum file isn't there, re-patch the package.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
166 [ ! -e "$SHA1FILE" ] && patch_package
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
167 return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
168 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
169
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
170 # Check the sha1 list from the previous extract. If the source is already
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
171 # up to date (including patches), keep it.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
172
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
173 SHA1TAR="$(sha1file "$FILENAME")"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
174 SHALIST=$(cat "$SHA1FILE" 2> /dev/null)
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
175 if [ ! -z "$SHALIST" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
176 then
1814
5ff5fea0ad7a Add EXTERNAL_PATCH_DIR option for applying patches from outside aboriginal
Geoff Salmon <gsalmon@se-instruments.com>
parents: 1642
diff changeset
177 for i in "$SHA1TAR" $(sha1file $(gather_patches))
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
178 do
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
179 # Is this sha1 in the file?
1582
6a8114c148d1 Lots of small improvements: check for toybox instead of busybox for host-tools $PATH adjustment, better manifest generation (with toybox in list), make package_cache function to find extracted source (so root-image.sh works when packages contains linux-git snapshot), fix more/record-commands.sh path adjustment.
Rob Landley <rob@landley.net>
parents: 1553
diff changeset
180 if [ -z "$(echo "$SHALIST" | grep "$i")" ]
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
181 then
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
182 SHALIST=missing
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
183 break
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
184 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
185 # Remove it
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
186 SHALIST="$(echo "$SHALIST" | sed "s/$i//" )"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
187 done
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
188 # If we matched all the sha1sums, nothing more to do.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
189 [ -z "$SHALIST" ] && return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
190 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
191
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
192 # Re-extract the package, deleting the old one (if any)..
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
193
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
194 echo -n "Extracting '$PACKAGE'"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
195 (
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
196 UNIQUE=$(readlink /proc/self)
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
197 trap 'rm -rf "$BUILD/temp-'$UNIQUE'"' EXIT
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
198 rm -rf "$SRCTREE/$PACKAGE" 2>/dev/null
1410
7373c4c7ecee Don't create build/packages _and_ $SRCTREE, they should be the same and when they aren't the first is wrong.
Rob Landley <rob@landley.net>
parents: 1358
diff changeset
199 mkdir -p "$BUILD/temp-$UNIQUE" "$SRCTREE" || dienow
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
200
1515
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
201 { tar -xv${DECOMPRESS} -f "$FILENAME" -C "$BUILD/temp-$UNIQUE" &&
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
202 # Wildcards only expand when they ponit to something that exists,
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
203 # and even then they have to be an argument to a command.
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
204 TARDIR="$(readlink -f "$BUILD/temp-$UNIQUE"/*)" &&
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
205 touch "$TARDIR/$SHA1NAME"
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
206 } | dotprogress
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
207
1515
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
208 [ -e "$BUILD/temp-$UNIQUE"/*/"$SHA1NAME" ] &&
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
209 mv "$BUILD/temp-$UNIQUE/"* "$SRCTREE/$PACKAGE" &&
1515
63d3f491b92f Make tarball extract failure detection not rely on "kill" command.
Rob Landley <rob@landley.net>
parents: 1486
diff changeset
210 echo "$SHA1TAR" >> "$SHA1FILE"
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
211 )
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
212
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
213 [ $? -ne 0 ] && dienow
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
214
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
215 patch_package
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
216 }
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
217
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
218 # Confirm that a file has the appropriate checksum (or exists but SHA1 is blank)
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
219 # Delete invalid file.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
220
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
221 confirm_checksum()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
222 {
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
223 SUM="$(sha1file "$SRCDIR/$FILENAME" 2>/dev/null)"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
224 if [ x"$SUM" == x"$SHA1" ] || [ -z "$SHA1" ] && [ -f "$SRCDIR/$FILENAME" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
225 then
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
226 if [ -z "$SHA1" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
227 then
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
228 echo "No SHA1 for $FILENAME ($SUM)"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
229 else
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
230 echo "Confirmed $FILENAME"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
231 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
232
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
233 # Preemptively extract source packages?
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
234
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
235 [ -z "$EXTRACT_ALL" ] && return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
236 extract_package "$BASENAME"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
237 return $?
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
238 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
239
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
240 # If there's a corrupted file, delete it. In theory it would be nice
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
241 # to resume downloads, but wget creates "*.1" files instead.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
242
1486
d398569effd2 Suppress some warnings.
Rob Landley <rob@landley.net>
parents: 1461
diff changeset
243 rm -f "$SRCDIR/$FILENAME"
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
244
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
245 return 1
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
246 }
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
247
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
248 # Attempt to obtain file from a specific location
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
249
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
250 download_from()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
251 {
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
252 # Return success if we already have a valid copy of the file
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
254 confirm_checksum && return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
255
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
256 # If we have another source, try to download file from there.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
257
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
258 [ -z "$1" ] && return 1
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
259 wget -t 2 -T 20 -O "$SRCDIR/$FILENAME" "$1" ||
1486
d398569effd2 Suppress some warnings.
Rob Landley <rob@landley.net>
parents: 1461
diff changeset
260 (rm -f "$SRCDIR/$FILENAME"; return 2)
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
261 touch -c "$SRCDIR/$FILENAME"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
262
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
263 confirm_checksum
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
264 }
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
265
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
266 # Confirm a file matches sha1sum, else try to download it from mirror list.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
267
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
268 download()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
269 {
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
270 FILENAME=`echo "$URL" | sed 's .*/ '`
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
271 [ -z "$RENAME" ] || FILENAME="$(echo "$FILENAME" | sed -r "$RENAME")"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
272
1335
ffb07a48a960 If sha1sum isn't installed, download falls back through all the mirrors repeatedly downloading stuff, without explaining why it failed.
Rob Landley <rob@landley.net>
parents: 1300
diff changeset
273 if [ -z "$(sha1sum < /dev/null)" ]
ffb07a48a960 If sha1sum isn't installed, download falls back through all the mirrors repeatedly downloading stuff, without explaining why it failed.
Rob Landley <rob@landley.net>
parents: 1300
diff changeset
274 then
ffb07a48a960 If sha1sum isn't installed, download falls back through all the mirrors repeatedly downloading stuff, without explaining why it failed.
Rob Landley <rob@landley.net>
parents: 1300
diff changeset
275 echo "Error: please install sha1sum" >&2
ffb07a48a960 If sha1sum isn't installed, download falls back through all the mirrors repeatedly downloading stuff, without explaining why it failed.
Rob Landley <rob@landley.net>
parents: 1300
diff changeset
276 exit 1
ffb07a48a960 If sha1sum isn't installed, download falls back through all the mirrors repeatedly downloading stuff, without explaining why it failed.
Rob Landley <rob@landley.net>
parents: 1300
diff changeset
277 fi
ffb07a48a960 If sha1sum isn't installed, download falls back through all the mirrors repeatedly downloading stuff, without explaining why it failed.
Rob Landley <rob@landley.net>
parents: 1300
diff changeset
278
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
279 echo -ne "checking $FILENAME\r"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
280
1552
c3b91b70cc42 Remove USE_ALT option, I have a better idea (upcoming patch).
Rob Landley <rob@landley.net>
parents: 1540
diff changeset
281 # Update timestamp on tarball (if it exists) so cleanup_oldfiles keeps it
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
282
1552
c3b91b70cc42 Remove USE_ALT option, I have a better idea (upcoming patch).
Rob Landley <rob@landley.net>
parents: 1540
diff changeset
283 touch -c "$SRCDIR"/"$FILENAME" 2>/dev/null
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
284
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
285 # Give package name, minus file's version number and archive extension.
1257
1461d20bb6a7 Genericize the noversion() regex a bit, and make sure to only apply patches in the sources/patches directory that end in ".patch".
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
286 BASENAME="$(noversion "$FILENAME")"
1553
da21ff27331e Add ability to check out repositories in packages, IGNORE_REPOS to build from tarballs anyway, a FAQ entry on it, and more/repo.sh to set up/update some of the common ones.
Rob Landley <rob@landley.net>
parents: 1552
diff changeset
287 ! is_in_list "$BASENAME" "$IGNORE_REPOS" &&
da21ff27331e Add ability to check out repositories in packages, IGNORE_REPOS to build from tarballs anyway, a FAQ entry on it, and more/repo.sh to set up/update some of the common ones.
Rob Landley <rob@landley.net>
parents: 1552
diff changeset
288 [ -d "$SRCDIR/$BASENAME" ] && echo "Using $SRCDIR/$BASENAME" && return 0
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
289
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
290 # If environment variable specifies a preferred mirror, try that first.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
291
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
292 if [ ! -z "$PREFERRED_MIRROR" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
293 then
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
294 download_from "$PREFERRED_MIRROR/$FILENAME" && return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
295 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
296
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
297 # Try original location, then mirrors.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
298 # Note: the URLs in mirror list cannot contain whitespace.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
299
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
300 download_from "$URL" && return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
301 for i in $MIRROR_LIST
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
302 do
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
303 download_from "$i/$FILENAME" && return 0
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
304 done
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
305
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
306 # Return failure.
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
307
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
308 echo "Could not download $FILENAME"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
309 echo -en "\e[0m"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
310 return 1
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
311 }
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
312
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
313 # Clean obsolete files out of the source directory
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
314
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
315 START_TIME=`date +%s`
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
316
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
317 cleanup_oldfiles()
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
318 {
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
319 # wait for asynchronous downloads to complete
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
320
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
321 wait
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
322
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
323 for i in "${SRCDIR}"/*
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
324 do
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
325 if [ -f "$i" ] && [ "$(date +%s -r "$i")" -lt "${START_TIME}" ]
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
326 then
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
327 echo Removing old file "$i"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
328 rm -rf "$i"
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
329 fi
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
330 done
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents:
diff changeset
331 }