Mercurial > hg > aboriginal

annotate sources/include.sh @ 1560:6bdc83f49920 1.2.1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Rename record-commands wrapper directory "record-commands". (Note: this means more/clean.sh takes it out.)
author Rob Landley <rob@landley.net>
date Tue, 13 Nov 2012 13:57:44 -0600
parents e2f722cc97a6
children 6a8114c148d1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
840
96b276eda7f1 Comment tweaks.
Rob Landley <rob@landley.net>
parents: 798
diff changeset
1 #!/bin/echo "This file is sourced, not run"
96b276eda7f1 Comment tweaks.
Rob Landley <rob@landley.net>
parents: 798
diff changeset
2
1207
4021fb1183d7 Environment sanitizing screwed up the cross-compiler.sh wrapper. Fix it, rename PROGRAM_PREFIX to TOOLCHAIN_PREFIX, and put a guard around sources/include.sh so it's safe to include multiple times.
Rob Landley <rob@landley.net>
parents: 1187
diff changeset
3 if ! already_included_this 2>/dev/null
4021fb1183d7 Environment sanitizing screwed up the cross-compiler.sh wrapper. Fix it, rename PROGRAM_PREFIX to TOOLCHAIN_PREFIX, and put a guard around sources/include.sh so it's safe to include multiple times.
Rob Landley <rob@landley.net>
parents: 1187
diff changeset
4 then
1504
eb4d0124767a Fix record-commands.sh regression with oldpath going recursive when sources/incldue.sh is source twice.
Rob Landley <rob@landley.net>
parents: 1502
diff changeset
5 alias already_included_this=true
1207
4021fb1183d7 Environment sanitizing screwed up the cross-compiler.sh wrapper. Fix it, rename PROGRAM_PREFIX to TOOLCHAIN_PREFIX, and put a guard around sources/include.sh so it's safe to include multiple times.
Rob Landley <rob@landley.net>
parents: 1187
diff changeset
6
840
96b276eda7f1 Comment tweaks.
Rob Landley <rob@landley.net>
parents: 798
diff changeset
7 # Set up all the environment variables and functions for a build stage.
96b276eda7f1 Comment tweaks.
Rob Landley <rob@landley.net>
parents: 798
diff changeset
8 # This file is sourced, not run.
138
7dffe9648a56 Installing bash but not using it. Yeah, there's a political agenda at work
Rob Landley <rob@landley.net>
parents: 125
diff changeset
9
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents: 1229
diff changeset
10 # Include config and source shell function files.
351
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
11
399
4e11cfb255e2 Break config out from include.sh, so environment variables users can edit to affect the build are collected together in one place with nothing else in it.
Rob Landley <rob@landley.net>
parents: 398
diff changeset
12 [ -e config ] && source config
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
13
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents: 1229
diff changeset
14 source sources/utility_functions.sh
744
759adf5a0fe9 Refactor so include.sh mostly just sets environment variables. Move read_arch_dir to function and call it explicitly (no more need for $NO_ARCH). Make blank_tempdir a function, called explicitly by stages when needed, with some sanity checks. Insert prerequisite tests to later stages so they can detect failure early and provide an explicit erro rmessage, and have those tests happen before blanking $WORK dir, to preserve debugging info. Make buildall.sh depend on prerequisite tests rather than trying to avoid calling later stages (and thus do flow control from asynchronous context). Add FAIL_QUIET option so buildall.sh doesn't spam the log with the new prerequisite error messages.
Rob Landley <rob@landley.net>
parents: 743
diff changeset
15 source sources/functions.sh
1253
d4b251e78fc0 Break out download_functions.sh into their own file.
Rob Landley <rob@landley.net>
parents: 1229
diff changeset
16 source sources/download_functions.sh
398
a7daae215647 Implement unstable package download logic, with examples. (Try "USE_UNSTABLE=uClibc,linux ./download.sh" for example.)
Rob Landley <rob@landley.net>
parents: 396
diff changeset
17
997
491eec5512dd Slightly tweaked bugfix from Wolfgang Denk: clean out environment variables from the host that would screw up the build.
Rob Landley <rob@landley.net>
parents: 984
diff changeset
18 # Avoid trouble from unexpected environment settings
491eec5512dd Slightly tweaked bugfix from Wolfgang Denk: clean out environment variables from the host that would screw up the build.
Rob Landley <rob@landley.net>
parents: 984
diff changeset
19
1175
886a2ea90bc1 Add sanitize_environment to unset unrecognized environment variables.
Rob Landley <rob@landley.net>
parents: 1174
diff changeset
20 [ -z "$NO_SANITIZE_ENVIRONMENT" ] && sanitize_environment
997
491eec5512dd Slightly tweaked bugfix from Wolfgang Denk: clean out environment variables from the host that would screw up the build.
Rob Landley <rob@landley.net>
parents: 984
diff changeset
21
972
cfa6262528f3 Genericize native build.
Rob Landley <rob@landley.net>
parents: 910
diff changeset
22 # List of fallback mirrors to download package source from
cfa6262528f3 Genericize native build.
Rob Landley <rob@landley.net>
parents: 910
diff changeset
23
1331
29e1df07e99a impactlinux.com is just an alias for landley.net now, remove it from mirror list.
Rob Landley <rob@landley.net>
parents: 1253
diff changeset
24 MIRROR_LIST="http://landley.net/code/aboriginal/mirror http://127.0.0.1/code/aboriginal/mirror"
972
cfa6262528f3 Genericize native build.
Rob Landley <rob@landley.net>
parents: 910
diff changeset
25
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
26 # Where are our working directories?
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
27
984
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
28 export_if_blank TOP=`pwd`
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
29 export_if_blank SOURCES="$TOP/sources"
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
30 export_if_blank SRCDIR="$TOP/packages"
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
31 export_if_blank PATCHDIR="$SOURCES/patches"
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
32 export_if_blank BUILD="$TOP/build"
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
33 export_if_blank SRCTREE="$BUILD/packages"
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
34 export_if_blank HOSTTOOLS="$BUILD/host"
1560
6bdc83f49920 Rename record-commands wrapper directory "record-commands". (Note: this means more/clean.sh takes it out.)
Rob Landley <rob@landley.net>
parents: 1544
diff changeset
35 export_if_blank WRAPDIR="$BUILD/record-commands"
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
36
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
37 # Set a default non-arch
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
38
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
39 export WORK="${BUILD}/host-temp"
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
40 export ARCH_NAME=host
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
41
351
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
42 # What host compiler should we use?
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
43
984
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
44 export_if_blank CC=cc
351
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
45
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
46 # How many processors should make -j use?
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
47
900
266dc7ea04c2 If there's enough memory (half a gig per processor), use 1.5x as many CPUs to try to keep 'em busy.
Rob Landley <rob@landley.net>
parents: 897
diff changeset
48 MEMTOTAL="$(awk '/MemTotal:/{print $2}' /proc/meminfo)"
351
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
49 if [ -z "$CPUS" ]
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
50 then
553
80f3356577fc Move kernel building from mini-native to package-mini-native in preparation for initramfs packaging option.
Rob Landley <rob@landley.net>
parents: 538
diff changeset
51 export CPUS=$(echo /sys/devices/system/cpu/cpu[0-9]* | wc -w)
351
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
52 [ "$CPUS" -lt 1 ] && CPUS=1
900
266dc7ea04c2 If there's enough memory (half a gig per processor), use 1.5x as many CPUs to try to keep 'em busy.
Rob Landley <rob@landley.net>
parents: 897
diff changeset
53
1476
a4714d7d6769 Tweak CPUS calculation again: don't overcommit when using hyper-threading.
Rob Landley <rob@landley.net>
parents: 1331
diff changeset
54 # If we're not using hyper-threading, and there's plenty of memory,
a4714d7d6769 Tweak CPUS calculation again: don't overcommit when using hyper-threading.
Rob Landley <rob@landley.net>
parents: 1331
diff changeset
55 # use 50% more CPUS than we actually have to keep system busy
900
266dc7ea04c2 If there's enough memory (half a gig per processor), use 1.5x as many CPUs to try to keep 'em busy.
Rob Landley <rob@landley.net>
parents: 897
diff changeset
56
1476
a4714d7d6769 Tweak CPUS calculation again: don't overcommit when using hyper-threading.
Rob Landley <rob@landley.net>
parents: 1331
diff changeset
57 [ -z "$(cat /proc/cpuinfo | grep '^flags' | head -n 1 | grep -w ht)" ] &&
a4714d7d6769 Tweak CPUS calculation again: don't overcommit when using hyper-threading.
Rob Landley <rob@landley.net>
parents: 1331
diff changeset
58 [ $(($CPUS*512*1024)) -le $MEMTOTAL ] &&
a4714d7d6769 Tweak CPUS calculation again: don't overcommit when using hyper-threading.
Rob Landley <rob@landley.net>
parents: 1331
diff changeset
59 CPUS=$((($CPUS*3)/2))
351
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
60 fi
7f529baf0b57 Reorganize include.sh to put config options at the top, minor cleanups.
Rob Landley <rob@landley.net>
parents: 340
diff changeset
61
984
9840847885e8 Add export_if_blank and make lots of build paths overrideable.
Rob Landley <rob@landley.net>
parents: 973
diff changeset
62 export_if_blank STAGE_NAME=`echo $0 | sed 's@.*/\(.*\)\.sh@\1@'`
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
63 [ ! -z "$BUILD_VERBOSE" ] && VERBOSITY="V=1"
615
7523d1f3b818 Some slight paranoia against multiple host-tools.sh runs that get interrupted in the middle. (Yeah, in general that's not guaranteed to work, but still...)
Rob Landley <rob@landley.net>
parents: 608
diff changeset
64
1544
e2f722cc97a6 Make busybox build by default, switch override knob to BUSYBOX=1 to use defconfig busybox, always use toybox for oneit, cleanup/fix record-commands logic.
Rob Landley <rob@landley.net>
parents: 1510
diff changeset
65 export_if_blank BUILD_STATIC=busybox,toybox,binutils,gcc-core,gcc-g++,make
1083
cb4dbdb7f2cd Make BUILD_STATIC take comma separated list of packages, or "all" or "none". Default behavior should remain the same.
Rob Landley <rob@landley.net>
parents: 1077
diff changeset
66
352
1782b77fae15 Add command logging. Set RECORD_COMMANDS=1 to log every command line run
Rob Landley <rob@landley.net>
parents: 351
diff changeset
67 # Adjust $PATH
1782b77fae15 Add command logging. Set RECORD_COMMANDS=1 to log every command line run
Rob Landley <rob@landley.net>
parents: 351
diff changeset
68
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
69 # If record-commands.sh set up a wrapper directory, adjust $PATH again.
1510
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
70
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
71 export PATH
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
72 if [ -z "$OLDPATH" ]
352
1782b77fae15 Add command logging. Set RECORD_COMMANDS=1 to log every command line run
Rob Landley <rob@landley.net>
parents: 351
diff changeset
73 then
1504
eb4d0124767a Fix record-commands.sh regression with oldpath going recursive when sources/incldue.sh is source twice.
Rob Landley <rob@landley.net>
parents: 1502
diff changeset
74 export OLDPATH="$PATH"
1510
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
75 [ -f "$HOSTTOOLS/busybox" ] &&
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
76 PATH="$(hosttools_path)" ||
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
77 PATH="$(hosttools_path):$PATH"
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
78
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
79 if [ -f "$WRAPDIR/wrappy" ]
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
80 then
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
81 OLDPATH="$PATH"
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
82 mkdir -p "$BUILD/logs"
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
83 [ $? -ne 0 ] && echo "Bad $WRAPDIR" >&2 && dienow
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
84 PATH="$WRAPDIR"
8e98296410f3 Another attempt to make more/record-commands.sh and host-tools.sh play together.
Rob Landley <rob@landley.net>
parents: 1509
diff changeset
85 fi
352
1782b77fae15 Add command logging. Set RECORD_COMMANDS=1 to log every command line run
Rob Landley <rob@landley.net>
parents: 351
diff changeset
86 fi
1544
e2f722cc97a6 Make busybox build by default, switch override knob to BUSYBOX=1 to use defconfig busybox, always use toybox for oneit, cleanup/fix record-commands logic.
Rob Landley <rob@landley.net>
parents: 1510
diff changeset
87 export WRAPPY_LOGPATH="$BUILD/logs/cmdlines.$ARCH_NAME.early"
352
1782b77fae15 Add command logging. Set RECORD_COMMANDS=1 to log every command line run
Rob Landley <rob@landley.net>
parents: 351
diff changeset
88
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
89 # Create files with known permissions
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
90 umask 022
608
ab76d4cdd392 Jean Wolter pointed out that running RECORD_COMMANDS=1 twice could create circular symlinks. Hopefully fixed now, and some related cleanups/clarifications.
Rob Landley <rob@landley.net>
parents: 597
diff changeset
91
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
92 # Tell bash not to cache the $PATH because we modify it. (Without this, bash
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
93 # won't find new executables added after startup.)
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
94 set +h
352
1782b77fae15 Add command logging. Set RECORD_COMMANDS=1 to log every command line run
Rob Landley <rob@landley.net>
parents: 351
diff changeset
95
897
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
96 # Disable internationalization so sort and sed and such can cope with ASCII.
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
97
850da666acc6 Redo command recording: no longer RECORD_COMMMANDS config entry, now sources/more/record-commands.sh run to set up the wrapper (either before or after host-tools.sh). General cleanup and simplification of code this interacted with, plus clean up bit rot in the reporting.
Rob Landley <rob@landley.net>
parents: 854
diff changeset
98 export LC_ALL=C
1207
4021fb1183d7 Environment sanitizing screwed up the cross-compiler.sh wrapper. Fix it, rename PROGRAM_PREFIX to TOOLCHAIN_PREFIX, and put a guard around sources/include.sh so it's safe to include multiple times.
Rob Landley <rob@landley.net>
parents: 1187
diff changeset
99
4021fb1183d7 Environment sanitizing screwed up the cross-compiler.sh wrapper. Fix it, rename PROGRAM_PREFIX to TOOLCHAIN_PREFIX, and put a guard around sources/include.sh so it's safe to include multiple times.
Rob Landley <rob@landley.net>
parents: 1187
diff changeset
100 fi # already_included_this