Toybox vs Busybox, Rob Landley (rob@landley.net) https://patreon.com/landley ---------------------------------------------------------------------------- Video: https://www.youtube.com/watch?v=MkJkyMuBm3g Description I gave to the program committee: So why would you use toybox instead of busybox, the gnu tools, or something else? When you build an embedded system, what are your options and what do they mean? This talk is an attempt to survey, compare, and contrast. Time they gave me to answer: 35 minutes. (Scheduled against 18 other panels!) Outline - motivation and goals (why am I doing this, what am I trying to accomplish?) - busybox vs toybox is A) Aboriginal Linux vs mkroot, B) Linux vs Android - What _did_ I do? - busybox timeline - license issues - GPLv3, 0BSD - toybox timeline - wilderness, targeting android, merge, 1.0, need to redo AOSP - tech Quick look at the code Quick look at _using_ each project I was maintainer of busybox, left to create toybox. I have extensive experience with both, and obviously I would say use toybox. Why? - motivation and goals (why am I doing this sort of thing)? What was I trying to accomplish? longer version at https://landley.net/aboriginal/history.html knoppix - linux from scratch + tomsrtbt https://en.wikipedia.org/wiki/Knoppix https://en.wikipedia.org/wiki/Tomsrtbt http://www.linuxfromscratch.org/lfs/view/stable/ If I could save Knoppix 100M of their 700M CD budget, they could work wonders - I made build systems 2001-2006 Firmware Linux https://landley.net/aboriginal/old replaced 20 LFS packages with busybox: bzip2, coreutils, e2fsprogs, file, findutils, gawk, grep, inetutils, less, modutils, net-tools, patch, procps, sed, shadow, sysklogd, sysvinit, tar, util-linux, vim. still todo at that time: bash, diffutils... That built a uClibc system and booted it under User Mode Linux. Then QEMU came out and I got into cross compiling, started over... 2006-2016 Aboriginal Linux https://landley.net/aboriginal/about.html Aboriginal Linux: 7 packages 7 packages: busybox, uclibc, linux, gcc, binutils, make, bash That built its own toolchain, and I'd stayed on the last GPLv2 releases of gcc/binutils/make/bash which got increasingly stale. No external cross compilers also provided _native_ compilers for the target, until I made puppy eyes at https://github.com/richfelker/musl-cross-make... 2016-2019 mkroot (now part of toybox, "make root") previously at https://github.com/landley/mkroot theoretically 4 packages: kernel, command line, compiler, libc - linux, toybox, tinycc/qcc, musl/bionic? currently builds toybox+linux, boot to shell prompt todo: https://landley.net/code/qcc (glue qemu tcg to tinycc) http://landley.net/hg/qcc/file/tip/todo/todo.txt todo: write make/awk/flex/yacc for toybox. todo: mount native compiler from .sqf files ala https://landley.net/aboriginal/control-images todo: new cfront: llvm C target https://github.com/JuliaComputing/llvm-cbe Aboriginal Linux drove busybox development, and early toybox development, circa 2001-2011. Replaced by Android (2011-now) and mkroot (2016-now). Busybox timeline: - March 2001: my first busybox bug report (git 8b960321393f) - Sep 2003: my first patch to the list (8aac05bfe5ff) - Jan 4, 2004: First patch committed by me (they forced me to use the source control system instead of mailing patches to the list: 40ec4aeb8e26) - October 4, 2004: Busybox 1.00 release 9395ca4f6832 - Erik goes limp, busybox "finished", work on uClibc and buildroot. I poke him http://lists.busybox.net/pipermail/busybox/2005-August/049489.html - August 17, 2005: I put together a 1.01 release "accumulated bugfixes". - I created the tarball, Erik uploaded it. - That month I posted to the list 137 times, Erik posted 8. You can see which way this is going. - Jan 14, 2006: 1.1.0 release, first dev release under new maintainer. http://lists.busybox.net/pipermail/busybox/2006-January/051694.html - Feb 14, 2006: I became busybox maintainer (a253e7361f8f) - https://landley.livejournal.com/26529.html - In practical terms, it meant I get root access (sudo) on the server. - Erik was happy to be out: https://landley.livejournal.com/29498.html - March 27, 2006: Announce SFLC representing busybox (0949d3063889) - I inherited a "hall of shame" page, found pro-bono legal representation. https://git.busybox.net/busybox/tree/docs/busybox.net/shame.html?id=fe3d844bb158 https://www.networkworld.com/article/2285948/lawsuit-charging-gpl-violation-is-first-ever.html - Sep 30, 2006: Resigned as busybox maintainer over GPLv3 dispute - https://lwn.net/Articles/202106/ (busy busy busybox) - https://busybox.net/~landley/forensics.txt - handoff to Denys Vlasenko after 1.2.2 release Why did I switch from busybox to toybox? I left busybox due to an argument over GPLv3. https://lwn.net/Articles/202106/ (busy busy busybox) Took all the fun out of it (reminded me of "SCO Disease") Did toybox as a hobby for a bit, no real expectation anybody use it. Even came back to busybox for a while, on and off, circa 2008-2011 sent toybox patch to busybox, fixed their vi, nbd-client... http://lists.busybox.net/pipermail/busybox/2008-October/067397.html http://lists.busybox.net/pipermail/busybox/2010-September/073311.html https://landley.net/notes-2015.html#30-07-2015 Tim Bird "bentobox" project, 2011. https://www.elinux.org/Busybox_replacement_project I already knew Android was taking over (main->mini->micro->phone) http://www.catb.org/~esr/halloween/halloween9.html#id2867629 http://catb.org/~esr/writings/world-domination/world-domination-201.html http://landley.net/notes-2011.html#26-06-2011 But Tim thought it was worth doing something about, and he'd contacted the big players already to confirm nobody ELSE was doing it, so... Alright, if nobody else is going to fix this, I'll give it a go. - imagine an 8 year old girl in india inherits her mother's old phone, pulls a USB hub, usb keyboard, usb mouse out of trash, exports display to HDTV with chromecast (or USB->HDMI adapter). - That's the HARDWARE of a workstation. Just needs software. - She should be able to recompile AOSP under Android, using what comes in the box. (Old phone, slow internet, don't require downloading lots of crap from a play store version that may not still be up.) - Having an Android device should be sufficient to become a first class Android developer. Other systems may be _faster_, but there's nothing she shouldn't be able to DO on an Android device that NEEDS a PC she hasn't got. - Busybox (1.0 in 2004) predates Android (released 2007). If Android wanted to use busybox, it already would. Waiting around for it to change its mind: not gonna happen. Toybox timeline: Sep 27, 2006: git init toybox - Still working to make Aboriginal Linux rebuild itself under itself, didn't quite do so with busybox yet, so write new toybox commands to replace the busybox commands that aren't "there" yet. 2007, 2008: lots of work to advance that goal, mostly on a-l mailing list. http://lists.landley.net/pipermail/aboriginal-landley.net/ Apr 24, 2008: last commit to my tinycc fork https://landley.net/code/tinycc - I'd maintained it on and off for about 3 years. - Aboriginal Linux was stuck on the last GPLv2 releases of gcc/binutils, which were increasingly stale. Tinycc only the 3rd compiler ever to build Linux kernel https://bellard.org/tcc/tccboot.html https://bootlin.com/pub/video/2008/ols/ols2008-rob-landley-linux-compiler.ogg - But it didn't build a current vanilla kernel, had lots of pending todo items. Upstream abandoned (Fabrice Bellard went on to do QEMU) and repository was in CVS (ick), so I decided todo them in my own fork: https://bellard.org/tcc/tccboot_readme.html https://landley.net/hg/tinycc - After ~3 years I got tired of fighting zombie "upstream" that overshadowed my project Linux Weekly news covered THEIR releases, but not mine. yet they never advanced towards my goals (build unmodified Linux kernel and LFS) except by occasionally copying part (but not all) of my code. - emotional parallels to toybox/busybox should be obvious. Dec 13, 2008: FSF lawsuit against Cisco absolutely INSANE. Disgusted with GPL enforcement. https://landley.net/notes-2008.html#13-12-2008 Jan 19, 2009: Busybox 0.0.8 release. Toybox development pauses: lost interest. - Feb 26, 2009: ponder rewriting toybox in lua https://landley.net/notes-2009.html#26-02-2009 - June 18, 2009: blogged about toybox being mothballed, compare to busybox https://landley.net/notes-2009.html#06-08-2009 - Aug 24, 2009: Blogged that I should work on busybox again, but it was no fun https://landley.net/notes-2009.html#24-08-2009 - Jan 5, 2010: Blogged about maybe folding toybox into busybox https://landley.net/notes-2010.html#05-01-2010 April 12, 2010: Met with Denys Vlasenko in person at ELC, explain toybox to him - https://landley.net/notes-2010.html#12-04-2010 - May 9, 2010: Denys adds toybox-style config to busybox (git 7fb68f199f03) - not how I would have done it, but... less bad. - Aug 13, 2010: submit toybox patch.c to busybox (bb git 1bbc0cd7f293) Sep 5, 2010: Aboriginal Linux 1.0 release using busybox - rebuilds itself under itself, from source. That was the goal I'd been working towards. Now what? - New goal distro bootstrapping (build fedora/debian/gentoo under it). - turns out to be hard. Fedora/debian/gentoo only build under themselves. Circular dependencies, completely undocumented. Gento worst of the lot! - define new goals for 2.0: https://landley.net/aboriginal/about.html#next - replace GPL packages with BSD packages - New goal distro bootstrapping (build fedora/debian/gentoo under it). - turns out to be hard - Oct 10, 2011: my most recent commit to busybox repository (39ec6a2ad5da) Nov 13, 2011: blog entry about Tim Bird's "bentobox" project, poke at relicense http://landley.net/notes-2011.html#13-11-2011 - Nov 14, 2011: more blog: create new toybox mailing list, relicense to BSD see also http://lists.landley.net/pipermail/toybox-landley.net/2011-November/003802.html - Nov 15, 18, 19, 20, 2011: blog entries about making roadmap. https://landley.net/toybox/roadmap.html - Dec 3, 2011: blog, applying aboriginal's record-commands analysis to toybox https://landley.net/notes-2011.html#03-12-2011 - Dec 16, 2011: blog about how the GPL is dying due to GPLv3 https://landley.net/notes-2011.html#16-12-2011 Jan 18, 2012: convinced musl-libc maintainer to switch from lgpl to bsd license Complained on my blog: https://landley.net/notes-2012.html#18-01-2012 Followed up in email and IRC, convinced him BSD was better way to go, he did May 14, 2012: Got permission from Fabrice Bellard to relicense tinycc as BSD https://landley.net/notes-2012.html#14-05-2012 Feb 17, 2013: Document new Aboriginal Linux goals (git 84ae89737649) Now have (potential) BSD licensed userspace! toybox, musl, tinycc! - new goals for 2.0: https://landley.net/aboriginal/about.html#next - replace GPL packages with BSD packages (and 7 packages -> 4 packages) - distro bootstrapping (fedora, debian, gentoo etc) - Tackle Android (AOSP) as the toughest "distro" hairball to unravel. Feb 21, 2013: ELC "why is toybox" talk. https://landley.net/talks/celf-2013.txt Jun 4, 2013: Ask Googles' head of open source development to merge toybox He said no: https://twitter.com/landley/status/342138988779433984 Kept working anyway. (Plan: squashfs approach. Convince each "distro" to install it, then get it upstream once everybody's already using it.) Nov 7, 2013: Wrote a "history of toybox" blog entry to correct Wikipedia https://landley.net/notes-2013.html#07-11-2013 2015: Google merges toybox https://lwn.net/Articles/629362/ (android gets a toybox) 2017, 2018: Made it build with Android NDK, etc. http://lists.landley.net/pipermail/toybox-landley.net/2017-April/008970.html Nov 2018: Android "hermetic" builds. https://twitter.com/landley/status/1064582061464449026 Why you might like toybox. - simplest possible development environment. - Android workstations. - less legacy cruft than busybox - no #ifdefs in .c or .h files except pending - help text for every command. - complete usable documentation, should be man page replacement. - same as menuconfig help text - SPOT: single point of truth - nommu support: one codepath, use vfork() always, deal with it. - proud of ps, ls - toybox uuencode vs busybox uuencode - busybox does complicated tricks to produce smaller binaries. - license clarity - 0BSD license. A BSD license (OpenBSD suggested template license and I got permission from Marshall McKusick to call it 0BSD) that's also a public domain equivalent license. (No stuttering problem.) - Busybox ping is BSD and GPL, but it wasn't dual license from same people. - I had to police SO much license nonsense as Busybox maintainer... - Portions of it build on FreeBSD and MacOSX. Source code and infrastructrue comparison: busybox top level dir: 15 files, 33 directories explain arch/ qemu_multiarch_testing klibc-utils/minips.c: a thing it says it shouldn't be doing. - toybox derefrences one level of symlinks when it can't understand a name. So if you "ln -s ps minips" and run minips, toybox acts like ps. Generic solution to the problem, not special case whack-a-mole. examples/linux-2.6.30_proc_self_exe.patch (from 2009) examples/devfsd.conf examples/udhcp: why doesn't it work without one? names of other packages: coreutils, findutils, e2fsprogs, klibc-utils... Toybox hasn't got an "upstream". It is its own, legitimate, standalone implementation of command line utilities. - bell labs, SysV, BSD, Coherent, Minix, Linux. libbb: 147 files, a megabyte of code. libbb/common_bufsiz.c config FEATURE_USE_BSS_TAIL toybox top level dir: 7 files, 6 directories LICENSE: the toybox license (zero clause BSD, a public domain equivalent) README: documentation, start reading here www: The website files. Most of the documentation lives here. Config.in: top kconfig file that "sources" others. "make menuconfig" starts here. The --help text comes from here too. configure: sets environment variables. People want to ./configure; harmless Makefile: type "make help" to see your options. Should just be a convenience, the scripts/*.sh do the actual work and can be run directly. scripts: shell scripts called by the Makefile. Also 4 .c files built and run on the host as part of the build; those binaries don't ship. kconfig: snapshot of the menuconfig infrastructure from 2.6.12, one of my big remaining todo items is to write a replacement from scratch. configure: sets environment variables. People want to ./configure; harmless tests: the test suite: make defconfig; make test_seq. (P.S. "make change") Try "VERBOSE=fail make test_sed" or "VERBOSE=nopass make tests" main.c top level toybox file. The main() function is at the end of this, rest of the file is setup code for the toybox multiplexer (command dispatcher) 259 lines, 7.6 k. toys.h Top level #include file, included by each app. #includes all the others. lib: shared code avilable to all apps. du -s lib: 216k, in 19 fiiles. toys: source code to all the actual commands ls -lSr toys/*/*.c | grep -v pending ps.c is 5 commands (ps, top, iotop, pgrep, pkill) At 1935 lines it's the largest file, about twice as large as I like to see. Next is sed at 1066, just over. wc toys/*/{ps,sed,tar,bzcat,find}.c Infratructure: busybox menuconfig, scroll down through sources menu, through coreutils suboptions mean two busybox builds may behave very differently. "Does busybox date support -I" is not a question with one answer. This is a design decision busybox made. toybox menuconfig toybox global settings: sane defaults, can safely ignore whole dir very few suboptions, trying to get rid of them. "How this command behaves in toybox" should have ONE answer. Building with various toolchains CROSS_COMPILE=/dir/prefix- LDFLAGS=--static make distclean defconfig toybox tested with musl-cross-make, android ndk, AOSP prebuilts... make sed, make test_sed, make change make help Adding a new command is adding a single file. I usually copy hello.c https://landley.net/toybox/code.html#adding walk through hello.c adding a new toys/dir menuconfig is just: 1) mkdir toys/newdir 2) echo "description line" > toys/newdir/README 3) make distclean menuconfig toybox tries to have clean code: lib/portability.h and lib/portability.c - entry point of busybox it's in libbb/appletlib.c (obviously, I should have guessed) SINGLE_APPLET_MAIN -> ENABLE_FEATURE_INDIVIDUAL Marshalling information between symbols, vs SPOT Micromanaging: calculate bunzip2 inclusion threshold? FAST_FUNC? MAIN_EXTERNALLY_VISIBLE #if 0 main() function is in the #else case of an ifdef on line 1033, with UNUSED_PARAM macros, followed by an #if 0 block, followed by #ifdef salad compare with toybox: the only top level .c file is main.c which ends with function main() toybox tries to have no #ifdefs in .c files. $ grep '#if' *.c toys/*/*.c | grep -v pending | wc -l 16 - that's too many, trying to remove about half - they _belong_ in lib/portability.h note: more comments than code. Trying to explain what it's doing. toybox lib/portability.{h,c} The busybox license situation - libbb/appletlib.c is GPLv2 "or later". Why? Busybox is v2? - toybox "ping.c" has GPL boilerpate at start and BSD boilerplate at end - GPL says no additional restrictions, BSD says copy specific blob of text into all drived works, we pretend these don't conflict. No troll has sued over this yet. That goddess SCO didn't think of it. Which brings us back to licensing: Licensing! Left because GPLv3. Android no GPL in userspace. (Trademark licensing guidelines.) Wrote new apache licensed bluetooth daemon to replace GPL one. https://lwn.net/Articles/597293/ - not gonna happen Apple "great GPL purge". http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/ Stayed on gcc 4.2.1 with binutils 2.17 in xcode for 5 years while sponsoring LLVM. Android built "M" with LLVM, deprecated GCC, and yanked from NDK. GPL terminal node.... "GPL vs bsd" is actually copyleft vs public domain. Public domain _adjacent_ license proliferation/fragmentation. Zero clause BSD GPLv3 killed copyleft. There's no such thing as "the" GPL anymore: the linux kernel and samba implement 2 ends of same protocol, are both GPL, can't share code. A "GPLv2 or later" project can't accept code from _either_. - Fragmentation increasing: GPL-next, AGPL... Nobody under 30 thinks GPL a good idea anymore. Copyleft is riding down the half-life of existing GPL codebases and developers. - Percentage of repositories licensed AT ALL declining: https://speakerdeck.com/benbalter/open-source-licensing-by-the-numbers?slide=41 Younguns lumping software copyrights in with software patents, "too dumb to live", opt out entirely. - problem: while we're all waiting for the Boomers to die and get universal basic income, we can't use your code in the meantime. - Solution: promote public domain equivalent licensing, to replace copyleft, as the way to "fight back" against IP overreach. - Problem: most existing public domain equivalent licenses terrible, scare off either hobbyists or corporations. I created a new license for toybox In 2011 I used the OpenBSD suggested template license as my "BSD license" https://www.openbsd.org/policy.html It was far and away the simplest "BSD" license. Easy to read/understand. In 2013 I removed half a sentence, creating a new license: http://lists.landley.net/pipermail/toybox-landley.net/2013-March/004589.html https://github.com/landley/toybox/commit/ee86b1d8e25c I got permission from Kirk McKusick to call new license "Zero Clause BSD" SPDX approval in 2015 (SPDX: 0BSD) https://lists.linuxfoundation.org/pipermail/spdx-legal/2015-June/001443.html Richard Fontanna at OSI made a mistake: https://lists.linuxfoundation.org/pipermail/spdx-legal/2015-November/001566.html Richard Fontanna's defense of his mistake caused much market confusion: https://github.com/david-a-wheeler/spdx-tutorial/issues/1 https://github.com/github/choosealicense.com/issues/464 OSI brought into alignnment last year: http://lists.opensource.org/pipermail/license-review_lists.opensource.org/2018-September/003519.html Github added 0BSD to choose-a-license plumbing https://github.com/github/choosealicense.com/issues/642 Public domain equivalent license: CC0, unlicense, wtfpl simple statement ala libtommath/libtomcrypt in dropbear use public domain and stamp another license on it and THEN it's ok? I almost went with CC0, but got strong pushback due to... FUD campaign against public domain When copyright was extended to cover binaries by: https://en.wikipedia.org/wiki/Apple_Computer,_Inc._v._Franklin_Computer_Corp. the main competitor to the newly created shrinkwrap software gold rush was 30+ years of accumulated public domain code. They created all the FUD they could around it, the way bottled water sellers badmouth tap water. OSI's lawyer Larry Rosen did an outright hit piece in 2003: https://www.linuxjournal.com/article/6225 Paragraph 5 compares releasing code into the public domain to dumping trash by the side of a highway, I.E. littering, implies it should be a prosecutable offsense. The lawyer for the Open Source Initiative said this! https://www.ted.com/talks/clay_shirky_on_institutions_versus_collaboration Hence creation of "public domain adjacent" licenses, which are almost public domain but require you to copy a specific blob of license text into derived works Results in "the stuttering problem". https://android.googlesource.com/platform/system/core.git/+/711e776cc21d/toolbox/NOTICE (Because the dates changed, and a strict reading of the license says...) Kindle "paperwhite" had 325 pages of about->license pulldown. So... what license applies to any specific piece of code? Are you sure? Wanted to use a "BSD license" for a _reason_: Lawyers dislike being cut out of future business: https://www.openwall.com/lists/musl/2016/03/23/11 But BSD is good license because AT&T vs BSDI spent LOTS on lawyers. https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc._v._Berkeley_Software_Design,_Inc. Traditional "GPL vs BSD" axis (actually copyleft vs public domain) - BSD used as "shorthand" for public domain adjacent (ISC, MIT, Apache...) Name recognition! Already 2 clause, 3 clause, 4 clause... zero clause gets rubber stamped. - it's a family, exact wording varies. Result: public domain equivalent that _looks_ public domain adjacent. A) camouflage! B) must explain why half-sentence so vital, nowhere to hide I.E. a corporate-friendly public domain equivalent license. https://en.wikipedia.org/wiki/Public-domain-equivalent_license Toybox has explicit license text at the top of the project, and copyright notices on all the files, to make the lawyers _comfortable_. - the internet cares about attribution, not ownership. (Very good at finding plagiarism.) The copyright notice is attribution, "Thanks X for this one!" - Note: only the first paragraph of 0BSD means much, the second is there for camouflage (look like a BSD) and again to make lawyers comfortable. Disclaimer paragraphs are left over from the 1983 shrinkwrap gold rush when existing licenses were all for custom software written for big iron (which didn't depend on copyright but on contract law; they got paid _then_ wrote it). These contracts were for installing brand new (hence lightly tested) software on $50 million of hardware, and if your installation techs unexpectedly took the machine offline for 3 days you cost them a year's salary in depreciation alone, and they'd totally sue. This never applied to a home appliance (do you sign a beverage waiver in case it leaks and stains your couch?), but the contract boilerplate was copied over by the early garage operations to "look professional", and became https://www.snopes.com/fact-check/grandmas-cooking-secret/ The disclaimer was there in the OpenBSD license and was kept to minimize the difference between 0BSD and OpenBSD. It makes the license look bigger, look professional, look meaty, soothes the lawyers... but people are giving legal and medical advice on youtube to millions of subscribers, half the free phone games are trying to mine bitcoins in the background, windows needed to suddenly reboot twice a day for 30 years, and you downloaded something for free off the web. This is not the same legal context as bespoke development for 1982 mainframes. Meanwhile, 0BSD isn't an unreasonable burden for open source programmers, and it's a recognized option on github now: https://choosealicense.com/licenses/0bsd/ If you want to "opt out" of copyright, use 0BSD licensing.