Sigh. Me and my big mouth.

The tl;dr version: somebody was an asshole on IRC and I responded in kind for about 15 seconds before /ignoring them, randomly rolled a critical hit on the parting shot, and thus got kicked out of a position in Funtoo development I'd never asked for in the first place. Oh well.

So hanging out on the #funtoo channel, one of the devs suddenly started going off on a random unprovoked rant about how "If welfare worked, I'd lose weight when thin people exercise", and so on.

This pissed me off, since my sister and her four kids have been on food stamps and living in heavily subsidized housing ever since her husband left her for the wife of some military guy deployed to Iraq. I've sent her tens of thousands of dollars over the years, in the past year alone I bought each niecephew a netbook, and flew all five of them to florida to see their great grandparents with us over christmas. My father and grandfather have also each sent her five figure amounts, but I haven't got the money to actually _support_ her, and if she moves she'd lose custody of her kids.

Another friend was recently homeless (she moved to Austin for a job right as the recession hit and the job wasn't here anymore when she got here because the _company_ went away, then she was tied to an apartment lease in a strange city but couldn't find a new job during her three months of saved living expenses, had her car reposessed shortly before she was evicted from her apartment, had her purse stolen her first week on the street with all her photo ID in it, and spent about nine months camped in a park before I found her). She's currently unemployed again (I helped her put her life back together enough to get a minimum wage clerical job but it only lasted a couple months), and now she's trying to survive a bad infection of both kidneys without health insurance (this is apparently why she hasn't been sleeping well since before christmas; as with all poor people she only went to the doctor when the problem wouldn't go away on its' own for a long time, so it's pretty advanced by the time it's diagnosed). She's on 1500 miligrams of antibiotics a day, which I gave her most of the money to pay for (she borrowed the rest from a friend, another dancer she met dancing at a strip club; which was the only job she could get without photo ID and turns out to pay horribly when four different managers want to be "tipped out" to the tune of $50 each every night; she wound up _losing_ money half the time, especially since her skin's a bit sun-damaged from holding a sign on street corners, which might make maybe half minimum wage on a good day and literally nothing on a bad day).

The clinic she went to (the emergency room just tried to give her Vicodin and send her away) says she really _needs_ to be on intravenous antibiotics, but she can't afford them, and the next appointment to see if she can qualify for medicaid isn't until friday. Last I heard one of her kidneys had already shut down. I'm seriously worried she's going to die, of something entirely treatable, because she hasn't got health insurance.

This is not "some anecdote". Her name is Heather, her 26th birthday is a week from Friday. I was hoping to get her dental work for her birthday (living on the streets is really hard on the teeth, first place she went to said a dozen teeth can't be saved and need to come out), but after the other medical bills I can't afford it.

Another friend is dead broke and moved back in with her father because she can't find a job (her degree is in mortuary science, kinda specific), and can't really search for one from his suburban house without a car. (She previously lived with her mother in NYC, and thus never got a driver's license. She's terrified of having to move back in with her mother "because of the roaches".) Her father recently had his _second_ sudden massive abdominal surgery (the details are horrific, but he survived and is doing well enough to have the colostomy bag removed soon), so now she's taking care of him. He's been underemployed since the dot-com crash caused his business to fold, and back on the market in his 50's he hit the age discrimination our industry's full of, so he had to start social security early (hence is getting less of it), then he tried to take up the post-mortgage collapse "refinancing" plans that basically meant the bank told him to stop paying his mortgage until the process completed, and then tried to reposess his house when he complied. (Ongoing legal battle there is in something like its third year, involving every form of malfeasance on the part of the bank you can imagine, including up through the "robo-signers".)

Another friend is currently broke and unemployed after high school, due to a motorcycle accident that hospitalized her (her knee is still screwed up due to severed nerves) that prevented her from starting college on time, and thus cost her a scholarship. She moved from Arkansas to Louisiana recently (more or less couch surfing), and tried to check into a mental hospital (feeling suicidal) which wouldn't take her.

Another friend's been hospitalized a couple times for stress (related to his father winding up in jail basically for life) and a back injury (he's in his 20's but fell down the stairs in his apartment), but luckily he had health insurance. Except now he's unemployed and paying something like $1000/month for Cobra, which has eaten through his savings and is currently going on credit cards (along with his rent and other living expenses) while he job hunts. He's a fairly recent college graduate with only ~3 years experience on his resume, so even though he's probably smarter than me it's a lot harder for him to find a job without moving to a strange city where he has no friends.

This is not an exhaustive list. My ex-roommate Reese was homeless for a while, she got back on her feet after I let her "rent" a room from me for a year and only pay for one month of that. Back when I dabbled in real estate in the late 90's I rented a place to a guy named Tim who had been homeless for a while before I met him (dug himself out of it waiting tables, eventually saved up and bought a trailer in a trailer park). My brother tried his own dabbling in real estate and the mortage crisis happened while he was trying to fix up and sell four properties (one of which he was living in), they all wound up getting reposessed, and he lost his job, so he moved back in with his father for a while.

So when this guy on the Funtoo list started randomly mouthing off about how safety net programs didn't help _him_ and were thus worthless, I told him to "die in a fire" before slash-ignoring him. This was my mistake.

A couple hours Daniel Robbins informed me that the guy's father _did_ die in a fire (about five years ago), and the police suspect him of arson and raided his house last week, so it was the most hurtful possible thing I could have said to him (which I honestly didn't know), and I "couldn't represent the Funtoo project" anymore, which actually comes as something of a relief.

I never actually asked to be on the Funtoo core team, and warned him I probably wouldn't have time to do much with it, I was just working on a technology he found useful. I've wanted to get bootstrap-gentoo working for _years_ and still do, and it's not actually that _hard_ since he and I did about the first third of it in a single evening when he visited Austin last month. But the bits I don't know are in _gentoo_, and learning the guts of gentoo just isn't all that interesting to me, so it's been on my todo list for years and hasn't quite made it to the top yet.

The reason it not being on this core team anymore comes as a relief is I no longer feel _guilty_ about not spending enough time on it. Funtoo wanted me to maintain wiki pages and regression test their "metro" image builder tool (which is another highly integrated thing like catalyst that's almost useless for the bootstrapping I want to do: it's simply not compatible with anything other than itself). These are all interesting todo items, but I have no time! (Heck, I never did set up a standalone funtoo system, and deleted my funtoo chroot while I was in Florida because my fiddling around had glitched it and my netbook was out of space anyway. Now I don't have to feel guilty about not finding time to set it back up.)

That said, yeah I shouldn't have said that to whoever that guy was, wouldn't have if I'd known the context, and would happily apologize to him. Turnabout is fair play and all, but I also had the option to be _better_ than him, and probably should have taken it. To do otherwise makes Wil Wheaton sad.

(I think the real lesson I'm taking away from all this is I shouldn't let it get to be afternoon without having eaten anything all day if there's any chance I'll have to interact with people.)

Other bugs in aboriginal linux:

1) Powerpc threading segfaults immediately on application launch. (Non-threaded stuff seems to work fine.)

2) Sparc threading hangs, and the g++ "hello world" build fails with an uknown link type.

3) If the network card config fails the CPUS count is still set to 3, but without distcc the builds don't ge distributed, and boards with only 256 megs of ram can't do -j 3 builds locally without dying.

All of these were there in the 1.1.0 release (and in fact sparc dynamic linking didn't work at all) so they're not _regressions_ and thus don't hold up the release, but I should fix both next time around. (And the proper fix to #3 is really to fix the network cards.

Still debugging the 3.2 kernel. Apparently everybody's network drivers went bye-bye and need a vendor symbol now (not just the intel stuff), and one of my long term todo items has been to switch more board emulations over to gigabit ethernet (which has less overhead and thus works faster than 100baseT or 10baseT emulations).

Unfortunately, enabling the intel E1000 driver on armv5l panics the kernel at some point after it loads. (Sometimes. Other times, it boots up but the interface still doesn't work, I don't think the device probe actually finds one even when I tell qemu to provide one using the same arguments that work on i686. Possibly the qemu code for that is target-specific, even though it's a PCI device.)

I got an ssh account on Daniel Robbins' fire-breathing 16-way server with 48 gigs of ram (actually an openvz container), and managed to wget the 3.2 kernel over there yesterday. Today I can't download it to my laptop from kernel.org, so I copied the one from that server.

Yes, the 3.2 kernel is out and even today kernel.org is melting under the strain of serving it. I'm told the local mirrors (eu.kernel.org and such) aren't back up yet. I never got a release out with 3.1 (upgraded the repository to that right after cutting 1.1.0 in October), so I waited for 3.2 to come out and now I'm trying to get 3.2 out promptly to make staying in sync with kernel releases a bit easier.

One of the perl removal patches needed to be rediffed, but that was just to remove fuzz; otherwise still works just fine. The sparc relocation fix for 3.1 is already upstream, so yank that.

In 3.2, the network card config symbols have been deeply screwed up for some reason: now you need to specify which vendor the cards belong to. If you want the E1000 driver, you have to add CONFIG_ETHERNET and CONFIG_NET_VENDOR_INTEL, for no apparent reason. I don't know why they did this. Ask commit dee1ad47f2ee7.

I'm mucking about with the early boot code of a Texas Instruments board at work, which means I'm reading through the guts of U-boot.

Bootable kernels tend to make an ELF image, and then run it through "objcopy -O binary elffile binfile" to create a fully linked runnable binary blob of raw machine language, with a fixed load address and start address, that can actually run unmodified on the hardware.

In Linux, this ELF file is called "vmlinux", and in U-boot it's "u-boot", and both are still around after the build if you want to play with them. (QEMU can cheat slightly: it contains an ELF loader that does what objcopy does, and in the process it can determine what load address and start address to use from the ELF metadata. So "qemu -kernel vmlinux" (or u-boot) works on some targets. One of my todo items is to make it work on all of them, the fiddliness is how to feed in the kernel command line.)

The start address generally corresponds to the symbol "_start" in the ELF file (it's actually a field in the ELF header, but the location of _start is the default that field gets set to by the ELF tools), which is some assembly function that does the really low level setup necessary before it can jump to C code. You have to set the processor into the right mode, make sure problematic hardware we haven't configured yet is switched off (disable interrupts and Memory Management Unit), and set the stack pointer to a chunk of memory so C has local variables and so the first C function can call other functions and return from them.

The first thing the C code generally does set up the DRAM controller to start refreshing memory at the correct rate, so data written to DRAM actually stays there. Until the DRAM controller is set up (which is nontrivial, since the same hardware has to work with different brands/speeds/sizes of memory that get refreshed in different ways), all reads from DRAM return random garbage.

This raises a couple of interesting questions. The first is "How does u-boot run before the DRAM init happens?" By running out of something other than DRAM. Your board has to have some non-volatile memory such as ROM or Flash to provide a boot program when you switch it on, which has the disadvantage of being insanely slow compared to DRAM, but once the DRAM is initialized you can copy the rest of your code into DRAM and run from there.

The other question is: where do you get memory for the C stack? In the olden days, C's need for a stack led BIOS vendors to write the DRAM controller init routines in assembly language using no storage other than the CPU registers, which was horrible and is why BIOS development used to be black magic nobody understood. Then the "coreboot" project came up with the a trick of repurposing the processor's data cache via a static TLB entry, so a chunk of address ranges were stored in cache instead of DRAM, providing enough stack space to run a DRAM controller setup function in C. (And there was much rejoicing.) You only need to do it the old way on processors with no L1 or L2 cache (which are obsolete these days because the latency of DRAM fetch from chips a couple inches away does not mix well with modern clock speeds: just sending a signal down that much wire is several clock cycles' round trip, let alone having the DRAM circuitry actually look stuff up).

In general, having more assembly code than necessary raises maintenance problems: it's extremely awkward to write and debug assembly compared to C, and every processor has its own slightly different assembly language which few people know all that well. You generally want to Jump to C code as fast as possible, where the potential number of people who can review/maintain your code increases by an order of magnitude and you have many more tools at your disposal.

Alas, somebody needs to tell Texas Instruments this.

In this case, Texas Instruments went with an overcomplciated 3 layer approach, plus extra hardware. It includes a ROM in its system-on chip and builds U-boot twice. The ROM loads the first instance of U-boot (called "MLO" for some reason) from an SD card into 256k of dedicated on-chip memory (which is not the same as the L2 cache, and is thus mostly wasted after boot time). The ROM has to have a device driver to talk to the SD card through an MMC bus and parse the FAT filesystem to find and read the MLO file. The MLO instance of U-boot has to have its own MMC+SD+FAT driver to load a bigger U-boot into DRAM after it's initialized the DRAM refresh circuitry. And then that U-boot has to have a third instance of the same controller in order to load Linux from the SD card.

Yes, really.

In the u-boot board I'm working on right now, _start is defined in the file "arch/arm/cpu/arm_cortexa8/start.S", and it does _not_ do the minimal work necessary to jump to C. Instead, Texas Instruments wrote extensive setup code in assembly language.

The _start code begins with a branch to the label "reset:", jumping past some memory used to store variables. The reset routine sets the cpu to SVC32 mode, and then has a large blob to "copy vectors to mask ROM indirect addr" inside an #ifdef CONFIG_OMAP34XX that I really hope doesn't apply to the board I'm using. (I doubt Wolfgang Denk would allow such a horror upstream into his code, most likely it's only present in the vendor fork I'm using.)

Next it then optionally calls cpu_init_crit. This disables the processor cache and MMU (in case something left them on), and then calls lowlevel_init which is board-specific init (in this case in arch/arm/cpu/arm_cortexa8/ti81xx/lowlevel_init.S . What does "optionally" mean here? It's in an #ifndef CONFIG_SKIP_LOWLEVEL_INIT, but that doesn't seem to be set in our board's include/config.h or the files sourced from it.)

Lots of plumbing in lowlevel_init.S involves NOR flash, which our board config doesn't seem to have enabled either. The lowlevel_init: label is at line 383 in the source I'm looking at, and minus the NOR bits it does two things: 1) Set the stack pointer to the end of the first block of physical memory (SRAM0_START+SRAM0_SIZE-4). 2) Try to figure out if we're running from DRAM already or not.

Bootable kernels tend to make an ELF image, and then run it through "objcopy -O binary elffile binfile" to create a fully linked runnable binary blob.

The rest of this blog entry just explains that statement.

The ELF file format is a container format (like zip/tar/cpio or one the "ar" tool does for *.a static libraries). Executable files, shared libraries, and the *.o files created by compiling source code are all different kinds of ELF files.

Instead of a bunch of separate files, this format is optimized to contain chunks of program code. The archive's metadata describes what kind of code it contains (armv7 little endian 32 bit using the Thumb2 extensions...), and lists a bunch of named "symbols" describing the chunks of data in the archive. This includes "sections" of memory (with attributes like read-only or zero filled), plus the variables and functions that live in those sections. Each variable or function includes a size in bytes, the section name it lives in, the offset within that section it starts at, and so on. A bunch of tools can show this data, but "objdump -x" and "readelf -a" the big ones.

The Linux kernel contains an ELF loader that can parse ELF files and load them into memory, sometimes delegating to a dynamic linker (another program listed in the ELF headers as responsible for dealing with this program).

But running hardware needs raw machine language, so at boot time you need a full linked binary blob, a "load address" to copy it into memory at, and a "start address" to jump to. (Sometimes the start address is the same as the load address. Sometimes a jump to the real start address is inserted at the very start of the file.)

Bootable kernels tend to make an ELF image, and then run it through "objcopy -O binary elffile binfile" to create a fully linked runnable binary blob with a known load address and start address. In the case of Linux, the ELF file is called "vmlinux". In the case of u-boot, the elf file is called "u-boot". They're still around at the end of the build, and you can play with 'em if you like. (If you ever hook a debugger up to a JTAG interface to debug a running kernel, loading the ELF file into the debugger gives it all the symbol information to provide symbol and function names instead of raw memory addresses for everything.)

A "relocateable" program can be loaded and run from any memory location, but it has to either be REALLY carefully programmed (only using relative addresses that are a given distance forward or backward from the current location), or use some sort of wrapper has to run on it to adjust all its addresses with regards to where it's been loaded this time around. The relocation code has to be one big function using entirely local variables: it can't access any globals or call any other functions because those would have to be relocated, which is a chicken and egg problem.

Another common kind of wrapper decompresses the program, so the binary you actually load can be compressed and thus much smaller. Both kind of wrappers run first, and then jumps to the real starting location once they've doen their work on the runnable image. This is a form of relocation, but the wrapper is just copying the output to a known location so it doesn't actually have to patch all its jump and read/write instructions that use an address: those can still be fixed at build time.