Met mark around lunchtime and got the bank account for the new company open. Now to figure out whether to try to launch it slowly part-time over the next couple years while working a day job (like Mark's doing), or jump in with both feet. The second's probably more likely to work, but as of yet I have no actual customers for it...
Ok, the new patch is in, and that should be a toybox 0.0.4 release. I'm test-building Firmware Linux with it to make sure everything else is happy. (I think I already fixed touch... Yup, checkin 218.) I have one more todo item for the toybox release (the readlink -f stuff Roberto Foglietta's been sending me) but that should be out this evening.
The two other projects I might syncopate a bit, because although scheduled releases are great, having releases of all three of my projects at the same time could get a little inconvenient going forward. And in any case, I need to do a little more banging on both firmware and tinycc.
Heard from the Google recruiter today, who says they might be starting a new branch in Austin. I've heard this rumor before, but it would be nice if it became true someday. The main reason I haven't gone to work for Google is they keep wanting me to move to California. (The job the recruiter contacted me about is, once again, in california. And they want somebody to manage kernel developers, which I could do but which combines two things I've only flirted with before. (Management, and kernel development.) I've done both, but they tend to grab your attention to the exclusion of all else, and I've preferred to remain a generalist. I suppose as I get older, trending towards management is sort of inevitable...
Switching back to a day schedule, mostly offline.
The downside of netflix is the frequency with which you get damaged DVDs, and have to stop watching stuff halfway through until they send you a new disk three days later. (Or, in the case of the first pressing of "The Christmas Invasion", until you get your own box set as a wedding present, three months after they yanked the entire production run but have yet to re-issue it or even comment on when it _might_ become available.)
This is second "House" DVD we've had to stop watching halfway through an episode due to a scratched up disk. They're very nice about replacing them, but it still won't be here before Tuesday at the earliest, and the episode had just gotten to a good bit. Altogether, we've gotten at least a half dozen unwatchable DVDs over the past year, which we've had to send back.
Yet another corner case in "patch". I'd assumed there would always be at least one non-patch line between each patch, so what I was doing was naievely reading each hunk until the first line that didn't start with a "+- " char, and then trimming any extra context lines (more than the starting context line).
This was _really_stupid_, and I honestly don't know what I was thinking. It was small, simple, and wrong. Here's how that breaks when you cat multiple patch files together:
--- filename +++ filename @@ -42,7 +42,7 @@ context context context -line +line context context context --- filename +++ filename ...
The length of the patch is given in the @@ line at the top: the -xx,7 means 7 lines of old content, the +xx,7 means 7 lines of new content. Use that and concatenating patches is no problem (and context line trimming isn't necessary either, so it actually probably winds up smaller.)
My code's even already parsing @@ lines, although I just through it would just be for error reporting purposes. The -42 and +42 are the starting line at which to expect the start of the hunk (in the old and new file, respectively), which is only really useful to tell you at what offset you did find the hunk. I even thought about searching for multiple patch locations and taking the closest one, but traditional patch just locks to the first three matching lines of context and fails the hunk if the rest of the patch doesn't match. (Unless of course you implement "fuzz factor" support, which is often a bad idea.)
There is a corner case where repeated context lines could make it miss a patch location and fail to apply. If the first three context lines are "aab" and the file has "aaab", then matching the first two and failing the third would flush those three already-evaluated lines from the previous supposed match to the output, and they wouldn't be re-scanned for another match later down. This is an "I can fix that, but is it worth enlarging/complicating the algorithm"?
I could make the thing _darn_ reliable (and fuzz factor implementation trivial) by loading the whole file into memory and running each hunk against the in-ram copy. It could apply overlapping or out-of-order hunks, use the closest match in case of duplicate match sites... But it would also eat much more memory at runtime, and I'm trying to do embedded software...
Maybe I could do something with mmap(MAP_PRIVATE)? Eh, worry about it later, get this implementation working first. And _that_ means a rewrite of the hunk reading code.
Hanging out at the Starbucks behind the arboretum. Today they have a woman playing a harp. She's playing "tea for two" on said harp.
Refactoring tinycc. Yeah, I know I should get a release out first but I really can't make much more changes to the code until I clean it up to the point where I understand what the heck it's _doing_. The current organization is haphazard, undocumented, and full of cross-dependencies in the strangest places, uses way too many single character variable names, buckets of global variables with names like "ind", "loc", and "vtop", and instead of breaking up the code into multiple files that could be separately compiled, it #includes other .c files which can't build on their own because they depend on static functions and static global variables from the #including files. Yeah, cleanup time.
Currently changing the names of the functions in i386/gen.c so they all start with a gen_ prefix. (Some already did, some started with just the letter g, and a few had neither.) This is a finite and hopefully somewhat self-contained change, so I know when I'm done. Trying to figure out along the way what each function does (and comment it as I go) which is a bit nonobvious in places. Take gsym() and gsym_addr() for example (now gen_sym() and gen_sym_addr()). The first is a wrapper calling the second with a global variable for its second argument. The second takes two arguments, both integers, but what are they? The first indicates a symbol somehow, and the second is an address, but in what format? They're both integers, should they be longs on a 64-bit host?
Looking at the calls to the function, tcc.c has block() which has an if statement for TOK_WHILE() that ends with a call to gsym_addr(). Of course the variables used in this area of the code are a, b, and d with no further explanation and no comments.
Ok, it looks like what these are doing is _linking_ a symbol. I note that gsym_addr() is probably going to be really unhappy running on a host that traps on access to unaligned integer pointers, but for the moment I'll add a comment and move on. What the heck does the comment "output a symbol" mean in this context? What actually gets output? They're patching addresses that have already been output.
The harp lady is now playing an Evanescence song. The "these wounds won't seem to to heal" one. And singling along to it. Now she's doing a classical piece I recognize but couldn't name. Beethoven, possibly. (I know Trans siberian orchestra covered it, and it's often played around christmas.) Next a Roxette song, "listen to your heart". This is actually a better selection than I expected from harp music.
I wonder what oad() stands for. Output with additional data, perhaps? What it _does_ is gen_inst_le32() so that's what I'll call it, I guess...
Right, renaming pass first, _then_ try to clean up the code. (Gotta do them seperately so when I screw stuff up I have a chance of tracking down _which_ change broke things, and debugging it.)
And patch broke in the uClibc build, because if a hunk is right up against the end of the file there aren't as many trailing context lines as starting context lines. (There may not, in fact, be any.) So fewer ending than starting context lines means match end of file. The start of file case is sort of already handled by the current code; with zero context lines we match immediatey (start of file unless it's not the first hunk, which is a "don't go there" case). Trimming trailing context lines shouldn't hurt too much because the start of the file is easy to find reliably.
Finally collating my todo lists to produce a master todo list. Not _close_ to complete yet, of course. :)
At Epoch I learned that I'm not the first person to do a parallel bzip2 implementation. It's even in Ubuntu's "universe" repository, I just never noticed before. Can't say I'm surprised, it is an obviously parallelizable algorithm (Independent 900k chunks. Ok, the only way I can see they're parallelizing decompression beyond 2-3 threads is via an unreliable heuristic. I haven't looked at the code yet, but I'm just guessing that they're searching ahead in the data for frame headers, hoping there are no false positives in the actual compressed data. I wonder how they do handle false positives?)
Then again, toybox isn't exactly trying to break new ground, is it? Neither was busybox. The point is I can do a _better_ job, or at least a smaller and simpler one. :)
Implemented the rest of patch add/remove in toybox. Comment from 60 seconds ago, "That may actually have worked". Firmware Linux seems to be building with it. Fingers crossed... I still need to implement "move" support, ala git. And I need to go re-read Brahm Cohen's blog (the creator of bittorrent) to see what his new diff algorithm was all about. But hey... Progress.
Broke the option parsing logic of tinycc into a separate file. I know I shouldn't do cleanups until after this release, but the urge to wield a shovel at it is enormous. I have Jack Shang's macro parsing fix and am trying to figure out if it's the correct fix (it works, but the author himself isn't sure it's the right way to fix it), and backing up and tracing through the context of what the token parsing logic is doing, the urge to refactor and clean up is HUGE.
Heh. Now that I'm using the toybox commands during the build, FWL hit the problem that "touch" isn't creating files. I knew about that one from the test suite already, time to fix it...
Merry christmas.
Banging on tinycc a bit, on and off. Ok, mostly off doing other things. (Fade and I spent 2 hours driving around town trying to find an open restaurant for lunch before discovering that the IHOP within walking distance of home was open. The only other candidate was a Furr's in south austin with a line stretching outside the building. The cheesecake pancakes are REALLY GOOD, especially with strawberry syrup.)
I learned from Wikipedia that Sandman:The Dream Hunters was actually based on Pu Songling's Strange Stories From A Chinese Studio, which I thought I ought to read. Will report back.
- Neil Gaiman, author of Sandman: The Dream Hunters
So, after much head-scratching trying to figure out what I screwed up in the bunzip refactoring, I find out that the version I'd added to toybox never worked with the testcase I'm currently trying (a semi-current stable Linux kernel tarball). So I've backed up to my old micro-bunzip.c and I'm debugging against that, but it's reading data in different sized chunks in places which screws up the trivial divergence finding technique of "do a printf of each read starting position and bit length, and diff the output".
On the bright side, all this bashing through two slightly different implementations and head scratching over "what does this bit do?" is refamiliarizing me with intricate subtle code I haven't really looked at closely since I wrote it in 2003. I should write up an explanation of how the darn bunzip algorithm works and send it to the lwn guys or something.
The bunzip problem turned out to be missing curly brackets.
Huh, I appear to be allergic to parsnips.
Fade and I went to see Sweeney Todd at the Alamo Drafthouse today, and of course when the Drafthouse is showing a musical where the protagonists murder people and make meat pies out of the bodies, the drafthouse is going to serve meat pies. I had the rabbit and Fade had the duck. Halfway through the movie I started to get a weird reaction I haven't had for a few years, my neck aching and feeling swollen, and like I'm not getting enough blood to my brain (tired, unable to focus, stuffed up ears). The last time I seriously felt this way I traced it to some only vaguely identified vegetables. This time, the drafthouse's ingredients list for the rabbit pie mentions parsnips, something I don't normally have.
I was a bit out of it for my meeting with Will (spent half of it trying to poke the right side of my neck to unblock it), and then took a 2 hour nap. Then had LOTS OF TEA. Feeling better now. (Tea. Marvelous stuff. Diuretic, among other things. Plus if adrenaline is a treatment for allergic reactions that close up your lungs, I can see caffiene helping ones that close up the blood vessels in your neck.)
Once again bit by vi's most annoying and useless feature: the tendency to accidentally change the case of random text when you typo something (in this case, a dozen screens worth of text), and not notice. That was a totally unnecessary half-hour of cleanup. ("Undo" doesn't help if you don't spot the lower-casing until you've done an hour or two of editing on other parts of the file.)
Got the first major refactoring in my threaded rewrite of bunzip2 to the point where it compiles again. (And segfaults, but that's to be expected.) I'd estimate that once I get this debugged and checked in, that'll be somewhere around 1/2 what I need to do to get a 2-thread version of bunzip2 implemented.
The altorithm makes two major passes over the data. The first reads from the source file, and does huffman, run-length, and move-to-front encoding. It has to do all three to figure out where the end of each block is, because the blocks are 900k before compression (and should thus decompress to an even 900k), but the only way to find out how much _compressed_ data is in the block is to decompress it. The second pass undoes the burrows-wheeler transform and calculates the CRCs. The obvious parallelism is to split those two passes and have one reading into a buffer while the other writes the previous buffer, and then swap. If the huffman decode turned out to be twice as fast as the burrows-wheeler step (and given the speed ratio between gzip and bzip that's possible), then two burrows wheeler steps could easily happen in parallel.
For right now, I'm just doing two (one read thread, one write thread) to simplify things. And I'm making it so the same code still works uniprocessor with config switch.
Watched "The Face of Evil" on youtube. Off to feed Mark's cats.
The first tenth doctor DVD finaly arrived through netflix, and we're watching the commentary track of "the christmas invasion", and it's "video commentary", meaning there's a darn pop-up window (showing three people with party hats huddled around amicrophone) blocking 1/4 of the picture. I keep wanting to grab it and drag it out of frame, but alas it doesn't work that way...
Mark informed me that multicast is still used by the "internet 2" people (both of them, apparently) to do videoconferencing.
Yay, Mark got a job!
Remembered that the Wii acts as a youtube video viewer, which was finally enough incentive to fire it up. Watching "The Curse of Peladon" (third doctor). Alas, when I fullscreen the video, A) the frame rate drops to about 5 updates/second, B) it doesn't go quite fullscreen, there's still a large wii icon bar across the bottom I haven't figured out how to dismiss, yielding black bars along the sides of the video to maintain the aspect ratio. So it's using just over half the screen, and can't even update that much at full speed. Sigh.
I remember when people used to care about "multicast traffic" and the m-bone and such; a way of sending one packet to multiple recipients. For the record, that stuff's dead. Youtube doesn't use any of that, and neither did Napster before it. Instead they send a unique copy of the data to each user, via a standard TCP/IP connection to a web server. This is called "unicast", although the name doesn't get used much for the same reason the names for Earth's moon and sun (Luna and Sol, respectively) don't get used much.
These days you download video the same way you download any other web content, there's a file at a URL and you download the whole thing from beginning to end. Even Youtube is doing this behind the scenes, if you want to dig through a bit of Javascript to find the URL. (Things like bittorrent are more efficient from a pure downloading perspective, but since they download data out of order you can't stream it and view while downloading.)
The failure of multicast was 100% predictable to those of us working in this area circa 2001. We got some details wrong; for example those of us who thought cable TV companies might be less stupid expected more regional cacheing. Although Youtube/Google have multiple data centers full of rackmounted PCs serving data into big fiber optic connections, they only have a dozen or so enormous data centers around the world instead of a small one in each city. But that's details.
The reason multicast traffic died was essentially due to improvements in data compression technology. The first modern multimedia distribution system was Napster, and Napster's trick was the mp3 compession format. Modern modern multimedia-specific data compression formats can crunch multimedia stuff down to about 1/10th of the size of the data compression formats that came before it. So a 650 megabyte CD became around 50 megabytes of MP3s, making individual songs (3-4 megabytes) feasible to download circa 1998 via standard TCP/IP connections, even over dialup. Instead of fancy downloading technology, better data compression made the multimedia files an order of magnitude smaller.
Of course data compression can't handle missing data, and broadcast techniques drop packets. The original purpose of things like RealAudio was to do without the data that didn't make it through, reconstruct the missing bits as best as possible, and gracefully degrade the output. They put huge amounts of effort into making this work. But with better data compression, sending 1/10th as much data produced better output, as long as it was sent reliably.
The DRM people loved multicast, of course. A normal file that lives at a URL can be downloaded via right clicking on the link your browser and going "save as", and there's really nothing the server can do to stop this. Viewing and downloading are the same action. With multicast the data is sent to special applications that do nothing but display it, and they probably don't get a complete copy.
The final nails in the coffin were that A) you need special servers to send multicast data, while any web server will do for unicast, B) podcasting: separating the download from the viewing means any old program can view the data, and you can take it with you on portable devices that aren't even connected to the net.
The only reason this is worth commenting on is the sheer amount of _effort_ that went into multicast. The are internet RFCs, the Linux kernel has a multicast stack, RealAudio built its original business model on it (and of course Microsoft blindly copied them). And within 10 years, it dried up and blew away because it's just not how the internet works.
Met Mark for lunch at a strip club. The $2 steak and fries were good (and I don't feel bad about taking advantag of them after buying a $4.50 diet coke). We filed the LLC paperwork for Impact, and then my driver's side car window broke again, so I took it to Lamb's. They have to replace the regulatory thingy they put in back in May. Part's still under warantee but labor was only guaranteed for 6 months. Oh well.
One more swing at patch (I keep getting distracted from it). Got add and remove implemented, although not debugged yet. I need to make it understand the git "move" syntax too, because renaming files is one of those things people actually do, and "delete whole file, insert whole file" is a _horrible_ way to represent that.
I need to work on the Impact website. I need to put together a current TODO list, considering the last five I have floating around are all highly incomplete and full of stale entries, and I'm not quite sure where they all are anyway. (Yes "collate todo lists" is a perennial todo item of mine.)
Heard back from the author of "Lipitor, thief of memory" who agrees Terry Pratchett might be suffering from the problem his book's about, and asked said author to email Terry directly. (Well Terry said, "I know it's a very human thing to say "Is there anything I can do", but in this case I would only entertain offers from very high-end experts in brain chemistry." This guy at least has an MD and wrote a book about this specific problem, which is more than I can say. :)
The Tin Can Tools guys are prototyping a USB-powered hammer board based on some suggestions of mine on #edef on freenode. They're calling it "nail", of course.
Still banging on patch. Less Dr. Who watching today, but the night is young, as they say. (Prenatal, even. The sun is still up.)
Expanded the architecture section of kernel.org/doc, and slightly tweaked and htmlized my gplv2 compliance howto (which isn't official until the SFLC lawyers sign off on it).
I seem to have inspired a thing. Not entirely sure how, but I look forward to seeing it in action.
Incorporating Impact's going to cost $300 to do an LLC. Sigh. Oh well, needs to be done...
Watched "Tomb of the cybermen" (an old Patrick Troughton Dr. Who episode) on youtube. Wow did it suck. I suppose it's a bit like complaining that "the great train robbery" didn't have much of a plot by modern standards. The three main characters (The Doctor, Jamie, and to an extent Victora although she had nothing to work with) could all act, but none of the guest stars could survive in a modern commercial, let alone series. (I don't mind the horrible special effects, but the _acting_ has to be at least half decent.) Still, "The gunfighters" remains unassailble as the worst Dr. Who episode of all time.
Banging on patch.c. Got it working... except for inserting new files, which breaks a couple of rules. (Patching a file that doesn't exist, zero context lines.) Deleting an entire file is another unimplemented corner case, again zero context lines and deleting the resulting file. Also, right now I'm using the +++ line as the filename, which works for everything except comparing a file with /dev/null to delete it. Sigh.
Fade pointed me at this marvelous review of Windows XP as an upgrade from Vista, covering Look & Feel, Performance, Device Support, Reliability, Gaming, and Multimedia:
Conclusion
To be honest there is only one conclusion to be made; Microsoft have really outdone themselves in delivering a brand new operating system that really excels in all the areas where Vista was sub-optimal. From my testing, discussions with friends and colleagues, and a review of the material out there on the web there seems to be no doubt whatsoever that that upgrade to XP is well worth the money. Microsoft can really pat themselves on the back for a job well done, delivering an operating system which is much faster and far more reliable than its predecessor. Anyone who thinks there are problems in the Microsoft Windows team need only point to this fantastic release and scoff loudly.
Well done Microsoft!
So I'm working on building toybox commands as individual files, like I did for busybox way back when. Unfortunately, this fights with the config system. Do I say "scripts/individual.sh hello" to build one command, or do I have the user switch on what they want with menuconfig and then autodetect that and build the enabled features? I'd lean towards the first (and come up with a way to say "make all_individual") except that commands have sub-features, and you need menuconfig to specify which sub-features you want.
However I do it, dealing with toys/toylist.h turns out to be a pain. Just trying to do this by hand (come up with an individual.c and gcc invocation to build hello.c) The second half of that file has been extensively designed to do something very specific: creating lists of information about all the commands in toybox. It's quite well-suited to its original purpose. What it's not well-suited for is cherry picking an individual entry from those lists without instantiating the whole list first.
For the individual command case, I want to have toy_list[] contain just one entry, and set toys=toy_list, and drop out the rest of the list. But the initialization of that list is controlled by USE_BLAH() macros, which come from the config system. I need to switch off all the USE_ macros except the ones for the current command, and within the C code I don't even know what they all are. Distinguishing that requires a sed preprocessor step.
Possibly I could do a big stack of ? : operators the way I'm doing a big stack of || operators, so that toy_list is just the one entry I need. I could even skip the first NEWTOY() declaration (for the toybox command, which has no USE_ guard) by doing 0 && in front of the whole lot (the precedence between that and ? : works out ok in this instance.) Except for two problems: 1) thing={blah,blah,blah} syntax is only accepted in an initializer during a declaration, so sticking (1 ? {blah} : {blah}) around it may not work B) what would I test against? Making a preprocessor symbol that's 1 for this command is easy enugh, but how do I get zero symbols for all the other ones?
I could break down and just run sed against the header file to chop out the declaration I need, but part of the reason I'm doing this is that "sed" doesn't work the same way on various different platforms. I still don't care about Cygwin but I do care about MacOS X. So adding my sed implementation to toybox and building it as a separate executable without having to go through the config system (which uses sed) at the start of the build makes sense. But if the scripts/individual build needs sed then I'm in chicken-and-egg land...
Got the first half of patch checked into toybox. The patch output is going to stdout, but it seems to be applying hunks properly now. With an offset even, albeit no fuzz factor yet.
I need to think more about the offset logic. Right now it doesn't care where the hunks say they are (in terms of starting line number), it just tries to apply them all in order based on context lines. This is probably good enough for most real world uses, although the downside is that a single failed hunk means all the other hunks in the patch fail too. In order to improve on it I'd probably want to read in all the hunks for a file and try to apply them in parallel. Worry about it later...
Reviewed Roberto Foglietta's readlink patch adding [-fem] (alas, incorrectly due to the gnu behavior being complicated and persnickety in ways that are not at all obvious upon casual inspection). In the process I noticed that MacOS X has a "readlink" command that essentially has no common options with the gnu version. (It shares -n, but that's it. Their -f implements totally different functionality, and there is no -e or -m.) So the question of whether or not this is worth doing at all comes up. I still lean towards adding -f.
Got the high-gain antennas source at least minimally reviewed. It's stock BusyBox 1.01 with three lines of tweaks, some bundled config files, and they didn't run "make distclean" since the last time they built it. Other than that, it's stock 1.01. Whether or not that corresponds to the binaries they've been distributing, I couldn't tell you. I have no idea how their firmware images are packaged, and I haven't got one of the devices to install it on and take a look. The file command says "data" when I point it at the firmware images. Wheee. I know a number of people who slap a jtag on these sort of devices and reverse engineer the hardware for fun, and they could almost certainly extract the appropriate binaries... if I had the hardware.
Yay, new laptop charger. Dell replaced it under warantee, and got it to me next day via DHL. I'm impressed. (I didn't even buy the extra service plan, this was straight warantee coverage. There are some advantages to buying new hardware instead of used...)
Went to CompUSA's going out of business sale during my day and half offline. Most of the stuff they have is still more expensive than Fry's even with the extra 10-20% off, but I bought Fade a copy of the Mac VMWare for christmas. (And gave it to her, since she already gave me my PSP and she might as well start using it now.)
I really need to learn to use branches in mercurial. Roberto Foglietta sent me a readlink patch I want to apply, but it hits some of the files I've already changed for patch. But patch isn't ready to go in yet...
Jon Stewart continues to be amusing, even on cspan.
Oh come on, Terry Pratchett gets diagnosed with Alzheimer's right after being put on cholesterol lowering Statin drugs which are known to screw up memory. And nobody makes the connection?
My laptop charger has stopped charging. I have no idea why. Great. I've plugged it into three outlets, some of which are powering lights, and it won't light up. Not good...
My laptop finally failed to suspend again last night. I let it get to 2% battery before hibernating, and after grinding away for five minutes trying to suspend the battery gave out. Software suspend gets really really slow when you've done it several dozen times without rebooting. (Mostly the "free as much memory by swapping it out" phase, the "write out what's left" phase is still relatively fast. Fragmented swap file, I'd guess.
So I lost six desktops crammed full of todo windows. Let's see, what are my current pending projects:
And that's just the immediately pending stuff. I've actually been swap-thrashing about as much as my laptop has, trying to get all these done at once. The bottom three have kind of gotten knocked down until I get releases out of each of the top three. And once I get the _bottom_ three done, then I need to seriously job hunt.
Implementing patch for toybox, I am learning _so_ much about the weird corner cases of diff and patch. (This is a fairly common occurrence, I learned tons about mount implementing it, learned sed by implementing it, and so on. For one thing, unified diffs can't have hunks out of order. (If they do, the gnu version of patch fails with a specific error message for this. If you reorder and then renumber them, the first one applies with an offset and the second fails.) So it's doing a simple one pass substitution.
Two cute blonde women stopped me and spent half an hour talking with me. Sadly, this does not fall in the "good things" column for the day.
Remember when I visited Austin back in May and got the inspection updated? May 2007, so the inspection would expire 05/08? It turns out that Lamb's on Far West put the "8" in the month field and the "5" in the year field, so it says it expired in August 2005. Yeah, I got pulled over. Plus the glove compartment was empty (there was a huge pile of papers in there, where did it all _go_?) so there's no insurance paperwork to give them. (We make the car extortion payment, really.) And the registration sticker on the car expired last month, although their computer did note that the registration was updated (just the new sticker they mailed to us isn't on the car yet).
The whole thing becomes about a $10 fine once I cough up the appropriate paperwork, but the sad part is I did all the right steps to make my car legal. (I wonder if Lamb's has inspection records. I _did_ get it inspected, when I got the driver's side window and the air conditioning fixed. The inspection sticker I had before that nominally expired _after_ the one currently on my car.)
Ok, this guy has more time on his hands than I do. (The lights are apparently running on one of those hammer boards. The music is Trans Siberian Orchestra.)
Why is the GPL like Star Trek? The even numbered ones are better. Why is the GPL like Star Wars? The new version 15 years later just doesn't measure up to the old stuff.
An entire swat team (eight or nine people in green army camoflage fatigues with grey t-shirts over them that say "Austin SWAT") are having lunch at Chick-fil-a. I wonder why?
Oh wow, the assembly produced by tinycc is horrible. There's an "fldz" instruction and yet it's doing "fldl 0x0"... Ok, after this release, there's some serious cleanup to do. But for now, I've ignored Firmware Linux for too long. The wrapper needs to be updated so the x86-64 toolchains are less brittle.
*blink* *blink* *blink*... And I broke patch. The gnu one. I have no idea how I did this, but there's a file that it's _insisting_ contains a reversed patch, yet it doesn't. (The 2.6.13 patch to the linux kernel source to make "init=/tools/bin/init.sh" not break.) It's removing a hunk of code from init/main.c, and said hunk of code is in init/main.c. If I let it go "assume -r" like it prompts, it adds a second copy of said hunk of code. Yes, the patch has minus signs. (Minus signs remove code. Plus signs add code.) There's no -r being passed on the patch command line. I checked the starting line number, it's only two lines off. What the...?
I think it was the upgrade to 7.10 that did it, actually. Neither my scripts nor my patches changed. (I upgraded the kernel tarball to 2.6.24-rc4 but that's just standard preemptive stuff, there's no obvious
Right, screw it. Time to implement patch in toybox. This is just _silly_.
Hmmm... Toybox only really has wrappers for file descriptor functions, not FILE * functions. I hate to have two parallel sets of infrastructure that do the same thing in a project designed around being small and simple. The file descriptor functions are inherently more simple: it's what the OS is doing under the covers anyway, there's no need to worry about an internal buffer not being flushed... Unfortunately, doing getline() without a pushback buffer boils down to single byte reads, which are painfully inefficient. (If you overshoot, you either have to maintain your own buffer of the extra data you read, or hope that your filehandle is seekable, which pipes like stdin aren't.)
Finally made expressions like 'long l = ("blah" && 0)' resolve properly at compile time as a contant in tinycc. Now I've gotten distracted into installing wine and spending five minutes poking at it to see if ./win32-tinycc is actually doing anything interesting. (Answer: no, it's building x86 Linux binaries. Apparently the only way to tell tcc to build Windows output was to compile it on a Windows host.)
I've ranted here before about HOST != TARGET, and in my build system rewrite I've confined all the weird "you can you run code for an x86 target on an x86-64 host but the library paths are /lib32 instead of /lib" stuff to ./configure and not in the C code. But from a purely pragmatic viewpont, I can't test the win32 backend unless I use Linux to do it. I haven't got a Windows machine. With two brief exceptions (the time it took me to complete "starsiege" and about three months keeping around a World of Warcraft partition), I've never had a windows machine. I went from DOS to OS/2 to Linux with the occasional ricochet off of Solaris, AIX, JavaOS, etc. The closest I've come to supporting Windows was writing code in Java or Python that could be deployed on it, or changing the output of my CGI scripts to work around bugs in Internet Explorer.
I've now inherited a project that has a win32 backend, and it seems a shame to rip it out, but I simply don't do Windows. So I'm playing with Wine...
tinycc 0.9.25-pre1 is out. It only builds on i386 at the moment, but that one's building itself from source and passing its test suite, so it's time for a snapshot. Real release still at the end of the month. I need to get at least arm working, possibly win32 as well. (I'll care about c67 when I find a user and an emulator.)
Added the test suite back to tinycc, using the new build infrastructure. This is just the first pass, the second pass is to automate building tinycc with itself and see what the output of _that_ is. But the first pass seems to have three separate instances of breakage, according to the diff. Wheee.
Right now, you need to "ln -s i386-tinycc tinycc" in order to run make/test.sh because it tries to use the "native compiler" symlink, and the build isn't making that yet. This is because it's not trying to automatically determine what host you're running on. This sort of belongs in ./configure, but is just complicated enough I have the urge to put it somewhere else. (Mostly configure is "if environment variable X isn't set, set it to a default value". Adding in actual logic that _does_ stuff goes against the grain, because I want people to feel free to edit configure to whatever they need, or set all its variables themselves as environment variables. Alas, I need some logic for certain things, and this is one of them. The abnormal library paths for x86 binaries on an x86-64 host are another...
Anyway, once I get the known bugs worked out and have it in a working state I can regression test, then it's time to tackle the 8 gazillion warnings it produces while compiling...
Fixed the octal parsing logic, which never quite worked after I consolidated it last month but I couldn't test until now. (And _this_ is why I've been holding off fixing up the warnings: even simple changes break things. How people ever wrote programs before they had their own machine to actually test it on is one of those great heroic endeavours of history like the people who recited the entire Illiad from memory. (Of course they could spend months learning one story, and make a career out of a dozen or so, and these days the rest of us have a bookshelf full of 300 of the suckers.)
Lots of things are broken on x86-64, but most of these are actually due to x86 not really being a native compiler. You need a native compiler for -run to work, you need a native compiler to run the test suite. (You can compare the output of a 64-bit build of the test binary with the output of a 32-bit build, but anything that uses sizeof() or align() is going to be different. I fed CFLAGS="-m32" into test/test.sh, but either I'm doing it wrong or gcc on Ubuntu doesn't want to produce a 32-bit binary. Possibly I need to install something. Luckily, I have a 32-bit xubuntu installed in an image file under qemu, which makes testing so much easier.)
Another fun thing that makes testing easier is the following script:
(cd toybox/toybox && hg serve -d -p 8000 -a 127.0.0.1) (cd firmware/firmware && hg serve -d -p 8001 -a 127.0.0.1) (cd tinycc/tinycc && hg serve -d -p 8002 -a 127.0.0.1)
Not only does this let me fire up a web view of my projects when I'm not online (via http://127.0.0.1:8002 and such) but it lets me check them out from qemu ala "hg clone http://10.0.2.2:8002 tinycc" and then "hg pull -u" to update them. It turns out that 10.0.2.2 in qemu is an alias for the host system's loopback. (This is also what makes the distcc trick work in the FWL build, I need to finish that and check it in...)
Hanging out at Subway today, for a change of pace. No outlets or net access here, but free soda refills and 3 hours of battery life in ye newe laptop. I'm loving it. (No, McDonalds, you can't own common phrases out of the language. And this applies to you too Nike, with the "just do it" thing, which was a line Ford Prefect to Arthur Dent in the BBC hitchhiker's guide to the galaxy in 1980, among other things. It was even vaguely shoe-related, telling Arthur to prepare for hyperspace by lying down and grabbing a towel between his ankles...)
While walking Mark through the toybox code yesterday I noticed that checkin 186 is broken. The default toys.exitval is 1, not 0, because this is what error_exit() returns. This means that half the commands (ironically enough including "true") are exiting with 1 instead of 0 on success. Which is broken.
The question is how to fix it. I'd rather not patch each command (and add back the 4 bytes to each one), but instead change the default value and have error_exit() and friends check for 0. This means that error_exit() can't print an error message but then exit(0). I think I'm ok with this, since I can't come up with an actual use case that would need to do this, but I've been pondering it a bit before making the change...
I'm watching Mark export his iPhone's cellular network connection to his macbook. That's... both sick and amazingly cool. Support for all this is apparently built in. [Correction: Mark informs me it's not built in, it's software he installed. Considering he's got a shell prompt on his iPhone, I should have guessed this.]
Hmmm... Granularity problem in the tinycc configuration. The logic in tinycc itself is adding "/lib" and "/include" to a base path ala "/lib/tinycc". But the ./configure script isn't passing in "/lib/tinycc", it's passing in separate "/lib/tinycc/lib" and "/lib/tinycc/include" paths.
On the one hand, having just one install location is simple, and simple is good. But explicitly specifying each path both provides more control and more visibility. It's an aesthetic decision as much as anything, both ways work fine but what should the design be? These are always the tough ones...
Ideally I'd just have ./configure prepend these paths to the library and header search paths it's already assembling, but unfortunately the Linux kernel build wants to know where the compiler headers are, but to supply its own headers for everything else. So they have to stay separate. Which means the fully orthogonal separate paths aren't fully orthogonal if we want to build an unmodified Linux kernel, so I might as well take advantage of "simple", above.
Ok, that was a fun couple hours of debugging.
Trying to make a new test script for tcc to build and run the test suite, I noticed it was trying to link against "tinycc-1" rather than "tinycc-i386". The i386 comes from TINYCC_TARGET which is initialized from "$1" in make.sh, and that seemed a natural place for a 1 to be coming from. But it wasn't, the macro was set properly, and the thing built just fine, but with the wrong value. Huh?
Fast forward two hours later, after a dozen debugging blind alleys (and re-learning that #warning doesn't evaluate macros). It finally hit me: TINYCC_TARGET was evaluating to i386, and i386 is itself a macro in gcc that evaluates to "1" on i386. It's one of the compiler builtins.
This didn't come up when it was quoted. Sigh. And this is likely to be a problem on _every_ platform. Even though there's just the one user of this, I can't pass it in as "tinycc-i386" either, because unless I put quotes around that the i386 is matched and turned into a 1. Sigh. Can't assemble it out of substrings or pass it in as part of a bigger string, I _must_ put quotes around it if I want "i386" to wind up in the result. And putting it in a subdirectory named "i386" has the same problem.
I note that I always test for __i386__ and friends, because #defining i386 or arm is just too dangerous. But the gcc guys do it anyway. I note that this built just fine on an x86-64 host. :)
Although the Schlotzky's at UT has free wireless, it's no substitute for Chick-fil-a. Among other things, its sodas taste very strongly of chlorine.
I've heard a half-dozen variants of speculating on what, exactly, the reindeer games Rudolph was excluded from consisted of. I am aware that a preoposition is considered a poor word to end a sentence with. These days we're allowed to boldly split infinitives that nobody using "them" as a gender neutral singular pronoun has split before, too. Isn't it great not to have a language institute? The lingua franca is English, in large part thanks to L'academie Francaise pruning French back until there's nothing left. Although laws mandating a language's continued use do help to drive home how deeply popular it must be among its native speakers, to need a squirrel cage like that in order to survive. (Welsh is having an actual popular revival, possibly pushing it past Klingon. Good for them. I wonder how many new words English will steal from it?)
Huh, "./i386-tinycc -run hello.c" is segfaulting on x86-64. Works fine on x86. I'm guessing this is a 32-bit vs 64-bit thing. I used to think it wrote a file in /tmp and execed it, but no, mallocs a big buffer to hold the executable and calls it as a function pointer. Cool, but not gonna work when host!=target no matter what fancy BINFMT_MISC configuration you've set up. Even running x86 code on x86-64 requires an exec to switch modes.
Spent several hours in the evening trying to get wordpress working for fade's new blog. Eventually asked Mark for help. I spent two hours making no progress (fighting with MySQL administration), he fixed it in about six minutes, including installing the rest of wordpress.
Apparently it's possible to switch VT support off in the bios, meaning a core 2 duo would claim not to have the extensions KVM needs, which could explain why kvm was degrading to normal (slow) qemu. What I don't know is whether suspending to disk, switching this on, and resuming from disk would make the kernel suddenly demand prozac.
The downside of having a laptop with 2 gigs of ram and 4 gigs allocated to swap, now that I've figured out now to close the lid and unplug it from the wall until the suspend has started actually writing to disk, is that I haven't actually rebooted in weeks. I have six desktops full of windows. I have windows open for kernel documentation, firmware linux, toybox, tinycc, the 64-bit paper with eric, playing with the hammer board, a copy of Amarok open with a half-finished podcast (an interview with Randy Millholland of "Something Positive" from 2005), probably about a hundred unrelated web page tabs open in (count count count) eight different Konqueror windows, kmail with seven half-written replies...
A reboot is actually a bit of a production. An _unexpected_ reboot is a pain the ass.
Ah, it appears that my problem is that not all Core 2 Duo processors have the VT extensions. The T5300 does not. Oh well, time to look at kqemu I guess...
Ok, dear whoever made the "kqemu-common" package: where's the rest of it? Kqemu is a kernel module, this package doesn't contain the kernel module. The only other package that comes up on a search for "kqemu" is the source code. Why bother with a kqemu-common if there's nothing _there_? There's no "recommends" packages, nothing...
So, tinycc path logic. It builds "hello world" again. Yay.
Still trying to figure out what the intention was in some of this code. I've almost certainly broken the Win32 support because I can't test it and I'm not quite sure how it's intended to work.
I also want to add some of the toybox "which" logic to tinycc, because I want to look for the compiler include and lib directory relative to the tinycc executable. Locating your current executable on Linux is either a question of doing a readlink() on /proc/self/exe, or looking at argv[0] and using it directly if it has a "/" in it or trying to find it in $PATH. In toybox, that's about 16 lines of code from which_in_path() and 33 lines of find_in_path(). Still, I can do that later, lemme just get it basically working first...
Done. Wheee. It builds "hello world" again, on both x86 and x86-64 hosts. Not to clean up the warnings, revisit the octal escape logic, update the test suite...
The correct invocation to use "netcat -f" with the serial port on the hammer board turns out to be:
stty 115200 -F /dev/ttyUSB0 stty raw -echo -ctlecho -F /dev/ttyUSB0 stty raw -echo ./toybox netcat -f /dev/ttyUSB0 stty cooked echo
Added much -v logic to tinycc.
Darn it. I can't get C to do something I want it to do. Right now I'm doing:
cc '-DBLAH="string"' tcc.c
And then in the source, going:
char *x = "prefix-" BLAH;
And letting string concatenation turn that into "prefix-string" for me. What I _want_ to do is take the quites out of the macro definition because the actual value of the string is a shell environment variable, so I need someting like
-DBLAH='"'"$BLAH"'"'
Which is just insane. Worse, if I try to echo that line out for debugging purposes the extra quotes get stripped. It's very hard to both print and execute that line from a shell script, because if it's evaluated twice quotes vanish.
Unfortunately, my first attempt at supplying macros without quoting them (cc -DBLAH=string) was to have C code that looks like:
#define TOSTR(x) #x char *x = "prefix-" TOSTR(BLAH)
But that gives me "prefix-BLAH", not "prefix-string". I can't figure out how to get the darn macro to be evaluated before it's converted into a string. Grrr. Frustrating.
Yay, the ##c channel on freenode knew the answer:
#define TOSTR(x) TOSTR2(x) #define TOSTR2(x) #x
And _that_ gives me a TOSTR() macro that does what I want.
Ah, that's where that article went: milk the new oil.
Oh brilliant. Just brilliant. They marked the bug invalid. This is the "kde thinks there's no network connection when there is" bug, which dozens of people have hit and was "status high" before being marked invalid. What is WRONG with these people? Yeah, Napoleon said never attribute to malice what can be adequately explained by stupidity, but at this point that's actually starting to be an argument _for_ interpreting it as malice if you ask me... AAAAAAAAHHHHHHH!!!!!!
Maybe I should try xubuntu...
Cleaning up tinycc. Nobody's done this in years. (For example, tcc_add_include_path() can't return nonzero: there's exactly one return statement in the function and it returns zero as a constant. Therefore the code checking the return value and printing an error if it's nonzero can never trigger. Of course the reason I noticed this is I'm removing all the seprate add_blah_path() functions and implementing a single add_dynarray_path() function that operates on a common structure type, but that's a side issue...)
At Starbucks', trying not to listen to a man use religion to convince a young couple to buy insurance from him. Really. It's not often you hear "God's plan for you" and "government regulation of Google" in the same conversation.
My wife has finally gotten fed up with Livejournal, and is backing up her journal to move it somewhere else. She was annoyed at the random deletion of communities without explanation or appeal, and all links becoming mouseover popups that block the page you're reading, and the mandatory content filtering which recently added a click-through so that when reading her journal anonymously, you get a click through asking you to confirm that you're 14 years old. (Really.) But the last straw appears to have been six apart selling livejournal to a Russian company with connections to both the censorship-happy Russian government and Russian organized crime. (Admittedly, lots of overlap there.)
Neither of the above two paragraphs strikes me as things that belong in a normal day.
Added netcat to toybox, and the serial port in qemu seems to be behaving itself. I'll try the board again when I get home from starbucks. Going through the "low hanging fruit" list and adding "chroot", I wasted several minutes trying to figure out why it was complaining about:
char *default[] = {"/bin/sh",0};
Before noticing that "default" is a C keyword.
I find myself in need of "netcat -f" to play with the new ARM "hammer" board. (It's either use netcat -f or try to get minicom working, and I've forgotten which way you swing the chicken.)
Are the arm "hammer" developers trying to make some sort of baking soda related pun? Huh. Anyway...
Unfortunately, using the netcat version in BusyBox 1.2.2 results in a seriously misbehaving serial port connection to the hammer board, and I'm not sure if it's a bug in the old busybox version, or in the software on the board, or in the Ubuntu kernel, or what? I don't poke at BusyBox anymore thanks to Bruce, and I've now written two netcat instances (one in 2001 and again when I rewrote the busybox version), so trying to debug busybox's netcat is uninteresting (even though I wrote it), so this is a good excuse for me to add this to toybox. I can get it working under qemu then try to attack the hammer board again.
I should also go through the "low hanging fruit" section of my toybox todo list and see how many of those I can get done in one sitting. And then, finish the path logic rewrite over on tinycc.
Right, off to do that. (Somewhere away from the cats.)
Walking into Starbucks reminded me of the Hitchhiker's Guide to the Galaxy. Specifically, the part: "And if we're unlucky? The Vogon captain might want to read us some of his poetry first." Sunday afternoons all the coffee shops seem to have distracting "entertainment" and no free tables. Even the coffee shop inside Barnes & Noble was full. Wound up in a booth at McDonald's. (Yay laptop with actual battery life.)
Hmmm... Couple of interesting corner cases. The OLDTOY() macro needs an argument description string just like the NEWTOY() macro does. In theory different commands that use the same basic function can have different arguments. In practice A) nothing does yet, B) their command_data structures would conflict if they diverged too much (although that's nothing a typecast to another structure type can't fix). This comes up because "netcat" has a largeish number of arguments, and doing an OLDTOY(nc, netcat) for it requires repeating this longish option string. Modern compilers should be smart enough to merge these strings (I wonder if tinycc knows to merge duplicate strings yet) but it's a bit of a maintenance burden keeping them in sync by hand if they change.
An alternative would be to pass NULL to oldtoy and have that mean use the old option string, although it's not immediately obvious to me the best way to actually implement that. Hmmm...
The other corner case is that the option parsing logic doesn't know to make "<2" conditional on "-f" not having been supplied. For now I can just do the test by hand, but if it comes up again I may need to complicate the option parsing logic some more. Sigh...
Spent today under the weather, got nothing done.
Both toybox and tinycc are getting contributors, and traffic on the mailing lists. On the one hand, this is a really positive development and I want to encourage it as much as possible. On the other hand, my ability to wander off into my little corner and focus exclusively on one problem for three days is now being interrupted by editorial duties. Replying to other people's questions and reviewing their patches are priority interrupts, not getting around to them for a week is bad form.
Meaning spending a day under the weather and getting nothing done leaves me with more work tomorrow. Wheee...
I may finally have to give up on kmail. No matter what I try, I can't get it to show me the email address of a message I'm reading. Not unless I either reply to the message, or mouse over it look at the info field at the bottom fo the screen. This is _sad_. It used to do this, but now it's decided to be more outlook-like and hide information, and I can't find a way to make it _stop_ doing this in the configuration anywhere.
Yes, this is enough to make me switch email clients. First of all, I've always had an easier time remembering email addresses than names, and this makes that go away, which is _deeply_ annoying on a personal level. It also makes certain mailing lists unusable (such as the Dell desktops mailing list: is this message from a dell employee replying to a question, or is this message from a user asking a question? Can't see unless I can see if the email is from somebody @dell.com..). Maybe thunderbird's developers aren't so amazingly stupid they think an email program shouldn't even have the _option_ to show email addresses?
Met with Mark today. Spent three hours talking about C with him.
The Hammer board from the Tin Can Tools guys arrived. Broke it open and spent some time frowning at the contents of the CD it came with. I'll probably actually power it up and try it out tomorrow.
Starbucks continues to have a good "hot chocolate with orange syrup". Still no blackberry syrup. Lemon cake's nice, but pricey and I got an end piece when I ordered a slice.
Finally got loopfiles() checked into toybox. Amusing how a function so small takes longer than big ones. (It was almost, but not quite, right for the longest time. Handling the empty list case with a seperate if() outside the loop turned out to be a lot cleaner than trying to handle it inside the loop and having to perform the same end-of-list test three times.
Banging on tinycc for a bit. The dynarray logic is all wrong, there are two related variables for each one (a pointer array, and a length), but they're not in a structure. So basically everything that adds to, frees, or iterates over the contents of one of these suckers needs a seprate function because it needs two unrelated variables and is thus harder to genericize. However, include_paths, sysinclude_paths, and library_paths are char *, but other things (cached_includes, loaded_dlls, sections...) are various struct pointers. Hmmm...
I need to fix the dynarray stuff to genericize the logic I added for handling colon separated paths. I did it for library_paths but not the two include_paths, and they're almost exactly the same code, I refuse to block copy this...
Mark says that the Darwin 8 install ISO (I.E. text mode MacOS, runnable under qemu) lives here, but it needs a login. Oh yeah, Apple's interested in actual open source development on this... Made an account, now it wants me to read a big legal thingy and agree to it (sigh)...
And it won't boot under qemu. It made it as far as some ACPI thing, then hung. Ten minutes, no CPU usage. Wheee...
It's difficult to be particularly enthusiastic about the windows output mode of tinycc. About half of it is hidden behind #ifdef WIN32 meaning if you try to cross compile from a Linux box, it probably won't work. It's also full of endianness assumptions, meaning if you try to cross compile from a big-endian machine it probably won't work. And then there's the x86-64 host I'm trying to make work: Windows uses LLP64 instead of LP64 (meaning a pointer won't fit in a long integer on 64-bit windows, which it's guaranteed to do on any Unix including MacOS X). I really can't bring myself to care about 64-bit windows, but I'm making the rest of it 64 bit.
On the one hand, it would make my job way, way easier to just drop all the win32 code. But there are users, and it sort of worked a few months ago...
Oh well, for now, keep it there but not much I can do about the bit-rot just now. Once I get ELF and PE properly untangled (and start adding Mach-o), perhaps I can try to test it under Wine...
Finally got the new toybox list created. Time to move the them.com lists over...
I have driven my new laptop (with 2 gigs of ram) into swap thrashing. I knew it was only a matter of time...
Is anybody else kind of frightened that this whole biofuels thing will drive the price of food through the roof? Yay renewable resources, but if we dug up half the farmland in the country because it had oil under it we'd all be terrified of the effects, yet half the corn and soybean crops going to fuel automobiles for the forseeable future doesn't make anyone even slightly nervous?
Of course making fuel from plants that were bred for _taste_ isn't the world's most efficient idea. You want ethanol? Figure out how to ferment kudzu already. Corn really isn't very good for this. Ethanol as a whole actually sucks pretty badly, it only has about 2/3 the energy per gallon as gasoline anyway. Vegetable oil biodiesel works ok (just about a 1:1 replacement for conventional diesel fuel, with very little processing), and we were already making plastic out of soybeans so there's a certain amount of poetic symmetry here. But making fuel from food is still a kludge. Fuel production needs its own plants specially bred to do _that_ well, and that's going to take a while. And even then, land's a finite resource: the fabled stockpiles of government cheese and butter ran out back when milk passed $3/gallon, and I believe paying farmers not to grow food (so they don't exhaust the aquifers under the midwest) ran out this past year. Atlanta's already running out water. It would be nice if somebody competent was in charge of worrying about any of this.
I suppose corn syrup no longer being cheaper than cane sugar or honey is something we can look forward to from a taste perspective, and McDonalds is valiantly holding the line on its 99 cent double cheeseburgers, but face it: inflation's running something like 10% annually. I know the shrub administration disagrees with this, but since when can anyone involved in that mess do math? Yeah, "volatile food and energy prices" can triple in the seven years since the last regime change and that's ignored because nobody ever has to actually pay for food or energy. Gas is now $3/gallon, milk's more expensive than that, the falling dollar is even raising the prices of cheap imported chinese things made out of lead, the mortgage industry collapse means housing prices may have STOPPED climbing sky high for the moment... What on earth are they measuring?
And think about the falling dollar for a moment: dollars buy less around the world, dollars buy less here too, but we're not having massive inflation on a scale last seen under the Carter administration (which was triggered, ironically enough, by the OPEC oil embargo of the 1970's). Riiiight.
I'm in an interesting ping-pong state at the moment. "Ooh, new toybox patches from Charlie, must process. Now what window do I have toybox source code open in... Ah, this window has RFC 1321 describing the MD5 algorithm, need to finish reading that... Of course I need to finish sha1sum up first, where did I leave off on that? Ah yes, I was editing catv.c to do generic "iterate through file arguments" code... Hey, phone call from Stu telling me that my tinycc project was on the Reddit front page (for about five minutes, fallen to page 3 by the time I looked), I was banging on that all last night and should finish it up. Left off in nasty tangled path logic. Frown at that for a bit... And _this_ window has a .bat file for compiling tinycc on Windows with mingw, except that won't work with the current version (although the fixes are mostly just simplifications) and I haven't got a test environment for that. Speaking of test environmnets, over in THIS window is a Firmware Linux build that failed because the darn gcc path logic went all wonky again, substituting the x86-64 linker during an armv4l cross compile. Why didn't it do that last time? Right, frown at that later, too many open windows. Why hasn't the git repository on kernel.org updated in 8 days? Linus must be on vacation. I still have a half-dozen documentation patches to push upstream and a large todolist to go through there before I can consider the kdocs project at a good stopping point... And you don't want to _know_ about my web browser tabs. (Still haven't finished the 64-bit paper for Eric, either.)
Progress is being made on all of these, but in 3-minute increments. (So what do I do about it? Update my blog, and rant about how I miss the McDonalds' $1.99 "American meal" back in y2k.)
Phone call about a possible job tomorrow. Off getting the car fixed. The usual.
I wonder if I can con Mark into adding a MacOS X backend to tinycc? He has a testing environment for it, and does understand how OS X executes applications.
Another good article on the long drawn-out death of the music industry. Most of the articles I've collected on this are a bit old now (such as this and this, which I linked to when I wrote this. I note that was written so long ago that this hadn't been written yet...
Of course these days, Weird Al has personally uploaded every video he's ever made to youtube. (And yes, he had to get youtube to put "don't download this song" back up after explaining irony to them.)
I despise apache2. They decided to totally rearchitect the thing to make threading work better on Windows NT, dragged the Linux world along for the ride, and made gratuitously incompatible changes to the configuration files while they were at it so an apache1 howto doesn't tell you how to make it work in apache 2. Plus they changed the name of the executable so you keep needing to remember to append a "2" to things.
The reason I'm despising it right now rather than ignoring it is setting up a self-signed https certificate is more trouble than it's worth, and when apache refuses to work it often won't tell you _why_. Tired, off to do something else now.
Broke down and posted on the old tcc mailing list. I try not to, but when there's actually a thread about me that's... er... "based on a lack of information", shall we say... Mentioned the new tinycc mailing list while I was there.
Checked in the sha1.c I did on the trip to the toybox archive. Might get to hook it up as an actual sha1sum command this evening, who knows?
Finally got the car's oil changed, about 4000 miles late. Lamb's says that the front brake pads are worn out (suspected that), the right front tire has a chunk taken out of it (ah, interstates), and the steering and radiator fluids need changing (wheee). Probably about $500 altogether, deal with it on Monday.
It turns out "another brick in the wall" has a music video. Ok, the whole of Pink Floyd's "the wall" is a giant music video for said album. I'm watching the scenes of students jumping into a meat grinder and other students taking fire axes to their classrooms (now the building's on fire) and remembering that my high school Chorus teacher showed us this movie in class one day. (It's musical!) All I remember is that it was better than having to deal with class. (Oh, now they're throwing their teacher in the fire... Aw, it was only a dream sequence.) Turned int a deeply screwed up and fairly depressing movie about halfway through. No wonder they showed it to us in high school.
House MD was much better...
Long drive. Got a few chances to poke at sha1.c but mostly I was either driving or sleeping. Got in around 11pm, reassured the cats a bit, collapsed into bed.
On the road back to Austin. At the moment Fade's driving and I'm trying to get some work done on my laptop.
The public domain sha1 code works much better if you A) tell it you're on a little endian machine, B) change the various instances of "long" to "int" on a 64 bit machine. Now it's producing the right answers, so I can start untangling it and shrinking it down.
The SFLC fundraising letter I mentioned yesterday is now online. (Due to me emailing them and asking them if they'd put it up, but that's only mildly cheating.)
I'm used to treating my communications with them as "more or less NDA" by default, since they're lawyers and attourney/client priviledge and all that. (And due to previous experience with other laywers that I _someday_ hope to be able to talk about.) But if your fundraising letter is meant to be kept secret? You're doing it wrong. So I asked. :)
I note that I still haven't given them money either. But I did turn down a share in a five figure settlement and told them to keep it, so I'm not _that_ guilty about this.
Somebody dug up Richard Stallman's policy statement on why gcc must be a huge tangled hairball. Not just monolithic, but without clean separation between any of its pieces. This turns out to be intentional: to prevent anyone else from using parts of gcc with backends or front ends under a different license (like the Stanford Checker did anyway, or GCCfs, or the first versions of LLVM...)
In reality, this tangled hairballness of gcc is probably the main reason it's turned into such a bloated unintelligible pig. There's a Linux Weekly News article on this topic today. (Yes, I mentioned my tinycc project in the comments. Cleaned up its web page, too.)
Darn it, I haven't gotten the revamp of the 64-bit paper done. Eric did his pass in a day or so, but mine's a boatload of research that spawns several tangents with each new paragraph and has to be edited back down. I wonder how much I can do on the car trip, without internet access? I was hoping to focus on tinycc and toybox on the trip back...
So the the SFLC has filed two more lawsuits. Good for them.
I know the enforcement actions are in my and Erik's name, but we delegated it all to them and it's mostly just "Can we proceed against _blah_? See attached document for details," and then giving them permission and rewriting the quotes in the press releases into something I actually might have said.
We've delegated it like this because Erik's hall of shame just didn't work, and trying to do it yourself can eat your life. These days gpl@busybox.org goes straight to them and we don't even see the complaints until the lawyers ask us for a green light to file against somebody (with attached "why we're doing this" references which I then try to clear half an hour to read). They even include a prepaid fedex envelope when it's time to sign and return actual documents.
I don't mean to sound disinterested: somebody has to keep the industry honest and I'm glad to do my part. I choose to release code under GPLv2 and I'd like the license to _mean_ something. But I'm coding, writing a research paper, and Eric and I are going to try to make time to get a doclifter release out before I have to head back to Austin. I'm very glad that somebody _else_ can do all the heavy lifting on this, which includes figuring out who needs suing and who doesn't.
I suppose I should put a link to their promo button on my site somewhere, although since it links to their main page rather than their donations page I'm not sure how useful it would really be. (They sent out a nice fundraiser email last month explaining that they get the majority of their funding from vendors, but if they don't get _some_ individual contributions they might lose their public charity status, and they don't want to be just a vendor mouthpiece anyway. I considered sending 'em something, but I haven't got a paypal account, and I like being able to say "and it hasn't cost me anything" when talking about my relationship with the SFLC. Still have the email marked as "todo" in kmail, for what that's worth. :)
Good thing about Ubuntu 7.10: the left and right scroll buttons in the Konqueror tab bar work again. (You couldn't scroll left without a mouse wheel in Konqueror 7.04, and my laptop's touchpad hasn't got one.)
I ordinarily wouldn't be interested in something that close to rap, but That Calls for a Wilhelm Scream is just inspired. (Hey, I'm a sucker for the movie making process. Not so much actually watching movies as what goes on behind the camera. It started as an interest in special effects technology and got seriously fanned by Bernie Brillstein's book Where Did I Go Right?)
So the obvious thing to do (for now) with the tinycc build thingy is probably to create a config.h file with the appropriate #defines and just have the source #include that. Don't force the shell to try to deal with strings with quotes in them, it's not good at that.
You know, I never knew that the Fifth Doctor was married to Trillian. (and this was amusing too. And this is hilarious.)
Yes, the flash plugin is working again!
Finally got a chance to look at the public domain sha1.c code and it doesn't work. Or at least it gives a different sha1sum than the sha1sum command. Hmmm... It's much easier to start with something that works. Hard to debug this if it's doing several hundred complex mathematical steps before I get a result I can compare against something known. Probably I should go look at the wikipedia description of the algorithm or something...
So the sound on Eric's motherboard still doesn't work yet due to a driver bug in Ubuntu 7.10. He had a backup sound card he's had in a box for something like three years, but the sound output of that skips randomly a couple times a minute. So today he went out and bought another cheap sound card, plugged it in, and it has the same skipping problem. Turns out both of the pci cards have a cmi8738 chip and the 64-bit driver for the cmi8738 is buggy. So that's three different instances of sound hardware, two different types of hardware (the onboard sound is an intel 945 chipset), and Ubuntu 7.10 can't competently drive _any_ of them.
Oh yeah, ready for the desktop. Right.
P.S. Dear kmail developers: for years now, kmail has shown the actual
email address ala "Some Name
Slept until about noon-thirty after ditching the Arkham game at 2am. (The owner of the game apparently took it home at 3, so I didn't miss much. It "plays much faster" with only 5 players instead of 8, they were maybe half finished by that point.) Decided not to buy a Sunday badge at this point, hanging out with my laptop in the lobby instead, trying to catch up on Mt. Todolist.
Building libtinycc.a in a canadian cross is fundamentally awkward. If you build a sparc cross-compiler and then build an arm-targeted tinycc with the sparc cross compiler, you do indeed get a sparc hosted tinycc that outputs binaries for arm. But to build libtinycc.a you need an arm cross compiler that runs on your host, and where does that come from? In theory we should always be able to build a native cross-compiler for the target; if we can't then we don't support that target anyway, so it doesn't matter. But knowing exactly when to do this so it's not unnecessary work is slightly tricker...
Speaking of tricky, getting a shell script to be properly verbose is a minefield. In theory you can just do:
function showcmd()
{
echo "$@"
eval "$@"
}In practice, that doesn't work if your command is "CC=cc" assigning a variable, and if you do anything like:
showcmd $CC -DTHING='"'"$THING"'"'
The carefully arranged quotes that should result in the C code doing the equivalent of:
#define THING "contents-of-$THING"
get stripped out by the "eval", which evaluates quotes _again_ and doesn't pass them on to the program you're running.
Hmmm... The closest to what I want seems to be "set -x". This has the unfortunate side effect that instead of printing "commandline" it prints "+ commandline", but maybe there's a way to turn that off. (And there is, "set PS4=".) And of course it prints "+ set +x" when I turn it off again. Wheee...
Orthogonal behavior: wouldn't it be nice to be able to beat it out of gnu bloatware? And you wonder why I want to use something else. While gnu make continues to be crap, I'm starting to see why people don't use gnu bash for this either: it's crap too.
I could write a C version of "showcmd" that would do what I want, but compiling something in order to build is a bit silly. I could trivially do this in python, but do I really want to make that a build requirement?
Why is "show all the commands you shell out to, but not the internal plumbing" such a hard concept for bash to cope with? The dos command.com used to be able to do this one (and had @command to suppress it).
The prettiest girl at this convention asked to borrow my laptop so she could view a rough cut of her newest music video. There are times it's good to be a geek. :)
Short term toybox todo items: xstrcpy(), attribute_printf, and look at this...
1:30 am at Philcon. Exhausted when I got here (and suffering from way too big of a dinner) and took a nap, now everything's pretty much closed up until morning (including of course registration). Oh well, pound on laptop. The internet access here is $10/day. I can wait until Sunday, I think.
Still happy I figured out how to get the flash plugin working yesterday. (See tail end of yesterday's post.) My 7.10 laptop slowly approaches the level of usability that 7.04 had...
Taking a whack at updating my writing page, at least a little. Updating my User Mode Linux HOWTO for 2.6.23. Found _another_ problem with the Defective Annoying Shell (/bin/dash) while doing this: my example User Mode Linux command line:
./linux rootfstype=hostfs rw init=/bin/sh
And when UML boots up, there's no command prompt. Why? Because of dash. It'll respond to commands you type, but there's no prompt so you have to figure this out for yourself. Why is there no prompt? Because /dev/stdin isn't a tty (because /dev/console has always historically been weird for reasons I've never tracked down any reasonable explanation for). It's sitting there waiting for a command (with no script to run and an initial nonblocking read from stdin returns nothing) but it can't figure out it's in interactive mode so it gives the user no hint it's waiting for a command. Presumably it thinks you went "/bin/dash < script.sh" or some such.
Sigh. Very much not a bash replacement. Horrible judgement call on Ubuntu's part pointing /bin/sh at that piece of trash, but that was a bug they introduced in 7.04, not a new one in 7.10.
[Many hours later] Eric Flint's talk is good. He says he has a series of articles called "against big brother" on Baen's Universe, that should be enough to Google for it later. (His GoH talk at Philcon is basically what's going to be in his next article there, apparently.)
The con suite has caffeine free Mountain Dew. I'm still boggling, hours later. Spent more money than I should in the dealer's room. The Voltaire concert was great. (I wonder what it would take to get him to Penguicon?)
Playing "Arkham Asylum" or some such in the gaming room, not particularly impressed (as can be guessed from the fact I've pulled out my laptop and am updating my blog instead. (The gaming room is full of people yelling at the top of their lungs, for no readily apparent reason. It is not fun.)
Poking at the tinycc make files a bit, but not getting much done. Occurred to me that "./make --clean" is stupid, there should be a "./clean.sh" and each command should be a separate script and they should all be as small and simple as possible, so once again I'm refactoring the new build system. Presumably I'll have all of this out of the way at some point.
Woken up at 8:30 this morning (7:30 am Texas time) by a phone call from a recruiter. One blog post and there's blood in the water, apparently. :)
Darn it, Kmail used to have a "file->new->main window" menu option that would let me open a second kmail window open looking at a different folder. Apparently this feature went away in the increasingly misnamed "upgrade" to Kubuntu 7.10. I wouldn't mind so much if switching windows didn't take upwards of 30 seconds (linux-kernel, 74,582 unread messages, and my own email box currently goes back to 2004), and didn't tend to lose my place if I fetch new messages while not looking at that folder.
Spent far too much of the day replying to email and working on the revision to the 64-bit paper. Much research. I have a huge pile of stuff I want to do to tidy up the documentation thing, and to get tinycc ready for a release, and work on toybox (got some of that done yesterday, at least), and firmware linux (Mark's building it under MacOS X), but right now I'm about to head out to Philcon, and may not have net access again until late sunday night.
When I get back to Austin, _then_ I hope to be able to tackle the todo list. I fear I've gotten to the point where working through my todo list spawns new todo items faster than I dispatch them, but still. Nothing for it but to press on!
Figured out how to get the darn flash plugin working. I had to uninstall the "gnash" package. No matter what I tried, gnash insisted on trying to handle every flash file (instead of the actually functional flash plugin) until I not only shot it in the head but removed all traces of it from the system. You _cannot_ use the two side by side, gnash neither offers nor respects any configuration options. (I can watch the daily show again. Yay! Ok, there's a writer's strike, but it's the principle of the thing and a bit of an archive at this point.)
Off to Philcon...
And so the Documentation job ends, although they're being really nice about it and paying me through the end of the month. I've written up three different "I think the documentation job is over" blog entries over the past month, but deleted them instead of posting each time because I wasn't quite sure enough to make it a reality by effectively putting a resignation letter in my blog.
Got a phone call this morning. It's nice to have closure.
This was one of the stranger jobs I've had. The Linux Foundation didn't quite seem to know what they wanted me to do. "There isn't enough documentation for the Linux kernel" is a problem description, but not a job description. I think they figured that out _after_ hiring me. The pervasive uncertainty has been my main source of stress about this job ever since.
Then again, it took me three months of working on documentation full time just to find the edges of the problem space, and figuring out what needed to be done was essentially my day job. The fact this led me in a different direction than the Linux Foundation guys expected isn't entirely surprising.
I suppose it didn't help that the guy who hired me, Tom Hanrahat, quit to join Microsoft before I could even sign the contract. Then I was reporting directly to the CEO, which didn't work because he was constantly busy and traveling. Then I was reporting to the technical advisory board, a committe full of Linux developers elected to join a mailing list, each of whom thought I was someone else's responsibility. (Reporting to a mailing list you're not allowed to subscribe to is a surreal experience.) Then I was reporting to the newly hired CTO, who turned out to be busy and constantly traveling (only managed to actually make 2 of the first 5 weekly phone calls he set up, although he was very apologetic about it). Just before the six month mark, I got a phone call explaining I needed closer supervision (something I'd actually asked for earlier), and was given a specific assignment with a deadline for the first time since the interview process. I heard back from them exactly twice between the time I handed it in and this morning's phone call. (Once one of the tech board members asking for a different patch format so he could rebuild the xml on his own machine, and once the CTO assigning me more work with only slightly more specificity than "do more of that", half of which I'd already done by the time he contacted me.)
On the plus side, I had a remarkably free hand for six months to improve kernel documentation. I did the best I could, and I still think that locating, collecting, collating, and indexing the huge piles of existing documentation out there is a necessary first step before trying to fill in the holes about what isn't properly documented (at least in any sort of systematic fashion). Plus if you can't _find_ the documentation on something, then it isn't documented no matter what somebody wrote in a blog post back in 2005, or once posted about on linux-kernel, or what the -git commit comment for the feature says.
The Linux Foundation had apparently expected I'd become the kernel's Documentation subsystem maintainer, and that people would naturally start to write and submit more of the stuff once there was one. (Or maybe I could pester them into doing so, or inspire them by example, or some such.) Unfortunately, that's not how maintainers work. A maintainer's function is editorial, they filter through a slush pile of submissions, reject the unacceptable ones, and polish what's left into shape to publish. Doesn't matter whether it's software or print media, the editorial function remains the same. As Alan Cox says, "A maintainer's job is to say no."
Besides, documentation has the worst case of bikeshedding in the entire kernel. Everybody who can read documentation thinks they know how to write it, and how it should be organized. If I tell somebody like Greg Kroah-Hartman "I really don't think foreign language translations should be added to the Documentation directory, they should go on the web", and he ignores me and adds them anyway because he disagrees, what am I supposed to do about it? (Answer: focus on a web page that I _can_ organize into a coherent whole.)
I still need to reach a good stopping point before moving on to other projects. I was trying to do that by the end of October, but they interrupted me to work on SCSI documentation instead and then the 7.10 install ate a week and change, and now I'm at Eric's. The kernel.org/doc page needs about another week or two of work before it's a paticularly useful resource for other people.
I'll start job hunting when I'm back in Austin after Thanksgiving.
Still up in Pennsylvania, visiting Cathy and Eric Raymond.
Finally got mailman set up. There is now a new mailing list for my tinycc project. It might even work.
Mail was bouncing all night because I forgot to run genaliases after adding mailman's virtual domain database to the postfix main.cf. The thyrsus stuff worked because that's the primary domain of sendmail, so that goes in alias_maps instead of virtual_alias_maps. Plus I forgot to change PUBLIC_ARCHIVE_URL in mm_cfg.py when I told apache the archive alias was "lists" instead of "pipermail". It's probably still broken in various corner cases. (I didn't do the recipient delimeter change because it says it's optional, I don't think I enabled that, I don't quite know what it does yet.) But I can has mailing lists now!
Fade is here. Yay!
Fade tells me that the espresso shots packaged in little coffee creamer containers are now on sale, so we don't have to get them from the Exxon anymore.
Got mailman working today. I _think_ it's working, I'll poke at it a bit more before moving over the lists I have on them.com. Eric has a list for the jargon file now, too.
Meanwhile, Eric's been doing his pass updating the 64-bit paper. I tackle this tomorrow.
Eric bought new hardware today, and we spent far too much of the day juggling machines and setting things up.
Got 15 minutes to spend with tinycc and fixed the \n problem (it was a thinko on my part). It now builds and runs "hello world" just fine. Now to get it to build itself, and the test suite. But first, hacking on the build infrastructure a bit more to get all the environment variables set in a separate configure script sourced from make. (And if people run ./configure it'll set the environment variables. But it tests each environment variable first, and won't re-set ones that are already set, so you can override these in your actual environment if you'd like.)
Except that it's not quite that simple. An x86-64 host is also an x86 (32-bit) target, except that the shared libraries are in /usr/lib32 instead of /usr/lib. So if I build an i386-tinycc on an x86-64 host, I need to set the linker's library search path to /usr/lib32 instead of /usr/lib. (Note that varies by _host_, not target. At runtime it all works the same, the library search path is supplied by ld-linux.so.2 when the file actually executes).
Speaking of the runtime linker, tccelf.c has this lovely #ifdef block:
/* name of ELF interpreter */ #ifdef __FreeBSD__ static char elf_interp[] = "/usr/libexec/ld-elf.so.1"; #else #ifdef TCC_ARM_EABI static char elf_interp[] = "/lib/ld-linux.so.3"; #else static char elf_interp[] = "/lib/ld-linux.so.2"; #endif #endif
So FreeBSD sets the runtime linker to /usr/libexec/ld-elf.so.1, ARM EABI sets it to ld-linux.so.3, and everybody else uses ld-linux.so.2. Except that uClibc needs ld-uClibc.so.0. But anyway, that shouldn't be in the C source, that should go in ./configure or something. But that varies by target, which means it would have to re-source ./configure each time through the loop. But _that_ conflicts with the logic in ./configure that won't re-set a variable that's already been set. Hmmm...
All this is fixable. I just need to think about it. I'm trying to come up with a cleaner compiler design than the horrible mess gcc imposes upon people, and I'm still in the process of identifying and discarding all the broken assumptions both Fabrice and myself got from too much exposure to the mass of fundamental conceptual wrongness that is GCC's path logic. (Don't mince words Bones, tell me what you really think.)
Fade arrives tomorrow. Looking forward to this.
That's the second time software suspend (or "hibernate" as Ubuntu insists on calling it) decided it didn't want to suspend. Instead it wanted to blank the screen, do something that occasionally accessed the disk in short bursts, but never flush everything to disk and never power down. It's actually _less_ reliable now than it was in 7.04 (which failed due to lack of allocated disk space, but seldom for other reasons).
Once again, 7.10 is a regression. And you can't blame this one on KDE.
Out of morbid curiosity, I tried to boot the "Indiana" preview (I.E. GNU/Solaris) under qemu. First I tried "qemu -cdrom indiana.iso" but the default memory qemu emulates (128 megabytes) is too little for the solaris kernel to initialize (!). Added "-m 256" to that command line and it went further along, until it went into a slow endless loop doing:
WARNING: /pci@0,0/pci-ide@1,1/ide@1 (ata1): timeout: abort device, target=0 lun=0
Went and played with ReactOS instead. Its livecd boots up to a desktop. Whee. Doesn't do anything, but hey. It's much easier than trying to get Wine to do anything. (Normally, installing this sort of thing is the hard part. But with Wine, getting it to _run_ anything is rocket science.)
Ha! Finally got tcc running under x86-64 to compile a runnable "hello world". It seems to have trouble parsing the \n, but otherwise it worked.
I had to set the paths by hand, though:
TINYCC_CRTPATH=/usr/lib32 HOST=i386 TINYCC_LIBS=/lib32:/usr/lib32 ./make --fast ./i386-tinycc -I /home/landley/tinycc/tinycc/include hello.c ./a.out
On an x86-64 host, the 32 bit libraries are in /usr/lib32 (rather than /usr/lib where they are on an actual x86 host). This means that "x86 on x86-64 system" is actually a slightly different build target. Wheee. Not _quite_ sure how to gracefully configure this yet. Need sleep first.
This means the library search path, the C runtime path (where crt1.o/crti.o/crtn.o live, which isn't a search path but a specific location) all had to be adjusted (above). The header path (where #includes live) are the usual /usr/include because the asm directory of that is full of #ifdefs which include the 32 or 64 bit versions (although since I'm not installing tcc yet, I need to feed it the -I to add the compiler includes from the build directory). And the runtime linker path is the standard "/lib/ld-linux.so.2" because the 64-bit version is apparently "/lib/ld-linux-x86-64.so.2". (They couldn't change that and remain compatible with existing binaries.) I need to make that configurable too, so it can build against uClibc.
Right now there's a lot of #ifdef logic in the source to provide values for various targets. I need to think through how I _want_ all this to configure.
And fix whatever's wrong with \n parsing...
I'd like to thank Jim Tan for the patch he sent, but alas:
host b.mx.jtan.com[207.106.84.147] said: 554 5.7.1 Seems like a
dynamic IP address. See http://www.jtan.com/blocklist/ (in reply to MAIL
FROM command)
Eric Raymond's FIOS line has had the same static IP for a year now, yet some of the black hole lists out there are still broken. This is not my problem, but it's sad I can't think a contributor.
Jim: if you manage to read this, thanks for the patch. I've applied it. And may I introduce you to this marvelous thing called postgrey?
No, I don't know what those A things are, it's a bug in Kmail that inserts high asci crap in the whitespace of anything I cut and paste out of my email. (Makes it really hard to compile C code since gcc dies when it encounters this, yet it doesn't show up in vi for some reason.) Yet another thing to be thankful to Kubuntu about, although this was broken in 7.04 too...
Spent rather a lot of the day setting up a new grelber (the server several incoming ports of the above FIOS line forward to). It went live to mail bouncing (uninitialized alias and greylisting databases) and web pages not resolving (feed the behind-the-firewall IP address to the apache virtual hosting setup, not the outside-the-firewall IP address), but I think those are sorted now.
Did some cleanup work on my new tinycc thing. I need to get it building (and working) on x86-64 before I do another release. (It doesn't have to output x86-64 binaries, it just has to run on an x86-64 host to produce x86 and arm binaries.) Working on it. In addition to the zillions of sign mismatch warnings, there are now "pointer assignment from integer of different size" warnings which are probably actually important. Waffling between cleaning all that up immediately, vs making the minimal changes to get it linking against the right library paths so I can build and test "hello world" before making extensive changes that might potentially break it, without being able to test between each change.
Either way, it's probably not likely to work out of the box. Oh well, I can fix that. (And then learn enough x86-64 assembly to make a new target. Not looking forward to it, but it has to be done.)
Spoke to Fabrice Bellard (ok, typed, he was on freenode as "hint") and made sure he didn't have a major objectin to my tcc->tinycc fork. He said I could do what I like with tcc. Yay. (While I technically didn't do anything that would require extra permission, this makes me feel much much much much better. The Bruce thing left deep scars. Luckily, Fabrice ain't a Bruce.)
I now have two active contributors to toybox, one of whom has queued up a lot of patches for me in his own mercurial repository. I'd like to encourage this, but I couldn't quite manage to clone his repository through static-http. (For the record, I'm a big fan of the mercurial export and import commands. Annotated patches that maintain revision control history. Perfect!)
Went back and repaired the html markup in the November 3 and November 7 entries. The downside of editing my blog in vi is I tend not to view it much in an actual web browser. (It's mostly notes-to-self anyway. Mostly I keep it as is for the historical record, but if I'm going to use the package list in the June 17 entry as a guide to upgrading the next time I install, I'm going to add four or five packages to it when I find out that yes, I had to add those too. A balance between historical accuracy of what I wrote at the time, vs updating my notes to match reality...)
Rsyncing 40 gigabytes of backups from the old grelber to my laptop, so I can sort them into some kind of order. It's copying (via rsync) at just under 1 megabyte per second, and I am so _totally_spoiled_ because that's too slow. I have a laptop that has over 40 gigabytes of currently unused space to copy them to, I have a wireless connection that can go that fast, I am currently doing the download on battery power without particularly worrying about it (the indicator says I'll have to worry in 1 hour, 29 minutes)...
(Smacks gums, shakes walking stick.) I started with a 300 baud modem. (Both ways!) Hooked up to a commodore 64. (In the snow! Or at least with video output that sparkled because of some kind of interference with the character set rom that could cause spurious sprite collisions. Also, the picture was kind of fuzzy if you tried to display red or do an 80 column character display which meant each character was 4 pixels wide, on a monitor that was essentially a high end televison without a tuner. And the display got _really_ upset when mom ran the vacuum cleaner.) It had 38911 basic bytes free and we were _thankful_ to have it! (Just under 39k, for those of you counting along at home. Plus a 4k block at 0xC000, and you could swap out the rom banks to read the memory underneath by zeroing all the bits at location 1, although this meant you lost both the OS and BIOS. Yes, basic can be an OS. Or at least a program loader with built in command language and very special purpose editor and monitor features. Imagine the love child of grub and bash, only much smaller than either of those. And with the screen's character buffer substituting for a command line history. (You want to re-run a previous command, you cursor up, and the cursor moves _up_ and screen scrapes what you typed earlier.)
Those were the days. Boy did they suck, in retrospect. Which is why I'm spoiled by this measly 1 megabyte per second transfer rate. (1000000/38911 is 25.6996736141 according to Python.)
Right, bedtime...
Today looks like it's going to be _insanely_ busy. Oh well, not entirely unexpected.
Something I did installed exim4 on my laptop (possibly an autodependency of mailman?), and it was preventing my ssh email forwarder from binding to port 25 on loopback. That took some head scratching to track down. Yay "lsof -i".
Trying to replace docproc.c with a shell script. This involves understanding how "make xmldocs" works, which is slightly non-obvious. (Writing new code is easy. Reading existing code is always the time consuming bit.)
Argh. When applying a directory full of nicely broken out patches to one's temporary directory, be sure to specify the temporary directory and not the source directory which a mercurial repository that has pending changes in there that haven't been broken out yet. Sigh. There went an unnecessary half-hour of cleanup work, since I did it _twice_. (First I thought patch wasn't breaking hard links. I'm updating make/functions.sh so the temporary copy of the Linux directory uses hard links, which are much faster to create and eat less space.)
Eric is still preoccupied with Wesnoth development. (They apparently just had a release, which he's happy about the bug count being lower than usual in for a C++ application.) Showed him the C++ Frequently Questioned Answers page (much chortling while he read bits of it) and talked about what would be involved in porting more of Wesnoth to Python. This led me to investigate Python OpenGL bindings, which led me to PyOpenGL, which Ubuntu has prepackaged as "python-opengl", which is associated with the truly useless python-opengl-doc package. (How do I write "hello world" code that uses this? Download a demo source tarball from their website, note that it has a 0 byte readme, compile it, note that it won't run if you don't install it and the installation must be run as root and requires a third party installation script that doesn't come with it. Not likely.) But the website's documentation page linked to an OpenGL Tutorial which has a python translation of the example code, and that theoretically can run standalone. Downloaded it, ran it, and it segfaulted in glutInitDisplayMode(). (Not a python traceback, a segfault. Which really shouldn't happen in python code.) This is a python opengl app to open an empty window. It doesn't even try to draw a polygon (that's lesson 2). And the package segfaults.
Eric's of the opinion that if I download it and build it from source, the segfault will probably go away. Let me just say that I am so deeply impressed by the Ubuntu 7.10 release right now, words fail me. So do random Ubuntu packages, in unexpected ways. Right, on to other things...
AAAAAAARRGH! Speaking of compiling things from source, the x86_64 cross compilers I built for gcc are broken. I didn't notice because all the mini-native stages built, because the breakages is in the path logic again. (If you move the toolchain to another directory, it goes "boing" again.)
It's using the host assembler for all targets, because once again the path logic sprug a leak. Because none of the 15 places (!) it's looking for the assembler actually contain it if you relocate the cross-compiler-$ARCH directory, and thus it falls back to the one in $PATH. I can make the wrapper adjust the $PATH for its child process so the right one is first, but then collect2 can't find ld (which we just adjusted the path to _add_!) I can delete collect2 and symlink it to the correct ld during toolchain prep, but I wanted to add C++ support to this thing someday...
And you wonder why I'm rooting for pcc and banging on tcc? This thing is _broken_...
But anyway, I _did_ confirm that Ubuntu x86-64 does indeed run 32-bit executables, no problem. Good to know. And I can tell tcc to build against /usr/include (which has an asm directory full of #ifdefs for i386 and i686) and link against /lib32, and output 32-bit exectuables which I can run right here to test.
Still getting used to the fact that the video on my laptop is now the correct 1280x800, not 1024x768 stretched to a cinema display. This is a _good_ thing about the upgrade to 7.10, I guess I was just spoiled by the preinstalled 7.04 where just about everything worked out of the box.
Arrived at Eric and Cathy's. Too late to get any work done, I'll catch up Friday and over the weekend.
A big advantage of the new make/make.sh is that it won't delete the old htmldocs directory when it's not rebuilding it, so I should be able to rsync updated versions with impugnity (or something spelled similarly to that). (Technically I separated them so htmldocs is no longer a symlink to Documentation/DocBook, so rebuilding Documentation doesn't delete the old htmldocs. Wrote an actual installer to move files from one to the other.)
Well it's a big advantage to _me_. I've accidentally blanked the http://kernel.org/docs/htmldocs directory and left it that way for days enough times to not want to do that again.
The merge of the i386 and x86_64 architectures introduced a new architecutre, x86, which is "special". Every other architecture has an arch/$ARCH/ directory containing Kconfig and Makefile entries. If you try to "make ARCH=walrus allnoconfig" it complains "No rule to make target `.../arch/walrus/Makefile'." meaning if you _added_ that it would use it without modification to the rest of the kernel infrastructure.
But the new x86 directory doesn't have a "Kconfig". Instead it has Kconfig.i386 and Kconfig.x86_64, and it added special case code to the top level makefile to check for these and redirect them. Yet x86 itself isn't usable as an architecture, if you try to allnoconfig it you get "can't find file arch/x86/Kconfig". Because other architectures like powerpc can combine 32-bit and 64-bit versions in a single directory and still work like all the other architectures, but x86, isn't that special? It's no longer detected by the script that generates http://kernel.org/doc/menuconfig because it's just that special.
Meanwhile, while trying to document menuconfig and I just realized I can't _run_ menuconfig until I get network access again. Kubuntu installed curses, but not curses-dev. (Wouldn't it be nice if there was some way to tell it "install the -dev for every package that is otherwise installed? I have the _libraries_ but not the header files. Considering that Firefox alone is 32 megabytes, and KDE is enormous, why are they quibbling over header files?)
Speaking of firefox, here's what "aptitude show firefox" has to say:
Description: lightweight web browser based on Mozilla Firefox is a redesign of the Mozilla browser component, similar to Galeon, K-Meleon and Camino, but written using the XUL user interface language and designed to be lightweight and cross-platform. This browser was previously known as Firebird and Phoenix. This is a build of a random development version (aka trunk). It is ment for preview and not for production use.
No, I didn't build it myself. That's the version that comes with 7.10. And no, my cut-and-paste didn't respell "meant" either.
Tenessee seems to have a disproportionate number of cute girls. I wonder why?
Poking at the kernel build, trying to make "ARCH=x86" build. It seems to want to default to i386, so I'm making a list of all the places I need to force it to x86-64. It's getting -march=i386 from _somewhere_, and the only place that's set is arch/x86/Makefile.cpu, which is only included from Makefile_32, which is included from Makefile (all in the same directory), but tweaking Makefile to include Makefile_64 instead doesn't seem to have any effect. (Possibly commenting out an include directive doesn't work. I'd hate makefiles much less if I could stick a printf in them to see what variables are set to at a given point, but they're explicitly designed not to allow that, which is sad.)
It has something to do with bouncing off of scripts/Makefile.build... Ah, that needs to blank KBUILD_CFLAGS. Except that breaks the build later on. (Breaks the x86_64 build, too.) Right, worry about that later.
So "sed -n 's/^!.//p' Documentation/DocBook/*.tmpl" produces a list of 300 file imports of which 184 are sourced for Exported symbols, and 111 for Internal symbols. (There are also 4 F references and 1 D reference.) Sorting for unique files gives 271, so several are imported more than once. let's take a random look at one, kernel/resource.c which is imported in kernel-api.tmpl line 307:
<sect1><title>Resources Management</title>
!Ikernel/resource.c
!Ekernel/resource.c
</sect1>
So, A) this file is being imported twice, once for each kind of symbol, B) the symbols aren't marked as to whether or not they're exported, and what type of export it is.
Ok, so half of the function of docproc.c is to find all the exported symbols in the files. As far as I can tell, this is several hundred lines of C to duplicate approximately the following shell script snippet:
for i in $(sed -n 's/^![EI]//p' Documentation/DocBook/scsi_midlayer.tmpl) do echo $i egrep -o "EXPORT_SYMBOL(|_GPL)[ \t]*\(.*\)" $i done
Watching somebody write C to avoid a five line shell script is... Um... Painful. Oh well, at least they didn't do this part of it in perl.
So docproc.c shells out with 'kernel-doc -docbook -nofunction function_name filename", using -nofunction for internal functions and -function for external functions. Can kernel-doc take a mix of those? (No of course not, that would be too easy.) Reading the perl... What a mess. In addition to the xml output type being used, this thing can also output html, gnome, man, and text. This script is carrying along huge amounts of code that are never used. Of this 2007 line perl script, maybe only half that is actually needed.
Ow. Headache. (I can't read much perl in a sitting. I can untangle some pretty bad code, but my brain rebels against even the cleanest perl. It's evil.) Ok, back to driving...
Starting the long drive up to Eric and Cathy's. I planned the trip when I thought the documentation job was over, but I have my laptop with me and can work on stuff along the way.
Still fighting with Kubuntu 7.10. They've broken vim in new ways every release for a while (obviously a distro put out by people who don't like vi), but the bit I can't figure out how to undo is that it stopped scanning embedded vi comments, ala:
/* vi: ts=4 :*/
That sets the tabstop to 4, which is what I use when I write code. (Yeah, the kernel says you should indent 8 spaces. This is one of the reasons I don't write much kernel code. Until I spent time on busybox I used to use two spaces per indent.)
It's hard to look up something like that when you're not sure exactly what it's called. Possibly I can set something in the config file to bring it back, but if so I don't know what. Several things they removed (like syntax highlighting) were yanked at compile time. I tried looking for a bigger vim package, but my first attempt didn't help matters... Ah, there's a "vim-full". Maybe that's worth a try... (Sigh, they built it against the gnome libraries. It's a command line text editor, and they infected it with Gnome. That's sad. If I wanted to use kate, I'd use kate.)
Nope, it _still_ isn't reading the tabstop reset out of the file, which the previous version was doing.
Not a happy Kubuntu user. The networkmanager bug shows no progress, and none of the ones I've filed have been responded to, that I've noticed.
[Many hours later]
I love mercurial. I'm on the road, running from battery, no net access to be seen (this truck stop claims to have some, but it doesn't work), and checking the updated make scripts into the kernel documentation repository, locally on my laptop. This is _so_cool_.
I also like my new laptop. As much as I'm annoyed at Kubuntu for once again breaking stuff that USED TO WORK, moving from 32-bit to 64-bit is like getting a whole new laptop, only 6 months after actually getting a whole new laptop. I can has battery life!
I'd give Kubuntu a lot more leeway if the breakage I've experienced recently had anything to do with moving from 32-bits to 64-bits, but it mostly didn't. The hard drive thing was arogance. Knetworkmanager is just stupidly designed. I'm repeatedly told that using flash 9 via 32-bit nspluginwrapper works for other people, so why doesn't it work for me? Why does vim's feature set contract so drastically each version? And so on...
In each of these cases, half the problem is that whatever it is has been automated to the point where there's no easy way to diagnose the problem. It either works, or a failure occurs deep in the guts of something and I have to pull out my UberGeek hat to figure out what's actually gone wrong, let alone how I might possibly fix it. The hard drive thing just lurks until the drive fails. The browser plugin recognition thing is so automated I can't point it at a plugin sitting in a known location, or get it to say WHY it doesn't want to load it (or if it even noticed it was there). The whole knetworkmanger experience is layers of opaque breakage (silently ignoring my hardware, silently crippling konqueror, and I can't get either to tell me anything, they just _fail_ with no explanation at all)...
Oh well. Still, yay Mercurial, yay battery life, yay 64-bits. (And some things are better in the new kubuntu. For one thing, the "dynamic" frequency scaling no longer seems to get stuck at 800 mhz.)
It's monday, back to the salt mines.
So, when the scsi documentation thing came in as an interrupt, the kdocs make/make.sh was on the floor in pieces being converted from a big shell script into a series of smaller shell scripts. That's why kdocs site hasn't updated in a few days (including getting the new scsi_midlayer.tmpl into the htmldocs page).
The advantage of this breakup is that the big script did lots of things and sometimes I want it to do just _one_ of those things (such as rebuild index.html from master.idx). Doing all those things together is very very slow. (For example, make htmldocs is a 5 minute job even on my laptop. The "many sections" version is _way_ slower than the -nochunks version, I think that it's re-parsing the entire xml file for every separate html file it outputs. And -j 2 makes no difference because this part of the kernel build is wrapping a shell script. Anyway...) The search for RFC entries in source comments is almost that slow as well, and I didn't want to make that an unconditional stage, but adding more "--short" style flags with different granularity was getting silly.
So I broke the script up, but the downside of having lots of small shell scripts turns out to be lots of duplication. The checks that $LNXDIR and $WEBDIR are set get a bit repetitive after the fifteenth one of those; common code seems like a good thing.
So when I was almost done breaking them up, I glued all the little scripts back together into one big include file and I'm making it define lots of bash functions(). So if I need to call one or two of the things, I can source the file and call the functions by hand. (Or make a wrapper script with arbitrary granularity, in about 30 seconds.)
Of course I broke stuff doing two extensive conversions back to back without without ever reaching a good testing point, and now I'm debugging it all again. I'd have finished this last week if I hadn't lost several days upgrading my laptop to the pile of bugs that is Kubuntu 7.10.
It's a monday...
Tried easybuntu to see if that could install the flash plugin. Nope, it hasn't got a config for Gutsy because that one "just works" for a definition of works that does not actually involve working.
Wow, 64-bit mode builds very very slowly compared to 32-bit mode on the same hardware. Probably the 64 bit integer sizes are blowing through cache more quickly (almost certainly doing bad things to the stack). Or it could just be gcc, which is what's been trying to build for what, half an hour now? I haven't timed it...
Nope, it's gcc. All the other packages seem to build about as fast on x86-64 as they did on x86, but gcc builds waaaaaay slower.
[12 hours later] Ok, finally killed the "./forkbomb.sh --fork" session. For some reason, m68k died with an internal compiler error (maybe that one doesn't work on an x86-64 host, have to see if it's reproducible), and none of the others had made it out of the cross compiler build after being left to run overnight. (The most advanced one was trying to build uClibc.) Back on x86 on the same hardware, an entire forkbomb parallel build of all targets took about 3 hours total.
At a while guess, the CFLAGS="--param ggc-min-expand=0 --param ggc-min-heapsize=8192" I'm doing is starving 64-bit gcc, at least when building itself. That optimization works fine on 32-bit platforms, and makes the insn-attrtab build of gcc _not_ trigger the OOM killer on a virtual machine with only 128 megabytes of memory. I'll try making that conditional on [ "$(uname -m)" != x86_64 ] and see what happens. (It would be nicer to test sizeof(long) without having to compile anything, but I haven't figured out how yet. Nothing obvious under /proc or /sys presents itself, sysinfo is useless, none of the standard environment variables shown by "set" indicates this...)
Ooh, that's _much_ happier. Built x86-64 in 24 minutes, 6 seconds (with only 27 minutes "user", so not taking as much adantage of SMP as it could, but that's what --forkbomb is for.)
So, that optimization is only correct for 32 bit systems. Maybe gcc-min-heapsize=16384 is a better on 64 bit? I can play with this...
Darn it, I had a mechanism to tell vi not to call fsync() on its .swp file every hundred keystrokes (In a heavily I/O bound system causing the process to pause for 20 seconds or so, a _pain_ when you're typing). But I apparently didn't note it in my blog. Yeah, the new install's doing that and I forget how to switch it off. It was some "set" command in the vimrc file...
And I can't search the help because they broke that into a separate package I haven't installed yet (vim-doc) and I'm at a starbucks which has T-mobile branded lack of internet access. (And no more blackberry flavor syrup until spring.) Great...
Ok, whereis vim, "strings /usr/bin/vim | grep sync", and that contains "swapsync". Try ":set swapsync=n"... well, it took it. But no, I just got the darn pause again. That didn't work. Hang on, but was it a bad pause? I _am_ doing a forkbomb build in the background, with the memory constraints I'd added switched off. The occasional three second pause while vi gets knocked to swap is kind of expected. It's a twenty second pause twice in the same minute that gets on my nerves... (Yes, I'm typing to see what the pause behavior is like. So far, seems reasonable.)
Kubuntu 7.10 still sucks: the saga continues.
I shut down the laptop last night rather than trying to suspend, on the theory that Kubuntu might think it's windows and that behaving differently after a reboot was somehow a good thing. But no, knetworkmanager is still broken this morning.
Went into the system settings menu, fired up network settings, switched off "zeroconf network browsing". Nope. All the other buttons are greyed out. Ah, eventually figured out I had to scroll down to see the "administrator mode" button (despite my display being bigger than 1024x768 now and the window maximized: it's designed for a display _bigger_ than 1024x768. Brilliant.) Went in and told both network cards (which _this_ thing can also find; EVERYTHI