At the time, IBM researchers found that most of these errors were coming from impurities in the chip packaging itself, which is a solvable problem.  But a small number of errors were caused by cosmic radiation; high-energy particles generated by supernovae and other events in deep space.  The earth’s atmosphere and magnetic field shield us from most of these particles, but a few will always get through.  While reading about this in high school, it occurred to me that this could be a serious problem for computers in space.  So I wrote up an experiment proposal for NASA’s Space Shuttle Student Involvement Project, and was accepted as a finalist.  (While presenting my work at NASA Ames, an engineer asked me if he could blame all of his bugs on cosmic rays.  I replied with what I had calculated as the error rate for a single 4kb chip: “Sure, once every sixteen years.”)   The problem remains, but most hardware manufacturers consider it a one-in-a-million event, and pretend it doesn’t exist.  Crashes due to software bugs are vastly more common.

Thus it is interesting to hear of a new long-term study by Google on DRAM errors.  Google has so much computing capacity that it is uniquely positioned to perform such research (and it’s gratifying that they take the time to do so).  Surprisingly, the study found error rates 15 times higher than previously expected, but they also found that most of those errors were coming from the same group of chips, suggesting that manufacturing variability is a cause.  The study also found that age is a factor, with error rates sharply increasing at 20 months.  This could be due to chip designers cutting their silicon margins to the bone, knowing that due to Moore’s law, commercial hardware has a very limited lifetime these days.

What I took away from that early work was that nothing is perfect.  If you’re programming a computer and you put a byte of data into memory, the chances are extremely good that you’ll get the same byte back.  But it’s not 100% guaranteed.  And if it’s VERY IMPORTANT that you get the same byte back, you should invest in fault tolerance of some sort.  Spacecraft computers work around this problem with special hardened construction, redundant “voting” systems, and error-correcting codes in memory (which are an effective solution to the DRAM error problem, though it costs an extra 25% to store the correcting information).  I sometimes  wonder why these reliability lessons from space aren’t incorporated more into our everyday machines, but until software quality improves, it’s usually cheaper to just reboot.

• DRAM study turns assumptions about errors upside down (via Ars Technica)

One thing I really like about Armadillo is that from the beginning they’ve done their engineering in small increments, fixing what doesn’t work as they go along.  They’ve been at this a long time and their logs are fascinating to read; full of endless problems with metallurgy, thermodynamics, supersonic flow, control… all encountered through thousands of tests, and all eventually solved.  Founder John Carmack, who made his fortune from DOOM and other video games puts it best:

When asked what lessons the traditional aerospace community should learn from their success, Carmack answered, “You learn so much more by getting out there and doing things than you do sitting at a desk running a CAD program. You can’t even imagine some of the things that wind up going wrong. It’s the unknown unknowns that get you. You wind up getting things done by going out there and trying it, accepting levels of failure and you beat the problem into submission by working on it over and over and over again. And when you can build your operations tempo up to doing things every day, that’s what we want to see in the aerospace world.”

Exactly.  They’ve definitely earned this one, and the million dollar prize ain’t bad either.

• Armadillo Aerospace Qualifies for Northrop Grumman Lunar Lander Challenge

via OnOrbit

BeOS was an extraordinary operating system available in the mid-90s.  Lean, modern, media-savvy, and blazingly fast, it was a joy to use.  BeOS went through a number of evolutionary stages, running on proprietary, then Apple, then PC hardware; but each effort ran into brutal anticompetition tactics from both Apple and Microsoft.  (And to be fair, BeOS suffered from poor marketing of an admittedly difficult-to-market product.  Why should you buy something that comes “free” with your computer, even if it is mediocre, if it means giving up the benefits of ubiquity?)  In desperation, the company (Be) changed their focus to “internet appliances” instead of general-purpose desktop software, but this failed when the dot-com bubble burst.  Despite heroic efforts within and outside the company, Be went bankrupt and was sold to Palm (which did very little with the BeOS technology), and a small handful of BeOS revival efforts eventually faded away.

Except for one, the Haiku project.  Unlike some revival efforts, which used illegally stolen code, or proposed creating Linux windowing systems that just looked like BeOS, the Haiku project rewrote BeOS from the ground up.  Because they didn’t have access to all the minute details of the real BeOS internals, Haiku is not binary-compatible with BeOS.  But it IS source-compatible (existing code can be recompiled to run on Haiku), and has the same great look and feel that made BeOS such a pleasure to use.

It’s a valid question as to whether it’s worthwhile to try to revive orphaned operating systems, especially for desktop use.  There’s definitely a critical mass where the number of users vs. the amount of development feed each other; below that limit an OS is not likely to survive.  But there’s something to be said for a healthy ecosystem requiring diversity, in both the natural and computer worlds.  Without competition, the few remaining choices become stagnant and bloated.  Sometimes something revolutionary is needed to keep everyone on their toes.

• In the Round: Haiku Alpha Released (a good summary of the history of Be, BeOS, and the Haiku project).
• Haiku project announces availability of Haiku R1 Alpha 1 (announcement and download link from Haiku-os.org)

Via OSNews.com

So how did Steven Colbert get this picture of me kiteboarding?  I don’t even remember that day (though it looks like I should have.)

“Of all domestic animals the cat is the most equivocal and suspicious. He is kept, not for any amiable qualities but purely with a view to banish rats, mice, and other noxious vermin from our houses, granaries, etc. Although cats, when young, are playful and gay, they possess at the same time an innate malice and perverse disposition, which increases as they grow up, and which education teaches them to conceal, but never to subdue. Constantly bent upon theft and rapine, though in a domestic state, they conceal all their designs; seize every opportunity of doing mischief, and then fly from punishment. “They easily take on the habits of society, but never its manners, for they have only the appearance of friendship and attachment. This disingenuity of character is betrayed by the obliquity of their movements and the ambiguity of their looks. In a word, the cat is totally destitute of friendship; he thinks and acts for himself alone. … Their sleep is light; and they often assume the appearance of sleeping when in reality they are meditating mischief.”

SFGate: Deuce of Diamonds, by Jon Carrol

Bill worked for NASA back when they took the Aeronautics in their name seriously. He revolutionized air-traffic control software, and then left to pursue his dream of making an airplane for everyone, one that used advanced software to make it easy enough for anyone to fly. (His initial project towards this goal was to make a box that could be installed in an aircraft that had just one big red button on it. If you hit the button the box would immediately take over and safely land the plane at the nearest airport.) He and his wife Susan (just as brilliant, just as nice, not quite as intense) owned a small company in Boulder which did this and other work. I worked with them the summer of 2001 on a tiny unmanned airplane that they’d use to gather weather data, but was also unofficially testing out the core software that would eventually run Bill’s dream. The first time I met Bill, I dressed up nicely for a formal meeting. He came jogging in in a runner’s tank top and shorts, drenched with sweat. He sat down, toweled off, and started right in on the technical details he needed help on. I was very impressed by his philosophy of what was really important, and his refusal to do what wasn’t. I also remember Susan asking me how much money I wanted to make. I threw out a number. She said that’s not enough, and threw out a bigger number. This is the kind of people they were.

Immediately after 9/11 most of their weather contracts were canceled as government money moved to homeland defense. They could have bought into that but refused to on karmic principles. They hung on for a bit but eventually had to sell the company. Bill was devastated by having to let his friends go, and said he’d never work for the government again. They moved to Florida (I was very sorry to see them leave Boulder), and they eventually lived on a 50′ catamaran. Bill made the catamaran into a technological wonder, even putting in a home theater because Susan loved movies. This year they decided to move back onto dry land so their daughter could go to a real high school. Susan got an executive position at a biofuels company in Boston, and Bill was working on a new aviation startup in Kansas (he liked the boat but really missed flying). He was an avid runner and biker, and sadly a few months ago he was killed by a drunk driver while riding his bike.

Bill touched a lot of people’s lives.  The memorial dinner was about 20 people, most from the old company, many of which I had met but hadn’t seen since 2001.  I was honored to be invited since I had only known Bill a little while, but Susan once said he and I were simpatico. (Aside from consulting for the company, I also helped Bill with his annual high-tech Halloween show in his front yard. It was the year 2001, so the theme was 2001 A Space Odyssey.)  Heard lots of great stories tonight, and feel for Susan and her two children’s enormous loss.  I wish I had been able to spend more time with him, Bill’s the kind of person I want to be when I grow up.

1. Use the time of a total stranger in such a way that he or she will not feel the time was wasted.
2. Give the reader at least one character he or she can root for.
3. Every character should want something, even if it is only a glass of water.
4. Every sentence must do one of two things – reveal character or advance the action.
5. Start as close to the end as possible.
6. Be a sadist. No matter how sweet and innocent your leading characters, make awful things happen to them — in order that the reader may see what they are made of.
7. Write to please just one person. If you open a window and make love to the world, so to speak, your story will get pneumonia.
8. Give your readers as much information as possible as soon as possible. To heck with suspense. Readers should have such complete understanding of what is going on, where and why, that they could finish the story themselves, should cockroaches eat the last few pages.

“Hello, babies. Welcome to Earth. It’s hot in the summer and cold in the winter. It’s round and wet and crowded. At the outside, babies, you’ve got about a hundred years here. There’s only one rule that I know of, babies — ‘God damn it, you’ve got to be kind.’ ”

Via

The six weeks went by quickly (with all the drama of any good engineering project), and on March 29th the regional competition began at the University of Denver’s Magnus Arena. The rookie Fairview team did an outstanding job in both robot construction and driving, even being called out by the announcers for helping push another robot up a ramp at the end of a match.

After a day and a half of competitions, the top 32 robots went on to elimination matches. Unfortunately, Fairview was ranked 33rd! The team was a little bummed out that they had come so close to continuing. But then a higher-ranked team chose Fairview to be on their team (due to their excellent driving skills and teamwork), so they got to continue.

In the elimination matches (which were best-of-three), Fairview lost the first match, won the second (and got to continue!), but lost the third 140 to zero (ouch). But it was a heck of a ride. This was Fairview’s first FIRST competition, and the students are already planning for next season.

Final standings (Fairview is team #2036)
Lots of pictures in the Flying Circuits Gallery

12/24/06: Receive shiny new Thinkpad T60p.

1/23/07: Finally realize only one speaker is working (the left one). Call Lenovo. They are very apologetic, arrange for warranty repair.

1/25/07: Empty box arrives. Pack up computer. Call DHL to pick up box.

2/1/07: DHL picks up the box after several no-shows.

2/7/07: Wrong computer returned (DHL’s fault, dropped off wrong but identical box). Mixup cleared up several hours later.

2/7/07: Dead speaker now works, but has obvious problems (consistently pops loundly on mute/unmute, and doesn’t work at all at a specific volume level). Only the repaired speaker has these problems, the left one works fine as it always has. Since the same thing happens in both Windows and Linux, signs point to defective hardware. (Wish I had noticed this before the first service call, but the dead speaker masked the problem.)

2/9/07: Call Lenovo. They are very apologetic, promise to escalate to next level.

2/12/07: Empty box arrives. Pack up computer. Include very specific note about problem, including the instruction to not return it until the problem is solved. Take box to DHL (not messing around with pickup again).

2/13/07: Tracking shows computer arrives at Lenovo service and leaves six hours later.

2/14/07: Repaired computer left on doorstep while I wasn’t home (the first one properly required a signature.) And for some reason, the box is unsealed which is very disturbing.

2/14/07: Power it up, but my heart sinks when I hear the same “pop” from the speaker. Absolutely NOTHING has changed on the computer. There’s a generic “Repair Action Report” included, but nothing indicating what was or wasn’t done and why. I’m livid; I need this laptop operational NOW.

2/14/07: Call Lenovo. Very apologetic. Promise to escalate to next level. I say that’s what was promised the last time. They suggest I speak to a manager. Hold for 30 minutes. No managers available, told manager will call me personally in 24 hours.

2/15/07: 24 hours, no call from manager.

2/16/07: Call Lenovo. Ask to speak to manager. Hold for 5 minutes. No manager available, told my case is now on “hot line” status, they will be calling me. Say they were supposed to call me in 24 hours but didn’t. Now told that they’ll call within 72 hours. Ask for number for “hot line” people. Told that they have no contact information for them. Ask what to do if they don’t call within 72 hours. Told to call back and they’ll escalate it again.

2/16/07: “Repaired” speaker fails (I can’t make this stuff up). At least I don’t hear the pop now! (But the problem is still very apparent on headphones).

2/19/07: 72 hours have gone by without a call. Call Lenovo. They put me on hold briefly, then say the person in charge of my case is “unavailable”. I ask for contact information, they say they don’t have it, only his name (“Kier”). I ask what to do. They say to wait for his call. I say he hasn’t called in the timeframe(s) I’ve been given, what if he never calls, am I just stuck? Sounds like it. I ask for suggestions on what to do. They give me the number for the repair center, saying I could call them and see what happened during the repairs. (Aside from the fact that this is someone else’s job, they’ve “repaired” my machine twice and obviously couldn’t fix it, so calling them doesn’t seem like it would be productive.) Am told that when a machine gets sent in for a third time is when it is escalated. (I was previously told this would happen the second time, but once is too much: this should have been caught at the factory.) I mention that I bought a Thinkpad because of its reputation for build and service, but so far I haven’t seen it. Apologies, etc. but they’ve done all they can do (?!). Am now thinking of an end-run to corporate, but I see that the gentleman in charge of American operations recently resigned…

2/21/07: Write letter to the interim head of Lenovo’s US operations, hoping that he’s interested in the state of his company’s service system, and hoping he can get things moving from the top (since I seem to be stuck trying to get things moving from the bottom).

2/27/07: No response to letter and no call from Lenovo. Call Lenovo. Tell another person the whole story. Put on hold for 10 minutes. Person assures me that someone from the hotline will call today. Gets cell number and email to be sure they can reach me. As of 10PM still no call, message or email, but they may have a different definition of “day” than I do.

2/28/07: A CALL FROM IBM! (and within 24 hours!) They say that two parts were ordered for my machine, and one is still on backorder. This is complete news to me, if anyone had ever bothered to tell me, either in the first place or during one of my calls since, that they were waiting for parts and (I’m guessing) returned my laptop for me to use in the meantime, much of the above wouldn’t have happened. They say the case number is the same as before (but still give me a new one), give me a new phone number to call (woohoo!), and say they’ll call me when the part is in. My confidence is still low, but at least there’s some movement. Surprise me, IBM!

3/2/07: SUCCESS! IBM did surprise me (in a good way). A very helpful IBM field technician named Steve called me, said he had received the parts that had been ordered for my machine, and scheduled an appointment to replace the T60’s mainboard at my location (!) I was surprised because nobody had informed me what to expect (par for the course), so I expected to send my machine in again (which I was nervous about given the previous events). Steve came over with a new speaker and motherboard, a toolkit, and a field ESD mat. I demonstrated the problem, and he astutely theorized that the click was actually blowing out the right speaker, which certainly fit the chain of events. He quickly disassembled the machine, swapped the CPU to the new motherboard, and put everything back together (no extra screws). It took about 45 minutes, and when he was done the audio worked PERFECTLY. I was extremely impressed. Thank you Steve (we had a great geek talk while he was working), and thank you IBM for escalating this to a tech who could fix the problem. (Additionally, the same day I got a phone message from IBM executive services following up on my letter to corporate. I will call them back to tell them that I’m happy with the resolution, but also that there are deficiencies in their service process).

3/7/2007 The nice gentleman from IBM corporate has called me back several times but I haven’t been available during business hours to talk to him. Finally catch his call and give him a quick synopsis of the issues and outcome. I say that by far the main problem I encountered was communications with the customer. He is very apologetic and says he will check with the people who worked my case and try to determine why they repeatedly failed to contact me. After I hang up I wish I had also added that they should look into a different carrier than DHL.

I’m VERY happy with the final result, but this still took much more time and stress than it needed to. Lessons learned:

• These things come with problems from the factory. If I ran the zoo, that would be the first place I’d work to improve things (inspection should have caught this problem).
• The front-line repair system should have caught the secondary problem (loud click) while testing their repair of the first problem (repair speaker).
• I don’t know if it was DHL or the front-line repair people, but someone is sloppy when it comes to return packaging and shipping (my guess is both of them). DHL shipping was very fast, but the handoffs were repeatedly a PITA.
• Both the front-line and escalated service divisons need to communicate better with the customer. If the front-line tech had told me that they were waiting on parts (they even sent a repair report to list such things which wasn’t used), or the escalated system had called me (as they were supposed to) and told me the same thing, I would have been happy to let them do their jobs. As it was, I thought they had abandoned this customer.
• There is a severe communications disconnect between the front-line service system and the escalated system. I had no way of reaching the escalated system directly when they didn’t call me (it took two weeks and four calls to get a response out of them).
• The on-site repair service is definitely worth it if you can afford it, but it would be nice to see that level of customer care extended to all issues at all levels.

The great thing is that the issue is solved, and I now feel that the laptop is finally mine and I can commit to fully moving my work over to it. Thanks for listening, flame off.

From the title of this article I thought computer science legend Andrew Tanenbaum would be discussing one of my interests; user interfaces and software design for non-computer literates (see the remarkable design of the OLPC). However, he was actually speaking about another of my interests; the sorry state of what passes for consumer operating systems today.

Tanenbaum, who developed a small teaching OS called Minix which was a precursor to Linux, recently spoke at a Linux conference about a fundamental and escalating problem with modern operating systems: the fact that they’re big and getting bigger automatically means that they’ll be increasingly buggy. He supports a return to the fundamentals: small modules of code which can be kept reasonably bug-free, and isolating these modules from each other so a failure in one doesn’t cascade to others. Hopefully the Linux developers he spoke to take this advice to heart; Linux isn’t immune from this effect.