Welcome to Mike Grusin's

“The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.”
-Douglas Adams, Mostly Harmless

Bitflips

Filed under: Uncategorized — mgrusin at 9:43 am on Thursday, October 8, 2009

65595_memorychipMy first space “work” was with, what was back then, large-scale DRAM memory chips.  (By “large” I’m talking 4096 bits per chip; this was the very beginning of the personal computer revolution.  You kids get off my lawn!)  Shrinking the circuitry was increasing the capacity of these devices by leaps and bounds, but unfortunately this was also causing an alarming rise in random, radiation-induced errors.  Smaller circuitry carries a smaller amount of charge, making it increasingly susceptible to the charged particles that are always zipping around and through us.  A charged particle hits a memory cell, and suddenly what was once a “1” is now a “0”, and your bank balance (or anti-lock braking system or rocket ignitor) is now better or (more likely) worse than you left it.

At the time, IBM researchers found that most of these errors were coming from impurities in the chip packaging itself, which is a solvable problem.  But a small number of errors were caused by cosmic radiation; high-energy particles generated by supernovae and other events in deep space.  The earth’s atmosphere and magnetic field shield us from most of these particles, but a few will always get through.  While reading about this in high school, it occurred to me that this could be a serious problem for computers in space.  So I wrote up an experiment proposal for NASA’s Space Shuttle Student Involvement Project, and was accepted as a finalist.  (While presenting my work at NASA Ames, an engineer asked me if he could blame all of his bugs on cosmic rays.  I replied with what I had calculated as the error rate for a single 4kb chip: “Sure, once every sixteen years.”)   The problem remains, but most hardware manufacturers consider it a one-in-a-million event, and pretend it doesn’t exist.  Crashes due to software bugs are vastly more common.

Thus it is interesting to hear of a new long-term study by Google on DRAM errors.  Google has so much computing capacity that it is uniquely positioned to perform such research (and it’s gratifying that they take the time to do so).  Surprisingly, the study found error rates 15 times higher than previously expected, but they also found that most of those errors were coming from the same group of chips, suggesting that manufacturing variability is a cause.  The study also found that age is a factor, with error rates sharply increasing at 20 months.  This could be due to chip designers cutting their silicon margins to the bone, knowing that due to Moore’s law, commercial hardware has a very limited lifetime these days.

What I took away from that early work was that nothing is perfect.  If you’re programming a computer and you put a byte of data into memory, the chances are extremely good that you’ll get the same byte back.  But it’s not 100% guaranteed.  And if it’s VERY IMPORTANT that you get the same byte back, you should invest in fault tolerance of some sort.  Spacecraft computers work around this problem with special hardened construction, redundant “voting” systems, and error-correcting codes in memory (which are an effective solution to the DRAM error problem, though it costs an extra 25% to store the correcting information).  I sometimes  wonder why these reliability lessons from space aren’t incorporated more into our everyday machines, but until software quality improves, it’s usually cheaper to just reboot.

Armadillo Aerospace reaches beta (with a large bonus)

Filed under: Uncategorized — mgrusin at 9:35 am on Tuesday, September 15, 2009

CRW_7604Another X-Prize won!  Armadillo Aerospace is the first team to win the Lunar Lander Challenge (but the demonstration window is still open, and other teams could still share the prize money).

One thing I really like about Armadillo is that from the beginning they’ve done their engineering in small increments, fixing what doesn’t work as they go along.  They’ve been at this a long time and their logs are fascinating to read; full of endless problems with metallurgy, thermodynamics, supersonic flow, control… all encountered through thousands of tests, and all eventually solved.  Founder John Carmack, who made his fortune from DOOM and other video games puts it best:

When asked what lessons the traditional aerospace community should learn from their success, Carmack answered, “You learn so much more by getting out there and doing things than you do sitting at a desk running a CAD program. You can’t even imagine some of the things that wind up going wrong. It’s the unknown unknowns that get you. You wind up getting things done by going out there and trying it, accepting levels of failure and you beat the problem into submission by working on it over and over and over again. And when you can build your operations tempo up to doing things every day, that’s what we want to see in the aerospace world.”

Exactly.  They’ve definitely earned this one, and the million dollar prize ain’t bad either.

via OnOrbit

Haiku (finally) reaches alpha

Filed under: Uncategorized — mgrusin at 5:20 pm on Monday, September 14, 2009

haikuIt’s taken a while, but Haiku, an open-source project to replace an orphaned operating system called BeOS, has finally reached alpha stage.  To understand why this is exciting, you need to know a little history:

BeOS was an extraordinary operating system available in the mid-90s.  Lean, modern, media-savvy, and blazingly fast, it was a joy to use.  BeOS went through a number of evolutionary stages, running on proprietary, then Apple, then PC hardware; but each effort ran into brutal anticompetition tactics from both Apple and Microsoft.  (And to be fair, BeOS suffered from poor marketing of an admittedly difficult-to-market product.  Why should you buy something that comes “free” with your computer, even if it is mediocre, if it means giving up the benefits of ubiquity?)  In desperation, the company (Be) changed their focus to “internet appliances” instead of general-purpose desktop software, but this failed when the dot-com bubble burst.  Despite heroic efforts within and outside the company, Be went bankrupt and was sold to Palm (which did very little with the BeOS technology), and a small handful of BeOS revival efforts eventually faded away.

Except for one, the Haiku project.  Unlike some revival efforts, which used illegally stolen code, or proposed creating Linux windowing systems that just looked like BeOS, the Haiku project rewrote BeOS from the ground up.  Because they didn’t have access to all the minute details of the real BeOS internals, Haiku is not binary-compatible with BeOS.  But it IS source-compatible (existing code can be recompiled to run on Haiku), and has the same great look and feel that made BeOS such a pleasure to use.

It’s a valid question as to whether it’s worthwhile to try to revive orphaned operating systems, especially for desktop use.  There’s definitely a critical mass where the number of users vs. the amount of development feed each other; below that limit an OS is not likely to survive.  But there’s something to be said for a healthy ecosystem requiring diversity, in both the natural and computer worlds.  Without competition, the few remaining choices become stagnant and bloated.  Sometimes something revolutionary is needed to keep everyone on their toes.

Via OSNews.com

Boulder, Colorado

Filed under: Ephemera — mgrusin at 1:23 am on Thursday, September 10, 2009

Things I will miss when I move away from Boulder Colorado:

  • Driving on Cherryvale road, past the old house with the man on the porch who waves back at you, Peter’s Pumpkin Patch, and fields with the Flatirons in the background and deer and hawks and late summer wildflowers.
  • Being lucky enough to live in a light, airy treehouse on a creek and a bike path.
  • Being able to ride your bike most places in town without entering traffic.  And at certain times of the day, being able to get across town faster on a bike than in a car.
  • The pair of juvenile owls who learned to fly and hoot in the big cottonwoods just off our porch.
  • The Parkway Cafe, with the aquariums and great waitresses and the best Eggs Benedict in town.
  • My mom.
  • Birthday dinners at my aunt’s house.  My uncle’s perfectly-grilled steaks.  My cousin’s X-box.
  • Dropping by the Tropical Bird Farm and handing out nuts to everyone.
  • The human-scale housing developments (I can’t afford to live in them, but it’s encouraging to see them).
  • Skiing in whiteout powder that has blocked the arriving crowds and makes you laugh out loud.
  • J.B. Saunders, the best electronic parts store in the state within walking distance of my home.
  • Camping.
  • The International Scout.
  • The Boulder Outdoor Cinema.
  • All the science going on in the area, and the people who understand it.
  • People who see something they can do to make things better, then actually do it. (Community Cycles!)
  • People who think up completely oddball ways to have fun, then actually do it. (Ignite! Thursday night cruisers! Nerf wars!)
  • Having the resources of a major university just down the street.
  • Colorado Space Grant, birthplace of the next generation of explorers.
  • Growth limits, which have made this city beautiful (but expensive).
  • Summers when the bulk of the students are gone.

Things I won’t miss when I move away from Boulder Colorado:

  • The weekend the students return.
  • Growth limits, which have made this city expensive (but beautiful).
  • The abject wealth.  Friends saying “we’re selling our house, you should buy it!” when it’s listed at 1.2 million.
  • 28th street turning into Anymall America.
  • Pearl street turning into Upscale Anymall America.
  • RIP the beloved local places that have come and gone: Tom’s Tavern, The Last American Diner, The Hoffbrau, Daylight Donuts, Double Nickel, Philly Junction, Aria, etc.  (Most forced out by outrageous rent inflation and replaced by national chains that can operate at a loss.)
  • The completely insane building department, which lets everyone get away with murder except you.
  • Whole Foods, where people’s brains shut off as they enter the building.
  • RIP Annual Trash Day, where people used to put anything and everything out by the curb for city pickup and great scrounging.  (Permanently canceled due to liability and funding.)
  • RIP the old recycling center, which had better parking, tolerated scrounging, and had a more community feel. (Replaced by a “modern” recycling center which is poorly designed and doesn’t allow scrounging).
  • Complete respect for your constitutional rights, as long as you agree with us.
  • The moral high ground.  We’re all sinners.

It does explain some bruises

Filed under: Uncategorized — mgrusin at 2:48 pm on Monday, June 8, 2009

colbert090605

So how did Steven Colbert get this picture of me kiteboarding?  I don’t even remember that day (though it looks like I should have.)

History repeating

Filed under: Engineering,Space — mgrusin at 8:44 am on Tuesday, May 26, 2009

Administration changes are always times of upheaval at NASA.  A panel, led by former Lockheed-Martin CEO Norm Augustine, is currently (5/09) reviewing the state of NASA for president Obama.

Mr. Augustine led a very similar review in 1990, and their report is interesting to read in light of the last 19 years of NASA history.  (It appears that many, if not all of their recommendations were followed.)  I particularly liked the conclusion, which puts risk and blame in perspective:

“We believe that the legacy our generation should leave to the future is that we pioneered the exploration of space, and thereby made important discoveries that will prove of benefit to all mankind.  However, space activity is inherently difficult — involving advanced technology and taking place over great distances.  It demands reliance upon machines, often very complex machines, which are designed, tested and operated by mortals.  It involves rewards which may be intangible.

As we labor under such challenges, we should insist upon excellence.  We should strive for perfection.  We should demand the utmost of those to whom we entrust our space endeavor.  But we should be prepared for the occasional failure.  If we as a nation are to place a greater premium on letting nothing go wrong, on not making errors, and on ridiculing those who strive but occasionally fail, than we place upon seeking potentially great accomplishments, then we have no business in space.”

Report of the Advisory Committee On the Future of the U.S. Space Program  (December 1990)

An Innate Malice

Filed under: Uncategorized — mgrusin at 8:47 am on Sunday, October 26, 2008

A surprisingly accurate assessment, found hidden at the end of an article about new gem-quality artificial diamonds and allegedly from an old Encyclopedia Britannica:

“Of all domestic animals the cat is the most equivocal and suspicious. He is kept, not for any amiable qualities but purely with a view to banish rats, mice, and other noxious vermin from our houses, granaries, etc. Although cats, when young, are playful and gay, they possess at the same time an innate malice and perverse disposition, which increases as they grow up, and which education teaches them to conceal, but never to subdue. Constantly bent upon theft and rapine, though in a domestic state, they conceal all their designs; seize every opportunity of doing mischief, and then fly from punishment. “They easily take on the habits of society, but never its manners, for they have only the appearance of friendship and attachment. This disingenuity of character is betrayed by the obliquity of their movements and the ambiguity of their looks. In a word, the cat is totally destitute of friendship; he thinks and acts for himself alone. … Their sleep is light; and they often assume the appearance of sleeping when in reality they are meditating mischief.”

SFGate: Deuce of Diamonds, by Jon Carrol

Bill Nedell

Filed under: Uncategorized — mgrusin at 12:11 am on Wednesday, September 12, 2007

Went to a memorial dinner for a friend tonight, Bill Nedell. I’ve only met a handful of people like him, a certified genius who wanted to change the world and had the talent and energy to do it. He loved flying, and loved life, and dove headfirst into everything he did.

Bill worked for NASA back when they took the Aeronautics in their name seriously. He revolutionized air-traffic control software, and then left to pursue his dream of making an airplane for everyone, one that used advanced software to make it easy enough for anyone to fly. (His initial project towards this goal was to make a box that could be installed in an aircraft that had just one big red button on it. If you hit the button the box would immediately take over and safely land the plane at the nearest airport.) He and his wife Susan (just as brilliant, just as nice, not quite as intense) owned a small company in Boulder which did this and other work. I worked with them the summer of 2001 on a tiny unmanned airplane that they’d use to gather weather data, but was also unofficially testing out the core software that would eventually run Bill’s dream. The first time I met Bill, I dressed up nicely for a formal meeting. He came jogging in in a runner’s tank top and shorts, drenched with sweat. He sat down, toweled off, and started right in on the technical details he needed help on. I was very impressed by his philosophy of what was really important, and his refusal to do what wasn’t. I also remember Susan asking me how much money I wanted to make. I threw out a number. She said that’s not enough, and threw out a bigger number. This is the kind of people they were.

Immediately after 9/11 most of their weather contracts were canceled as government money moved to homeland defense. They could have bought into that but refused to on karmic principles. They hung on for a bit but eventually had to sell the company. Bill was devastated by having to let his friends go, and said he’d never work for the government again. They moved to Florida (I was very sorry to see them leave Boulder), and they eventually lived on a 50′ catamaran. Bill made the catamaran into a technological wonder, even putting in a home theater because Susan loved movies. This year they decided to move back onto dry land so their daughter could go to a real high school. Susan got an executive position at a biofuels company in Boston, and Bill was working on a new aviation startup in Kansas (he liked the boat but really missed flying). He was an avid runner and biker, and sadly a few months ago he was killed by a drunk driver while riding his bike.

Bill touched a lot of people’s lives.  The memorial dinner was about 20 people, most from the old company, many of which I had met but hadn’t seen since 2001.  I was honored to be invited since I had only known Bill a little while, but Susan once said he and I were simpatico. (Aside from consulting for the company, I also helped Bill with his annual high-tech Halloween show in his front yard. It was the year 2001, so the theme was 2001 A Space Odyssey.)  Heard lots of great stories tonight, and feel for Susan and her two children’s enormous loss.  I wish I had been able to spend more time with him, Bill’s the kind of person I want to be when I grow up.

Short story

Filed under: Uncategorized — mgrusin at 10:39 am on Saturday, April 14, 2007

Since we’re all storytellers at some level, I liked this list of Kurt Vonnegut’s (1922 – 2007) rules for short stories:

  1. Use the time of a total stranger in such a way that he or she will not feel the time was wasted.
  2. Give the reader at least one character he or she can root for.
  3. Every character should want something, even if it is only a glass of water.
  4. Every sentence must do one of two things – reveal character or advance the action.
  5. Start as close to the end as possible.
  6. Be a sadist. No matter how sweet and innocent your leading characters, make awful things happen to them — in order that the reader may see what they are made of.
  7. Write to please just one person. If you open a window and make love to the world, so to speak, your story will get pneumonia.
  8. Give your readers as much information as possible as soon as possible. To heck with suspense. Readers should have such complete understanding of what is going on, where and why, that they could finish the story themselves, should cockroaches eat the last few pages.

“Hello, babies. Welcome to Earth. It’s hot in the summer and cold in the winter. It’s round and wet and crowded. At the outside, babies, you’ve got about a hundred years here. There’s only one rule that I know of, babies — ‘God damn it, you’ve got to be kind.’ ”

Via

The Knights of Colbert

Filed under: Uncategorized — mgrusin at 12:36 pm on Friday, April 6, 2007

I’ve had the honor this year of helping out Fairview High School with their entry in the FIRST Robotics Challenge, the Colbertron.  Every year the FIRST organization (For Inspiration and Recognition of Science and Technology) announces a challenging new contest, and give teams of students just six weeks to build, program and test their entry. This year’s contest, “Rack and Roll”, required the robots to pick up inflated tubes and place them on a rack for points. Extra points could be gained by raising robots up to 12″ off the ground at the end of the match.

The six weeks went by quickly (with all the drama of any good engineering project), and on March 29th the regional competition began at the University of Denver’s Magnus Arena. The rookie Fairview team did an outstanding job in both robot construction and driving, even being called out by the announcers for helping push another robot up a ramp at the end of a match.

After a day and a half of competitions, the top 32 robots went on to elimination matches. Unfortunately, Fairview was ranked 33rd! The team was a little bummed out that they had come so close to continuing. But then a higher-ranked team chose Fairview to be on their team (due to their excellent driving skills and teamwork), so they got to continue.

In the elimination matches (which were best-of-three), Fairview lost the first match, won the second (and got to continue!), but lost the third 140 to zero (ouch). But it was a heck of a ride. This was Fairview’s first FIRST competition, and the students are already planning for next season.

Final standings (Fairview is team #2036)
Lots of pictures in the Flying Circuits Gallery

Next Page »