December 03, 2018

All Code Works By Accident

Tweet storm from CTO of VividCortex, Baron Sc​hwartz:
Something that isn’t a “Law,” but has held true through my entire career: when you instrument and measure something, you always learn surprising things.

--The code is doing WHAT?
--That query usually runs in microseconds but sometimes hours?
--We have a server nobody knew about?

This isn’t just database-monitoring-related. It’s true of any system, like… application code; off the shelf software; company finances; my personal health and vital signs—whatever.

Example: what happens to my bodyweight in the 2-3 days after a night of bad sleep? SURPRISE!!!

An example from early in my career as a developer. I discovered that the MySQL protocol has a notion of “warnings” and the number of warnings generated by a query is exposed in the Perl driver. I was curious, so I put a wrapper around our DB access library to check for warnings.

If you’re a software developer, this might not surprise you, but I’m sure some of you wouldn’t believe the broken things I found. I don’t just mean “broken like a tire with low pressure, but the car still drives.” I mean, like the car is in the ditch upside down and nobody knew.

In this specific example, some of the fundamental underpinnings of what the business was supposed to be doing, weren’t working. At all.

The surprises come from realizing your mental model isn’t reality. You think you have awareness of this, but it’s order-of-magnitude wrong. (Meta: your mental model of your mental models’ brokenness, is itself broken.)

You never understand how badly your model of the world and the world itself, have diverged until you measure. Then you’re aghast. But it’s just business as usual. Pick yourself up, go on.

One of the things I learned from that incident was that the stuff I thought was the fundamental, non-negotiable underpinnings of the business was just a nice-to-have.

I didn’t find just ONE thing broken, though. I found dozens, immediately. And many more in subsequent years.

Simple things reveal these totally invisible ways in which your mental models are broken. Quick: pick a server, SSH in, look at any log. Any one. Odds are you’ll find something ghastly.

Pick your system and your tools, do it any way you like. Dig into the bank account statements, look at HR data, whatever. (We recently got a gas+water bill from the city for an astronomical amount.) There’s always broken, weird stuff because NOBODY KNOWS HOW ANYTHING WORKS.

And a lot of what’s broken doesn’t even need to be fixed, despite the engineer’s impulse to make things perfect. The broken world turns on its axis every 24h anyway.
In reply someone said, "All code works by accident." Ha.

No comments: