Hi kid! You don’t know me, but I’m one of those “Internet expert” kind of guys they interview when something bad happens. I won’t bore you with the details, but let’s just say that I make big computer systems for a living and I know how they work.

I’m not your mom or dad, or a teacher, or your church leader, or your coach, or a cop. I don’t know you, either, and honestly, I don’t care about you personally so much that I’d want to scare you or exaggerate things or otherwise lie to you.

I love the Internet, and I’m pretty proud of this amazing place that my friends and I have built. There’s a lot of great stuff on it, and I truly think it’s one of the best things that people from all over the world have ever come together to create. I think we did a pretty good job, for the most part.

The thing is, it’s also easy for bad things to happen on the Internet. I’m not talking about stuff like child predators or terrorists or hackers trying to steal your iTunes credits. Yeah, those things exist. But yes, the news exaggerates them a lot to scare you and to make you want to watch more of the news (see how that works?). I don’t want to do that. The truth is, you’re way more likely to have trouble from the wrong people seeing things you’ve written or pictures you’ve sent than you are from any Stranger Danger.

Social networks are awesome. I use the same ones you do - and some that you don’t even know about yet - to talk to my friends the same way you talk to yours. I think they’re great. However. Every single one of them tells the same lie: that you can click a “keep all my stuff private!” checkbox and all your stuff will stay private. If you only hear one thing I’ve said today, let it be this:

They. Are. Lying.

Oh, they don’t mean to. The people who made Facebook and Twitter and WhatsApp are smart people who try awfully hard to do a good job. However, making giant computer systems like that is super difficult and it only takes one itty bitty mistake in a giant tangle of a million moving pieces for it all to break. When you see a setting like “only share this with my friends”, what it really means is “should we try to keep this private and hope that we did everything 100% correct and didn’t screw something up somewhere?”

But none of that really matters anyway, not when your “friend” can take a screenshot of your messages on their phone. Say you just told your BFF about your crush. You checked the little “don’t share this!” box and the computer guys did their job and the privacy stuff works just like it’s supposed to. And because they think it’s funny, your BFF clicks the buttons to take a screenshot so they can tease you about it later. Guess what: now there’s another copy of your message, but this one doesn’t have that little “don’t share this!” box next to it.

The rule in computing is “if you can read it, you can copy it”. There are some smart people who waste their time trying to break that rule so that you can’t make copies of movies or music or video games, but, well… did you pay for every one of those songs on your iPhone? Yeah, thought so. That’s what I mean, though. If it’s easy to copy songs and movies and video games, how hard is it to copy a screenshot of your text message conversation or - ahem - one of “those” pictures?

So after all that, here’s what I tell my own kids:

  • If it’s on the Internet, everyone can see it. No exceptions. Everyone. I’m saying this as one of the guys who helped make the Internet. Trust me on this one.
  • Do not put stuff on the Internet if you don’t want everyone to see it.
  • Before you put stuff on the Internet, imagine who the worst possible person to see it in the entire world would be. What if your teacher read it? What if your mom or dad saw the picture? If the answer is “oh, wow, that would really suck”, then don’t put it on the Internet.
  • If it’s on a computer or cell phone or iPad, it’s on the Internet. You wouldn’t believe how many ways there are for a note or photo to get automatically backed up or copied around to some computer somewhere that you don’t have any control over.
  • This one is mainly (but not only) for girls: odds are, some day he won’t be your boyfriend anymore. Do you want a pissed off ex-boyfriend to have “those” pictures to share at school?
  • Don’t. Reuse. Your passwords. I read the news stories you don’t, the ones about how some instant messaging company had their password database hacked into and stolen. That means someone has a whole list like “CoolKid23 uses the password MyDogStinks”, and then they go around to other websites and try to log in with those same usernames and passwords. Make up different passwords for each website and chat program you use. Write them down on a piece of paper and leave it at home if you have to, or use a “password manager” program to do it for you. Yes, I know this sounds paranoid and geeky. But I’m telling you this with my hand on my heart: this is important. It’s something you have to do. It’s a pain in the butt, but that’s just the way it is.

Have I scared you? If I did, I’m sorry. There are people who want to scare you because that’s how they think they can get you to listen. I’m not one of them. But I do want to tell you the truth about how things on the Internet actually work. This is important stuff, and it’s only going to get more important as we share more of our lives with our friends on the Internet.

I’m still proud of this great big network we’ve built, and I use it every day of my life. The Internet has a lot of exciting things to offer. Use them and have fun! Just be smart, and be a little suspicious before you send a message or a picture. Don’t share the things you don’t want the whole world to see.

OK? OK. Glad we had this talk. Now do your homework.

This is my letter to the FCC on September 12, 2014 regarding the upcoming net neutrality decision making process:

I am a Comcast customer, and I am paying them for a 100 million bit per second connection. Comcast has a monthly data cap of 300 billion bytes (or about 3 trillion bits) per month. At the speeds I’m paying full price for, I can use up my entire monthly data allotment in about 8 hours.

More simply, my monthly Comcast payment entitles me to use my Internet connection at full speed for one third of one day per month.

Esteemed colleagues, I find it disingenuous that Comcast and their peers claim that they need to charge more to carry the services I want to use, all while constricting my paid usage to one ninetieth of my connection’s capacity and raking in record profits. There is simply no fiscal credibility to their claims and I urge you to look upon them with due skepticism.

The FCC has received millions of letters supporting net neutrality rules against Internet slow lanes. Most of these have been form letters written by various citizen-friendly organizations and submitted by casual site visitors. Most of the individually written letters are various restatements of why net neutrality is important. All of those are good, but it’s also important to remind readers of these letters that anti-free-market groups like NCTA and its constituents have no legitimate counterarguments. They claim to need Internet slow and fast lanes to make money, but the industry makes huge amounts of money while delivering some of the worst Internet service in the developed world.

Comcast earned 3.3 billion dollars in net income in the second quarter of 2014, all while allowing customers to use only one ninetieth of the utility they’ve paid for. The only valid explanation for their strident opposition to net neutrality is sheer greed.

No good article about the Bay Area misses jokes at hipsters in their hoodies, whether biking through The Mission or chairing board meetings in The Valley. It’s an easy laugh and a nodding wink to your audience to assure them that you’re on their side, that you know how silly grown adults look in their kiddie jackets. But consider:

  • San Francisco is walkable, and people take advantage of it. My stroll from the bus terminal to my office is about a mile, and the sidewalks teem the whole way.
  • Layering is crucial. The weather changes rapidly from warm to cold, gray and windy and then back. Clothes have to adapt from comfortably light to guarding from the elements quickly and easily.
  • The city is humid. A light sweat from walking stays on you, and nylon clothes become waterlogged and sticky within blocks.
  • The city is windy. Synthetic fleece jackets are great, until a breeze picks up and cuts through the coarse cloth. I’ve never been so cold as when I was near the shore in a thick fleece.
  • Sun gives way to drizzle in minutes. Between the rain and the wind, it’s always smart to pack a hat.

Distilled, that means the ideal outerwear is of natural fiber to let sweat through while keeping wind out. It has a zipper and can go from breezy to windproof. It has a hat.

You know: a hoodie.

The humble jacket is a perfect fit for the local climate, where the weather is rarely great but is never bad. They’re warm in the winter and protective in the summer. You can buy one for a few dollars from street vendors, or spend more for a handmade work of urban art.

Making fun of a San Franciscan for wearing a hoodie is like teasing a Minnesotan for wearing a coat and scarf. Yes, we love our hooded jackets. Why shouldn’t we?

Originally published on the Crittercism Engineering Blog and reprinted with permission.

by Kirk Strauser on April 8, 2014

CAP theorem hates you and wants you to be unhappy

Some guy who isn’t fun at parties came up with the CAP theorem, which basically says it’s impossible to be consistent and available at the same time. In short, things will break and clients will lose access to a storage backend, or units in a storage cluster will lose the ability to talk to their peers. Maybe those servers are crashed. Even worse, maybe they’re up and running but a network outage means they can’t reach each other, and each part is still accepting writes from clients. Our enemy, CAP theorem, says we have to choose between:

  • Keeping our data consistent, at the price of not being able to make progress when parts of the database cluster are unavailable.
  • Keeping the database cluster available, at the price of some parts of it being out of sync with others and resolving any conflicts later.

Consistency brings pain

In any case, we have to decide what happens when we want to write to a record. Let’s assume for demonstration sake that a record is a bag of opaque data that the backing store doesn’t really understand; imagine a JSON blob, or a bit of XML, or whatever other datatype your favorite database doesn’t natively support.

Let’s also assume we have a consistent database. Either it’s a single server that’s running or not running, or it’s a cluster that only accepts requests if all nodes are online and synchronized. In short, we can always trust our database to Do The Right Thing.

Here’s how consistent workflows evolve from the most innocent of intentions.

First attempt: blind writes

We want to write a record, so we write it! Easy-peasy.

  1. Write out an entire record
  2. Profit

Second attempt: read-update-write

Ouch! Two requests want to update the same record. Both of them write out its entire contents, but only the last one wins.

  1. Request A writes out {"foo": "bar"}
  2. Request B writes out {"baz": "qux"}
  3. Request A cries salty tears
  4. Request B gloats and gets punched

That’s not good. The answer, then, is surely to read what’s there, update it, and write the results back out:

  1. Request A fetches the record with its initial value of {}
  2. Request A updates the record to {"foo": "bar"}
  3. Request A writes the record with the its new value
  4. Request B fetches the record with A’s value of {"foo": "bar"}
  5. Request B updates the record to {"foo": "bar", "baz": "qux"}
  6. Request B writes the record with the combined value

They shake hands and go home. And at 2AM, the Ops pager goes off because every write requires a read to get the pre-existing value. But let’s pretend IO is free and infinite. This algorithm is chock-full of race conditions. At our scale, here’s what’s going to happen many times per second:

  1. Request A fetches the record with its initial value of {}
  2. Request B fetches the record with its initial value of {}
  3. Request A updates the record to {"foo": "bar"}
  4. Request B updates the record to {"baz": "qux"}
  5. Request A writes the record with only its new value
  6. Request B writes the record with only its new value, overwriting A’s

And now we’re right back where we started.

Third attempt: locks everywhere!

Looks like we’ll need to lock each record before updating it so that only one request can mutate it at a time. We care about uptime so we have a highly available distributed locking system (ZooKeeper, Redis, a highly motivated Amazon Mechanical Turk, etc.). Now our workflow looks like:

  1. Request A acquires a lock on the record
  2. Request B attempts to acquire the same lock, but fails
  3. Request A fetches the record with its initial value of {}
  4. Request A updates the record to {"foo": "bar"}
  5. Request A writes the record with only its new value
  6. Request A releases the lock
  7. Request B attempts to acquire the same lock, and succeeds this time
  8. Request B fetches the record with A’s value of {"foo": "bar"}
  9. Request B updates the record to {"foo": "bar", "baz": "qux"}
  10. Request B writes the record with the combined value
  11. Request B releases the lock

That actually worked! Of course, it took two reads and two writes of the database and fives calls to the lock manager and Ops wants to set fire to your cubicle because their call duty phone won’t stop buzzing.

But let’s assume that we have a free and infinite lock manager. What happens if Request A never completes the transaction and releases its lock, maybe because of network problems, or the node it was on died, or it couldn’t write to the database, or [insert your own pet scenario here]. Now we can’t make progress on B’s request until the lock expires, or until we break the lock and potentially overwrite A’s updates. For all our efforts, we’re still not in a much better place than we started.

Side note about the locking manager

Any distributed lock manager has to solve all of the same problems we’re listing here. Even if we use this pattern, we haven’t made the root problem go away: we’ve just shifted it to another piece of software. The CAP theorem means that a lock manager optimized for consistency has to sacrifice availability, so in the event of a network outage or failed locking manager node we still can’t get any work done.

But eventual consistency brings joy and unicorns!

Consistency is critical for many reasons. I’d much rather queries to my bank be slow or unavailable than incorrect! There are times and places when we want the properties that consistency buys, regardless of its price.

But there aren’t many of them at our scale.

What we want is eventual consistency, or a promise that the database will make its best effort to make its records return their current values. This pattern extends to both the database we use to store our records, to the way in which we generate and process those records.

Solution: journaled updates

Instead of treating our record like an atomic chunk of data, we’ll treat it like a list of atomic chunks of data representing updates that clients want to make.

  1. Request A appends its update of {"foo": "bar"} to whatever happens to already be in the record
  2. Request B appends its update of {"baz": "qux"} to the record
  3. Much later, if ever, Request C fetches all the values from the record and combines them into a final data structure. In pseudocode:
def materialize(query):
    result = dict()
    for key, value in query.records():
        result[key] = value
    return result

In our example, that would fetch the list of updates [{"foo": "bar"}, {"baz": "qux"}] and combine them into a single record like {"foo": "bar", "baz": "qux"}. This is a very fast operation for any sane amount of updates.

Our primary usage pattern is “write many, read rarely”. Most of the events recorded by our system will never be viewed individually, but might be used to calculate trends. This solution allows us to trade a small (but fast and easy) bit of post-processing for a huge bit of not having to worry about requests clobbering each other, locking semantics, or mixing reads with writes.

Ordering is still hard

This solution isn’t magic, though. It doesn’t define how to reconcile conflicts between updates, and we still have to make those decisions.

Time and time again

The simplest method is to store each record’s timestamp and replay them in order. However, it’s impossible to guarantee ordering between timestamps generated across more than one host. Two database servers might be off from each other by a few seconds, and NTP only seems like a solution until you’ve actually tried to count on it. The one situation where this is feasible is when requests to update a given record are all generated by the same client. In this case, we can use the client-generated timestamp to reasonably represent the correct ordering of updates.

Understand our data

Another approach is to make smart decisions about the data we’re storing. Suppose a certain key, foo, may only ever increase. Given a set of updates like [{"foo": 23, "foo": 42, "foo": 17}], the correct resolution would be {"foo": 42}. This requires an understanding of the data, though, and isn’t something we’d want to pursue for customer-generated inputs.


Math says you can’t have both consistency and availability. At our scale, availability wins the argument and eventual consistency carries the day.

I’m a sucker for the idea of ritual. When I learn about a traditional, labor-intensive practice like shining shoes, oiling boots, or a complicated car washing regimen, I’m always drawn to try it myself. I imagine having the same meditative experience as the person convincing me to try their routine: feeling a connection to my ancestors, appreciating the finer things, tasting the rewards of patience, and such. So when I read an article about wet shaving a year ago, I could hardly wait to get started.

Razor, brush & stand

In practice, though, I hate ritual. I’ll pay a few bucks to have someone else shine my shoes. San Francisco Bay Area climate isn’t very hard on boots, whether I’ve diligently oiled them or not. Automatic car washes are popular for a reason. Basically, I run out of patience for things that take too long just for the sake of taking too long.

One recent morning, I found myself wondering if I actually enjoyed wet shaving or if I’d be better off going back to a can of foam and an 8-bladed disposable razor. Millions of guys do it the new way, after all - should I rejoin them?

No. For me, wet shaving is clearly better for two specific reasons:

  • It’s way cheaper. It’s like the laser printer business model of charging more up front but offering dirt cheap supplies. After the initial purchase, consumables cost less than $10 a year.
  • I haven’t had a single ingrown hair since I started. Modern razors always leave me with a few bumps on my neck and cheekbones, but that problem has completely disappeared.

Yes, it takes longer than I’d like and still carries more trappings of ritual than I care to think about. Still, it’s a little luxury that’s measurably nicer and I don’t think I’ll give it up.

I use and happily recommend:


Garrick Dee wrote another nice introduction to the subject at the Grooming Essentials blog.