(Yes, it’s badly dated Ballmer joke)
When we installed our new setup at our new data center, the sales team pushed redundant network equipment as a major selling point (our previous environment had only a single firewall and load balancer, but granted, the new team happened to be Cisco resellers.) I didn’t give it a lot of thought, since we’d never had a problem in three years outside of a 20 minute planned outage to do some maintenance, but the pricing didn’t make it overly restrictive, so we went for it.
Was it overkill? Based on 37Signals’ outage on Friday, I’m a bit happier with our decision:
Here’s the thing: the web world is largely an industry of self-taught developers and designers who have no concept of how to read a network diagram. “LAMP” doesn’t really go beyond the application, and even then, I’m seeing more advice on how to make an app scalable than I am about reliability, especially from a hardware perspective. Unless they’re lucky enough to have someone on the team with relevant experience (I know we can’t afford a dedicated network engineer, but we at least have our hosting managed by a team of geniuses at another company, and it’s well worth the cost), the average startup is at the mercy of whoever’s wiring the system up.
Our previous setup was put in place just before I got involved. It had a single firewall, a single load balancer, and all security was based on static IP rules in the firewall. There was no network-level redundancy, I have no idea if the servers’ NICs were teamed, we had no VPN, etc., and we were at one of the top hosts in the country. It’s quite possible that the options came up and decisions were made on a cost basis, but not once did an account manager try to sell me any upgrades or changes. We had an engineer assigned to us, but unless I found out about a technology and asked about it, I wouldn’t get any advice.
Our new digs are a lot more solid (though we’re still limited to a single physical location), and after Friday’s Basecamp outage I feel a lot better about our extra insurance. To be clear, I feel for the 37Signals gang, and I’ve obviously spent a lot of time being a single point of failure away from being in their shoes. My point is that we as an industry need to get better at sharing the nuts and bolts and wiring best practices in addition to things like software design patterns and security concepts.
Are there any sites out there today that can give someone a good grounding without requiring some kind of networking certification to get past the first page? HowtoForge seems like a start.
Leave a Reply