Building Scalable Web Sites: Building, Scaling, and Optimizing the Next Generation of Web Applications

Author: Cal Henderson
All Stack Overflow 13

Building Scalable Web Sites: Building, Scaling, and Optimizing the Next Generation of Web Applications


Review Date:


by anonymous   2017-10-22
There are [whole books]( on this subject. This is too broad for a question on Stack Overflow.
by anonymous   2017-08-20

Over the past year I've had to come up to speed on this question for a project my company's working on, and I've found these resources extremely helpful: Todd Hoff's; Scalable Internet Architectures, by Theo Schlossnagle; and Building Scalable Web Sites, by Cal Henderson. in particular will point you to many good presenations, tutorials, books, and papers, and is a great place to start. All of the advice is practical, and based on experience at sites like Flickr, Twitter, and Google.

BTW, scalability is not performance. A perfectly scalable system is one that has a fixed marginal cost to add additional users or capacity.

by anonymous   2017-08-20

Building Scalable Web sites has a nice section on handling email. It's written by a Flickr developer.

alt text

by Ryan Doherty   2017-08-20

I've worked on a few sites that get millions/hits/month backed by PHP & MySQL. Here are some basics:

  1. Cache, cache, cache. Caching is one of the simplest and most effective ways to reduce load on your webserver and database. Cache page content, queries, expensive computation, anything that is I/O bound. Memcache is dead simple and effective.
  2. Use multiple servers once you are maxed out. You can have multiple web servers and multiple database servers (with replication).
  3. Reduce overall # of request to your webservers. This entails caching JS, CSS and images using expires headers. You can also move your static content to a CDN, which will speed up your user's experience.
  4. Measure & benchmark. Run Nagios on your production machines and load test on your dev/qa server. You need to know when your server will catch on fire so you can prevent it.

I'd recommend reading Building Scalable Websites, it was written by one of the Flickr engineers and is a great reference.

Check out my blog post about scalability too, it has a lot of links to presentations about scaling with multiple languages and platforms:

by AlexC04   2017-08-19
I'd never heard of the C10K ( but I love scaling issues and this looks to be some fascinating reading. Thanks!

EDIT Interesting annecdote... Cal Henderson is the author of "Building Scalable Websites" and has worked in "the trenches" as the lead engineer of FLICKR and developer at B3TA.

Apparently his newest company is using node.js for their game engine.

Assuming he's planning on scaling to a reasonable size, that seems to be a pretty resounding endorsement that there's at least something going for it. I mean ... that guy's got a bit of experience in working at scale.

by prakash   2017-08-19
You might want to read Cal Henderson's book Building Scalable Web Sites:
by bootload   2017-08-19
great list. I'll take some time to go through this. To this I'd add the

- JOS software reading list [0]

- JOS management reading list [1]

There are gaps in these lists but pretty much cover the best for both programmers & business - types. But the lists needs updating as I note pg's 'Painters & Hackers' is not listed nor is Cal Hendersons 'Building Scalable Websites' [3].


[0] Joel On Software 'Book Review, reading list'

[1] Joel On Software, 'Management reading list'

[3] Cal Henderson, 'Building Scalable Web Sites, 978-0596102357':