[Discuss] AWS Linux server scaling question

[drew.vanzandt at gmail.com (Drew Van Zandt)]

Up front: While I'm solid on Linux for personal use, and use it daily, and
have been sysadmin for a tiny company with just one internal server, I've
never been what I consider a "real" sysadmin.

I'm working with an all-volunteer group trying to figure out what our
cheapest AWS setup to handle things for ~3500 users is.  Right now the
webserver, which is basically just a Wordpress install, is running on a
Micro instance.  When we released a half dozen documents that everyone
wanted right away last week, of course it fell over a couple of times,
(slow, nonresponsive, reboot needed once, etc.)  I joined the tech team
pretty much the day before this happened.

memcached was added sometime during the explosion.

We're regrouping for next time this happens, and given the limited admin
experience of everyone on the team, there are some disagreements on the
best approach.  I'll present a few of the options under discussion, if
anyone cares to comment I'd be happy to hear discussion.

Option A:
* Upgrade to a Small instance (1.7GB instead of 610MB)
* Tune apache prefork config so the max number of clients does not cause
much swapping at full load
* Tune Apache: FollowSymlinks, AllowOverrides None (symlinks is security
risk, accepting that)
* Add indexes for any SQL query WHERE clauses that are common to the setup
* Tune PHP and MySQL a bit for the limited available memory

Option B:
* Use several micro instances with load balancing
* Apache/PHP/SQL tuning identically per instance

Option C:
* Two Small instances
** One runs Apache/wordpress
** One runs MySQL and lighttpd for static files (lighttpd may be
unnecessary complexity)
** Each optimized as per A

Other items considered:
Amazon S3 for static files
AWS DB-specific instance for SQL
Medium instance
Varnish
113 other caching solutions

Based on making embedded Linux handle things with tiny, tiny memory models,
I'm inclined to think that A would suffice for the load I expect folks to
throw at it - the only reason even the Micro instance couldn't mostly
handle it is that it had stock settings on everything, no tuning at all.  A
Small instance is more than enough.

Based on having survive office politics for years, I'm inclined to think
that C is perfect, we're likely to get budget since Things Just Happened,
and it should be able to handle loads far beyond what I ever expect our
users to throw at it.  (Though we do get surges of 500+ users at high
activity for a few days just before conventions...I would expect C with
tuning to handle even that pretty gracefully.)

In either A or C I'm inclined to ask for an identical config but with Micro
instances for release prestaging/testing, I should be able to stress those
with just a script on my server, and test load handling - the Small version
should be strictly (and vastly) better than the prestaged test.

Are we thinking along the right lines?

*
Drew Van Zandt
Cam # US2010035593 (M:Liam Hopkins R: Bastian Rotgeld)
Domain Coordinator, MA-003-D.  Masquerade aVST
*