Scaling a Rails Application from the Bottom Up

Jason Hoffman: CTO, Joyent

• Rails Conference Page Description

RoR Apps

• LA Times
• Telemundo
• Katrina apps

Scalability Issues

• How many large scale apps are there in the world? Not many.
• How many are monolithic? Not many.
• The total server environment will consist of more than 25 components, including DNS, database, DHCP, auditing, login, etc. Rails usually encompasses only 2 of these components.
• Big bottlenech for Rails apps has been load balancers

Power

Power is usually the limiting factor

• $25 per amp for 208V power, $500 per month for 20amp drop (CA)
• 20 amps x 208v = 4160 watts
• 80 safely usable = 3328 watts
• Dell 1850 takes 440watts at $0.16/kwh = $617/1 year = $1850/3 years
• 100 kilowatts can power 250 - 400 servers
• Hard to get a utility to drop more than 100 kilowatts to one facility

Network

• 10Gbps is the most you can get
• 1Gbps is more realistic
• 100Mbps gives about 10 million unique hits per day
• 100Mbps is about $5k to $8k per month for something "good"
• Apache, Lighttpd can do 1k-15k static or proxy requests/second
• Hardware load balancer like big-IP can do 20k - 100k fine

10% Rule

• 10% of revenue should be spent on servers, storage and bandwidth (not including people)
• So code performance of application to meet this target

Costs Breakpoints

• Including people costs, it is generally cheaper once you start spending $20k - $30k/month to do it in house

Colocation

• Use colocation.com

Rack Costs

• $500-750/mo for the rack
• $500-1000/mo for power
• $1000/mo for bandwith
• $2000/mo for leased hardware
• Total: $6500/mo for 20 systems in a rack on a lease

Operation Systems

• Likes Solaris
• Likes ZFS

 Running a Rails Process

• New event driven Mongrel
• JRuby in Glassfish (Create Rails WAR)
• Sequia does DB clustering

Running Mongrel

• 16GB RAM (more than enough), 4 AMD CPU machines
• 4 virtual "containers" on them
• Each container, 10 mongrels (each Mongrel takes 5% to 10% of CPU, so 10 per CPU)
• Each does about 50 requests/second
• Joyent runs on 600 Mongrels, Twitter runs on 200-300.

Joyent Example

• Front end load balancer: F5 Big-IP costs $15k that scales across couple thousand mongrels and gives Layer 7 packet inspection
• Apache mod_proxy_balancer only can handle about 3 or 4 mongrels, so it isn't scalable. This is because the mod_proxy_balancer will max out first (say at 150 requests/sec)
• SW load balancers: HA-Proxy, ????
• Web server + LB Proxy: Nginx
• Alternatives to RDMS: Memcache, LDAP, J-EAI, file system , MySQL INNODB tables
• J-EAI: XMBB Jabber message queue
• LDAP: Fast, memory efficient, uses Sun's LDAP
• File System: No more than 10k, ideally 4k, files/subs in a single directory. Good hash: 16 top level and 256 subs, gives 2096 buckets. Or do 256x256.
• Remember DNS: Federation by DNS is easy to split your customers into pools. Uses PowerDNS

Summary

• Use DNS
• Use great load balancers
• Event-driven mongrels
• In addition to RDMS use: LDAP, J_EAI, file system
• Rails process should only be doing Rails
• Static assets should be coming from static servers
• Go layer7: rails process should be doing one controller
• Federate and separate as much as you can