MaxWiki Web Site

ScalingTwitter

Sun, 01 Nov 2009 00:06:46 Z

Preconceived Notions

Scaling on Rails is hard is a preconceived notion

Rails is not inherently difficult to scale

The Story

Twitter started as side-project of Odeo

South-by-southwest... not sure what this is

Traffic spike happened unexpectedly in March

Configuration

Just one database (master/slave - slave for reporting)

32 cores - sun machines

120 mongrels

16GB of memcache

MySQL on one 8 core box

Traffic

200-300 connections / seconds

800 spike per second

MySQL 2,400 queries/second

Solution

memcache!!!

cache-fu

extended cache stats

custom API caching

complex issues of caching of data because data is different for each users

Examples

denormalize ...didn't understand this

Starling... distributed queuing system - written from scratch (aren't there others?)

...something about how fast disk access is...

Advice

Talk to other people in the community

JRubyOnRails

Sun, 01 Nov 2009 00:06:32 Z

JRuby on Rails: A Taste of Honey (for the Enterprise)

Charles Nutter, Thomas Enebo

Session Link

Status

Most Rails apps will now work on JRuby
Most pure Ruby Gems work: Redcloth, etc
Rails commands work
Ruby DB MySQL adapter works (others close)
Extensions that work: Mongrel, OpenSSL, YAML, RMagickJr

Advantages of JRuby

Supports Java Unicode strings, Ruby and Java strings go back and forth safely
JRuby supports Ruby's Thread. Same as a system thread so can spin off threads easily. Can pool threads.
Access Java libraries
JDBC drivers with JNDI connection pooling
Supports more databases than Rails, but most still don't pass all tests
Can be faster than Ruby (after warmup)

Disadvantages of JRuby

Doesn't support SQLITE3?
PostgreSQL still fails some tests
Oracle, DB2 and MSSQL still not working
Native extensions (C extensions) not supported, but people can port them or call to equivalent Java library
RMagick not supported (but they are working on it)
Command line likely slower (because of JVM warmup)
Migrations don't work on all DBs

Building, Running, Deploying

Install JRuby from tar
Add to path
Install with jruby -S gem install rails -y
Install other stuff
Can run with native DB, to install JDBC modify environment.rb
Run as normal
However using Mongrel means that each one needs JVM too, so it takes a lot memory

Deploy to App Server

Install goldspide plugin
jruby -S rake war:standalone:create
Deploy to glassfish or tomcat
Can run GlassFish as a gem (Alpha)

RailsConfRealNumber

Sun, 01 Nov 2009 00:06:21 Z

response time

use 90%, not average (90% is where most people will be)

complicated automation

needs to be programmed

lost of detail on randomization of parallel request streams

use minimal Ruby code to create test drivers and reports

most tests... just use production monitoring tools

uses real-time character output to watch tests running

scaling

if you can bring down your site with one computer, you're already screwed

need a cluster

use UDP packets, can be broadcast, stateless

...tuned out...

user testing

use Google analytics

write scripts for each page

simulate production environment in test environment

RailsConfPlugins

Sun, 01 Nov 2009 00:06:09 Z

The Basics

lib - part of load path

install.rb/uninstall.rb - only evaluated once (script plugin)

Re-Use

Pros and cons

Speaker is author of engine plugins

DRY that can be shared

Add behavior to existing Ruby classes

Plugins are a bout Ruby

Example - add copyright function to all plugins

step one - use include module at the beginning of class

step two - use include in ActiveRecord::Base

step three - AR::Base.send( :include, Module ) [will be disallowed in Ruby 2]

include does not add class method; therefor add class method, then extend the class with the class methods module

include+extend -> use self.included to include class method module; call class_eval and do extend within it

this is what plugins are all about!!

Change behavior of Rails

Plugins are sanction method to change Rails

Example html_escape

a module is inserted above a class, and below the superclass - does not override class behaviour

redefine methods of a class - don't do this!

use alias_method

in Rails - use alias_method_chain (this isn't very simple, but very convenient)

Engine plugin adds to Dependencies class

Adding behavior of Rails

Example validates_ in AR

Like attr_accessor in Ruby classes - not a keyword, it's a method

You can call any class method in the class definition!!

Make method into module, then extend ActiveRecord::Base, and you can use it in class definition

Reference

lazyatom.com/plugins

<%= email 'james.adam@gmail.com' %>

Slingshot

Sun, 01 Nov 2009 00:05:56 Z

Slingshot

Connection agnostic Rails applications

Installs on Windows or Mac, updates automatically
Runs locally, syncs data back to server
Drag-and-drop
Update application code

Using Slingshot

Drop in a Rails app
Write ActiveRecord transport layer (2 to 30 lines of code, describing how to sync)

ScriptaculousInsideOut

Sun, 01 Nov 2009 00:05:43 Z

Prototype

protoypejs.org - excellent documentation

Enumerables - array extensions for iterators

Object.extend

Function.bind

DOM Utilities

Animation Engine

Time-based

http://script.aculo.us/dist/

Effect.Move Effect.Fade

Core vs. Combo effects.

Life Cycle = initialization - animation loop - finishing up

...speaker walks through code...

...does demos...

AngelsAndDaemons

Sun, 01 Nov 2009 00:05:31 Z

Angels & Daemons

"Supporting your rails application with custom daemons"

Session Description

Sample Project

http://thoughtbot.com/projects/ldap-ar-gateway

Daemonizing

def daemonize
Kernel.fork and Kernel.exit
Process.setsid
Kernel.fork and Kernel.exit
File.umask 0
Dir.chdir '/'
ObjectSpace.each_object(IO) {|io| io.close rescue nil}
STDIN.open '/dev/null'
STDOUT.open '/dev/nul', a
end

-- or --

require 'daemons'
Daemonis.daemonize
loop {
---
}

Interacting with Rails

require 'environment.rb'

if not defined? RAILS_ROOT
raise 'Error'
end

ActiveRecord:;Base.allow_concurrency = true #Otherwise get MySQL lost connection error

Alternatives

Rakes tasks
./script/runner

These are easy and good for single-shot or periodic tasks but overlapping execution can be a problem.

BackgroundDRB

Is good, early versions are buggy

CustomRailsHelpers

Sun, 01 Nov 2009 00:05:20 Z

Views are a Mess

Rails apps are pretty well designed, except for the Views.

Reasons - mix of languages, HTML, Ruby, JavaScript (every web framework has this problem).

Refactoring views involves writing helpers.

Helpers are not Well Understood

Making them work takes practise.

HTML Helpers

Mutliline string concatenation, or multiline string.

But, built-in helpers use content_tag( :tagname, :class => 'classname') - better than HTML, because now we're not mixing languages.

Another built-in helper: concat, capture.

Render partial from helper!

Reusability comes from reusing something

Rather than designing for reusability...

But for helpers, use html_options hash, and pass on to HTML helpers.

If generating multiple tags, html options should apply to outermost tag, and css selectors can be used to define inner css.

Helpers that generate JavaScript

Avoid putting javascript event handlers into HTML tags. Use observe_field prototype helper instead. Detangles javascript from HTML tag.

Learn JavaScript

Use objects, user Protoype (well documented now).

Form Builders

Introduced with Rails 1.2.

Demo

Lots of JavaScript !!!

WikiSpam

Sun, 01 Nov 2009 00:05:06 Z

Munge external links

Pissed off users. Stopped spammers for 2 weeks.

Content based rejection

Reject multiple sequential 2-byte characters on english language wiki. Stopped spammers for 2 weeks.

Problem was rejection notice - tells spammer that things don't work, so they try something else

Tarpit database

Identify messages as spam, but them in tarpit. Spammers get shown spam, we and Google see normal content.

Need to detect spammers. Based on IP addresses... (1) no reverse DNS lookup (90% of spam does not have set it up); (2) comment poster not logged in.

Worked well for some time.

Spammer profile

Poker, mortgage etc. posts.

Obvious spammers.

Hidden spam - hidden div, height of text = 1px, etc. Need to disallow styles in wiki.

...various amusing examples of spammers....

Spammers may put secret hashes on their posts to be able to check success through google.

Cross-site scripting (e.g. via an attachment on another site). May bypass javascript filter by spelling JaVaScript with mixed case.

Spammer myths

No-follow links will stop spammers? Does deny google-juice to spammer, but they will try anyway.

CAPTCHA? PWNtcha defeates this 80-100%, 40% for good once. Or, human defeat - people do this for 1$/hour.

Spam is created by SpamBots? No, there are a lot of humans doing this.

Blocking IP addresses? No more - botnets. Botnets rent out time. Botnets use multiple IP addresses - e.g. 7 hour attach, changing 300 pages, with 120 IP addresses (likely, infected PC's).

Ruse Wiki

Spam Resistant Wiki

Integrated tar pit - individual pages can be marked as spam.

RSS Feeds to detect changes to the site.

Approval of posts by users promotes or demotes author.

IP filtering - anonymous users can be marked as spam by ip.

Content filtering of URL's.

"No minor changes" - minor changes are not listed in history: very effective.

Continuous Arms Race !

RealWorldLessions

Sun, 01 Nov 2009 00:04:54 Z

Lessons From The Real World

Dirk Elmendorff, co-founder of Rackspace

twitter - rails app on rackspace the steps...

build team

Somebody who knows rails (an "anchor") is easier.

Need to understand where on rails learning curve team members are.

Buy the book!

Meta-programming can wait

Focus on conventions

brings as example rackspace app for kathrina relief

build app

get it to play with others

successful apps change over time

use tests to get back into the app ("interactive documentation")

TDD ... haven't gotten there yet. I

n the meantime, put stubs into test suites.

Have to support IE !

Selenium lets you use Ajax etc. in browser...

....at this point I tuned out....

LegacyRailsTesting

Sun, 01 Nov 2009 00:04:40 Z

Adding Tests to Legacy Rails Apps

Evan (Rabble) Henshaw-Plath, Hacker, Independent

Adding tests to legacy rails apps

Do legacy apps exist - rails < 3 years old!

"Who has time for tests?"

Testing == Health Food - good for you but not yummy

Testing == debugging!

Don't do it all at once - you won't pull it off.

Create test database

Use migrations

Scaffolded tests don't work... need to fix them first

Find a bug - write a test;

step by step Refactor a method - write a test

Test one thing at a time - short 3-5 line things (one assertion)

Running tests - rake or directly

GOOD IDEA: use development.log to build tests (has parameters, sql, etc.)

Test coverage

rake stats rcov - amazing tool, good graphs

other tools: heckel

autotest - great testing tool for immediate feedback

Fixtures, mocks, pull from database, all have pros and minuses

zentest..?

Heckle: Take Your Tests' Lunch Money

Kevin Clark, Powerset "

Mutation Tester"

Randomly permutates your code, then... runs your tests.

If tests don't fail, code is useless, or no tests are written.

heckle SomeController -> output is list of mutations that passed the tests.

2revolutions

Sun, 01 Nov 2009 00:04:25 Z

2 revolutions - Alan Francis

XP expert, particularly interested in culture and business impact 2004 - got a baby and a Mac, started with Ruby, working at ThoughtWorks there seems to be cross-over between XP people and Ruby people

Parallels between RoR and XP

more than software - "change the world", "programming should be fun" frameworks are extractions from existing projects/programs creation myth and cult of personality reaction to what went on before (php, j2ee) brash teenager personality, opinionated less process (XP), less code (RoR)

Rails as a Religion

Quote by Bill Captuo - "religion is the energy that builds around a good idea" Religion is the wrong word - "movement" is better (rf. Apple Dictionary) Movements arise as a reaction to something that was there before.... "RUP" in the case of "XP" Rails' enemy is J2EE. Parallel is Spring - more light-weight than J2EE. When will Rails become the enemy of something else.....?

Growing Up

When revolutions loose their brash teenager attitudes... they've either arrived or fizzled out.

Ba'h'ais

Bahais believe all religious leaders/prophets are manifestations of the same god. Analogy: Rails is just one other development in the mold of structured programming, objects, patterns, and XP. Will Rails remain stuck in teenage land? Ward Cunningham quote - long term success will not come from repeating our dogma's louder.

DHH response

David was in the room, and didn't quite buy it ... felt that Rails was actually a train of revolutions itself.

ScaleWithEC2

Sun, 01 Nov 2009 00:04:11 Z

Scale With Amazon EC2

RightScale: Web site that manages Amazon EC2

Common Use of EC2

Upload data to S3
Put request in SQS work queue
Fork new servers when needed
Put changed data back to S3
Put message in SQS when done (or POST to web)

RightScale Info

RightScale monitors work queue and adds or subtracts servers, puts log files in S3 (which is important because when the EC2 instance goes away, all logs are lost) and produces audit records
RightGrid and RightWorker manages the EC2 processes
RightScale runs 100% on EC2 itself.
Have to have a slave DB from day one, since the master can go away. RightScale also backs up DB every 10 minutes to S3.
Slowest part is parsing XML with reXML

EC2 Experience

Dynamic IP, need Dynamic DNS service
Very stable: 3 instances with greater than 170 days uptime, dozen instances with greater than 90 days uptime
Have seen spontaneous reboots
Amazon is very response
It's addictive, watch your credit card

CommunityFocussedApps

Sun, 01 Nov 2009 00:03:46 Z

Building Community-focused Apps with Rails Dan Benjamin

cork'd - developer & designer working together
500 users after a couple of hours, up to 20k users
acquired.

cork'd startup data center - mac mini in friend's computer closet

stay agile: resist big infrastructure

build right team

have a revenue stream - ads don't count

focus on simplicity

don't release public beta (do private beta)

resist cool features

web design - make sure all the action is "above the fold"

avoid user-entered data (misspelling, etc)

real-time collaboration (not real-space - the two founders met each
other during acquisition discussions)

re-use plugins
- attachment_fu
- acts_as_taggable
- ym4r_geocode

Google makes a big difference in getting noticed
- wine community didn't find out about it through blogs etc
- cork'd is high on google
- make sure metatags and description are appropriate

sign up should be obvious and minimal - ask for everything but ask for nothing!

keep non-members from doing the cool things to force signups

keep the coming back for more

make frequent improvements, and avoid "big releases"

respond positively to your members - be nice!!! always! apologize first

share your api

CleanCode

Sun, 01 Nov 2009 00:03:27 Z

Clean Code

Don't do big restructure, improve incrementally.
- One of the berst ways to ruin a probem is to make massive changes in the name of improvement
- Keep the system running at all times
- Make a tiny change, then run the tests
Always check-in code a little better than it was before.
Testing: rspec, test_unit
Open/Closed Principal: open for extension but closed for modification - or - create a barrier between those things that change and those things that don't.
Professionals write tests first
Professionals cleanup code
Professionals know that the only way to go fast -- is to go well

Is it worth it to clean up code?

Bad code gets harder and harder to clean as time goes by.
You should clean code as soon as it gets messy.
This takes less time than debugging code later
Nothing has a more profound and long-term degrading effect than bad code:
- Bad schedules can be redone
- Bad requirements can be redefined
- Bad team dynamcis can be resolved

Analogy: What is the quickets way to be done with dinner?

After dinnner, get up and walk away and don't clear or do the dishes
Or, keep the dishes clean all the time
Dinner will take longer, but the whole process of preparing dinner will be quicker