BlogmyQuery – BMQ

E-commerce runs on secrets. Those secrets let you update your blog, shop at Amazon and share code on GitHub. Computer security is all about keeping your secrets known only to you and the people you choose to share them with.

We’ve been sharing secrets for centuries, but the Internet runs on a special kind of secret sharing called public-key cryptography. Most secret messages depend on a shared secret—a key or password that everyone agrees on ahead of time. Public-key cryptography shares secret messages without a shared secret key and makes technologies like SSL possible.

Cryptography is a scary word: it conjures thoughts of complex equations and floating-point arithmetic. Cryptography does have a lot of math, but it’s more about keeping and sharing secrets.

A Simple Secret

Telling my best friends a secret is easy: I find a private place and whisper it in their ears. As long as no one is listening in, I’m totally secure. But the Internet is full of eavesdroppers, so we need codes.

We’ve all been inventing codes since we were children. I created this simple number code (actually a cipher) when I was 5:

a=1, b=2, c=3, d=4, e=5…

It fooled my friends, but not my parents. Simple substitution ciphers are based on a lack of knowledge. If you know how they work, then you can decode every message. The experts call this “security through obscurity.� Letter and number substitutions don’t work on the Internet, because anyone can look them up on Wikipedia. For computer security, we need codes that are still secure even if the bad guys, or your parents, know how they work.

The most secure code is still easy to use: a “one-time pad.� One-time pads have been used for centuries, so they don’t even need computers. They played a big part in World War II, when each pad of paper with the key numbers was used only once.

Let’s say I wanted to send you this secret message:

I love secrets

First, I’d turn the message into numbers using my simple cipher from when I was 5. (I’ve heard rumors that other people had this idea first, but I don’t believe it.)

Then I’d mash my keyboard to generate a random key string for my one-time pad.

Now I can add the two strings together. If my number is greater than 26, I would just wrap it around to the beginning. So, i(9) + e(5) = n(14), and o(15) + t(20) = i(35 - 16 = 9). The result is an encrypted string:

Decrypting the string to get the secret back is easy. We just subtract the one-time pad: n(14) - e(5) = i(9). Follow that pattern through the entire message, and you can securely share a secret. You don’t even need a computer: just work it out with a pen and paper.

This function is called a symmetric-key algorithm, or a shared-key algorithm, since it uses the same key to encrypt and decrypt the message. Modern systems can safely use the pad more than once, but the basic idea is the same.

The one-time pad is totally secure because the bad guys don’t know how we got the encoded letter. The n could be i + e, j + d or any other combination. We can use our shared secret (the one-time pad) once to share another secret.

But there’s a fatal flaw. We need to share the one-time pad ahead of time before we can start sharing secrets. That’s a chicken-and-egg problem because we can’t share the pad without worrying that someone will snoop. If the bad guys get the one-time pad, then they would be able to read everything.

One-time pads help me share secrets with my best friends, but I can’t use them with strangers such as Amazon or Facebook. I need a way to share something publicly that doesn’t compromise my one-time pad. I need a public key.

The Public-Key Backpack

Public-key encryption focuses on a single problem: how do I prove that I know something without saying what it is? An easy concept to help us understand this is a backpack full of weights.

I want to prove that I know which weights are in my pack, but I don’t want to tell you what they are. Instead of showing you all of the weights separately, I’ll just tell you the total. Now you can weigh the pack and see if I’m right without ever opening it.

If the pack weighs 20 kilos, then you wouldn’t know if it has one 20-kilo weight, twenty 1-kilo weights or something in between. With a large number, you can be pretty confident that I know what’s in the pack if I know the total; you don’t have to see inside. The weight of the backpack is the public part, and the individual weights are the private part.

This basic backpack enables us to share a secret without really sharing it. If we each have a backpack, then we can both share secrets.

The backpack works well enough for smaller numbers, but it isn’t useful in the real world. Backpack algorithms were a mere curiosity for decades. Then RSA changed everything.

RSA

RSA was the first public-key encryption system that worked in the real world. Invented more than 30 years ago, it coincided with the introduction of the more powerful computers that were needed to run the big numbers. RSA is still the most popular public-key encryption system in the world.

The basic premise of RSA is that factoring large numbers is difficult. Let’s choose two prime numbers: 61 and 53. I’m using the numbers from Wikipedia’s article on “RSA� in case you want more details.

Multiply these two numbers and you get 3233:

61 × 53 = 3233

The security of RSA comes from the difficulty of getting back to 61 and 53 if you only know 3233. There’s no good way to get the factors of 3233 (i.e. the numbers that multiply to make the result) without just looking for all of them. To think of this another way, the weight of our backpack is 3233 kilos, and inside are 61 weights weighing 53 kilos each. If you make the resulting number large enough, then finding the numbers that produced it would be very difficult.

Public And Private Keys

Unlike the one-time pad, RSA uses the public key to encrypt information and the private key to decrypt it. This works because of the special relationship between the public and private keys when they were generated, which allows you to encrypt with one and decrypt with the other.

You can share the public key with anyone and never reveal the private key. If you want to send me a secret message, just ask for my public key and use it to encrypt the message. You can then send it to anyone you want, and you’ll know that I’m the only one who can decrypt the message and read it.

I could send you a message in the same way. I would ask for your public key, encrypt the message using it and then send it to you to decrypt. The popular program Pretty Good Privacy (PGP) worked like that. We’re secure as long as we both keep our private keys private.

Exchanging keys is made even easier by special key servers that allow you to search for people and find their public keys.

Public-key encryption also works in reverse to provide digital signatures. Let’s say I want to write a message and prove that I wrote it. I just encrypt it with my private key and post it. Then anyone who wants to check can decrypt it with my public key. If the decryption works, then it means I have the private key and I wrote the message.

RSA is relatively simple: take two numbers (the private key), apply some math, and get a third number (the public key). You can write out all of the math in a few lines, and yet RSA changed the world. Business doesn’t work on the Internet without public-key encryption.

RSA And HTTPS

We use public-key encryption every day with HTTPS. When you access Facebook, Twitter or Amazon with HTTPS, you’re using a simple encryption mechanism like the one-time pad, but you’re creating the pad with public-key encryption. Without HTTPS, anyone else at Starbucks could read your credit-card number, Facebook password or private email while sipping a latte.

Amazon has a certificate from a company named VeriSign. The certificate certifies that Amazon is Amazon, and it contains its public key. Your browser creates a special key just for that session and encrypts it using Amazon’s public key. Then it sends it over the Internet, knowing that only Amazon can decrypt the session key. Once you’ve exchanged that secret key, you can use it as the one-time pad to protect your password and credit-card number.

You could keep using public-key encryption for the whole session, but because of all the math, it’s much slower than the one-time pad.

RSA And GitHub

Another place many of us use RSA is GitHub. Every time you push a change to GitHub or pull from a master branch, GitHub has to make sure you have permission to make the change. It gets its security through a secure command shell using RSA.

Remember when you set up your GitHub account and followed some commands to generate keys?

You used the SSH-Keygen tool to generate a new RSA private/public key pair. Then you went to your GitHub account page and entered your public key.

Now, when GitHub needs to authenticate you, it asks your computer to sign something with your private key and return the signed data. With your public key, GitHub can confirm that the signature is authentic and could only have been produced by someone who has the corresponding private key—even though GitHub itself doesn’t have that private key.

That’s better than a simple password because nobody can snoop it. And if GitHub ever gets hacked, your private key won’t be in danger because only you have it.

Sharing Passwords

When WordPress.org was “hacked�, it wasn’t really hacked. WordPress plugin developers, like everyone else, have accounts on other websites. They also reuse their passwords. When hackers cracked those other websites, they used the stolen passwords to log into WordPress.org and make malicious changes to plugins.

Most people use the same user name and password on multiple websites. That makes your website only as secure as everyone else’s. Public-key encryption changes that. Because you never have to share your private key, it doesn’t matter if other websites get hacked. If an attacker breaks into GitHub and gets your public key, they can’t use it to impersonate you or log in as you on other websites. Only someone with your private key can do that, which is why your private key remains safe on your computer. Using public-key cryptography makes GitHub much more secure.

GitHub Gets Hacked

GitHub was hacked recently, but not because the encryption failed. Real-world security breaches are caused by problems in implementation, not in math.

In this case, the hacker was able to exploit a hole and add his public key to the Ruby on Rails repository. Once the key was added, GitHub used it to verify the hacker’s identity and granted him access. We’re lucky this hacker was friendly and told GitHub about the issue.

Once the problem was fixed, you could keep using your private key because GitHub never had it to lose; it stayed on your machine. Public keys saved GitHub from serious problems.

The weakest link in GitHub’s security was in the mechanism that allowed clever users to add public keys to other projects without being authorized. The math was perfect, but the implementation wasn’t.

Public Keys In The Wild

Knowing the fundamentals is essential (you might say the key) to writing secure applications. The math is complex, but the basics are simple:

• There are two main types of encryption: shared-key encryption, such as a one-time pad, and public-key encryption, which uses public and private keys.
• Shared-key encryption is faster, but sharing the keys is difficult.
• RSA is the most popular public-key encryption algorithm, but a few others are in general use, as well as some cool experimental systems.
• Public-key cryptography works best in combination with other technologies.
• Don’t ever share your private key with anyone.

When it comes time to implement public-key cryptography in your application, don’t. RSA and other algorithms are already implemented in all major languages. These libraries include extra security features such as padding and salts, and they have a lot of testing behind them.

Most security flaws come from poor implementations and misunderstanding about the libraries. You don’t have to write your own cryptography libraries, but you do have to know the fundamentals so that you can use the ones that are out there.

Illustrations in this article were provided by Robb Perry.

(al) (km)

© Zack Grossbart for Smashing Magazine, 2012.

When my WordPress plugin had only three users, it didn’t matter much if I broke it. By the time I reached 100,000 downloads, every new update made my palms sweat.

My first goal for the WordPress Editorial Calendar was to make it do anything useful. I was new to JavaScript and PHP and didn’t really know what I could pull off. In a few days I had a proof of concept. In a few more I had a working version and was asking friends to install it. The calendar worked… sort of.

I spent three times as much time fixing bugs as I did coding. Once the plugin worked, I wrote unit tests to make sure it kept working.

The unit tests for my calendar use QUnit, but they really use just three functions: test, expect and ok. This article shows you how to integrate unit tests into your WordPress plugin, where to write unit tests and how they can help you.

QUnit Basics

Unit tests follow a basic pattern: do something, then check the results. (“Is this variable 4 when it should be 5?â€� “Does this line of my table show up where it’s supposed to?â€�)

function myTest() {
    test('My test run', function() {
        expect(2);
        ok(5 === 5, 'This test will always pass');
        ok(5 === 6, 'This test will always fail');
    });
}

This structure forces you to think of your code in simple units that return true or false. The test function starts a test run with two arguments: the title for this test run, and the function containing the tests. The expect function tells QUnit how many tests are in the run. If we call too few or too many, it causes an error.

The ok function performs the test of the expression. It takes two arguments: a boolean indicating whether the test was successful, and a message.

Test runs are added to a special list section, which shows the total number of tests, whether each test passed or failed, and how long the tests took.

A Real Unit Test

WordPress plugins complicate testing by using a combination of PHP and JavaScript. Even passing a simple value from PHP to JavaScript gets tricky.

The function below finds a WordPress preference with the get_option function and creates a JavaScript variable with the resulting value.

function getOption($myOption, $myVariable) {
    if (get_option($myOption) != "") {
        ?><script type="text/javascript">
            <?php echo($myVariable); ?> = <?php echo(get_option($myOption)); ?>;
        <php
    }
}

Now we’ll call it to get the name of the blog and set it to a variable named myBlogName.

getOption("blogname", "myBlogName");

Little helper functions like these glue your plugin together, but they always worry me. Did I access the JavaScript variable with the right name or did I mistype it? Did I use the same name twice? A simple unit test makes all of these worries go away.

function wpPrefTest() {
    test('Get access to WordPress preferences', function() {
        expect(1);
        ok(myBlogName, 'The variable (myBlogName) should be available.');
    });
}

This unit test checks whether the variable myBlogName exists. We could also look for a specific value or compare it to something else from the application.

Once you have this unit test, you’ll never need to worry about getting the blog’s name. It will always be there, and you’ll find out fast if you ever break it.

Integrating QUnit With WordPress

Testing in special development environments isn’t accurate. I wanted to add QUnit directly to my calendar, but I didn’t want to make the page’s size larger. The solution is a URL parameter and a little PHP to include QUnit only when I need it:

wp_enqueue_script( "qunit", path_join(WP_PLUGIN_URL, basename( dirname( __FILE__ ) )."/lib/qunit.js"), array( 'jquery' ) );
wp_enqueue_script( "edcal-test", path_join(WP_PLUGIN_URL, basename( dirname( __FILE__ ) )."/edcal_test.js"), array( 'jquery' ) );

This tells WordPress to include the QUnit JavaScript and my unit tests from edcal_test.js. I could have just embedded the script’s reference directly on my page, but I might have run into trouble if other plugins on the same page use QUnit.

You can see the full source code here.

The next step was to make sure that these scripts load only when I need them. To do this, I wrapped the code in a check for a URL parameter:

if ($_GET['qunit']) {
    wp_enqueue_script( "qunit", path_join(WP_PLUGIN_URL, basename( dirname( __FILE__ ) )."/lib/qunit.js"), array( 'jquery' ) );
    wp_enqueue_script( "edcal-test", path_join(WP_PLUGIN_URL, basename( dirname( __FILE__ ) )."/edcal_test.js"), array( 'jquery' ) );
}

This loads the scripts only if I’m running unit tests, and everything else in the plugin stays the same. You can run the unit tests at any time just by adding &qunit=true to the end of the URL. That’s a good thing because my unit tests actually change what’s going on in the blog.

You can run the Editorial Calendar’s unit tests in your browser right now. Scroll down to see the unit test results at the bottom of the page.

The PHP makes sure that my scripts get to the browser. The last step is to call them from my JavaScript. Once again, I want to call them only if we’re in unit test mode. So, I add a small function to get the parameters from the URL:

getUrlVars: function() {
    var vars = [], hash;
    var hashes = window.location.href.slice(window.location.href.indexOf('?') + 1).split('&');
    for (var i = 0; i < hashes.length; i++) {
        hash = hashes[i].split('=');
        vars.push(hash[0]);
        vars[hash[0]] = hash[1];
    }
    return vars;
}

And then I call my unit tests if the QUnit parameter is there:

jQuery(document).ready(function() {
    if (edcal.getUrlVars().qunit) {
        edcal_test.runTests();
    }
});

This ensures that we call the unit tests only if they’re available.

The last step is to set up a space for the unit test’s output. QUnit writes its test results into specially designated tags on your HTML page. You could embed these tags directly in your HTML output, but because they need to be there only when QUnit is active, I create the HTML in JavaScript instead.

jQuery('head').append('<link>');
css = jQuery('head').children(':last');
css.attr({
    rel: 'stylesheet',
    type: 'text/css',
    href: '../wp-content/plugins/edcal/lib/qunit.css'
});

jQuery('#wpbody-content .wrap').append('<div id="edcal-qunit"></div>');

jQuery('#edcal-qunit').append(
    '<h1 id="qunit-header">WordPress Editorial Calendar Unit Tests</h1>' +
    '<h2 id="qunit-banner"></h2>' +
    '<div id="qunit-testrunner-toolbar"></div>' +
    '<h2 id="qunit-userAgent"></h2>' +
    '<ol id="qunit-tests"></ol>' +
    '<div id="qunit-fixture">test markup</div>');

QUnit needs a list tag, a couple of divs and a style sheet.

Now we’re ready to write our first test.

The First Unit Test

The first calendar unit tests scroll the calendar up and down and make sure we see the correct number of days.

moveTests: function() {
    var curSunday = edcal.nextStartOfWeek(Date.today()).add(-1).weeks();
    edcal.moveTo(Date.today());

    test('Move to today and check visible dates', function() {
        expect(2);
        ok(edcal_test.getFirstDate().equals(curSunday.clone()),
           'firstDate should match ' + curSunday);
        ok(edcal_test.getLastDate().equals(
           curSunday.clone().add(edcal.weeksPref).weeks().add(-1).days()),
           'lastDate should match ' + curSunday);
    });
}

Our first test moves the calendar to today and checks to see if the first and last days are what we expect. We set up a variable, move the calendar and start the test by calling the test function.

In this case we want to make sure the dates are correct, so we compare the date from the calendar to the one we expect and then pass the result to the ok function. The test succeeds if they match and fails if they don’t.

This test may seem simple, but a lot is going on under the hood. We’re testing date handling, drawing and the fundamental arithmetic of the calendar.

Unit tests can do anything. The WordPress Editorial Calendar unit tests automate the plugin like a robot. They cover everything a user could do with the calendar.

What To Unit Test

I write a lot more unit tests for JavaScript than I do for compiled languages. In Java and C++, the compiler catches many of my mistakes, whereas JavaScript lets you pass a string when you meant to pass a number and lets you call a function with two arguments when it needs three.

Here’s the basic list of areas I test in JavaScript applications:

• Tests that make sure the program does what it is meant to do
  These tests ensure the basic functionality keeps working; they do not test interactions. (The calendar lets you drag and drop posts, but writing unit tests for dragging and dropping wouldn’t make sense; instead, we would focus on what happens after the drop event.)
• Tests that make sure the program doesn’t do what it’s not meant to do
  These tests ensure the program fails properly when it gets garbage.
• A unit test for every major bug I’ve found
  These tests ensure that none of these bugs creep back in.

APIs and other clear boundaries in the code lend themselves well to unit tests, as do utility functions that you call from many places in the application. With the calendar, this means testing calendar movements and testing how we create and modify posts in WordPress, like this:

1. Move the calendar and check the dates;
2. Create a post and make sure it was created properly;
3. Edit the post we just created and make sure it saves properly;
4. Move the post and make sure it shows up at the correct date;
5. Move the post from two places at the same time and make sure we get the proper warning message;
6. Delete the post from the server and make sure it isn’t there anymore.

The penultimate test covers an error condition in which two people move the same post at the same time. Unit tests work well for this kind of error, because the error type is difficult to test and manual testing is less likely to uncover problems.

For your own application, you should have at least one unit test for every data-changing operation that users can perform. I like to add them for all of the places where a user can get data, too. It might sound like a lot of work, but you can cut down on it by writing single tests that cover multiple areas.

Asynchronous Unit Tests

Many of these combination unit tests make AJAX calls. QUnit provides a special function for handling AJAX called asyncTest. This function works just like test, but it pauses the test run at the end of the function. The QUnit framework waits until your AJAX call completes and you call the start function before restarting the test run.

The asyncTest function handles all of the tests that edit posts from the calendar, including deleting the post at the end:

asyncTest('Delete the post created for testing', function() {
    expect(1);

    edcal.deletePost(edcal_test.post.id, function(res)
    {
        equals(jQuery('#post-' + res.post.id).length, 0,
               'The post should now be deleted from the calendar.');
        start();
    });
});

When you restart the test framework you can call more tests. Calling the next test function at the end of the previous one chains them together, and it supports calling all of your tests with just a call to the first function.

These tests, which call AJAX, ensure that the integration between the JavaScript on the client side and the PHP on the back end works properly.

That’s Not a Unit Test

When I first learned to write unit tests in C++ the rule was this: a single test should only call code in a single module or CPP file. That is, a unit test should test one unit of code.

Changing posts from unit tests violates that rule. Instead of just testing JavaScript, I’m really testing JavaScript, PHP, WordPress itself and MySQL all at once. That makes it an automated integration test.

Integration tests aren’t traditional unit tests, but they work well for WordPress plugins. When I create a post, I’ll know that the AJAX code in my plugin works as well as the JavaScript code. Covering a larger portion of the application with fewer tests makes it easier to focus on what I should be testing.

What Not To Unit Test

You could write unit tests forever, but some are more useful than others. Here are some guidelines.

• Don’t unit test the UI.
  The test has to run by itself. It can’t wait for you to click a button or look at something to make sure it appears correctly.
• Don’t unit test performance.
  Tests take a variable amount of time on different machines and browsers. Don’t write unit tests that depend on a function being returned in a set amount of time.
• Don’t unit test code from other projects.
  Adding unit tests for WordPress or a jQuery plugin you depend on might be tempting, but it rarely pays off. If you want to contribute unit tests to WordPress.org that’s great, but your unit tests should check that your plugin works.

The Editorial Calendar has 26 unit tests, at about 3,500 lines of code. That may not sound like much, but they’ve saved many of my releases.

Unit Tests Saved My Butt

I didn’t add unit tests until the thirteenth release of my plugin. By then, the calendar had a couple of hundred users and was growing fast. My plugin was working, and I was getting close to release 1.0.

Instead of adding new features, I took in a new framework, added special code to load it, wrote 381 lines of unit tests and integrated all of this into the plugin. It seems like a lot of work, but it saved my butt.

Right before a release, I wrote some harmless-looking PHP code like the following to get the JSON data that represented a set of posts to show up in the calendar:

function edcal_postJSON($post) {
    setup_postdata($post);
    ?>
    {
        "date" : "<?php the_time('d') ?><?php the_time('m') ?><?php the_time('Y') ?>",
        "title" : <?php echo($this->edcal_json_encode(get_the_title())); ?>,
        "author" : <?php echo($this->edcal_json_encode(get_the_author())); ?>
    },
    <?php
}

function edcal_posts() {
    header("Content-Type: application/json");

    global $post;
    $args = array(
        'posts_per_page' => -1,
        'post_status' => "publish&future&draft",
        'post_parent' => null // any parent
    );

    $myposts = query_posts($args);

    ?>[
        <?php
        $size = sizeof($myposts);
        for($i = 0; $i < $size; $i++) {
            $post = $myposts[$i];
            edcal_postJSON($post, $i < $size - 1);
        }
    ?> ]
    <?php
}

I ran the code and everything worked. I was about to release the new version but ran my unit tests first to make sure. They failed. Can you spot the bug? I didn’t.

I was returning a JSON array, but the last item in the array had a trailing comma. That’s invalid JSON. Firefox accepts it, but Safari, Chrome and IE don’t. I almost shipped a broken plugin to over half of my users.

Now I run the unit tests on all major browsers whenever I release a new version. Any time WordPress releases a new version, I run the unit tests. WordPress 3.3 broke the calendar — and I found out exactly why in 15 seconds.

Popular Plugins Are Stable Plugins

Most WordPress plugins are free and open source, but free doesn’t always mean cheap. The total cost of ownership of unstable plugins is more than people would pay. That’s a fancy way of saying that users will run from your plugin if it’s a bug fest.

My plugin became popular because of its features, but stability kept it popular. People remember one buggy release for a long time. If the Editorial Calendar deletes or corrupts posts from just one blog, thousands of people would stop using it. And they’d be justified.

Unit tests add the stability you need to support the multitude of browsers, mobile devices and dark corners involved in any JavaScript application. They’re easy to write, and they pay you back: because you find the bugs, and your users don’t.

(al)

© Zack Grossbart for Smashing Magazine, 2012.

Before drawing anything in a browser, ask yourself three questions:

1. Do you need to support older browsers?If the answer is yes, then your only choice is Raphaël. It handles browsers all the way back to IE 7 and Firefox 3. Raphaël even has some support for IE 6, although some of its underlying technology cannot be implemented there.
2. Do you need to support Android?Android doesn’t support SVG, so you’ll have to use Paper.js or Processing.js. Some rumors say that Android 4 will handle SVG, but the majority of Android devices won’t support it for years.
3. Is your drawing interactive?Raphaël and Paper.js focus on interaction with drawn elements through clicking, dragging and touch. Processing.js doesn’t support any object-level events, so responding to user gestures is very difficult. Processing.js can draw a cool animation on your home page, but the other tools are better for interactive applications.

Paper.js, Processing.js and Raphaël are the leading libraries for drawing on the Web right now. A couple of others are up and coming, and you can always use Flash, but these three work well with HTML5 and have the widest support among browser vendors.

Choosing the right framework will determine the success of your project. This article covers the advantages and disadvantages of each, and the information you need to make the best choice.

All of the code in this article is open source and can be run on the demo page that accompanies this article.

Overview

It’s all JavaScript once the page runs, but the frameworks take different paths to get there. Raphaël is written directly in JavaScript, but Paper.js uses PaperScript, and Processing.js uses its own script. They all support Firefox, Chrome and Safari, but Internet Explorer is an issue — Paper.js and Processing.js use the canvas tag and thus require IE 9.

PaperScript is a JavaScript extension that makes it possible to write scripts that don’t pollute the global namespace. This cuts down on JavaScript conflicts. PaperScript also supports direct math on objects such as Point and Size: you can add two points together as if they were numbers.

Processing.js is based on a framework named Processing, which runs in the Java Virtual Machine. You define int and float instead of var, and you can use classes with Java-style inheritance. While the Processing.js script looks a little like Java, it’s more like JavaScript and doesn’t require many of the more complex features of Java.

Using all three libraries is easy if you have some familiarity with JavaScript.

Getting Started

Start by importing each library. The process for setting each up is a little different.

Setting Up Paper.js

<head>
<script src="paper.js" type="text/javascript" charset="utf-8"></script>
<script type="text/paperscript" canvas="paperCircle" src="paper_circle.pjs" id="script"></script>
</head>
<body>
<canvas id="paperCircle" class="canvas" width="200" height="200" style="background-color: white;"></canvas>

Paper.js specifies a script type of text/paperscript and the ID of the canvas tag that you’ll draw on. It uses that ID to know where to draw.

Setting Up Processing.js

<head>
<script src="processing.js" type="text/javascript" charset="utf-8"></script>
</head>
<body>
<canvas width="200" height="200" class="canvas" data-processing-sources="processing_circle.java"></canvas>

Processing.js uses the data-processing-sources attribute of the canvas tag to import your drawing. I use a .java extension for Processing’s source file so that my editor color-codes it properly. Some authors use a .pde or .pjs extension. It’s up to you.

Setting Up Raphaël

<head>
<script src="raphael-min.js" type="text/javascript" charset="utf-8"></script>
<script src="raphael_circle.js" type="text/javascript" charset="utf-8"></script>
</head>

Raphaël is imported like any other JavaScript file. It works well with jQuery’s ready function or any other JavaScript framework.

Now we can start drawing.

Object-Oriented Drawing

Both Paper.js and Raphaël use object-oriented drawing: you draw a circle and get back a circle object. Processing.js draws the circle and doesn’t give you anything back. The following simple example makes it clear. Let’s start with a circle in the middle of the screen at point 100,100.

Paper.js:

var circle = new Path.Circle(new Point(100, 100), 10);
circle.fillColor = '#ee2a33';

Raphaël:

var paper = Raphael('raphaelCircle', 200, 200);
var c = paper.ellipse(100, 100, 10, 10);
c.attr({'fill': '#00aeef', 'stroke': '#00aeef'});

Processing.js:

void setup() {
   size(200, 200);
}

void draw() {
   background(#ffffff);
   translate(100, 100);
   fill(#52b755);
   noStroke();
   ellipse(0, 0, 20, 20);
}

Each code snippet draws the same circle. The difference is in what you can do with it.

Paper.js creates the circle as a path object. We can hold onto the object and change it later. In Paper.js, circle.fillColor = 'red'; fills our circle with red, and circle.scale(2) makes it twice as big.

Raphaël follows Paper.js’ object-oriented model. In Raphaël, we can change the color of our circle with circle.attr('fill', 'red');, and scale it up with circle.scale(2, 2);. The point is that the circle is an object that we can work with later.

Processing.js doesn’t use objects; the ellipse function doesn’t return anything. Once we’ve drawn our circle in Processing.js, it’s part of the rendered image, like ink on a page; it’s not a separate object that can be changed by modifying a property. To change the color, we have to draw a new circle directly on top of the old one.

When we call fill, it changes the fill color for every object we draw thereafter. After we call translate and fill, every shape will be filled with green.

Because functions change everything, we can easily end up with unwanted side effects. Call a harmless function, and suddenly everything is green! Processing.js provides the pushMatrix and popMatrix functions to isolate changes, but you have to remember to call them.

Processing.js’ no-objects philosophy means that complex drawings run faster. Paper.js and Raphaël contain references to everything you draw, and so the memory overhead created by complex animations will slow down your application. Processing.js contains no references to drawn elements, so each shape takes up a tiny amount of memory. Memory overhead pays off if you need to access an object later, but it’s overkill if you don’t. Paper.js gives you a way out of this with the Symbol object and by rasterizing objects, but you have to plan ahead to keep the app running fast.

The object-oriented versus no-objects philosophy has implications for everything you do with these libraries. It shapes the way each library handles animations.

Let’s Make It Move

Rotating circles aren’t very interesting, so we’ll make a square rotate around a circle.

Animation in Processing.js

Processing.js supports animation with the predefined setup and draw functions, like this:

float angle = 0.0;
void setup() {
   size(200, 200);
   frameRate(30);
}

void draw() {
   background(#ffffff);
   translate(100, 100);
   fill(#52b755);
   noStroke();
   ellipse(0, 0, 20, 20);

   rotate(angle);
   angle += 0.1;
   noFill();
   stroke(#52b755);
   strokeWeight(2);
   rect(-40, -40, 80, 80);
}

The setup function is called once when the application starts. We tell Processing.js to animate with a frame rate of 30 frames per second, so our draw function will be called 30 times every second. That rate might sound high, but it’s normal for making an animation look smooth.

The draw function starts by filling in the background of the canvas; it paints over anything left over from previous invocations of the draw function. This is a major difference with Processing.js: we are not manipulating objects, so we always have to clean up previously drawn shapes.

Next, we translate the coordinate system to the 100,100 point. This positions the drawing at 100 pixels from the left and 100 pixels from the top of the canvas for every drawing until we reset the coordinates. Then, we rotate by the specified angle. The angle increases with every draw, which makes the square spin around. The last step is to draw a square using the fill and rect functions.

The rotate function in Processing.js normally takes radians instead of degrees. That’s why we increase the angle of each frame by 0.2, rather than a higher number such as 3. This is one of many times when trigonometry shows up in this method of drawing.

Animation in Paper.js

Paper.js makes this simple animation easier than in Processing.js, with a persistent rectangle object:

var r;

function init() {
   var c = new Path.Circle(new Point(100, 100), 10);
   c.fillColor = '#ee2a33';

   var point = new Point(60, 60);
   var size = new Size(80, 80);
   var rectangle = new Rectangle(point, size);
   r = new Path.Rectangle(rectangle);
   r.strokeColor = '#ee2a33';
   r.strokeWidth = 2;
}

function onFrame(event) {
   r.rotate(3);
}

init();

We maintain the state of our square as an object, and Paper.js handles drawing it on the screen. We rotate it a little for each frame. Paper.js manages the path, so we don’t have to redraw everything for each frame or keep track of the angle of rotation or worry about affecting other objects.

Animation in Raphaël

Animations in Raphaël are written in standard JavaScript, so Raphaël doesn’t have specific functions for handling animation frames. Instead, we rely on JavaScript’s setInterval function.

var paper = Raphael('raphaelAnimation', 200, 200);
var c = paper.ellipse(100, 100, 10, 10);
c.attr({
   'fill': '#00aeef',
   'stroke': '#00aeef'
});

var r = paper.rect(60, 60, 80, 80);
r.attr({
   'stroke-width': 2,
   'stroke': '#00aeef'
});

setInterval(function() {
   r.rotate(6);
}, 33);

Raphaël is similar to Paper.js in its object-oriented approach. We have a square, and we call a rotate function on it. Thus, we can easily spin the square with a small amount of code.

Interaction

Raphaël shines when you need to enable interactivity in a drawing. It provides an event model similar to JavaScript’s, making it easy to detect clicks, drags and touches. Let’s make our square clickable.

Interactions With Raphaël

var paper = Raphael('raphaelInteraction', 200, 200);
var r = paper.rect(60, 60, 80, 80);
r.attr({'fill': '#00aeef', 'stroke': '#00aeef'});

var clicked = false;

r.click(function() {
   if (clicked) {
      r.attr({'fill': '#00aeef', 'stroke': '#00aeef'});
   } else {
      r.attr({'fill': '#f00ff0', 'stroke': '#f00ff0'});
   }
   clicked = !clicked;
});

The click function in Raphaël works like jQuery, and you can add it to any object. Once we get the click event, changing the color of the square is easy. Raphaël has more functions to support dragging, hovering and all of the other user interactions you expect from JavaScript.

Interactions With Paper.js

Paper.js has a different way of managing interactions, but it’s still pretty easy:

var hitOptions = {
   fill: true,
   tolerance: 5
};

function init() {
   var point = new Point(60, 60);
   var size = new Size(80, 80);
   var rectangle = new Rectangle(point, size);
   r = new Path.Rectangle(rectangle);
   r.fillColor = '#ee2a33';
}

function onMouseUp(event) {
   var hitResult = project.hitTest(event.point, hitOptions);

   if (hitResult && hitResult.item) {
      if (hitResult.item.clicked) {
         hitResult.item.fillColor = '#ee2a33';
      } else {
         hitResult.item.fillColor = '#f00ff0';
      }

      hitResult.item.clicked = !hitResult.item.clicked;
   }
}

init();

Paper.js deals with mouse gestures through a concept called “hit testing.� A hit finds the point under the mouse cursor and figures out which object it lies above. Hit options enable you to define how the hit works: you can set options for such things as how close the mouse has to be, and whether the middle of the object counts or only the edge. We can extend this hit test to any object or group of objects in Paper.js.

The Paper.js team added object-level events similar to Raphaël’s a few weeks ago. The events should show up in the next release.

Interactions With Processing.js

Processing.js makes detecting mouse clicks tricky. It doesn’t support object-level events or hit testing, so we’re pretty much on our own.

float bx;
float by;
int bs = 20;
boolean bover = false;
boolean clicked = false;

void setup() {
   size(200, 200);
   bx = width/2.0;
   by = height/2.0;
   noStroke();
   fill(#52b755);
   frameRate(10);
}

void draw() {
   background(#ffffff);

   // Test if the cursor is over the box
   if (mouseX > bx-bs && mouseX < bx+bs &&        mouseY > by-bs && mouseY < by+bs) {
      bover = true;
   } else {
      bover = false;
   }

   translate(100, 100);
   rect(-40, -40, 80, 80);
}

void mousePressed() {
   if (bover) {
      if (clicked) {
         fill(#52b755);
      } else {
         fill(#f00ff0);
      }
      clicked = !clicked;
   }
}

Once Processing.js draws the square, it forgets about it. We want the color of the square to change when we click on it, but the script doesn’t know that, so we have to do all of the calculations ourselves. The draw function detects the mouse cursor’s position and does the math to determine whether it lies within the square.

The code is not too bad for the square, but our circle would need Ï€r2. And more complex shapes such as ovals, curves and compound shapes would require even more math.

No Clear Winner

Each framework has its advantages. Between them, the features make for cool demos and even cooler applications.

Showing Off Paper.js

Paper.js excels at manipulating complex shapes. It can turn, twist and transform any object in hundreds of ways. These transforms make it easy to convert objects based on interactive gestures. The new Google Music Tour, which makes colored lines beat in time to music, shows how one can make complex changes on simple shapes.

The other wow factor in Paper.js is its support of raster graphics. Paper.js can completely change the way images are drawn — including by turning them into spirals and Q*bert boards.

Showing Off Processing.js

Processing.js’ biggest feature is speed, making it possible to draw complex animations on slower machines. Many examples are out there, but the fluidity of Processing.js animations shows up best in Ricardo Sánchez’s koi pond.

The swishing of the tails and waving of the bodies make the koi look very natural. Processing.js makes this easy, with support for curves and customized animations.

Processing.js also supports complex drawing elements such as shading, lighting and 3-D transforms. If you want to create complex animations in canvas very quickly, then Processing.js is the clear winner.

Showing Off Raphaël

The best feature of Raphaël is its support for Internet Explorer 7 and 8. If your application has to run on older browsers, then Raphaël is the only option.

The other big feature of Raphaël is its community. Raphaël is older than Paper.js and Processing.js and thus has had more time to build examples, tutorials and user support. It has built-in support for easing, animation transforms and the event handlers that we saw in the interaction example; it also has a comprehensive charting library.

Raphaël also has the best tooling support.

The Tools

If you’ve worked with Flash, the lack of tools for these frameworks will disappoint you. Many of the frameworks will edit SVG images, but none of them offer a drag-and-drop method for creating applications.

A few simple tools are out there, but they are more like proofs of concept than actual products. Adobe is working on a tool named Edge, but it has a long way to go.

If you want to drag and drop, then Web animations aren’t for you yet. Right now, this method of drawing is more like video-game programming. Writing code to draw a circle is tougher than clicking and dragging, but it scales to more complex applications and some fun stuff.

Let’s Build Something Real

So far, we’ve looked at some simple examples, seen the best features of each platform and looked at how to choose the right one. Each framework has pluses and minuses, but judging them is difficult until you create an actual application.

To compare each framework, I’ve drawn some gears. Each gear is made up of two circles, with a set of teeth around the outer circle.

When the shapes are all given the same color, they look just like a gear.

Every gear will rotate a little with each frame of the animation. The first gear will be given a speed, and the rest will move relative to it. The gears will arrange, mesh and rotate together with a crazy amount of trigonometry. Put them together and you’ve got a complex gear system.

Paper.js:

Processing.js:

Raphaël:

Well, that wasn’t quite Raphaël. The rotate function work different in Raphaël than it does in Paper.js and Processing.js. Raphaël doesn’t support rotation around a fixed point. Instead, the teeth of the gears are drawn and redrawn independently, and they fly through the air instead of rotating around the center. The only way to really turn the gear would be to draw the entire gear as a single path, and that takes more math than I’m willing to write. If anyone wants to give it a try, everything is open source.

The Future Of Web Drawing

We gamble on every new technology that we learn: we hope that it catches on and that our investment pays off. Technologies rise and fall on their respective merits, but other factors comes into play, such as vendor support and business uses. The future of our industry is almost a guessing game.

Right now, Flash looks like a bad investment. Flash has great tools, years of development and a large community, but even Adobe is moving away from it.

SVG is in a similar situation. Browsers support it now, but it isn’t getting a lot of attention.

Every browser vendor is working hard to render canvas faster, to use hardware acceleration and to better support libraries such as Paper.js and Processing.js. All mobile devices support canvas, and their developers are working to improve it.

(al)

© Zack Grossbart for Smashing Magazine, 2012.

The Web is just starting to use animation well. For years, animated GIFs and Flash ruled. Text moved and flashed, but it was never seamless. Animations had boxes around them like YouTube videos. HTML5 canvas changes everything about Web animation.

The canvas element makes it possible to integrate drawings and animations with the rest of your page. You can combine them with text and make animations interactive. This drawing mechanism is powerful, but very low-level.

Animations get more power and need less coding when you combine the canvas tag with higher-level libraries such as Paper.js. This article introduces HTML5 animation and walks you through creating an animation of dandelion seeds blowing in the wind.

Neat Is Easy, But Messy Is Hard

Computers love clean. They make spreadsheets, do statistics and plot multivariate curves; they always color inside the lines.

In the real world, even simple things are messy. Leaves falling from trees, water splashing — all the little interactions around us feel simple because we’re used to them; but little bursts of wind are actually messy and unpredictable.

For this article, we’ll animate dandelion seeds blowing in the breeze.

Dandelions are tricky because we all know what they look like: we’ve touched them and blown their seeds off. Commonplace objects produce instant recognition and feelings. I don’t have to tell you what dandelions are — you just know. Dandelions are a chaos of seeds piled on top of each other.

(Image: Arnoldius)

Our dandelion animation will never reproduce the complexity of the real thing, and it will work better if we don’t try: make it too close to real and it will feel funny. Instead, we’ll create a stylized dandelion that makes the right impression without all of the details.

Paper.js

Drawing simple shapes with the canvas tag, without any special drawing libraries, is easy. Create your canvas:

<canvas id="canvas" width="300" height="300"></canvas>

Then add a little JavaScript.

// Get our canvas
var canvas = $('#canvas')[0].getContext("2d");

// Draw a circle
canvas.beginPath();
canvas.arc(100, 100, 15, 0, Math.PI*2, true); 

// Close the path
canvas.closePath();

// Fill it in
canvas.fill();

Cheat sheets for canvas show you the basics, but when you get into more serious drawing, you’ll want a higher-level library, such as Paper.js.

Paper.js is a JavaScript library for drawings and animations. It’s based largely on Scriptographer, a scripting language for Adobe Illustrator. You can write JavaScript with Paper.js, but most of the time you’ll be working with a JavaScript variant called PaperScript.

Paper.js calls itself “The Swiss Army Knife of Vector Graphics Scripting,� and the “vector� part is important.

There are two basic types of graphics, vectorized and rasterized. Rasterized graphics are like the pictures you take with your camera: big rectangles with maps denoting the color of each pixel. Enlarge them and you’ll get blurry dots.

Vector graphics are like connect-the-dots pictures: they’re sets of lines and shapes that give instructions on how to draw the image at any size. Using vector graphics, you can make an image of the letter Z really big and it will still look sharp. If you turned it into a rasterized graphic by taking a picture of it and then blowing it up, the letter would get all blurry.

Vector graphics libraries are perfect for animation because they make resizing, rotating and moving objects easy. They’re also much faster, because the program has instructions for drawing each object instead of needing to figure it out.

The Paper.js examples page shows some of the amazing things you can do with vectorized graphics.

The dandelion is a complete functioning example, and you can see it all running on the example page. You can also change the code by clicking the “Edit� button, see your changes live, and copy and paste the code to your own website. Over the course of the article, we’ll explain each part of the code in turn, but please note that in order to run the code yourself, you will need to head over to the example page and copy and paste it to your own environment.

Drawing Our Dandelion

The first step is to import our JavaScript and PaperScript files.

<script src="paper.js" type="text/javascript" charset="utf-8"></script>
<script type="text/paperscript" canvas="canvas" src="dandelion.pjs" id="script"></script>

The PaperScript code for running the animation is declared as text/paperscript. Now we’re ready to start drawing.

The first part of our dandelion is the stem. The stem is the green arc, with a circle on the top for the bulb. We’ll make both shapes with a path, a list of shapes, points and lines that the browser is instructed to display.

Paths are the basic building blocks of animation. They render lines, curves and polygons. You can also fill them in to make complex shapes. Our path looks like this:

var path = new Path();
path.strokeColor = '#567e37';
path.strokeWidth = 5;

var firstPoint = new Point(0, 550);
path.add(firstPoint);

var throughPoint = new Point(75, 400);
var toPoint = new Point(100, 250);
path.arcTo(throughPoint, toPoint);

Our path is an arc, so it needs three points: the start, the end and a midpoint to arc through. Three points are enough to define any arc we need. The arcTo function draws the line between them. The path item also supports styling information, such as stroke color and stroke width; #567e37 and 5 will make our arcing line green and thick. Paper.js supports the same color definitions as CSS.

We can add a few more items to make it all easier to see:

path.fullySelected = true;

var circle = new Path.Circle(throughPoint, 5);
circle.fillColor = '#CC0000';

Fully selecting the path will display some lines to show us the arc; the red circle shows us where the through point is.

The stem ends with a circle to show the bulb of the flower and give us a place to attach all of the seeds. Circles are much easier in Paper.js than in direct canvas.

var bulb = new Path.Circle(toPoint, 10);
bulb.fillColor = '#567e37';

One line of code draws our circle, one more makes it green, and now we’re ready to add our seeds.

Drawing The Seeds

Each seed has a bulb, a little stem and a wispy part on top.

(Image: Hmbascom)

Our seed starts with a small oval for the bulb and an arc for the stem. The oval is a rectangle with rounded corners:

var size = new Size(4, 10);
var rectangle = new Rectangle(p, size);
var bottom = new Path.Oval(rectangle);
bottom.fillColor = '#d0aa7b';

The seed stem is another arc, but this one is much thinner than the flower stem:

var stem = new Path();
stem.strokeColor = '#567e37';
stem.strokeWidth = 1;
stem.add(new Point(p.x + 2, p.y));

var throughPoint = new Point(p.x + 4, p.y - height / 2);
var toPoint = new Point(p.x + 3, p.y - height);
stem.arcTo(throughPoint, toPoint);

The wisps are more arcs with a circle at the end of each line. Each seed has a random number of wisps that start at the top of the stem arc and curve out in different directions. Randomness makes them look a little bit messy and thus more natural. Each seed gets a random number of wisps, between 4 and 10.

for (var i = 0; i < random(4, 10); i++) {
    path = new Path();
    path.strokeColor = '#fff3c9';
    path.strokeWidth = 1;

    var p1 = new Point(p.x, p.y);
    path.add(new Point(p1.x + 2, p1.y + 2));

    // Each flutter extends a random amount up in the air
    var y = random(1, 5);

    // We draw every other stem on the right or the left so they're
    // spaced out in the seed.
    if (i % 2 == 0) {
        throughPoint = new Point(p1.x + random(1, 3), p1.y - y);
        toPoint = new Point(p1.x + random(5, 35), p1.y - 20 - y);
    } else {
        throughPoint = new Point(p1.x - random(1, 3), p1.y - y);
        toPoint = new Point(p1.x - random(5, 35), p1.y - 20 - y);
    }

    path.arcTo(throughPoint, toPoint);

    // Now we put the circle at the tip of the flutter.
    circle = new Path.Circle(toPoint, 2);
    circle.fillColor = '#fff3c9';
}

Now that we’ve drawn the seed, we need to manage it; later, we’ll want to move and rotate it. The seed is made up of a lot of parts, and we don’t want to have to manage each one separately. Paper.js has a nice group object. Groups associate a set of objects together so that we can manipulate them all at once.

var group = new Group();
group.addChild(bottom);
group.addChild(stem);

this.group = group;

The last step is to package our seed into a reusable object called Seed. We add all of the code we’ve been writing to a new function with the name Seed and add a function to create the initial variables. This example calls that function create, but you can name it anything you want.

function Seed() {

    this.create = function (/*Point*/ p, /*boolean*/ shortStem) {
    …

The create function draws the seed at the specified Point, and the shortStem boolean tells us whether this is a short stem. We’ll look at short-stemmed seeds a little later.

These types of functions don’t work as constructors in JavaScript, but are supported in PaperScript.

var seed = new Seed()
seed.create(new Point(100, 100), false);

Our seeds will look like this when we draw them:

The Seed object draws our random dandelion seeds. Now we can add them to our flower.

Adding A Little Chaos

The seeds will look better when we space them out around the circle of our dandelion bulb to feel like a halo of seeds. The bulb is a circle, and the circle is a path, so we can get each point on the path.

var bulb = new Path.Circle(toPoint, 10); bulb.fillColor = '#567e37';

var angle = 360 / bulb.length;
var seeds = [];

for (var i = 0; i < bulb.length; i++) {
    var seed = new Seed()
    seed.create(bulb.getPointAt(i));

    // Rotate each seed so that it points out from the bulb
    seed.rotate(i * angle);
    seeds.push(seed);
}

This will make a circle of seeds around the bulb but leave a space in the middle. We’ll add a few more seeds to fill in the center. We’re giving the center seeds short stems so that they show the white of the wisps more than the beige of the stems.

for (var i = 0; i < 18; i++) {
    var seed = new Seed()
    var point = new Point(toPoint.x + random(-3, 3),
                          toPoint.y + random(-3, 3));
    seed.create(new Point(toPoint), true);
    seed.rotate(random(0, 360));
    seeds.push(seed);
}

The seeds in the middle will bunch randomly and make our dandelion look nicely messy. Now we can make them blow off.

Animating The Seeds

Wind pushes seeds in complex patterns, and two seeds will never blow off the same way. We want to make them look real, so we’ll need a little more randomness.

Reproducing real wind is much too complicated, so we’ll make the seeds float off in a random-looking pattern. Each seed is assigned a random point on the right side of the screen as a final destination:

this.dest = new  Point(1800, random(-300, 1100));

The rotateMove function pushes each seed toward its destination point and rotates it. We can work with our Seed object as a group to rotate and move it with one function.

this.rotateMove = function(/*int*/ angle) {
    if (this.group.position.x < 850 && this.group.position.y < 650) {
        var vector = this.dest - this.group.position;
        this.group.position += vector / 150;

        this.angle += angle;
        this.group.rotate(angle);
    } else {
        this.isOffScreen = true
    }
}

This function will move the seed until it’s off the screen. Calling rotateMove for each frame of our animation will make the seed float across the screen.

Paper.js gives us an easy way to make animations with the onFrame function; when we implement onFrame, Paper.js will call it for every frame of our animation. With each frame, we iterate over each seed and move it across the screen.

function onFrame(event) {
    for (var i = 0; i < seedCount; i++) {
        if (!seeds[i].isOffscreen()) {
            seeds[i].rotateMove(random(2, 4));
        }
    }
}

The seeds slide and rotate a little closer to the destination point with each frame of the animation. Starting all of the seeds at the same point and ending them far apart makes them space out nicely as they move.

We don’t want all of the seeds to fall off at once, so we’ll use a timer to make them drift away.

function start() {
    var id = setInterval(function() {
        seedCount++;
        if (seedCount === seeds.length) {
            clearInterval(id);
        }
    }, 1000);
}

The timer waits for one second before releasing the next seed, giving our dandelion a nice floaty feel.

Some green grass and blue sky as a background image for our canvas puts it all into context. Now we have a dandelion with seeds floating on the breeze.

See the dandelion running here. You can edit and run the source code as part of the animation or download it from the dandelion GitHub page.

Paper.js In The Real World

Paper.js has some impressive examples and a nice coding model, but you should know a few gotchas before using it on your website.

It Doesn’t Work In Old Browsers

All Paper.js drawings use the canvas tag and require HTML5. This means that you need Internet Explorer 9+, Firefox 4+, Safari 5+ or Chrome. If your website must support older browsers, then you won’t be able to use canvas.

There’s no way around this requirement; if you need older browsers, you’re out of luck. As the Paper.js website says, “Let’s go forward!.�

Performance Can Be Slow

Paper.js can make a browser grind to a halt even if the browser supports HTML5. Pixar renders Buzz and Woody on giant server farms — all you get is your user’s cheap MacBook.

Not only are laptops slower than server clusters, but browsers make things worse by rendering the canvas tag with the CPU instead of the GPU. Games like Halo and Rage take advantage of the graphics processor on your video card to render rocket launchers and mutants. The CPU is less efficient with graphics, so the same computer that handles complex video games smoothly can make floating dandelion seeds look slow and jerky.

Make sure to test all of your animations with slower hardware, and watch the CPU usage. Use groups to minimize the calculations, and be very careful about what you do in each invocation of the onFrame function.

Mobile Devices Are Slower

Mobile performance is even worse. Most mobile devices support canvas, but they are mostly too slow to render canvas animations well. Even more powerful devices, like the iPad 2, can’t handle the dandelion seeds smoothly.

It Doesn’t Support Object-Level Events

Other drawing libraries, such as SVG (see below), support object-level mouse and keyboard events. Those events make it easy to respond when a path or a polygon is clicked, hovered over or touched.

The canvas tag doesn’t support object-level events. Paper.js has some basic functionality for hit testing, but it’s very low-level. You can listen for mouse and keyboard events on the whole canvas, but you’ll need to handle mapping those events to individual controls.

What About SVG?

The SVG (Scalable Vector Graphics) specification was defined over 10 years ago, but came to the forefront with support libraries such as Raphaël.js, which make it easy to generate SVG images with JavaScript. SVG is powerful, works well for smaller images, and is supported all the way back to Internet Explorer 7 with conversion to VML (Vector Markup Language). SVG is the best choice if you need to support older browsers.

The real issues with SVG are speed, future support and mobile devices. Every browser maker is actively working on making canvas faster. Safari 5 already offers hardware acceleration with the GPU for canvas, and the rest are working on it. SVG is also unsupported on Android devices.

There’s a growing community around canvas, the new technology that vendors are focusing on. They’re adding new features, fixing bugs and making it better every day.

Other Canvas Drawing Libraries

Paper.js isn’t the only option for canvas. Processing.js, from the creator of jQuery, ports the Processing programming language to JavaScript. It supports animations and has many examples.

The three.js engine supports canvas and the WebGL library, and it focuses more on 3-D drawings. Google Dart will also support canvas with built-in rendering objects.

Paper.js is a mature library with a very supportive community on the Paper.js Google Group and many impressive and well-documented examples. Check out some of the amazing things people are doing with it.

More Paper.js Examples

Our dandelion is just the beginning. Below are a few other impressive animations written in Paper.js.

• Examples, Paper.jsPaper.js has a page full of incredible examples. Voronoi is one of the best. Make sure to press the space bar and see the paths. More examples are in the GitHub folder.
• Nardove, Ricardo SánchezJellyfish and a koi pond written with Processing.js. Wait a minute, the jellies are shy.
• “Node Garden in Paper.js,â€� Andrew Berg
• “The HBO Recycling Programâ€�An infographic that I created using Paper.js to show how often different HBO series use the same actors.
• “20 Multi-Touch Gestures You Should Learn Today,â€� Zack GrossbartI created the interactive tutorial with the help of Paper.js.

Where’s your Paper.js amazingness?

(al)

© Zack Grossbart for Smashing Magazine, 2011.

Google Search likes simple, easy-to-crawl websites. You like dynamic websites that show off your work and that really pop. But search engines can’t run your JavaScript. That cool AJAX routine that loads your content is hurting your SEO.

Google’s robots parse HTML with ease; they can pull apart Word documents, PDFs and even images from the far corners of your website. But as far as they’re concerned, AJAX content is invisible.

The Problem With AJAX

AJAX has revolutionized the Web, but it has also hidden its content. If you have a Twitter account, try viewing the source of your profile page. There are no tweets there — just code! Almost everything on a Twitter page is built dynamically through JavaScript, and the crawlers can’t see any of it. That’s why Google developed AJAX crawling.

Because Google can’t get dynamic content from HTML, you will need to provide it another way. But there are two big problems: Google won’t run your JavaScript, and it doesn’t trust you.

Google indexes the entire Web, but it doesn’t run JavaScript. Modern websites are little applications that run in the browser, but running those applications as they index is just too slow for Google and everyone else.

The trust problem is trickier. Every website wants to come out first in search results; your website competes with everyone else’s for the top position. Google can’t just give you an API to return your content because some websites use dirty tricks like cloaking to try to rank higher. Search engines can’t trust that you’ll do the right thing.

Google needs a way to let you serve AJAX content to browsers while serving simple HTML to crawlers. In other words, you need the same content in multiple formats.

Two URLs For The Same Content

Let’s start with a simple example. I’m part of an open-source project called Spiffy UI. It’s a Google Web Toolkit (GWT) framework for REST and rapid development. We wanted to show off our framework, so we made SpiffyUI.org using GWT.

GWT is a dynamic framework that puts all of our content in JavaScript. Our index.html file looks like this:

<body>
   <script type="text/javascript" language="javascript"
   src="org.spiffyui.spsample.index.nocache.js"></script>
</body>

Everything is added to the page with JavaScript, and we control our content with hash tags (I’ll explain why a little later). Every time you move to another page in our application, you get a new hash tag. Click on the “CSS� link and you’ll end up here:

http://www.spiffyui.org#css

The URL in the address bar will look like this in most browsers:

http://www.spiffyui.org/?css

We’ve fixed it up with HTML5. I’ll show you how later in this article.

This simple hash works well for our application and makes it bookmarkable, but it isn’t crawlable. Google doesn’t know what a hash tag means or how to get the content from it, but it does provide an alternate method for a website to return content. So, we let Google know that our hash is really JavaScript code instead of just an anchor on the page by adding an exclamation point (a “bang�), like this:

http://www.spiffyui.org#!css

This hash bang is the secret sauce in the whole AJAX crawling scheme. When Google sees these two characters together, it knows that more content is hidden by JavaScript. It gives us a chance to return the full content by making a second request to a special URL:

http://www.spiffyui.org?_escaped_fragment_=css

The new URL has replaced the #! with ?_escaped_fragment_=. Using a URL parameter instead of a hash tag is important, because parameters are sent to the server, whereas hash tags are available only to the browser.

That new URL lets us return the same content in HTML format when Google’s crawler requests it. Confused? Let’s look at how it works, step by step.

Snippets Of HTML

The whole page is rendered in JavaScript. We needed to get that content into HTML so that it is accessible to Google. The first step was to separate SpiffyUI.org into snippets of HTML.

Google still thinks of a website as a set of pages, so we needed to serve our content that way. This was pretty easy with our application, because we have a set of pages, and each one is a separate logical section. The first step was to make the pages bookmarkable.

Bookmarking

Most of the time, JavaScript just changes something within the page: when you click that button or pop up that panel, the URL of the page does not change. That’s fine for simple pages, but when you’re serving content through JavaScript, you want give users unique URLs so that they can bookmark certain areas of your application.

JavaScript applications can change the URL of the current page, so they usually support bookmarking via the addition of hash tags. Hash tags work better than any other URL mechanism because they’re not sent to the server; they’re the only part of the URL that can be changed without having to refresh the page.

The hash tag is essentially a value that makes sense in the context of your application. Choose a tag that is logical for the area of your application that it represents, and add it to the hash like this:

http://www.spiffyui.org#css

When a user accesses this URL again, we use JavaScript to read the hash tag and send the user to the page that contains the CSS.

You can choose anything you want for your hash tag, but try to keep it readable, because users will be looking at it. We give our hashes tags like css, rest and security.

Because you can name the hash tag anything you want, adding the extra bang for Google is easy. Just slide it between the hash and the tag, like this:

http://www.spiffyui.org#!css

You can manage all of your hash tags manually, but most JavaScript history frameworks will do it for you. All of the plug-ins that support HTML4 use hash tags, and many of them have options for making URLs bookmarkable. We use History.js by Ben Lupton. It’s easy to use, it’s open source, and it has excellent support for HTML5 history integration. We’ll talk more about that shortly.

Serving Up Snippets

The hash tag makes an application bookmarkable, and the bang makes it crawlable. Now Google can ask for special escaped-fragment URLs like so:

When the crawler accesses our ugly URL, we need to return simple HTML. We can’t handle that in JavaScript because the crawler doesn’t run JavaScript in the crawler. So, it all has to come from the server.

You can implement your server in PHP, Ruby or any other language, as long as it delivers HTML. SpiffyUI.org is a Java application, so we deliver our content with a Java servlet.

The escaped fragment tells us what to serve, and the servlet gives us a place to serve it from. Now we need the actual content.

Getting the content to serve is tricky. Most applications mix the content in with the code; but we don’t want to parse the readable text out of the JavaScript. Luckily, Spiffy UI has an HTML-templating mechanism. The templates are embedded in the JavaScript but also included on the server. When the escaped fragment looks for the ID css, we just have to serve CSSPanel.html.

The template without any styling looks very plain, but Google just needs the content. Users see our page with all of the styles and dynamic features:

Google gets only the unstyled version:

You can see all of the source code for our SiteMapServlet.java servlet. This servlet is mostly just a look-up table that takes an ID and serves the associated content from somewhere on our server. It’s called SiteMapServlet.java because this class also handles the generation of our site map.

Tying It All Together With A Site Map

Our site map tells the crawler what’s available in our application. Every website should have a site map; AJAX crawling doesn’t work without one.

Site maps are simple XML documents that list the URLs in an application. They can also include data about the priority and update frequency of the app’s pages. Normal entries for site maps look like this:

<url>
   <loc>http://www.spiffyui.org/</loc>
   <lastmod>2011-07-26</lastmod>
   <changefreq>daily</changefreq>
   <priority>1.0</priority>
</url>

Our AJAX-crawlable entries look like this:

<url>
   <loc>http://www.spiffyui.org/#!css</loc>
   <lastmod>2011-07-26</lastmod>
   <changefreq>daily</changefreq>
   <priority>0.8</priority>
</url>

The hash bang tells Google that this is an escaped fragment, and the rest works like any other page. You can mix and match AJAX URLs and regular URLs, and you can use only one site map for everything.

You could write your site map by hand, but there are tools that will save you a lot of time. The key is to format the site map well and submit it to Google Webmaster Tools.

Google Webmaster Tools

Google Webmaster Tools gives you the chance to tell Google about your website. Log in with your Google ID, or create a new account, and then verify your website.

Once you’ve verified, you can submit your site map and then Google will start indexing your URLs.

And then you wait. This part is maddening. It took about two weeks for SpiffyUI.org to show up properly in Google Search. I posted to the help forums half a dozen times, thinking it was broken.

There’s no easy way to make sure everything is working, but there are a few tools to help you see what’s going on. The best one is Fetch as Googlebot, which shows you exactly what Google sees when it crawls your website. You can access it in your dashboard in Google Webmaster Tools under “Diagnostics.�

Enter a hash bang URL from your website, and click “Fetch.� Google will tell you whether the fetch has succeeded and, if it has, will show you the content it sees.

If Fetch as Googlebot works as expected, then you’re returning the escaped URLs correctly. But you should check a few more things:

• Validate your site map.
• Manually try the URLs in your site map. Make sure to try the hash-bang and escaped versions.
• Check the Google result for your website by searching for site:www.yoursite.com.

Making Pretty URLs With HTML5

Twitter leaves the hash bang visible in its URLs, like this:

http://twitter.com/#!/ZackGrossbart

This works well for AJAX crawling, but again, it’s slightly ugly. You can make your URLs prettier by integrating HTML5 history.

Spiffy UI uses HTML5 history integration to turn a hash-bang URL like this…

http://www.spiffyui.org#!css

… into a pretty URL like this:

http://www.spiffyui.org?css

HTML5 history makes it possible to change this URL parameter, because the hash tag is the only part of the URL that you can change in HTML4. If you change anything else, the entire page reloads. HTML5 history changes the entire URL without refreshing the page, and we can make the URL look any way we want.

This nicer URL works in our application, but we still list the hash-bang version on our site map. And when browsers access the hash-bang URL, we change it to the nicer one with a little JavaScript.

Cloaking

Earlier, I mentioned cloaking. It is the practice of trying to boost a website’s ranking in search results by showing one set of pages to Google and another to regular browsers. Google doesn’t like cloaking and may remove offending websites from its search index.

AJAX-crawling applications always show different results to Google than to regular browsers, but it isn’t cloaking if the HTML snippets contain the same content that the user would see in the browser. The real mystery is how Google can tell whether a website is cloaking or not; crawlers can’t compare content programmatically because they don’t run JavaScript. It’s all part of Google’s Googley power.

Regardless of how it’s detected, cloaking is a bad idea. You might not get caught, but if you do, you’ll be removed from the search index.

Hash Bang Is A Little Ugly, But It Works

I’m an engineer, and my first response to this scheme is “Yuck!� It just feels wrong; we’re warping the purpose of URLs and relying on magic strings. But I understand where Google is coming from; the problem is extremely difficult. Search engines need to get useful information from inherently untrustworthy sources: us.

Hash bangs shouldn’t replace every URL on the Web. Some websites have had serious problems with hash-bang URLs because they rely on JavaScript to serve content. Simple pages don’t need hash bangs, but AJAX pages do. The URLs do look a bit ugly, but you can fix that with HTML5.

Further Reading

We’ve covered a lot in this article. Supporting AJAX crawling means that you need to change your client’s code and your server’s code. Here are some links to find out more:

• “Making AJAX Applications Crawlable,â€� Google Code
• “Session History and Navigation,â€� HTML5 specification, section 5.4
• sitemaps.org
• Google Webmaster Tools
• Spiffy UI source code, with a complete example of AJAX crawling.

Thanks to Kristen Riley for help with some of the images in this article.

(al)

© Zack Grossbart for Smashing Magazine, 2011.