BlogmyQuery – BMQ

Forums have been around forever, so it should come as no surprise that several plugins for the popular publishing platform WordPress provide this feature, as well as support for integrating other forum software. One project, however, has a special place in the WordPress community, and that is bbPress. This is the software created by WordPress founder, Matt Mullenweg, as a lightweight system for the WordPress.org support forums. In true open-source fashion, the bbPress project was born (at bbpress.org, of course) as a lightweight standalone alternative for forums.

The problem is that the project never really kept up the pace; and while the WordPress community wanted to use it, and bbPress saw some promising spurts of development, it never really caught up to the alternatives. Most of us who needed a forum went either with a plugin alternative that integrated perfectly or with forum software such as Vanilla.

The Facebook-inspired community plugin BuddyPress changed all that. BuddyPress, which adds groups and other membership functionality to a blog, started to ship with bbPress integrated in it. Perhaps unknowingly, some WordPress bloggers who had community features powered by BuddyPress were actually running a version of bbPress, which is enabled in the BuddyPress interface. It worked — and continues to work — great actually; because although bbPress, as standalone forum software, is way behind the competition in terms of features, sometimes all you need is a lightweight alternative, which was the idea behind bbPress all along.

bbPress 2.0 changed it all again, because bbPress has now been officially reborn as a plugin for WordPress, something that had been in the works for quite some time. This is where we stand today, with a fresh release of the first version of the bbPress plugin. In the coming weeks (or right now, depending on when you’re reading this), the plugin will get proper documentation and more support for cool functionality. That shouldn’t stop you from giving it a go right away, because getting started and taking advantage of its core functionality is easy enough.

Before we move on, we need to clear up some nomenclature:

• bbPress is a plugin for WordPress, and is sometimes referred to as bbPress 2.0 for clarity.
• bbPress 1.0 is a standalone forum that integrates with WordPress (and the BuddyPress plugin) but does not reside in WordPress’ core.
• BuddyPress is a separate plugin for WordPress that integrates with the bbPress plugin.
• BuddyPress still ships with bbPress, but you can connect to your bbPress plugin forums if you want to.

Yes, it’s all a bit messy.

Getting bbPress Up And Running

Installing bbPress is easy, because it’s available in WordPress’ plugin directory. Either install it from within WordPress, using the “Add new plugin� feature, or via FTP if you prefer to (or must) upload plugins. Then, activate the plugin, and you’re all set!

Well, not quite. You’ll want to look at some settings before starting to use the forums.

bbPress settings.

You’ll notice a new “Forums� menu under “Settings� in the admin area, along with the brand new sections “Forums,� “Topics� and “Replies,� all sporting bee-inspired icons.

Let’s look at the “Forums� settings pane first, shown above. Here you have an assortment of settings for your forums, such as whether to allow anonymous posts, how long posters should be able to edit their posts, and how many topics to show per page.

The “Archiveâ€� and “Single Slugsâ€� settings are important. These define the URLs of your forums, the posts, and the tags for posts. Choose something that fits your set-up; if you’re running an English-language website, then the default settings will probably do, but you can fine tune to your needs. Remember to go to Settings → Permalinks after making any change to the slugs, and rebuild the permalink structure by clicking the “Save Changesâ€� button on that page. If you ever have problems viewing the forums, give this a shot because it might be an issue with the permalinks, and rebuilding them might help. Also, make sure to press the “Saveâ€� button in Settings → Forums.

Where are your forums, then? Well, you’ll already know that from the Settings → Forums page, because they are located at the base slug assigned for the forums. By default, it would be forums, so you’d find them at yourdomain.com/forums/. Do yourself a favor and use pretty permalinks, because although bbPress will work without them, the URLs will look so much better if they’re pretty. That Google will thank you is just a bonus (note: an actual thank-you from Google is not guaranteed).

The forums page, without any forums unfortunately.

There we go: all set up and ready to go. Too bad there aren’t any forums, nor posts… yet!

Managing bbPress Forums

Getting bbPress set up and ready to go is a breeze, but if you actually want some action in your brand new forums, then you’ll need to create a forum. This is easily done under “Forums� in the admin area. Just click “New Forum,� and you’ll get a familiar-looking screen to create a forum.

Create a forum.

This is pretty self-explanatory. The one thing you’ll need to be careful with is the box in the top-right corner. These are the settings that enable you to control whether a forum is open or closed, whether it is a forum or a category, and who should see it. When you have created multiple forums, the “Parent� and “Order� options will show up, allowing you to nest forums (much like “Pages�) and sort them (also like Pages).

To make a long story short, with a few forums created, users will soon be able to post in your forums. Depending on your settings, they may need to sign up, but that’s a different matter and depends on what kind of website you’re running.

A forums page.

Managing “topics,� which are new posts, and “replies,� which are replies to topics, is easy enough. These show up under their respective sections in the WordPress admin area, and they behave much like posts and comments. That’s no surprise because bbPress has the same model as standard posts and Pages, using custom post types. This will also make it easy to style the bbPress forum should you want to, something we’ll look at more closely later.

Finally, one thing to know when running bbPress on a non-English website: localization projects are on GlotPress, and you can get a translation by using the options at the bottom of the entry for your selected language. You’ll need to upload these to the wp-content/plugins/bbpress/bbp-languages/ folder, and the file should be called bbpress-sv_SE.mo, where sv_SE should be swapped for your language of choice. Hopefully, we’ll be able to store these files in the wp-content/languages/ folder later, but this doesn’t work for me right now.

Extending bbPress

Although bbPress is now a WordPress plugin and not a standalone system, you’ll find plugins that extend its functionality. Quite a few actually: for displaying the latest posts in widgets, adding signatures and whatnot.

Your starting point for bbPress-related plugins is the plugins section of the bbPress website and, of course, the WordPress plugin directory (begin with a search).

One thing, though: make sure any plugin you choose is made for bbPress 2.0 (i.e. the plugin version). Older plugins made for the 1.x branch will not work.

BuddyPress And bbPress

BuddyPress, the plugin that enables you to create your own Facebook-like community on a WordPress website, work just great with bbPress. That should come as no surprise because the plugin still ships with the forum component (bbPress) built in. But this forum component is for enabling forums for your BuddyPress groups. Groups are exactly what they sound like: members can join them, even create their own (depending on your settings), and discuss various topics in them. With forums enabled for groups, every group will get a forum. This is still true with BuddyPress 1.5, despite there being a standalone bbPress plugin now. If you want forums for your BuddyPress-powered groups, then either choose an existing bbPress installation or install one in the BuddyPress settings. And yes, this is a bit confusing.

The settings page for BuddyPress forums.

With bbPress 2.0 and the shift from standalone forum software to WordPress plugin, you can rest assured that BuddyPress and bbPress still work well enough together. The option for installing forums site-wide is on the settings page for the BuddyPress forum, and it actually installs the bbPress plugin, rather than rely on the built-in forum component in the BuddyPress plugin. BuddyPress and the bbPress plugin integrate nicely out of the box, but not for group forums. Instead, your posts in the forums will show up in the BuddyPress activity stream; surely we’ll see some cool plugins in the future that leverage both BuddyPress and the bbPress plugin, tying the two even closer together.

All in all, there is no reason not to combine bbPress with BuddyPress if you need more community features than just a forum on your website.

Making bbPress Look Good

While your forums will work well out of the box, as you no doubt have gathered from the screenshots earlier in this article, you might want to make bbPress better suit the look of your website. You’ve already seen the default styles of bbPress, which you can tweak easily enough: just add CSS to your theme’s style sheet!

Doing this is easy: just inspect the code of the forums with your favorite Web inspector (such as Firebug or the built-in inspector in Chrome or Safari), and find the classes that you’ll need to style the forums.

The ul.bbp-forums class gives you control.

If you want more control, perhaps to break from the default layout of the forums, you can add additional template files to your WordPress theme. The bbPress plugin is already compatible with Twenty Ten, the previous default theme. In the bbpress folder, look at the files in bbp-themes/bbp-twentyten/ and you’ll get an idea what you can do. Simply changing the theme to Twenty Ten (instead of Twenty Eleven, which was shown earlier in this article) will give us something different and more attuned to our theme.

We get a different look when using Twenty Ten.

How you style the forums will depend on how much you want to deviate from the default look and feel. If everything is where it should be, then you’ll be able to make the forums looks good and fit your theme just by adding styles to your theme’s style sheet. But if you want to move things around a lot, then you’ll probably have to create your own template files. Consult the files in the bbpress/bbp-themes/bbp-twentyten/ folder to get an idea of what can be done, while we wait for bbPress to publish proper documentation. Because forums are really just a custom post type, you’ll likely be able to find your way around if you’ve worked with them before.

Three Websites That Use bbPress

Want to see some bbPress forums in action, other than bbPress.org itself or Twenty Ten and Twenty Eleven themes with the plugin activated?

WordPress.org

WordPress Forums

While using bbPress on WordPress.org might not exactly qualify as eating one’s own dog food, this is where it started after all.

Dropbox

Dropbox Forums

The syncing service Dropbox has been using bbPress forums for quite some time, with a pretty simple, standard look. This is just the standalone version, and it shows that bbPress is ready for prime time.

WPCandy

Discussions on WPCandy

The forums section of WPCandy is a great example of how bbPress can be easily integrated in an existing WordPress theme.

What’s Next?

Personally, I’m thrilled to see bbPress become a WordPress plugin. We’ve seen plugins that add forum features to WordPress in the past, but I haven’t been comfortable running any of them, to be honest. Whenever I’ve needed forums, I’ve used software such as the excellent Vanilla. Some people have suggested the BuddyPress plugin, but that’s a bit much if all you need is a simple forum for discussions.

With bbPress 2.0, this isn’t an issue anymore, and although documentation isn’t available yet, getting started is easy enough. You’ll probably want to add features to your forums, and that’s easy with additional plugins. And because bbPress is really just a custom post type for your WordPress website, using actual registered users, you can use existing plugins to achieve things such as moderator privileges and whatnot. We can anticipate a boom of bbPress-compatible plugins in the near future that will make our forums even better and more interesting.

For now, let’s play with what we have, which is usually more than enough.

(al)

---

© Thord Daniel Hedengren for Smashing Magazine, 2011.

There are many ways to skin a redesign (I think that’s how the saying goes). On a philosophical level, I agree with those who advocate for realigning, not redesigning, but these are mere words when you’re staring a design problem in the face with no idea where to start. This article came out of my own questions about how to make the realignment philosophy practical and apply it to my day-to-day work — especially when what’s needed is more than a few tweaks to the website here and there.

I propose an approach to redesign through realignment, by using a framework adapted from Edward Tufte’s principles on the visual display of quantitative information.

But first, a little context.

Redesign Through Realignment

Let’s recap the redesign versus realign argument. Here is Cameron Moll in “Good Designers Redesign, Great Designers Realign�:

The desire to redesign is aesthetic-driven, while the desire to realign is purpose-driven. One approach seeks merely to refresh, the other aims to fully reposition and may or may not include a full refresh.

A realignment can include a full refresh, but the starting point is not the visual layer. The starting point is an understanding of the website’s users and their objectives, of market trends and of brand strategy. These are the hard questions that guide a realignment — not a desire to try out some new fonts or see whether a +1 button would look good on the home page.

But surely a visual refresh can be beneficial, too? What’s the danger in giving users something new to look at? In an essay that builds on Cameron’s article, Francisco Inchauste sums up the problem as follows:

Great designers adjust an existing work with little disruption of the foundational design for a goal or purpose. The end result is a modification to the design that improves the user experience. Good designers, on the other hand, recreate existing work focusing on the aesthetic, with a misunderstood notion that it will always improve it. However, they end up disrupting and/or damaging the user’s experience, making no real impact with the effort.

The main problem with big redesigns, therefore, is that, even though objectively the UX might have been improved, users are often left confused about what has happened and are unable to find their way. In most cases, making “steady, relentless, incremental progress� on a website (to borrow a phrase of John Gruber) is much more desirable. With this approach, users are pulled gently into a better experience, as opposed to being thrown into the deep end and forced to sink or swim.

So, if we agree that a realignment is preferable to a redesign (and I’m sure we never will, but let’s assume we do for the sake of this article), a big question remains unanswered: What happens when a realignment requires major changes to the website? What happens when small tweaks aren’t enough, when a website’s UX is so far gone that you’re tempted to scrap everything and start over?

One way to go about it is to use a continual realignment process to redesign the website. Build a vision, and know where you’re going in the long term; but get there incrementally, not with a big-bang release. Remaining rooted in the realignment approach ensures that the focus remains purpose-driven, even if the process results in major visual changes. “That’s fine,� you say, “but how do you do it? Where do you begin with such a project?� Let’s now look at one possible approach to redesigning through realignment.

Edward Tufte And The Data-Ink Ratio

I’ve always been intrigued by Edward Tufte’s principles for visualizing large quantities of data. The Visual Display of Quantitative Information is one of my favorite books of all time. Recently, I’ve been wondering whether its principles could be applied to Web design and, specifically, the realignment process. The deeper I got into it, the more I realized that within Tufte’s principles lies a goldmine for people who make websites.

One of Tufte’s central principles in The Visual Display of Quantitative Information is what he calls data-ink:

Data-Ink is the non-erasable core of a graphic, the non-redundant ink arranged in response to variation in the numbers represented.

Before we unpack what this means for Web design, it’s important to note that Tufte’s work applies specifically to information graphics and the display of quantitative data, not to the design of graphical user interfaces. However, when carefully interpreted and applied to the field of Web design, the principles are extremely useful.

With that in mind, I propose the concept of data-pixels for the design of user interfaces to mirror Tufte’s data-ink for information graphics. In the context of Web design, we can then think of data-pixels as the simplest and most desirable path that a user can take through a flow (the “non-erasable core of an interfaceâ€�). It is what would remain in “focus modeâ€� — if nothing else existed on the screen, just the design elements that enable users to get from one screen to the next.

For example, on a payment screen, data-pixels would be the credit-card fields, text labels and “Pay nowâ€� button. Nothing else. This is obviously not possible — you need headers, payment summaries, tooltips, trust seals, etc. But the “core dataâ€� are the elements of the page that we cannot remove without the user getting stuck with no ability to recover.

From this irreducible point of the design, you can start to add other elements as necessary, and this is what Tufte’s work is all about. Much of what Tufte espouses is finding the right data-ink ratio (or what we’ll call data-pixel ratio) for quantitative data, so that the core data can shine through. He lays out five principles for data-ink. Here is an overview and how these principles can be adapted to redesign through realignment.

Principle 1: Above All Else, Show the Data

Data graphics should draw the viewer’s attention to the sense and substance of the data, not to something else.

– Edward Tufte, The Visual Display of Quantitative Information

During a realignment, we should be guided by the principle that every page should be focused on the core data and the primary task that users need to take in that particular flow. Anything else is noise and should be added only after very careful consideration.

Craig Mod has written a great article titled “The Shape of Our Future Book.� In it, he describes the “quiet confidence� that a Kindle has when it is woken from its sleep state (compared to the iPad in particular), and then he addresses the data-pixel issue as follows:

I think the same concept of “quiet confidence� can be applied to data. Namely, in designing user experiences, we need to produce data that doesn’t draw attention to itself explicitly as data.

This doesn’t mean that design has to be boring or that aesthetics are not important. It means that we need to be mindful that any layer of design we add to the core data has to serve a specific function and cannot distract from the data itself.

Image credit: Sarah Blake

Principle 2: Erase Non-Data Pixels, Within Reason

While it is true that these boring [pixels] sometimes help set the stage for the data action, it is surprising […] how often the [pixels] themselves can serve as their own stage.

– Edward Tufte, The Visual Display of Quantitative Information

Once the context for the realignment is set by the overarching principle of focusing on core data, it’s time to evaluate the design and start improving the data-pixel ratio.

The first step is to look for ways to erase non-data pixels — the parts of the design that don’t directly apply to the user’s primary task. Look for elements that cannot be connected to guiding a user to the desired outcome, such as:

• Colors that don’t support the visual hierarchy through contrast;
• Typefaces that draw attention to themselves for no good reason;
• Gratuitous imagery (stock photography in particular) that does nothing more than break up the page (consider using white space and proximity grouping to create natural breaks instead);
• Alignment, sizes and color contrast that draw unwanted attention (for example, indented paragraphs, large social-media icons, a bright ad that doesn’t fit the website’s visual style).

One website that could certainly benefit from the removal of non-data pixels is doHome. Notice the gratuitous reflection of the navigation menu, the unnecessary stock photography, and the fact that so little content on the page tells you what the website is about:

MailChimp does a good job of limiting the non-data pixels on its sign-up page. Clear language indicates what to do; large fields with top-aligned labels ensure that you complete the form quickly; some information is included about the plan you’re signing up for; and the button to sign up is big and high-contrast.

Squarespace’s sign-up page removes as much non-data pixels as possible to put all the focus on the task at hand. The background disappears as soon as you click the “Sign up� link, and you’re presented with the following zero-distraction form:

Google+ is another example of a design that employs minimal non-data pixels. As Oliver Reichenstein says:

It is extremely difficult to keep a complicated user interface so light, white and free of lines, boxes and ornaments. The content hierarchy is always clear, color definitions and consistent and clear without labeling them.

This list isn’t exhaustive, but it illustrates the purpose of the principle: to critically evaluate the visual elements in order to strip out what isn’t necessary.

Principle 3: Erase Redundant Data-Pixels, Within Reason

Gratuitous decoration and reinforcement of the data measures generate much redundant data-[pixels].

– Edward Tufte, The Visual Display of Quantitative Information

“Redundant data-pixels� refer to elements of the design that are repeated without good reason. Some examples:

• Rows of products in a table with an “Add to cartâ€� button next to each one.
  (Consider using check boxes, with one “Add to cart� button at the bottom.)
• Animation as a way to draw attention to an element.
  (Consider using high-contrast color and size instead.)
• “Are you sure you want to do this?â€� dialogs for simple tasks such as adding a product to a cart.
  (For potentially catastrophic actions, like deleting an account, this type of dialog is, of course, appropriate.)

Below is an example from the offline word, courtesy of Allan Kent. The chart shows the price of parking per hour. Surely a simple sign that reads “R10 per hour� would suffice?

Note the addition of “within reason� to each of the data-pixel principles. Tufte himself acknowledges that redundant pixels are sometimes necessary:

Redundancy, upon occasion, has its uses: giving a context and order to complexity, facilitating comparisons over various parts of the data, perhaps creating an aesthetic balance.

The exception, then, is when, in Tufte’s words again, “redundancy has a distinctly worthy purpose�. A “Pay now� button at the top and bottom of a check-out page could be an example of this. One of the buttons is redundant, yet it introduces efficiency so that users don’t have to scroll up or down to place their order.

The guiding principle here is to strive for a minimalist aesthetic, adding redundant pixels only when they serve a larger purpose (for example, when they’re essential to the brand’s promise or to user efficiency).

Principle 4: Maximize the Data-Pixel Ratio, Within Reason

Every [pixel] on a graphic requires a reason. And nearly always that reason should be that the [pixel] presents new information.

– Edward Tufte, The Visual Display of Quantitative Information

Once you’ve erased as many non-data pixels and redundant data pixels as possible, the next step is to figure out what (if anything) is missing from the design. The goal of this principle is to add more pixels to the design, if necessary.

There should always be a reason for adding elements to a design, and the reason will usually be that those elements provide information and/or functionality that increases usability. Some examples:

• Breadcrumbs that tell users where they are on the website and give them an easy way to get back to where they came from.
• An aesthetic layer of color, typography, layout and so on, to ensure consistency between brand perception and the website.
• Hover states or tooltips to provide appropriate guidance or contextual help to users.

An example of necessary pixels is relevant inline error messages. Consider the sign-up form on Quora below. Very few non-data pixels are on the page. The form’s layout is simple, with no extraneous decoration. But if you try to enter only your first name, the page instantly reminds you that a full name is required:

One could argue that this isn’t technically part of the “core data� of the design. Quora could have let this slide and either allowed accounts with first names only or sorted it out after users have signed up. But it has decided that data integrity is important from the start, so it has added this real-time check.

A little closer to home, this is what the header of kalahari.com (where I currently work) looked like when I started at the company:

Identifying the non-data pixels in this design is easy: very large radii on the rounded corners, color that grabs too much attention, too many unimportant links, etc. After maximizing the data-pixel ratio to put the focus on the core data (which is search), we ended up with this header:

Principle 5: Revise and Edit

Probably, indeed, the larger part of the labour of an author in composing his work is critical labour; the labour of sifting, combining, constructing, expunging, correcting, testing: this frightful toil is as much critical as creative.

– T.S. Eliot, Selected Essays, 1917–1932

Tufte quotes T.S. Eliot to describe the relentless effort of editing and revising in graphic design work, and it’s certainly true for Web design.

Once you’ve completed a cycle through these principles, it’s time to go back and start again. Every realignment cycle exposes new opportunities to “above all else, show the data�. UX is a never-ending cycle of improvement, and following a realignment process bakes this constant cycle into the strategy in a very natural way.

Summary

As I mentioned at the beginning of this article, there are many approaches to redesigning a website. The hardest part often is knowing how and where to start. As I’ve shown, Edward Tufte’s timeless principles for the visual display of data can be adjusted and used as a framework to get over that initial hump and serve as a catalyst for a cycle of continual improvement through realignment.

What other models or approaches are there to frame a realignment project? How do you get started?

Front page image source: Elliot Jay Stocks

(al)

---

© Rian van der Merwe for Smashing Magazine, 2011.

The profession that dares not speak its name needs you. Digital design is the wonder of the world. But the world hasn't bothered to stop and wonder about web workers—the designers, developers, project managers, information architects slash UX folk, content strategists, writers, editors, marketers, educators, and other professionals who make the web what it is. That’s where you come in. Take the survey!

As Web developers, we have an affinity for developing with JavaScript. Whatever the language used in the back end, JavaScript and the browser are the primary language-platform combination available at the user’s end. It has many uses, ranging from silly to experience-enhancing.

(Image: Viktor Hertz)

In this article, we’ll look at some methods of manipulating JavaScript to determine various network characteristics from within the browser — characteristics that were previously available only to applications that directly interface with the operating system. Much of this was discovered while building the Boomerang project to measure real user performance.

What’s In A Network Anyway?

The network has many layers, but the Web developers among us care most about HTTP, which runs over TCP and IP (otherwise known jointly as the Internet protocol suite). Several layers are below that, but for the most part, whether it runs on copper, fiber or homing pigeons does not affect the layers or the characteristics that we care about.

Network Latency

Network latency is typically the time it takes to send a signal across the network and get a response. It’s also often called roundtrip time or ping time because it’s the time reported by the ping command. While this is interesting to network engineers who are diagnosing network problems, Web developers care more about the time it takes to make an HTTP request and get a response. Therefore, we’ll define HTTP latency as the time it takes to make the smallest HTTP request possible, and to get a response with insignificant server-processing time (i.e. the only thing the server does is send a response).

Cool tip: Light and electricity travel through fiber and copper at 66% the speed of light in a vacuum, or 20 × 10⁸ metres per second. A good approximation of network latency between points A and B is four times the time it takes light or electricity to travel the distance. Greg’s Cable Map is a good resource to find out the length and bandwidth of undersea network cables. I’ll leave it to you to put these pieces together.

Network Throughput

Network throughput tells us how well a network is being utilized. We may have a 3-megabit network connection but are effectively using only 2 megabits because the network has a lot of idle time.

DNS

DNS is a little different from everything else we care about. It works over UDP and typically happens at a layer that is transparent to JavaScript. We’ll see how best to ascertain the time it takes to do a DNS lookup.

There is, of course, much more to the network, but determining these characteristics through JavaScript in the browser gets increasingly harder.

Measuring Network Latency With JavaScript

My first instinct was that measuring latency simply entailed sending one packet each way and timing it. It’s fairly easy to do this in JavaScript:

var ts, rtt, img = new Image;
img.onload=function() { rtt=(+new Date - ts) };
ts = +new Date;
img.src="/1x1.gif";

We start a timer, then load a 1 × 1 pixel GIF and measure when its onload event fires. The GIF itself is 35 bytes in size and so fits in a single TCP packet even with HTTP headers added in.

This kinda sorta works, but has inconsistent results. In particular, the first time you load an image, it will take a little longer than subsequent loads — even if we make sure the image isn’t cached. Looking at the TCP packets that go across the network explains what’s happening, as we’ll see in the following section.

TCP Handshake and HTTP Keep-Alive

When loading a Web page or image or any other Web resource, a browser opens a TCP connection to the specified Web server, and then makes an HTTP GET request over this connection. The details of the TCP connection and HTTP request are hidden from users and from Web developers as well. They are important, though, if we need to analyze the network’s characteristics.

The first time a TCP connection is opened between two hosts (the browser and the server, in our case), they need to “handshake.â€� This takes place by sending three packets between the two hosts. The host that initiates the connection (the browser in our case) first sends a SYN packet, which kind of means, “Let’s SYNc up. I’d like to talk to you. Are you ready to talk to me?â€� If the other host (the server in our case) is ready, it responds with an ACK, which means, “I ACKnowledge your SYN.â€� And it also sends a SYN of its own, which means, “I’d like to SYNc up, too. Are you ready?” The Web browser then completes the handshake with its own ACK, and the connection is established. The connection could fail, but the process behind a connection failure is beyond the scope of this article.

Once the connection is established, it remains open until both ends decide to close it, by going through a similar handshake.

When we throw HTTP over TCP, we now have an HTTP client (typically a browser) that initiates the TCP connection and sends the first data packet (a GET request, for example). If we’re using HTTP/1.1 (which almost everyone does today), then the default will be to use HTTP keep-alive (Connection: keep-alive). This means that several HTTP requests may take place over the same TCP connection. This is good, because it means that we reduce the overhead of the handshake (three extra packets).

Now, unless we have HTTP pipelining turned on (and most browsers and servers turn it off), these requests will happen serially.

We can now modify our code a bit to take the time of the TCP handshake into account, and measure latency accordingly.

var t=[], n=2, tcp, rtt;
var ld = function() {
   t.push(+new Date);
   if(t.length > n)
     done();
   else {
     var img = new Image;
     img.onload = ld;
     img.src="/1x1.gif?" + Math.random()
                         + '=' + new Date;
   }
};
var done = function() {
  rtt=t[2]-t[1];
  tcp=t[1]-t[0]-rtt;
};
ld();

With this code, we can measure both latency and the TCP handshake time. There is a chance that a TCP connection was already active and that the first request went through on that connection. In this case, the two times will be very close to each other. In all other cases, rtt, which requires two packets, should be approximately 66% of tcp, which requires three packets. Note that I say “approximately,â€� because network jitter and different routes at the IP layer can make two packets in the same TCP connection take different
lengths of time to get through.

You’ll notice here that we’ve ignored the fact that the first image might have also required a DNS lookup. We’ll look at that in part 2.

Measuring Network Throughput With JavaScript

Again, our first instinct with this test was just to download a large image and measure how long it takes. Then size/time should tell us the throughput.

For the purpose of this code, let’s assume we have a global object called image, with details of the image’s URL and size in bits.

// Assume global object
// image={ url: …, size: … }
var ts, rtt, bw, img = new Image;
img.onload=function() {
   rtt=(+new Date - ts);
   bw = image.size*1000/rtt;    // rtt is in ms
};
ts = +new Date;
img.src=image.url;

Once this code has completed executing, we should have the network throughput in kilobits per second stored in bw.

Unfortunately, it isn’t that simple, because of something called TCP slow-start.

Slow-Start

In order to avoid network congestion, both ends of a TCP connection will start sending data slowly and wait for an acknowledgement (an ACK packet). Remember than an ACK packet means, “I ACKnowledge what you just sent me.� Every time it receives an ACK without timing out, it assumes that the other end can operate faster and will send out more packets before waiting for the next ACK. If an ACK doesn’t come through in the expected timeframe, it assumes that the other end cannot operate fast enough and so backs off.

This means that our throughput test above would have been fine as long as our image is small enough to fit within the current TCP window, which at the start is set to 2. While this is fine for slow networks, a fast network really wouldn’t be taxed by so small an image.

Instead, we’ll try by sending across images of increasing size and measuring the time each takes to download.

For the purpose of the code, the global image object is now an array with the following structure:

var image = [
	{url: …, size: … }
];

An array makes it easy to iterate over the list of images, and we can easily add large images to the end of the array to test faster network connections.

var i=0;
var ld = function() {
   if(i > 0)
      image[i-1].end = +new Date;
   if(i >= image.length)
      done();
   else {
      var img = new Image;
      img.onload = ld;
      image[i].start = +new Date;
      img.src=image[i].url;
   }
   i++;
};

Unfortunately, this breaks down when a very slow connection hits one of the bigger images; so, instead, we add a timeout value for each image, designed so that we hit upon common network connection speeds quickly. Details of the image sizes and timeout values are listed in this spreadsheet.

Our code now looks like this:

var i=0;
var ld = function() {
   if(i > 0) {
      image[i-1].end = +new Date;
      clearTimeout(image[i-1].timer);
   }
   if(i >= image.length ||
         (i > 0 && image[i-1].expired))
      done();
   else {
      var img = new Image;
      img.onload = ld;
      image[i].start = +new Date;
      image[i].timer =
            setTimeout(function() {
                       image[i].expired=true
                    },
                    image[i].timeout);
      img.src=image[i].url;
   }
   i++;
};

This looks much better — and works much better, too. But we’d see much variance between multiple runs. The only way to reduce the error in measurement is to run the test multiple times and take a summary value, such as the median. It’s a tradeoff between how accurate you need to be and how long you want the user to wait before the test completes. Getting network throughput to an order of magnitude is often as close as you need to be. Knowing whether the user’s connection is around 64 Kbps or 2 Mbps is useful, but determining whether it’s exactly 2048 or 2500 Kbps is much less useful.

Summary And References

That’s it for part 1 of this series. We’ve looked at how the packets that make up a Web request get through between browser and server, how this changes over time, and how we can use JavaScript and a little knowledge of statistics to make educated guesses at the characteristics of the network that we’re working with.

In the next part, we’ll look at DNS and the difference between IPv6 and IPv4 and the WebTiming API. We’d love to know what you think of this article and what you’d like to see in part 2, so let us know in a comment.

Until then, here’s a list of links to resources that were helpful in compiling this document.

• Greg’s Cable Map, Greg Mahlknecht
  Aggregated information about undersea network cables.
• “Transmission Control Protocol,â€� Wikipedia
  This page links to all of the RFCs.
• TCP/IP Illustrated, Vol. 1: The Protocols, by Richard Stevens
  A link to the book on Amazon.
• “Slow-Start,â€� Wikipedia
  Also, see the links above for congestion control and Nagle’s Algorithm.
• “Bandwidth Images Sizes,â€� Google Spreadsheet
  This is based on the research done while building Boomerang.
• Boomerang
  The Boomerang project on GitHub, where much of this has been implemented.

(al)

---

© Philip Tellis for Smashing Magazine, 2011.

Disclaimer: This post by Jeremy Keith is one of the many reactions to our recent article on the pursuit of semantic value by Divya Manian. Both articles are published in the Opinion column section in which we provide active members of the community with the opportunity to share their thoughts and ideas publicly.

Divya Manian, one of the super-smart web warriors behind HTML5 Boilerplate, has published an article called Our Pointless Pursuit Of Semantic Value.

I’m afraid I have to agree with Patrick’s comment when he says that the abrasive title, the confrontational tone and strawman arguments at the start of the article make it hard to get to the real message.

But if you can get past the blustery tone and get to the kernel of the article, it’s a fairly straightforward message: don’t get too hung up on semantics to the detriment of other important facets of web development. Divya clarifies this in a comment:

Amen, this is the message the article gets to. Not semantics are useless but its not worth worrying over minute detail on.

The specific example of divs and sectioning content is troublesome though. There is a difference between a div and a section or article (or aside or nav). I don’t just mean the semantic difference (a div conveys no meaning about the contained content whereas a section element is specifically for enclosing thematically-related content). There are also practical differences.

A section element will have an effect on the generated outline for a document (a div will not). The new outline algorithm in HTML5 will make life a lot easier for future assistive technology and searchbots (as other people mentioned in the comments) but it already has practical effects today in some browsers in their default styling.

Download the HTML document I’ve thrown up at https://gist.github.com/1360458 and open it in the latest version of Safari, Chrome or Firefox. You’ll notice that the same element (h1) will have different styling depending on whether it is within a div or within a section element (thanks to -moz-any and -webkit-any CSS declarations in the browser’s default stylesheets).

So that’s one illustration of the practical difference between div and section.

Now with that said, I somewhat concur with the conclusion of “when in doubt, just use a div”. I see far too many documents where every div has been swapped out for a section or an article or a nav or an aside. But my reason for coming to that conclusion is the polar opposite of Divya’s reasoning. Whereas Divya is saying there is effectively no difference between using a div and using sectioning content, the opposite is the case: it makes a big difference to the document’s outline. So if you use a section or article or aside or nav without realising the consequences, the results could be much worse than if you had simply used a div.

I also agree that there’s a balance to be struck in the native semantics of HTML. In many ways its power comes from the fact that it is a limited—but universally understood by browsers—set of semantics. If we had an element for every possible type of content, the language would be useless. Personally, I’m not convinced that we need a section element and an article element: the semantics of those two elements are so close as to be practically identical.

And that’s the reason why right now is exactly the time for web developers to be thinking about semantics. The specification is still being put together and our collective voice matters. If we want to have well-considered semantic elements in the language, we need to take the time to consider the effects of every new element that could potentially be used to structure our content.

So I will continue to stop and think when it comes to choosing elements and class names just as much as I would sweat the details of visual design or the punctation in my copy or the coding style of my JavaScript.

(vf)

---

© Jeremy Keith for Smashing Magazine, 2011.