In a perfect world, cross-browser testing would be straightforward. We would download a legacy version of a browser, run it, and be able to instantly test our pages and scripts without a single care in the world. The reality of cross-browser testing, though, is very different. Issues such as runtime conflicts when running multiple versions of the same browser and inaccurate third-party testing tools mean we can spend hours just evaluating whether a testing set-up is anywhere near reliable.
I’m a user-interface developer at AOL (yes, we’re not dead yet!), and in this multi-part post I’ll take you through the exact set-up we use to accurately test content that will be potentially viewed by up to millions of users with a very diverse set of browsers. This set-up is similar to the one used by some of my colleagues at Opera, Mozilla and Google, so, fingers crossed, we’re doing this optimally.
A quick note before we begin. Setting up accurate testing for Internet Explorer as outlined in this post requires a bit of effort. So, please check your website analytics first to ensure that a sufficient number of IE users visit your website in the first place to warrant this effort.
Internet Explorer 6 To 10
Let’s begin with our old friend, Internet Explorer (IE). As most of us know, running multiple versions of the original Internet Explorer executables on the same system is very difficult due to issues ranging from runtime version conflicts to operating-system incompatibility. In truth, I don’t think Microsoft ever considered a scenario in which developers needed a way to achieve this back when it was first conceiving IE 6, 7 and 8.
This has left developers in a chasm of uncertainty, forced through trial and error to discover a way to accurately test what is (for better or worse) the world’s most widely used set of browsers.
In this section, I’ll take you through some IE testing options that you may be using or have heard of before. I’ll explain why they might not be reliable; and then I’ll present the solutions we currently use at AOL for production websites.
In case you’re interested, our team generally uses IE 7 as a baseline, although we do also test stable, beta and preview or dev-channel versions of Chrome, Firefox, Opera, Safari and, of course, IE 8 to 10.
Our reason for using IE 7 as a baseline comes down to our global website analytics: IE 6, 7 and 8 are the most common browsers used to access our websites. However, we stopped supporting IE 6 as of a few months ago. The reality is that IE 6 has major compatibility issues with modern technologies, and our team has found that we are able to deliver projects up to 20% more quickly when we don’t have to worry about catering a basic experience to it.
Most Third-Party IE Testing Tools Are Unreliable
A Google search for third-party IE testing tools will result in at least ten different options, nine of which are likely to be unreliable. Let’s go through a sample of them, in case you’re using one of them to test against staging or production websites at your work.
The following reviews may sound a little harsh, but the purpose is to stress the importance of using approaches that have been well engineered, tried and tested over time.
IETester
Unfortunately, IETester is probably the most popular third-party tool that I see designers and developers use to test multiple versions of IE. When I first tried it, I too was drawn by the promise of a single application that would solve all of my IE testing woes. However, all that glitters is not gold.
The tool has a number of inconsistencies when testing IE 6, 7 and 8, with none providing a 1:1 experience of the original browsers. Our and many other teams have confirmed not only that the rendered output of IETester varies significantly from any real version of IE, but that pop-up interactions cause failures, Flash and CSS filters don’t work in user mode, conditional CSS comments often fail, and switching between versions makes it very prone to crashing.
In short, it’s unreliable, and this should be enough for any developer to consider alternatives.
Multiple IE
TredoSoft’s Multiple IE is another tool that often shows up when searching for an IE testing solution. Unfortunately, it too suffers from a number of issues, including inaccurate rendering. One common complaint is that people experience IE 5.5 rendering bugs even when they’re just testing IE 6; this certainly is not something we want to deal with at a time when many of us are trying to ditch IE 6 completely.
Multiple IE isn’t updated regularly either, and if the long thread of user issues experienced with it since its release doesn’t put you off, then consider that the tool’s IE textboxes actually break in a number of circumstances.
To the best of my knowledge, both IETester and Multiple IE rely on an exploit known as DLL redirection in order to bypass issues with DLL naming conflicts, allowing the tools to attempt to run standalone copies of IE. I would recommend avoiding such tools, because implementing a completely sandboxed environment for IE that is as accurate as running the originals independently is very difficult.
Expression Web SuperPreview
Next up is Microsoft’s Expression Web SuperPreview, which claims to simplify the process of testing and debugging layouts across multiple versions of Web browsers.
Unfortunately, you won’t be able to test complete user interaction, JavaScript, DOM manipulation, animation or much else with this tool. We live in an age when the Web can be very dynamic and, in some cases, highly interactive. A tool like this might work for baseline testing, but not for accurate cross-browser testing.
BrowserLabs, Browsershots, BrowserCam
I personally use Adobe’s BrowserLabs — or sometimes Browsershots, if a static layout test is absolutely necessary. But again, neither of these options allow you to test interactions with your pages. The same goes for BrowserCam. These services simply weren’t designed for this purpose, but we still regularly see designers and developers using them as if they were.
I’m not in any way saying to flat out avoid these services, but rather that they’re inadequate for complete cross-browser testing. Designers and developers need to know exactly what a user sees when they interact with their website. No visitor will be using a static page renderer, so why rely on one yourself?
Windows Virtual PC
One other solution you’ll probably come across is Microsoft’s Virtual PC, with time-bombed images for IE6 to 8.
For a brief time this was considered to be the answer to cries from developers for a better solution. Unfortunately, it is by far the most inadequate (and demanding) solution to testing that I’ve seen proposed in the past few years. At least 12 GB of disk space is required to install all of the images, and the images regularly expire during the year.
A cross-browser testing environment for legacy browsers should have reasonable system requirements and, for the most part, not require regular manual updating in order to continue using it. Because the Virtual PC option fails these criteria, I can’t recommend using it either.
Browser And Document Modes
We’ve looked at third-party tools, but what about Microsoft’s current solutions to these problems?
Both IE 9 and IE 10 PP2 support switchable document and browser modes via the F12 Developer Tools for cross-version compatibility testing. To be specific, browser-version testing here is made possible using a kind of emulation.
“Document mode� determines what features a page has access to and what can be adjusted based on the page’s doc type, X-UA-compatible meta tag and headers. For example, the standards document mode allows the page to take advantage of IE’s implementation of ECMAScript 5 (ES5), while the IE 7 and 8 standards modes offer an alternative experience.
“Browser mode,� on the other hand, emulates different IE browser version behaviors and can be changed directly from the IE Developer Tools. Emulation is achieved in a few different ways, but it includes altering both the document mode and the user-agent string. In case you’re wondering, the UA string is adjusted here to ensure that code that relies on UA sniffing functions as though the correct version of the browser were being used.
It is worth noting that IE 9 shipped with a newer JavaScript engine, called Chakra. While the browser itself supports a number of compatibility modes, because the JavaScript engine itself differs significantly from what shipped with IE 6, 7 and 8, there are acknowledged differences between the experience in IE 9 and testing in a standalone browser.
That aside, there are, unfortunately, a number of quirks in the way that both document mode and browser mode function. Our team has run into scenarios where IE returns the wrong user-agent string to the server; and, as with the third-party tools, there have been several instances of inaccuracies with the expected rendering when tested against the original browsers.
Finally (and rather strangely), these modes have a number of issues of their own that are not present in IE 6, 7 or 8, making it even more difficult to establish whether the issues experienced are specific to a browser version or just the mode being used.
For these reasons, I would urge developers not to rely strictly on document or browser mode for their cross-version testing needs.
Accurate IE Testing Solutions
Option 1: Virtual Machines + IECollection
We’ve reviewed a number of solutions that don’t provide an accurate means of testing multiple versions of IE. So, what does work? The answer lies in using dedicated virtual machines (VMs) and a tool called IE Collection.
Out of the box, Utilu’s IE Collection offers the following:
- A tested collection of standalone versions of IE, including versions 1 through 9;
- Confirmed support for accurate rendering when compared against the original IE executables;
- Confirmed support (to date) for the correct IE JavaScript engine implementations that shipped with the originals;
- Confirmed support for the correct UA strings being detectable (not that you should be UA-string testing in the first place — however, if needed, it’s there);
- Support for both the 32-bit and 64-bit versions of Windows XP, 2003, Vista and 7;
- Access to the IE Developer Toolbar, which comes with the standard IE Collection set-up; this is compatible with IE 5 and above, but you also have the option to install Firebug Lite if you prefer that.
Although the majority of IE versions are supported and function reliably within the collection, there are known issues with versions of IE 7 when used under Windows Vista or Windows 7 (as noted on Utilu’s website). I’ll discuss how we handle this limitation shortly, but let’s first briefly go over virtual machines.
A virtual machine (VM) is a sandboxed “guest� operating system that run within a normal “host� operating system. It effectively enables you to install and run a dedicated copy of almost any other operating system on your current one and to share a configurable set of memory and disk resources between the two. For example, a Windows 7 VM could easily be run on Mac OS X, as could an Ubuntu VM be run on Windows, and so on.
You’ll need two things to get started using VMs: a virtualization tool to create VMs on your OS, and a CD or image of the operating system that you wish to run as a virtual machine. Your company should be able to furnish you with Windows licenses relatively easily (… famous last words). If you’re a college student, you could probably get licenses for Windows XP SP3, Vista and 7 for free through the MSDN Academic Alliance program.
You could set up a number of different configurations to test Windows and IE, but if you’re using IE Collection, I would generally recommend the following:
- Windows 7 for IE 6, 8, 9, 10;
- Windows XP (SP3) for IE 7.
You shouldn’t need to support anything below IE 6 nowadays, but IE 4 to 5.5 also work fine on XP SP3.
At AOL, we use a combination of VirtualBox, Parallels and VMware for our virtualization needs. I prefer Sun’s VirtualBox because it has the following advantages:
- It’s free, providing a lower barrier to entry for developers who cannot afford commercial licenses for Parallels or VMware;
- In my experience, it tends to perform more optimally than Parallels or VMware under the same configuration settings;
- It has a marginally better UI than the alternatives.
That said, VMware is considered better if you’re working with a standalone server. At the end of the day, all three options are equally valid, and both Parallels and VMware come with trial versions that you can easily evaluate. Choose whichever option best suits your needs.
Pro tip: Remember to configure your VM to use the minimim amount of memory needed to test, unless you are using a dedicated testing box. By default, not all virtualization tools are configured to do this. This will simply ensure that the rest of your OS runs more smoothly.
Here are the three steps to setting this up:
- If you’re a Mac user, I recommend using a 32-bit Windows 7 Pro or Ultimate image (or installation CD), with a minimum of 17 GB of disk space allocated for the installation. 512 MB of virtualized RAM (which can be adjusted in the settings panel) should work fine for testing on a non-dedicated testing machine, but use more if feasible. You can create this as a new VM using VirtualBox, which I have configured to run at a resolution of between 800 × 600 and 1024 × 768 pixels. Windows users have two options. If you’re already running a version of Windows 7, skip to step 2, where we’ll begin installing IE Collection. If you’re running a different version of Windows and would like to run a virtualized copy of Windows 7 on top of it, then follow the steps above for Mac users. Mac and Windows users should boot up their VM or desktop Windows installation before proceeding to step 2.
- Next, download IE Collection from Utilu’s website, and then install Windows 7 on either your VM or your primary system, depending on your set-up. This will give you access to IE 6, 8 and 9. I usually install IE 10 for testing purposes separately, but bear in mind that it is only in the platform preview phase at the time of writing. Ideally, install only the bare essentials to test on any VM in order to avoid excessive start-up times.
- Mac and Windows users: to address the issue of IE 7 not working on Windows 7, you’ll have to create another virtual machine with Windows XP installed (ideally with SP3). Because no reliability issues have been reported for running either IE Collection’s IE 7 or the original IE 7 on this OS, your bases should be covered here. As stated in my previous instructions, obtain a Windows XP image or installation CD, and create a new VM using VirtualBox. Boot it up, install IE7 through the link above, and you should be good to go.
That’s it really! Some developers prefer to run IE 6 and 7 on XP (step 3) rather than on Windows 7 or Vista, because XP accounts for the majority of users who have yet to upgrade to IE 8 or 9. This, too, is completely valid; the set-up is flexible enough to support a little personal preference.
Next, for the sake of being thorough, let’s look at some alternatives to the set-up I’ve recommended above.
Option 2: Dedicated VMs for Each Version of IE
Pragmatic developers might wish to maintain a dedicated VM for each version of IE being tested. On these VMs, rather than using any third-party tools, they would install the original, official IEs instead.
The reasoning behind this is that it potentially minimizes the risk of individual browser configurations or overrides from interfering with one another, and it ensures that you get only the exact version of IE that you wish to test (and, in turn, the corresponding JavaScript engine in that version).
If you’re setting up dedicated VMs for individual releases of IE, I recommend the following set-up:
- Windows XP (SP3) for IE 6,
- Windows Vista (SP2) for IE 7,
- Windows Vista (SP2) for IE 8,
- Windows 7 for IE 9 and 10.
Knowing now about dedicated VMs, you might be wondering why we don’t use this set-up at AOL. I can only speak for my team, but we moved away from this approach because running three to four VMs on the same system and still using it efficiently for other tasks can become unmanageable.
With a minimum configuration of 512 MB of RAM per VM, we would be looking at around 2 GB of RAM (if not more) just for testing purposes. This set-up might work if you currently have 8 GB or more of system memory. But if not, there’s still hope.
We could always get a dedicated box just for testing purposes, and run all of our virtual machines directly on there instead. I recently acquired a second box at work for this very purpose. But at the end of the day, both options 1 and 2 are set-ups you can rely on.
Option 3: VNC or RDP + Dedicated Testing Boxes
A few companies (such as ZURB) prefer to use a form of remote desktop access to allow designers and developers to log into a dedicated testing box or set of boxes for cross-browser testing, over a network. The benefit of this is reduced demand on memory for each client. And this can be implemented in a few ways; VNC and RDP are the first that come to mind.
Virtual network computing (VNC) is a platform-independent desktop sharing system that allows you to remotely control another computer. It transmits both keyboard and mouse events from one system to another, relaying screen updates back in the other direction over a network. Similarly, RDP allows users to access applications over a network, but it is a proprietary protocol developed by Microsoft. If you happen to be running Mac OS X, VNC is already built in, but a number of VNC clients are also available in the wild, such as RealVNC. On the RDP side of things, Mac users can use Apple’s own Remote Desktop and Remote Desktop Connection by Microsoft.
You might be wondering what the differences are between VNC and RDP. RDP is semantic and is more aware of controls, fonts and other graphical primitives, while VNC is a little “dumber� in this regard. VNC, though, permits a session to be shared on the target machine, which might be useful for multi-user demos; whereas RDP does not, as far as I’m aware.
With the decreased memory requirements, you might then wonder why we don’t opt to use remote desktops for testing at AOL. The primary reason is that, in some cases, testing over a network can result in packets (effectively, frames) being dropped, which can affect our ability to test highly interactive applications or websites. Also, websites that make use of rich-media experiences based on canvas, SVG or otherwise aren’t reliably tested over networks for this reason. But don’t let our experience put you off. I know of teams in other companies that haven’t run into these issues, and remote testing might be suitable for some networks more than others.
Yet another alternative: If your team is unable to purchase or rent dedicated boxes for remote testing, there are third-party services that readily host all versions of Windows, along with historic versions of IE and other browsers, for a subscription fee. CrossBrowserTesting.com is one such option that might be worth exploring.
Conclusion
I hope this guide has come in handy. Please feel free to share your own stories and experiences of cross-version testing IE, because they may help other readers decide which option to try out first.
In part 2 of this post, we’ll look at cross-browser testing modern browsers. See you then!
(al)
© Addy Osmani for Smashing Magazine, 2011.
