You’re probably familiar with the whole European Commission/Microsoft browser case. Long story, but basically the EC accused Microsoft of harming consumer choice by bundling Internet Explorer with Windows, Microsoft offered to present users with a browser ballot, I wrote about how a ballot wasn’t a great solution but should be designed well, yadda yadda. Last month, Microsoft agreed to show the browser ballot choices in random order, rather than fixed. Definitely an improvement!
I still wondered what effect, if any, item order and other variables would have on what browser people would choose. Would certain design changes on the ballot have a big impact on users? And how could Firefox optimize its space on the ballot to be most effective?
Mechanical Turk: The Odds are Good, the Goods are Odd
My colleague Alex Faaborg suggested that I try using Amazon’s Mechanical Turk to get some answers to these questions. Mechanical Turk lets users create tasks that they need humans (called workers) rather than computers to perform. Workers select tasks that look interesting to them and are paid a small amount for each task completed. I was pretty skeptical about using any pay-for-answers service at first. My fear was that workers would be a strange grabbag of too-technically-competent-to-be-representative people who would go through tasks at lightening speed, barely reading them, to get the most profit out of their time. But I was curious about how well the system worked and tried it out with a series of tests. Some of these presented workers with ballots, and others asked workers to compare different aspects of Mozilla’s branding within the ballot.
I was actually amazed at how well the Turk performed, and especially at the ludicrous speed at which workers responded. Within four hours I had the time of about four hundred workers, each spending an average of three minutes on a short seven-question survey that I thought would take 20 seconds. To my surprise, the vast majority of free-form answers were filled in with thoughtful detail. Numbers of people this large, for that kind of time, for so little money (each worker was paid ten cents USD) was pretty sweet. It’s frightening how the hive mind’s torrential force is so easily bribed.
The bad news is that as suspected, the Turk workers are a strange sample. They skewed heavily towards living in North America and being more technical than an average users. Of my workers, 49.35% were already using Firefox (compared with 31.34% worldwide), and only 18.44% were using Internet Explorer (compared with 55.53% worldwide). However, those free-form answers were pretty effective at separating geeks from n00bs, and it’s n00bs I was interested in.
So what are the results already?
Ok, ok. For the majority of tests, I presented users with a browser ballot with some small variables changed – maybe a different Firefox logo, a different description, etc – and had users select a browser as though they were being given the ballot when starting Windows. Unsurprisingly, no change to Firefox’s logo nor description on the ballot made a statistically significant difference to what browser workers selected. But people were in the mood for change: 29.09% of people wanted to switch browsers when presented with the ballot. Internet Explorer users were most likely to want a new browser – 38.03% of them changed teams. Opera users were most loyal, with 92% staying on Opera. 70.53% of Firefox’s users wanted to stay with Firefox.
Of those Firefox users who wanted to switch, 51.79% wanted to try Chrome. Of Chrome users who wanted to switch, 50% wanted to try Firefox. It’s not too surprising that there’s crossover – both Chrome and Firefox attracted the more tech-saavy users who followed browser innovation to some degree. What’s interesting is that the main reasons people gave for switching to Firefox and Chrome from any browser were related to brand recognition. Nearly everyone who wanted to switch to Firefox said they’d “heard good things” about it, or that an acquaintance had recommended it. Most switching to Chrome mentioned using and liking Google’s search engine and being curious to see what else Google could do. In Firefox’s case, recognition was based on recommendation. In Chrome’s case, recognition was all about the familiar logo associated with successful searching.
Of Internet Explorer users – the only ones who will actually see the ballot – 22.53% wanted to switch to Chrome, 8.45% to Firefox, and 7.04% to Opera. I should note that I believe the high Chrome percentage is partially because this sample of people make part of their income through the browser and are thus more familiar with the Google brand than your average European Joe. However, it does show a huge momentum towards Chrome, fueled by that all-important brand recognition.
So what else made people switch browsers? Workers had many associations with browsers – even those they didn’t use. They described both Chrome and Firefox as fast, easy to use, and innovative. Most users sticking with Internet Explorer described it as being familiar, or that they were content with their current browsing experience. However, there was one association that workers only made with Firefox: security. Nearly every comment relating to security and privacy was in support of Firefox, and most of those who would consider switching to Firefox mentioned this as a personal priority.
Below is a graph of how the workers responded to the ballot. Unsurprisingly, most people voted for the browser they were using, shown by a strong top-left to bottom-right diagonal.
If these results were extrapolated, it would dramatically change the browserscape of Europe. Internet Explorer would lose 38.03% of its European market share, Chrome would gain 10.11% the European share, and Firefox would gain 3.79%. Again, these results are highly skewed towards Chrome because of the Mechanical Turk sample, but I do think that Chrome will probably emerge as a big winner of the ballot screen. The Google brand is well known and well established, and I believe it will cause users to recognize Google as a brand they already use thus try Chrome.
This is fascinating! Thanks!
–Tristan
This is a really great writeup, Boriss, and I like the graphics you used to present the results (someone’s been reading her Tufte!). Is your data based purely on responses from the EU, or a combination of EU and North American readers? It would be good to see if a “worker’s” home country has an effect on brand recognition (I suspect so!)
Thanks for a really fascinating post – would it be possible to see the survey questions you asked participants? I’m curious what kinds of questions you used in the lead up to presenting the ballot.
This is all exactly what I’d expect, really, and the analysis seems pretty much spot-on, with the exception of the reasoning behind why there were so many Chrome choices by IE users. I’d suspect that the answer lies more along the lines of people not having any idea what a browser is, not reading text, and thinking that Google is itself in fact “the Internet”, and that thus when you ask them something about how they want to use the Internet, they pick Google.
-Max
Great post, Boriss!
The ballot could, in pretty short order (Microsoft has committed to completing the roll-out to the installed base over a period not greater than 5 months) put Firefox in first place for browser usage in Europe.
The installed base (all current Windows users in the EEC with IE set as their default) will see this ballot rolled out over 5 months and your investigation makes some (I think) compelling estimates of how that will fall out.
But I have another question that might be equally important to investigate. Microsoft has also agreed to present the ballot as part of the normal first or near first run experience for every new Windows machine that ships in the EEC for the next 5 years. This case covers two groups of people, I think. First, it covers the “upgrade” or “second machine” people who are already online and who are buying a new PC. Second, it covers the actual growth of the Web case where people are getting their first computer and going online on their own computer for the first time.
The second cohort will be difficult to measure with something like Mechanical Turk because by definition they’re not online yet. (If you have any ideas how we might learn about them, I’m all ears.)
But the first group, people getting a second or a new PC, might actually behave differently from the installed base when they confront the ballot. Do you think there could be enough difference to warrant a follow-up Mechanical Turk or other study to see if there’s anything we should be doing later in the year after the installed base is all finished with the ballot and we’re just dealing with new PCs?
The reports I’ve read from Gartner and iSuppli suggest that ~100 million new PCs are purchased each year in Europe and my estimate is that 1/3rd of those are first-time purchases. That suggests that somewhere around 150-200 million new PC users will see this ballot over the next 5 years. It might be nice to think about how that might unfold for users.
Nice work.
Some deft wordplay here:
“38.03% of them changed teams. Opera users were most loyal, with 92% staying on Opera. 70.53% of Firefox’s users wanted to stay with Firefox”
Means 30% wanted to change teams from FF and 28% from Chrome. And IE was 38%. It does nudge my perception a little.
Good to see that Firefox spreads through word of mouth. That’s positive. **** brand recognition. Internet is not a Texan ranch. The negative thing is I’m yet to convince a single person to switch, maybe the problem is with me.:-(
And good news in the security area, seen in the light of some negative press last year. Some website made assertion that “FF was the most exploited browser, etc.” And Forbes included FF on its software in danger of exploitation list.
Google is FF’s main sponsor, whose pledge of not to be evil is less heard nowadays. Hmmm. The hope lies with the Community. The Geek shall inherit the Internet.
Opera, the Sushmita Sen of the Internet. Why doesn’t she get the spotlight she deserves? She should’ve cornered 20% at least. Not that I succeeded in sticking with opera. I never seem to get infected with the x-factor that ties down the 92%.
BTW, I typed this on The Fox.
Great analysis, however I was surprised by your explanation of Chrome success: “Most switching to Chrome mentioned using and liking Google’s search engine and being curious to see what else Google could do”
That’s nice to see curious people, but also: isn’t Google also leading a _huge_ marketing campaign to boost Chrome breakthrough (precisely when this ballot screen is launched)? I don’t know the situation in the US, but that’s what Tristan says when seeing the situation in Europe: (french) http://standblog.org/blog/post/2010/01/08/Publicites-Google
Maybe the turks desire of “being curious to see what else Google could do” were somewhat influenced? Like Mrs. Google whispering nice words on their ears through giant billboards in the subway?
What do you think?
Ronan Jouchet –
I think it’s possible the marketing campaign is having some effect. Google builds its brand recognition both through its search engine and advertising. However, Chrome’s market share has not yet gone up significantly since the beginning of the ad campaign. Further study would be needed to see what, if any, effect it is having.
Thanks for sharing good information…!!
Thank you nice article…
Good day very cool site!! Guy .. Excellent .. Wonderful .. I will bookmark your website and take the feeds also?I’m satisfied to seek out numerous helpful info right here within the put up, we need work out extra techniques in this regard, thanks for sharing. . . . . .
The services offered by SEO company India possess a quick turnaround time of 4 to 5 days.
Outsourcing the job to India will ensure the global standards are maintained in any way
cost. These links will help you advertise your products and services in addition to increase your brand awareness.