One more reason why Comscore and other “surveys” are unreliable

And that’s being kind…

Over the past few days the CA Security Advisor Blog has been posting about the spyware which is installed by Sears and KMart when you join their “community”. Spyware which leads back to Comscore and which, in essence, tracks every single web action – including secured transactions, that the infected users take. And it is this very pool of spyware infested users which Comscore then relies upon to make sweeping statements about the traffic and online activity across the Internet.

This is not a minor issue. These seriously flawed and troubling methods result in the numbers which, in turn, get cited as fact at major events (such as from keynote speakers on stage at AdTech NYC this past Fall) and quoted heavily in the major press on and offline. Further these comscore numbers are then used to drive much of online ad spending.

On the Pho list I wrote the following analysis a few months back, I am quoting my email in the entirety,  the context was a discussion on the list (which is focused on digital music) on the Radiohead In Rainbows experiment and a comscore report that claimed that 60% of all users who visited the Radiohead site had downloaded the music without paying at all. A statement which the band itself vigorously rejected.

A few observations and further fodder for discussion.

This past week I spent the last four days at Ad:Tech NYC. (I was
covering Ad:Tech for my friend Allen Stern’s blog, Centernetworks –
see http://centernetworks.com/tag/adtech for my coverage).

At MULTIPLE times over the course of the conference, most notably at
many of the keynote presentations from senior ad industry leaders, the
comscore study was cited without question as being authoritative and
proof that most people won’t pay.

I see any number of very serious flaws in Comscore‘s processes and methods.

Here are a few.

1. The underlying, basic assumption of any survey is that your sample
population can serve as a proxy and basis to extrapolate up to the
whole population. HOWEVER I think that especially online today this is
dangerously flawed. People do not act independently – instead people
are deeply influenced by the behaviors of their peers – and online
this effect can be multiplied many, many fold. In my own personal,
online networks literally dozens upon dozens of people have sent
twitters, emails, and written blog posts about In Rainbows so my
awareness of it (and the purchases of it often down to the exact price
paid) is extremely high. Amongst my circle – a very very high
percentage of people likely have visited the site,and most have
downloaded the album (and in most cases paid over $5 for it)

2. On a related front – about zero percentage of my population I
mention above are part of comscore‘s surveys population. Indeed given
that they: install an explicit piece of spyware (with permission – but
inherently it is spying on your every action) in return for “server
based virus scanning, sweepstakes and helping the internet” I doubt
anyone I know would participate – nor would I allow or suggest it to
ANYONE I know or advice. Not to mention that almost certainly most
corporate security processes would not allow such technology on
corporate machines (and with extremely good reasons).

Thus almost certainly their survey population, though over 2M are
almost entirely home/personal computers (even while more and more
workers have internet access at work and use that access for some
personal use). Furthermore since they are installing tracking in the
browser they miss: people with multiple browsers which they use on the
same machine, potentially people who have multiple logins to the same
computer (parents sharing a computer with children for example),
people with multiple computers, people with multiple internet
connected non-pc devices (i.e. browsing via a game console for
example), mobile phone (such as iPhone) access.

I use multiple browsers on both my Vista tablet and my iMac desktop –
not to mention my occasional use of Parallels on the mac. I also make
extensive use of my iPhone’s web access.

3. I would need to know much more in depth technical details of how
their browser plugin works – but on Vista computers to take one very
large example by default the OS firewall will block many types of
outbound reporting by applications without authorization (but this may
happen as part of their install).

4. I find it somewhat telling that nowhere which I could find at least
on http://comscore.com could I find a means to choose to join their
survey population (they may do this deliberately in that they want to
have some “randomness” to their survey population.

BUT this implies that they are using online ads and other means to
attract people to join their survey population.

I don’t know about the rest of you, but almost all savvy Internet
users I know generally never click on any survey driven offer (and/or
never give anything accurate in an online survey). I certainly don’t
answer surveys via online ads. Nor most via emails or popups on a
given page. Occasionally I will follow up from a conference by filling
out their survey of attendees (but I would NEVER allow such a survey
to install spyware on my computer).

5. While comscore is tracking a survey population designed to measure
the “typical” Internet user (though with billions of “internet users”
this alone may be essentially meaningless on a global scale) Radiohead
never intended to reach the “typical” user.

Radiohead wants to reach radiohead fans first – mostly current fans
but also to grow and gain new fans. They sing in English so a large
portion of their fans speak English (though now 100’s of millions of
people online do not) – further while they tour worldwide almost
certainly countries and cities where they have performed in the past
have more fans than those where they have not.

With the millions of existing radiohead fans (people who have bought
their past albums, gone to their shows etc) I would guess that the
percentage who visited their website is very very high – and that the
percentage who paid is also quite high.

Shannon