TrendWatch Blog
CMS Watch Releases Web Analytics Association Vendor Compliance Results
23-Apr-2009 --
When the Web Analytics Association (WAA) reviewed the 2008 edition of the CMS Watch Web Analytics Report, they noted that while the "report is well worth the price," there "was no effort to discern vendors' current or planned compliance with the WAA standards." The WAA standards represent an effort by the Association to set common definitions and descriptions for the various measurement concepts used by the web analytics solutions. A new version was released in September, 2008.
It was a fair criticism, and we decided to address this in the 2009 edition of our research. We asked all twenty vendors we evaluated to provide the same point-by-point explanation of their solution in the manner set out by IndexTools' (now Yahoo! Web Analytics) Dennis Mortensen.
The results: 50 percent of the vendors responded. They are
Amethon
Bango
Analytics
Coremetrics
Google
Analytics
Intellitracker
Lyris
HQ: ClickTracks
Mobilytics
Nedstat
Unica
WebTrends
We put all their responses into a matrix that's included as an appendix in our report. But you can now download the matrix here for free. See our release about it here.
We were pleased to see that most vendors we contacted wanted to participate. At the same time it was disappointing that not all did, especially mega-vendor Omniture. Omniture indicated that they would be releasing this information on their own sometime this year. We gave a pass to the video analytics vendors, Visible Measures and GlanceGuide, since the WAA Standards did not apply to them in a complete manner. While there is certainly debate among web analytics insiders as to the relative value of the Web Analytics Definitions, we think you can derive some “on the ground” benefits from using them as a reference in a couple of key ways.
1. Vendor evaluations
You’ll see when comparing responses that not all vendors answer "Yes"
in their being able to calculate all of the WAA standard metrics. This may come
as a bit of a surprise to those who have sat through demos where vendors spin
out beautiful reports, or to those who don’t really think about their
metrics requirements until after the vendor selection process. For example,
Repeat Visitors is not a metric you can get in Google Analytics, Lyris HQ:ClickTracks
or Nedstat. This becomes even more interesting in viewing the responses of mobile
analytics vendors who indicate that the metrics are collected, but not available
in reports.And, those vendors who do indicate compliance with the WAA standards,
may choose wording that you not find to be entirely clear, or may by omission
of details, invite additional questions, such as: "What do you mean by
that?" Or, "Can you explain that in more detail?" Or, "Could
you give me an example?" The definition of a "Page" provides
some illustration. Most vendors answer that their definition of a Page is the
same as the WAA, "A page is an analyst definable unit of content."
That in of itself is a bit obtuse, but the vendors by and large don’t
try to explain in any detail how "they" define a page; much less files
or media content. Only WebTrends describes in their definition, "Images,
sound files and video are not considered pages."
You may also ask vendors whether they provide reports that are compliant with the definitions "out of the box", or whether they require calculation
2. Report analysis
How many times have you been asked by your web analytics report consumers, "how
are unique visitors calculated" or "how is visit duration defined,"
or a range of other questions that are central to coming to terms with report
accuracy and intention? Referring to the vendor responses provides a basis for
your reference, as well as establishing a benchmark for your own report development.
You may also find that using these definitions as the basis for report specifications
prior to development helps educate your report consumers and helps to constrain
multiple iterations of report review cycles caused by a lack of understanding
of metrics definitions and terminology. Are the Web Analytics Association
Web Analytics Definitions a gold standard? No, but they are a useful tool in
your evaluation of vendors, as well as your understanding of the metrics and
reports you can produce once you have the solution in production. We invite
you to take a look for yourself and check out how your current solution stacks
up on its own and against the other vendors. We think it makes for some interesting
reading.
Again, you can download the WAA Compatibility ratings here.
- Submitted by: Phil Kemelor, Contributing Analyst
Join the conversation
Get a Free Sample
Wondering about CMS Watch research? Sign up to receive free samples of any of our products.