A Scenario-based Approach to Evaluating CMS Vendors

By Tony Byrne at 2006-10-18 10:17:00 |

There is no "best" web content management system. I don't believe in magic quadrants, leaders-and-laggards, and other horserace-style evaluation approaches. Neither should you. The best CMS for you is the one that best matches your needs — your budget, scope, and type of project — in short, the one that fits best for your web publishing and interaction scenarios.

Like everything else in business, software development comprises hard choices and trade-offs. Explicitly or not, software developers have particular scenarios in mind as they opt for one approach or another. Usually a product gets deeply imprinted by an influential early customer or a dominant set of internal advocates. In any case, vendors' design choices will decisively impact how customers use their products. But there are few universal norms here. That slick new Ajax interface may bring browser limitations. A cool personalization service may have unpleasant security, performance, and licensing implications.

So what's a prospective customer to do in a world where product "fit" is situational? I think the answer lies in matching your needs against common patterns. Then you can assess vendor inclinations against those patterns to see whether a product has evolved with your scenarios in mind.

In the recent update to The CMS Report, we've identified twelve common scenarios against which vendors can be judged. What follows is a short distillation of our thinking.

The limits of functional evaluations

For the past five years we've emphasized functional services — specifically, what CMS products actually do. Here's our latest list of functional criteria for web content management projects.

Web Content Management Lifecycle
Production Phase	Delivery Phase
Roles & Groups	Page Generation
User Interfaces	Site Search
Authoring & Transformation	Personalization
Aggregation	Caching & Replication
Tagging	Multichannel & Syndication
Workflow	Micro-applications
Repository Services	Retention
System Reporting	Site Analytics
Templating
Globalization
Content & Code Promotion

We still pay close attention to these features, because among them, you can understand in good detail how a particular product really works. Still, we found this approach incomplete, inasmuch as it doesn't tell you the deeper story behind the product or the vendor.

So a couple of years ago we added a second set of criteria that we call "vendor intangibles." Upon buying a CMS product or service, you are committing to ideally a long-term relationship with a supplier. It's essential to get to know that supplier.

These intangibles include:

• Maintenance & Support
• Services & Channel
• Integration & Partnerships
• Technology Vision
• Product Stability

But still, the methodology remains incomplete, because it doesn't explicitly answer key situational questions. When does a larger or smaller vendor make better sense? When does workflow matter more or less? More pointedly, when does a robust (read: complex) workflow subsystem become a drawback rather than an advantage? The answers lie in fleshing out common scenarios.

Twelve universal scenarios

CMS scenarios vary as widely as there are web publishing regimes, or put another way, as widely as there are websites. After talking to hundreds of customers over the years, we came up with twelve common scenarios for implementing a CMS. They revolve around twelve different types of web properties, broken into three broad categories.

Simpler

• Corporate Brochure Site
• Community-oriented Site
• Basic Interactive
• SMB/Departmental Intranet

Mid-range

• Enterprise Intranet
• Interactive Marketing
• Microsites

Complex

• Global Intranet
• Enterprise E-business
• Global Enterprise
• Multichannel Publishing
• Ultra-large Single Site

You can find a longer discussion of these scenarios in the Report, but let's look at just the "Community-oriented Site" scenario in more detail.

This is a fairly well-known scenario. Clubs, associations, civic groups, and other membership organizations frequently need an online destination to communicate and share opinions and information. More than just a simple forum, the site is typically owned and managed by the community leader, who may publish updates and other unstructured content. But the core of the system may include wikis, threaded discussion, blogs, photo-sharing, chat, polls, surveys, ratings, and other interactive services. At some level, these sites are really small portals, but most portal software — especially commercial portal technology — is too complex for this purpose. Meanwhile, some CMS tools — especially open source frameworks — target this scenario explicitly. In addition to providing a variety of authoring services, this scenario emphasizes role and ggroup Management, to be able to distinguish carefully among different classes of visitors and contributors.

Community-oriented Site
Editorial Model	Tends to be highly de-centralized and in fact may rely substantially on community (a.k.a., visitor) input.
Content Model	Highly unstructured; limited set of content types
Functional Emphasis On	Role & Group Management Authoring & Transformation Repository Services Site Search Micro-applications Linux support or Hosted model
Typical Adopters	Clubs Associations and other membership groups
System Modifications	Semi-frequent tweaks. Configurations set by novice IT staffer or technical webmaster.

You can imagine how the publishing rhythms and needs of a Community-oriented Site would differ from Corporate Brochure or Interactive Marketing site. To go back to our workflow question above, if workflow is required at all in this scenario, it must be very simple to craft and run, probably at the expense of advanced features like branching and collections.

To be sure, these 12 scenarios are abstractions. In practice your own web publishing effort is likely to represent variants or some hybrid combination of scenarios. And our use cases overlap somewhat. Nonetheless, they are useful for understanding which types of CMS packages tend to work better for which type of projects.

Looking at vendors

So let's talk about products. Many vendors — commercial and open source — don't like to focus on scenarios. Their sales and marketing people don't want to be shoehorned into a market niche.

To be sure, larger vendors can show you customers across broad industry types and scenarios. But the fact remains that products have profiles that constitute their "sweet spot," and our research suggests it's possible to identify those profiles.

More pointedly, if a vendor tells you they're good at nearly everything, beware: it may well mean that they're good at nothing, and significant implementation effort and ongoing support liabilities will fall to your team. In our experience, CMS products almost never fit plausibly across more than 8 of these 12 scenarios.

So let's take a look at how we profile one vendor, Interwoven, and its TeamSite / LiveSite CMS products (see chart, left).

As you can see, Interwoven tends to fit more plausibly for complex scenarios. We are not suggesting that Interwoven presents an ideal choice for backing your Enterprise E-business properties or your Ultra-large Single Site, just that TeamSite appears to be tacitly architected and developed with these use-cases in mind. Similarly, we would not automatically discount it for Multichannel Publishing or Global Intranets, but you should still carefully investigate its fit against your versions of those scenarios.

We would never counsel using Interwoven for any simpler scenario. It's not impossible to use TeamSite for those scenarios, just overkill. You could drive a huge dump truck on your daily commute to work. But should you? (You can read more about how we came to this analysis for Interwoven in our free sample of the Report)

Implications

In addition to facilitating a short list of potential vendors, a scenario-based approach has several important implications for your product selection process more broadly.

1. You may find it difficult or impossible to find one CMS tool well-suited for the multiple scenarios in your enterprise. This does not mean that you absolutely preclude standardizing on a single supplier. But you should understand the internal trade-offs your choice presents, and recognize explicitly when a tool will not fit well for certain scenarios, so the enterprise can make suitable accommodations.
2. A good functional match may present a bad fit for other reasons. You may like the workflow and repository services of a particular CMS product, but it may lack the object orientation that enables the easy cloning and ongoing management of Microsites. If your enterprise does public advocacy work or sells consumer goods (two traditional areas where we find Microsites), that product might present a mis-match.
3. There is great value in understanding how a vendor perceives the problem of managing web content. Some traditional document management vendors still tend to see it as a problem of exposing document stores through intranet or extranet interfaces. For some scenarios, that is indeed the sum of it, but many web publishing teams require much more. This is another way of asking: what do your prospective suppliers really mean when they say "web content management"? Ask, and you'll hear some very different answers.
4. Getting references from enterprises "like us" remains important, but perhaps "like us" does not automatically mean firms in your particular industry, but rather other customers who share similar scenarios. A direct competitor has deployed Stellent successfully for their global intranet. That doesn't make Stellent the right tool for your multichannel publishing efforts. Better information could come from the experiences of different firms re-purposing content the way you anticipate doing in your enterprise.

Taking a scenario-based approach does not absolve you from gathering business requirements — though hopefully it will release you from the slavish adherence to functional checkboxes that remains all too common in our industry.

Functional matrices typically don't help you differentiate effectively. And believe me, modern CMS tools really do differ markedly. Understanding how different products align against common business scenarios helps you peer more deeply into their situational strengths and weaknesses. And it will start to give you some clues about how those products will work with the most important websites of all: yours.