Testing
CMS Acceptance Testing
by Lisa Welchman
15-Nov-2004
Ideally your organization is undertaking some degree of user testing while your CMS implementation is underway. But before the system launches, it will need a rigorous and likely more comprehensive review by all the key stakeholders through formal "acceptance testing." When creating an acceptance test plan for web content management systems you must address three concerns.
Does the content management system:
- Meet the expressed business and functional requirements?
- Execute as intended?
- Satisfy the business and functional requirement in a usable way?
Let's look at each question in turn.
Satisfying the Requirements
Like most major software projects, many requirements can fall on the floor during the development phase of a content management system implementation.
Sometimes this is intentional. Explicit compromises are frequently made somewhere between defining business requirements and a functional specifications phase of the implementation, while the design phase typically entails a second check regarding viable expectations on the part of both content managers and the development team.
In properly run implementations, these compromises are discussed, agreed upon, and documented. It is during the heat of the development phase, though (with an implementation deadline fast approaching) that reality often sets in. Functionality that developers thought that they could build ends up being unbuildable. Frequently development efforts happen in a modular way -- one team working on the "templates," others on "workflow," others on "content," and yet another team on "publishing," so small, undocumented tweaks in one area can often cause system train wrecks later. Or, developers simply forget about a set of requirements. It happens.
Of course, there are many standard software development lifecycle practices that apply during a CMS software implementation. However, it's important to remember that web content management is still a relatively new field. I've found that sometimes these practices are not always followed by junior developers who honed their development skills in internet time, or by project teams who may be working together for the first time.
So the first thing you should do is make sure that the developed system meets all of the requested functionality. Dig out your business and functional requirements documents. Ideally, you have expressed these as use-cases or scenarios, which are designed almost explicitly to be tested. If you did a good job in this area, your initial requirements have created the superstructure of an initial test plan for you. In any case, your task is to create and go down a check list jointly (users and developers) to make sure nothing important got swept under the carpet or compromised away.
Software Integrity
Are there bugs in your CMS? Of course there are -- it's software -- but are they roaches or gnats? Does the system hang every time you run a certain workflow? Does the CMS actually publish content? Do titles associated with content objects actually appear properly in the rendered HTML...and so on. You will need to run through your scenarios or iterate through the checklist you made above. Most organizations are familiar with this sort of software testing.
Unfortunately many development teams unveil half-baked software to their end users. In-house development teams may have underestimated the complexity of integrating software they just recently learned, and now face a hard deadline. Outside consultants may have overstated their expertise in the CMS solution (and now face a hard deadline). Vendor professional services teams may want to quickly "stand up" an implementation in an effor to recognize software revenues faster. Or something just went wrong with the software.
But remember that content managers are the people who are going to give your CMS implementation a good name or bad. So here's some simple-sounding but important advice: don't show your end users a system that's not ready and call it acceptance testing. It will confuse them, and likely scare them too. Sure, you should have a non-technical user or two (who resides on the core implementation team ) take a look at things as you are developing, even if you are not following a formal spiral model. But don't open the curtain for acceptance testing until you have something that looks and acts like the system the typical content manager thinks they are going to get.
It naturally follows that if the system doesn't doesn't meet all its key requirements or execute properly, then you'd be better off not moving forward towards the last dimension of acceptance testing -- user acceptance -- until those issues are addressed.
User Acceptance Testing
Do you remember why you started this CMS project in the first place? It was probably so that you could publish web content more quickly, or create suitable publishing audit trails, or distribute website updates to non-technical users, or any number of other useful business goals.
For user acceptance testing you need to go back to this core success criteria and create test cases that will validate that you've met those goals in a sustainable manner. I like to define these types of step-by-step tests early on in the process, prior to or during design (and again, if you have already created formal scenarios, then you have already done most of the work here).
This is because it's important for users to understand before any code gets written exactly how they are going to work in the system and how it will impact their job. For instance you might design a workflow that triggers email notifications every time a particular type of content gets added or modified. During user acceptance testing all of the players involved will need to see that functionality in action as part of their daily routine to judge if it meets their expectations. (Some users may have no idea all those messages are coming, may not understand them, or may not know what to do with them.)
You can make up your own test plan but at a minimum it should contain the following information.
| TEST CASE | 1.1 | ||
| Task | Creating a new press release | ||
| Business Event | Public Affairs Liaison needs to post a press release to the website | ||
| Use Case/Step | User Action | Actual Response | Expected Response |
| 1 | User logs in as... | ||
| 2 | Clicks "New Object" icon... | ||
Conclusion
Proper content management software testing is crucial to the success of your implementation. You may be tempted to skip or expedite the requirements checks and user acceptance aspects of testing in order to reach the end of the implementation ("hit the date"). But if these aspects are glossed over, you run a serious risk of not meeting your business goals. And remember: it's never too late. Even if you are well into (or close to the end of) your CMS implementation you can still draft testable use cases and confirm requirements with your content managers. They'll appreciate the effort -- and you'll appreciate the results...