CMS Watch Enterprise Search Feed

A new (and wearable) Content Technologies Subway Map

tbyrne@cmswatch.com(Tony Byrne) — Wed, 23 Jul 2008 14:55:00 -0400

A new season brings an updated vendor map:

We added a Yellow Line -- for XML & Component Content Management vendors, and reflected some other station changes.

And now, if you like what you see, you and your wall can wear it. Our new store at Cafe Press offers t-shirt and posters of various sizes, along with other CMS Watch tchotchkes.

Regarding the latter, perhaps you already own your fill of mugs and mousepads, but can you ever have enough beer steins? Bring it to the next event where we're speaking and we'll fill it up with the closest available brew. ;-)

Infrastructure Updates for SharePoint

shawn_shell@consejoinc.com(Shawn Shell) — Fri, 18 Jul 2008 00:20:00 -0400

Through the SharePoint product team's MSDN blog, Microsoft announced that it had released a significant infrastructure update for SharePoint (and related technologies like Project Server that leverages SharePoint components). The update seems to primarily address three areas:

Search functionality and search-related performance (like index performance).

Content Deployment bug fixes (which hopefully will correct a series of irritating bugs related to deploying content from one SharePoint environment to another in web content management scenarios). These are include the hotfix packs Microsoft released for content deployment back in May of this year.

General interface and performance improvements. In reading the three or four pages in Microsoft's site that aimed to describe what was actually included, it was difficult to pinpoint what these "improvements" actual mean to SharePoint administrators. However, Microsoft describes them as "...fixes and product performance updates driven by customer feedback which have resulted in significant platform performance improvements..." Again, I was unable to nail what precisely has changed or how significant the improvements were.

What's interesting, at least with regard to search, is that it seems the "ancillary" search products like Search Server 2008 (and it's "free" sibling Search Server Express 2008) are driving updates to SharePoint's search technology. As mentioned in the SharePoint Report 2008, Microsoft has invested heavily in improving SharePoint search. In fact, historically, it seemed as if SharePoint Search was the the parent of these independent search tools, but it now appears as if "the student [has become] the master" as Darth Vader said to Obi Wan.

In particular, SharePoint is getting Search Server's federated search capabilities and "a unified search dashboard." From what I saw at the last SharePoint conference, both of these search products borrowed very heavily from the SharePoint interface construct, but improved the visibility of certain configuration settings. In particular, I liked the ease with which you could configure the federated search.

However, these changes call into question how this will all play out within the Shared Services provider and whether administrators who are struggling to figure out where to go to change search settings -- at the site, site collection, Central Administration (in the Application or Operation tab) or in Shared Services. While most key search settings reside in Shared Services, SharePoint has search-relate configuration in spread over virtually every administrative interface. My hope is that this "unified search dashboard" brings some order to search within SharePoint.

In the end, these changes (along with the FAST search integration) also add more evidence to the theory that Microsoft is going to decouple search from SharePoint entirely (and potentially the Office team) -- making SharePoint a client technology. As I blogged about in a post on the completion of the FAST acquisition, Microsoft seems to be leaning very heavily towards and independent search product team. And just to add fuel to the conspiratorial fire, this type of organizational structure might make sense if, say, Microsoft were to acquire a large Internet-centric search company (although it begs the question what they'd do with all of this overlapping technology).

Narrowcasting to your feed aggregator

tbyrne@cmswatch.com(Tony Byrne) — Fri, 11 Jul 2008 15:39:00 -0400

We're pleased that CMS Watch now covers ten different technologies, but I suspect that many of you take an interest in only one or two families of tools. If that's you, here's a list of technology-specific RSS feeds that will just send relevant postings to your reader or aggregator.

Digital Asset Management: http://www.cmswatch.com/RSS/cmswatch.channel.xml/DAM

ECM Suites: http://www.cmswatch.com/RSS/cmswatch.channel.xml/ECM

E-mail Archiving & Management: http://www.cmswatch.com/RSS/cmswatch.channel.xml/E-mail

Enterprise Portals: http://www.cmswatch.com/RSS/cmswatch.channel.xml/Portal

Enterprise Search: http://www.cmswatch.com/RSS/cmswatch.channel.xml/Search

SharePoint: http://www.cmswatch.com/RSS/cmswatch.channel.xml/SharePoint

Social Software: http://www.cmswatch.com/RSS/cmswatch.channel.xml/Social

Web Analytics: http://www.cmswatch.com/RSS/cmswatch.channel.xml/Analytics

Web CMS / WCM: http://www.cmswatch.com/RSS/cmswatch.channel.xml/CMS

XML & Component Content Management: http://www.cmswatch.com/RSS/cmswatch.channel.xml/CCM

Enterprise Search Scalability: A Big Issue

kthomas@cmswatch.com(Kas Thomas) — Wed, 9 Jul 2008 13:02:00 -0400

I was talking to a search vendor the other day who said something that really got my attention. He remarked that a customer recently came to him and asked what it would take, in terms of software, hardware, and time, to index 30 billion documents. Mind you, this was not some hypothetical exercise. The question came from somebody whose company actually has 30 billion documents under management.

Consider the dimensions of the problem. Assuming (for purposes of argument) you could index a thousand documents per second on one machine, it would take a full year just to build the index for 30 billion docs. If the solution scales linearly, building (or rebuilding) the index would keep a 100-machine server farm busy for the better part of a week.

That's considering the scenario in a static context only. In the real world, of course, documents are revised (some frequently, others never, most somewhere in-between). New docs enter the system. Old ones are dropped. Otherwise-unchanged docs are moved to new locations. Unless you can update your index(es) incrementally, in real time, as docs are added, deleted, modified, or moved, you have an index shelf-life problem.

The traditional answer to the shelf-life problem is to rebuild the index every few days (or every night, if resources allow). At the level of ten or twenty thousand docs, a total rebuild of the index every few days isn't a huge issue. But when you get beyond something like a few million docs, performing total-rebuilds on a frequent basis quickly becomes a worst practice (if indeed it's practicable at all). At some point, you need the ability to do incremental indexing.

But, someone will ask, can't you just throw more machine resources (threads, memory, cycles) at the problem? Yes and no. If you're spidering files over the wire, bandwidth exhaustion becomes an issue. If you're indexing files locally, there's an OS-imposed limit on how many files you can have open at once. There's also the question of how much file data you can hold in memory. The reason this is important is that some search systems (quite a few, actually) need to load an entire document into memory before the doc can be indexed. If you're indexing 10-megabyte PDFs, it might not matter how many threads you have available. (Note, incidentally, that most docs occupy a lot more space in memory than on disk.) And anyway, the CPU can execute only so many instructions per second, no matter how many docs you can load at once.

I bring all this up for a couple of reasons. First, if you're shopping for a search solution, you need to regard the various vendors' performance claims with more than a modicum of caution. No two search scenarios are the same, obviously. But more than that, the parameters that affect scalability and performance are numerous and non-obvious (and their interactions subtle), tending to moot most performance claims straight out of the gate.

Takeaway No. 1: If you care about performance (and you should), do your own testing. Insist on it as part of any product evaluation.

Takeaway No. 2: Get your programmers involved in the evaluation process early. Some of these issues require computer-science expertise to evaluate properly.

Also (very important), when shopping for a search solution, don't buy for your present needs. Shop for your future needs. Your company probably has ten times more content under management today than it had just five years ago. Five years from now, it could have ten times more than today. Will your search solution scale appropriately? More particularly, how will it scale? Will it scale linearly? Will it hit a brick wall?

If I were searching for a search solution, I'd ask every vendor a few simple questions:

How big is your biggest customer installation and what did it take to build it?
Can your system do incremental indexing? How often is a full rebuild required?
Does your indexer need to read a document into memory (whole) before indexing it, or can files be stream-processed?
What's the largest document your system can index without either choking or stopping after a particular number of characters?
How does indexing performance change as the index gets bigger? (Not just "does it slow down?" but how does it slow? Linearly? Exponentially? If it's the latter, you're going to hit a brick wall.)
And: Do you support 64-bit architectures?

Those are just a few conversation-starters. For more (lots more), be sure to consult our Enterprise Search Report 2008. (You can get a free sample of it online here.) And if you end up evaluating one or more search offerings in depth, please drop us a line and let us know what you learned. We're always interested in your feedback.

How Fast is Attivio?

bloem@radagio.com(Adriaan Bloem) — Sat, 5 Jul 2008 09:55:00 -0400

The Fast Search & Transfer news continues to keep me on my toes. The Norwegian business weekly Dagens Næringsliv seems to have come up with some decent evidence (English translation) of many things everybody already suspected -- and a couple of new ones.

It covers the Convera acquisition and the "separate" but surprisingly coincidental deal where Convera bought several million's worth of Fast software it didn't need. Or as rival Autonomy (which was also in the running for buying Convera) has pointed out, Fast pumped up its revenues for that quarter with part of the money it paid for Convera, then got back for licenses. The DN article also covers a few other very suspicious deals, and some outright fraud. It's now even getting to the point where calling Fast "the Enron of Norway" is getting long in the tooth.

While that train wreck was unfolding before my eyes in slow motion, my fellow analyst Theresa Regli pinged me last February about a new enterprise search company called Attivio. Information Today raved about their new product AIE, with analysts quoted as saying things like "they are moving rapidly to develop tools that will eliminate many of the practical barriers to easily and efficiently deploy robust enterprise search solutions," with the unique selling point of "data integration plus search and content processing," a "hot niche for the next few years."

Since I'm always interested to find out more about robust enterprise search tools to fill hot niches for the next few years, I scrolled down to read what the Attivio CTO would explain about how the product would achieve what "should have been solved by the integration of text search and XML into relational database managers such as Oracle." As it turns out, it is based on a "mash-up" of open source Apache Lucene and "licensed commercial software."

As described in our Enterprise Search Report and on this blog, Lucene itself is just a Java text search API. To be able to actually gather, convert, and query content you need many more components. It is perfectly feasible to put together a working enterprise search product around the core Lucene JAR (as demonstrated by IBM's Omnifind Yahoo! Edition). But in order to get there, and to have Lucene index, for instance, Office documents and PDFs, you will have to first convert those documents to text. The filters to perform that conversion can be bought from other vendors, based on open source such as pdftohtml, or you'll have to build them yourself, which is a lot of work. There aren't too many vendors building their own filters, or even just modifying open source to do so. So if you do build the filters needed to use Lucene yourself, you'd probably like to mention this as an advantage, and as Attivio states, "we developed our own Microsoft Office, WordPerfect, and PDF connectors to improve performance and reach deeper into the files than the conventional converters."

Since, like most enterprise search products, Lucene isn't based on a database and couldn't even connect to such content without help, it isn't surprising Attivio had to develop a "unique RDBMS data loader" which "indexes the tables individually." This, again, is presented as a major advantage -- remember, converting documents and integrating structured and unstructured data are "a hot niche."

I remember seeing a vendor at a conference a few years back, with banners jokingly stating its product was "buzzword compliant!" Attivio certainly seems to have that skill down. The engineering effort is marketed as a "technology mashup," "breaking down silos" between "open source and commercial software." "We have lived with the challenge of having to choose between the precision of databases and the richness of search for a long time, but no longer" sounds great, but I don't see Oracle and Thunderstone's RDBMS-based solutions breaking out in a sweat just yet.

Maybe my over-exposure to marketing materials and flashy demos has turned me into a cynic, and Attivio's downloadable trial version will have to do at least a decent job to convince me of the product's added value. Fortunately, that free download is "coming soon!" Yes, I'm sorry, I'm finding it increasingly hard to turn off that cynicism, especially when I turn back to the DN article about Fast Search & Transfer. Attivio was founded by former Fast employees and the Attivio CTO is Sid Probstein, formerly vice president of technology at FAST. More importantly, Attivio's CEO is Ali Riaz, who was COO at Fast but unexpectedly left the company in late 2006. Well, in hindsight, perhaps not so unexpectedly, though DN quotes him as saying "I had nothing to gain from manipulation of the accounts. I had no shares in the company. I wanted shares and quit because I didn't get any. If you want to find out what's wrong with the accounts, you need to look at those who could gain from it. And it wasn't me."

Dagens Næringsliv doesn't appear to agree with Riaz, however; if you want the full analysis of the what and why, I suggest you read the article. I myself find it surprising that the CEO of a technology startup backed by $6.2 million in venture capital would drive an Audi R8, but that doesn't mean anything (other than that I'm envious of his car). I also find it surprising a former Fast COO would be co-owner of a company reselling Fast licenses, but walking like an Enron duck and quacking like an Enron duck doesn't necessarily mean that it's really anything like Enron. And Attivio's clouding the core technology in marketing hyperboles and buzzword compliance is slightly disconcerting, but many renowned companies engage in the same practice.

DN quotes Riaz as saying "you should be much better at praising the people who have success, instead of pushing them down." And I would certainly love to be proven wrong by Attivio's software; as soon as I get my hands on the trial download I requested, I will let you know if it lives up to the high expectations. As one of my teachers in school once told me, "I'm known for being cynical, or even sarcastic -- myself, I prefer to call it healthy skepticism and mild irony."

Being a cynic isn't a lot of fun -- but for now, I would advise you to be at least healthily skeptical of what Attivio has to offer.

Best bets: a worst practice?

kthomas@cmswatch.com(Kas Thomas) — Thu, 26 Jun 2008 21:50:00 -0400

As one of the authors of our Enterprise Search Report 2008, I've been spending a lot of time lately looking at search technology and talking to folks who care deeply about the subject (i.e., vendors and their customers).

One thing everyone seems to agree on is that providing relevant results to the user is a very hard problem indeed. People don't want to enter a few keywords and then get 10,000 hits on documents that contain those keywords, 9,999 of which might be irrelevant. They want pointers to the one or two documents that are relevant.

The signal-to-noise problem is so thorny that many enterprise search products include an optional feature known as "best bets." The idea is that certain very common searches should point to particular documents (or intranet pages) that are known, or presumed, to apply. Imagine that a lawyer working for a large legal firm logs into the company portal and searches on "poison pill." A thousand hits might come back, of which 990 are related to medications, allergic reactions, toxicity, malpractice, and so on, even though all the person was really looking for was a link to the company's "merger and acquisitions" resource page. ("Poison pill" is a term for tactics a company can use to fend off hostile takeover attempts.) The idea of "best bets" is that you rig the system to promote the company's "M&A resources" link to the top of the hit list whenever someone does a search on "poison pill."

Sometimes "best bets" refers to presenting the user with a recommendation when, say, several repositories exist, one or more of which could be better-suited to a given search than the others. ("Would you like to search the Parts Catalog for this?") This is more of a navigational scenario. That's not really what I'm talking about here. I'm talking about the practice of biasing search results by hard-coding certain answers to certain common queries.

Setting up "best bets" is typically a manual process. A person in IT will use search analytics to determine the most common search queries and the most-followed links associated with them. Then those associations will be captured in a database and wired into the search software in such a way that when a user issues a query for which a best bet already exists, the best-bet link(s) will automatically be shown at the top of the results page (either as a regular hit or under a separate heading of "Best Bets").

Not everyone thinks the "best bets" mechanism is a good idea. The problem is that, fundamentally, it's a hack. It's arguably the worst kind of hack in that it involves serious amounts of human intervention. Someone has to create the best-bet database. (Typically there will be hundreds, if not thousands, of best-bet links.) Then the database has to be updated and kept fresh as user needs change and documents are added to or dropped from the system.

In point of fact, the search software should do all this for you. After all, that's its job: to return relevant results (automatically) in response to queries. Why would you sink tens (or hundreds) of thousands of dollars into an enterprise search system only to override it with a manually assembled collection of point-hacks?

Sure, search is a hard problem. But if your search system is so poor at delivering relevant results that it can't figure out what your users need without someone in IT explicitly telling it the answer, maybe you should search for a new search vendor. (And for help with that, see Enterprise Search Report 2008, a free sample of which is available right here.)

The value of archiving and the limitations of e-discovery

aps@cmswatch.com(Alan Pelz-Sharpe) — Thu, 19 Jun 2008 13:49:00 -0400

Today we read about yet another major financial scandal allegedly exposed through the discovery of an e-mail message from a fund principal that apparently stated that their fund was going to be"'toast."

The first thing I thought about this was that (if true) it was a fantastically stupid communication to put in an e-mail exchange. Secondly, I wondered why it took so long to find this mail -- surely such high-profile financial managers would have their mail exchanges monitored automatically and an exchange like this should have rung every major alarm bell in the firm within seconds. Of course they could have been using an external system to get around that; we don't know at present. But this case once more highlights the limitations of e-mail monitoring (discussed here the other day) and e-discovery, and conversely the value of content archiving.

E-discovery is in many regards simply glorified Enterprise Search technology, but with the added ability to apply legal holds to data. Just as Enterprise Search is limited by the quality and location of the content it indexes, so too are e-discovery tools. Though in the case of e-discovery the limitations are often more severe: evidence may or may not be conveniently located in an e-mail message, as seems to be the case at Bear Stearns. More commonly evidence has to be culled from not only e-mail stores, but also from instant messaging systems, document systems, ERP systems, financial and business applications, external drives, and so on. The idea that e-discovery is limited to mail -- as many vendors (and worryingly many buyers) seem to think -- is naive in the extreme. Yet this misplaced belief is based on the reality that the bulk of the data you will have to search will indeed be mail. Mail represents the largest form of data in any organization, typically by an order of magnitude (10x) or more.

But here's the rub. Most of that e-mail mountain consists of redundant data or as the technical terms goes, "crap." As we discuss at length in our E-mail Archiving & Management Report, typically 80% of mail data consists of a duplication. Yet any search tool has to treat each piece of data equally, slowing the process down massively and shooting discovery costs through the roof. How much more sensible to use an archiving method to capture, filter, and reduce that volume -- and ease the burden and cost of discovery?

What did we learn today from the Bear Stearns scandal? Not much really, other than mail (and messages) continue to the be the key "gotcha" elements of the data mountain, and that we need to monitor and manage them ever more closely. Though the monitoring elements are far from mature, EAM tools today archive and filter very efficiently indeed. The need to take mail and mail content seriously is now an imperative, and building a strategy, agreeing methods and policies, and selecting the right tools -- however complex -- is a must.

FAST clarification

bloem@radagio.com(Adriaan Bloem) — Tue, 3 Jun 2008 09:43:00 -0400

Apparently, when I previously wrote about Carrie's hand reaching out from the grave, the metaphor was too subtle. So let me make it absolutely clear: I'm not hoping the FAST turmoil will "just go away" so I won't be bothered by it anymore. I was pointing out the fact that the consequences could be further reaching than it being "just another investor soap", and the acquisition of FAST by Microsoft won't be enough to bury it.

Now that I've ruined the joke by explaining it anyway, let me add that this is no laughing matter. Of course, while it is much more fun to ridicule the situation without having to be able to back it up, I'll have to stick to the known facts: FAST is under investigation by the police. By stating just that, I estimate the readers of this blog to understand this as a serious warning.

When I say I'd rather discuss the technology, again, I imply rather then explain (to do a good job of that takes quite a few pages). So let me explicitly state some of the questions arising.

Now that FAST is turning out to be more of a Tucker Torpedo than a Cadillac (and to quote Wikipedia: "[Tucker's] Accessories Program raised funds by selling accessories before the car was even in production"): what is it Microsoft actually bought?

Customers (who were, it now seems, fraudulently overvalued, and many of which are on the Linux platform), whom Microsoft probably won't support in the long run?
Street-cred in enterprise search, which is now seriously marred by the consistently negative reports about FAST, compounded by suspicions that MS may have once again been blinded by competition with Google?
A vision on enterprise search, in the form of a vice president of Enterprise Search possibly implicated in a police investigation?

These have been widely reported, and there doesn't seem to be a whole lot more of value that came for Microsoft's $1.2 billion than technology. Which is what prompted my widely misunderstood remark that I'd rather discuss just that piece of the puzzle. Surely, it would have been possible to acquire the technology at a friendlier price point, say $23 million (oh wait! they got that as a free bonus), but it raises enough questions on its own. Which parts of ESP aren't third party, bought, or open source? How would these be integrated, or will a separate product line emerge? What are Microsoft's long-term views on enterprise search?

I've criticized Microsoft's unclear strategy and roadmap for MOSS in the past, and once again, Redmond seems to be covered in fog. Since search solutions are written off in years, not months, knowing what future development holds for the software is very relevant to existing and prospective customers. I'd think twice before investing in an implementation until the clouds subside.

On a side note, kudos to Dave Kellogg for being the first to actually respond to my previous post under his own name. I received a fair bit of anonymous mail on this subject, and it seems paradoxical to be criticized for not speaking out loud enough by those who choose to remain incognito themselves. Though this post is not about Dave, as much as his post is not about me, I would like to point out that my name contains three A's (not two), and I'm not a 20-something English major. Yes, CMS Watch does have maps and charts, and no, we don't dismiss business problems -- we just prefer not to place too much emphasis on the intricate details of GAAP accounting procedures on this blog...;-)

CMS Watch Competition Winner

aps@cmswatch.com(Alan Pelz-Sharpe) — Mon, 2 Jun 2008 09:45:00 -0400

You may remember a while back we launched our little competition to come up with a new name for our vendor positioning chart. We had some great (and varied) responses from all over the world. And it took quite an internal debate to decide on the eventual winner, but decide we did. And the winner is...Laurence Hart who offered us the name "Cross Check." Laurence, a bottle of champagne is yours.

Over the next month we will continue working with our designer to revamp the chart, and of course to rename it -- so look out for the Cross Check in all our report updates this year.

FAST and the Furious

bloem@radagio.com(Adriaan Bloem) — Fri, 30 May 2008 01:16:00 -0400

I vividly remember being terrified as a child by the movie Carrie. The closing scene -- where Carrie's hand reaches out from the grave -- left a lasting impression.

Now I'm a technology analyst, not a financial analyst, and certainly no Stephen King. But sometimes it's hard to ignore what goes on in our industry beyond the realm of ones and zeros, and when Wired reports it (or, if you prefer to read about it in Norwegian, Aftenposten), I should at least casually mention it. The central character here, FAST Search & Transfer, is not exactly dead -- just acquired, by Microsoft.

After enduring many difficulties last year, FAST's problems turned out to be more than just accounting headaches. So much so, in fact, that the Kredittilsynet (the Norwegian financial authority) found the irregularities important enough to refer the case to Økokrim.

Økokrim describes itself as "The Norwegian National Authority for Investigation and Prosecution of Economic and Environmental Crime." A bit of a mouthful which I can abbreviate to "the police." In other words, it is now a criminal investigation.

Of course, it would be thrilling to continue this story by recounting past rumors about FAST, and what Microsoft did or did not know about it before they started talking about an acquisition. Followed by an interpretation of the meaningful silences in the conference call where the two companies jointly announced the deal. And to round it off, a discourse on how this will affect the future of enterprise search for Microsoft.

But in fact, I hope we've pretty much heard the last of it and can return to simply discussing the merits and demerits of the technology. So even though I have this lingering image of the last scene of "Carrie" in the back of my mind, after this short break, we'll return to our regularly scheduled programming.

Dining at the intersection of Search and Retention

kthomas@cmswatch.com(Kas Thomas) — Thu, 22 May 2008 17:41:00 -0400

Lawyers were well represented (you might say) at this year's Enterprise Search Summit in New York. At times, ESS felt more like an e-discovery conference with analytics and social-computing side-tracks rather than a search conference featuring a few e-discovery sessions.

Based on what I saw at the Search Summit, there seems to be a renewed awareness, at ever-higher levels in the corporate responsibility chain, that in a litigious business environment "enterprise search" is not just a knowledge-management tactic or a productivity aid, but a survival imperative. You will be sued some day. (It's not a matter of "if," but when.) During the discovery phase of the suit, you're going to provide (and also receive from the other side) bewilderingly immense amounts of data. Without good search technology, sifting through the data isn't just tedious but nightmarishly expensive.

I didn't get a chance to attend any e-discovery sessions at the Search Summit. It didn't matter. At lunch, I happened to sit down next to litigation technology consultant (and ESS presenter) Jeff Flax. We had an illuminating chat about search and discovery in the context of records retention.

Flax noted that many companies that have records retention policies aren't following them. He sees a "pack rat" syndrome: a tendency to let expired records remain in the morgue past the "save-till" date. The problem with this is that files that have been declared obsolete or marked for disposition, but have not yet been physically destroyed, are still subject to subpoena. "A good lawyer will ask for expired documents during discovery," Flax notes.

Lawyers are also demanding data in its "native state": Not text dumps or PDFs or other derivative forms of the data, but the data as it actually exists. "If I'm a lawyer and I'm requesting someone's e-mails on a certain subject," says Flax, "I don't want the e-mails as text files, I want the original e-mail archive in binary form so I can pick apart the bits and get at all the header and footer and other information in context."

Sometimes physical media must be handed over in discovery so that deleted files can be detected and recovered. "I've seen cases where browser search queries from many years back, supposedly no longer on disk, have been recovered forensically," Flax told me. "And then certain keyword clumps are detected, and those query patterns can become admissible in court."

It turns out that the data in a search index (the index built by a search engine) can often be used to reconstruct a document even after the document itself has been irretrievably lost. Takeaway: A document can't be considered fully destroyed until you've destroyed its search-index data as well. (I wonder how many retention policies take this into account? Doubtless very few.)

If you're concerned about e-mail retention (and if you're not, you should be), you might want to look into our latest offering: The E-mail Archiving & Management Report 2008.You'll find that the report divides vendors (roughly) along three lines: policy-centric, archive-centric, and SaaS-based. (You can see a free sample here.)

My advice? Never pass up a chance to have lunch with a litigation technology expert. You'll be inundated with food for thought.

Enterprise search: free as in free beer?

bloem@radagio.com(Adriaan Bloem) — Fri, 16 May 2008 13:23:00 -0400

Searching information -- really, how hard can it be? So, why wouldn't you go out and get a search engine that's for free? Well, to stick to the analogy of "free beer," you might wake up in the morning with a headache, only to find your wallet gone.

Of course, I'm paraphrasing the definition of "free software". Richard Stallman's example is used to point out the ambiguity of the term "free" in the English language. With free software, "you should think of free as in free speech, not as in free beer." Nevertheless, you should be warned: both open source beer (now in version 3.3) and free commercial beer have the potential for leaving you with a bit of a hangover.

If you really think enterprise search is a simple commodity -- and I will only comment on that with the obligatory statement that readers of our Enterprise Search Report will probably know better than that -- getting a free product would be ideal to get your feet wet (albeit somewhat sticky). I get invited to BYOB enterprise search parties a lot, and usually come up with Apache Lucene, IBM Omnifind Yahoo! Edition, and Microsoft Search Server 2008 Express. Let's get a closer taste of each.

Apache Lucene. Lucene is open source, which you are free to use. The problem is, it's not a complete enterprise search product -- it's a "text search engine API." What you get is a Java JAR with the core functionality of a search engine. In typical hardcore Java developer understatement this is described as "you write the easy stuff, the UI and the process of selecting and parsing your data files to pump them into the search engine, yourself." To developers that doesn't sound too difficult -- it's a library they'd be able to use to create search functionality for many applications. As they embark on that journey, however, many will find out they'll have to become experts on enterprise search to get their implementation to perform basic tasks any Google user has come to expect. Index Word documents? You'll have to convert those to text first. Remove stop words or perform spell checking? You'll have to get some more jars to fit that in. And that familiar user interface isn't so easy to replicate, either.

Of course, there's a couple of more "pre-packaged," Lucene-based engines (such as Nutch and Solr), but they'll only take you so far on that long and winding road. There's some excellent examples of what you can achieve with Lucene, but many more of how hard it can be to get there.

IBM Omnifind Yahoo! Edition (or OY!E). The Google appliances have the Google brand behind them, which must have got the IBM people thinking the Yahoo! brand would be excellent marketing for their free-to-use search engine. In fact, it's neither IBM nor Yahoo's technology, but Lucene wrapped in other open source software. A few commercial bits thrown in create a product that's easy to install and run. It will actually do many of the things Lucene will make you work hard to accomplish: it comes with support for several languages and quite a few source content filters. For users, it looks like a regular web search engine; for admins, there's a nicely designed and intelligible interface. In short, it does most of the things a Google Mini appliance will do -- but for free.

So what's the catch? Well, the license (by the way, what license?) limits you to 500,000 documents and 5 collections. After that, you can "upgrade" to other Omnifind products. But since the technology across the Omnifind line-up is completely different, this is the same as starting from scratch, and you'll pay for the privilege. I've been critical of the limitations of Google's appliances in the past, and sure, the 50,000 document limit of the entry-level Google Mini is a lot less than OY!E's half a million. But that comparison isn't really fair, considering the fact the Mini actually comes with the hardware to run the queries on for a mere $2,990. And don't think you'll be able to run IBM's software on an old abandoned test server you have available -- OY!E will need more power than the single blade Google Mini or Thunderstone Appliance to match the performance. Tellingly, I wasn't able to dig up an example of an OY!E implementation to mention while researching the Enterprise Search Report (if you know of one, let me know).

Microsoft Search Server 2008 Express. Microsoft's free offering is basically the same software as the non-Express version, but then there's the seemingly innocent limitation: one server only. I wouldn't want to continue the theme of this post by saying this is akin to handing out free samples of beer to get you hooked; suffice it to say that if you start to run the Express version in a production environment, there will, no doubt, come a time when a single server won't be enough anymore. When you've come to rely on the solution, you'll suddenly have to shell out for the licenses. As I've said before, having a free lunch isn't necessarily a bad thing; just remember that you'll probably have to pay for the beer the lunch comes with.

So, this might all start sounding like advice your mother gave you: never take anything from a stranger, and certainly no free alcoholic beverages. Don't forget, however, that I'm Dutch, and I've certainly developed a taste for enterprise search. Free beer sounds too good to be true, but it could certainly get your party started; just remember to drink in moderation, and never, ever, drink and drive.

Vendor criticism of CMS Watch

aps@cmswatch.com(Alan Pelz-Sharpe) — Fri, 16 May 2008 10:43:00 -0400

As you know at CMS Watch we write critical product evaluations to help you avoid expensive procurement and deployment mistakes. We write reports that detail both the warts and merits of big vendors like EMC, Oracle, Xerox and IBM -- through to smaller specialist vendors like Hyland, Autonomy and Nuxeo. Readers of our reports often ask me "what did vendor x say when they read that!" The assumption, sometimes correct, is that vendors freak out on reading such criticism.

In an industry whereby most of the "independent analysts" are heavily dependent on revenues from the very firms they claim to be "independent" of, it's unusual to see truly critical research get published. So it becomes a surprise to both buyers and sellers when they read such criticism. In our reports we widely distribute the compliments and brickbats -- if something is truly terrible we will tell you.

But most of the time it is not a case of bad technology versus good technology. Rather it is a case of good fit versus bad fit: a product that could become an outstanding performer in a larger legal firm may make a terrible fit in a mid-sized manufacturing and ERP-centric environment. Hence we urge you the reader to study all the alternatives and balance them out, rather than look at one preferred vendor in isolation.

Speaking of isolation, the marketing groups of some vendors seem to operate in in a kind of vacuum. I guess it's part of the job for them to drink their own Kool Aid, but some of them seem to think it's part of their job to attack and stop any criticism of their product or company. At CMS Watch we're often on the receiving end of that wrath; that stinks sometimes, but so be it. Just as it is the vendor's job to wax lyrical about the joys of their product, so too is it ours to unearth the reality. If you want to get an insight into this particular dynamic, whether you're a curious end user or a vendor AR (Analyst Relations) person, check out the article I published today.

SEO: furthering the case for better content hygiene

tregli@cmswatch.com(Theresa Regli) — Fri, 9 May 2008 16:12:00 -0400

The worlds of SEO and enterprise search are not as far apart as you might think. Let me explain. Along with my colleague Alan, I had the pleasure of attending the upbeat Internet World UK conference earlier this month in London. What I enjoy most about attending such large, diverse events is going to sessions about technologies related to, but not directly about, the technologies I cover -- and so I found myself attending numerous sessions on SEO, or Search Engine Optimization.

Though there was an inquisitive crowd present for my talk, the topic I spoke about -- enterprise search -- is much lower on the totem pole for e-marketers (the principal attendees at this event) than SEO. A few showed up, in fact, wanting to better understand the nuances of website search vs. enterprise search vs. SEO. Are there universal things we can do to simultaneously improve all things, I was asked? Indeed. Despite the differences in searching the web, searching within your enterprise, and making your own website more findable by search engines, many of the same best practices apply.

As readers of The Enterprise Search Report know, good content hygiene is essential to good search results. Consistent content structure, metadata, and simple things like clear and meaningful document titles all help search engines work better -- be they the public (e.g., Google) website, or enterprise kind. But in most cases, content managers don't know where to start with their clean-up: it's an often overwhelming task. As such, content clean-up continues to get cast aside ("too much work"), and search technology vendors make it seem that much less important when they promote technology as the panacea to content woes. Don't believe it for a second.

A few more tidbits from the event:

These days, 80% of e-commerce transactions start with a web search, says Dan Cohen, Head of SEO for MSN UK.
The #1 hindrance to SEO is poor content and code (which also highly contributes to poor web and intranet search results)
Creating clear, topic-related content "hubs" on your web site is the next most important thing for SEO
As readers of our Web Analytics Report know, analytics should be a core part of your e-marketing pie, to get a clear picture of what your customers are looking for and where they're getting stuck.

Bottom line: cleaning up your content can improve your website search results and your Google ranking.

Budget time: How much should I set aside for software licenses?

info@jboye.dk(Janus Boye) — Sat, 3 May 2008 17:14:00 -0400

When budget-building time comes up, many technology customers face the interesting question of how much money to put aside for new software licenses. Even without looking at specific vendors, you might have to tell your manager some ballpark figure for expected license costs.

As an analyst I'm frequently asked about license prices. A recent interesting discussion among peers challenged my views and provided helpful feedback that might assist you in arriving at the right numbers in today's marketplace:

List prices aside, buyers can presently obtain significant discounts on enterprise portals and on Web CMS tools. This may be caused by the increased SharePoint infiltration. A commentary in February on big software discounts and recent numbers from Vignette seems to confirm this trend. SharePoint licensing for websites is the exception that proves the rule. In general if the Web CMS comes from an ECM vendor, it will be more expensive -- potentially way more expensive

With enterprise search at the high end, the reverse is true. The marketplace is seeing strong demand at the moment. Many enterprise-tier search offerings come only as a bundled offering, so there is little list pricing to benchmark against. Deals quickly run into the millions of Euros in large, global, and complex enterprises.

Among the huge array of mid-market vendors across different content technologies -- many them local/regional in footprint -- you can typically find solutions that meet the needs of even organization-wide deployments in most enterprises, but at a factor of five (or more) cheaper than the higher-end solutions

If you are willing to serve as a reference client or appear on the customer list -- or better within a press release -- this is very valuable for the vendor and should help you to get significant discounts. (And of course as you look to evaluate vendors and they provide such testimonials, you should also understand how this game is played.)

Remember that enterprise deals entail complex negotiation and pricing models that ultimately boil down to what the salesperson thinks you can afford. Perhaps needless to say, but still: Implementation costs are higher than licensing costs and open source projects are not necessarily cheaper just because you might save licensing costs.

Thanks to Alan Pelz-Sharpe, Apoorv Durga, James Robertson and Martin White.

Thoughts on SharePoint and FAST Search

shawn_shell@consejoinc.com(Shawn Shell) — Fri, 25 Apr 2008 18:11:00 -0400

In our SharePoint Report 2008 we discuss SharePoint's shortcomings and strengths in the search space. While SharePoint 2007's search capabilities have been improved over the 2003 product, it's still not "enterprise class" for a variety of different reasons.

Clearly Microsoft saw this same shortcoming (both in SharePoint and it's overall search offerings) and announced that they were going to acquire enterprise search vendor FAST Search and Transfer (more information on FAST can be found in our Enterprise Search Report 2008).

For SharePoint users, this brings up a few opportunities and issues. In a previous blog post about the SharePoint conference, I highlighted the presentation that FAST employees gave. This presentation showed nifty new Silverlight-enabled search Web Parts. These Web Parts demonstrated several capabilities that FAST brings to the SharePoint world, like: content spotlighting, multimedia search, and taxonomy management.

The last capability is one that I believe would be particularly interesting for SharePoint users, since taxonomy management represents a challenging area for most SharePoint implementations -- SharePoint taxonomies are very rigidly based on physical structure of the SharePoint sites and leave little flexibility for a more logical taxonomic structure (Redmond folks would argue that the content query Web Part might help, but it's not a solution). That said, what Microsoft hasn't really provided is good guidance on what users can expect to see and any migration path between SharePoint search and FAST. Unfortunately, that really hasn't changed: Microsoft hasn't released any roadmap for FAST's integration, though you could argue it's still early.

That aside, what could the FAST acquisition mean to SharePoint customers? Here are a few of my thoughts:

The FAST acquisition was officially completed today. This is a good thing, since Microsoft can now get down to the business of integrating the company's technology with the rest of Microsoft's products. However, it's not clear what, if any, impact this acquisition will have in the very short term. Judging by the Groove acquisition, which preceded the SharePoint 2007 release by some months, an acquisition after the release will not yield any updates in the core product until SharePoint vNext (probably around 2010 or so) and beyond. However, I'd be willing to bet that some elements of the demo given at the SharePoint conference make their way to sites like Codeplex or as free downloads on Microsoft's site.
The former CEO of FAST will assume the role of VP of Enterprise Search. His responsibility will include all search products: SharePoint search, Search Server Express and FAST ESP. So I guess the "Enterprise Search" moniker might need to be removed or rewritten on the SharePoint search page; FAST is, by far, the new "enterprise" search product at Microsoft. What's interesting here is that Microsoft has historically brought Office-related technologies under one roof; just look at what happened to SharePoint specifically -- that product was once its own product group. As they integrate FAST, it would appear that this announcement suggests Microsoft might break out search into its own dedicated team and make SharePoint a "customer." This opens up the possibility of decoupling SharePoint from any particular search technology -- perhaps a pipe dream, but we can always hope.
In a blog entry on the Enterprise Search blog, Microsoft stated that the FAST offering will continue its Linux and Unix support. The blog entry was quick to reinforce the message that Microsoft does not want to support or wish to invest in Linux or Unix solutions. While they would like to "delight a core part of FAST's customer base," they are openly hoping those customers will convert to Windows and .NET. This does call into question whether this Linux/Unix support will be long for the world. In the short term, however, Microsoft can boast a better product from which to launch a play to be a real contender in the enterprise search space; see our related blog entries and article on FAST here. In the long term, Microsoft will have to come to grips with the fact that enterprises will continue to leverage non-Windows technology and if Microsoft wants to benefit from that revenue, they should consider continuing FAST's support for those technologies (their recent earnings announcement, and the subdued guidance for the year, may reinforce that message).
Microsoft won't support SharePoint on Linux <gasp!>. This is probably not a surprise to anyone who is even vaguely familiar with Microsoft. However, the fact that FAST (currently) is supported on Linux may introduce greater content aggregation and, certainly, search capabilities within SharePoint. Let's hope that Microsoft sees it that way.

In general, the FAST acquisition, for SharePoint, will likely have little impact short term. Over the longer term, it's clear that the Office 14 version of SharePoint will be substantially improved in the search area (depending on the SharePoint product team's willingness to implement the new technology). I would personally like to see some add-ons in the near term, since that would improve search within organizations that may have both tools.

Stay tuned for more on this topic as the integration progresses.