Enterprise Search Vendor Landscape, Circa 2008

tregli@cmswatch.com(Theresa Regli and Adriaan Bloem) — Tue, 19 Feb 2008 00:16:00 -0500

You might be tempted to select enterprise search vendors for your shortlist based on their supposed "leadership" status in the market -- status either conferred by analyst firms or assumed by the vendors themselves. However, CMS Watch analyst Theresa Regli argues that you need to look more closely at product and vendor alike -- and understand where both are headed -- to properly evaluate your longterm risks and opportunities in an evolving marketplace...

Search the X-Files: unknown entities

bloem@radagio.com(Adriaan Bloem) — Fri, 21 Sep 2007 07:46:00 -0400

If you're in the market for search technology, you probably hear a lot about faceted browsing, guided navigation, refining, clustering, categorization, and so on. Many of today's search engines attempt to present more than just keyword search. That's fine if your content has high-quality structured metadata, but what if you throw in thousands of Word documents where the "author" is defined as John Doe? The truth may be out there -- but the answer is buried deep in the text.

Distilling things like people, email addresses, and company names from source content is what is known as "entity extraction." Vendors may tell you that yes, their search interface pivots off that kind of data (e.g., for guided navigation), but don't worry: they can extract the unknown entities even if you throw in large files nobody ever bothered to tag right. Enterprise search will create and then reveal structure where once there was chaos.

Of course, this is not at all the black magic it is made out to be. Finding relevant entities is usually accomplished through a combination of pattern-matching and dictionaries. An email address will contain the "@" symbol, and it's pretty safe to say that if it's followed by a dotted domain name, you've got your address. If "John" is in your dictionary of first names, the next capitalized word will probably be the surname. This also means that entity extraction is language- and even country-specific. A representative of Fast Search & Transfer's professional services told me about the challenges the company faced finding a fail-safe way of distilling German street addresses, which have a very different and much less formal structure than those in, say, North America.

Many vendors, of course, won't like you to be distracted with the details of their "automagical" ways of achieving this. Their method may be English- and US-specific, but hey, so what -- if your company is based in the US and content comes in English, you're fine. In reality, things are never that easy though.

I was running a test of ISYS:web against the CMS Watch website, and was pleasantly surprised to see the out-of-the-box installation correctly identified several countries, and had no problems finding out that Tony Byrne is an actual person. It even managed to extract Janus Boye's somewhat more exotic Danish name. Unsurprisingly, Apoorv Durga was a bit too outlandish and my ego wasn't hurt when Adriaan Bloem wasn't ranked among the people. But you really don't want to provide Theresa Regli with cannon fodder by ignoring her (which it did), while on the other hand, I can't recall ever having met "Read More," international man of mystery, now a full-fledged person in my search engine.

This is not to say you should bash ISYS for this -- the company is the first to admit its methods aren't infallible, and many vendors at a much higher price point don't even offer similar technology, instead relying on third-party tools. What it does mean, however, is you shouldn't take claims that "it's all taken care of" at face value. Investigate whether languages and countries relevant to you are supported, and better still, test against your own content. Then assume you are committing yourself to near constant system training and tweaking.

Failing that, some search products will allow you to specify additional criteria (with ISYS, for instance, "Theresa" was easily added with a [pre] construct in a text-based configuration file). Others enable you to define completely new entities and patterns from scratch (such as FAST's processing in Python, or Endeca's XSL and Perl). Be very aware, though, that sending in a Mulder agent to investigate your X-Files might be a costly, ongoing adventure, lasting nine seasons of suspense.

Verity Acquires 80-20 Software

tbyrne@cmswatch.com(Tony Byrne) — Tue, 14 Jun 2005 11:35:00 -0400

"Big 4" enterprise search vendor Verity has acquired desktop search vendor 80-20. The growth of desktop search is compelling major vendors to provide solutions that span from desktop to enterprise indexes. Will independent desktop search vendors ISYS and Copernic be swallowed next? The latter has conveniently spun off its server-based search tools as a separate company.

Industry Solutions for Search

tbyrne@cmswatch.com(Tony Byrne) — Tue, 8 Feb 2005 00:00:00 -0500

As we've noted previously, intelligence and law enforcement have proven fertile ground for enterprise search vendors. One supplier, ISYS, has found a very cozy niche with search solutions for local law enforcement. ISYS wins in part on the strength of its longstanding desktop search product (you can read more about ISYS in our Enterprise Search Report). There are two small stories here: first, with the right engineering resources, toolkits like ISYS can can be molded into viable industry solutions; and second, those solutions increasingly encompass a continuum of search search services from the desktop to intranet to enterprise...

CMS Watch ISYS Feed

Enterprise Search Vendor Landscape, Circa 2008

Search the X-Files: unknown entities

Verity Acquires 80-20 Software

Industry Solutions for Search