CMS Watch Thunderstone Feed

How Fast is Attivio?

bloem@radagio.com(Adriaan Bloem) — Sat, 5 Jul 2008 09:55:00 -0400

The Fast Search & Transfer news continues to keep me on my toes. The Norwegian business weekly Dagens Næringsliv seems to have come up with some decent evidence (English translation) of many things everybody already suspected -- and a couple of new ones.

It covers the Convera acquisition and the "separate" but surprisingly coincidental deal where Convera bought several million's worth of Fast software it didn't need. Or as rival Autonomy (which was also in the running for buying Convera) has pointed out, Fast pumped up its revenues for that quarter with part of the money it paid for Convera, then got back for licenses. The DN article also covers a few other very suspicious deals, and some outright fraud. It's now even getting to the point where calling Fast "the Enron of Norway" is getting long in the tooth.

While that train wreck was unfolding before my eyes in slow motion, my fellow analyst Theresa Regli pinged me last February about a new enterprise search company called Attivio. Information Today raved about their new product AIE, with analysts quoted as saying things like "they are moving rapidly to develop tools that will eliminate many of the practical barriers to easily and efficiently deploy robust enterprise search solutions," with the unique selling point of "data integration plus search and content processing," a "hot niche for the next few years."

Since I'm always interested to find out more about robust enterprise search tools to fill hot niches for the next few years, I scrolled down to read what the Attivio CTO would explain about how the product would achieve what "should have been solved by the integration of text search and XML into relational database managers such as Oracle." As it turns out, it is based on a "mash-up" of open source Apache Lucene and "licensed commercial software."

As described in our Enterprise Search Report and on this blog, Lucene itself is just a Java text search API. To be able to actually gather, convert, and query content you need many more components. It is perfectly feasible to put together a working enterprise search product around the core Lucene JAR (as demonstrated by IBM's Omnifind Yahoo! Edition). But in order to get there, and to have Lucene index, for instance, Office documents and PDFs, you will have to first convert those documents to text. The filters to perform that conversion can be bought from other vendors, based on open source such as pdftohtml, or you'll have to build them yourself, which is a lot of work. There aren't too many vendors building their own filters, or even just modifying open source to do so. So if you do build the filters needed to use Lucene yourself, you'd probably like to mention this as an advantage, and as Attivio states, "we developed our own Microsoft Office, WordPerfect, and PDF connectors to improve performance and reach deeper into the files than the conventional converters."

Since, like most enterprise search products, Lucene isn't based on a database and couldn't even connect to such content without help, it isn't surprising Attivio had to develop a "unique RDBMS data loader" which "indexes the tables individually." This, again, is presented as a major advantage -- remember, converting documents and integrating structured and unstructured data are "a hot niche."

I remember seeing a vendor at a conference a few years back, with banners jokingly stating its product was "buzzword compliant!" Attivio certainly seems to have that skill down. The engineering effort is marketed as a "technology mashup," "breaking down silos" between "open source and commercial software." "We have lived with the challenge of having to choose between the precision of databases and the richness of search for a long time, but no longer" sounds great, but I don't see Oracle and Thunderstone's RDBMS-based solutions breaking out in a sweat just yet.

Maybe my over-exposure to marketing materials and flashy demos has turned me into a cynic, and Attivio's downloadable trial version will have to do at least a decent job to convince me of the product's added value. Fortunately, that free download is "coming soon!" Yes, I'm sorry, I'm finding it increasingly hard to turn off that cynicism, especially when I turn back to the DN article about Fast Search & Transfer. Attivio was founded by former Fast employees and the Attivio CTO is Sid Probstein, formerly vice president of technology at FAST. More importantly, Attivio's CEO is Ali Riaz, who was COO at Fast but unexpectedly left the company in late 2006. Well, in hindsight, perhaps not so unexpectedly, though DN quotes him as saying "I had nothing to gain from manipulation of the accounts. I had no shares in the company. I wanted shares and quit because I didn't get any. If you want to find out what's wrong with the accounts, you need to look at those who could gain from it. And it wasn't me."

Dagens Næringsliv doesn't appear to agree with Riaz, however; if you want the full analysis of the what and why, I suggest you read the article. I myself find it surprising that the CEO of a technology startup backed by $6.2 million in venture capital would drive an Audi R8, but that doesn't mean anything (other than that I'm envious of his car). I also find it surprising a former Fast COO would be co-owner of a company reselling Fast licenses, but walking like an Enron duck and quacking like an Enron duck doesn't necessarily mean that it's really anything like Enron. And Attivio's clouding the core technology in marketing hyperboles and buzzword compliance is slightly disconcerting, but many renowned companies engage in the same practice.

DN quotes Riaz as saying "you should be much better at praising the people who have success, instead of pushing them down." And I would certainly love to be proven wrong by Attivio's software; as soon as I get my hands on the trial download I requested, I will let you know if it lives up to the high expectations. As one of my teachers in school once told me, "I'm known for being cynical, or even sarcastic -- myself, I prefer to call it healthy skepticism and mild irony."

Being a cynic isn't a lot of fun -- but for now, I would advise you to be at least healthily skeptical of what Attivio has to offer.

Enterprise search: free as in free beer?

bloem@radagio.com(Adriaan Bloem) — Fri, 16 May 2008 13:23:00 -0400

Searching information -- really, how hard can it be? So, why wouldn't you go out and get a search engine that's for free? Well, to stick to the analogy of "free beer," you might wake up in the morning with a headache, only to find your wallet gone.

Of course, I'm paraphrasing the definition of "free software". Richard Stallman's example is used to point out the ambiguity of the term "free" in the English language. With free software, "you should think of free as in free speech, not as in free beer." Nevertheless, you should be warned: both open source beer (now in version 3.3) and free commercial beer have the potential for leaving you with a bit of a hangover.

If you really think enterprise search is a simple commodity -- and I will only comment on that with the obligatory statement that readers of our Enterprise Search Report will probably know better than that -- getting a free product would be ideal to get your feet wet (albeit somewhat sticky). I get invited to BYOB enterprise search parties a lot, and usually come up with Apache Lucene, IBM Omnifind Yahoo! Edition, and Microsoft Search Server 2008 Express. Let's get a closer taste of each.

Apache Lucene. Lucene is open source, which you are free to use. The problem is, it's not a complete enterprise search product -- it's a "text search engine API." What you get is a Java JAR with the core functionality of a search engine. In typical hardcore Java developer understatement this is described as "you write the easy stuff, the UI and the process of selecting and parsing your data files to pump them into the search engine, yourself." To developers that doesn't sound too difficult -- it's a library they'd be able to use to create search functionality for many applications. As they embark on that journey, however, many will find out they'll have to become experts on enterprise search to get their implementation to perform basic tasks any Google user has come to expect. Index Word documents? You'll have to convert those to text first. Remove stop words or perform spell checking? You'll have to get some more jars to fit that in. And that familiar user interface isn't so easy to replicate, either.

Of course, there's a couple of more "pre-packaged," Lucene-based engines (such as Nutch and Solr), but they'll only take you so far on that long and winding road. There's some excellent examples of what you can achieve with Lucene, but many more of how hard it can be to get there.

IBM Omnifind Yahoo! Edition (or OY!E). The Google appliances have the Google brand behind them, which must have got the IBM people thinking the Yahoo! brand would be excellent marketing for their free-to-use search engine. In fact, it's neither IBM nor Yahoo's technology, but Lucene wrapped in other open source software. A few commercial bits thrown in create a product that's easy to install and run. It will actually do many of the things Lucene will make you work hard to accomplish: it comes with support for several languages and quite a few source content filters. For users, it looks like a regular web search engine; for admins, there's a nicely designed and intelligible interface. In short, it does most of the things a Google Mini appliance will do -- but for free.

So what's the catch? Well, the license (by the way, what license?) limits you to 500,000 documents and 5 collections. After that, you can "upgrade" to other Omnifind products. But since the technology across the Omnifind line-up is completely different, this is the same as starting from scratch, and you'll pay for the privilege. I've been critical of the limitations of Google's appliances in the past, and sure, the 50,000 document limit of the entry-level Google Mini is a lot less than OY!E's half a million. But that comparison isn't really fair, considering the fact the Mini actually comes with the hardware to run the queries on for a mere $2,990. And don't think you'll be able to run IBM's software on an old abandoned test server you have available -- OY!E will need more power than the single blade Google Mini or Thunderstone Appliance to match the performance. Tellingly, I wasn't able to dig up an example of an OY!E implementation to mention while researching the Enterprise Search Report (if you know of one, let me know).

Microsoft Search Server 2008 Express. Microsoft's free offering is basically the same software as the non-Express version, but then there's the seemingly innocent limitation: one server only. I wouldn't want to continue the theme of this post by saying this is akin to handing out free samples of beer to get you hooked; suffice it to say that if you start to run the Express version in a production environment, there will, no doubt, come a time when a single server won't be enough anymore. When you've come to rely on the solution, you'll suddenly have to shell out for the licenses. As I've said before, having a free lunch isn't necessarily a bad thing; just remember that you'll probably have to pay for the beer the lunch comes with.

So, this might all start sounding like advice your mother gave you: never take anything from a stranger, and certainly no free alcoholic beverages. Don't forget, however, that I'm Dutch, and I've certainly developed a taste for enterprise search. Free beer sounds too good to be true, but it could certainly get your party started; just remember to drink in moderation, and never, ever, drink and drive.

Enterprise Search Vendor Landscape, Circa 2008

tregli@cmswatch.com(Theresa Regli and Adriaan Bloem) — Tue, 19 Feb 2008 00:16:00 -0500

You might be tempted to select enterprise search vendors for your shortlist based on their supposed "leadership" status in the market -- status either conferred by analyst firms or assumed by the vendors themselves. However, CMS Watch analyst Theresa Regli argues that you need to look more closely at product and vendor alike -- and understand where both are headed -- to properly evaluate your longterm risks and opportunities in an evolving marketplace...

Shame on Google

tbyrne@cmswatch.com(Tony Byrne) — Tue, 6 Sep 2005 16:52:00 -0400

Google has removed a direct competitor, Thunderstone Software, from its index. Thunderstone markets an enterprise search appliance that competes directly with Google's own appliance. Neither Yahoo! nor MSN have blacklisted the company. As of Monday, 05 September, the following queries yielded no hits from Google for the Thunderstone website at thunderstone.com:

Thunderstone (live) (05 Sept, 2005)
Thunderstone appliance (live) (05 Sept)
software site:thunderstone.com (live) (05 Sept)
Thunderstone -- Yahoo (live) (05 Sept)
Thunderstone -- MSN (live) (05 Sept)

In theory, Google could have removed thunderstone.com for spamming the index, but that seems quite unlikely (Technorati shows a scant 12 links to thunderstone.com). At this writing, Google has not yet responded to a request for comments. I've never cottoned to conspiracy theories and don't believe in any future Google-led dystopia, but it's chilling -- dare I say "evil?" -- when a public search provider de-indexes a competitor of its enterprise product line.

David to Google's Mini Goliath?

tbyrne@cmswatch.com(Tony Byrne) — Wed, 2 Feb 2005 03:00:00 -0500

Longtime search vendor Thunderstone innovated the search appliance approach. But pity the vendor that has to stand up to Google. Today, Thunderstone announced an alternative to Google Mini. Same price, same doc limit, more features, and arguably stronger technology with a deeper history. It is a sad fact of the software business, though, that better technology doesn't always win. Still, we think Thunderstone will stick around...