RSS Syndication
Syndication on the Cheap with RSS
by Karl Fast
06-Aug-2001
If you want to add news headlines to your site or make your headlines available to others, you can take the high road or the low. The high road is what you'd expect: high quality, lots of features, expensive, and marketing literature that make it sound both wonderful and complex. But since this is the Internet, there is a compelling low road alternative that's simple, easy to implement, and free (or cheap).
The low road alternative is based on Rich Site Summary (RSS)--an open XML format for syndicating news headlines. Each item in an RSS news feed includes the title of the article, a brief description, the date it was published, and a link back to the article. Note that RSS does not include the article text itself, only a link to it. If you need full content syndication, take the high road (and reach for your wallet).
Stunning Simplicity
Although it sounds complex, RSS is stunningly simple. It's just an XML file sitting on a remote web server, similar to the HTML documents, GIFs, and other files on a web server. The only differences are (1) it's an XML file and (2) it's meant to be read by a software agent, not viewed directly in the browser.
But note what's not required: no special servers, commercial databases, or proprietary content management systems. RSS is popular precisely because it's easy to understand, easy to implement, isn't overloaded with unnecessary features, and isn't owned or controlled by a corporation or some Byzantine standards organization. RSS evolved organically, not officially. Perhaps this is why it has become the de-facto standard for delivering news headlines over the Internet.
You can use whatever is convenient to generate the RSS file, meaning that RSS can be integrated into any web site no matter how simple or sophisticated the back-end technology. The truly desperate could even create the RSS file by hand in a text editor!
Adding News Feeds to Your Site
How does a headline get from the RSS file on a remote server to your web site? The RSS distribution model works like this. The website that is providing the syndicated news headlines creates an RSS file listing up to 15 items, or stories. Each item has a title, a description, and the URL linking to the full text of the story.
Structure of a fictional item syndicated by salon.com
There is also some general information about the RSS file itself--the "channel"--including the name of the channel, when it was last updated, and the URL of the RSS file. When a new story is added to the site the RSS file is updated to reflect this change.
| Sample XML Format |
|
<?xml version="1.0"> <news-feed> <channel>Salon</channel> <url>http://www.salon.com/news.xml</url> <date>2001-APR-04</date> <news-item> <title>Wheelchairs, pig guts, computers...</title> <desc>Let me tell you what happened in... </desc> <url>http://www.salon.com/people/feature... </url> </news-item> ........ </news-feed> |
Structure of a fictional item syndicated by salon.com
Once the RSS file is made public, anyone is free to pull it down, read the list of headlines, reformat them, and post them to their site. The software required to do this is very simple and most programmers with even a little bit of web development experience should be able to quickly write a program that grabs an RSS file, parses it, and spits out some reformatted HTML. Perl hackers can get started even faster thanks to excellent tutorials at Perl.com and WebReference, and the wonderful XML::RSS module by Jonathan Eisenzopf and Rael Dornfest.
The RSS Content Chain, from syndicator to host site and back
Using Aggregators to Combine Newsfeeds
Most RSS-based newsfeeds come from smaller independent sites, but several big names --including Salon, Wired, and the Washington Post--also provide them.
If you want to include RSS-based newsfeeds on your own site, you could go to each syndication site and try to track down the RSS file yourself. But that's harder than it sounds, because most sites don't do a good job of publicizing the URL for their RSS feeds. Thankfully, several sites aggregate multiple RSS feeds into a single newsfeed which you can then split up and filter and republish however you wish.
A nice advantage to using an aggregator is that they provide thematically-focused feeds on a single subject (like "E-commerce") from multiple news sources. Two of the best-known aggregator sites are Meerkat and Moreover.
Meerkat is run by O'Reilly & Associates and was developed by Rael Dornfest. It focuses strictly on technology-related news. The Meerkat site offers plenty of information on how to exploit their aggregated feed for your own purposes.
The service provided by Moreover is more ambitious. It doesn't limit itself to just technology news and does some work to categorize the news stories. Adding news from Moreover is especially easy thanks to an online wizard that takes you through the process from start to finish in less than five minutes. (The wizard provides with sample Javascript code to insert into your HTML pages or templates). Moreover also has fee-based services that give you richer content and advanced tools for getting more out of their feed.
Of course RSS is not the last word in syndicated news headlines and it does have its limits. Not everybody provides an RSS feed. RSS only carries headlines, not content. And it doesn't have a mechanism for categorizing the headlines or extending RSS for your own needs (these last two limitations are being addressed in new versions of RSS).
If you consider these to be serious limitations then you might need to take the high road. If so, you might want to try a commercial vendor like Factiva or take a closer look at the the commercial services provided by Moreover. There is no doubt that some people need this type of high quality service, but for the rest of us there's always the glories of RSS.
Resources on the Web
The Evolution of RSS by Andrew King
RSS Delivers the XML Promise by Derrick Story
Syndicate Your Content with RSS by Kevin Savetz
RSS and You by Chris Nandor
Using RSS Newsfeeds by Jonathan Eisenzopf
The RSS DevCenter
RSS Frequently Asked Questions for Content Providers, by Alis Marsden