(picture)

January 05, 2005

Real News in RSS

Some notes on how to find "real news" -- by which I mean newspaper/newswire material rather than weblog commentary -- and consume it with RSS. An aide-memoire.

I don't deal with "what" or the "why" of RSS here. (Also I don't care what newsreader you use).

I've probably missed some very major sources here. If so, call me, please!

Newspapers, radio, etc

Most of the big mainstream media outlets roll their own RSS feeds. Most of these are simply organized by category, and don't allow you to produce a feed of "search results within Business News", for example.

NYT. Guardian. BBC. CNN. NPR.

If you can't find RSS feeds for your newspaper/etc, try searching for them at Syndic8 - a very big catalog; at BlogLines - a catalog of all the feeds bloglines users subscribe to, which is just about everything on the planet; or maybe at Technorati, which is a weblog/feed tracker.

Local and Specialist sources

If you have a local/regional newspaper without a feed, it'll be worth calling their IT guys and ask them very nicely to set up an RSS feed. It's easy, but usually best done by the IT department's perl hacker, rather than by the editorial people. Then you can also ask for a keyword search feature...!

Many of the best specialist sources (Alertnet, for example) don't produce RSS at all. Again, it's worth asking them to. Alertnet is big enough to be syndicated various places, though, as we'll see.

If you're completely out of luck, it's sometimes possible to hire someone to write a "scraper": a small Web program which reads a site's HTML and produces rough-and-ready RSS from it. Check the howto docs at syndic8 for some pointers.

Keyword Search

Yahoo News is the place to go. Start at their Advanced News Search page. Enter your keywords/conditions, and press the search button. On the right-hand side of the results page you'll see an orange button: View as RSS: . Click that button, to bring a page of RSS into your browser window. Copy the URL of this page, and subscribe to it.

I haven't tried Yahoo's language-specific search. I tried selecting specific categories, and it seemed to be OK, but I suspect you might be better to refine the search terms than to restrict to particular categories. Try both, and see.

Google News has keyword search, and vast amounts of news, but no RSS. Here's an experimental scraper. Results are mixed. If you use a too-generic search, you'll likely find many many versions of exactly the same news item - from all the local newspapers across America who republish the AP wire. But a search which includes a source-specific keyword (tsunami alertnet), or exclusions (tsunami -"press release"), can work really well.

MSN Search, in Beta, I haven't tried it.

Weblogs

Often, you can find a good weblog you're specifically interested in. Example: http://bdheart.blogspot.com/. Finding them might be harder; use word-of-mouth, or the weblog search engines.

Feedster, Blogdigger and Technorati not only search lots of weblogs, they also produce search results as an RSS feed. The search at Bloglines is good. PubSub should also work for this, but hasn't come through for me yet.

Serendipity Engines

Try http://flickr.com/photos/tags/....someword... or http://del.icio.us/tag/...someword.... You might get lucky.

Commerical News Providers

Finally, there are dozens of good news providers who'll take your money and deliver you news. I've dealt with Moreover and Factiva before; they have content, and moreover produce good RSS.