SAN FRANCISCO, Dec 3 – Google News on Wednesday launched a "news-specific crawler" that lets online media automatically keep stories, photos or video out of its index.
The announcement comes a day after the California-based Internet giant said it is letting publishers limit the number of online pages people can view after being routed to their websites by Google\’s search engine.
Publishers have always been able to block Google from including their website content in the search engine index.
Google senior business product manager Josh Cohen said in a blog post that a new "web crawler" extends that option to Google News.
Web crawlers are automated programs that scour the Internet for content and then index it in databases routinely mined for results to online search queries.
News website publishers can fill out online forms telling Google\’s crawler which content, if any, can be indexed.
Similar directives can be given to Google\’s main search engine, which operates separately from the firm\’s online news aggregation service.
"Most people put their content on the web because they want it to be found, so very few choose to exclude their material from Google," Cohen said.
"But we respect publishers\’ wishes. If publishers don\’t want their websites to appear in web search results or in Google News, we want to give them easy ways to remove it."
Google, under fire from Rupert Murdoch and some other newspaper owners, said Tuesday it will let publishers set a limit on the number of articles people can read for free through its search engine.
Google\’s announcement came as the News Corp. chairman, who has threatened to block the Internet giant from indexing his newspapers, and other US media heavyweights gathered in Washington to discuss journalism in the Internet age.
Murdoch has blasted Google and other news aggregators for "stealing" stories without sharing advertising revenue and has reportedly been holding talks with Microsoft about making News Corp.\’s content accessible exclusively through the software giant\’s new search engine, Bing.
Acknowledging that "creating high-quality content is not easy and, in many cases, expensive" Google said in a blog post it is changing its "First Click Free" program.
First Click Free directs readers from Google or Google News to a story on a newspaper\’s website but prevents them from having unrestricted access.
Google noted, however, that some readers were "abusing" the program by returning to Google or Google News and clicking through to other stories.
"Previously, each click from a user would be treated as free," Cohen said. "Now, we\’ve updated the program so that publishers can limit users to no more than five pages per day without registering or subscribing."
