I have been asked about what we need for robotic access to publishers’ sites. Several publishers are starting to allow robotic access to their Open material. (Of course the full BBB declarations logically require this, but in practice many publishers haven’t made the connection). So let’s assume a publisher who espouse Open Access and allows robotic access to their site. Is, say, CC licence enough?
There are no moral problems with CC, but the use of robots has additional technical problems, even when everyone agrees they want it to happen....
I can see roughly two types of robotic behaviour:
systematic download for mining or indexing....It would be highly desirable to minimise repetitious indexing and an enthusiastic publisher could put their XML material in a proper repository framework with a RESTful API (rather than requiring HTML screen-scraping of PDF-hack-and-swear). In return there could be a list of acknowledged robots so that these could act as “proxies” or caches.
Random access from links in abstracts or citations. This is likely to happen when the bot is in PMC/UKPMC, or crystaleye, and discovers an interesting abstract and goes to the full-text on a publishers site. The bot may have been created by an individual researcher for a single one-time purpose.
So I’d like to come up with (three?) laws of mining robotics. Here’s a first shot:
A publisher should display clear protocols for robots, with explanations of any restrictions and lists of any regular mining bots.
A data-miner should use software that is capable of honouring machine-understandable guidance from servers. The robots should be prepared to use secondary sites.
Mining software should be Open Source and should honour a common set of public protocols.
But I would like suggestions from people who have been through this…
Posted by
Peter Suber at 9/20/2007 02:15:00 PM.
The open access movement:
Putting peer-reviewed scientific and scholarly literature
on the internet. Making it available free of charge and
free of most copyright and licensing restrictions.
Removing the barriers to serious research.
I recommend the OA tracking project (OATP) as the best way to stay on top of new OA developments. You can read the OATP feed on a blog-like web page or subscribe to it by RSS, email, or Twitter. You can also help build the feed by tagging new developments you encounter.
