This report is a follow-up to the McCown et al. article in IEEE Internet Computing two years ago, in which the researchers investigated the percentage of URLs from OAI records in Google, Yahoo and MSN search indexes. We were interested in whether Google in particular had increased the number of OAI-based resources in its search index.
To this end, we used a slightly different methodology using the OAIster metadata corpus to see what percentage of the corpus was found in the Google search index only. OAIster harvests and aggregates OAI metadata with links to digital resources – those without links to digital objects are removed during our transformation and indexing process....
Google's indexing does not seem to have retrieved more of the hidden web since the publication of the McCown, et al. article in 2006. We would venture to conclude that Google has not endeavoured to increase their support and access to OAI materials. Even taking into account the caveats, we would also conclude that aggregations of OAI records are as valuable for user research purposes as they were at least two years ago.
From our own experience, we know that providing the OAIster records in bulk to Google proved problematic for them, and eventually they requested only the OAIster URLs instead of the complete metadata. We are not, at this point, certain that Google is using these URLs (crawling them) for addition to their search index.
It is also interesting to note that Google has recently dropped support of OAI for website indexing. Given the resulting numbers from our investigation, it seems that Google needs to do much more to gather hidden resources, not less. (Granted, the OAI for Sitemaps feature may not have been an appropriate approach for Google.) ...
We would...like to encourage other OAI aggregators to run their metadata against the Google index, to prove or disprove our conclusions. Our source code and raw data are available upon request.
Update (7/29/08). Also see the note on this article in Wired Campus, the blog of the Chronicle of Higher Education. See especially the comments from readers.
Posted by
Peter Suber at 7/16/2008 01:13:00 PM.
The open access movement:
Putting peer-reviewed scientific and scholarly literature
on the internet. Making it available free of charge and
free of most copyright and licensing restrictions.
Removing the barriers to serious research.
I recommend the OA tracking project (OATP) as the best way to stay on top of new OA developments. You can read the OATP feed on a blog-like web page or subscribe to it by RSS, email, or Twitter. You can also help build the feed by tagging new developments you encounter.
