PS: Good question. I'd like to know too. The web pages we're talking about are already OA at the NSF site.
Update. Bill Hooker at Open Reading Frame wrote to the NSF webmaster and got a direct answer:
NSF blocks all indexing of the site between 7AM and 7PM ET, our peak traffic hours, for the convenience of our users. However, there is no block on the site from 7PM to 7AM ET. This is standard policy for most high traffic sites. The owner of [the Wayback Machine] need only comply with our policy in order to index our pages.
(Thanks, Bill.)
Posted by
Peter Suber at 7/07/2006 09:56:00 AM.
The open access movement:
Putting peer-reviewed scientific and scholarly literature
on the internet. Making it available free of charge and
free of most copyright and licensing restrictions.
Removing the barriers to serious research.
I recommend the OA tracking project (OATP) as the best way to stay on top of new OA developments. You can read the OATP feed on a blog-like web page or subscribe to it by RSS, email, or Twitter. You can also help build the feed by tagging new developments you encounter.