Keep UI archivable by Heritrix web crawler

Hi @ojsbsb,

As long as the crawl results aren’t public, that’s fine. (The test data closely resembles scholarly content so I’d hate to see it get somehow added to a search engine; that’s why we’ve got no-index set in the .htaccess file.) The feedback would be welcome. Note that the current beta level front-end for readers is fairly minimal.

Regards,
Alec Smecher
Public Knowledge Project Team