https://github.com/lfarrell/Constraint-Analysis

The State Library of North Carolina, in collaboration with the North Carolina State Archives, has been archiving North Carolina state agency web sites since 2005 and social media since 2009.  Since then, we have crawled over 82 million documents and archived 6 terabytes of data.