A quick note because I have to get out the door and go to work… gah.
I recently removed the restrictions on my robots.txt file.
This site has been denying access to most bots for the past year:
# Welcome to Tregeagle
######################
User-agent: archive.org_bot
Allow: /
User-agent: ia_archiver …