Difference between revisions of "Aboutus:Bot"
(→How do I prevent the bot from gathering info about my site) |
|||
Line 1: | Line 1: | ||
{{DISPLAYTITLE:The AboutUs Bot}} | {{DISPLAYTITLE:The AboutUs Bot}} | ||
− | The main job of the AboutUs Bot is to generate basic pages and analysis about websites. | + | The main job of the AboutUs Bot is to generate basic initial pages and analysis about websites. The bot pulls initial page data once when a page is first create. Website analysis may be pulled multiple times but is cached to prevent continuous access by the bot. We want the bot to be well behaved, if you are seeing otherwise please [[help/contact contact us]] and let us know. |
− | == | + | ==User-Agent String== |
− | + | The AboutUs Bot User-Agent string contains the following: | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
:: '''<nowiki>AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)</nowiki>''' | :: '''<nowiki>AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)</nowiki>''' | ||
Line 18: | Line 11: | ||
The current AboutUs Bot version is <strong>Harpy</strong>. | The current AboutUs Bot version is <strong>Harpy</strong>. | ||
− | + | ==Blocking the AboutUs Bot== | |
+ | Using a [[Learn/How-To-Use-Robots.txt|robots.txt file]], you can choose to not have the About Us Bot access your website. This doesn't mean that we won't create a page for your website. Our members still have the opportunity to contribute their own content describing your site. | ||
+ | |||
+ | To prevent the AboutUs Bot from accessing your site in the future, please include the following lines in your /robots.txt file. | ||
+ | |||
+ | :: '''User-agent: AboutUsBot''' | ||
+ | :: '''Disallow: /''' | ||
− | + | Other supported bot prevention methods | |
:The AboutUs Bot will also honor a rule like this in your robots.txt file: | :The AboutUs Bot will also honor a rule like this in your robots.txt file: | ||
Line 28: | Line 27: | ||
''For more information about [[Learn/How-To-Use-Robots.txt|robots.txt]], read [[Learn/How-To-Use-Robots.txt|this article]].'' | ''For more information about [[Learn/How-To-Use-Robots.txt|robots.txt]], read [[Learn/How-To-Use-Robots.txt|this article]].'' | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Revision as of 22:15, 3 December 2013
The main job of the AboutUs Bot is to generate basic initial pages and analysis about websites. The bot pulls initial page data once when a page is first create. Website analysis may be pulled multiple times but is cached to prevent continuous access by the bot. We want the bot to be well behaved, if you are seeing otherwise please help/contact contact us and let us know.
User-Agent String
The AboutUs Bot User-Agent string contains the following:
- AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
For example:
- AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
The current AboutUs Bot version is Harpy.
Blocking the AboutUs Bot
Using a robots.txt file, you can choose to not have the About Us Bot access your website. This doesn't mean that we won't create a page for your website. Our members still have the opportunity to contribute their own content describing your site.
To prevent the AboutUs Bot from accessing your site in the future, please include the following lines in your /robots.txt file.
- User-agent: AboutUsBot
- Disallow: /
Other supported bot prevention methods
- The AboutUs Bot will also honor a rule like this in your robots.txt file:
- User-agent: *
- Disallow: /
For more information about robots.txt, read this article.