perl-WWW-RobotRules - Database of robots.txt-derived permissions
| Website: | http://search.cpan.org/dist/WWW-RobotRules/ |
|---|---|
| License: | GPL+ or Artistic |
| Vendor: | Alcance Libre, Inc. |
- Description:
This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>. Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.
Packages
| perl-WWW-RobotRules-6.02-18.fc14.al.src [15 KiB] |
Changelog
by Fedora Release Engineering (2018-02-09):
- Rebuilt for https://fedoraproject.org/wiki/Fedora_28_Mass_Rebuild |