Categories


robots.txt class

Raiting 0.00

Description

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Information
Title: robots.txt class
Author: toong
Price / Cost: GPL
Platform / OS: Independent
Version: 1
File Size: 3 KB
Last Updated: 2005-12-02
URL: Click to Visit
Web site Language: English
Total Votes: 5
Current Rating: 0.00
Total Hits #: 70
Listings ID #: 41075

Need Scripts Message

We at Need Scripts, help You For Free, You can help us to serve you even better!

  1. Put a link, banner or button of Need Scripts on your website.
  2. Become a Need Scripts Sponsor or Advertiser.
  3. Tell your newsgroup and/or mailing list about Need Scripts.