FREEWARE

The web crawler is a program that automatically traverses the web by downloading the pages and following the links from page to page. A general purpose of web crawler is to download any web page that can be accessed through the links.  
 
This process is called web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a website, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvest ing e-mail addresses (usually for spam).  
 
A web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.  
 
Requires the Java Runtime Environment

Please Login...

Rate it

Share or Bookmark

Jordy Video Downloader   1.04 Jordy Video Downloader is a free multi-segment download accelerator which downloads videos from sites like YouTube, DailyMotion, Google Videos, Break, MetaCafe,…

Raptor   v2.11b Raptor is a FREE Download Manager for Rapidshare, Letitbit, Deposit, Uploading... Automate your Free User downloading. Setup is fast and easy. Simply enter all your…

WinGet   3.0 WinGet is a download manager that can increase download speeds by up to 300 percent. You may automatically resume broken downloads if your connection is lost or if your…

Dead Simple Downloader   v2.00.4750 Easy paste and go downloader, reliable, robust, lightweight architecture / resources. Program Features: * Smart, built-in network management for reliable, hassle-free…

Torrent2Exe   1.0.10 Torrent2Exe will make torrent downloads a breeze even for computer novices. The application is a useful alternative to installing a client on your PC to download…

Gigaget   1.21 Gigaget utilizes cutting-edge Net Grid technology that improves your download speed by 7-10 times. It features:  Free and safe  100% free  No adware or spyware  No…

TSG NetLoader 1.0   Multi Threaded Internet File Download Manager, Includes URL File filter , Passive FTP download, Page Scan and list.  Intergrates into IE Right Click.

Octopus   3.11 Octopus is absolutely free FTP/HTTP download manager. It can download each file through several connections concurently. So it accelerates downloading. It is small and…

  More Related...