• P-ISSN 0974-6846 E-ISSN 0974-5645

Indian Journal of Science and Technology


Indian Journal of Science and Technology

Year: 2015, Volume: 8, Issue: 36, Pages: 1-7

Original Article

Design of Web Crawler for the Client - Server Technology


Search engines store information locally with the purpose of deliver quick, accessible search abilities. This information is collected by Web crawler. Web crawling is necessary for the maintenance of complete and latest web document gathering for a web search tool. The web document modifies its content on regular basis hence it becomes necessary to build up a successful framework which could identify these sorts of changes proficiently in the most minimal scanning time to accomplish this modifications. The essential thought behind designing of such a web crawler is to find high quality web documents within limited time frame. The proposed system works on the Client-Server Technology it reduces the overlap problem and downloads high quality web pages. We can add many web crawler in parallel to download web page in parallel way.
Keywords: Client-Server Technology, Overlap, Search Engine, Web Crawler, Web Page


Subscribe now for latest articles and news.