An Approach for Identifying URLs Based on Division Score and Link Score in Focused Crawler
|
10.5120/643-899 |
Debashis Hati and Amritesh Kumar. Article: An Approach for Identifying URLs Based on Division Score and Link Score in Focused Crawler. International Journal of Computer Applications 2(3):48–53, May 2010. Published By Foundation of Computer Science. BibTeX
@article{key:article,
author = {Debashis Hati and Amritesh Kumar},
title = {Article: An Approach for Identifying URLs Based on Division Score and Link Score in Focused Crawler},
journal = {International Journal of Computer Applications},
year = {2010},
volume = {2},
number = {3},
pages = {48--53},
month = {May},
note = {Published By Foundation of Computer Science}
}
Abstract
The rapid growth of the World Wide Web (WWW) poses unprecedented scaling challenges for general-purpose crawlers. Crawlers are software which can traverse the internet and retrieve web pages by hyperlinks. The focused crawler of a special-purpose search engine aims to selectively seek out pages that are relevant to a pre-defined set of topics, rather than to exploit all regions of the Web. Focused crawler is developed to collect relevant web pages of interested topics from the Internet. Maintaining currency of search engine indices by exhaustive crawling is rapidly becoming impossible due to the increasing size of the web. Focused crawlers aim to search only the subset of the web related to a specific topic, and offer a potential solution to the problem. In our proposed approach, we calculate the link score based on average relevancy score of parent pages (because we know that the parent page is always related to child page which means that for detailed information any author prefers the child page) and division score (means how many topic keywords belong to division in which particular link belongs). After finding out link score, we compare the link score with some threshold value. If link score is greater than or equal to threshold value, then it is relevant link. Otherwise, it is discarded. Focused crawler first fetches that link which has greater value compared to all link scores and threshold.
Reference
- X. Zhang, T. Zhou, Z.Yu and D.Chen, “URL Rule Based Focused Crawlers”, IEEE International Conference on e-Business Engineering, 2008.
- A. Pal, D. S. Tomar and S.C. Shrivastava. “Effective Focused Crawling Based on Content and Link Structure Analysis”, (IJCSIS) International Journal of Computer Science and Information Security, Vol. 2, No. 1, June 2009.
- Y. Zhang, C. Yin and F. Yuan. “An Application of Improved PageRank in Focused Crawler”, Fourth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD 2007IEEE).
- Q. Cheng, W. Beizhan and W. Pianpian. “Efficient focused crawling strategy using combination of link structure and content similarity”, IEEE 2008.
- X. Chain and X. Zhang. “HAWK: A Focused Crawler with Content and Link Analysis”, IEEE International Conference on e-Business Engineering, 2008.
- S. Chakrabarti, M. van den Berg and B. Dom. “Focused crawling: a new approach to topic-specific Web resource discovery”, 8th International WWW Conference, May 1999.
- M. Yuvarani, N. Ch. S. N. Iyengar and A. Kannan, “LSCrawler: A Framework for an Enhanced Focused Web Crawler based on Link Semantics” in Proceedings of the 2006 IEEE/WIC/ACM International Conference on WebIntelligence.
- Novak, B., “A survey of focused web crawling algorithms”, in Proceedings of SIKDD 2004 at Multiconference IS. 2004, ACM Press: Slovenia. p. 55-58.
- Sergey, B., Lawrence, Page. “The anatomy of a largescale hypertextual Web search engine”, Computer Networks and ISDN Systems 1998. 30(1-7): p. 107-117.
- Davison, B.D. “Topical locality in the Web”, in Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval 2000: Athens, Greece. p. 272-279.
- Altingovde, I.S., Ulusoy, O. “Exploiting interclass rules for focused crawling”, IEEE Intelligent Systems, 2004. 19(6): p. 66-73.
Key words
