Employing a novel content-based similarity measure for a machine learning-driven focused crawler

Authorsعطیه جبل عاملی, محمد مهدی محمدی
Conference Titlethe 6th National Conference on Applied Research in Computer Engineerinag and Informmation Technology
Holding Date of Conference2020-02-13 - 2020-02-14
Event Place1 - تهران
Presented byدانشگاه خواجه نصیرالدین طوسی
PresentationSPEECH
Conference LevelNational Conferences

Abstract

The volume of the World Wide Web is growing rapidly, reaching a point where governing data is challenging. Search engines are used to collect data across the web for users. Web crawlers as the major part of search engines are then used to retrieve relevant data on the web according to the user requests. Accordingly, a focused crawler considers a predefined subject and retrieves corresponding relevant pages. In this paper, we propose an efficient focused web crawling approach, which uses a combination of a content-based similarity measure and a Naive Bayes learning classifier in order to find relevant pages to a particular subject. Our first experimental studies show satisfactory improvements where accuracy and recall are increased by 4% and 1% respectively.

tags: Focused crawler, Web crawler, Naive Bayes classification, Relevant page, TF-IDF criteria