Volume 15, Issue 3 (12-2018)                   JSDP 2018, 15(3): 3-12 | Back to browse issues page

XML Persian Abstract Print

Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:

Yadolahi M, Zargari F, Farhoodi M. Automatic Evaluation of Video Search engines in Persian Web domain based on Majority Voting. JSDP. 2018; 15 (3) :3-12
URL: http://jsdp.rcisp.ac.ir/article-1-606-en.html
Iran Telecom Research Center
Abstract:   (261 Views)

Today, the growth of the internet and its high influence in individuals’ life have caused many users to solve their daily needs by search engines and hence, the search engines need to be modified and continuously improved. Therefore, evaluating search engines to determine their performance is of paramount importance. In Iran, as well as other countries, extensive researches are being performed on search engines. To evaluate the quality of search engines and continually improve their performance, it is necessary to evaluate search engines and compare them to other existing ones. Since the speed plays an important role in the assessment of the performance, automatic search engine evaluation methods attracted grate attention. In this paper, a method based on the majority voting is proposed to assess the video search engines. We introduced a mechanism to assess the automatic evaluation method by comparing its results with the results obtained by human search engine evaluation. The results obtained, shows 94 % correlation of the two methods which indicate the reliability of automated approach.
In general, the proposed method can be described in three steps.
Step 1: Retrieve first k_retrieve results of n different video search engines and build the return result set for each written query.
Step 2: Determine the level of relevance of each retrieved result from the search engines
Step 3: Evaluating the search engines by computing different evaluation criteria based on decisions on relevance of the retrieved videos by each search engine

 Clearly, the main part of any evaluation system with the goal of evaluating the accuracy of search engines is the second step. In this paper, we have tried to present a new solution based on the aggregation of votes in order to determine whether a result is relevant or not, as well as its level of relevance. For this purpose, for each query the return results from different search engines are compared with each other, and the result returned by more than m of the search engines (m ; and the result of which their URLs (after the normalization) are similar to the normalized URL from the m-1 of the other search engines, are considered as the relevant results. At the second level, the retrieved results will be compared in terms of content. In this way, after calculating the address-like similarity, all the results are transmitted to the motion vector extraction component to extract and store the motion vector.
In the content based similarity algorithm, the set of motion vectors is initially considered as a sequence of motion vector. We, then, try to find the greatest similarity of the smaller sequence with the larger sequence. After this step, we will report the maximum similarity of the two videos. The process of finding the maximum similarity is that we consider a window with a smaller video sequence length. In this window we calculate and hold the similarity of two sequences. In the proposed method, after identifying the similarity between the return results of different search engines, their level is ranked at three different levels: "unrelated" (0), "quantitatively related" (1) and "related" (2). Since Google's search engine is currently the world’s largest and best-performing search engine, and most search engines have been compared to it, and are also trying to achieve the same function, the first five Google search engine results are get the minimum relevance, by default, "slightly related". Then the similarity module is used to evaluate the similarity of the retrieved n results of the tested search engines.

Full-Text [PDF 6698 kb]   (145 Downloads)    
Type of Study: Research | Subject: Paper
Received: 2017/11/6 | Accepted: 2018/09/15 | Published: 2018/12/19 | ePublished: 2018/12/19

1. [1] Moosavi Sobhan, Azimzadeh Masoumeh, Mahmo-udi Maryam, Yari Alireza, A Comprehens-ive and Effective Framework for the Assessment of Persian Search Engines, The 18th Annual National Conference of the Computer Society, Tehran, Esfand 1391.
2. [2] Azimzadeh Masoumeh, Somoori Shahriar, Yari Alireza, Qualitative and qualitative comparison of search engines in the Persian Web domain, 18th National Computer Society Conference, Tehran, Esfand 1391.
3. [3] R. Badie, M. Azimzadeh, M.S. Zahedi, S. Samuri, "Automatic evaluation of search engines: Using webpages' content, web graph link structure and websites' popularity" Seventh International Symposium on Telecommunica-tions ( IST2014), September 09-11, 2014. [DOI:10.1109/ISTEL.2014.7000766]
4. [4] Maryam Mahmoudy, Mohammad Sadegh zahedi, Masomeh Azimzadeh, "Evalua-ting the retrieval effectiveness of search engines using Persian navigational queries", Seventh Interna-tional Symposium on Telecommunica-tions (IST2014), September 09-11, 2014.
5. [5] I. Soboroff, C. Nicholas, and P. Cahan, "Ranking retrieval systems without relevance judgments," in Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, 2001, pp. 66-73. [DOI:10.1145/383952.383961]
6. [6] S. P. Harter, "Variations in relevance assessm-ents and the measurement of retrieval effective-ness," JASIS, vol. 47, pp. 37-49, 1996. https://doi.org/10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3 [DOI:10.1002/(SICI)1097-4571(199601)47:13.0.CO;2-3]
7. [7] A. Spink and H. Greisdorf, "Regions and levels: measuring and mapping users' relevance judgments," Journal of the American Society for Information science and Technology, vol. 52, pp. 161-173, 2001. https://doi.org/10.1002/1097-4571(2000)9999:9999<::AID-ASI1564>3.0.CO;2-L [DOI:10.1002/1097-4571(2000)9999:99993.0.CO;2-L]
8. [8] S. Wu and F. Crestani, "Methods for ranking information retrieval systems without relevance judgments," in Proceedings of the 2003 ACM symposium on Applied computing, 2003, pp. 811-816. [DOI:10.1145/952532.952693] [PMCID]
9. [9] J. Callan, M. Connell, and A. Du, "Automatic discovery of language models for text databases," in ACM SIGMOD Record, 1999, pp. 479-490. [DOI:10.1145/304181.304224]
10. [10] A. Chowdhury and I. Soboroff, "Automatic evaluation of world wide web search services," in Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, 2002, pp. 421-422. [DOI:10.1145/564376.564474]
11. [11] S. M. Beitzel, E. C. Jensen, A. Chowdhury, and D. Grossman, "Using titles and category names from editor-driven taxonomies for automatic evaluation," in Proceedings of the twelfth international conference on Information and knowledge management, 2003, pp. 17-23. [DOI:10.1145/956863.956868]
12. [12] F. Can, R. Nuray, and A. B. Sevdik, "Automatic performance evaluation of Web search engines," Information processing & management, vol. 40, pp. 495-514, 2004. [DOI:10.1016/S0306-4573(03)00040-2]
13. [13] T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay, "Accurately interpreting clickth-rough data as implicit feedback," in Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, 2005, pp. 154-161.
14. [14] Y. Liu, Y. Fu, M. Zhang, S. Ma, and L. Ru, "Automatic search engine performance evaluation with click-through data analysis," in Proceedings of the 16th international conference on World Wide Web, 2007, pp. 1133-1134. [DOI:10.1145/1242572.1242731]
15. [15] Y. Liu, M. Zhang, L. Ru, and S. Ma, "Automatic query type identification based on click through information," in Information Retrieval Techno-logy, ed: Springer, 2006, pp. 593-600. [DOI:10.1007/11880592_51]
16. [16] T. Joachims, "Evaluating Retrieval Performance Using Clickthrough Data," ed: Citeseer, 2003.
17. [17] G. Mood, "Boes, Introduction to the theory of statistics," McCraw-Hill Statistics Series, 1974. [PMID]
18. [18] H. Sharma and B. J. Jansen, "Automated evaluation of search engine performance via implicit user feedback," in Proceedings of the 28th annual international ACM SIGIR con-ference on Research and development in information retrieval, 2005, pp. 649-650. [DOI:10.1145/1076034.1076172]
19. [19] R. Ali and M. S. Beg, "Automatic performance evaluation of web search systems using rough set based rank aggregation," in Proceedings of the First International Conference on Intelligent Human Computer Interaction, 2009, pp. 344-358. [DOI:10.1007/978-81-8489-203-1_34]
20. [20] R. Nuray and F. Can, "Automatic ranking of information retrieval systems using data fusion," Information processing & management, vol. 42, pp. 595-614, 2006. [DOI:10.1016/j.ipm.2005.03.023]
21. [21] H. Sadeghi. "Automatic Performance Evaluation of Web search Engines using judgements of Meta search Engines", Online Information Review, ISSN:1468-4527,Emerald Publishing Limited, pp.957-971. (2011).
22. [22] Tawileh W, Griesbaum J, Mandl T. Evaluation of five web search engines in Arabic language. Proceedings of LWA. (2010).
23. [23] Lewandowski, Dirk. Evaluating the retrieval effectiveness of Web search engines using a representative query sample. Journal of the Association for Information Science and Technology (2015). [DOI:10.1002/asi.23304]
24. [24] Bar-Ilan J, Levene M. A method to assess search engine results. Online Information Review 35(6), 854-868. (2011). [DOI:10.1108/14684521111193166]
25. [25] Keyvanpour M, alamdar F. Effective browsing of image search results via diversified visual summarization by Clustering. JSDP. 2012; 8 (2) :57-74

Add your comments about this article : Your username or Email:

Send email to the article author

© 2015 All Rights Reserved | Signal and Data Processing