Volume 19, Issue 1 (5-2022)
JSDP 2022, 19(1): 19-38 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Sadr H, Pedram M M, Teshnehlab M. Efficient Method Based on Combination of Deep Learning Models for Sentiment Analysis of Text. JSDP 2022; 19 (1) : 2
URL: http://jsdp.rcisp.ac.ir/article-1-1060-en.html
URL: http://jsdp.rcisp.ac.ir/article-1-1060-en.html
Department of Electrical and Computer Engineering Faculty of Engineering, Kharazmi University Tehran, Iran
Abstract: (1429 Views)
People's opinions about a specific concept are considered as one of the most important textual data that are available on the web. However, finding and monitoring web pages containing these comments and extracting valuable information from them is very difficult. In this regard, developing automatic sentiment analysis systems that can extract opinions and express their intellectual process has attracted considerable attention in recent years. Sentiment analysis is considered as one of the most active research areas in the field of natural language processing which tries to classify a piece of text containing opinions based on its polarity and determine whether an expressed opinion about a specific topic, event or product is positive or negative.
Since about a decade ago, many studies have been carried out to investigate the effects of traditional classification models, such as Support Vector Machine (SVM), Naïve Bayes, Logistic Regression, etc. in the task of sentiment analysis. Although machine learning models have achieved great success in this filed, they are still confronted with some limitations, notably manual feature engineering requirements. In other words, the classification performance of machine learning models is highly dependent on the extracted features and they play an important role in obtaining higher classification accuracy. To deal with these problems, deep learning models have been extensively employed as an alternative to traditional machine learning models and have achieved impressive results. It is worth mentioning that despite the remarkable performance of these methods, they are still confronted with some limitations and they are on their first steps of progress.
Therefore, the goal of this paper is to propose a combinational deep learning model that can overcome their problems as well as utilizing their benefits. In this regard, an efficient method based on combination of convolutional and recursive neural networks is proposed in this paper that employs a generalized recursive neural network, where an intermediate feature is obtained by combining children's nodes, as an alternative of pooling layer in attention-based convolutional neural network with the aim of capturing long term dependencies and decreasing the loss of local information. Based on empirical results, the proposed method with the accuracy of 53.92% and 92.89% respectively on SST1 and SST2 datasets not only outperforms other existing models but also can be trained much faster.
Since about a decade ago, many studies have been carried out to investigate the effects of traditional classification models, such as Support Vector Machine (SVM), Naïve Bayes, Logistic Regression, etc. in the task of sentiment analysis. Although machine learning models have achieved great success in this filed, they are still confronted with some limitations, notably manual feature engineering requirements. In other words, the classification performance of machine learning models is highly dependent on the extracted features and they play an important role in obtaining higher classification accuracy. To deal with these problems, deep learning models have been extensively employed as an alternative to traditional machine learning models and have achieved impressive results. It is worth mentioning that despite the remarkable performance of these methods, they are still confronted with some limitations and they are on their first steps of progress.
Therefore, the goal of this paper is to propose a combinational deep learning model that can overcome their problems as well as utilizing their benefits. In this regard, an efficient method based on combination of convolutional and recursive neural networks is proposed in this paper that employs a generalized recursive neural network, where an intermediate feature is obtained by combining children's nodes, as an alternative of pooling layer in attention-based convolutional neural network with the aim of capturing long term dependencies and decreasing the loss of local information. Based on empirical results, the proposed method with the accuracy of 53.92% and 92.89% respectively on SST1 and SST2 datasets not only outperforms other existing models but also can be trained much faster.
Article number: 2
Keywords: Sentiment analysis, Deep Leaning, Convolutional neural network, Recursive neural network, Attention mechanism
Type of Study: Research |
Subject:
Paper
Received: 2019/08/15 | Accepted: 2021/01/10 | Published: 2022/06/22 | ePublished: 2022/06/22
Received: 2019/08/15 | Accepted: 2021/01/10 | Published: 2022/06/22 | ePublished: 2022/06/22
References
1. [1] H. Sadr, M. M. Pedram, and M. Teshnehlab, "A Robust Sentiment Analysis Method Based on Sequential Combination of Convolutional and Recursive Neural Networks," Neural Processing Letters, pp. 1-17, 2019. [DOI:10.1007/s11063-019-10049-1]
2. [2] H. Sadr, M. M. Pedram, and M. Teshnelab, "Improving the Performance of Text Sentiment Analysis using Deep Convolutional Neural Network Integrated with Hierarchical Attention Layer," International Journal of Information and Communication Technology Research, vol. 11, no. 3, pp. 57-67, 2019.
3. [3] Mohades Deilami, Fatemeh, Hossein Sadr, and Morteza Tarkhan. "Contextualized Multidimensional Personality Recognition using Combination of Deep Neural Network and Ensemble Learning." Neural Processing Letters 2022: 1-18. [DOI:10.1007/s11063-022-10787-9]
4. [4] V. Vyas and V. Uma, "Approaches to sentiment analysis on product reviews," in Sentiment Analysis and Knowledge Discovery in Contemporary Business: IGI Global, 2019, pp. 15-30. [DOI:10.4018/978-1-5225-4999-4.ch002]
5. [5] Kalashami, Mahsa Pourhosein, Mir Mohsen Pedram, and Hossein Sadr. "EEG Feature Extraction and Data Augmentation in Emotion Recognition." Computational Intelligence and Neuroscience 2022. [DOI:10.1155/2022/7028517] [PMID] [PMCID]
6. [6] S. M. H. Chowdhury, S. Abujar, M. Saifuzzaman, P. Ghosh, and S. A. Hossain, "Sentiment Prediction Based on Lexical Analysis Using Deep Learning," in Emerging Technologies in Data Mining and Information Security: Springer, 2019, pp. 441-449. [DOI:10.1007/978-981-13-1501-5_38]
7. [7] Soleymanpour, Shiva, Hossein Sadr, and Mojdeh Nazari Soleimandarabi. "CSCNN: cost-sensitive convolutional neural network for encrypted traffic classification." Neural Processing Letters , pp.3497-3523, 2021. [DOI:10.1007/s11063-021-10534-6]
8. [8] Sadr, Hossein, and Mojdeh Nazari Soleimandarabi. "ACNN-TL: attention-based convolutional neural network coupling with transfer learning and contextualized word representation for enhancing the performance of sentiment classification." The Journal of Supercomputing 2022, pp. 1-27, 2022. [DOI:10.1007/s11227-021-04208-2]
9. [9] H. Sadr, M. Nazari, M. M. Pedram, and M. Teshnehlab, "Exploring the Efficiency of Topic-Based Models in Computing Semantic Relatedness of Geographic Terms," International Journal of Web Research, vol. 2, no. 2, pp. 23-35, 2019.
10. [10] H. Sadr, M. M. Pedram, and M. Teshnehlab, "Multi-View Deep Network: A Deep Model Based on Learning Features From Heterogeneous Neural Networks for Sentiment Analysis," IEEE Access, vol. 8, pp. 86984-86997, 2020. [DOI:10.1109/ACCESS.2020.2992063]
11. [11] H. Sadr, M. N. Soleimandarabi, M. Pedram, and M. Teshnelab, "Unified Topic-Based Semantic Models: A Study in Computing the Semantic Relatedness of Geographic Terms," in 2019 5th International Conference on Web Research (ICWR), 2019: IEEE, pp. 134-140. [DOI:10.1109/ICWR.2019.8765257]
12. [12] V. D. Van, T. Thai, and M.-Q. Nghiem, "Combining convolution and recursive neural networks for sentiment analysis," in Proceedings of the Eighth International Symposium on Information and Communication Technology, 2017: ACM, pp. 151-158. [DOI:10.1145/3155133.3155158]
13. [13] N. C. Dang, M. N. Moreno-García, and F. De la Prieta, "Sentiment Analysis Based on Deep Learning: A Comparative Study," Electronics, vol. 9, no. 3, pp. 483, 2020. [DOI:10.3390/electronics9030483]
14. [14] H. Sadr and M. Nazari Solimandarabi, "Presentation of an efficient automatic short answer grading model based on combination of pseudo relevance feedback and semantic relatedness measures," Journal of Advances in Computer Research, vol. 10, no. 2, pp. 1-10, 2019.
15. [15] J. Islam and Y. Zhang., "Visual Sentiment Analysis for Social Images Using Transfer Learning Approach," 2016 IEEE Int. Conf. Big Data Cloud Comput. (BDCloud), Soc. Comput. Netw. (SocialCom), Sustain. Comput. Commun., pp. 124130, 2016. [DOI:10.1109/BDCloud-SocialCom-SustainCom.2016.29]
16. [16] X. Ouyang, P. Zhou, C. H. Li, and L. Liu, "Sentiment Analysis Using Convolutional Neural Network," Comput. Inf. Technol. Ubiquitous Comput. Commun. Dependable, Auton. Secur. Comput. Pervasive Intell. Comput. (CIT/IUCC/DASC/PICOM), 2015 IEEE Int. Conf., pp. 23592364, 2015. [DOI:10.1109/CIT/IUCC/DASC/PICOM.2015.349]
17. [17] R. Yin, P. Li, and B. Wang, "Sentiment Lexical-Augmented Convolutional Neural Networks for Sentiment Analysis," IEEE Second International Conference on Data Science in Cyberspace, 2017. [DOI:10.1109/DSC.2017.82] [PMCID]
18. [18] R. Socher, Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning, "Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions," Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics., 2011.
19. [19] R. Socher, B. Huval, C. D. Manning, and A. Y. Ng, "Semantic Compositionality through Recursive Matrix-Vector Spaces," Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics., 2012.
20. [20] R. Socher, A. Perelygin, and Wu, "Recursive deep models for semantic compositionality over a sentiment treebank," Proceedings of the conference on empirical methods in natural language processing (EMNLP), 2013.
21. [21] Q. Huang, X. Zheng, R. Chen, and Z. Dong, "Deep Sentiment Representation Based on CNN and LSTM " International Conference on Green Informatics, 2017. [DOI:10.1109/ICGI.2017.45]
22. [22] A. Hassan and A. Mahmood, "Deep Learning approach for sentiment analysis of short texts," in Control, Automation and Robotics (ICCAR), 2017 3rd International Conference on, 2.17 IEEE, pp. 705-710. [DOI:10.1109/ICCAR.2017.7942788]
23. [23] A. Timmaraju and V. Khanna, "Sentiment Analysis on Movie Reviews using Recursive and Recurrent Neural Network Architectures," 2017.
24. [24] X. Wang, W. Jiang, and Z. Luo, "Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts," 2016.
25. [25] V. D. Van, Œ. Thai, and M.-Q. o. Nghiem, "Combining Convolution and Recursive Neural Networks for Sentiment Analysis," 2018. [DOI:10.1145/3155133.3155158]
26. [26] S. M. Rezaeinia, R. Rahmani, A. Ghodsi, and H. Veisi, "Sentiment analysis based on improved pre-trained word embeddings," Expert Systems with Applications, vol. 117, pp. 139-147, 2019. [DOI:10.1016/j.eswa.2018.08.044]
27. [27] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Distributed Representations of Words and Phrases and their Compositionality, Nips," 2013.
28. [28] O. Irsoy and C. Cardie, "Deep recursive neural networks for compositionality in language," in Advances in neural information processing systems, 2014, pp. 2096-2104.
29. [29] Y. Zhang and B. Wallace, "A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification," arXiv preprint arXiv:1510.03820, 2015.
30. [30] Y. LeCun, Y. Bengio, and G. Hinton, " Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, May, 2015. [DOI:10.1038/nature14539] [PMID]
31. [31] C. DU and L. HUANG, "Sentiment Classification Via Recurrent Convolutional Neural Networks," DEStech Transactions on Computer Science and Engineering, no. cii, 2017. [DOI:10.12783/dtcse/cii2017/17268]
32. [32] N. Kalchbrenner, E. Grefenstette, and P. Blunsom, "A convolutional neural network for modelling sentences," arXiv preprint arXiv:1404.2188, 2014. [DOI:10.3115/v1/P14-1062]
33. [33] Y. Kim, "Convolutional neural networks for sentence classification," arXiv preprint arXiv:1408.5882, 2014. [DOI:10.3115/v1/D14-1181]
34. [34] W. Yin and H. Schütze, "Multichannel variable-size convolution for sentence classification," arXiv preprint arXiv:1603.04513, 2016. [DOI:10.18653/v1/K15-1021] [PMCID]
35. [35] K. S. Tai, R. Socher, and C. D. Manning, "Improved semantic representations from tree-structured long short-term memory networks," arXiv preprint arXiv:1503.00075, 2015. [DOI:10.3115/v1/P15-1150]
36. [36] F. Kokkinos and A. Potamianos, "Structural attention neural networks for improved sentiment analysis," arXiv preprint arXiv:1701.01811, 2017. [DOI:10.18653/v1/E17-2093]
37. [37] Y. Wang, M. Huang, and L. Zhao, "Attention-based LSTM for aspect-level sentiment classification," in Proceedings of the 2016 conference on empirical methods in natural language processing, 2016, pp. 606-615. [DOI:10.18653/v1/D16-1058]
38. [38] Sadr, Hossein, Mir M. Pedram, and Mohammad Teshnehlab. "Convolutional neural network equipped with attention mechanism and transfer learning for enhancing performance of sentiment analysis." Journal of AI and Data Mining 9.2, 2021 : 141-151.
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
