Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Entropy-Based Trust Management System for Mitigating Malicious Behaviors in Trust Management Systems, Considering Information Ethics Theory 3 22 FA Amir Khoshkbarchi Amir kabir University of Technology a.khoshkbarchi@aut.ac.ir N Hamid Reza Shahriari Amir kabir University of Technology shahriari@aut.ac.ir Y 10.61186/jsdp.20.4.3 Trust management systems are used in interactive environments, where an agent needs to make a decision about using a service. Due to the preponderance of these systems, malicious entities have strong incentives to influence trust management systems and divert their decisions. In spite of approaches presented in previous trust models to mitigate the malicious activities, many of them could not cope with the problem efficiently. For example, tackling the variable behavior of agents is a common failure point for many trust models. Moreover, no rigid, flexible and adaptive general approach has been presented and the problem somehow remains. This paper presents a novel approach to prevent malicious actions and identify anomalies using an entropy-based trust management system. The system is capable of recognizing the intrinsic characteristics of actions, determining whether they are malicious or not. To achieve this, the information environment is divided into four main parts based on entropy changes. Trust calculation in this system relies on an entropy structure derived from information ethics theory. To enhance the system’s resistance and resilience against malicious behavior, it is important to understand the nature of the actions performed by the agents. To accomplish this, we define the patterns of entropy changes for the four parts and use these patterns to identify and refine the nature of actions as good, bad, or insignificant. The simulation-based experimental results indicate that the proposed system shows promising performance in terms of accurately calculating trust and detecting malicious behavior. Specifically, the proposed system exhibits a 10 percent advantage over well-known trust systems with regards to swiftly adapting to environmental changes and diverse agent behaviors. Moreover, the observed experiments have displayed a notable trend in the fluctuation of good, bad, and insignificant actions. The results indicate a consistent increase in the number of good actions and a corresponding decrease in bad actions. Put simply, the method demonstrates improvement over time through repeated system implementations. This improvement can be attributed to the agents’ heightened honesty as they gain a better understanding of the nature of their actions. Additionally, the provision of feedback on their behavior plays a pivotal role in reinforcing more accurate decision-making within the system. Trust, Information Ethics Theory, Entropy, Service Discovery, Service-Oriented environment http://jsdp.rcisp.ac.ir/article-1-1370-en.html http://jsdp.rcisp.ac.ir/article-1-1370-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Online Learning for Imbalanced Data Streams with Concept Drift by Belief Theory and Chaotic Function 23 34 FA Javad Hamidzadeh Sadjad University J_Hamidzadeh@sadjad.ac.ir Y Mohammad Ali Rashidi Mahmoodi Sadjad University ma.rashidi191@sadjad.ac.ir N Mona Moradi Sadjad University monamoradi0@gmail.com N 10.61186/jsdp.20.4.23 Continual learning from data streams is a pivotal aspect of machine learning, requiring the development of algorithms capable of adapting to incoming data. However, the ongoing evolution of data streams presents a formidable challenge as previously acquired knowledge may become outdated. This challenge, known as concept drift, demands timely detection for the effective adaptation of learning models. While various drift detectors have been proposed, they often assume a relatively balanced class distribution. In scenarios with imbalanced data streams, these detectors may exhibit bias toward majority classes, overlooking shifts in minority classes. Moreover, the imbalance among classes can change over time, with roles shifting between majority and minority classes, especially when relationships among classes become complex due to overlapping regions. In this paper, a novel classification method is introduced for imbalanced streaming data affected by concept drift. The proposed method continuously monitors arriving streams to detect and adapt to both imbalances and concept drift. Upon receiving a new block of data, the proposed method employs the k-means clustering approach to identify non-dense regions and performs oversampling for minority classes. Cluster centers are selected using the belief function to address overlapping issues between majority and minority classes. Utilizing a chaotic approach, the new sample is added based on its neighborhood and the size of thresholds that cover time intervals and classification errors. Finally, the label prediction process is done by ensemble learning and weighted majority voting. Experiments conducted on benchmark datasets from the UCI database evaluate the performance of the proposed method using Leave-One-Out (LOO) validation and comparisons with state-of-the-art methods. The results demonstrate the superiority of the proposed method across various evaluation criteria, highlighting its effectiveness in addressing imbalanced streaming data with concept drift. Belief Theory, Concept Drift, Data Stream, Imbalanced Data, Online Classification http://jsdp.rcisp.ac.ir/article-1-1246-en.html http://jsdp.rcisp.ac.ir/article-1-1246-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 A New Approach for Digital Image Segmentation with Genetic Algorithm and Random Forest 35 44 FA Fariba Namiranian Yazd University fariba.namiranian75@gmail.com N AliMohammad Latif Yazd University alatif@yazd.ac.ir Y 10.61186/jsdp.20.4.35 In this study, a new method for image segmentation by genetic algorithms and random forest is resented. The main objective of image segmentation is to distinguish different components within an image, achieved by labeling pixels based on shared characteristics. In this novel approach, these distinguishing features are derived through the application of image filters (Gabor filters). The random forest algorithm is then employed as a classifier to perform image segmentation according to extracted features from these filters. The image filters utilized come with various hyperparameters, and tuning of these parameters significantly enhances the algorithm's performance. The proposed methodology distinguishes itself by employing a genetic algorithm to fine-tune the hyperparameters of Gabor filters. In this context, the hyperparameters are treated as genes within the chromosome of the genetic algorithm. The success of this optimization is evaluated using f1-score, a metric derived from the random forest algorithm's execution in image segmentation. This step ensures that the selected hyperparameters contribute to optimal segmentation results. The achievement of this research lies not only in the implementation of this novel approach but also in surpassing the performance of other investigated methods through the enhancement of the f1-score in image segmentation. Key to the success of the proposed method is the careful consideration of hyperparameters and their role in defining the characteristics crucial for accurate image segmentation. The use of genetic algorithms not only automates this parameter tuning process but also ensures that the algorithm adapts and evolves to find the most suitable values for the hyperparameters of Gabor filters. As a result, the research contributes to the broader field of image segmentation by providing a robust and effective methodology, demonstrating superior performance compared to alternative methods. In conclusion, this study introduces an approach to image segmentation, leveraging the synergies between genetic algorithms, random forest, and image filters. The research not only emphasizes the importance of hyperparameter tuning but also showcases the effectiveness of the proposed methodology through the optimization of Gabor filter parameters. The overall impact of this work is evident in the improved f1-score achieved in image segmentation, establishing it as a noteworthy advancement in the field. Image Segmentation, Genetic Algorithm, RImage Segmentation, Genetic Algorithm, Random Forest Algorithm, Gabor Filter, Feature Extractionandom Forest Algorithm http://jsdp.rcisp.ac.ir/article-1-1339-en.html http://jsdp.rcisp.ac.ir/article-1-1339-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Improved Ensemble Learning Model by Swarm Intelligence for Mobile Subscribers’ Churn Prediction 45 66 FA bijan moradi bijanmoradi52@yahoo.com N Mehran Khalaj mkhalaj@rkiau.ac.ir Y Ali Taghizadeh Harat N 10.61186/jsdp.20.4.45 In today’s competitive world, companies need to analyze, identify and predict the behavior of their customers and respond to their demands earlier than their competitors. Moreover, in many industries such as mobile telecommunications, the cost of maintaining existing customers (customer retention) is much lower than the cost of attracting a new customer. Therefore, the problem of identifying customers who are going to leave the company, so-called Customer Churn Prediction (CCP), and preventing them by offering Incentives is essential in these industries. In this direction, researchers have presented competent methods using data mining and artificial intelligence tools to identify potential churners. Machine learning (ML) methods are one of the most powerful and widely used techniques to deal with the CCP problem, since they can properly extract and learn complex relationships between the customers’ attributes and their churn intention. Artificial Neural Networks (ANNs), Support Vector Machines (SVMs), Decision Trees (DTs), Logistic Regression (LR), and Naïve Bayes (NB) are among the well-known ML models utilized in numerous studies to tackle the CCP problem. Also, ensemble learning techniques such as Adaboost, Gradient Boost, and Extreme Gradient Boost (XG_boost) have been widely used to solve the CCP problem since they can aggregate the capabilities of multiple ML models. Hence, in order to improve the process of predicting customer churn, in this paper we propose a novel ensemble learning based approach, which is designed based on the two-level stacking technique. We employ six prominent ML models in each level of our proposed ensemble model including MLP, SVM-RBF, DT, NB, KNN, and LR. We also benefit from the Gray Wolf Optimization (GWO) algorithm as an efficient swarm intelligence based search algorithm to select the most effective features and also adjust the hyper-parameters in the proposed model. We have implemented our proposed model using Python and simulated it on two well-known customer churn datasets in the telecom market (IBM_Telco and Duke_Cell2Cell) to evaluate its performance. In this direction, we first demonstrated the optimal features’ subset and the parameter values obtained from applying the GWO algorithm on each dataset. Next, we compared the performance of the proposed ensemble model with each of the base learners using common evaluation criteria including accuracy, precision, recall, F1 score and AUC. The results show that the proposed ensemble model can collect the capabilities of all the base learners and it works better than each of the basic ML models. Afterward, we compared the obtained results from the suggested model with the common ensemble models (Adaboost, Gradient_boost, XG_boost, and Cat_boost) The experimental results show the superiority of the proposed method over other evaluated ensemble models in all the evaluation criteria. Eventually, our method is compared with two recent CCP approaches introduced in the literature. This analysis reveals that, except for the recall criterion in the Duke_Cell2Cell dataset, our introduced method achieves superior results compared to the considered approaches in both datasets.   Customer Churn Prediction, Mobile Telecommunication, Ensemble Learning, Swarm Intelligence, Gray Wolf Optimization Algorithm http://jsdp.rcisp.ac.ir/article-1-1344-en.html http://jsdp.rcisp.ac.ir/article-1-1344-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Improving Persian Opinion Mining based on Polarity and Balancing Positive and Negative Keywords (case study: Digikala reviews for mobile) 67 88 FA Mahdieh Vahedipoor vahedipoor.m@qut.ac.ir N Mahboubeh Shamsi shamsi@qut.ac.ir N Abdolreza Rasouli Kenari rasouli@qut.ac.ir Y 10.61186/jsdp.20.4.67 In recent years, the massive growth of generated content by the users in social networks and online marketing sites, allows people to share their feelings and opinions in a variety of opinions about different products and services. Sentiment analysis is an important factor for better decision-making that is done using natural language processing (NLP), computational methods, and text analysis to extract the polarity of unstructured documents. The complexity of human languages and sentiment analysis have created a challenging research context in computer science and computational linguistics. Many researchers used supervised machine learning algorithms such as Naïve Bayes (NB), Stochastic Gradient Descent (SGD), Support Vector Machine (SVM), Logistic Regression (LR) Random Forest (RF), and deep learning algorithms such as Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM). Some researchers have used Dictionary-based methods. Despite the existence of effective techniques in text mining, there are still unresolved challenges. Note that user comments are unstructured texts; Therefore, in order to structure the textual inputs, parsing is usually done along with adding some features, linguistic interpretations and removing additional items, and inserting the next terms in the database, then extracting the patterns in the structured data and finally the outputs will evaluate and interpret. The imbalance of data with the difference in the number of samples in each class of a dataset is an important challenge in the learning phase. This phenomenon breaks the performance of the classifications because the machine does not learn the features of the unpopulated classes well. In this paper, words are weighted based on the prescribed dictionary to influence the most important words on the result of the opinion mining by giving higher weight. On the other hand, the combination of the adjacent words using n-gram methods will improve the outcome. The dictionaries are highly related to the domain of the application. Some words in an application are important but in mobile comments are not impressive. Another challenge is the unbalanced train data, in which the number of positive sentences is not equal to the number of negative sentences. In this paper, two ideas are applied to build an efficient opinion mining algorithm. First, we build a precise dictionary for mobile Persian comments, and the second idea is to balance the positive and negative comments in train data. In summary, the main achievements of the current research can be mentioned: creating a weighted comprehensive dictionary in the field of mobile phone opinions to increase the accuracy of opinion analysis, balancing positive and negative opinions to improve the accuracy of opinion analysis, and eliminating the negative effect of overfitting and providing a precise approach to Determining the polarity of users' opinions about mobile phones using machine learning and recurrent deep learning algorithms. This new method is presented on mobile phone products from the Digikala site and Senti-Pers data. The result is performed with Naive Bayesian, Support Vector Machine, Stochastic Gradient Descent, Logistic Regression, Random Forest, and deep learning methods such as Convolutional Neural Network and Long Short-Term Memory based on parameters such as Accuracy, Precision, Retrieval, and F-Measure. The proposed method increases accuracy on Digikala, with NB between 10% and 34% and SVM between 5% and 24%, SGD between 7% and 38%, LR between 5% to 38%, and RF between 4% Up to 22% and CNN by 4%. The results show an accuracy increment on Senti-Pers, with NB between 12% and 46% and SVM between 5% and 46%, SGD between 5% and 35%, LR between 6% to 46%, and RF between 4% Up to 46%.   Sentiment Analysis, Opinion Mining, Machine Learning, Deep Learning, Polarity http://jsdp.rcisp.ac.ir/article-1-1202-en.html http://jsdp.rcisp.ac.ir/article-1-1202-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Using movie genres and Demographic Information to improve movie recommendation systems 89 106 FA samad mohammadi Department of Computer Engineering, Islamic Azad University, Central Tehran Branch, Tehran, Iran mohamadi601@gmail.com N vahe aghazarian Department of Computer Engineering, Islamic Azad University, Central Tehran Branch, Tehran, Iran aghazarian@iauctb.ac.ir Y alireza hedayati Department of Computer Engineering, Islamic Azad University, Central Tehran Branch, Tehran, Iran samad601@gmail.com N 10.61186/jsdp.20.4.89 Movie recommendation systems are efficient tools to help users find their relevant movies by investigating the previous interests of users. These systems are established on considering the ratings of users provided for movies in the past and using them to predict their interests in the future. However, users mainly provide insufficient ratings leading to make a problem called data sparsity. This problem makes reducing the effectiveness of movie recommendation systems. On the other hand, other available data such as genres of movies and demographic information of users play a vital role in assisting recommenders in order to better produce recommendations. This paper proposes a movie recommendation method utilizing the movies’ genres and users’ demographic information. In particular, we propose an effective model to evaluate the user’s rating profile and determine the minimum number of ratings required to produce an accurate prediction. Then, appropriate virtual ratings are incorporated into the profiles with insufficient ratings to expand them. These virtual ratings are calculated using similarity values between users obtained by genres of movies and demographic information of users. Furthermore, an effective measure is introduced to determine how much an item is reliable. This measure guarantees the virtual ratings’ reliability. Finally, unknown ratings for target user are predicted based on the expanded rating profiles. Experiments performed on two well-known movie recommendation datasets demonstrate that the proposed approach is more efficient than other compared recommenders. We propose a movie recommender system in this paper by employing the genres of movies and demographic information of users to address the above-mentioned challenges. To this end, first of all, a model is developed in order to determine whether the target user’s rating profile is appropriate to produce accurate recommendations or not. In other words, the developed model determines how many ratings are required for each user to generate an accurate prediction with a high probability. This criterion is used to demonstrate that a rating profile contains sufficient ratings for producing reliable recommendations or not. Then, the quality of rating profiles containing insufficient ratings is boosted using an effective profile expansion technique which incorporates some virtual ratings to these profiles. These virtual ratings are calculated using the similarity values between users which are computed according to the genres of movies and demographic information of users. Moreover, the reliability values of users and items are calculated using appropriate reliability measurements to guarantee that the incorporated virtual ratings are reliable. Experimental results on two movie recommendation datasets indicate the superiority of the proposed approach in respect to other models. In the following, we provide a list of the main contributions of this paper: We develop a model in order to evaluate the users’ rating profiles and determine how many ratings are required for generating an accurate prediction. We propose a powerful profile expansion technique which incorporates some virtual ratings to user-item ratings matrix for improving its quality. Movies’ genres and users’ demographic information are used as additional data in the proposed movie recommender system. The reliability measures of users and items are used in the proposed method to guarantee the reliability of calculated virtual ratings. The proposed method generates a denser user-item ratings matrix than the original matrix which results in alleviating data sparsity problem significantly.    The remaining parts of this paper are structured as follows: in section 2, related works are investigated, section 3 includes the details of the proposed method, section 4 refers to the discussion of experimental results, and section 5 provides some conclusions about the paper Movie recommendation systems are efficient tools to help users find their relevant movies by investigating the previous interests of users. These systems are established on considering the ratings of users provided for movies in the past and using them to predict their interests in the future. However, users mainly provide insufficient ratings leading to make a problem called data sparsity. This problem makes reducing the effectiveness of movie recommendation systems. On the other hand, other available data such as genres of movies and demographic information of users play a vital role in assisting recommenders in order to better produce recommendations. This paper proposes a movie recommendation method utilizing the movies’ genres and users’ demographic information. In particular, we propose an effective model to evaluate the user’s rating profile and determine the minimum number of ratings required to produce an accurate prediction. Then, appropriate virtual ratings are incorporated into the profiles with insufficient ratings to expand them. These virtual ratings are calculated using similarity values between users obtained by genres of movies and demographic information of users. Furthermore, an effective measure is introduced to determine how much an item is reliable. This measure guarantees the virtual ratings’ reliability. Finally, unknown ratings for target user are predicted based on the expanded rating profiles. Experiments performed on two well-known movie recommendation datasets demonstrate that the proposed approach is more efficient than other compared recommenders. Recommender systems, movies, cold start, data sparsity, demographic information, genre http://jsdp.rcisp.ac.ir/article-1-1332-en.html http://jsdp.rcisp.ac.ir/article-1-1332-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Farsi Question and Answer Dataset (FarsiQuAD) 107 120 FA Javad ForutanRad Forutanrad@gmail.com Y Maryam HourAli mhourali@mut.ac.it N MohammadAli KeyvanRad keyvanrad@aut.ac.ir N 10.61186/jsdp.20.4.107 A fast and accurate response to questions posed in natural language is a fundamental objective in the advancement of question and answer systems. These systems involve computers comprehending textual content and questions, and subsequently, delivering precise answers to users. Despite significant advancements in this field, there remains room for improvement, particularly when dealing with languages other than English, such as Persian. In this article, we present the Persian language question and answer dataset, known as FarsiQuAD. This dataset was meticulously crafted by human annotators, drawing from Persian Wikipedia articles. FarsiQuAD is made available in two versions: Version 1 comprises over 10,000 questions and answers, while Version 2 offers an extensive collection of over 145,000 rows. This dataset is designed to seamlessly integrate with the English version of SQuAD and other databases in various languages adhering to this standard, and it is open to the public. These data serve as valuable resources for the development of artificial intelligence models based on deep learning and for the enhancement of Persian language question and answer systems. The research findings reveal that the FarsiQuAD dataset is capable of providing answers to questions posed in the natural Persian language with an exact matching accuracy of 78% and an F1 score of 87%. However, there is still room for improvement in achieving even higher accuracy levels. This project arises from the critical need for non-English languages to have access to more data for training deep learning models, especially in the domain of factoid questions. Hence, the primary objective of this article is to introduce the newly created dataset. Prior to this effort, well-known datasets like SQuAD predominantly focused on English, and similar datasets has been developed in other languages, including French, German, Korean, and Japanese. Nevertheless, the dearth of question datasets in the Persian language was evident. The quality and diversity of questions are pivotal aspects, and as this dataset continues to grow, it will contribute to the broader landscape of research in this domain, allowing for valuable cross-linguistic comparisons and integration with research conducted in other languages.   Question And Answer Dataset, Question And Answer systems, Reading comprehension, Deep Learning, Natural Language Processing http://jsdp.rcisp.ac.ir/article-1-1337-en.html http://jsdp.rcisp.ac.ir/article-1-1337-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Aspect-Based Sentiment Analysis using the Attentional Encoder Network 121 128 FA somayeh karimi Shahrood University of Technology somayehkarimi6@yahoo.com Y Fatemeh Jafarinejad Shahrood University of Technology jafarinejad@shahroodut.ac.ir N 10.61186/jsdp.20.4.121 Natural language processing is growing significantly and has gained much attention with the advent of the World Wide Web and search engines, and researchers have witnessed an explosion of information in different languages. Sentiment analysis is one of the most active fields of study in natural language processing that focuses on text classification and is used to identify, extract and analyze subjective information from text sources. Aspect-based sentiment analysis is a text analysis technique that classifies comments by aspect and identifies the sentiment associated with each aspect. This analysis can be used to automatically analyze the feedback of customers' comments to different parts of goods or services and help employers to focus on points that need quality improvement. In this paper, we will introduce a new architecture based on deep learning for aspect-based sentiment analysis. This architecture will use an attention-encoder network-based model with multiple multi-head attention and a pointwise convolutional transform (which is a parallelizable and interactive alternative to LSTM and is applied to compute hidden states of input embeddings). Testing this architecture on three different datasets, including restaurants and laptops, SemEval 2014 Task 4 and ACL 14 Twitter dataset, in all three datasets, the polarity of emotions is positive, neutral and negative, which is compared with modern methods of sentiment analysis. Based on the aspect, it will show the high accuracy of this method. For example, the aspect-based sentiment analysis on the Laptop dataset has shown 79.15% accuracy, which has increased the accuracy by 4.24% compared to modern methods.   Aspect-Based Sentiment Analysis (ABSA) , Deep Learning, Attentional Encoder Network, Multi-Head Attention (MHA) http://jsdp.rcisp.ac.ir/article-1-1321-en.html http://jsdp.rcisp.ac.ir/article-1-1321-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Maximize Score in stochastic match-3 games using reinforcement learning 129 140 FA Mehdy Roayaei Ardakany Tarbiat Modares University mroayaei@modares.ac.ir Y Ali Afroughrh Tarbiat Modares University ali74afrougheh@gmail.com N 10.61186/jsdp.20.4.129 Computer games have played an important role in the development of artificial intelligence in recent years. Throughout the history of artificial intelligence, computer games have been a suitable test environment for evaluating new approaches and algorithms to artificial intelligence. Different methods, including rule-based methods, tree search methods, and machine learning methods (supervised learning and reinforcement learning) have been developed to create intelligent agents in different games. Games have been used as a suitable environment for trial and error, testing different artificial intelligence ideas and algorithms. Among these researches, we can mention the research of Deep Blue in the chess game and AlphaGo in the game Go. AlphaGo is the first computer program to defeat an expert human Go player. Also, Deep Blue is a chess-playing expert system is the first computer program to win a match, against a world champion. In this paper, we focus on the match-3 game. The match-3 game is a popular game in cell phones, which consists of a very large random state space which makes learning difficult. It also has random reward function which makes learning unstable. Many researches have been done in the past on different games, including match-3. The aim of these researches has generally been to play optimally or to predict the difficulty of stages designed for human players. Predicting the difficulty of stages helps game developers to improve the quality of their games and provide a better experience for users. Based on the approach used, past works can be divided into three main categories including search-based methods, machine learning methods and heuristic methods. In this paper, an intelligent agent based on deep reinforcement learning is presented, whose goal is to maximize the score in the match-3 game. Reinforcement learning is one of the approaches that has received a lot of attention recently. Reinforcement learning is one of the branches of machine learning in which the agent learns the optimal policy for choosing actions in different spaces through its experiences of interacting with the environment. In deep reinforcement learning, reinforcement learning algorithms are used along with deep neural networks. In the proposed method, different mapping mechanisms for action space and state space are used. Also, a novel structure of neural network customized for the match-3 game environment has been proposed to achieve the ability to learn large state space. The contributions of this article can be summarized as follow. An approach for mapping the action space to a two-dimensional matrix is presented in which it is possible to easily separate valid and invalid actions. An approach has been designed to map the state space to the input of the deep neural network, which reduces the input space by reducing the depth of the convolutional filter and thus improves the learning process. The reward function has made the learning process stable by separating random rewards from deterministic rewards. The comparison of the proposed method with other existing methods, including PPO, DQN, A3C, greedy method and human agents shows the superior performance of the proposed method in the match-3 game.   deep reinforcement learning, random game, match-3, large state space http://jsdp.rcisp.ac.ir/article-1-1345-en.html http://jsdp.rcisp.ac.ir/article-1-1345-en.pdf

Research Center on Developing Advanced Technologies Signal and Data Processing 2538-4201 2538-421X 20 4 2024 3 1 Regularized Knowledge Transfer for Multi-Agent Reinforcement Learning 141 160 FA Niloofar Alavi Urmia University of Technology nilooalavi504@it.uut.ac.ir N Jafar Tahmoresnezhad Urmia University of Technology j.tahmores@it.uut.ac.ir Y 10.61186/jsdp.20.4.141 Reinforcement learning (RL) refers to the training of machine learning models to make a sequence of decisions on which an agent learns by interacting with its environment, observing the results of interactions and receiving a positive or negative reward, accordingly. RL has many applications for multi-agent systems, especially in dynamic and unknown environments. However, most multi-agent reinforcement learning (MARL) algorithms suffer from some problems specifically the exponential computational complexity to calculate the joint state-action space, which leads to the lack of scalability of algorithms in realistic multi-agent problems. Applications of MARL can be categorized from robot soccer, networks, cloud computing, job scheduling, and to optimal reactive power dispatch. In the area of reinforcement learning algorithms, there are serious challenges such as the lack of application of equilibrium-based algorithms in practice and high computational complexity to find equilibrium.  On the other hand, since agents have no concept of equilibrium policies, they tend to act aggressively toward their goals, which it results the high probability of collisions. Consequently, in this paper, a novel algorithm called Regularized Knowledge Transfer for Multi-Agent Reinforcement Learning (RKT-MARL) is presented that relies on Markov decision process (MDP) model. RKT-MARL unlike the traditional reinforcement learning methods exploits the sparse interactions and knowledge transfer to achieve an equilibrium across agents. Moreover, RKT-MARL benefits from negotiation to find the equilibrium set. RKT-MARL uses the minimum variance method to select the best action in the equilibrium set, and transfers the knowledge of state-action values across various agents. Also, RKT-MARL initializes the Q-values in coordinate states as coefficients of current environmental information and previous knowledge. In order to evaluate the performance of our proposed method, groups of experiments are conducted on five grid world games and the results show the fast convergence and high scalability of RKT-MARL. Therefore, the fast convergence of our proposed method indicates that the agents quickly solve the problem of reinforcement learning and approach to their goal.   Multi-agent reinforcement learning, Knowledge transfer, Meta and Nash equilibriums, Regularization, Sparse interactions, Agents negotiations. http://jsdp.rcisp.ac.ir/article-1-1056-en.html http://jsdp.rcisp.ac.ir/article-1-1056-en.pdf