Volume 18, Issue 3 (12-2021)                   JSDP 2021, 18(3): 109-126 | Back to browse issues page


XML Persian Abstract Print


Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:

Hourani O, Moghadam Charkari N, Jalili S. An Ensemble Multiview learning method for visual object decoding from fMRI brain data. JSDP 2021; 18 (3) :109-126
URL: http://jsdp.rcisp.ac.ir/article-1-1022-en.html
Department of Computer Engineering, Tarbiat Modares University
Abstract:   (1661 Views)
In the past two decades, the applications of computational neuroscience have been increasingly growing. Breaking the neural code is a crucial open problem in computational neuroscience. Various research groups attempt to provide an efficient method to decode human brain activity using fMRI data. The output of these methods is a computational model that can assign brain signals to an external stimulus; in this study, visual object recognition has been investigated. The brain decoders are used in many applications, such as the brain-computer interface or detecting specific mental illnesses. In general, brain fMRI data have a high spatial and temporal resolution that increases the number of features of the problem. Proper feature extraction from brain images is a challenging and time-consuming process. Consequently, the convergence of learning algorithms takes a long time to create an appropriate model. So, breaking down the feature space is highly recommended. We proposed new multi-view learning to solve the brain decoding problem. This approach splits the feature space based on mutual information and finds an appropriate ensemble classification model that detects the related visual object to neural activities in the brain.
The proposed method clusters the feature space based on mutual information and splits it into coherent sub-spaces, views. For each feature view, a support vector machine model is learned in parallel; the used SVM version can generate a vector of probabilities for each class. At the test phase, the feature space of test data is divided similarly to the training data, and each model generates a probabilistic vector for the test instances. Then, these vectors are combined in the decision profile matrix. The decision fusion is employed by the ordered weighted averaging (OWA) approach. The proposed multi-view learning methods achieved higher accuracy rates than the single view model. The main advantage of the MV model is that it can run in parallel, making it counterproductive to deal with the high-dimensional problems based on the divide and conquer strategy. The optimization phase to detect the most acceptable parameters for each model is obtained using the simulated annealing, SA, algorithm. We have employed three real fMRI datasets of the human brain to assess the proposed method, obtained from the Openneuro website. Also, the leave-one-run-out cross-validation approach has been carried out to evaluate the proposed method in the intra-subject scenario. Criteria such as accuracy rate and confusion matrix have been undertaken to analyze the results. The single feature view obtains an accuracy rate of more than 50%. While in the ensemble model, the accuracy rate in most subjects is more than 90%.
Full-Text [PDF 1170 kb]   (516 Downloads)    
Type of Study: Applicable | Subject: Paper
Received: 2019/05/28 | Accepted: 2021/09/4 | Published: 2022/01/20 | ePublished: 2022/01/20

References
1. [1] J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, "Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex," Science (80-. )., vol. 293, no. 5539, pp. 2425-2430, 2001. [DOI:10.1126/science.1063736] [PMID]
2. [2] T. Naselaris, K. N. Kay, S. Nishimoto, and J. L. Gallant, "Encoding and decoding in fMRI," Neuroimage, vol. 56, no. 2, pp. 400-410, 2011. [DOI:10.1016/j.neuroimage.2010.07.073] [PMID] [PMCID]
3. [3] H. Uchida et al., "Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders," Neuron, vol. 60, no. 5, pp. 915-929, 2008. [DOI:10.1016/j.neuron.2008.11.004] [PMID]
4. [4] ali mahloojifar, "Classification of Parkinson Disease Based on Inter - and Intra -Regional Biomarkers of the Brain Motor Network Using Resting State fMRI Data," Signal Data Process., vol. 11, no. 2, 2015.
5. [5] P. Dayan and L. F. Abbott, Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series, 2001.
6. [6] J. L. Gallant and K. N. Kay, "I can see what you see.," Nat. Neurosci., vol. 12, no. 3, p. 245, 2009. [DOI:10.1038/nn0309-245] [PMID]
7. [7] N. Branan, "Reading Minds," Sci. Am. Mind, vol. 20, no. 6, pp. 8-8, 2009. [DOI:10.1038/scientificamericanmind1109-8a]
8. [8] O. Sporns, "Network analysis, complexity, and brain function," Complexity, vol. 8, no. 1, pp. 56-60, 2002. [DOI:10.1002/cplx.10047]
9. [9] J. V. Haxby, "Multivariate pattern analysis of fMRI: The early beginnings," Neuroimage, vol. 62, no. 2, pp. 852-855, Aug. 2012. [DOI:10.1016/j.neuroimage.2012.03.016] [PMID] [PMCID]
10. [10] J. V. Haxby, A. C. Connolly, and J. S. Guntupalli, "Decoding Neural Representational Spaces Using Multivariate Pattern Analysis," Annu. Rev. Neurosci., vol. 37, no. 1, pp. 435-456, 2014. [DOI:10.1146/annurev-neuro-062012-170325] [PMID]
11. [11] D. J. Heeger and D. Ress, "WHAT DOES f MRI TELL US ABOUT NEURONAL ACTIVITY?," Nat. Rev. Neurosci., vol. 3, no. February, pp. 142-151, 2002. [DOI:10.1038/nrn730] [PMID]
12. [12] K. Smith, "Brain imaging: fMRI 2.0," Nature, vol. 484, no. 7392, pp. 24-26, 2012. [DOI:10.1038/484024a] [PMID]
13. [13] P. S. Bradley and O. L. Mangasarian, "Feature selection via concave minimization and support vector machines.," in ICML, 1998, vol. 98, pp. 82-90.
14. [14] H. R. Holger Mohr, Uta Wolfenstelle, Steffi Frimmel, H. Mohr, U. Wolfensteller, S. Frimmel, H. Ruge, and H. R. Holger Mohr, Uta Wolfenstelle, Steffi Frimmel, "Sparse regularization techniques provide novel insights into outcome integration processes," Neuroimage, vol. 104, pp. 163-176, 2015. [DOI:10.1016/j.neuroimage.2014.10.025] [PMID]
15. [15] J. V. Haxby et al., "A common, high-dimensional model of the representational space in human ventral temporal cortex," Neuron, vol. 72, no. 2, pp. 404-416, 2011. [DOI:10.1016/j.neuron.2011.08.026] [PMID] [PMCID]
16. [16] D. D. Cox and R. L. Savoy, "Functional magnetic resonance imaging (fMRI) 'brain reading': Detecting and classifying distributed patterns of fMRI activity in human visual cortex," Neuroimage, vol. 19, no. 2, pp. 261-270, Jun. 2003. [DOI:10.1016/S1053-8119(03)00049-1]
17. [17] T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, and J. L. Gallant, "Bayesian Reconstruction of Natural Images from Human Brain Activity," Neuron, vol. 63, no. 6, pp. 902-915, 2009. [DOI:10.1016/j.neuron.2009.09.006] [PMID] [PMCID]
18. [18] J. S. Guntupalli, M. Hanke, Y. O. Halchenko, A. C. Connolly, P. J. Ramadge, and J. V Haxby, "A model of representational spaces in human cortex," Cereb. cortex, vol. 26, no. 6, pp. 2919-2934, 2016. [DOI:10.1093/cercor/bhw068] [PMID] [PMCID]
19. [19] A. Lorbert and P. J. Ramadge, "Kernel hyperalignment," in Advances in Neural Information Processing Systems, 2012, pp. 1790-1798.
20. [20] X. Ma, C.-A. Chou, H. Sayama, and W. A. Chaovalitwongse, "Brain response pattern identification of fMRI data using a particle swarm optimization-based approach," Brain Informatics, vol. 3, no. 3, pp. 181-192, 2016. [DOI:10.1007/s40708-016-0049-z] [PMID] [PMCID]
21. [21] M.-H. Kao, A. Mandal, and J. Stufken, "Constrained multiobjective designs for functional magnetic resonance imaging experiments via a modified non-dominated sorting genetic algorithm," J. R. Stat. Soc. Ser. c (applied Stat., vol. 61, no. 4, pp. 515-534, 2012. [DOI:10.1111/j.1467-9876.2011.01036.x]
22. [22] D. E. Osher, R. R. Saxe, K. Koldewyn, J. D. E. Gabrieli, N. Kanwisher, and Z. M. Saygin, "Structural Connectivity Fingerprints Predict Cortical Selectivity for Multiple Visual Categories across Cortex," Cereb. Cortex, vol. 26, no. 4, pp. 1668-1683, 2016. [DOI:10.1093/cercor/bhu303] [PMID] [PMCID]
23. [23] S. Sun, "A survey of multi-view machine learning," Neural Comput. Appl., vol. 23, no. 7-8, pp. 2031-2038, 2013. [DOI:10.1007/s00521-013-1362-6]
24. [24] R. Polikar, "Ensemble based systems in decision making," Circuits Syst. Mag. IEEE, vol. 6, no. 3, pp. 21-44, 2006. [DOI:10.1109/MCAS.2006.1688199]
25. [25] O. Sagi and L. Rokach, "Ensemble learning: A survey," Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 8, no. 4, p. e1249, 2018. [DOI:10.1002/widm.1249]
26. [26] M. N. Hebart, K. Görgen, and J.-D. Haynes, "The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data," Front. Neuroinform., vol. 8, p. 88, Jan. 2015. [DOI:10.3389/fninf.2014.00088] [PMID] [PMCID]
27. [27] K. Seeliger et al., "Convolutional neural network-based encoding and decoding of visual object recognition in space and time," Neuroimage, no. July, pp. 1-14, 2017.
28. [28] M. P. Eckstein et al., "Neural decoding of collective wisdom with multi-brain computing," Neuroimage, vol. 59, no. 1, pp. 94-108, 2012. [DOI:10.1016/j.neuroimage.2011.07.009] [PMID]
29. [29] L. I. Kuncheva and J. J. Rodríguez, "Classifier ensembles for fMRI data analysis: an experiment," Magn. Reson. Imaging, vol. 28, no. 4, pp. 583-593, 2010. [DOI:10.1016/j.mri.2009.12.021] [PMID]
30. [30] W. J. Faithfull, J. J. Rodríguez, and L. I. Kuncheva, "Combining univariate approaches for ensemble change detection in multivariate data," Inf. Fusion, vol. 45, no. January 2018, pp. 202-214, 2019. [DOI:10.1016/j.inffus.2018.02.003]
31. [31] S. Sun, C. Zhang, and Y. Lu, "The random electrode selection ensemble for EEG signal classification," Pattern Recognit., vol. 41, no. 5, pp. 1680-1692, May 2008. [DOI:10.1016/j.patcog.2007.10.023]
32. [32] T. M. Cover and J. A. Thomas, Elements of information theory. John Wiley & Sons, 2012.
33. [33] S. Guia csu, Information Theory with Applications. New York : McGraw-Hill, 1977.
34. [34] Y. Saeys, I. Inza, and P. Larranaga, "A review of feature selection techniques in bioinformatics," Bioinformatics, vol. 23, no. 19, pp. 2507-2517, Oct. 2007. [DOI:10.1093/bioinformatics/btm344] [PMID]
35. [35] C. A. Chou, K. Kampa, S. H. Mehta, R. F. Tungaraza, W. A. Chaovalitwongse, and T. J. Grabowski, "Voxel selection framework in multi-voxel pattern analysis of fMRI data for prediction of neural response to visual stimuli," IEEE Trans. Med. Imaging, vol. 33, no. 4, pp. 925-934, Apr. 2014. [DOI:10.1109/TMI.2014.2298856] [PMID]
36. [36] C. Cabral, M. Silveira, and P. Figueiredo, "Automatic classification of cognitive states," 1st Port. Meet. Biomed. Eng. ENBENG 2011, no. February, 2011. [DOI:10.1109/ENBENG.2011.6026089]
37. [37] V. Gómez-Verdejo, M. Martínez-Ramón, J. Florensa-Vila, and A. Oliviero, "Analysis of fMRI time series with mutual information," Med. Image Anal., vol. 16, no. 2, pp. 451-458, 2012. [DOI:10.1016/j.media.2011.11.002] [PMID]
38. [38] O. Hourani, N. M. Charkari, and S. Jalili, "Voxel selection framework based on meta-heuristic search and mutual information for brain decoding," Int. J. Imaging Syst. Technol., vol. 29, no. 4, pp. 663-676, Jun. 2019. [DOI:10.1002/ima.22353]
39. [39] M. Jenkinson, C. F. Beckmann, T. E. J. J. Behrens, M. W. Woolrich, and S. M. Smith, "Fsl," Neuroimage, vol. 62, no. 2, pp. 782-790, 2012. [DOI:10.1016/j.neuroimage.2011.09.015] [PMID]
40. [40] R. Buyya, G. M. Mohay, P. Roe, and IEEE Computer Society., "Sun Grid Engine: towards creating a compute power grid," in Proceedings of the 1st International Symposium on Cluster Computing and the Grid, 2001, p. 704.
41. [41] R. S. Desikan et al., "An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest," Neuroimage, vol. 31, no. 3, pp. 968-980, 2006. [DOI:10.1016/j.neuroimage.2006.01.021] [PMID]
42. [42] J. L. Lancaster et al., "Automated Talairach atlas labels for functional brain mapping," Hum. Brain Mapp., vol. 10, no. 3, pp. 120-131, 2000. https://doi.org/10.1002/1097-0193(200007)10:3<120::AID-HBM30>3.0.CO;2-8 [DOI:10.1002/1097-0193(200007)10:33.0.CO;2-8]
43. [43] P. Talairach, J. & Tournoux, "Co-planar stereotaxic atlas of the human brain," Clin. Neurol. Neurosurg., vol. 91, no. 3, pp. 277-278, 1989. [DOI:10.1016/0303-8467(89)90128-5]
44. [44] M. W. Woolrich, B. D. Ripeky, J. M. Brady, and S. M. Smith, "Temporal Autocorrelation in Univariante Linear Modelling of FMRI Data," Neuroimage, vol. 14, no. 6, pp. 1370-1386, 2001. [DOI:10.1006/nimg.2001.0931] [PMID]
45. [45] J. V Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, "Distributed and Overlapping Representations of Face and Objects in Ventral Temporal Cortex," Science (80-. )., vol. 293, no. 5539, pp. 2425-2430, 2001. [DOI:10.1126/science.1063736] [PMID]
46. [46] K. J. Duncan, C. Pattamadilok, I. Knierim, and J. T. Devlin, "Consistency and variability in functional localisers," Neuroimage, vol. 46, no. 4, pp. 1018-1026, 2009. [DOI:10.1016/j.neuroimage.2009.03.014] [PMID] [PMCID]
47. [47] J. M. Walz, R. I. Goldman, M. Carapezza, J. Muraskin, T. R. Brown, and P. Sajda, "Simultaneous EEG-fMRI Reveals Temporal Evolution of Coupling between Supramodal Cortical Attention Networks and the Brainstem," J. Neurosci., vol. 33, no. 49, pp. 19212-19222, 2013. [DOI:10.1523/JNEUROSCI.2649-13.2013] [PMID] [PMCID]
48. [48] C. A. Chou, K. Kampa, S. H. Mehta, R. F. Tungaraza, W. A. Chaovalitwongse, and T. J. Grabowski, "Voxel selection framework in multi-voxel pattern analysis of fMRI data for prediction of neural response to visual stimuli," IEEE Trans. Med. Imaging, vol. 33, no. 4, pp. 925-934, Apr. 2014. [DOI:10.1109/TMI.2014.2298856] [PMID]
49. [49] M. Yousefnezhad and D. Zhang, "Multi-Objective Cognitive Model: a Supervised Approach for Multi-subject fMRI Analysis," Neuroinformatics, vol. 17, no. 2, pp. 197-210, 2019. [DOI:10.1007/s12021-018-9394-9] [PMID]
50. [50] H. R. Holger Mohr, Uta Wolfenstelle, Steffi Frimmel, "Sparse regularization techniques provide novel insights into outcome integration processes," Neuroimage, vol. 104, pp. 163-176, 2015. [DOI:10.1016/j.neuroimage.2014.10.025] [PMID]

Add your comments about this article : Your username or Email:
CAPTCHA

Send email to the article author


Rights and permissions
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

© 2015 All Rights Reserved | Signal and Data Processing