Traditional regularization parameters such as L1 and L2 are added to the cost function for neural network learning to improve learning ability and generate sparsity in the solution. L2 regularization adds  the squared value of the weights to the cost function, whereas L1 regularization adds the absolute value of the weights. This study proposes an online gradient method with a novel double regularization (OGDr) for enhancing the learning ability of pi-sigma neural networks (PSNNs). The L1 and L2 regularization methods are combined in the double regularization method, which is frequently used in several machine learning frameworks. To improve the suggested method’s performance learning ability, we applied the XOR problem, parity problem, Gabor function problem, and sonar benchmark challenge. The numerical examples of cases, OGL1, and OGL2 were compared. The OGDr has a good learning accuracy, according to numerical statistics. In addition, unlike OGL1 and OGL2, the error decreases monotonically, and the gradient of the error function approaches zero throughout learning.

online gradient method, pi-sigma neural networks, double regularization, L2 regularization.

Received: September 8, 2022; Accepted: October 29, 2022; Published: December 6, 2022

How to cite this article: Khidir Shaib Mohamed, Osman Abdalla Adam Osman, Khalid Makin, Mohammed Nour A. Rabih and D. S. Muntasir Suhail, Training pi-sigma neural network using double regularization, Far East Journal of Electronics and Communications 26 (2022), 1-16. http://dx.doi.org/10.17654/0973700622002

This Open Access Article is Licensed under Creative Commons Attribution 4.0 International License

Refernces:

[1] Y. Shin and J. Ghosh, The pi-sigma network: an efficient higher-order neural network for pattern classification and function approximation, IJCNN-91-Seattle International Joint Conference on Neural Networks, IEEE, Vol. 1, 1991, pp. 13-18.
[2] M. Heywood and P. Noakes, A framework for improved training of sigma-Pi networks, IEEE Transactions on Neural Networks 6(4) (1995), 893-903.
[3] S. Dehuri and S. B. Cho, Evolutionarily optimized features in functional link neural network for classification, Expert Systems with Applications 37(6) (2010), 4379-4391. https://doi.org/10.1016/j.eswa.2009.11.090.
[4] C. K. Li, A sigma-pi-sigma neural network (SPSNN), Neural Processing Letters 17(1) (2003), 1-19. https://doi.org/10.1023/A:1022967523886.
[5] E. Akdeniz, E. Egrioglu, E. Bas and U. Yolcu, An ARMA type pi-sigma artificial neural network for nonlinear time series forecasting, Journal of Artificial Intelligence and Soft Computing Research 8 (2018), 121-132.
https://doi.org/10.1515/jaiscr-2018-0009.
[6] N. Panda and S. K. Majhi, Improved spotted hyena optimizer with space transformational search for training pi-sigma higher order neural network, Computational Intelligence 36(1) (2020), 320-350.
https://doi.org/10.1111/coin.12272.
[7] N. A. Husaini, R. Ghazali, N. M. Nawi and L. H. Ismail, Pi-sigma neural network for temperature forecasting in Batu Pahat, International Conference on Software Engineering and Computer Systems, Springer, Berlin, Heidelberg, 2011. https://doi.org/10.1007/978-3-642-22191-0_46.
[8] E. Bas, E. Egrioglu and E. Kolemen, A novel intuitionistic fuzzy time series method based on bootstrapped combined pi-sigma artificial neural network, Engineering Applications of Artificial Intelligence 114 (2022), 105030.
https://doi.org/10.1016/j.engappai.2022.105030.
[9] A. Krogh and J. Hertz, A simple weight decay can improve generalization, Advances in Neural Information Processing Systems, 1991, pp. 950-957.
[10] A. S. Weigend, D. E. Rumelhart and B. A. Huberman, Generalization by weight-elimination applied to currency exchange rate prediction, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks, IEEE, 1991, pp. 2374-2379. doi: 10.1109/IJCNN.1991.155287.
[11] J. Moody and T. Rögnvaldsson, Smoothing regularizers for projective basis function networks, Advances in Neural Information Processing Systems, 1996, pp. 585-591.
[12] E. Phaisangittisagul, An analysis of the regularization between L2 and dropout in single hidden layer neural network, 2016 7th International Conference on Intelligent Systems, Modelling and Simulation (ISMS), IEEE, 2016, pp. 174-179. doi: 10.1109/ISMS.2016.14.
[13] Y. Liu, D. Yang and C. Zhang, Relaxed conditions for convergence analysis of online back-propagation algorithm with L2 regularizer for sigma-Pi-sigma neural network, Neurocomputing 272 (2018), 163-169.
https://doi.org/10.1016/j.neucom.2017.06.057.
[14] Z. Wei, Q. Li, J. Wei and W. Bian, Neural network for a class of sparse optimization with L0-regularization, Neural Networks 151 (2022), 211-221. https://doi.org/10.1016/j.neunet.2022.03.033.
[15] Y. Zhang, Y. Shi, L. Ma, J. Wu, L. Wang and H. Hong, Blind natural image deblurring with edge preservation based on L0-regularized gradient prior, Optik 225 (2021), 165735. https://doi.org/10.1016/j.ijleo.2020.165735.
[16] F. Wu and W. Bian, Accelerated iterative hard thresholding algorithm for $$ l_0 $$ l0 regularized regression problem, J. Global Optim. 76(4) (2021), 819-840. https://doi.org/10.1007/s10898-019-00826-6.
[17] J. Kuha, AIC and BIC: comparisons of assumptions and performance, Sociol. Methods Res. 33(2) (2004), 188-229. https://doi.org/10.1177/0049124103262065.
[18] S. Mc Loone and G. Irwin, Improving neural network training solutions using regularization, Neurocomputing 37(1-4) (2001), 71-90.
https://doi.org/10.1016/S0925-2312(00)00314-3.
[19] M. Schmidt, G. Fung and R. Rosales, Fast optimization methods for l1 regularization: a comparative study and two new approaches, European Conference on Machine Learning, Springer, Berlin, Heidelberg, 2007. https://doi.org/10.1007/978-3-540-74958-5_28.
[20] Z. Xu, H. Zhang, Y. Wang, X. Chang and Y. Liang, L1/2 regularization, Science China Information Sciences 53(6) (2010), 1159-1169.
https://doi.org/10.1007/s11432-010-0090-0.
[21] Y. Liang, H. Chai, X. Y. Liu, Z. B. Xu, H. Zhang and K. S. Leung, Cancer survival analysis using semi-supervised learning method based on Cox and AFT models with L1/2 regularization, BMC Medical Genomics 9(1) (2016), 1-11. https://doi.org/10.1186/s12920-016-0169-6.
[22] H. K. Jiang and Y. Liang, The L1/2 regularization network Cox model for analysis of genomic data, Computers in Biology and Medicine 100 (2018), 203-208. https://doi.org/10.1016/j.compbiomed.2018.07.009.
[23] H. Zou and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol. 67(2) (2005), 301-320. https://www.jstor.org/stable/3647580.
[24] L. Wang, J. Zhu and H. Zou, The doubly regularized support vector machine, Statist. Sinica 16 (2006), 589-615. https://www.jstor.org/stable/24307560.
[25] W. Shen, J. Wang and S. Ma, Doubly regularized portfolio with risk minimization, Twenty-eighth AAAI Conference on Artificial Intelligence, 2014. https://doi.org/10.1609/aaai.v28i1.8906.
[26] P. Milanez-Almeida, A. J. Martins, R. N. Germain and J. S. Tsang, Cancer prognosis with shallow tumor RNA sequencing, Nature Medicine 26(2) (2020), 188-192. https://doi.org/10.1038/s41591-019-0729-3