Maximum Penalized Likelihood Estimation of Shannon and Past Entropy under Diverse Sampling Schemes for the Geometric Distribution

Entropy measures are crucial in measuring uncertainty and unpredictability in complex systems and have found applications in various fields, including reliability theory, economic statistics, cybersecurity, pattern recognition, and artificial intelligence. This paper explores the estimation of Shannon and Past entropy of the Geometric distribution with Maximum Penalised Likelihood Estimation (MPLE), a method that aims to reduce overfitting and increase robustness in small or sparse samples. In contrast to traditional Maximum Likelihood Estimation (MLE), MPLE employs penalty functions to balance the accuracy and complexity of the model. There are two penalty structures considered: a Quadratic (ridge-type) penalty and a Lasso-type penalty. The results of the proposed estimators are compared under a complete sampling scheme, a right-censored sampling scheme, and a Type I censored sampling scheme, using extensive Monte Carlo simulations of 10,000 iterations with different sample sizes (n = 25, 50, 75) and parameter values (0.15, 0.5, 0.9). The criteria applied in evaluation are bias, mean square error (MSE) and relative efficiency (RE). The findings repeatedly indicate that the Quadratic Penalty 1 (QP1) estimator is the best with the lowest bias, least MSE and maximum RE in all cases. It is essential to note that QP1 can sustain a RE of above 80% with moderate sample sizes (n = 50) and mid-range parameter values. The results are helpful in entropy applications, especially in the design of cryptographic systems, the analysis of random number generators, and reliability analysis. The suggested MPLE framework enhances the accuracy of estimates and is robust, thereby contributing to improved predictive performance and the development of safe systems across a wide range of disciplines.