Application of a nonparametric technique for testing the hypothesis of independence of random variables in conditions of a large volume of statistical data

The problem of testing the hypothesis about the independence of random variables in conditions of large volumes of statistical data is considered. The results of solving the problem are necessary when estimating probability densities of random variables and synthesizing information processing algorithms. A nonparametric technique is proposed for testing the hypothesis about the independence of random variables in a sample containing a large amount of statistical data. The technique is based on compression of the initial statistical information by decomposition of the range of values of random variables. The generated data array consists of the centers of sampling intervals and the corresponding frequencies of observations from the original sample. The information obtained is used in the construction of a nonparametric pattern recognition algorithm corresponding to the maximum likelihood criterion. The evaluation of the distribution laws in classes is carried out under the assumption of independence and dependence of the compared random variables. When restoring the laws of distribution of random variables in classes, regression estimates of probability densities are used. Under these conditions, estimates of the probabilities of pattern recognition errors in classes are calculated. According to their minimum value, a decision is made on the independence or dependence of random variables. The technique was applied in the analysis of remote sensing data of forest areas, linear and nonlinear dependencies between pairs of spectral characteristics of the objects of study were determined.

Keywords:

1. Pugachev V. S., Teoriya veroyatnostej i matematicheskaya statistika [Probability theory and mathematical statistics: study guide], Moscow, Fizmatlit Publ., 2002, 496 p. (In Russ.)

2. Lapko A. V., Lapko V. A., Bakhtina A. V., Optoelectronics, Instrumentation and Data Processing, 2022, vol. 57, no. 6, pp. 639– 648. https://doi.org/10.3103/S8756699021060078

3. Lapko A. V., Lapko V. A., Bakhtina A. V., Measurement Techniques, 2022, vol. 65, no. 1, pp. 17–23. https://doi.org/10.1007/s11018-022-02043-2

4. Lapko A. V., Lapko V. A., Bakhtina A. V. Comparison of the methodology for hypothesis testing of the independence of two-dimensional random variables based on a nonparametric classifier. Scientific and Technical Information Processing, 2022, no. 1, pp. 45–56. (In Russ.)

5. Lapko A. V., Lapko V. A. Optoelectronics, Instrumentation and Data Processing, 2014, vol. 50, no 2, pp. 148–153. https://doi.org/10.3103/S875669901402006X

6. Parzen E. Annals of Mathematical Statistics, 1962, vol. 33, nо. 3, pp. 1065–1076. https://doi.org/10.1214/aoms/1177704472

7. Epanechnikov V. A. Theory of Probability & Its Applications, 1969, vol. 14, no. 1, pp. 156–161. https://doi.org/10.1137/1114019

8. Sturgess H. A., Journal of the American Statistical Association, 1926, vol. 21, pp. 65–66. https://doi.org/10.1080/01621459.1926.10502161

9. Heinhold I., Gaede K. W., Ingeniur-Statistic, München, Wien, Springler Verlag, 1964, 352 p. (In German)

10. Lemeshko B. Yu., Chimitova E. V., On the selection of the number of intervals in the criteria of agreement of type χ2, Industrial Laboratory. Diagnostics of Materials, 2003, vol. 69, no. 1, pp. 61– 67. (In Russ.)

11. Hacine Gharbi A., Ravier P., Harba R., Mohamadi T., Pattern Recognition Letters, 2012, vol. 33, no. 10, pp. 1302–1308. https://doi.org/10.1016/j.patrec.2012.02.022

12. Scott D. W., Multivariate Density Estimation: Theory, Practice, and Visualization, 2nd Edition, NY, John Wiley & Sons, 2015, 384 p.

13. Devroye L., Lugosi G., Test, 2004, vol. 13, no. 1, pp. 129– 145. https://doi.org/10.1007/BF02603004

14. Lapko A. V., Lapko V. A., Measurement Techniques, 2013, vol. 56, no. 7, pp. 763–767. https://doi.org/10.1007/s11018-013-0279-x

15. Lapko A. V., Lapko V. A., Measurement Techniques, 2016, vol. 59, no. 2, pp. 122–126. https://doi.org/10.1007/s11018-016-0928-y

16. Lapko A. V., Lapko V. A., Measurement Techniques, 2019, vol. 62, no. 1, pp. 16–22. https://doi.org/10.1007/s11018-019-01579-0

17. Rudemo M., Empirical choice of histograms and kernel density estimators, Scandinavian Journal of Statistics, 1982, vol. 9, no. 2, pp. 65–78.

18. Hall P., Annals of Statistics, 1983, vol. 11, no. 4, pp. 1156– 1174. https://doi.org/10.1214/aos/1176346329

19. Bowman A. W., Biometrika, 1984, vol. 71, no. 2, pp. 353– 360. https://doi.org/10.1093/BIOMET/71.2.353

20. Jiang M., Provost S. B., Journal of Statistical Computation and Simulation, 2014, vol. 84, no. 3, pp. 614–627. https://doi.org/10.1080/00949655.2012.721366

21. Dutta S., Communications in Statistics – Simulation and Computation, 2016, vol. 45, no. 2, pp. 472–490. https://doi.org/10.1080/03610918.2013.862275

22. Heidenreich N.-B., Schindler A., Sperlich S., AStA Advances in Statistical Analysis, 2013, vol. 97, no. 4, pp. 403–433. https://doi.org/10.1007/s10182-013-0216-y

23. Li Q., Racine J. S., Nonparametric Econometrics: Theory and Practice, Princeton, Princeton University Press, 2007, 768 p.

24. Sharakshaneh А. S., Zheleznov I. G., Ivnitskij V. А., Slozhnye sistemy, Moscow, Vysshaya shkola Publ., 1977, 247 p. (In Russ.)

25. Dvorkin B., European program GMES and the challenging constellation of Sentinel satellites, Geomatics, 2011, no. 3, pp. 14– 26. (In Russ.)

26. Goryainov V. B., Pavlov I. V., Tsvetkova G. M., Teskin O. I. Matematicheskaya statistika: textbook for universities, Moscow, MGTU im. N. E. Baumana Publ., 2001, 424 p. (In Russ.)