izmertechИзмерительная техникаIzmeritel`naya Tekhnika0368-10252949-5237ФГУП "ВНИИФТРИ"10.32446/0368-1025it.2024-4-23-31izmertech-2140Research ArticleОПТИКО-ФИЗИЧЕСКИЕ ИЗМЕРЕНИЯOPTICOPHYSICAL MEASUREMENTSВосстановление изображений объектов: метод реконструкции с использованием цифровых внеосевых голограмм и генеративно-состязательной нейронной сетиReconstructing images of objects: method for reconstructing images from digital off-axis holograms based on a generative adversarial neural networkhttps://orcid.org/0000-0001-5914-7275КирийС. А.KiriyS. A.

Кирий Семен Алексеевич.

Москва

Semen A. Kiriy.

Moscow

semakiriy@kiraksa.ruhttps://orcid.org/0009-0004-5455-9181СвистуновА. С.SvistunovA. S.

Свистунов Андрей Сергеевич.

Москва

Andrey S. Svistunov.

Moscow

svistunov.andrey.sergeevich@gmail.comhttps://orcid.org/0000-0003-0914-9736РымовД. А.RymovD. A.

Рымов Дмитрий Андреевич.

Москва

Dmitriy A. Rymov.

Moscow

rymov.d.a@gmail.comhttps://orcid.org/0000-0002-7369-1565СтариковР. С.StarikovR. S.

Стариков Ростислав Сергеевич.

Москва

Rostislav S. Starikov.

Moscow

rstarikov@mail.ruhttps://orcid.org/0000-0001-7816-5989ШифринаА. В.ShifrinaA. V.

Шифрина Анна Владимировна.

Москва

Anna V. Shifrina.

Moscow

avshifrina@gmail.comhttps://orcid.org/0000-0003-3556-2663ЧерёмхинП. А.CheremkhinP. A.

Черёмхин Павел Аркадьевич.

Москва

Pavel A. Cheremkhin.

Moscow

cheremhinpavel@mail.ruНациональный исследовательский ядерный университет «МИФИ»РоссияNational Research Nuclear University MEPhI (Moscow Engineering Physics Institute)Russian Federation202410062024042331Copyright © ФГУП "ВНИИФТРИ", 20242024ФГУП "ВНИИФТРИ"ФГУП "ВНИИФТРИ"https://www.izmt.ru/jour/about/submissions#copyrightNoticehttps://www.izmt.ru/jour/article/view/2140

Рассмотрено использование цифровой голографии для реконструкции изображений объектов, расположенных в различных сечениях 3D-сцены. Такая реконструкция позволяет исследовать различные материалы, характеризовать микрочастицы в некотором объёме среды, анализировать содержание микропластика в водоёмах. Предложен метод реконструкции изображений объектов с помощью цифровых внеосевых голограмм и генеративно-состязательной нейронной сети. Генеративно-состязательная нейронная сеть служит для восстановления сечений 3D-сцены, в которых расположены внеосевые объекты. Показано, что применение нейронных сетей повышает скорость и качество восстановления, а также уменьшает уровень шума изображения. Предложенный метод апробирован на численно синтезированных и оптически измеренных цифровых голограммах. Данным методом с помощью одной синтезированной голограммы восстановлены восемь сечений 3D-сцены. Получен средний индекс структурного сходства не менее 0,73. Экспериментально зарегистрированы наборы внеосевых цифровых голограмм фазовых объектов, выведенных на пространственно-временные модуляторы света для формирования сечений 3D-сцены. При восстановлении изображений объектов с применением оптически полученных голограмм средний индекс структурного сходства по сечениям сцены составил 0,83. Предложенный метод позволяет качественно восстанавливать изображения объектов и будет полезен при анализе микро- и макрообъектов, в том числе в медико-биологических приложениях и метрологии, при характеризации материалов, поверхностей и объёмных сред.

The reconstruction of object images that are located in 3D scene cross-sections using digital holography is described. The potential of generative adversarial networks for reconstructing cross-sections of 3D scenes composed of multiple layers of off-axis objects from holograms is investigated. Such scenes consist of a series of sections with objects that are not aligned with the camera’s axis. Digital holograms were used to reconstruct images of cross-sectional views of 3D scenes. It has been shown that the use of neural networks increases the speed and reconstruction quality, and reduces the image noise. A method for reconstructing images of objects using digital off-axis holograms and a generative adversarial neural network is proposed. The proposed method was tested on both numerically simulated and experimentally captured digital holograms. It was able to successfully reconstruct up to 8 cross-sections of a 3D scene from a single hologram. It was obtained that an average structural similarity index measure was equal to at least 0.73. Based on optically registered holograms, the method allowed us to reconstruct object image cross-sections of a 3D scene with a structural similarity index measure over cross-sections of a 3D scene of equal to 0.83. Therefore, the proposed technique provides the possibility for high-quality object image reconstruction and could be utilized in the analysis of micro- and macroobjects, including medical and biological applications, metrology, characterization of materials, surfaces, and volume media.

цифровая голографияреконструкция изображенийгенеративно-состязательная нейросетьвнеосевая голографияхарактеризация объектовмашинное обучение3D-сценапространственно-временной модулятор светаdigital holographyimage reconstructiongenerative adversarial networkoff-axis holographyobject characterizationmachine learning3D-scenespatial light modulatorРабота выполнена при финансовой поддержке Российского научного фонда (РНФ), грант № 22-79-10340.The work was supported by the Russian Science Foundation (RSF), Grant no. 22-79-10340.ReferencesSchnars U., Jueptner W. Digital holography: Digital hologram recording, numerical reconstruction, and related techniques. Springer, Berlin Heidelberg (2005). https://doi.org/10.1007/b138284Schnars U., Jueptner W. Digital holography: Digital hologram recording, numerical reconstruction, and related techniques. Springer, Berlin Heidelberg (2005). https://doi.org/10.1007/b138284Wang Z., Bianco V., Maffettone P. L., Ferraro P. Holographic flow scanning cytometry overcomes depth of focus limits and smartly adapts to microfluidic speed. Lab on a Chip, 23, 2316–2326 (2023). https://doi.org/10.1039/D3LC00063JWang Z., Bianco V., Maffettone P. L., Ferraro P. Holographic flow scanning cytometry overcomes depth of focus limits and smartly adapts to microfluidic speed. Lab on a Chip, 23, 2316–2326 (2023). https://doi.org/10.1039/D3LC00063JSingh V., Joshi R., Tayal S., Mehta D. S. Speckle-free common-path quantitative phase imaging with high temporal phase stability using a partially spatially coherent multi-spectral light source. Laser Physics Letters, 16, 025601 (2019). https://doi.org/10.1088/1612-202X/AAF179Singh V., Joshi R., Tayal S., Mehta D. S. Speckle-free common-path quantitative phase imaging with high temporal phase stability using a partially spatially coherent multi-spectral light source. Laser Physics Letters, 16, 025601 (2019). https://doi.org/10.1088/1612-202X/AAF179Calore D., Fraticelli N. State of the art offshore in situ monitoring of microplastic. Microplastics, 1, 640–650 (2022). https://doi.org/10.3390/MICROPLASTICS1040044Calore D., Fraticelli N. State of the art offshore in situ monitoring of microplastic. Microplastics, 1, 640–650 (2022). https://doi.org/10.3390/MICROPLASTICS1040044Zhang W., Li B., Zhang X., Shi C. Off-axis digital holography based on the Sagnac interferometer. Laser Physics Letters, 18, 035202 (2021). https://doi.org/10.1088/1612-202X/ABDECBZhang W., Li B., Zhang X., Shi C. Off-axis digital holography based on the Sagnac interferometer. Laser Physics Letters, 18, 035202 (2021). https://doi.org/10.1088/1612-202X/ABDECBBondareva A. P., Cheremkhin P. A., Evtikhiev N. N., et al. Measurement of characteristics and phase modulation accuracy increase of LC SLM “HoloEye PLUTO VIS”. Journal of Physics: Conference Series, 536(1), 012011 (2014). https://doi.org/10.1088/1742-6596/536/1/012011Bondareva A. P., Cheremkhin P. A., Evtikhiev N. N., et al. Measurement of characteristics and phase modulation accuracy increase of LC SLM “HoloEye PLUTO VIS”. Journal of Physics: Conference Series, 536(1), 012011 (2014). https://doi.org/10.1088/1742-6596/536/1/012011Upatnieks J., Leith E. N. Wavefront reconstruction with diffused illumination and three-dimensional objects. Journal of the Optical Society of America, 54, 1295–1301 (1964). https://doi.org/10.1364/JOSA.54.001295Upatnieks J., Leith E. N. Wavefront reconstruction with diffused illumination and three-dimensional objects. Journal of the Optical Society of America, 54, 1295–1301 (1964). https://doi.org/10.1364/JOSA.54.001295Park J., Kang H., Stoykova E. Twin-image problem in digital holography – a survey. Chinese Optics Letters, 12, 060013 (2014). https://doi.org/10.3788/COL201412.060013Park J., Kang H., Stoykova E. Twin-image problem in digital holography – a survey. Chinese Optics Letters, 12, 060013 (2014). https://doi.org/10.3788/COL201412.060013Yamaguchi I. Phase-shifting digital holography. Optics Letters, 22, 1268–1270 (1997). https://doi.org/10.1364/OL.22.001268Yamaguchi I. Phase-shifting digital holography. Optics Letters, 22, 1268–1270 (1997). https://doi.org/10.1364/OL.22.001268Olivier T., Momey F., Denis L., Fournier C. From Fienup’s phase retrieval techniques to regularized inversion for in-line holography: tutorial. Journal of the Optical Society of America A, 36, D62–D80 (2019). https://doi.org/10.1364/JOSAA.36.000D62Olivier T., Momey F., Denis L., Fournier C. From Fienup’s phase retrieval techniques to regularized inversion for in-line holography: tutorial. Journal of the Optical Society of America A, 36, D62–D80 (2019). https://doi.org/10.1364/JOSAA.36.000D62Zeng T., Zeng T., Zhu Y., et al. Deep learning for digital holography: a review. Optics Express, 29, 40572–40593 (2021). https://doi.org/10.1364/OE.443367Zeng T., Zeng T., Zhu Y., et al. Deep learning for digital holography: a review. Optics Express, 29, 40572–40593 (2021). https://doi.org/10.1364/OE.443367Cheremkhin P. A., Evtikhiev N. N., Krasnov V. V., et al. Machine learning methods for digital holography and diffractive optics. Procedia Computer Science, 169, 440–444 (2020). https://doi.org/10.1016/j.procs.2020.02.243Cheremkhin P. A., Evtikhiev N. N., Krasnov V. V., et al. Machine learning methods for digital holography and diffractive optics. Procedia Computer Science, 169, 440–444 (2020). https://doi.org/10.1016/j.procs.2020.02.243Situ G. Deep holography. Light Advanced Manufactoring, 3, 278–300 (2022). https://doi.org/10.37188/LAM.2022.013Situ G. Deep holography. Light Advanced Manufactoring, 3, 278–300 (2022). https://doi.org/10.37188/LAM.2022.013Черёмхин П. А., Рымов Д. А., Свистунов А. С., Злоказов Е. Ю., Стариков Р. С. Нейросетевые методы в цифровой и компьютерной голографии. Обзор. Оптический Журнал, 91, 62–78 (2024). http://doi.org/10.17586/1023-5086-2024-91-03-62-78Cheremkhin P. A., Rymov D. A., Svistunov A. S., Zlokazov E. Yu., Starikov R. S. Neural-network-based methods in digital and computer-generated holography. А review. Opticheskii Zhurnal, 91, 62–78 (2024). (In Russ.) http://doi.org/10.17586/1023-5086-2024-91-03-62-78Рымов Д. А., Черёмхин П. А., Стариков Р. С. Нейросетевая реконструкция сцен с цифровых голограмм на основе извлечения амплитуды и фазы. Оптический журнал, 89, 11–19 (2022). http://doi.org/10.17586/1023-5086-2022-89-09-11-19Rymov D. A., Cheremkhin P. A., Starikov R. S., Neural-network-enabled holographic image reconstruction via amplitude and phase extraction. Journal of Optical Technology, 89(9), 511–516 (2022). https://doi.org/10.1364/JOT.89.000511Pirone D., Sirico D., Miccio L., et al. Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning. Lab on a Chip, 22, 793–804 (2022). https://doi.org/10.1039/D1LC01087EPirone D., Sirico D., Miccio L., et al. Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning. Lab on a Chip, 22, 793–804 (2022). https://doi.org/10.1039/D1LC01087ERazi A., Chen X., Wang H., et al. DH-GAN: a physics-driven untrained generative adversarial network for holographic imaging. Optics Express, 31, 10114–10135 (2023). https://doi.org/10.1364/OE.480894Razi A., Chen X., Wang H., et al. DH-GAN: a physics-driven untrained generative adversarial network for holographic imaging. Optics Express, 31, 10114–10135 (2023). https://doi.org/10.1364/OE.480894Svistunov A. S., Rymov D. A., Starikov R. S., Cheremkhin P. A. HoloForkNet: Digital Hologram Reconstruction via Multibranch Neural Network. Applied Sciences, 13 (10), 6125 (2023). https://doi.org/10.3390/app13106125Svistunov A. S., Rymov D. A., Starikov R. S., Cheremkhin P. A. HoloForkNet: Digital Hologram Reconstruction via Multibranch Neural Network. Applied Sciences, 13 (10), 6125 (2023). https://doi.org/10.3390/app13106125Goodfellow I., Pouget-Abadie J., Mirza M., et al. Generative adversarial networks. Communications of the ACM, 63, 139–144 (2014). https://doi.org/10.1145/3422622Goodfellow I., Pouget-Abadie J., Mirza M., et al. Generative adversarial networks. Communications of the ACM, 63, 139–144 (2014). https://doi.org/10.1145/3422622Tang H., Liu H., Xu D., et al. AttentionGAN: unpaired image-to-image translation using attention-guided generative adversarial networks, IEEE Trans. Neural Networks Learning Systems, 34, 1972–1987 (2023). https://doi.org/10.1109/TNNLS.2021.3105725Tang H., Liu H., Xu D., et al. AttentionGAN: unpaired image-to-image translation using attention-guided generative adversarial networks, IEEE Trans. Neural Networks Learning Systems, 34, 1972–1987 (2023). https://doi.org/10.1109/TNNLS.2021.3105725Xia J., Zhang L., Zhai Y., Zhang Y. Reconstruction method of computational ghost imaging under atmospheric turbulence based on deep learning. Laser Physics, 34, 015202 (2023). https://doi.org/10.1088/1555-6611/AD0EBFXia J., Zhang L., Zhai Y., Zhang Y. Reconstruction method of computational ghost imaging under atmospheric turbulence based on deep learning. Laser Physics, 34, 015202 (2023). https://doi.org/10.1088/1555-6611/AD0EBFLeonov M. M., Soroka A. A., Trofimov A. G. Russian language speech generation from facial video recordings using variational autoencoder. Studies in Computational Intelligence, 1120, 489–498 (2023). https://doi.org/10.1007/978-3-031-44865-2_51Leonov M. M., Soroka A. A., Trofimov A. G. Russian language speech generation from facial video recordings using variational autoencoder. Studies in Computational Intelligence, 1120, 489–498 (2023). https://doi.org/10.1007/978-3-031-44865-2_51Fevens T., Jaferzadeh K. HoloPhaseNet: fully automated deep-learning-based hologram reconstruction using a conditional generative adversarial model. Biomedical Optics Express, 13, 4032–4046 (2022). https://doi.org/10.1364/BOE.452645Fevens T., Jaferzadeh K. HoloPhaseNet: fully automated deep-learning-based hologram reconstruction using a conditional generative adversarial model. Biomedical Optics Express, 13, 4032–4046 (2022). https://doi.org/10.1364/BOE.452645Ay B. Open-set learning-based hologram verification system using generative adversarial networks. IEEE Access, 10, 25114–25124 (2022). https://doi.org/10.1109/ACCESS.2022.3155870Ay B. Open-set learning-based hologram verification system using generative adversarial networks. IEEE Access, 10, 25114–25124 (2022). https://doi.org/10.1109/ACCESS.2022.3155870Kiriy S. A., Rymov D. A., Svistunov A. S., et al. Generative adversarial neural network for 3D-hologram reconstruction. Laser Physics Letters, 21, 045201 (2024). https://doi.org/10.1088/1612-202X/ad26ebKiriy S. A., Rymov D. A., Svistunov A. S., et al. Generative adversarial neural network for 3D-hologram reconstruction. Laser Physics Letters, 21, 045201 (2024). https://doi.org/10.1088/1612-202X/ad26ebPrilepsky J. E., Manuylovich E., Freire P., Turitsyn S. K. Artificial neural networks for photonic applications – from algorithms to implementation: tutorial. Advances in Optics and Photonics, 15, 739–834 (2023). https://doi.org/10.1364/AOP.484119Prilepsky J. E., Manuylovich E., Freire P., Turitsyn S. K. Artificial neural networks for photonic applications – from algorithms to implementation: tutorial. Advances in Optics and Photonics, 15, 739–834 (2023). https://doi.org/10.1364/AOP.484119Genty G., Salmela L., Dudley J. M., et al. Machine learning and applications in ultrafast photonics. Nature Photonics, 15, 91–101 (2020). https://doi.org/10.1038/s41566-020-00716-4Genty G., Salmela L., Dudley J. M., et al. Machine learning and applications in ultrafast photonics. Nature Photonics, 15, 91–101 (2020). https://doi.org/10.1038/s41566-020-00716-4Dmitriev E., Bukharskii N., Korneev P. Powerful elliptically polarized terahertz radiation from oscillating-laser-driven discharge surface currents.Photonics, 10(7), 803 (2023). https://doi.org/10.3390/photonics10070803Dmitriev E., Bukharskii N., Korneev P. Powerful elliptically polarized terahertz radiation from oscillating-laser-driven discharge surface currents. Photonics, 10(7), 803 (2023). https://doi.org/10.3390/photonics10070803Yu Z., Li X., Bai X., et al. Computational ghost imaging through a dynamic scattering medium based on a convolutional neural network from simulation. Laser Physics Letters, 20, 055204 (2023). https://doi.org/10.1088/1612-202X/ACC245Yu Z., Li X., Bai X., et al. Computational ghost imaging through a dynamic scattering medium based on a convolutional neural network from simulation. Laser Physics Letters, 20, 055204 (2023). https://doi.org/10.1088/1612-202X/ACC245Лоскутов А. И., Якимов В. Л., Карпушев С. И. и др. Модель контроля технического состояния бортовой аппаратуры космических аппаратов на основе значений телеметрируемых параметров переходных процессов. Измерительная техника, (6), 13–20 (2023). https://doi.org/10.32446/0368-1025it.2023-6-13-20Loskutov A. I., Yakimov V. L., Karpushev S. I., et al. Model for Monitoring the Technical Condition of Onboard Equipment of Space Vehicles Based on the Telemetry Parameters of Transient Processes. Measurement Techniques, 66(6), 384–391 (2023). https://doi.org/10.1007/s11018-023-02238-1Злоказов Е. Ю., Минаева Е. Д., Родин В. Г. и др. Методы синтеза дифракционных оптических элементов: оперативное и качественное формирование трёхмерных объектов из набора плоских сечений. Измерительная техника, (11), 45–51 (2023). https://doi.org/10.32446/0368-1025it.2023-11-45-51Zlokazov E. Yu., Minaeva E. D., Rodin V. G., Starikov R. S., Cheremkhin P. A., Shifrina A. V. Methods of diffractive optical element generation for rapid, high-quality 3D image formation of objects divided into a set of plane layers, Measurement Techniques, 66(11), (2024). https://doi.org/10.1007/s11018-024-02301-5Ronneberger T. B. O., Fischer P. U-Net: convolutional networks for biomedical image segmentation. Lecture Notes in Computer Science, 9351, 1–8 (2015). https://doi.org/10.1007/978-3-319-24574-4_28Ronneberger T. B. O., Fischer P. U-Net: convolutional networks for biomedical image segmentation. Lecture Notes in Computer Science, 9351, 1–8 (2015). https://doi.org/10.1007/978-3-319-24574-4_28Lecun Y. The MNIST database of handwritten digits, available at: http://Yann.Lecun.Com/Exdb/Mnist/ (accessed: 01 April 2024).Lecun Y. The MNIST database of handwritten digits, available at: http://Yann.Lecun.Com/Exdb/Mnist/ (accessed: 01 April 2024).Wang Z., Bovik A. C., Sheikh H. R., Simoncelli E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13, 600–612 (2004). https://doi.org/10.1109/tip.2003.819861Wang Z., Bovik A. C., Sheikh H. R., Simoncelli E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13, 600–612 (2004). https://doi.org/10.1109/tip.2003.819861Миниханов Т. З., Злоказов Е. Ю., Стариков Р. С., Черёмхин П. А. Временная динамика модуляции фазы жидкокристаллического пространственно-временного модулятора света. Измерительная техника, (12), 35–39 (2023). https://doi.org/10.32446/0368-1025it.2023-12-35-39Minikhanov T. Z., Zlokazov E.Yu., Starikov R. S., Cheremkhin P. A. Phase modulation time dynamics of the liquid-crystal spatial light modulator. Measurement Techniques, 66(12), (2024). https://doi.org/10.1007/s11018-024-02309-x

The authors declare that there are no conflicts of interest present.