Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement (2024)

research-article

Authors: Xiaoya Chong, Howard Leung, Qing Li, Jianhua Yao, Niyun Zhou

IEEE/ACM Transactions on Computational Biology and Bioinformatics, Volume 21, Issue 5

Pages 1299 - 1310

Published: 25 March 2024 Publication History

Metrics

Total Citations0Total Downloads1

Last 12 Months1

Last 6 weeks1

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

Manage my Alerts

New Citation Alert!

Please log in to your account

Get Access

    • Get Access
    • References
    • Media
    • Tables
    • Share

Abstract

Cryo-EM in single particle analysis is known to have low SNR and requires to utilize several frames of the same particle sample to restore one high-quality image for visualizing that particle. However, the low SNR of cryo-EM movie and motion caused by beam striking make the task very challenging. Video enhancement algorithms in computer vision shed new light on tackling such tasks by utilizing deep neural networks. However, they are designed for natural images with high SNR. Meanwhile, the lack of ground truth in cryo-EM movie seems to be one major limiting factor of the progress. Hence, we present a synthetic cryo-EM movie generation pipeline, which can produce realistic diverse cryo-EM movie datasets with low-SNR movie frames and multiple ground truth values. Then we propose a deep spatio-temporal network (DST-Net) for cryo-EM movie frame enhancement trained on our synthetic data. Spatial and temporal features are first extracted from each frame. Spatio-temporal fusion and high-resolution re-constructor are designed to obtain the enhanced output. For evaluation, we train our model on seven synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results show that DST-Net can achieve better enhancement performance both quantitatively and qualitatively compared with others.

References

[1]

E. Callaway, “The revolution will not be crystallized: A new method sweeps through structural biology,” Nature News, vol. 525, no. 7568, 2015, Art. no.

[2]

W. Kühlbrandt, “The resolution revolution,” Science, vol. 343, no. 6178, pp. 1443–1444, 2014.

[3]

J. Zhang, Z. Wang, Y. Chen, Z. Liu, and F. Zhang, “Memory-efficient and stabilizing management system and parallel methods for relion using CUDA and MPI,” in Proc. Int. Symp. Bioinf. Res. Appl., Springer, 2018, pp. 205–216.

[4]

J. Zhang, Z. Wang, Z. Liu, and F. Zhang, “Improve the resolution and parallel performance of the three-dimensional refine algorithm in RELION using CUDA and MPI,” IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 18, no. 2, pp. 583–595, Mar./Apr. 2021.

[5]

X. Li et al., “Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-EM,” Nature Methods, vol. 10, no. 6, pp. 584–590, 2013.

[6]

S. Q. Zheng, E. Palovcak, J.-P. Armache, K. A. Verba, Y. Cheng, and D. A. Agard, “MotionCor2: Anisotropic correction of beam-induced motion for improved cryo-electron microscopy,” Nature Methods, vol. 14, no. 4, pp. 331–332, 2017.

[7]

M. Madej et al., “Structural and functional insights into oligopeptide acquisition by the RagAB transporter from Porphyromonas gingivalis,” Nature Microbiol., vol. 5, no. 8, pp. 1016–1025, 2020.

[8]

J. Pan, H. Bai, and J. Tang, “Cascaded deep video deblurring using temporal sharpness prior,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3043–3051.

[9]

X. Wang, K. C. Chan, K. Yu, C. Dong, and C. Change Loy, “EDVR: Video restoration with enhanced deformable convolutional networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2019, pp. 1954–1963.

[10]

E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “FlowNet 2.0: Evolution of optical flow estimation with deep networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2462–2470.

[11]

D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8934–8943.

[12]

J. Dai et al., “Deformable convolutional networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 764–773.

[13]

X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets V2: More deformable, better results,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 9308–9316.

[14]

S. Nah, S. Son, and K. M. Lee, “Recurrent neural networks with intra-frame iterations for video deblurring,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 8102–8111.

[15]

Z. Zhong, Y. Gao, Y. Zheng, and B. Zheng, “Efficient spatio-temporal recurrent neural network for video deblurring,” in Proc. Eur. Conf. Comput. Vis., Springer, 2020, pp. 191–207.

[16]

Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 2472–2481.

[17]

Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 7, pp. 2480–2495, Jul. 2021.

[18]

J. Lehtinen et al., “Noise2Noise: Learning image restoration without clean data,” 2018,.

[19]

D. Tegunov and P. Cramer, “Real-time cryo-electron microscopy data preprocessing with warp,” Nature Methods, vol. 16, no. 11, pp. 1146–1152, 2019.

[20]

T. Bepler, K. Kelley, A. J. Noble, and B. Berger, “Topaz-Denoise: General deep denoising models for cryoEM and cryoET,” Nature Commun., vol. 11, no. 1, pp. 1–12, 2020.

[21]

W. Zheng and Y. Yang, “An unsupervised iterative model for single-particle cryo-EM image denoising based on siamese neural network,” in Proc. IEEE Int. Conf. Bioinf. Biomed., 2020, pp. 2390–2394.

[22]

J. Caballero et al., “Real-time video super-resolution with spatio-temporal networks and motion compensation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4778–4787.

[23]

T. H. Kim, M. S. Sajjadi, M. Hirsch, and B. Scholkopf, “Spatio-temporal transformer network for video restoration,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 106–122.

[24]

Y. Tian, Y. Zhang, Y. Fu, and C. Xu, “TDAN: Temporally-deformable alignment network for video super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3360–3369.

[25]

M. Tassano, J. Delon, and T. Veit, “FastDVDNet: Towards real-time deep video denoising without flow estimation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1354–1363.

[26]

T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” Int. J. Comput. Vis., vol. 127, no. 8, pp. 1106–1125, 2019.

[27]

S. Lin et al., “Learning event-driven video deblurring and interpolation,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 695–710.

[28]

W. Shen, W. Bao, G. Zhai, L. Chen, X. Min, and Z. Gao, “Blurry video frame interpolation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 5114–5123.

[29]

S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang, “Deep video deblurring for hand-held cameras,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1279–1288.

[30]

T. Hyun Kim, K. Mu Lee, B. Scholkopf, and M. Hirsch, “Online video deblurring via dynamic temporal blending network,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 4038–4047.

[31]

F. Förster, S. Pruggnaller, A. Seybert, and A. S. Frangakis, “Classification of cryo-electron sub-tomograms using constrained correlation,” J. Struct. Biol., vol. 161, no. 3, pp. 276–286, 2008.

[32]

R. Yao, J. Qian, and Q. Huang, “Deep-learning with synthetic data enables automated picking of cryo-EM particle images of biological macromolecules,” Bioinformatics, vol. 36, no. 4, pp. 1252–1259, 2020.

[33]

M. Xu, M. Beck, and F. Alber, “Template-free detection of macromolecular complexes in cryo electron tomograms,” Bioinformatics, vol. 27, no. 13, pp. i69–i76, 2011.

[34]

A. Singer and Y. Shkolnisky, “Three-dimensional structure determination from common lines in cryo-EM by eigenvectors and semidefinite programming,” SIAM J. Imag. Sci., vol. 4, no. 2, pp. 543–572, 2011.

[35]

E. Palovcak, D. Asarnow, Z. Yu, M. Campbell, and Y. Cheng, “Generating contrast in cryo-EM images with convolutional networks,” Improved Methods Single-Particle Cryogenic Electron Microsc., 2019, Art. no.

[36]

W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1874–1883.

[37]

L. Jiang, B. Dai, W. Wu, and C. C. Loy, “Focal frequency loss for image reconstruction and synthesis,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 13919–13929.

[38]

H. Xu et al., “Structural basis of nav1. 7 inhibition by a gating-modifier spider toxin,” Cell, vol. 176, no. 4, pp. 702–715, 2019.

[39]

S. He et al., “The structural basis of rubisco phase separation in the pyrenoid,” Nature Plants, vol. 6, no. 12, pp. 1480–1490, 2020.

[40]

N. Adachi et al., “Sub-3 Åresolution structure of 110 kDa nitrite reductase determined by 200 kv cryogenic electron microscopy,” bioRxiv, pp. 2020–7, 2020.

[41]

Y. Niu, H. Suzuki, C. J. Hosford, T. Walz, and J. S. Chappie, “Structural asymmetry governs the assembly and GTPase activity of McrBC restriction complexes,” Nature Commun., vol. 11, no. 1, pp. 1–17, 2020.

[42]

T. Zhou et al., “Cryo-EM structures of SARS-CoV-2 spike without and with ACE2 reveal a pH-dependent switch to mediate endosomal positioning of receptor-binding domains,” Cell Host Microbe, vol. 28, no. 6, pp. 867–879, 2020.

[43]

D. N. Grba et al., “Cryo-electron microscopy reveals how acetogenins inhibit mitochondrial respiratory complex I,” J. Biol. Chem., vol. 298, no. 3, 2022.

[44]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080–2095, 2007.

[45]

B. Ji and A. Yao, “Multi-scale memory-based video deblurring,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 1919–1928.

Index Terms

  1. Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement

    Index terms have been assigned to the content through auto-classification.

    Recommendations

    • Cryo-EM Study of ATP Induced Conformational Heterogeneity in Archaeal Chaperonin mmCpn

      Read More

    • Cryo-EM Structures and Biological Study of Pathogenic Alpha-Synuclein Fibrils

      Read More

    • Spatio-Temporal Convolutional Neural Network for Frame Rate Up-Conversion

      ISMSI '19: Proceedings of the 2019 3rd International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence

      The visual quality of the video is improved by realizing higher resolution and higher frame rate. In order to realize higher frame rate, we propose new frame rate up-conversion method using spatio-temporal convolutional neural network. In recent years, ...

      Read More

    Comments

    Information & Contributors

    Information

    Published In

    Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement (5)

    IEEE/ACM Transactions on Computational Biology and Bioinformatics Volume 21, Issue 5

    Sept.-Oct. 2024

    460 pages

    Issue’s Table of Contents

    1545-5963 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

    Publisher

    IEEE Computer Society Press

    Washington, DC, United States

    Publication History

    Published: 25 March 2024

    Published inTCBBVolume 21, Issue 5

    Qualifiers

    • Research-article

    Contributors

    Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement (6)

    Other Metrics

    View Article Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Total Citations

    • 1

      Total Downloads

    • Downloads (Last 12 months)1
    • Downloads (Last 6 weeks)1

    Reflects downloads up to 24 Oct 2024

    Other Metrics

    View Author Metrics

    Citations

    View Options

    Get Access

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    Get this Article

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement (2024)
    Top Articles
    Latest Posts
    Recommended Articles
    Article information

    Author: Foster Heidenreich CPA

    Last Updated:

    Views: 5612

    Rating: 4.6 / 5 (56 voted)

    Reviews: 87% of readers found this page helpful

    Author information

    Name: Foster Heidenreich CPA

    Birthday: 1995-01-14

    Address: 55021 Usha Garden, North Larisa, DE 19209

    Phone: +6812240846623

    Job: Corporate Healthcare Strategist

    Hobby: Singing, Listening to music, Rafting, LARPing, Gardening, Quilting, Rappelling

    Introduction: My name is Foster Heidenreich CPA, I am a delightful, quaint, glorious, quaint, faithful, enchanting, fine person who loves writing and wants to share my knowledge and understanding with you.