research-article
Authors: Xiaoya Chong, Howard Leung, Qing Li, Jianhua Yao, Niyun Zhou
IEEE/ACM Transactions on Computational Biology and Bioinformatics, Volume 21, Issue 5
Pages 1299 - 1310
Published: 25 March 2024 Publication History
Metrics
Total Citations0Total Downloads1Last 12 Months1
Last 6 weeks1
New Citation Alert added!
This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Manage my Alerts
New Citation Alert!
Please log in to your account
Get Access
- Get Access
- References
- Media
- Tables
- Share
Abstract
Cryo-EM in single particle analysis is known to have low SNR and requires to utilize several frames of the same particle sample to restore one high-quality image for visualizing that particle. However, the low SNR of cryo-EM movie and motion caused by beam striking make the task very challenging. Video enhancement algorithms in computer vision shed new light on tackling such tasks by utilizing deep neural networks. However, they are designed for natural images with high SNR. Meanwhile, the lack of ground truth in cryo-EM movie seems to be one major limiting factor of the progress. Hence, we present a synthetic cryo-EM movie generation pipeline, which can produce realistic diverse cryo-EM movie datasets with low-SNR movie frames and multiple ground truth values. Then we propose a deep spatio-temporal network (DST-Net) for cryo-EM movie frame enhancement trained on our synthetic data. Spatial and temporal features are first extracted from each frame. Spatio-temporal fusion and high-resolution re-constructor are designed to obtain the enhanced output. For evaluation, we train our model on seven synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results show that DST-Net can achieve better enhancement performance both quantitatively and qualitatively compared with others.
References
[1]
E. Callaway, “The revolution will not be crystallized: A new method sweeps through structural biology,” Nature News, vol. 525, no. 7568, 2015, Art. no.
[2]
W. Kühlbrandt, “The resolution revolution,” Science, vol. 343, no. 6178, pp. 1443–1444, 2014.
[3]
J. Zhang, Z. Wang, Y. Chen, Z. Liu, and F. Zhang, “Memory-efficient and stabilizing management system and parallel methods for relion using CUDA and MPI,” in Proc. Int. Symp. Bioinf. Res. Appl., Springer, 2018, pp. 205–216.
[4]
J. Zhang, Z. Wang, Z. Liu, and F. Zhang, “Improve the resolution and parallel performance of the three-dimensional refine algorithm in RELION using CUDA and MPI,” IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 18, no. 2, pp. 583–595, Mar./Apr. 2021.
[5]
X. Li et al., “Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-EM,” Nature Methods, vol. 10, no. 6, pp. 584–590, 2013.
[6]
S. Q. Zheng, E. Palovcak, J.-P. Armache, K. A. Verba, Y. Cheng, and D. A. Agard, “MotionCor2: Anisotropic correction of beam-induced motion for improved cryo-electron microscopy,” Nature Methods, vol. 14, no. 4, pp. 331–332, 2017.
[7]
M. Madej et al., “Structural and functional insights into oligopeptide acquisition by the RagAB transporter from Porphyromonas gingivalis,” Nature Microbiol., vol. 5, no. 8, pp. 1016–1025, 2020.
[8]
J. Pan, H. Bai, and J. Tang, “Cascaded deep video deblurring using temporal sharpness prior,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3043–3051.
[9]
X. Wang, K. C. Chan, K. Yu, C. Dong, and C. Change Loy, “EDVR: Video restoration with enhanced deformable convolutional networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2019, pp. 1954–1963.
[10]
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “FlowNet 2.0: Evolution of optical flow estimation with deep networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2462–2470.
[11]
D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8934–8943.
[12]
J. Dai et al., “Deformable convolutional networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 764–773.
[13]
X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets V2: More deformable, better results,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 9308–9316.
[14]
S. Nah, S. Son, and K. M. Lee, “Recurrent neural networks with intra-frame iterations for video deblurring,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 8102–8111.
[15]
Z. Zhong, Y. Gao, Y. Zheng, and B. Zheng, “Efficient spatio-temporal recurrent neural network for video deblurring,” in Proc. Eur. Conf. Comput. Vis., Springer, 2020, pp. 191–207.
[16]
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 2472–2481.
[17]
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 7, pp. 2480–2495, Jul. 2021.
[18]
J. Lehtinen et al., “Noise2Noise: Learning image restoration without clean data,” 2018,.
[19]
D. Tegunov and P. Cramer, “Real-time cryo-electron microscopy data preprocessing with warp,” Nature Methods, vol. 16, no. 11, pp. 1146–1152, 2019.
[20]
T. Bepler, K. Kelley, A. J. Noble, and B. Berger, “Topaz-Denoise: General deep denoising models for cryoEM and cryoET,” Nature Commun., vol. 11, no. 1, pp. 1–12, 2020.
[21]
W. Zheng and Y. Yang, “An unsupervised iterative model for single-particle cryo-EM image denoising based on siamese neural network,” in Proc. IEEE Int. Conf. Bioinf. Biomed., 2020, pp. 2390–2394.
[22]
J. Caballero et al., “Real-time video super-resolution with spatio-temporal networks and motion compensation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4778–4787.
[23]
T. H. Kim, M. S. Sajjadi, M. Hirsch, and B. Scholkopf, “Spatio-temporal transformer network for video restoration,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 106–122.
[24]
Y. Tian, Y. Zhang, Y. Fu, and C. Xu, “TDAN: Temporally-deformable alignment network for video super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3360–3369.
[25]
M. Tassano, J. Delon, and T. Veit, “FastDVDNet: Towards real-time deep video denoising without flow estimation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1354–1363.
[26]
T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” Int. J. Comput. Vis., vol. 127, no. 8, pp. 1106–1125, 2019.
[27]
S. Lin et al., “Learning event-driven video deblurring and interpolation,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 695–710.
[28]
W. Shen, W. Bao, G. Zhai, L. Chen, X. Min, and Z. Gao, “Blurry video frame interpolation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 5114–5123.
[29]
S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang, “Deep video deblurring for hand-held cameras,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1279–1288.
[30]
T. Hyun Kim, K. Mu Lee, B. Scholkopf, and M. Hirsch, “Online video deblurring via dynamic temporal blending network,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 4038–4047.
[31]
F. Förster, S. Pruggnaller, A. Seybert, and A. S. Frangakis, “Classification of cryo-electron sub-tomograms using constrained correlation,” J. Struct. Biol., vol. 161, no. 3, pp. 276–286, 2008.
[32]
R. Yao, J. Qian, and Q. Huang, “Deep-learning with synthetic data enables automated picking of cryo-EM particle images of biological macromolecules,” Bioinformatics, vol. 36, no. 4, pp. 1252–1259, 2020.
[33]
M. Xu, M. Beck, and F. Alber, “Template-free detection of macromolecular complexes in cryo electron tomograms,” Bioinformatics, vol. 27, no. 13, pp. i69–i76, 2011.
[34]
A. Singer and Y. Shkolnisky, “Three-dimensional structure determination from common lines in cryo-EM by eigenvectors and semidefinite programming,” SIAM J. Imag. Sci., vol. 4, no. 2, pp. 543–572, 2011.
[35]
E. Palovcak, D. Asarnow, Z. Yu, M. Campbell, and Y. Cheng, “Generating contrast in cryo-EM images with convolutional networks,” Improved Methods Single-Particle Cryogenic Electron Microsc., 2019, Art. no.
[36]
W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1874–1883.
[37]
L. Jiang, B. Dai, W. Wu, and C. C. Loy, “Focal frequency loss for image reconstruction and synthesis,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 13919–13929.
[38]
H. Xu et al., “Structural basis of nav1. 7 inhibition by a gating-modifier spider toxin,” Cell, vol. 176, no. 4, pp. 702–715, 2019.
[39]
S. He et al., “The structural basis of rubisco phase separation in the pyrenoid,” Nature Plants, vol. 6, no. 12, pp. 1480–1490, 2020.
[40]
N. Adachi et al., “Sub-3 Åresolution structure of 110 kDa nitrite reductase determined by 200 kv cryogenic electron microscopy,” bioRxiv, pp. 2020–7, 2020.
[41]
Y. Niu, H. Suzuki, C. J. Hosford, T. Walz, and J. S. Chappie, “Structural asymmetry governs the assembly and GTPase activity of McrBC restriction complexes,” Nature Commun., vol. 11, no. 1, pp. 1–17, 2020.
[42]
T. Zhou et al., “Cryo-EM structures of SARS-CoV-2 spike without and with ACE2 reveal a pH-dependent switch to mediate endosomal positioning of receptor-binding domains,” Cell Host Microbe, vol. 28, no. 6, pp. 867–879, 2020.
[43]
D. N. Grba et al., “Cryo-electron microscopy reveals how acetogenins inhibit mitochondrial respiratory complex I,” J. Biol. Chem., vol. 298, no. 3, 2022.
[44]
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080–2095, 2007.
[45]
B. Ji and A. Yao, “Multi-scale memory-based video deblurring,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 1919–1928.
Index Terms
Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement
Index terms have been assigned to the content through auto-classification.
Recommendations
- Cryo-EM Study of ATP Induced Conformational Heterogeneity in Archaeal Chaperonin mmCpn
Read More
- Cryo-EM Structures and Biological Study of Pathogenic Alpha-Synuclein Fibrils
Read More
- Spatio-Temporal Convolutional Neural Network for Frame Rate Up-Conversion
ISMSI '19: Proceedings of the 2019 3rd International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence
The visual quality of the video is improved by realizing higher resolution and higher frame rate. In order to realize higher frame rate, we propose new frame rate up-conversion method using spatio-temporal convolutional neural network. In recent years, ...
Read More
Comments
Information & Contributors
Information
Published In
IEEE/ACM Transactions on Computational Biology and Bioinformatics Volume 21, Issue 5
Sept.-Oct. 2024
460 pages
Issue’s Table of Contents
1545-5963 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Computer Society Press
Washington, DC, United States
Publication History
Published: 25 March 2024
Published inTCBBVolume 21, Issue 5
Qualifiers
- Research-article
Contributors
Other Metrics
View Article Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
Total Citations
1
Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)1
Reflects downloads up to 24 Oct 2024
Other Metrics
View Author Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Full Access
Get this Article
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderMedia
Figures
Other
Tables