Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion

doi:10.1109/TPAMI.2020.3026039

	Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion
	Jing Jin 1; Junhui Hou 1; Jie Chen 2; Huanqiang Zeng 3; Sam Kwong 1; Jingyi Yu4
	2022-04-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
EISSN	1939-3539
卷号	44 期号:4
发表状态	已发表
DOI	10.1109/TPAMI.2020.3026039
摘要	A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods. In addition, we illustrate the benefits and advantages of the proposed approach when applied in various LF-based applications, including image-based rendering and depth estimation enhancement. The code is available at https://github.com/jingjin25/LFASR-FS-GAF.
关键词	Image reconstruction Geometry Learning systems Image resolution Rendering (computer graphics) Estimation Cameras Light field deep learning depth estimation super resolution compression image-based rendering
URL	查看原文
收录类别	SCI ; EI ; SCIE
语种	英语
资助项目	Hong Kong RGC["9048123 (CityU 21211518)","9042820 (CityU 11219019)"] ; Natural Science Foundation of China[61871342,61871434] ; Basic Research General Program of Shenzhen Municipality[JCYJ20190808183003968]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000764815300014
出版者	IEEE COMPUTER SOC
来源库	IEEE
引用统计
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/165023
专题	信息科学与技术学院信息科学与技术学院_PI研究组_虞晶怡组
作者单位	1.Department of Computer Science, City University of Hong Kong, Hong Kong 2.Department of Computer Science, Hong Kong Baptist University, Hong Kong 3.College of Engineering, Huaqiao University, Quanzhou, China 4.School of Information Science and Technology, ShanghaiTech University, Shanghai, China
推荐引用方式 GB/T 7714	Jing Jin,Junhui Hou,Jie Chen,et al. Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2022,44(4).
APA	Jing Jin,Junhui Hou,Jie Chen,Huanqiang Zeng,Sam Kwong,&Jingyi Yu.(2022).Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,44(4).
MLA	Jing Jin,et al."Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 44.4(2022).