SA-AE for Any-to-Any Relighting

Zhongyun Hu; Xin Huang; Yaning Li; Qing Wang

doi:10.1007/978-3-030-67070-2_32

SA-AE for Any-to-Any Relighting

Zhongyun Hu
, Xin Huang
, Yaning Li
, Qing Wang

School of Computer Science

Northwestern Polytechnical University Xian

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

In this paper, we present a novel automatic model Self-Attention AutoEncoder (SA-AE) for generating a relit image from a source image to match the illumination setting of a guide image, which is called any-to-any relighting. In order to reduce the difficulty of learning, we adopt an implicit scene representation learned by the encoder to render the relit image using the decoder. Based on the learned scene representation, a lighting estimation network is designed as a classification task to predict the illumination settings from the guide images. Also, a lighting-to-feature network is well designed to recover the corresponding implicit scene representation from the illumination settings, which is the inverse process of the lighting estimation network. In addition, a self-attention mechanism is introduced in the autoencoder to focus on the re-rendering of the relighting-related regions in the source images. Extensive experiments on the VIDIT dataset show that the proposed approach achieved the 1st place in terms of MPS and the 1st place in terms of SSIM in the AIM 2020 Any-to-any Relighting Challenge.

Original language	English
Title of host publication	Computer Vision – ECCV 2020 Workshops, Proceedings
Editors	Adrien Bartoli, Andrea Fusiello
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	535-549
Number of pages	15
ISBN (Print)	9783030670696
DOIs	https://doi.org/10.1007/978-3-030-67070-2_32
State	Published - 2020
Event	Workshops held at the 16th European Conference on Computer Vision, ECCV 2020 - Glasgow, United Kingdom Duration: 23 Aug 2020 → 28 Aug 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12537 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Workshops held at the 16th European Conference on Computer Vision, ECCV 2020
Country/Territory	United Kingdom
City	Glasgow
Period	23/08/20 → 28/08/20

Keywords

Any-to-any relighting
Autoencoder
Deep learning
Lighting estimation
Self-attention mechanism

Access to Document

10.1007/978-3-030-67070-2_32

Cite this

Hu, Z., Huang, X., Li, Y., & Wang, Q. (2020). SA-AE for Any-to-Any Relighting. In A. Bartoli, & A. Fusiello (Eds.), Computer Vision – ECCV 2020 Workshops, Proceedings (pp. 535-549). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12537 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-67070-2_32

Hu, Zhongyun ; Huang, Xin ; Li, Yaning et al. / SA-AE for Any-to-Any Relighting. Computer Vision – ECCV 2020 Workshops, Proceedings. editor / Adrien Bartoli ; Andrea Fusiello. Springer Science and Business Media Deutschland GmbH, 2020. pp. 535-549 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{5689badfddf54e93bb1836c2e22b450f,

title = "SA-AE for Any-to-Any Relighting",

abstract = "In this paper, we present a novel automatic model Self-Attention AutoEncoder (SA-AE) for generating a relit image from a source image to match the illumination setting of a guide image, which is called any-to-any relighting. In order to reduce the difficulty of learning, we adopt an implicit scene representation learned by the encoder to render the relit image using the decoder. Based on the learned scene representation, a lighting estimation network is designed as a classification task to predict the illumination settings from the guide images. Also, a lighting-to-feature network is well designed to recover the corresponding implicit scene representation from the illumination settings, which is the inverse process of the lighting estimation network. In addition, a self-attention mechanism is introduced in the autoencoder to focus on the re-rendering of the relighting-related regions in the source images. Extensive experiments on the VIDIT dataset show that the proposed approach achieved the 1st place in terms of MPS and the 1st place in terms of SSIM in the AIM 2020 Any-to-any Relighting Challenge.",

keywords = "Any-to-any relighting, Autoencoder, Deep learning, Lighting estimation, Self-attention mechanism",

author = "Zhongyun Hu and Xin Huang and Yaning Li and Qing Wang",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; Workshops held at the 16th European Conference on Computer Vision, ECCV 2020 ; Conference date: 23-08-2020 Through 28-08-2020",

year = "2020",

doi = "10.1007/978-3-030-67070-2\_32",

language = "英语",

isbn = "9783030670696",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "535--549",

editor = "Adrien Bartoli and Andrea Fusiello",

booktitle = "Computer Vision – ECCV 2020 Workshops, Proceedings",

}

Hu, Z, Huang, X, Li, Y & Wang, Q 2020, SA-AE for Any-to-Any Relighting. in A Bartoli & A Fusiello (eds), Computer Vision – ECCV 2020 Workshops, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12537 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 535-549, Workshops held at the 16th European Conference on Computer Vision, ECCV 2020, Glasgow, United Kingdom, 23/08/20. https://doi.org/10.1007/978-3-030-67070-2_32

SA-AE for Any-to-Any Relighting. / Hu, Zhongyun; Huang, Xin; Li, Yaning et al.
Computer Vision – ECCV 2020 Workshops, Proceedings. ed. / Adrien Bartoli; Andrea Fusiello. Springer Science and Business Media Deutschland GmbH, 2020. p. 535-549 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12537 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SA-AE for Any-to-Any Relighting

AU - Hu, Zhongyun

AU - Huang, Xin

AU - Li, Yaning

AU - Wang, Qing

PY - 2020

Y1 - 2020

N2 - In this paper, we present a novel automatic model Self-Attention AutoEncoder (SA-AE) for generating a relit image from a source image to match the illumination setting of a guide image, which is called any-to-any relighting. In order to reduce the difficulty of learning, we adopt an implicit scene representation learned by the encoder to render the relit image using the decoder. Based on the learned scene representation, a lighting estimation network is designed as a classification task to predict the illumination settings from the guide images. Also, a lighting-to-feature network is well designed to recover the corresponding implicit scene representation from the illumination settings, which is the inverse process of the lighting estimation network. In addition, a self-attention mechanism is introduced in the autoencoder to focus on the re-rendering of the relighting-related regions in the source images. Extensive experiments on the VIDIT dataset show that the proposed approach achieved the 1st place in terms of MPS and the 1st place in terms of SSIM in the AIM 2020 Any-to-any Relighting Challenge.

AB - In this paper, we present a novel automatic model Self-Attention AutoEncoder (SA-AE) for generating a relit image from a source image to match the illumination setting of a guide image, which is called any-to-any relighting. In order to reduce the difficulty of learning, we adopt an implicit scene representation learned by the encoder to render the relit image using the decoder. Based on the learned scene representation, a lighting estimation network is designed as a classification task to predict the illumination settings from the guide images. Also, a lighting-to-feature network is well designed to recover the corresponding implicit scene representation from the illumination settings, which is the inverse process of the lighting estimation network. In addition, a self-attention mechanism is introduced in the autoencoder to focus on the re-rendering of the relighting-related regions in the source images. Extensive experiments on the VIDIT dataset show that the proposed approach achieved the 1st place in terms of MPS and the 1st place in terms of SSIM in the AIM 2020 Any-to-any Relighting Challenge.

KW - Any-to-any relighting

KW - Autoencoder

KW - Deep learning

KW - Lighting estimation

KW - Self-attention mechanism

UR - https://www.scopus.com/pages/publications/85101591298

U2 - 10.1007/978-3-030-67070-2_32

DO - 10.1007/978-3-030-67070-2_32

M3 - 会议稿件

AN - SCOPUS:85101591298

SN - 9783030670696

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 535

EP - 549

BT - Computer Vision – ECCV 2020 Workshops, Proceedings

A2 - Bartoli, Adrien

A2 - Fusiello, Andrea

PB - Springer Science and Business Media Deutschland GmbH

T2 - Workshops held at the 16th European Conference on Computer Vision, ECCV 2020

Y2 - 23 August 2020 through 28 August 2020

ER -

Hu Z, Huang X, Li Y, Wang Q. SA-AE for Any-to-Any Relighting. In Bartoli A, Fusiello A, editors, Computer Vision – ECCV 2020 Workshops, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 535-549. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-67070-2_32

SA-AE for Any-to-Any Relighting

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this