NeRSP CVPR24 Neural 3D Toe Fausiaina mo Mea Fa'ata'ita'i Fa'atonuga Tusi Taiala

Tusitala: Yufei Han, Heng Guo, Koki Fukai, Hiroaki Santo, Boxin Shi, Fumio Okura, Zhanyu Ma, Yunpeng Jia
Fa'atasiga: Beijing University of Posts and Telecommunications, Osaka University, Peking University

Fa'amatalaga: O le NeRSP o lo'o ofoina atu le fa'aleleia atili o foliga toe fa'aleleia i'uga mo fa'afanua fa'atusa pe a fa'atusatusa i metotia o lo'o iai.

Fa'atonuga o le Fa'aaogaina o Mea

Folasaga
O le NeRSP oloa ua mamanuina mo le toe fausia 3D o mea e atagia ai le faʻaaogaina o ata polarized seasea. E manumalo i luitau e fesoʻotaʻi ma le sailia o le teleview Feso'ota'iga ma fa'ate'a ese foliga mai le susulu i lalo o feso'ota'iga fa'atapula'aina.

Galuega Faatatau
NeRSP e musuia e Neural Radiance Fields (NeRF) ma isi neural 3D toe fausia auala. E fa'ata'ita'i fa'atusa foliga o luga e ala i le saini mamao fanua (SDF) ma fa'aogaina le su'eina o si'osi'omaga eseese ma le fa'aliliuina o le voluma e fa'aleleia atili ai le lelei o le toe fausia.
Fa'ata'ita'iga Fa'atusa o Ata Polarimetric
O lo'o tu'ufa'atasia e le NeRSP se fa'ata'ita'iga fa'ata'ita'iga fa'atusa e maua mai ai fa'ailoga photometric ma fa'ailoga mo le toe fa'aleleia.

FAQ:

Q: O le a le advantago le fa'aaogaina o le NeRSP mo le toe fa'aleleia o le 3D?
A: O lo'o ofoina atu e le NeRSP ni fa'ai'uga toe fa'aleleia o foliga mo fa'amana'oga fa'atusa pe a fa'atusatusa i metotia o lo'o iai ona o lana faiga fou e fa'aogaina ai ata fa'asolo.

Q: E fetaui lelei le oloa NeRSP mo luga ole laiga?
A: E ui o le NeRSP o loʻo faʻatulagaina muamua mo mea e atagia ai, e mafai foi ona tuʻuina atu le faʻamaoniaina o foliga faʻatusatusa mo luga o luga ole mea e faʻamaonia ai le tutusa o le photometric i le isi itu. views.

NeRSP:
Neural 3D Toe Fausiaina mo Mea Faʻataʻitaʻi Faʻatasi ma Ata Polarized Sparse

Yufei Han1† Heng Guo1†∗ Koki Fukai2† Hiroaki Santo2 Boxin Shi3,4 Fumio Okura2 Zhanyu Ma1 Yunpeng Jia1

Beijing Iunivesite o Falemeli ma Fesootaiga

A'oga Fa'au'u o Fa'amatalaga Saienisi ma Tekinolosi, Iunivesite o Osaka
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University 4National Engineering Research Center of Visual Technology, School of Computer Science, Peking University
- {hanyufei, guoheng, mazhanyu}@bupt.edu.cn shiboxin@pku.edu.cn
- {santo.hiroaki, okura, fukai.koki}@ist.osaka-u.ac.jp xibei156@163.com.

Avanoa

Matou te tuʻuina atu le NeRSP, o le Neural 3D toe faʻaleleia auala mo faʻataʻitaʻiga luga ma ata Sparse Polarized. O le toe fa'aleleia o luga ole ata e matua'i lu'itau e pei ona iai mafaufauga fa'apitoa view-faalagolago ma faapea ona solia le multiview tutusa mo le teleview mea fa'ataoto. I le isi itu, e tau leai ni fa'aoga o ata, e fai ma fa'atonuga o le pu'eina, e masani ona mafua ai le le atoatoa pe fa'aletonu i'uga ona o le leai o se feso'ota'iga fetaui. O lenei pepa o lo'o fa'atasi fa'atasi ai lu'itau o mea fa'aoga seasea ma mea e fa'amana'o e ala i le fa'aogaina o ata fa'a polarized. Matou te maua fa'ailoga photometric ma geometric mai le fa'ata'ita'iga fa'ata'ita'iga o ata ma le teleview azimuth tutusa, lea e fa'atasi fa'atasi ai le fa'ata'ita'iga fa'ata'ita'iga ile fa'ata'ita'iga ole neural. Faʻavae i luga o faʻataʻitaʻiga i luga oa matou faʻamaumauga faʻapipiʻi ma moni, matou te ausia taunuʻuga o le toe faʻaleleia o luga ma naʻo le 6 views e fai ma fa'aoga.

Folasaga

Teleview O le toe faʻaleleia o le 3D o se faʻafitauli taua ile vaʻaiga komepiuta (CV) ma sa suʻesuʻeina tele mo le tele o tausaga [14]. Faʻatasi ai ma le alualu i luma o faʻaaliga faʻaalia i luga ole laiga [27, 28] ma neural radiance fields [22], talu ai nei le teleview Metotia toe fausia 3D [5, 33, 38, 41] ua tele le alualu i luma. E ui i le malosi o le toe faʻaleleia o foliga, o le tele o le teleview leo (MVS) auala e faalagolago tele i le sailia o fetusiaiga i le va views, lea e sili ona lu'itau mo fa'amana'oga o luga ma fa'aoga seasea views.

Mo mea fa'afoliga, o le view-faalagolago i luga ole laiga-

Tusa sao. ∗ Tusitala tutusa.
Itulau o galuega: https://yu-fei-han.github.io/NeRSP-project/.

Ata 1. Fa'afouga toe fa'afo'isia o se mea fa'afofoga mai ata e 6 fa'ase'e fa'apola na pu'eina (laina pito i luga). E maua e le matou NeRSP se fa'ai'uga toe fa'aleleia o foliga e sili atu pe a fa'atusatusa i metotia o lo'o i ai e fa'atatau i mea fa'apipi'i (S-VolSDF [35]) po'o le fa'afoliga fa'afofoga (PANDORA [9]).

Pearance talepe le manatu tutusa photometric o loʻo faʻaaogaina i le faʻatusatusaga o fesoʻotaʻiga i le MVS. Ina ia foia lenei faʻafitauli, o auala fou e toe faʻaleleia ai neural 3D (faʻataʻitaʻiga, Ref-NeuS [13], NeRO [19], ma PANDORA [9]) faʻataʻitaʻiina manino le faʻataʻitaʻiga ma faʻatusatusa i le taimi e tasi le faʻataʻitaʻiga ma faʻafanua o le siosiomaga e ala i le faʻaliliuina faʻafefe. Ae ui i lea, maua ata mafiafia i lalo eseese views e mana'omia e fa'atino ma le fa'amaoni isi mea e le iloa e ese mai foliga, e pei o le albedo, talatala, ma le fa'afanua o le si'osi'omaga.

Mai le tele o mea e fai views, e masani ona luitauina le sailia o le tele o meaview fetusiaiga. Ae maise pe a fai ma sui view-faʻalagolago i reflectances, e faigata ona faʻamavae foliga mai le susulu i lalo o se numera faʻatapulaʻa o fesoʻotaʻiga, e oʻo atu ai i le faʻaogaina o foliga-faʻamalamalamaga [40]. Metotia fou toe fausia neural 3D mo seasea views (faʻataʻitaʻiga, S-VolSDF [35] ma SparseNeuS [20]) e manaʻomia le faʻavasegaina e faʻaaoga ai le faʻaogaina o le photometric, lea e mafai ona solia mo luga ole ata.

Ina ia foia faʻafitauli uma e lua, matou te fautua atu e faʻaaoga ata polarized seasea nai lo mea faʻaoga RGB. Aemaise lava, matou te tuʻuina atu le NeRSP, o le Neural 3D toe faʻaleleia auala e toe faʻaleleia ai foliga o Faʻaaliga Faʻaalia mai ata Sparse Polarized. Matou te faʻaogaina le faʻaogaina o le polarization (AoP) e maua mai ata polarized, lea e faʻaalia saʻo ai le azimuth angle o foliga i luga e oʻo atu i le π ma le π / 2 ambiguities. O lenei fa'ailoga fa'ailoga ua iloa e mafai ai ona fa'aopoopoview toe fa'aleleia o foliga e tusa lava po'o le a le mea o lo'o fa'aalia i luga, ae o foliga fa'atatau e fa'atatau i le fa'ailoga fa'ata'ita'i e le manino [6] i lalo ifo view faatulagaga. I le isi itu, o le photometric cue mai le polarimetric image formation model [2] fesoasoani i le toe fausia o neural surface (eg, PANDORA [9]) e ala i le faʻaitiitia o le eseesega i le va o le toe tuʻuina atu ma le puʻeina o ata polarized. Peita'i, o le fa'atusa fa'atatau e fa'atatau i le fa'ailoga photometric e fa'aletonu fo'i i lalo ole mea fa'aoga seasea ona o le le manino o le fa'aaliga. E le pei o le auala o loʻo i ai nei polarimetric-based PANDORA [9] mafaufau i le photometric cue naʻo, o lo tatou NeRSP o loʻo faʻaalia ai le tuʻufaʻatasia o faʻataʻitaʻiga geometric ma photometric faʻapipiʻi lelei i lalo le avanoa fofo mo foliga o luga, faʻaalia e aoga i le toe faʻaleleia o luga o le eleele e faʻavae i luga o mea e faʻaogaina. , e pei ona vaaia i le Ata 1.

E ese mai i le NeRSP fuafuaina mo le toe fausia o le 3D, sa matou fausia foi le Real-world MultiView Fa'ata'otoga ata fa'a polarized o lo'o iai meafaitino e 6 fa'atasi ai ma fa'ameamea 3D fa'aeleele (GT), fa'aigoa RMVP3D. E ese mai fa'amaumauga o lo'o i ai e pei o le PANDORA dataset [9] o lo'o tu'uina atu ai na'o ata fa'apipi'i, o fa'aoga GT meshes ma luga ole tulaga masani mo ta'itasi. view fa'atagaina se su'esu'ega fa'afuainumera o le teleview toe fausia 3D polarized.

I se aotelega, matou te agai i luma multiview Toe fa'aleleia 3D e ala i le tu'uina atu

NeRSP, o le auala muamua o loʻo fuafua e faʻaoga le faʻamatalaga polarimetric mo le toe faʻaleleia o luga o le eleele i lalo ole laititi. views;

se au'ili'iliga fa'apitoa o le fa'ailoga photometric ma fa'ailoga e maua mai ata fa'a polarized; ma
RMVP3D, o le muamua tele-lalolagi teleview fa'asologa o ata fa'apipi'i fa'atasi ma foliga GT mo su'esu'ega fa'atele.

Galuega fa'atatau

Teleview Toe fa'aleleia 3D ua su'esu'eina tele mo le tele o tausaga. Neural Radiance Fields (NeRF) [3, 22, 40] ua ausia le manuia tele i tala fou. view synthesis i tausaga talu ai nei. Fa'aosoina e le NeRF, neural 3D reconstruction method [24] o loʻo faʻatulagaina, pe a faʻataʻitaʻiina foliga o le pito i luga e ala i se fanua mamao saini (SDF). Amata mai le DVR [24], o auala mulimuli e faʻaleleia atili ai le faʻaleleia o foliga e ala i le suʻeina o le siʻosiʻomaga eseese [37], faʻasologa o le voluma [26, 33, 38], poʻo faʻamatalaga faʻaleleia foliga [18, 34]. O nei metotia e mafai ona maua ai le fa'amaoniaina o foliga fa'atatau i luga ole lavelave lea e fa'amaonia ai le tutusa o ata-metric i le isi itu views.

Toe fa'aleleia mo mea e fa'amana'o e faigata ona o le fa'aogaina o le photometric e le aoga. Metotia oi ai nei [5, 41, 42] faʻataʻitaʻia manino le view-fa'alagolago i le fa'ata'ita'iga ma fa'a'ese'ese le foliga, fa'amanino-ese'ese fa'amalama, ma mea e fa'afoliga e pei ole albedo ma le talatala. Peita'i, e le'o fa'amalieina fua fa'atatau o suiga o lo'o ta'ua i luga ona o le fa'amavaega e matua leaga lava. NeRO [19] fa'atonu le fa'aogaina o le vaeluaga o le fa'ata'ita'iga o le fa'atulagaina o ata ma fa'aleleia atili le lelei o le toe fa'aleleia o foliga e aunoa ma le mana'omia o matapulepule. Ae ui i lea, o auala o loʻo i luga e masani ona manaʻomia le puʻeina o ata mafiafia e faʻamautinoa ai le toe faʻaleleia o foliga faʻapitoa mo luʻitau o luga o le ata.

Toe faʻaleleia ma seasea views e taua mo fa'ata'ita'iga fa'atino e mana'omia ai le pu'eina lelei. Ona o le le lava o fetusiaiga mai le faatapulaaina views, e le mafai ona foia le le mautonu foliga- susulu, e taitai atu ai i le pisapisao ma le le mautonu toe faaleleia. O metotia o loʻo i ai nei e faʻafeiloaʻi ai lenei faʻafitauli e ala i le faʻaopoopoina o faʻasalalauga e pei o le faʻaogaina o le geometry i luga [25], loloto loloto muamua [10, 32], poʻo le faʻatonutonuina o le faʻasologa o tulaga [36]. O nisi o metotia [7, 20, 39] e fa'avaeina ai le toe fa'aleleia o le 3D e pei o se fa'afitauli fa'apitoa o le 3D lea e fa'aogaina ai ata na mua'i a'oa'oina e avea ma fa'amuamua lautele. S-VolSDF [35] fa'aoga le fa'asologa masaniview auala fa'ataoto e pei o le amataga ma fa'avasegaina le fa'ata'ita'iina o neural rendering fa'atasi ai ma se voluma fa'aletonu. Ae ui i lea, o loʻo faigata pea mo metotia o loʻo iai nei e toe faʻaleleia saʻo ai luga o ata.

Toe faʻaleleia e faʻaaoga ai ata polarized na suʻesuʻeina mo le tasi-view faatulagaga [1, 2, 16, 23, 29] ma le teleview faatulagaga [6, 8, 9, 11, 12, 43]. E le pei o ata RGB, o le AoP mai ata polarized e maua ai faʻamatalaga tuusaʻo mo le masani masani. nofofua-view foliga mai polarization (SfP) techniques aoga mai lenei meatotino ma faʻatusatusa le faʻafanua masani i lalo ole malamalama mamao tasi [21, 29] poʻo le malamalama masani e le iloa [1, 16]. Teleview SfP metotia [8, 43] foia le π ma le π/2 ambiguities i le AoP e faavae i luga o le tele.view matauga. PANDORA [9] o le neural 3D reconstruction method muamua e faʻavae i luga o ata polarized, faʻaalia e aoga i le toe faʻaleleia o foliga ma le faʻamalamalamaina. MVAS [6] toe maua foliga o luga mai le teleview fa'afanua azimuth, e feso'ota'i vavalalata ma fa'afanua AoP na maua mai

ata polarized. Ae ui i lea, o nei metotia e le suʻesuʻeina le faʻaogaina o ata polarized mo le toe faʻaleleia o luga o le eleele i lalo ole ata faʻapitoa.

Fa'ata'ita'iga Fa'atusa o Ata Polarimetric

A'o le'i su'esu'e i le auala ua fuafuaina, matou te fa'ailoa muamua le fa'ata'ita'iga o le fa'ata'ita'iga o ata ma maua mai le fa'ailoga ata-metric ma le fa'ailoga geometric i la matou metotia. E pei ona fa'aalia i le Ata 2, o se ata pu'upu'u fa'apipi'i pu'e pu'e pu'e fa'amauina ata mata'ina i tulimanu eseese e fa, fa'atasi ai ma ona fa'ailoga pika {I0, I45, I90, I135}. O ata nei e fa o loʻo faʻaalia ai le tulaga faʻapipiʻiina o moli na maua, lea o loʻo faʻatusalia o se 4D Stokes vector s = [s0, s1, s2, s3] faʻatusatusa e pei o.

Matou te fa'apea e leai se moli fa'ata'amilomilo fa'ata'amilomilo fa'apea ona tu'u ai le s3 i le 0. E mafai ona fa'aoga le Vector Stokes e fa'atatau le tulimanu o le polarization (AoP), ie.

Fa'avae i luga ole AoP ma Stokes vector, tatou te maua fa'atusa fa'ailoga fa'atusa ma fa'aata.

Fa'ailoga fa'atusa
O le AoP ϕa, o le azimuth angle o luga e mafai ona ϕa + π/2 poʻo ϕa + π, e taʻua o le π ma le π/2 ambiguity e faʻatatau i luga pe faʻapitoa pe faʻasalalau le pule. I lenei vaega, matou te faʻaalia muamua le faʻailoga geometric na aumaia e le teleview fa'afanua azimuth ona fa'alautele lea i le tulaga o AoP.

Mulimuli i le MVAS [6], mo se vaaiga mata x, lona luga masani n ma le azimuth tulimanu ϕ i le tasi meapueata view mulimuli i le mafutaga e pei o

o le R = [r1, r2, r3]⊤ o le fa'asolo o le meapueata. E mafai ona matou toe faʻatulagaina le Eq. (3) ia maua le so'otaga fa'alanotonu i le va o le fa'afanua masani ma se ve'a fa'atatau t(ϕ) e pei ona fa'amatalaina i lalo,

O le π ambiguity i le va o le AoP ma le azimuth angle e mafai ona foia masani e pei o Eq. (4) e tu pe a tatou faaopoopo le ϕ i le π. O le π/2 ambiguity e mafai ona fa'atalanoaina e ala i le fa'aogaina o se pseudo-projected tangent vector tˆ(ϕ) e faapea

Afai e tasi le vaaiga mata'ina x e matauina e le f views, e mafai ona matou fa'aputuina Eq. (4) ma le Eq. (5) faʻavae i luga o suiga eseese k ma mataʻituina AoPs, e taʻitaʻia ai se laina laina

T(x)n(x) = 0. (6)

Matou te faʻaogaina lenei faiga faʻasologa o la matou faʻailoga faʻailoga mo le tele-view toe fausia 3D polarized.

Fa'ailoga Photometric
Fa'apea o le fa'amalamalamaga o le si'osi'omaga na tupu e le'o fa'asalaina, e mafai ona fa'atusalia le Stokes vector o le fa'atonuga malamalama fa'alavelave ω.

si(ω) = L(ω)[1, 0, 0, 0]⊤, (7)

o le L(ω) o lo'o fa'aalia ai le malosi o le malamalama. O le moli alu ese e pu'eina e le meapueata fa'apolokalame ua fa'asolo fa'atasi ona o le ata. O lenei faagasologa o loʻo faʻataʻitaʻiina e ala i le 4 × 4 Muller matrix H. I lalo o le faʻamalamalamaina o le siosiomaga, o le Stokes vector o loʻo i fafo e mafai ona faʻatulagaina e avea ma mea taua o le mea na tupu Stokes vector faʻateleina ma le Muller matrix, ie

o le v ma le Ω e fa'ailoa ai le view fa'atonuga ma le vaega tu'ufa'atasi. I le mulimuli ai i le fa'ata'ita'iga polarized BRDF (pBRDF) [2], e mafai ona fa'a'ese'esea le ve'a Stokes i vaega fa'asalalau ma fa'apitoa e fa'ata'ita'iina e ala i le Hd ma le Hs tutusa, ie.

A maeʻa le faʻaulu mai PANDORA [9], e mafai ona tatou faʻatulagaina atili le gaosiga Stokes vector pei

pe a Ld =fΩ ρL(ω)ω⊤n T+i T−i dω o loʻo faʻaalia o le susulu felafolafoaʻi e fesoʻotaʻi ma le masani i luga ole n, faʻasologa o le Fresnel transmission [2] T+i,o ma T−i,o, diffuse albedo ρ , ma le tulimanu azimuth o le mea na tupu malamalama ϕn. Ls = fΩ L(ω) DG 4n⊤v dω o loʻo faʻaalia ai le susulu faʻapitoa e fesoʻotaʻi ma Fresnel reflect coefficients [2] R+ ma R−, le mea na tupu azimuth angle ϕh wrt le afa vector h = ω+v∥ω+v∥22, ma o le tufatufaina masani ma le faʻafefe o le D ma le G i le faʻataʻitaʻiga Microfacet [31].

Fa'amolemole siaki mea fa'aopoopo mo nisi fa'amatalaga. Faʻavae i luga o le faʻataʻitaʻiga o ata faʻataʻitaʻiga o loʻo faʻaalia i le Eq. (10), matou te fausia le faʻailoga photometric.

Auala ua fuafuaina

O la matou NeRSP e fa'aalu le teleview ata fa'a polarized, le ufimata fa'atusa tutusa o le mea fa'atatau, ma mea pu'e ata e fai ma fa'aoga ma fa'aalia foliga o le mea i luga ole mea o lo'o fa'aalia manino ile SDF. Matou te amata i le talanoaga i luga o faʻailoga photometric ma faʻailoga geometric i le fofoina o foliga toe faʻaleleia le le mautonu, sosoo ai ma le faʻatonuga i le fausaga o fesoʻotaʻiga ma le leiloa o galuega a la matou NeRSP.

Fa'aleaogaina ile toe fa'aleleia ole 3D
O le fa'ailoga geometric ma le fa'ailoga photometric e iai sona sao taua i le fa'aitiitia o le avanoa o fofo o foliga i luga ole laiga i lalo ole va'aiga. views. E pei ona faʻaalia i le Ata 3, matou te faʻaalia le fua faʻatatau i lalo ole 2 views ma fa'ailoga eseese. Tuuina atu naʻo ata RGB e fai ma faʻaoga (e fetaui ma le seti i NeRO [19] ma S-VolSDF [35]), o tuʻufaʻatasiga eseese o tulaga faʻaaliga, tulaga masani o luga, ma mea e atagia ai e pei ole albedo e mafai ona taʻitaʻia ai ata tutusa, talu ai. e na'o le lua fua RGB mo pito 3D ta'itasi i luga ole ave o meapueata. Fa'atasi ai ma Stokes vectors na maua mai ata fa'apolokalame, o le photometric cue e maua ai le 6 fua mo mata 3D ta'itasi (Stokes vector e 3 elemene), fa'aitiitia ai le maualuga o sui masani e le fetaui i le fa'ata'ita'iga o ata fa'atusa.

I le isi itu, e fa'avae i luga o fa'afanua AoP1 mai ata fa'apipi'i, e mafai ona fa'avasega fa'apitoa le tulaga e o'o atu i le π fa'alilolilo mo va'aiga mata'i uma i luga o le ave o le meapueata. Peita'i, e le'o manino lava le su'eina o le tulaga o lo'o feso'ota'i ai le ave o le meapueata se'i vagana le tolu view ua saunia [6]. O le mea lea, i lalo o seasea views seti (faataitaiga, 2 views i le Ata 3), o le fuafuaina o le tulaga o le vaaiga e fa'avae i luga ole fa'ailoga geometric po'o le photometric e tumau pea le le mautonu.
O la matou metotia e tu'ufa'atasia ai nei fa'ailoga e lua e maua mai ata polarized. E pei ona va'aia i le pito i lalo taumatau o le Ata 3, o le tulaga sa'o o le fa'aaliga e tatau ona fa'ata'atia i luga o le va'aiga o kulupu masani fa'ailoga e maua mai i fa'ailoga photometric ma geometric. E pei ona masani ai luga i s eseeseamptulaga ta'ita'i vaaiga e tulaga ese fuafuaina e fa'ailoga geometric, e mafai ona faigofie ona tatou iloa pe o le mata o loʻo i luga o le faʻatasi ma le fesoasoani a se faʻailoga photometric. I lenei auala, matou te faʻaitiitia ai le avanoa o fofo o le toe faʻaleleia o luga ole ata.

NeRSP
Fa'asologa o feso'ota'iga E pei ona fa'aalia i le Ata 4, o lo'o fa'aogaina e le matou NeRSP se fausaga feso'ota'iga tutusa ma PANDORA [9] na maua mai i le Ref-NeRF [30]. Mo se ave malamalama e sau mai le meapueata ogatotonu o ma le itu v, tatou sampO se pito i luga o le ave ma le mamao o le malaga, o lona tulaga e fa'ailoa i le xi = o + tiv. I le maeʻa ai o le faʻaogaina o le voluma o loʻo faʻaaogaina i le NeRF [25], o le Stokes vector s(v) ua matauina e mafai ona tuʻufaʻatasia e le voluma opacity σi ma le Stokes vectors i le s.ampta'ita'i pito i luga o le ave, ie

o fea fa'ailoa le fa'aputuina o felauaiga o le asampta'ita'i mata.

Faʻaosofia e le neural 3D reconstruction method NeuS [33] talu ai nei, matou te maua le voluma opacity mai se fesoʻotaʻiga SDF ma faʻapipiʻi foi le tulaga masani mai le gradient o le SDF. E fa'atatauina so(xi, v) i le samptaʻitaʻia taʻiala, matou te mulimuli i le faʻataʻitaʻiga o ata faʻatusa i le Eq. (10). Aemaise lava, o le diffuse radiance Ld e fesoʻotaʻi ma le diffuse albedo ma le Fresnel transmission coefficients, lea e faʻalagolago i tulaga o vaaiga ae e fesuisuiaʻi i le view faasinomaga. O le mea lea, matou te fa'aogaina se feso'ota'iga susulu fa'asalalau e fa'afanua ai le Ld mai fa'aaliga o vaega ta'itasi. O le specular radiance Ls e fesootai ma le specular lobe fuafuaina e le view itu, tulaga masani o luga, ma le talatala o luga. O lea matou te faʻaogaina ai le RoughnessNet e vaʻai ai le gaogao o luga. Faatasi ai ma le meapueata view fa'atonuga ma va'aiga i luga o le tulaga masani, matou te fa'atusatusaina le susulu fa'apitoa Ls mulimuli i le fa'apipi'iina o le fa'asologa o le tulaga fa'atulagaina na fa'atūina e Ref-NeRF [30]. O le tu'ufa'atasia o le Ld ma le Ls, matou te toe fa'aleleia le va'aiga o Stokes e mulimuli i le Fa'atusa. (10).

Galuega gau
O le gau o le photometric ua faauigaina o le L1 mamao i le va o ˆs(v) ua matauina ma toe fausia Stokes vectors s(v), ie,

o le V o lo'o fa'aalia uma ave o mea pu'epu'e i totonu o ufimata mea i 'ese'ese views. Mo le gau geometric. matou te su'e muamua le 3D scene point x i luga o le meapueata ave v se'ia pa'i i luga ona su'e lea o tulaga 2D-pixel fuafuaina i tulaga eseese. views. O le gau geometric o loʻo faʻamatalaina e faʻatatau i le Eq. (6), o lona uiga,

o lo'o fa'ailoa mai ai e le X ia feso'ota'iga uma o ave i totonu o ufimata mea i 'ese'ese views. E ese mai i le photometric ma geometric gau, matou te faaopoopo ufimata gau e vaavaaia e ufimata mea ma le Eikonal regularization gau. O le leiloa matapulepule ua faauigaina e faapea

o fea o lo'o fa'atusalia le ufimata va'ai ile k-th meapueata, o lona fa'atauga o le ufimata GT o lo'o fa'ailogaina o le Mk. O le TLM o lo'o fa'atusalia ai le leiloa o le cross-entropy binary.

lea o le ni,k o le luga masani e maua mai le SDF network i le i-th sampta'ita'i i luga ole k-th meapueata ave. O la matou NeRSP o loʻo vaʻavaʻaia e le tuʻufaʻatasiga o tuʻutuʻuga o loʻo i luga, ie

o le λe, λm, ma le λp o coefficient ia mo tulaga gau tutusa.

RMVP3D Fa'amaumauga
Ina ia fa'avasegaina fa'atusatusa le metotia ua tu'uina mai, matou te pu'eina se Real-world Multiview Fa'ata'otoga ata fa'apolokalame fa'atasi ma fa'atutusa fa'amaufa'ailoga moni o le eleele. Ata 5 (agavale) o loʻo faʻaalia ai le matou faʻapipiʻiina o le puʻeina, lea e aofia ai se mea pueata polarimetric, FLIR BFS-U3-51S5PC-C, faʻapipiʻiina i se tioata 12 mm ma se laina faʻasolosolo. Matou te faʻaaogaina le OpenCV mo le faʻaogaina o faʻamatalaga mataʻutia ma maua ata lanu 1224 × 1024 faʻatasi ai ma tulimanu polarizer i 0, 45, 90, ma 135 tikeri. I le taimi o le pueina o faʻamaumauga, matou te tuʻuina mea faʻatatau i le ogatotonu o le nofoaafi ma puʻeina ata e 60 i le mea faitino e ala i le faʻaogaina ma le lima o le mea pueata. Matou te aoina mea e 4 e fai ma sini: DOG, FROG, LION, ma PALL, e pei ona faʻaalia i le Ata 5 (ogatotonu). Mo le su'esu'ega fa'afuainumera, matou te fa'aaogaina se masini su'e leisa Creaform HandySCAN BLACK ma le sa'o o le 0.01 mm e maua ai le mata o le mea moni. Ina ia fa'aoga le mata i le ata na pu'eina views, matou te fa'aogaina muamua le PANDORA [9] e fa'atatau i se fa'asino foliga e fa'aaoga uma ai avanoa views ona fa'aogaina lea o le mata'itusi i le mea fa'atatau e ala i le ICP algorithm [4]. E ese mai le eleele-upu moni foliga ma teleview ata, matou te pu'eina fo'i le fa'afanua o le si'osi'omaga e fa'aaoga ai le 360-tikeri mea pu'eata THETA Z1, fa'amanuiaina le tele o iloiloga i le fua fa'amalamalamaga mo galuega fa'aliliuga fa'afeagai neural.

Fa'ata'ita'iga

Matou te iloiloina le NeRSP ma ni faʻataʻitaʻiga se tolu: 1) faʻatusatusa i le tele o loʻo iaiview 3D auala toe fausia fa'afuainumera i luga o se fa'amaumauga tu'ufa'atasi; 2) su'esu'ega ablation i le sao o fa'ailoga fa'atekinolosi ma photometric 3) su'esu'ega agava'a ma fa'atusatusaga i fa'amaumauga o le lalolagi moni. Matou te tuʻuina atu foʻi le BRDF ma tala view e maua ai mea faaopoopo.

Fa'amaumauga & Fa'avae
Fa'amaumauga. Matou te saunia ni fa'amaumauga moni e lua o le lalolagi: o le PAN-DORA dataset [9] ma la matou RMVP3D fuafuaina, lea o le PANDORA dataset [9] e fa'aaogaina mo le iloiloga lelei ona o le mea moni o le eleele e le o saunia. Matou te saunia foi se faʻapipiʻi faʻapipiʻiview polarized image dataset SMVP3D ma Mitsuba rendering engine [15], lea e aofia ai 5 mea faitino ma vaʻavaʻa-varying ma atagia atagia, e pei ona vaaia i le Ata 6. O mea faitino e faʻamalamalamaina e faʻafanua siʻosiʻomaga2 ma puʻeina e 6 views tufatufa fa'afuase'i i mea faitino. E ese mai i le fa'aliliuina o ata fa'apolokalame, matou te fa'atauina atu fo'i fa'ameamea, fa'afanua masani a le GT, ma fa'afanua AoP mo mea ta'itasi.

Fa'avae. O la matou galuega e foia le teleview Toe fa'aleleia 3D mo mea fa'afofoga e fa'atatau i ata fa'a polarized seasea. O le mea lea, matou te filifilia auala faʻafouina 3D toe faʻaleleia e faʻatatau i luga ole ata NeRO [19] ma seasea. views S-VolSDF [35]. O auala e lua o loʻo i luga e faʻavae i luga o faʻaoga ata RGB. Mo le teleview leo fa'avae i ata fa'a polarized, matou te filifilia le PANDORA [9] ma le MVAS [6] e fai ma a matou laina fa'avae. NeRO [19] e le mana'omia ni matapulepule e fai ma fa'aoga. Mo se faʻatusatusaga talafeagai, matou te aveese le pito i tua i ata RGB faʻatasi ai ma matapulepule tutusa aʻo leʻi tuʻuina atu i le NeRO [19]. Ina ia faʻatusatusa auala eseese, matou te faʻaogaina le mamao o le Chamfer (CD) i le va o le faʻatusatusaina ma le GT meshes, ma le mean angular error (MAE) i le va o le fua faʻatatau ma le GT masani i luga ole laiga. views e fai ma a matou fua fa'atatau.

Fa'afouga toe fa'aleleia i luga o se fa'amaumauga tu'ufa'atasi
E pei ona fa'aalia i le Laulau 1, matou te aoteleina le sese o fua fa'atusa o metotia o lo'o iai ma matou i luga o SMVP3D. O la matou metotia e ausia ai le mamao pito sili ona laititi o Chamfer i luga o mea uma e 5. Faʻavae i luga o faʻatusatusaga o foliga vaaia o loʻo faʻaalia i le Ata 7, NeRO [19] ma le S-VolSDF [35] e le mafai ona toe faʻaleleia saʻo faʻamatalaga o luga e pei ona faʻaalia i le tapunia. views. O se tasi o mafuaʻaga o le faʻamavaeina o foliga ma ata faʻaalia mai ata faʻapitoa e faigata tele mo nei metotia e faʻavae i luga o faʻamatalaga RGB. O le MVAS [6] ma le PANDORA [9] o loʻo faʻamatalaina faʻailoga faʻataʻitaʻi ma photometric o ata polarized, eseese. Peita'i, e le'o fa'amalieina le fa'amalieina o foliga fa'afoliga fa'atusa ona o le fa'anenefu o fa'ailoga fa'ata'ita'i ma ata fa'ata i lalo ole va'aiga. views faatulagaga. E pei ona faamamafaina i le tapunia views, manuia mai fa'ailoga geometric ma photometric, o la matou metotia e fa'aitiitia ai le avanoa o fofo o foliga fa'atatau, e o'o atu ai i le toe fa'aleleia o foliga sili ona talafeagai pe a fa'atusatusa i foliga GT.

E ese mai i le iloiloga o le mata ua toe fausia, matou te su'eina fo'i le fua fa'atatau masani. E pei ona fa'aalia i le Laulau 2, matou te aoteleina le uiga o mea sese o le fa'atatau i luga ole laiga ile 6 views mai auala eseese. E o gatasi ma fa'ai'uga o iloiloga i le Laulau 1, e maua e NeRSP fa'aletonu pito sili ona la'ititi i le averesi. Na matou matauina foi o taunuʻuga mai NeRO [19], MVAS [6], ma PANDORA [9] o loʻo i ai le tele o mea sese i luga o mea faitino ma faʻamatalaga lelei, e pei o mea DAVID ma DRAGON. I le avea ai ma se example, MVAS [6] o loʻo i ai le mamao lona lua pito sili ona laʻititi o Chamfer o loʻo faʻaalia i le Laulau 1, ae o le mea sese o le angular e sili atu i le 20◦. O se tasi o mafua'aga ona o auala o lo'o i ai nei e maua mai ai foliga lamolemole i le va'aiga views faʻatulagaina, lea e le o toe faʻaleleia lelei ai faʻamatalaga i luga e pei o le paʻu o le DRAGON.

Fuafuaga 1. Fa'atusatusaga o foliga toe fa'aleleia i luga o fa'amaumauga fa'akomepiuta su'esu'eina ile mamao ole Chamfer (↓). O mea sese la'ititi ma lona lua la'ititi e fa'ailoga mata'utia ma vase. “N/A” o lo'o fa'ailoa mai ai le fa'ata'ita'iga lea e le mafai ai e se metotia fa'apitoa ona tu'uina mai fa'ai'uga fa'atusa talafeagai.

Su'esu'ega tu'ufua
I lenei vaega, matou te faia se suʻesuʻega ablation e suʻe ai le aoga o faʻailoga geometric ma photometric. Avea le mea DRAGON e fai ma example, matou te fa'atinoina la matou metotia fa'atasi ma le leai o le photometric gau Lp ma le geometric gau Lg. E pei ona fa'aalia i le Ata 8, matou te fa'avasega le foliga ma fua fa'atatau masani i luga e ala i le fa'agataina o tu'utu'uga leiloa eseese. A aunoa ma le leiloa photometric, foliga le mautonu ona o le seasea views tupu. E pei ona faʻaalia mai le tapunia views, o le foliga latalata i le vae vae o loʻo i ai se mea faʻapipiʻi, aua e naʻo le lua e vaʻaia views mo lenei itulagi, ua le mafai ona faia se fofo tulaga ese mo foliga na o le faavae i luga o faafanua AoP [6]. A aunoa ma le gau geometric, tatou te maua foi i'uga fa'aletonu foliga ona e le lava le mata'ituina o ata e fa'apitoa ai le fa'aleagaina o foliga, fa'aaliga, ma fa'amalamalamaga. E ala i le tu'ufa'atasia o le photometric ma le geo-metric gau, o le matou NeRSP e fa'aitiitia ai le le mautonu o foliga toe ufiufi ma o le fa'atusatusaina o foliga e latalata i le GT, e pei ona fa'ailoa mai i le tapuni tapuni. views.

Ata 8. Su'esu'ega fa'asa'o ile 'ese'ese tu'uga gau. O laina pito i luga ma lalo e faʻaalia ai le faʻatusatusaina o foliga ma foliga masani, faʻatasi ai ma le mamao o le Chamfer ma le mea sese faʻailoga faʻailoga i le pito i luga o faʻataʻitaʻiga taʻitasi.

Toe faʻaleleia i luga o faʻamatalaga moni
E ese mai i faʻataʻitaʻiga faʻataʻitaʻiga o loʻo faʻaalia i le vaega muamua, matou te iloiloina foi la matou metotia i luga o faʻamaumauga o le lalolagi PANDORA dataset [9] ma le RMVP3D e faʻataʻitaʻi ai lona faʻaogaina i faʻataʻitaʻiga o le toe fausia o 3D.

Su'esu'ega fa'apitoa i fa'amaumauga PANDORA [9]. E pei ona faʻaalia i le Ata 9, matou te tuʻuina atu iloiloga faʻapitoa o le PAN-DORA dataset [9]. Pe a faatusatusa i foliga vaaia o ata ma taunuuga fuafuaina mai le S-VolSDF [35] ma le NeRO [19], o le foliga e le o atoatoa ona vavae ese mai le atagia, e oo atu ai i foliga faʻafefeteina e fesoʻotaʻi vavalalata ma le faʻasologa o ata. O le MVAS [6] ma le PANDORA [9] o loʻo i ai faʻataʻitaʻiga faʻalelei o foliga poʻo mea faʻapipiʻi foliga, ona o le faʻaogaina o faʻailoga faʻataʻitaʻi poʻo photometric i lalo o le faʻaogaina o le puʻeina. O a matou fa'ai'uga fa'atatau o foliga e leai ni fa'ailoga fa'apenei ma e fetaui lelei ma va'aiga ata.

Laulau 3. Su'esu'ega fa'atatau ile RMVP3D fa'atasi ai ma le va o le Chamfer (↓). O la matou metotia e ausia ai le laʻititi laʻititi i le averesi.

Metotia	MAILE	LEONA	RAE	PALO	averesi
NeRO [19]	9.11	10.74	6.21	3.87	7.48
S-VolSDF [35]	9.93	7.39	7.91	18.4	10.91
MVAS [6]	9.23	7.51	9.90	4.77	7.86
PANDORA [9]	14.3	15.04	11.27	3.96	11.14
NeRSP (O matou)	8.80	5.18	6.70	3.84	6.13

Iloiloga fa'atatau ile RMVP3D. E pei ona faʻaalia i le Laulau 3, matou te tuʻuina atu se iloiloga faʻatusatusa o le RMVP3D e faʻavae i luga o le mamao o Chamfer. E o gatasi ma le su'esu'ega fa'akomepiuta, e ausia e le matou NeRSP le la'ititi la'ititi fa'atatau sese i le averesi. O foliga vaaia o loʻo faʻaalia i le Ata 10 o loʻo faʻaalia atili ai o mea e faʻaalia e luʻitauina i le S-VolSDF [35] mo le faʻamavaeina o foliga mai le faʻaalia, e pei ona faʻaalia e le faʻafefe o le mea FROG i totonu o le tapuni. views. NeRO [19] ma PANDORA [9] o loʻo i ai faʻatusatusaga faʻatusatusa tutusa ma i matou i luga o le mea faigofie PALL. Mo foliga lavelave e pei o le LION, fa'aletonu foliga toe maua mai i nei metotia ona o le le lava view fa'atulagaina, a'o tatou latalata i le GT meshes, fa'aalia le aoga o la tatou metotia i le toe fa'aleleia o luga ole lalolagi i lalo ole mea e fa'aogaina.

Fa'ai'uga

Matou te tuʻuina atu le NeRSP, o se auala toe faʻaleleia 3D neural mo luga o le ata i lalo o ata polarized seasea. Ona o lu'itau o foliga-lalelei le mautonu ma le lavelave o le atagia, o metotia o loʻo i ai e tauivi ma mea e atagia ai poʻo seasea. views ma e le mafai ona foia faafitauli uma e lua i ata RGB. Matou te fautua atu e faʻaaoga ata polarized e fai ma faʻaoga. E ala i le tuʻufaʻatasia o faʻailoga geometric ma photometric na maua mai ata polarized, matou te faʻaititia le avanoa fofo o foliga faʻatatau, faʻatagaina mo le toe faʻaleleia lelei o le faʻaaliga faʻaalia i le itiiti ifo i le 6 views, e pei ona faʻaalia e avanoa lautele ma a matou faʻamaumauga.

Tapulaa
E le'o mafaufauina i lenei galuega le felafolafoa'iga ma le malamalama si'osi'omaga polarized, e ono a'afia ai le sa'o o le toe fausia. Na matou matauina se galuega lata mai NeISF [17] o loʻo taulaʻi i lenei autu, ma matou te fiafia i le tuʻufaʻatasia o a matou togi faʻapitoa ma lenei galuega i le lumanaʻi.
Fa'afetai
O lenei galuega sa lagolagoina e le Beijing Natural Science Foundation Project Numera Z200002, le National Nature Science Foundation of China (Grant Nu. 62136001, 62088102, 62225601, U23B2052), le Youth Innovative Research Team of BUPT Nu. 2023QHNTD02. (Fa'ameanoa Nu. JP22K17910 ma JP23H05491). Matou te faafetai ia Youwei Lyu mo talanoaga malamalama.

Fa'asinomaga

Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang, Rui Chen, Yiqin Wang, Lei Yan, Boxin Shi, ma Achuta Kadambi. Siepi loloto mai polarization. I le ECCV, itulau 554–571, 2020. 2
Seung-Hwan Baek, Daniel S Jeon, Xin Tong, ma Min H Kim. Fa'atasia le mauaina o le polarimetric SVBRDF ma masani. ACM TOG, 37(6):268–1, 2018. 2, 3, 4

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, ma Pratul P Srinivasan. Mip-NeRF: O se faʻataʻitaʻiga tele mo le faʻaogaina o faʻamalama o neural. I le ICCV, itulau 5855–5864, 2021. 2
Paul J Besl ma Neil D McKay. Metotia mo le resitalaina o foliga 3-D. I le Sensor fusion IV: fa'atonutonu fa'atusa ma fa'amaumauga fa'amaumauga, itulau 586–606, 1992. 6
Mark Pule, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, ma Hendrik Lensch. Neural-PIL: Neural mo'i fa'apipi'i fa'atasi mo le fa'aleagaina o ata. I NeurIPS, itulau 10691–10704, 2021. 1, 2

Xu Cao, Hiroaki Santo, Fumio Okura, ma Yasuyuki Matsushita. tele-View Azimuth Stereo e ala i le Tangent Space Consistency. I le CVPR, itulau 825–834, 2023. 2, 3, 4, 6, 7, 8
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, ma Hao Su. MVSNeRF: Toe fa'aleleia fa'ato'aga fa'aolaola fa'atele mai le tele-view fa'avili. I le CVPR, itulau 14124–14133, 2021. 2
Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, ma Jan Kautz. Polarimetric multi-view fa'avili. I le CVPR, itulau 1558–1567, 2017. 2

Akshat Dave, Yongyi Zhao, ma Ashok Veeraraghavan. Pandora: Polarization-fesoasoani neural decomposition o le susulu. I le ECCV, itulau 538–556, 2022. 1, 2, 4, 6, 7, 8
Kangle Deng, Andrew Liu, Jun-Yan Zhu, ma Deva Ra-manan. NeRF e vaavaaia loloto: E itiiti views ma aʻoaʻoga vave mo le leai o se totogi. I le CVPR, itulau 12882–12891, 2022. 2
Yuqi Ding, Yu Ji, Mingyuan Zhou, Sing Bing Kang, ma Jin-wei Ye. Polarimetric helmholtz stereopsis. I le ICCV, itulau 5037–5046, 2021. 2

Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, ma Ko Nishino. Polorimetric stereo masani. I le CVPR, itulau 682–690, 2021. 2
Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, ma Ying-Cong Chen. Ref-NeuS: A'oa'oga Fa'asagaga Fa'asao-Fa'a'au'au-Fa'aitiitia Neural Fa'asa'o mo le tele-View Toe Fausiaina ma Mafaufauga Loloto. arXiv muai lolomi arXiv:2303.10840, 2023. 1
Richard Hartley ma Andrew Zisserman. Fa'atele view geometry i le va'aiga komepiuta. Cambridge University Press, 2003. 1

Wenzel Iakopo. Mitsuba renderer, 2010. 6
Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, ma Qifeng Chen. Fa'asolo mai polarization mo vaaiga lavelave i le vao. I le CVPR, itulau 12632–12641, 2022. 2
Chenhao Li, Taishi Ono, Takeshi Uemori, Hajime Mihara, Alexander Gatto, Hajime Nagahara, ma Yuseke Moriuchi. NeISF: Neural Incident Stokes Field mo Geometry ma Mea Fa'atatau. arXiv muai lolomi arXiv:2311.13187, 2023. 8

Zhaoshuo Li, Thomas M¨uller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, ma Chen-Hsuan Lin. Neu-Colangelo: Toe Fausiaina o Neural Surface High-Fidelity. I le CVPR, itulau 8456–8465, 2023. 2
Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, ma Wenping Wang. NeRO: Neural Geometry ma le BRDF Toe Fausiaina o Mea Fa'afofoga mai le Teleview Ata. arXiv muai lolomi arXiv:2305.17398, 2023. 1, 2, 4, 6, 7, 8
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, ma Wenping Wang. SparseNeuS: Fa'asao fa'asao fa'aalaua'itele fa'alelei luga neural mai seasea views. I le ECCV, itulau 210–227, 2022. 2

Youwei Lyu, Lingran Zhao, Si Li, ma Boxin Shi. Faiga mai polarization ma fua mamao moli. IEEE TPAMI, 2023. 2
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, ma Ren Ng. NeRF: Fa'atusaina vaaiga e pei o neural radiance fanua mo view tuufaatasia. I le ECCV, itulau 405–421, 2020. 1, 2
Miyazaki, Tan, Hara, ma Ikeuchi. Fa'aliliuga fa'afeagai fa'avae fa'a polarization mai le tasi view. I le ICCV, itulau 982–987, 2003. 2

Michael Niemeyer, Lars Mescheder, Michael Oechsle, ma Andreas Geiger. Fa'aliliuga voluma eseese: A'oa'oina fa'atusa fa'atusa 3D e aunoa ma se va'aiga 3D. I le CVPR, itulau 3504–3515, 2020. 2
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, ma Noha Radwan. Reg-nerf: Fa'atulafonoina fanua susulu neural mo view tu'ufa'atasiga mai mea fa'aoga seasea. I le CVPR, itulau 5480–5490, 2022. 2, 4
Michael Oechsle, Songyou Peng, ma Andreas Geiger. UNISURF: Fa'atasia neural implicit surfaces ma fa'aola susulu mo le tele-view toe fausia. I le ICCV, itulau 5589–5599, 2021. 2

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, ma Steven Lovegrove. DeepSDF: A'oa'oina galuega mamao saini fa'aauau mo le fa'atusaina o foliga. I le CVPR, itulau 165–174, 2019. 1
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, ma Gordon Wetzstein. Fa'aaliga neural fa'a'ole'ole fa'atasi ai ma galuega fa'ato'aga fa'avaitaimi. I le NeurIPS, 2020. 1
William AP Smith, Ravi Ramamoorthi, ma Silvia Tozza. Maualuluga-mai-polarization ma le le iloa moli po'o albedo. IEEE TPAMI, 41(12):2875–2888, 2018. 2

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, ma Pratul P Srinivasan. Ref-NeRF: Fa'atulagaina view-faʻalagolago foliga mo fanua susulu neural. I le CVPR, itulau 5481–5490, 2022. 4, 5
Bruce Walter, Stephen R Marschner, Hongsong Li, ma Ken-neth E Torrance. Microfacet faʻataʻitaʻiga mo le faʻafefe i luga o luga o luga. I Taualumaga o le 18th Eurographics conference on Rendering Techniques, itulau 195–206, 2007. 4
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, ma Ziwei Liu. SparseNeRF: Fa'avasegaga loloto mo ni nai ata pu'upu'u view tuufaatasia. arXiv muai lolomi arXiv:2303.16196, 2023. 2

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, ma Wenping Wang. NeuS: A'oa'oina Neural Implicit Surfaces e ala i le Volume Rendering mo le tele-view Toe fa'aleleia. arXiv muai lolomi arXiv:2106.10689, 2021. 1, 2, 5
Yiqun Wang, Ivan Skorokhodov, ma Peter Wonka. HF-NeuS: Faʻaleleia le toe faʻaleleia o luga ole faʻaogaina o faʻamatalaga maualuga. I le NeurIPS, itulau 1966–1978, 2022. 2
Haoyu Wu, Alexandros Graikos, ma Dimitris Samaras. S-VolSDF: Fa'ase'e Tele-View Stereo Fa'atonuina o Neural Implicit Surfaces. arXiv muai lolomi arXiv:2303.17712, 2023. 1, 2, 4, 6, 7, 8

Jiawei Yang, Marco Pavone, ma Yue Wang. FreeNeRF: Fa'aleleia le Fa'aa'oa'oga Neural Fa'ato'aga Fa'atasi ma Fa'atonu Fa'atonu Fa'atonu. I le CVPR, itulau 8254–8263, 2023. 2
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, ma Yaron Lipman. Teleview toe fa'aleleia o luga neural e ala i le fa'a'ese'ese'ese o le geometry ma foliga vaaia. I NeurIPS, itulau 2492–2502, 2020. 2
Lior Yariv, Jiatao Gu, Yoni Kasten, ma Yaron Lipman. Le fa'aliliuina o le voluma o luga ole neural. I le NeurIPS, itulau 4805–4815, 2021. 1, 2

Alex Yu, Vickie Ye, Matthew Tancik, ma Angjoo Kanazawa. pixelNeRF: Neural radiance fanua mai se tasi pe itiiti ata. I le CVPR, itulau 4578–4587, 2021. 2
Kai Zhang, Gernot Riegler, Noa Snavely, ma Vladlen Koltun. NeRF ++: Suʻesuʻeina ma faʻaleleia fanua susulu neural. arXiv muai lolomi arXiv:2010.07492, 2020. 2
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, ma Noa Snavely. PhySG: Fa'aliliuga fa'afeagai fa'atasi ma Gaussians lapotopoto mo le fa'asa'oina ma le toe fa'amalamalamaina o meafaitino fa'avae. I le CVPR, itulau 5453–5462, 2021. 1, 2

Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul De-bevel, William T Freeman, ma Jonathan T Barron. NeR-Factor: Fa'avasegaina neural o foliga ma le atagia i lalo o se malamalama e le iloa. ACM TOG, 40(6):1–18, 2021. 2
Jinyu Zhao, Yusuke Monno, ma Masatoshi Okutomi. Polarimetric tele-view fa'aliliuga fa'afeagai. IEEE TPAMI, 2022. 2

Photometric ma fa'ailoga fa'ailoga o le NeRSP

Fa'atupuina o fa'ailoga fa'ailoga
E pei ona faʻaalia i le Ata S1, tuʻuina atu se vaaiga mataʻina e eseese views, lona luga masani i le sini view e mafai ona fa'atusalia e le azimuth ma tulimanu maualuga ϕ ma θ fa'asologa, ie,

O le sootaga i le va o le azimuth angle ma le elemene o luga masani e mafai ona faʻatulagaina e pei o

Ole luga ole mea masani ile fa'amoemoe view e mafai ona fa'atatau ile feliuliua'i ole mea masani ile puna view, ie ˆn = Rn. Tuuina atu o le rotation matrix mai le meapue fa'avasegaina o lo'o fa'atusalia R = [r1, r2, r3]⊤, Eq. (2) fa'avae ile ˆn e mafai ona fa'atusaina e pei o

r⊤1 n cos ϕ − r⊤ 2 n agasala ϕ = 0. (3)

I le mulimuli ai i le MVAS [2], e mafai ona matou toe faʻatulagaina le Eq. (3) ia maua le va o le va o luga masani ma le faʻataʻitaʻiina o le vete t(ϕ) e pei ona faʻamatalaina i lalo,

O lenei faaiuga i luga o le azimuth angle e mafai ona faʻalauteleina i le tulimanu o le polarization (AoP). O le π ambiguity e mafai ona fo'ia fa'alenatura e pei ole Fa'atusa. (4) e tu pe a tatou faaopoopo le ϕ i le π. O le π/2 ambiguity e mafai ona fa'atalanoaina e ala i le fa'aogaina o se pseudo-projected tangent vector tˆ(ϕ) e faapea.

Afai e tasi le vaaiga mata'ina x e matauina e le f views, e mafai ona matou fa'aputuina Eq. (4) ma le Eq. (5) faʻavae i luga o suiga eseese ma mataʻituina AoPs, e taʻitaʻia ai se laina laina

T(x)n(x) = 0. (6)

Matou te faʻaogaina lenei faiga faʻasologa o la matou faʻailoga faʻailoga mo le tele-view toe fausia 3D polarized.

Fa'atupuina ole fa'ailoga photometric
I le mulimulita'ia o le fa'ata'ita'iga BRDF fa'a polarized [1], e mafai ona fa'apalapala le ve'a fa'avevela i vaega fa'asalalau ma fa'apitoa fa'atusa fa'atusa e ala i le Hd ma le Hs tutusa, ie,

O le diffuse stokes vaega i lalo o se moli e tasi e mafai ona faia e pei

o le ρd o loʻo faʻaalia ai le albedo faʻasalalau, ϕn o le azimuth angle o le malamalama faʻafuaseʻi i luga o le vaalele e faʻasaga i luga o le mea masani, T + i, o ma T−i, o faʻaalia le faʻatusatusaga o faʻasologa o felauaiga o Fresnel [1] e fesoʻotaʻi ma le tulimanu i le va view faasinoga ma luga masani. I le mulimulitaia o manatu i le PANDORA [3], matou te toe tusia le diffuse stokes vector i lalo o le malamalama o le siosiomaga e pei

o fea ua fa'ailogaina o le susulu fa'asalalau. Nai lo le fa'atatauina mai le fa'atusa, o le fa'aaliga fa'asalalau e pei o se fesuia'iga va'ava'a e va'aia sa'o mai se vaega neural point na maua mai e se MLP fa'avae fa'amaopoopo. I le isi itu, o le specular stokes vector i lalo o se itu malamalama e tasi ω i le polarimetric BRDF model e mafai ona faauigaina

o le ρs o lo'o fa'aalia ai le specular albedo; O le D ma le G o loʻo faʻaalia ai le tufatufaina masani ma le faʻataʻitaʻiga i le Microfacet faʻataʻitaʻiga [8], lea e mafai ona faʻatonutonuina e ala i luga o le eleele; R+ ma le R− fa'ailoa mai le fa'atusatusaga o le fa'atusa o le Fresnel reflecting coefficients [1], lea e feso'ota'i ma le tulimanu i le va o le fa'afanua masani ma le fa'atonuga o le malamalama; ϕh o le mea na tupu azimuth angle wrt le afa vector. Mulimuli i manatu i le PANDORA [3], matou te toe tusia le specular stokes vector i lalo o le siosiomaga malamalama e pei

o fea e fa'aalia ai le susulu fa'apitoa. Faatasi ai ma le fa'atusatusaga o le masaa [5], e mafai ona tatou fa'aopoopoina le Ls ≈ ρsDG/4n⊤v *fΩ L(ω) dω. Tu'ufa'atasia ma le so'oga fa'asalalau fa'aalia i le Fa'atusa. (9), matou te fausia le faʻailoga photometric e faʻavae i luga o le faʻataʻitaʻiga o ata faʻataʻitaʻiga o loʻo mulimuli mai

Fa'amatalaga Fa'atinoga

O lenei vaega o loʻo tuʻuina atu ai faʻamatalaga faʻamatalaga o la matou Synthetic Multi-view Polarized image dataset SMVP3D ma faʻamatalaga aʻoaʻoga a NeRSP.

Fa'amaumauga
Matou te tuʻuina atu le SMVP3D, o loʻo i ai ata o mea faʻataʻitaʻi e lima i lalo o le malamalama masani. Mo mea taitasi, matou te tuuina atu le 48 views ma fa'amaumau fa'afanua masani o luga o le eleele (GT). Matou te faʻaogaina le Mit-suba3 [4] e fai ma afi faʻaliliu, faʻatasi ai ma le BRDF ituaiga seti i mea palasitika polarized i la matou faʻaaliga. Mo le dif-fuse albedo ρd, matou te fa'aogaina le 'albedo texture e fa'aleleia ai le moni o a matou fa'ai'uga. I le taimi lava e tasi, matou te tausia le specular albedo ρs i se tau tumau o le 1.0 ma faʻapipiʻi le gaogao i luga o le 0.05. O lenei faiga e fa'amautinoa ai le fa'afoliga tutusa i luga o luga o mea. O ata polarized e maua mai o lo'o tu'uina atu ile iuga ole 512 × 512 pixels.

Aoaoga
O le hyperparameters λg, λm, ma le λe i la tatou galuega leiloa ua seti i le 1, 1, ma le 0.1, faasologa. I le taimi o le aʻoaʻoga, matou te faʻaaogaina se taʻiala mafanafana e mulimuli i le PAN-DORA [3], lea mo le 1 taimi muamua, matou te manatu naʻo faʻamatalaga e leʻi faʻaogaina i le photometric cue ma faʻapea o le vaega faʻapitoa o le mea e 000. I mea uma. faʻataʻitaʻiga, matou te faʻaaogaina se iugafono o le 0 × 512 mo aʻoaʻoga ma suʻega i SMVP512D, ma le 3 × 512 mo faʻamaumauga o le lalolagi moni. O la matou metotia e masani ona faʻafesoʻotaʻi i le 612, 100 epochs, lea e tusa ma le 000 itula i luga o le Nvidia RTX 6 GPU, faʻatasi ai ma le mafaufau e faʻaaoga ile 3090, 8 MB.

BRDF fua fa'atatau ma toe fa'afouga i'uga

Ata S4 (pito i luga) o loʻo faʻaalia ai la matou faʻatusatusaga o mea gaoa, faʻasalalau, ma vaega faʻapitoa. E fai si pisapisao ia tala fa'atatau ona e na'o le 6 views. E tutusa ma Ref-NeRF [7] pe a faʻamalamalamaina le faʻamalamalamaina e ala i le IDE, e le mafai ona matou faia ni suʻega faʻamalamalamaina. O le mea lea, matou te faʻaalia le tala view fa'asologa o fa'ai'uga, e pei ona fa'aalia i le Ata S4 (lalo). Pe a faʻatusatusa i metotia o loʻo i ai, o a tatou toe faʻaliliuina ata e latalata atu i faʻamatalaga moni o le lalolagi.

Fa'ai'uga fa'aopoopo ile matou fa'amaumauga

I lenei vaega, matou te tuʻuina atu faʻamatalaga faaopoopo o le toe faʻaleleia o foliga ile SMVP3D ma le Real-world Multi-view Fa'asologa o ata fa'apolokalame RMVP3D.

Iloiloga ile SMVP3D
Matou te tu'uina atu fa'ai'uga lelei o le toe fa'aleleia o metotia fa'avae ma la matou faiga i le Ata S2. O fa'ai'uga mai le MVAS [2] e leai ni fa'amatalaga, ona e le'o amanaia le fa'ailoga photometric. A'o ofoina atu e NeRO [6] le toe fa'aleleia o foliga, e le mafai ona tu'uina atu se tulaga fa'atuatuaina mo mea e leai ni lanu, e pei o DAVID. S-VolSDF [9] e fa'aogaina se fa'a-mama-i-leleiView Stereo (MVS) fa'alatalata ma fa'aalia le fa'atupula'ia o fa'amatalaga fa'amatalaga i luga o mea faitino, ma o nisi taimi e o'o atu ai i le fa'auigaseseina o fa'amatalaga o mea e fai ma fa'atulagaga fa'avae. PANDORA [3] e faigata ona tu'u'ese'ese lelei fa'amatalaga albedo ma fa'amatalaga fa'apitoa, e o'o atu ai i taunu'uga le toe fa'aleleia. O la matou metotia, NeRSP, e fa'aoga lelei uma fa'ailoga photometric ma geometric, e maua ai le toe fa'aleleia e sili atu ona atagia ai le fausaga GT.

Matou te faʻaalia foʻi faʻamatalaga masani masani ma faʻasalalauga faʻasolosolo tutusa i le Ata S3, o loʻo faʻaalia ai o le NeRSP e sili atu le lelei o le toe faʻaleleia o faʻamaumauga mo luga o le ata ma le faʻaogaina o mea itiiti. views.

Iloiloga ile RMVP3D
I lenei vaega, matou te tuʻuina atu se isi mea toe faʻaleleia iʻuga ile RMVP3D. O le ata S5 o loʻo faʻaalia ai o NeRO [6], MVAS [2], ma NeRSP e mafai ona toe faʻaleleia saʻo se mea faʻataʻavalevale faigofie ma se mea e faʻaalia ai. I se faʻatusatusaga, e le mafai e le S-VolSDF [9] ma le PANDORA [3] ona faʻaumatia le albedo ma le vaega faʻapitoa o luga, e mafua ai le faʻalavelave i le faiga o le toe faʻaleleia o foliga. Ina ia iloa le eseesega i le toe fausia o taunuuga o NeRO [6], MVAS [2], ma NeRSP, matou te vaʻavaʻai i le Chamfer Distance mo meshes toe fausia e auala taʻitasi. E pei ona fa'aalia i le Ata S6, o le lanu o togi ta'itasi e fa'ailoa mai ai lona Va'a o le Chamfer, lea e tipi i le va o le 0 ma le 5 mm. O nei faʻataʻitaʻiga o loʻo faʻaalia ai o le toe faʻaleleia o mea sese e fesoʻotaʻi ma NeRSP e laʻititi pe a faʻatusatusa i isi auala e lua.

Su'esu'ega fa'a'ese'esega ile fa'afoliga o luga

O la matou metotia e faʻamoemoe i le toe faʻaleleia o luga, ma e mafai foi ona faʻaoga i le toe faʻaleleia o foliga i luga o luga. I le avea ai ma se example, matou toe tuʻuina atu le mea SNAIL ma lona specular albedo ρs faʻaititia mai le 1.0 i le 0.1. Ole mean angular error (MAE) ole fa'atatau ile mea masani ile 6 fa'aoga views mai auala eseese o loʻo faʻaalia ile Siata S1. Ole su'esu'ega fa'apitoa ole fua fa'atatau ile pito i luga ole laiga ma le fa'asoa sese fa'aitu'u tutusa o metotia 'ese'ese i lalo ole fa'aoga tutusa. view o lo'o fa'aalia i le Ata S7. O nei su'esu'ega ua fa'ailoa mai ai o le tele o metotia e fa'aleleia atili ai le lelei o le toe fa'aleleia o luga o luga o luga o mea talatala pe a fa'atusatusa i luga o le ata. Aemaise lava, o la matou metotia e tuʻuina atu i taimi uma le toe faʻaleleia o luga o le mea.

Su'esu'ega fa'amama ile #views

O le matou NeRSP e fa'amoemoe i le toe fa'aleleia o mea fa'afofoga i lalo o le tele o fa'aoga views. O faʻataʻitaʻiga o loʻo faʻaalia i le pepa autu e 6 seasea views e fai ma fa'aoga. E iloilo a matou metotia i lalo o numera eseese o faʻaoga views (ie, #views), matou te faia suʻesuʻega i le mea moni o le lalolagi LION i lalo o le seti o le 3, 6, 12, ma le 24. views. O le Ata S8 o lo'o fa'aalia ai foliga ua toe maua, a'o le su'esu'ega agava'a ma le Chamfer Distance o lo'o tu'uina atu ile Laulau S2.

I lalo o le fa'aogaina o mea laiti views, pei o le 3, o loʻo i ai nei auala e tauivi e toe faʻaleleia iʻuga talafeagai. E mafua ona latou taula'i atu i fa'ailoga photometric po'o fa'ailoga fa'ata'ita'i. Avea le S-VolSDF [9] e fai ma example, le foliga fuafuaina, e pei ona matauina i le latalata views, e matua'i a'afia i le fa'atusa tutusa. E mafua ai le le sa'o o siepi ona o le le mautonu o foliga- susulu i lalo ole seasea views. E ala i le fa'aogaina o fa'ailoga fa'atusa ma fa'ailoga photometric, o le matou NeRSP e fa'aitiitia ai le fa'anenefu i lalo ole mea fa'aoga. O se taunuuga, matou te ausia le toe faʻaleleia o foliga talafeagai. O lenei matauga e tumau pea le aoga pe a o'o i le numera o fa'aoga views e sili atu i le 12. E pei ona fa'aalia i le Laulau S2, o lo'o fa'aauau pea ona ausia e le matou NeRSP le La'ititi la'ititi la'ititi ma le fa'ateleina o le aofa'i o fa'aoga. views. O lo'o fa'aalia ai le aoga o la tatou metotia i luga o mea fa'afofoga i luga o le tele o vaega views.

Laulau S2. Su'esu'ega agava'a ile LION e fua ile Chamfer Distance (↓) ile fa'aoga eseese views.

#Views	NeRO [6]	S-VolSDF [9]	MVAS [2]	PANDORA [3]	NeRSP
3	34.48	31.50	23.96	24.44	24.01
6	10.74	7.39	7.51	15.04	5.18
12	5.50	6.80	5.31	12.1	4.29
24	4.96	6.14	5.32	12.5	4.11

Iloiloga o faʻamaumauga MVIR polarimetric

E ese mai i faʻataʻitaʻiga moni i le lalolagi i luga o le PANDORA dataset [3] ma la matou RMVP3D, matou te tuʻuina atu foi le iloiloga o le tele-view fa'asologa o ata fa'apipi'i o lo'o iai ile PMVIR [10]. E pei ona faʻaalia i le Ata S9, matou te vaʻavaʻai i le faʻaleleia o foliga mai PANDORA [3] ma matou, ave 6 seasea. views e fai ma fa'aoga. Talu ai e leai se foliga GT i lenei faʻamaumauga, matou te faʻaaogaina iuga mai le PMVIR [10] e fai ma faʻamatalaga, lea e manaʻomia ai le 31 ma le 56. views e fai ma fa'aoga mo le meapueata ma le va'aiga o le ta'avale, ta'itasi. Matou te matauina o matou taunuuga e sili atu ona talafeagai pe a faʻatusatusa ia i latou o loʻo faʻaaogaina PANDORA [3], faʻaalia le aoga o la matou metotia i le toe faʻaleleia o le 3D.