NeRSP CVPR24 Neural 3D Reconstruction bakeng sa Bukana ea Tataiso ea Lintho tse Reflective

Bangoli: Yufei Han, Heng Guo, Koki Fukai, Hiroaki Santo, Boxin Shi, Fumio Okura, Zhanyu Ma, Yunpeng Jia
Likamano: Beijing University of Posts and Telecommunications, Osaka University, Peking University

Kakaretso: Sehlahisoa sa NeRSP se fana ka liphetho tse ntlafalitsoeng tsa kaho ea sebopeho bakeng sa libaka tse khanyang ha li bapisoa le mekhoa e teng.

Litaelo tsa Tšebeliso ea Sehlahisoa

Selelekela
Sehlahisoa sa NeRSP se etselitsoe tokiso ea 3D ea lintho tse bonahatsang ho sebelisoa litšoantšo tsa polarized. E hlola mathata a amanang le ho fumana tse ngataview mangolo a ngollano le sebopeho se arohanyang ho tloha ho khanya tlas'a mangolo a fokolang.

Mosebetsi o Amanang
NeRSP e bululetsoe ke Neural Radiance Fields (NeRF) le mekhoa e meng ea neural 3D reconstruction. E fana ka mohlala oa sebopeho sa bokaholimo ka mokhoa o hlakileng ka sebaka sa sebaka se saennoeng (SDF) mme e sebelisa mokhoa o fapaneng oa ho lekola le ho fana ka molumo ho ntlafatsa boleng ba kaho bocha.
Polarimetric Image Formation Model
NeRSP e kenyelletsa mohlala oa sebopeho sa setšoantšo sa polarimetric ho fumana lintlha tsa photometric le geometric bakeng sa ho aha bocha.

LBH:

P: Advan ke engtagNa ea ho sebelisa NeRSP bakeng sa kaho bocha ea 3D?
A: NeRSP e fana ka liphetho tse betere tsa kaho ea sebopeho bakeng sa libaka tse bonts'ang ha li bapisoa le mekhoa e teng ka lebaka la mokhoa oa eona o mocha o sebelisang litšoantšo tse se nang polarized.

P: Na sehlahisoa sa NERSP se loketse libaka tse pharalletseng?
A: Le ha NeRSP e etselitsoe libaka tse khanyang, e ka fana ka khakanyo e kholisang ea sebopeho bakeng sa libaka tse pharalletseng moo photometric consistency e sebetsang hohle. views.

NeRSP:
Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images

Yufei Han1† Heng Guo1†∗ Koki Fukai2† Hiroaki Santo2 Boxin Shi3,4 Fumio Okura2 Zhanyu Ma1 Yunpeng Jia1

Univesithi ea Beijing ea Posts le Telecommunications

Sekolo sa Thuto ea Mahlale le Theknoloji, Univesithi ea Osaka
National Key Laboratory bakeng sa Multimedia Information Processing, School of Computer Science, Peking University 4National Engineering Research Center ea Visual Technology, School of Computer Science, University Peking
- {hanyufei, guoheng, mazhanyu}@bupt.edu.cn shiboxin@pku.edu.cn
- {santo.hiroaki, okura, fukai.koki}@ist.osaka-u.ac.jp xibei156@163.com.

Kakaretso

Re hlahisa NeRSP, mokhoa oa ho aha bocha oa Neural 3D bakeng sa libaka tse Reflective tse nang le litšoantšo tsa Sparse Polarized. Ho aha bocha bokaholimo ho na le phephetso e kholo joalo ka ha likhakanyo tse ikhethang li le teng view-e itshetlehile mme ka hona e tlola bongataview consistency for multiview setirio. Ka lehlakoreng le leng, ho kenya litšoantšo tse fokolang, joalo ka mokhoa o sebetsang oa ho hapa, hangata ho baka liphetho tse sa fellang kapa tse sothehileng ka lebaka la khaello ea mangolo a tsamaisanang. Pampiri ena ka kopanelo e sebetsana le liphephetso tsa ho kenella ka mokhoa o fokolang le libaka tse khanyang ka ho sebelisa litšoantšo tse polarized. Re fumana litlatsetso tsa photometric le geometric ho tsoa ho mohlala oa ho etsa setšoantšo sa polarimetric le tse ngataview azimuth consistency, eo ka kopanelo e ntlafatsang sebopeho sa bokaholimo se entsoeng ka boemeli bo hlakileng ba neural. Ho ipapisitsoe le liteko ho li-dataset tsa rona tsa maiketsetso le tsa 'nete, re fihlela liphetho tsa kaho ea boemo bo holimo ka 6 feela. views joalo ka ho kenya letsoho.

Selelekela

Tse ngataview Ho aha bocha 3D ke bothata ba mantlha ponong ea komporo (CV) mme e ithutoe haholo ka lilemo tse ngata [14]. Ka tsoelo-pele ea boemeli bo hlakileng ba holim'a metsi [27, 28] le masimo a mahlaseli a neural [22], mefuta e mengata ea morao tjena.view Mekhoa ea kaho ea 3D [5, 33, 38, 41] e hatetse pele haholo. Leha ho na le liphetho tse tlamang tsa ho hlaphoheloa ha sebopeho, tse ngata tse ngataview mekhoa ea stereo (MVS) e ntse e itšetlehile haholo ka ho fumana mangolo pakeng tsa views, e leng phephetso ka ho khetheha bakeng sa libaka tse khanyang le ho kenya letsoho ka mokhoa o fokolang views.

Bakeng sa libaka tse khanyang, the view-Dependent surface app-

Monehelo o lekanang. ∗ Mongoli ea ngollang.
Leqephe la morero: https://yu-fei-han.github.io/NeRSP-project/.

Setšoantšo sa 1. Ponahalo ea sebopeho sa bokaholimo bo khanyang ho tsoa ho litšoantšo tse 6 tse polarized tse nkang (mela e kaholimo). NeRSP ea rona e fihlela sephetho se betere sa kaho ea sebopeho ha se bapisoa le mekhoa e teng e sebetsanang le li-input tse nyane (S-VolSDF [35]) kapa reflective reflectance (PANDORA [9]).

Pearance e senya khopolo ea ho lumellana ha photometric e sebelisoang khakanyong ea mangolo ho MVS. Ho rarolla bothata bona, mekhoa ea morao-rao ea neural 3D ea kaho (mohlala, Ref-NeuS [13], NeRO [19], le PANDORA [9]) e etsa mohlala ka ho hlaka 'me ka nako e ts'oanang e hakanya limmapa tsa ponahatso le tikoloho ka phetolelo e fapaneng. Leha ho le joalo, ho fumaneha ha litšoantšo ka mefuta e fapaneng views e hlokahala ho sebetsana ka botšepehi le tse ling tse sa tsejoeng ntle le sebopeho, joalo ka albedo, roughness, le 'mapa oa tikoloho.

Ho tsoa ho tse fokolang views, hangata ho thata ho fumana bongata bo lekanengview ngollano. Haholoholo ha o emetse view-mekhabiso e itšetlehileng ka eona, ho thata ho arola sebopeho ho tloha ho khanya tlas'a palo e fokolang ea mangolo, e lebisang ho sa hlakang ha sebopeho sa mahlaseli [40]. Mekhoa ea morao-rao ea neural 3D ea kaho ea sparse views (mohlala, S-VolSDF [35] le SparseNeuS [20]) li hloka hore ho be le kamehla ho sebelisa photometric consistency, e ka senyehang bakeng sa libaka tse khanyang.

Ho rarolla mathata ana ka bobeli, re sisinya ho sebelisa litšoantšo tsa polarized tse fokolang ho fapana le ho kenya RGB. Ka ho khetheha, re sisinya NeRSP, mokhoa oa ho aha bocha oa Neural 3D ho khutlisa sebopeho sa Reflective ho tsoa litšoantšong tsa Sparse Polarized. Re sebelisa angle of polarization (AoP) e nkiloeng litšoantšong tse polarized, e bonts'ang ka kotloloho angle ea azimuth ea sebopeho sa bokaholimo ho fihla ho π le π/2 ambiguities. Leqheka lena la jiometri le tsebahala ka ho thusa tse ngataview kaho ea sebopeho ho sa tsotelehe thepa e bonahatsang bokaholimo, empa sebopeho se hakantsoeng se ipapisitse le leqheka la geometri ha se na moelelo [6] ha se letho. view litlhophiso. Ka lehlakoreng le leng, setšoantšo sa photometric se tsoang ho sebopeho sa setšoantšo sa polarimetric [2] se thusa ho tsosolosoa ha methapo ea kutlo (mohlala, PANDORA [9]) ka ho fokotsa phapang pakeng tsa litšoantšo tse entsoeng bocha le tse hapiloeng. Leha ho le joalo, sebopeho se hakantsoeng se ipapisitseng le "photometric cue" le sona ha se tsejoe hantle ka lebaka la ho se utloisisehe ha sebopeho. Ho fapana le mokhoa o teng oa polarimetric-based PANDORA [9] ha ho nahanoa ka mokhoa oa photometric feela, NeRSP ea rona e bonts'a kopanyo ea li-geometric le li-photometric cues ka katleho e fokotsa sebaka sa tharollo bakeng sa sebopeho sa bokaholimo, se bonts'itsoeng se sebetsa kahong bocha e bonahatsang holim'a metsi ho ipapisitsoe le lintlha tse nyane. , joalokaha ho bontšitsoe setšoantšong sa 1.

Ntle le NeRSP e reriloeng bakeng sa kaho bocha ea 3D, re boetse re hahile Multi-worldView Lethathamo la polarized la litšoantšo le nang le lintho tse 6 tse nang le meshes ea 3D e tsamaisanang le 'nete ea fatše, e bitsoang RMVP3D. E fapane le li-dataset tse seng li ntse li le teng joalo ka dataset ea PANDORA [9] e fanang ka litšoantšo tsa polarized feela, li-meshes tsa GT tse tsamaellanang le maemo a holimo bakeng sa e 'ngoe le e 'ngoe. view lumella tlhahlobo ea palo ea bongataview polarized 3D kaho bocha.

Ho akaretsa, re tsoela pele ka multiview 3D kaho bocha ka ho sisinya

NeRSP, mokhoa oa pele o fanang ka tlhahiso ea ho sebelisa tlhaiso-leseling ea polarimetric bakeng sa kaho e bonts'ang holim'a sebaka se tlase. views;

tlhahlobo e pharaletseng ea setšoantšo sa photometric le geometri e nkiloeng litšoantšong tsa polarized; le
RMVP3D, sehlopha sa pele sa lefatše sa 'neteview dataset ea polarized e nang le libopeho tsa GT bakeng sa tlhahlobo ea bongata.

Mosebetsi o amanang

Tse ngataview Kaho bocha ea 3D e 'nile ea ithutoa haholo ka lilemo tse mashome. Neural Radiance Fields (NeRF) [3, 22, 40] e fihletse katleho e kholo bukeng ea lipale view synthesis lilemong tsa morao tjena. E bululetsoe ke NeRF, mekhoa ea tsosoloso ea neural 3D [24] e hlahisoa, moo sebopeho sa bokaholimo se entsoeng ka mokhoa o hlakileng ka sebaka sa sebaka se saennoeng (SDF). Ho qala ho tloha DVR [24], mekhoa e lateloang e ntlafatsa boleng ba ho aha sebopeho ka mokhoa o fapaneng oa ho latella sebaka [37], tlhahiso ea molumo [26, 33, 38], kapa boemeli bo ntlafalitsoeng ba sebopeho [18, 34]. Mekhoa ena e ka fihlela khakanyo e kholisang ea sebopeho bakeng sa libaka tse pharalletseng moo foto-metric consistency e sebetsang hohle. views.

Ho haha bocha bakeng sa libaka tse khanyang ho thata kaha photometric consistency e fosahetse. Mekhoa e teng [5, 41, 42] ka ho hlaka mohlala oa view-e itšetlehileng ka ponahatso le ho arola sebopeho, li-luminations tse fapaneng tsa sebaka, le thepa ea ho bonahatsa joalo ka albedo le ho ba thata. Leha ho le joalo, likhakanyo tsa mefuta e ka holimo ha li khotsofatse kaha khaohano e na le boemo bo bobe haholo. NeRO [19] e fana ka tlhahiso ea ho sebelisa khakanyo ea kakaretso ea sebopeho sa sebopeho sa setšoantšo le ho ntlafatsa boleng ba kaho ea sebopeho ntle le ho hloka limaske tsa ntho. Leha ho le joalo, mekhoa e kaholimo hangata e hloka ho nka litšoantšo tse teteaneng ho netefatsa liphetho tse hlakileng tsa ho khutlisa sebopeho bakeng sa libaka tse nang le phephetso tse hlakileng.

Ho tsosolosoa ka mokhoa o fokolang views e bohlokoa bakeng sa maemo a sebetsang a hlokang ho ts'oaroa hantle. Ka lebaka la khaello ea mangolo a lekaneng a tsoang ho fokolang views, ho se hlaka ha sebopeho-hlaseli ho ke ke ha rarolloa, ho lebisang ho lerata le ho sotha sebopeho. Mekhoa e teng e sebetsana le bothata bona ka ho eketsa li-regulations tse kang boreleli ba geometry ea holim'a metsi [25], botebo bo tebileng pele ho [10, 32], kapa taolo ea maqhubu a khouto ea boemo [36]. Mekhoa e meng [7, 20, 39] e theha 3D ea kaho e nyane joalo ka bothata bo akaretsang ba 3D moo likarolo tsa setšoantšo se koetlisitsoeng esale pele li sebelisoa e le lintlha tse akaretsang. S-VolSDF [35] e sebelisa mefuta e mengata ea khaleview mokhoa oa stereo joalo ka ho qala le ho hlophisa ntlafatso ea neural rendering ka molumo oa monyetla. Leha ho le joalo, e ntse e le phephetso bakeng sa mekhoa ea hajoale ea ho khutlisa libaka tse khanyang ka nepo.

Kaho e ncha e sebelisang litšoantšo tsa polarized e ithutoe bakeng sa bobeli ba motho a le mong.view litlhophiso [1, 2, 16, 23, 29] le tse ngataview litlhophiso [6, 8, 9, 11, 12, 43]. Ho fapana le litšoantšo tsa RGB, AoP e tsoang litšoantšong tse polarized e fana ka lintlha tse tobileng bakeng sa maemo a holimo. Motho a le mong-view sebopeho ho tloha ho mekhoa ea polarization (SfP) e rua molemo thepa ena mme e hakanya sebaka se tloaelehileng tlas'a leseli le le leng le hōle [21, 29] kapa leseli la tlhaho le sa tsejoeng [1, 16]. Tse ngataview Mekhoa ea SfP [8, 43] e rarolla mathata a π le π/2 ho AoP a ipapisitse le bongata.view tse hlokometsoeng. PANDORA [9] ke mokhoa oa pele oa ho aha bocha 3D oa neural o thehiloeng ho litšoantšo tse polarized, o bonts'itsoeng o sebetsa hantle ho khutliseng sebopeho sa bokaholimo le khanya. MVAS [6] e khutlisa sebopeho sa bokaholimo ho tsoa ho tse ngataview limmapa tsa azimuth, tse amanang haufi-ufi le limmapa tsa AoP tse nkiloeng ho tsona

litšoantšo tse polarized. Leha ho le joalo, mekhoa ena ha e hlahlobe ho sebelisa litšoantšo tse nang le polarized bakeng sa ho aha bokaholimo bo bocha ka lithunya tse nyane.

Polarimetric Image Formation Model

Pele re kenella ka har'a mokhoa o reriloeng, re qala ka ho hlahisa mofuta oa sebopeho sa setšoantšo sa polarimetric mme re fumana sesupo sa foto-metric cue le geometric cue mokhoeng oa rona. Joalokaha ho bontšitsoe setšoantšong sa 2, khamera ea snapshot polarization e tlaleha maikutlo a setšoantšo ka li-angles tse 'nè tse fapaneng tsa polarization, ka litekanyetso tsa eona tsa pixel tse hlalositsoeng e le {I0, I45, I90, I135}. Litšoantšo tsena tse 'ne li senola boemo ba polarization ea mabone a amoheloang, a emeloang e le 4D Stokes vector s = [s0, s1, s2, s3] computed as

Re nahana hore ha ho na leseli le pota-potileng ka mokhoa o ts'oanelang ho abela s3 hore e be 0. Vector ea Stokes e ka sebelisoa ho compute angle of polarization (AoP), ke hore.

Ho ipapisitsoe le vector ea AoP le Stokes, re fumana lintlha tsa geometric le photometric ka ho tšoana.

Sebopeho sa geometri
Ho fanoe ka AoP ϕa, angle ea azimuth ea bokaholimo e ka ba ϕa + π/2 kapa ϕa + π, e tsejoang e le π le π/2 ambiguity ho latela hore na bokaholimo bo khethehile kapa bo atile. Karolong ena, re qala ka ho hlahisa lesela la geometri le tlisoang ke mefuta e mengataview 'mapa oa azimuth ebe o o fetisetsa ho AoP.

Ho latela MVAS [6], bakeng sa sebaka sa x, bokaholimo ba eona bo tloaelehile n le angle ea azimuth e lebelletsoeng ϕ khamera e le 'ngoe. view latela kamano e le

moo R = [r1, r2, r3]⊤ ke matrix a ho potoloha ha khamera. Re ka boela ra hlophisa Eq. (3) ho fumana kamano ea orthogonal lipakeng tsa maemo a holimo le a tangent vector t(ϕ) joalo ka ha ho hlalositsoe ka tlase,

Ho se hlaka ha π lipakeng tsa AoP le angle ea azimuth ho ka rarolloa ka tlhaho joalo ka Eq. (4) e eme haeba re kenyelletsa ϕ ka π. Ho se utloisisehe ha π/2 ho ka rarolloa ka ho sebelisa vector ea pseudo-projected tangent tˆ(ϕ) hore

Haeba setšoantšo se le seng ntlha x e bonoa ke f views, re ka beha Eq. (4) le Eq. (5) e ipapisitse le k lipotoloho tse fapaneng le ho shebella li-AoPs, tse lebisang tsamaisong ea mela

T(x)n(x) = 0. (6)

Re nka sistimi ena ea mela e le sesupo sa rona sa jiometri bakeng sa mefuta e mengata.view polarized 3D kaho bocha.

Tlhaloso ea Photometric
Ho nka hore leseli la tikoloho ea ketsahalo ha le na polarized, vector ea Stokes ea leseli la ketsahalo ω e ka emeloa e le

si(ω) = L(ω)[1, 0, 0, 0]⊤, (7)

moo L(ω) e bolelang matla a khanya. Leseli le hlahang le tlalehiloeng ke khamera ea polarization le fetoha polarized ka lebaka la ponahalo. Ts'ebetso ena e entsoe ka 4 × 4 Muller matrix H. Tlas'a khanya ea tikoloho, vector e tsoelang pele ea Stokes e ka etsoa e le karolo ea ketsahalo ea Stokes vector e atisitsoeng ka Muller matrix, ke hore.

moo v le Ω li supang view tataiso le sebaka sa bohlokoa. Ho latela polarized BRDF (pBRDF) mohlala [2], vector e tsoang ho Stokes e ka aroloa ka likarolo tse fapaneng le tse ikhethang tse entsoeng ka Hd le Hs ka ho tšoana, ke hore.

Ka mor'a ho tsoa ho PANDORA [9], re ka tsoela pele ho etsa tlhahiso ea Stokes vector e le

moo Ld =fΩ ρL(ω)ω⊤n T+i T−i dω e hlalosoa e le mahlaseli a pharalletseng a amanang le bokaholimo bo tloaelehileng n, Fresnel transmission coefficients [2] T+i,o le T−i,o, diffuse albedo ρ , le angle ea azimuth ea leseli la ketsahalo ϕn. Ls = fΩ L(ω) DG 4n⊤v dω e bolela mahlaseli a khethehileng a amanang le li-coefficients tsa Fresnel reflection [2] R+ le R-, ketsahalo ea azimuth angle ϕh wrt the half vector h = ω+v∥ω+v∥22, le kabo e tloaelehileng le lentsoe la moriti D le G ka mohlala oa Microfacet [31].

Ka kopo, sheba boitsebiso bo eketsehileng bakeng sa lintlha tse ling. Ho ipapisitsoe le mokhoa oa ho etsa setšoantšo sa polarimetric o bontšitsoeng ho Eq. (10), re theha setšoantšo sa photometric.

Mokgwa o sisintsweng

NeRSP ea rona e nka tse ngata tse ngataview litšoantšo tse polarized, mask a silhouette e tsamaellanang ea ntho e shebiloeng, le kh'amera e ema joalo ka ho kenya le ho hlahisa sebopeho sa bokaholimo ba ntho e emetsoeng ka mokhoa o hlakileng ka SDF. Re qala ka lipuisano tse mabapi le li-photometric cues le li-geometric cues ho rarolla bothata ba ho tsosolosa sebopeho, ho lateloa ke taelo ea sebopeho sa marang-rang le mosebetsi oa tahlehelo oa NRSP ea rona.

Ho se utloisisehe kahong e nyane ea 3D
Leqheka la geometric le cue ea photometric li bapala karolo ea bohlokoa ho fokotsa sebaka sa tharollo ea sebopeho sa bokaholimo ka tlase ho sekoti. views. Joalokaha ho bontšitsoe setšoantšong sa 3, re bontša khakanyo ea sebopeho tlas'a 2 views e nang le matšoao a fapaneng. Ho fanoe feela ka litšoantšo tsa RGB e le ho kenya letsoho (tse tsamaellanang le maemo a NeRO [19] le S-VolSDF [35]), mefuta e fapaneng ea maemo a sebaka sa liketsahalo, maemo a holimo, le thepa ea ho bonahatsa e kang albedo e ka lebisa tlhokomelong e tšoanang ea setšoantšo, kaha ho na le litekanyo tse peli feela tsa RGB bakeng sa lintlha ka 'ngoe tsa 3D haufi le lehlaseli la khamera. Ka li-vector tsa Stokes tse ntšitsoeng litšoantšong tsa polarized, setšoantšo sa photometric se tlisa litekanyo tse 6 bakeng sa ntlha e 'ngoe le e' ngoe ea 3D (Stokes vector e na le likarolo tse 3), e fokotsang bonkgetheng ba tloaelehileng ba holim'a metsi hore ba se ke ba lumellana le mokhoa oa ho theha setšoantšo sa polarimetric.

Ka lehlakoreng le leng, ho ipapisitsoe le limmapa tsa AoP1 ho tsoa litšoantšong tse polarized, re ka tseba ka mokhoa o ikhethileng sebaka se tloaelehileng ho fihlela ho sa hlaka ho π bakeng sa sebaka se seng le se seng se haufi le lehlaseli la khamera. Leha ho le joalo, e ntse e le taba e sa hlakang ho fumana sebaka seo lehlaseli la khamera le kopanang le sona ntle le ha karolo ea boraro view e fanoe [6]. Ka hona, ka tlase views setting (mohlala, 2 views setšoantšong sa 3), ho khetholla sebaka sa sebaka sa ketsahalo ho ipapisitse le leqheka la geometri kapa la photometric ho ntse ho sa hlaka.
Mokhoa oa rona o kopanya lintlha tsena tse peli tse nkiloeng litšoantšong tsa polarized. Joalo ka ha ho boletsoe karolong e ka tlaase-ho le letona ea setšoantšo sa 3, sebaka se nepahetseng sa sebaka sa sebaka se lokela ho ba le bokaholimo ba sona bo tloaelehileng mateanong a lihlopha tse tloaelehileng tsa bonkgetheng tse nkiloeng ho li-photometric le li-geometric cues. Joalo ka holimo ho tloaelehileng ho li-s tse fapanengamplintlha tse etelletsoeng pele li khethoa ka mokhoa o ikhethileng ke li-geometric cues, re ka tseba habonolo hore na ntlha e holim'a metsi ka thuso ea setšoantšo sa photometric. Ka tsela ena, re fokotsa sebaka sa tharollo ea ho aha bocha sebaka sa marang-rang se hlakileng.

NeRSP
Sebopeho sa marang-rang Joalokaha ho bontšitsoe setšoantšong sa 4, NERSP ea rona e sebelisa mohaho o tšoanang oa marang-rang le PANDORA [9] qalong e tsoang ho Ref-NeRF [30]. Bakeng sa lehlaseli la leseli le tsoang setsing sa khamera o le tataiso ea v, re sampLe ntlha e holim'a lehlaseli le sebaka sa maeto ti, sebaka sa eona se qotsitsoe ho xi = o + tiv. Ho latela tlhahiso ea molumo e sebelisoang ho NeRF [25], Stokes vector s(v) e hlokometseng e ka kopanngoa ke opacity ea molumo σi le li-vector tsa Stokes ho s.ampled lintlha hammoho le ray, ke hore

moo bolela phetisetso e bokelletsoeng ea joalo kaampntlha e etelletseng pele.

E susumetsoa ke mokhoa oa morao-rao oa neural 3D reconstruction NeuS [33], re fumana opacity ea molumo ho tsoa ho marang-rang a SDF hape re ntša bokaholimo bo tloaelehileng ho tsoa ho gradient ea SDF. Ho bala joalo(xi, v) ka kamplintlha tse etelletsoeng pele, re latela mohlala oa sebopeho sa setšoantšo sa polarimetric ho Eq. (10). Haholo-holo, mahlaseli a pharalletseng a Ld a amana le li-coefficients tsa phetiso ea albedo le Fresnel, tse ipapisitseng le maemo a sebaka empa li sa fetohe. view tataiso. Ka hona, re sebelisa marang-rang a mahlaseli a pharalletseng ho etsa 'mapa oa Ld ho tsoa ho likarolo tsa sebaka se seng le se seng sa ketsahalo. Lehlaseli le ikhethang la Ls le amana le lobe e ikhethang e khethiloeng ke view tsela, bokahodimo bo tlwaelehileng, le bokahodimo bo makgwakgwa. Ka hona, re sebelisa RoughnessNet ho bolela esale pele ho se be teng. Hammoho le khamera view tataiso le sebaka se boletsoeng esale pele se tloaelehileng, re hakanya khanya e khethehileng ea Ls ka mor'a ho theola mojule o kopantsoeng oa "positional encoding" o hlahisitsoeng ke Ref-NeRF [30]. Ho kopanya Ld le Ls, re tsosolosa vector e hlokometsoeng ea Stokes e latelang Eq. (10).

Mosebetsi oa tahlehelo
Tahlehelo ea photometric e hlalosoa e le sebaka sa L1 lipakeng tsa ˆs(v) le li-vector tsa Stokes s(v), ke hore,

moo V e bolelang mahlaseli ohle a khamera ka har'a limaske tsa ntho e fapaneng views. Bakeng sa tahlehelo ea geometri. re qala ka ho fumana 3D scene point x haufi le kh'amera ray v ho fihlela re ama bokaholimo ebe re fumana maemo a reriloeng a 2D-pixel libakeng tse fapaneng. views. Tahlehelo ea geometri e hlalosoa e thehiloe ho Eq. (6), ke hore,

moo X e bolelang mateano 'ohle a marang-rang a marang-rang ka har'a limaske tsa ntho e fapaneng views. Ntle le tahlehelo ea photometric le geometric, re eketsa tahlehelo ea mask e laoloang ke limaske tsa ntho le tahlehelo ea Eikonal ea kamehla. Ho lahleheloa ke mask ho hlalosoa e le

moo e emela maske a boletsoeng esale pele ho k-th khamera ray, eo boleng ba eona ba mask a GT bo hlalosoang e le Mk. BCE e emela tahlehelo ea binary cross-entropy.

moo ni,k e leng bokaholimo bo tloaelehileng bo tsoang ho marang-rang a SDF ho i-th sampled point along the k-th camera ray. NeRSP ea rona e laoloa ke ho kopanngoa ha mantsoe a ka holimo a tahlehelo, ke hore

moo λe, λm, le λp e leng li-coefficients bakeng sa mantsoe a tahlehelo a tsamaisanang.

Lethathamo la lintlha tsa RMVP3D
Ho lekola ka mokhoa o reriloeng, re hapa Multi-worldview Setšoantšo sa polarized meshes le meshes ea 'nete ea motheo. Setšoantšo sa 5 (ka ho le letšehali) se bontša mokhoa oa rona oa ho hapa, o kenyelletsang khamera ea polarimetric, FLIR BFS-U3-51S5PC-C, e nang le lense ea 12 mm le seporo sa potoloho. Re sebelisa OpenCV bakeng sa ho hlakola data e tala le ho fumana litšoantšo tsa mebala tse 1224 × 1024 tse nang le li-angles tsa polarizer ho 0, 45, 90, le 135 degrees. Nakong ea ho nkuoa ha data, re beha lintho tse shebiloeng bohareng ba seporo ebe re nka litšoantšo tse 60 ntho ka 'ngoe ka ho tsamaisa kh'amera ka letsoho. Re bokella lintho tse 4 e le lipakane: NTJA, FROG, TAU, le BALL, joalokaha ho bontšitsoe setšoantšong sa 5 (bohareng). Bakeng sa tlhahlobo ea lipalo, re sebelisa laser scanner Creaform HandySCAN BLACK e nepahetseng ea 0.01 mm ho fumana mesh ea 'nete ea motheo. Ho hokahanya mesh ho setšoantšo se hapiloeng views, re qala ka ho sebelisa PANDORA [9] ho hakanya sebopeho sa litšupiso re sebelisa tsohle tse teng views ebe o hokahanya mesh e hlahlobiloeng ho e hakantsoeng ka algorithm ea ICP [4]. Ntle le libopeho tsa 'nete tsa fatše le tse ngataview litšoantšo, re boetse re hapa 'mapa oa tikoloho re sebelisa kh'amera ea 360-degree THETA Z1, e ruisang molemo litlhahlobo tsa palo ea likhakanyo tsa khanya bakeng sa mesebetsi e amanang le neural inverse.

Liteko

Re lekola NeRSP ka liteko tse tharo: 1) papiso le tse ngata tse tengview Mekhoa ea kaho ea 3D ka bongata ho dataset ea maiketsetso; 2) boithuto ba ho ablation mabapi le tlatsetso ea mantsoe a tahlehelo ea jiometri le fotometric 3) litlhahlobo tsa boleng le palo ho li-dataset tsa lefats'e la nnete. Re boetse re fana ka BRDF le novel view e fella ka boitsebiso ba tlatsetso.

Li-datasets & Baseline
Lefapha la boitsebiso. Re lokisa lintlha tse peli tsa lefats'e la 'nete: dataset ea PAN-DORA [9] le RMVP3D ea rona e reriloeng, moo dataset ea PANDORA [9] e sebelisoang feela bakeng sa tlhahlobo ea boleng ha meshes ea 'nete ea motheo e sa fanoe. Re boetse re lokisetsa mefuta e mengata ea maiketsetsoview polarized image dataset SMVP3D with Mitsuba rendering engine [15], e nang le dintho tse 5 tse nang le sebaka se fapaneng le se bonahatsang ponahatso, jwalo ka ha se bontshitswe ho Setšoantšo sa 6. Lintho li bonesoa ke limmapa tsa tikoloho2 'me li hapuoe ke 6. viewe ajoa ka tšohanyetso ho potoloha lintho. Ntle le litšoantšo tse entsoeng ka polarized, re boetse re romella li-stokes vectors, limmapa tse tloaelehileng tsa GT, le limmapa tsa AoP bakeng sa ntho ka 'ngoe.

Baseline. Mosebetsi oa rona o rarolla mathata a mangataview Kaho e ncha ea 3D bakeng sa libaka tse khanyang tse ipapisitseng le litšoantšo tsa polarized. Ka hona, re khetha mekhoa ea morao-rao ea kaho ea 3D e shebileng libaka tse bonahatsang NeRO [19] le tse fokolang. views S-VolSDF [35]. Mekhoa e 'meli e kaholimo e ipapisitse le likenyelletso tsa setšoantšo sa RGB. Bakeng sa tse ngataview stereo ho ipapisitsoe le litšoantšo tse polarized, re khetha PANDORA [9] le MVAS [6] joalo ka metheo ea rona. NeRO [19] ha e hloke limaske tsa silhouette joalo ka ho kenya letsoho. Bakeng sa papiso e nepahetseng, re tlosa bokamorao ho litšoantšo tsa RGB tse nang le limaske tse lumellanang pele re kenya NeRO [19]. Ho bapisa mekhoa e fapaneng, re sebelisa sebaka sa Chamfer (CD) lipakeng tsa meshes e hakantsoeng le ea GT, le phoso e kholo ea angular (MAE) lipakeng tsa maemo a hakanyetsoang le a GT a fapaneng. views joalo ka metrics ea rona ea tlhahlobo.

Ho hlaphoheloa ha sebopeho ho dataset ea maiketsetso
Joalokaha ho bontšitsoe ho Lethathamo la 1, re akaretsa phoso ea khakanyo ea sebopeho ea mekhoa e teng le ea rona ho SMVP3D. Mokhoa oa rona o fihlella sebaka se senyenyane sa Chamfer haufi le lintho tsohle tse 5 tsa maiketsetso. Ho ipapisitsoe le likhakanyo tse bonts'itsoeng tsa sebopeho se bonts'itsoeng setšoantšong sa 7, NeRO [19] le S-VolSDF [35] li ke ke tsa fumana lintlha tsa bokaholimo ka nepo joalo ka ha li totobalitsoe ho koaletsoeng. views. Lebaka le leng le ka 'nang la etsahala ke hore ho arohana ha sebopeho le ho bonahatsa litšoantšo ho tloha litšoantšong tse fokolang ho thata haholo bakeng sa mekhoa ena e thehiloeng boitsebisong ba RGB feela. MVAS [6] le PANDORA [9] li bua ka lintlha tsa geometric le photometric tsa litšoantšo tsa polarized, ka thoko. Leha ho le joalo, libōpeho tse entsoeng bocha tse khanyang li ntse li sa khotsofatse ka lebaka la ho se hlaka ha litšoantšo tsa geometri le photometric tlas'a sparse. views ho beha. Joalo ka ha ho totobalitsoe ho koetsoeng-up views, ho rua molemo ho li-geometrical le li-photometric cues, mokhoa oa rona o fokotsa sebaka sa tharollo ea khakanyo ea sebopeho, e lebisang ho tsosoloso e utloahalang ka ho fetisisa ha e bapisoa le libopeho tsa GT.

Ntle le tlhahlobo ea mesh e entsoeng bocha, re boetse re leka liphetho tsa likhakanyo tse tloaelehileng. Joalo ka ha ho bonts'itsoe ho Lethathamo la 2, re akaretsa liphoso tse kholo tsa maemo a hakanyetsoang ho 6 views ho tsoa mekhoeng e fapaneng. Tumellanong le liphetho tsa tlhahlobo ho Lethathamo la 1, NeRSP e fihlela liphoso tse nyane ka ho fetesisa tsa angular ka karolelano. Hape re hlokometse hore liphetho tse tsoang ho NeRO [19], MVAS [6], le PANDORA [9] li na le liphoso tse kholoanyane ho lintho tse nang le lintlha tse ntle, tse kang DAVID le DRAGON lintho. Joalo ka mohlalaample, MVAS [6] e na le sebaka sa bobeli se senyenyane sa Chamfer se bontšitsoeng ho Lethathamo la 1, empa phoso e kaalo ea angular e feta 20◦. Lebaka le leng le ka bang teng ke mekhoa e teng e hlahisang libopeho tse boreleli ka har'a sparse views, moo lintlha tse holimo tse kang li-flakes tsa DRAGON li sa fumaneheng hantle.

Letlapa la 1. Papiso ea ho hlaphoheloa ha sebopeho ho dataset ea maiketsetso e hlahlojoang ke sebaka sa Chamfer (↓). Liphoso tse nyane haholo le tsa bobeli li ngotsoe ka mongolo o motenya le ho thallisoa. "N/A" e bolela teko moo mokhoa o itseng o sa khoneng ho hlahisa sephetho se nepahetseng sa khakanyo ea sebopeho.

Thuto ea Ablation
Karolong ena, re etsa boithuto ba ho fokotsa chelete ho lekola katleho ea maqheka a geometric le photometric. Ho nka ntho ea DRAGON joalo ka example, re tsamaisa mokhoa oa rona le ntle le tahlehelo ea photometric Lp le tahlehelo ea geometri Lg. Joalokaha ho bontšitsoe setšoantšong sa 8, re rera sebopeho le likhakanyo tse tloaelehileng tsa holim'a metsi ka ho thibela mantsoe a fapaneng a tahlehelo. Ntle le tahlehelo ea photometric, ho se hlaka ha sebopeho ka lebaka la ho fokola views etsahala. Joalokaha ho bontšitsoe ho tloha ho koetsoeng views, sebopeho se haufi le karolo ea leoto se na le maiketsetso a concave, kaha ho na le tse peli feela tse bonahalang views sebakeng sena, ha e khone ho etsa tharollo e ikhethang bakeng sa sebopeho se ipapisitseng le limmapa tsa AoP [6]. Ntle le tahlehelo ea jiometri, re boetse re fumana liphetho tse sothehileng tsa sebopeho kaha litšoantšo tse fokolang ha li lekane ho senya sebopeho, ponahalo le khanya. Ka ho kopanya tahlehelo ea photometric le geo-metric, NeRSP ea rona e fokotsa ho se utloisisehe ha ho koahela sebopeho hape 'me sebopeho se hakantsoeng se haufi le GT, joalo ka ha ho totobalitsoe ho koaletsoeng. views.

Setšoantšo sa 8. Thuto ea ho fokotsa chelete ka lipehelo tse fapaneng tsa tahlehelo. Mela e ka holimo le e ka tlaase e bona ka mahlo a kelello sebopeho se hakantsoeng le bokaholimo bo tloaelehileng, ka sebaka sa Chamfer le phoso e bolelang ea angular e ngotsoeng ka holimo ho setšoantšo se seng le se seng, ka ho latellana.

Ho hlaphoheloa ha sebopeho ho data ea 'nete
Ntle le liteko tsa maiketsetso tse bontšitsoeng karolong e fetileng, re boetse re lekola mokhoa oa rona ho li-dataset tsa lefats'e tsa PANDORA [9] le RMVP3D ho lekola hore na e sebetsa joang maemong a kaho bocha a 3D lefatšeng ka bophara.

Tlhahlobo ea boleng ho dataset ea PANDORA [9]. Joalokaha ho bontšitsoe setšoantšong sa 9, re fana ka litlhahlobo tsa boleng ba dataset ea PAN-DORA [9]. Ha ho bapisoa le ponahalo ea setšoantšo le liphetho tse hakantsoeng tse tsoang ho S-VolSDF [35] le NeRO [19], sebopeho ha sea arohana ka botlalo ho tloha ponaletsong, se lebisang ho libopeho tse makukuno tse amanang haufi-ufi le sebopeho sa ponahalo. MVAS [6] le PANDORA [9] li na le likhakanyo tsa sebopeho se boreleli ho feta kapa li-artifacts tsa sebopeho sa concave, ka lebaka la ho sebetsana le maqheka feela a geometri kapa a photometric tlas'a tlhophiso ea ho hapa e fokolang. Liphetho tsa rona tsa likhakanyo tsa sebopeho ha li na libopeho tse joalo tsa maiketsetso 'me li tsamaisana le litebello tsa litšoantšo haufi-ufi.

Lethathamo la 3. Tekolo ea lipalo ho RMVP3D ka Chamfer dis-tance (↓). Mokhoa oa rona o finyella phoso e nyenyane ka ho fetisisa ka karolelano.

Mokhoa	NTJA	TAUA	SEKHOHANE	POLO	Karolelano
NeRO [19]	9.11	10.74	6.21	3.87	7.48
S-VolSDF [35]	9.93	7.39	7.91	18.4	10.91
MVAS [6]	9.23	7.51	9.90	4.77	7.86
PANDORA [9]	14.3	15.04	11.27	3.96	11.14
NeRSP (ea Rona)	8.80	5.18	6.70	3.84	6.13

Tekolo ea palo ho RMVP3D. Joalo ka ha ho bonts'itsoe ho Lethathamo la 3, re fana ka tlhahlobo ea palo ea RMVP3D e ipapisitseng le sebaka sa Chamfer. Tumellanong le liteko tsa maiketsetso, NeRSP ea rona e fihlela phoso e nyane haholo ea khakanyo ka kakaretso. Libopeho tse bonts'itsoeng tse bontšitsoeng setšoantšong sa 10 li senola hape hore libaka tse bonahatsang li thata ho S-VolSDF [35] bakeng sa ho arola sebopeho ho tloha ponahalong, joalo ka ha ho totobatsoa ke bokaholimo ba ntho ea FROG e koetsoeng. views. NeRO [19] le PANDORA [9] li na le liphoso tsa likhakanyo tse tšoanang le rona ho ntho e bonolo ea BOLO. Bakeng sa libopeho tse rarahaneng tse kang LION, ho hlaphoheloa ha sebopeho se sothehileng ho fumanoa mekhoeng ena ka lebaka la ho fokola view tlhophiso, ha ea rona e le haufi le li-meshes tsa GT, tse bonts'ang katleho ea mokhoa oa rona oa kaho ea lefatše ea 'nete e bonahatsang holim'a lintho tse kenang.

Qetello

Re sisinya NeRSP, mokhoa oa ho aha bocha oa neural 3D bakeng sa libaka tse bonahatsang tlas'a litšoantšo tse pepenene. Ka lebaka la liqholotso tsa ho se hlaka ha sebopeho sa mahlaseli le ponahatso e rarahaneng, mekhoa e teng e thatafalloa ke ho ba le bokaholimo kapa bo fokolang. views mme ha e khone ho rarolla mathata ka bobeli ka litšoantšo tsa RGB. Re sisinya ho sebelisa litšoantšo tsa polarized e le ho kenya letsoho. Ka ho kopanya li-geometric le li-photometric cues tse nkiloeng litšoantšong tsa polarized, re fokotsa sebaka sa tharollo ea sebopeho se hakantsoeng, e leng se lumellang ho hlaphoheloa hantle ha sebaka se khanyang ka tse seng kae joaloka 6. views, joalo ka ha ho bonts'itsoe ke tse fumanehang phatlalatsa le li-database tsa rona.

Moedi
The inter-reflections le polarized tikoloho-lebone ha ho nkoa e le mosebetsing ona, e leng se ka susumetsa ho nepahala ha kaho ea sebopeho. Re hlokometse mosebetsi oa morao-rao oa NeISF [17] o shebaneng le sehlooho sena, 'me re thahasella ho kopanya melemo ea rona e fokolang le mosebetsi ona nakong e tlang.
Kananelo
Mosebetsi ona o ne o tšehetsoa ke Beijing Natural Science Foundation Project No. Z200002, National Nature Science Foundation ea Chaena (Grant No. 62136001, 62088102, 62225601, U23B2052), Youth Innovative Research Team ea BUPT No. (Thuto No. JP2023K02 le JP22H17910). Re leboha Youwei Lyu ka lipuisano tse nang le temohisiso.

Litšupiso

Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang, Rui Chen, Yiqin Wang, Lei Yan, Boxin Shi, le Achuta Kadambi. Sebopeho se tebileng ho tloha polarization. Ho ECCV, leqepheng la 554–571, 2020. 2
Seung-Hwan Baek, Daniel S Jeon, Xin Tong, le Min H Kim. Ho fumanoa ka nako e le 'ngoe ha polarimetric SVBRDF le mekhoa e tloaelehileng. ACM TOG, 37(6):268–1, 2018. 2, 3, 4

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, le Pratul P Srinivasan. Mip-NeRF: Boemeli ba mefuta e mengata bakeng sa masimo a mahlaseli a neural a anti-aliasing. Ho ICCV, leqepheng la 5855–5864, 2021. 2
Paul J Besl le Neil D McKay. Mokhoa oa ho ngolisa libopeho tsa 3-D. Ho Sensor fusion IV: laola paradigms le data structures, maqephe 586–606, 1992. 6
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, le Hendrik Lensch. Neural-PIL: Lebone la Neural pre-integrated bakeng sa ho bola ha ho bonahatsa. Ho NeurIPS, maqephe 10691–10704, 2021. 1, 2

Xu Cao, Hiroaki Santo, Fumio Okura, and Yasuyuki Matsushita. Multi-View Azimuth Stereo ka Tangent Space Consistency. Ho CVPR, leqepheng la 825–834, 2023. 2, 3, 4, 6, 7, 8
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, le Hao Su. MVSNERF: Kaho e potlakileng ea mabone a mabone a akaretsang ho tsoa ho mefuta e mengata.view setirio. Ho CVPR, maqephe 14124–14133, 2021. 2
Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, le Jan Kautz. Polarimetric multi-view setirio. Ho CVPR, maqephe 1558–1567, 2017. 2

Akshat Dave, Yongyi Zhao, le Ashok Veeraraghavan. Pandora: Polarization-aid neural decomposition of radiation. Ho ECCV, leqepheng la 538–556, 2022. 1, 2, 4, 6, 7, 8
Kangle Deng, Andrew Liu, Jun-Yan Zhu, le Deva Ra-manan. NeRF e hlokometsoeng ka botebo: E fokolang views le koetliso e potlakileng mahala. Ho CVPR, maqephe 12882–12891, 2022. 2
Yuqi Ding, Yu Ji, Mingyuan Zhou, Sing Bing Kang, le Jin-wei Ye. Polarimetric helmholtz stereopsis. Ho ICCV, maqephe 5037–5046, 2021. 2

Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino. Polarimetric e tloaelehileng ea stereo. Ho CVPR, leqepheng la 682–690, 2021. 2
Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, le Ying-Cong Chen. Ref-NeuS: Ambiguity-Reduced Neural Implicit Sur-face Learning for Multi-View Ho tsosolosa ka Boikutlo. arXiv preprint arXiv:2303.10840, 2023. 1
Richard Hartley le Andrew Zisserman. Tse ngatanyana view geometry ponong ea k'homphieutha. Cambridge University Press, 2003. 1

Wenzel Jacob. Mofani oa Mitsuba, 2010. 6
Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, le Qifeng Chen. Sebopeho sa polarization bakeng sa liketsahalo tse rarahaneng naheng. Ho CVPR, maqephe 12632–12641, 2022. 2
Chenhao Li, Taishi Ono, Takeshi Uemori, Hajime Mihara, Alexander Gatto, Hajime Nagahara, and Yuseke Moriuchi. NEISF: Neural Incident Stokes Field for Geometry le Material Estimation. arXiv preprint arXiv:2311.13187, 2023. 8

Zhaoshuo Li, Thomas M¨uller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, le Chen-Hsuan Lin. Neu-Colangelo: High-Fidelity Neural Surface Reconstruction. Ho CVPR, maqephe 8456–8465, 2023. 2
Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, le Wenping Wang. NeRO: Neural Geometry le BRDF Reconstruction of Reflective Objects from Multiview Litšoantšo. arXiv preprint arXiv:2305.17398, 2023. 1, 2, 4, 6, 7, 8
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, le Wenping Wang. SparseNeuS: Kaho e potlakileng ea neural surface reconstruction ho tloha ho sparse views. Ho ECCV, leqepheng la 210–227, 2022. 2

Youwei Lyu, Lingran Zhao, Si Li, le Boxin Shi. Sebopeho sa polarization le khakanyo ea mabone a hole. IEEE TPAMI, 2023. 2
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, le Ren Ng. NeRF: E emelang liketsahalo e le masimo a mahlaseli a neural bakeng sa view tlhaho. Ho ECCV, leqepheng la 405–421, 2020. 1, 2
Miyazaki, Tan, Hara, le Ikeuchi. Phetolelo e fapaneng e thehiloeng ho polarization ho tsoa ho e le 'ngoe view. Ho ICCV, leqepheng la 982–987, 2003. 2

Michael Niemeyer, Lars Mescheder, Michael Oechsle, le Andreas Geiger. Tlhahiso e fapaneng ea volumetric: Ho ithuta litšoantšo tse hlakileng tsa 3D ntle le tlhokomelo ea 3D. Ho CVPR, maqephe 3504–3515, 2020. 2
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, le Noha Radwan. Reg-nerf: Ho laola libaka tsa mahlaseli a neural bakeng sa view kopanyo ho tsoa ho tse kenang tse fokolang. Ho CVPR, maqephe 5480–5490, 2022. 2, 4
Michael Oechsle, Songyou Peng, le Andreas Geiger. UNISURF: Ho kopanya libaka tse hlakileng tsa neural le libaka tse khanyang bakeng sa mefuta e mengata.view kaho bocha. Ho ICCV, leqepheng la 5589–5599, 2021. 2

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, le Steven Lovegrove. DeepSDF: Ho ithuta mesebetsi e saenneng e saenneng bakeng sa boemeli ba sebopeho. Ho CVPR, maqephe 165–174, 2019. 1
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, le Gordon Wetzstein. Lits'oants'o tse hlakileng tsa neural tse nang le ts'ebetso ea nako le nako. Ho NeurIPS, 2020. 1
William AP Smith, Ravi Ramamoorthi, le Silvia Tozza. Bophahamo ho tloha ho polarisation ka mabone a sa tsejoeng kapa albedo. IEEE TPAMI, 41(12):2875–2888, 2018. 2

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, le Pratul P Srinivasan. Ref-NeRF: E hlophisitsoeng view- ponahalo e itšetlehileng bakeng sa masimo a mahlaseli a neural. Ho CVPR, maqephe 5481–5490, 2022. 4, 5
Bruce Walter, Stephen R Marschner, Hongsong Li, le Ken-neth E Torrance. Mefuta ea Microfacet bakeng sa refraction ka holim'a mafika. Ho Proceedings of the 18th Eurographics conference on Rendering Techniques, leqepheng la 195–206, 2007. 4
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, le Ziwei Liu. SparseNeRF: Boemo bo tebileng bo tebileng bakeng sa nova e thunngoeng hanyane view tlhaho. arXiv preprint arXiv:2303.16196, 2023. 2

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, le Wenping Wang. NeuS: Ho Ithuta Neural Implicit Surfaces ka Volume Rendering for Multi-view Kaho bocha. arXiv preprint arXiv:2106.10689, 2021. 1, 2, 5
Yiqun Wang, Ivan Skorokhodov, Peter Wonka. HF-NeuS: Kaho e ntlafalitsoeng ea bokaholimo ho sebelisoa lintlha tsa maqhubu a holimo. Ho NeurIPS, maqephe 1966–1978, 2022. 2
Haoyu Wu, Alexandros Graikos, le Dimitris Samaras. S-VolSDF: Sparse Multi-View Regularization ea Stereo ea Neural Implicit Surfaces. arXiv preprint arXiv:2303.17712, 2023. 1, 2, 4, 6, 7, 8

Jiawei Yang, Marco Pavone, le Yue Wang. FreeNeRF: Ho Ntlafatsa Phano ea Neural e fokolang e fokolang e nang le Free Frequency Regularization. Ho CVPR, maqephe 8254–8263, 2023. 2
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, le Yaron Lipman. Tse ngataview neural surface reconstruction ka ho arola geometry le ponahalo. Ho NeurIPS, leqepheng la 2492–2502, 2020. 2
Lior Yariv, Jiatao Gu, Yoni Kasten, le Yaron Lipman. Phatlalatso ea molumo oa libaka tse sa bonahaleng tsa neural. Ho NeurIPS, maqephe 4805–4815, 2021. 1, 2

Alex Yu, Vickie Ye, Matthew Tancik, le Angjoo Kanazawa. pixelNeRF: Masimo a mahlaseli a Neural ho tsoa setšoantšong se le seng kapa tse 'maloa. Ho CVPR, maqephe 4578–4587, 2021. 2
Kai Zhang, Gernot Riegler, Noah Snavely, le Vladlen Koltun. NeRF++: Ho sekaseka le ho ntlafatsa masimo a mahlaseli a neural. arXiv preprint arXiv:2010.07492, 2020. 2
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, le Noah Snavely. PhySG: Phetolelo e fapaneng le li-Gaussia tse chitja bakeng sa ho hlophisa le ho khantša lintho tse thehiloeng ho fisiks. Ho CVPR, maqephe 5453–5462, 2021. 1, 2

Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul De-bevel, William T Freeman, le Jonathan T Barron. NeR-Factor: Neural factorization ea sebopeho le ho bonahatsa tlas'a khanya e sa tsejoeng. ACM TOG, 40(6):1–18, 2021. 2
Jinyu Zhao, Yusuke Monno, le Masatoshi Okutomi. Polarimetric multi-view phetiso e fapaneng. IEEE TPAMI, 2022. 2

Lintlha tsa lifoto le tsa jiometri tsa NeRSP

Tšebeliso ea sebopeho sa geometri
Joalokaha ho bontšitsoe setšoantšong sa S1, ho fanoe ka sebaka sa ketsahalo se hlokometsoeng ke ba fapaneng views, bokaholimo ba eona bo tloaelehile ho sepheo view e ka emeloa ke azimuth le li-angles tse phahameng ϕ le θ ka ho latellana, ke hore,

Kamano pakeng tsa angle azimuth le element ea bokaholimo bo tloaelehileng e ka etsoa joalo ka

Sebaka se tloaelehileng ho sepheo view e ka baloa ka ho potoloha e tloaelehileng mohloling view, ke hore ˆn = Rn. Ha ho fanoa ka matrix a potolohang ho tsoa ho khamera e lekantsoeng e le R = [r1, r2, r3]⊤, Eq. (2) e ipapisitseng le ˆ n e ka etsoa joalo ka

r⊤1 n cos ϕ − r⊤ 2 n sebe ϕ = 0. (3)

Ho latela MVAS [2], re ka hlophisa bocha Eq. (3) ho fumana kamano ea orthogonal pakeng tsa sepakapaka se tloaelehileng le se boleloang ke tangent vector t(ϕ) joalokaha ho hlalositsoe ka tlase,

Qeto ena ka angle ea azimuth e ka fetisetsoa ho angle of polarization (AoP). The ambiguity ea π e ka rarolloa ka tlhaho joalo ka Eq. (4) e eme haeba re kenyelletsa ϕ ka π. Ho se utloisisehe ha π/2 ho ka rarolloa ka ho sebelisa vector ea pseudo-projected tangent tˆ(ϕ) hore

Haeba setšoantšo se le seng ntlha x e bonoa ke f views, re ka beha Eq. (4) le Eq. (5) e ipapisitse le lipotoloho tse fapaneng le ho shebella li-AoPs, tse lebisang tsamaisong ea mela

T(x)n(x) = 0. (6)

Re nka sistimi ena ea mela e le sesupo sa rona sa jiometri bakeng sa mefuta e mengata.view polarized 3D kaho bocha.

Ho hlahisoa ha setšoantšo sa photometric
Ho latela polarized BRDF model [1], vector ea stokes e ka aroloa ka likarolo tse fapaneng le tse ikhethang tse entsoeng ka Hd le Hs ka ho tšoana, ke hore,

Karolo ea diffuse stokes tlasa lebone le le leng e ka etsoa joalo ka

moo ρd e bolelang diffuse albedo, ϕn ke angle azimuth ea leseli la ketsahalo sefofaneng se pota-potileng holimo ho tloaelehileng, T+i,o le T−i,o e bolelang lipalo tsa li-coefficients tsa Fresnel [1] tse amanang le angle lipakeng view tataiso le bokahodimo bo tlwaelehileng. Ho latela maikutlo a PANDORA [3], re ngola bocha vector ea li-stoke tlas'a leseli la tikoloho joalo ka

moo e hlalosoa e le khanya e pharalletseng. Sebakeng sa ho bala ho tsoa ho equation, mahlaseli a pharalletseng joalo ka phetoho e fapaneng ea sebaka e entsoe 'mapa ka kotloloho ho tsoa karolong ea neural e ntšitsoeng ke MLP e thehiloeng ho coordinate. Ka lehlakoreng le leng, vector ea specular stokes tlas'a leseli le le leng ω mofuteng oa polarimetric BRDF e ka hlalosoa e le.

moo ρs e bolelang albedo e ikhethang; D le G li bolela lentsoe le tloaelehileng la kabo le moriti ka har'a mohlala oa Microfacet [8], o ka laoloang ke ho hlaka holimo; R+ le R− li bolela lipalo tsa li-coefficients tsa Fresnel reflection [1], tse amanang le angle pakeng tsa sebaka se tloaelehileng le tataiso ea leseli la ketsahalo; ϕh ke ketsahalo ea azimuth angle e nang le halofo ea vector. Ho latela maikutlo a PANDORA [3], re ngola bocha vector ea specular stokes tlasa leseli la tikoloho joalo ka

moo e bolela kganya e ikgethang. Ka khakanyo ea spilt-sum [5], re ka tsoela pele ho hakanya Ls ≈ ρsDG/4n⊤v *fΩ L(ω) dω. Ho kopanngoa le vector ea diffuse stokes e bontšitsoeng ho Eq. (9), re haha cue ea photometric ho latela mohlala o latelang oa ho etsa setšoantšo sa polarimetric

Lintlha tsa Phethahatso

Karolo ena e fana ka lintlha tsa phetolelo ea Synthetic Multi-view Polarized image dataset SMVP3D le lintlha tsa koetliso tsa NeRSP.

Lefapha la boitsebiso
Re fana ka SMVP3D, e nang le litšoantšo tsa lintho tse hlano tsa maiketsetso tse bonahatsang tlas'a khanya ea tlhaho. Bakeng sa ntho ka 'ngoe, re fana ka 48 views le ho rekota 'nete ea motheo e tsamaellanang (GT) holim'a limmapa tse tloaelehileng. Re sebelisa Mit-suba3 [4] joalo ka enjine ea ho fetolela, e nang le mofuta oa BRDF o hlophiselitsoeng ho lisebelisoa tsa polasetiki tse polarized ha re fetolela. Bakeng sa dif-fuse albedo ρd, re sebelisa sebopeho se fapaneng sa sebaka sa albedo ho ntlafatsa bonnete ba liphetho tsa rona tsa tlhahiso. Ka nako e ts'oanang, re boloka specular albedo ρs ka boleng bo sa fetoheng ba 1.0 mme re beha bohloeki ba bokaholimo ho 0.05. Mokhoa ona o etsa bonnete ba hore o na le mokhoa o ts'oanang oa ho bonahatsa lintho. Litšoantšo tse hlahisoang ke polarized li fanoa ka qeto ea 512 × 512 pixels.

Koetliso
Li-hyperparameters λg, λm, le λe ts'ebetsong ea rona ea tahlehelo li behiloe ho 1, 1, le 0.1, ka ho latellana. Nakong ea ts'ebetso ea koetliso, re sebelisa leano la ho futhumatsa ho latela PAN-DORA [3], moo bakeng sa nako ea pele ea 1, re nahanang feela ka boitsebiso bo sa tsitsang ka setšoantšo sa photometric mme re nka hore karolo e ikhethang ea ntho ke 000. Ka kakaretso. liteko, re sebelisa qeto ea 0 × 512 bakeng sa koetliso le liteko ho SMVP512D, le 3 × 512 bakeng sa li-datasets tsa lefatše la sebele. Mokhoa oa rona ka kakaretso o fetoha nako e ka bang 612, 100, e nkang lihora tse 000 ho Nvidia RTX 6 GPU, 'me mohopolo o ja ho pota 3090, 8 MB.

Khakanyo ea BRDF le ho fana ka sephetho bocha

Setšoantšo sa S4 (holimo) se hlahisa khakanyo ea rona ea likarolo tse thata, tse fapaneng le tse ikhethang. Likhakanyo li batla li le lerata ka lebaka la 6 feela views. Joalo ka Ref-NeRF [7] moo leseli le laoloang ka mokhoa o hlakileng ka IDE, re ke ke ra etsa liteko tsa ho khantša. Ka hona, re bontša buka ena view synthesis liphetho ho e-na, joalokaha visualized ho Feiga. S4 (ka tlaase). Ha ho bapisoa le mekhoa e teng, litšoantšo tsa rona tsa ho fetolela bocha li haufi le litebello tse tsamaellanang tsa lefatše la 'nete.

Liphetho tse ling ho li-dataset tsa rona

Karolong ena, re fana ka liphetho tse ling tsa ntlafatso ea sebopeho ho SMVP3D le Real-world Multi-view Setšoantšo sa polarized RMVP3D.

Tekolo ka SMVP3D
Re fana ka liphello tsa boleng bo bocha ba mekhoa ea mekhoa ea motheo le mokhoa oa rona ho Fig. S2. Liphetho tse tsoang ho MVAS [2] ha li na lintlha, kaha setšoantšo sa photometric ha se eloe hloko. Le hoja NeRO [6] e fana ka ntlafatso ea sebopeho se ntlafetseng, e hlōleha ho fana ka sebaka se ka tšeptjoang bakeng sa lintho tse se nang sebopeho, tse kang DAVID. S-VolSDF [9] e sebelisa Multi-View Mokhoa oa stereo (MVS) mme o bonts'a kutloelo-bohloko e eketsehileng ho leseli la sebopeho holim'a lintho, tseo ka linako tse ling li lebisang ho tolokeng hampe ha lintlha tsa sebopeho e le likarolo tsa sebopeho. PANDORA [3] e na le bothata ba ho arola ka mokhoa o atlehileng litaba tsa albedo le tse ikhethang, tse lebisang ho liphetho tse sa tšepahaleng tsa ho aha bocha. Mokhoa oa rona, NeRSP, o sebelisa ka nepo li-cues tsa photometric le geometric, e leng se fellang ka ho tsosolosoa ho bonahatsang sebopeho sa GT ka nepo.

Re boetse re bonts'a likhakanyo tse tloaelehileng tsa bokaholimo le likhatiso tse tsamaellanang tsa liphoso ho Setšoantšo sa S3, se bonts'ang khafetsa hore NeRSP e fihlela sephetho se betere sa kaho ea sebopeho bakeng sa libaka tse bonahatsang tse nang le lintlha tse fokolang. views.

Tekolo ka RMVP3D
Karolong ena, re hlahisa sephetho se seng sa kaho bocha ho RMVP3D. Setšoantšo sa S5 se bontša hore NeRO [6], MVAS [2], le NeRSP li ka tsosolosa ka nepo ntho e bonolo e chitja e nang le bokaholimo bo khanyang. Ka lehlakoreng le leng, S-VolSDF [9] le PANDORA [3] ha li khone ho senya albedo le karolo e ikhethang ea holim'a metsi, e leng se bakang ho sotha ha mokhoa oa ho tsosolosa sebopeho. Ho khetholla har'a liphello tsa tsosoloso ea NeRO [6], MVAS [2], le NeRSP, re bona Chamfer Distance bakeng sa meshes e tsosolositsoeng ka mokhoa o mong le o mong. Joalokaha ho bontšitsoe setšoantšong sa S6, 'mala oa ntlha e' ngoe le e 'ngoe o bontša Chamfer Distance ea eona, e phunyeletsoeng pakeng tsa 0 le 5 mm. Lipapiso tsena li bontša hore phoso ea kaho e amanang le NeRSP e nyane ha e bapisoa le ea mekhoa e meng e 'meli.

Thuto ea Ablation holim'a ho bonahatsa bokaholimo

Mokhoa oa rona o ikemiselitse ho aha bocha bokaholimo, hape o ka sebelisoa ho khutlisa sebopeho ka libaka tse mahoashe. Joalo ka mohlalaample, re fetolela hape ntho ea SNAIL ka albedo ρs ea eona e ikhethang ho tloha ho 1.0 ho isa ho 0.1. Phoso e bolelang ea angular (MAE) ea sebaka se hakantsoeng se tloaelehileng ho 6 input views ho tsoa mekhoeng e fapaneng e bontšoa ho Lethathamo la S1. Tekolo ea boleng ba khakanyo e tloaelehileng ea bokaholimo le kabo ea liphoso tse tsamaisanang le tse ling tsa mekhoa e fapaneng tlasa kenyeletso e tšoanang. view li bonts'itsoe setšoantšong sa S7. Liteko tsena li bonts'a hore mekhoa e mengata e ntlafatsa boleng ba kaho bocha libakeng tse makukuno ha li bapisoa le libaka tse bonahatsang. Haholo-holo, mokhoa oa rona o fana ka mokhoa o tsitsitseng ka ho fetisisa oa ho tsosolosa holim'a ntho.

Boithuto ba ho fokotsa chelete ho #views

NeRSP ea rona e ikemiselitse ho aha bocha libaka tse khantšang tlas'a tlhahiso e fokolang views. Liteko tse bontšitsoeng pampiring e kholo li nka 6 tse fokolang views joalo ka ho kenya letsoho. Ho lekola mokhoa oa rona tlasa lipalo tse fapaneng tsa ho kenya views (ke hore, #views), re etsa liteko ka ntho ea sebele ea lefats'e LION tlasa maemo a 3, 6, 12, le 24. views. Setšoantšo sa S8 se bonts'a libopeho tse khutlisitsoeng, ha tlhahlobo ea boleng e nang le Chamfer Distance e hlahisoa ho Lethathamo la S2.

Tlas'a tlhahiso e fokolang views, joalo ka 3, mekhoa e teng e sokola ho fumana liphetho tse hlakileng. Sena se bakoa haholo ke hore li tsepamisitse maikutlo ho matšoao a photometric kapa matšoao a geometri. Ho nka S-VolSDF [9] joalo ka example, sebopeho se hakantsoeng, joalo ka ha se hlokometsoe haufi-ufi views, e susumetsoa haholo ke sebopeho se lumellanang. Sena se lebisa ho libopeho tse fosahetseng ka lebaka la ho se hlaka ha sebopeho-lehlaseli tlas'a ho fokola views. Ka ho sebetsana le lintlha tsa jiometri le li-photometric, NeRSP ea rona e fokotsa ho se hlaka ka har'a lintho tse kenyelelitsoeng tse fokolang. Ka lebaka leo, re fihlella kaho e utloahalang haholoanyane ea sebopeho. Tlhahiso ena e lula e sebetsa ha palo ea ho kenya views e feta 12. Joalokaha ho bontšitsoe ho Lethathamo la S2, NeRSP ea rona e lula e finyella Chamfer Distance e nyenyane ka ho fetisisa ka palo e ntseng e eketseha ea ho kenya letsoho. views. Sena se bontša katleho ea mokhoa oa rona holim'a libaka tse khanyang holim'a mefuta e mengata ea views.

Lethathamo la S2. Tlhahlobo ea boleng ho LION e lekantsoeng ka Chamfer Distance (↓) tlas'a kenyelletso e fapaneng views.

#Views	NeRO [6]	S-VolSDF [9]	MVAS [2]	PANDORA [3]	NeRSP
3	34.48	31.50	23.96	24.44	24.01
6	10.74	7.39	7.51	15.04	5.18
12	5.50	6.80	5.31	12.1	4.29
24	4.96	6.14	5.32	12.5	4.11

Tekolo ea dataset ea polarimetric MVIR

Ntle le liteko tsa lefats'e la 'nete ho dataset ea PANDORA [3] le RMVP3D ea rona, re boetse re fana ka tlhahlobo ea mefuta e mengata.view litšoantšo tse polarized dataset e teng ho PMVIR [10]. Joalokaha ho bontšitsoe setšoantšong sa S9, re bona ka mahlo a kelello liphello tsa ho hlaphoheloa ha sebopeho ho tloha PANDORA [3] le ea rona, ho nka 6 sparse. views joalo ka ho kenya letsoho. Kaha ha ho na sebopeho sa GT sebakeng sena sa data, re sebelisa liphetho tse tsoang ho PMVIR [10] joalo ka referense, e nkang 31 le 56. views e le ho kenya letsoho bakeng sa khamera le sebaka sa koloi, ka ho latellana. Rea hlokomela hore liphetho tsa rona li utloahala ha li bapisoa le tse sebelisang PANDORA [3], tse bonts'ang katleho ea mokhoa oa rona oa ho aha bocha 3D.