ArticleAn integrated imaging sensor foraberration-corrected 3D photographyhttps:/doi.org/10.1038/s41586-022-05306-8Received:9 November 2021Jiachen Xie,Lu Fang&Qionghai Dai2Accepted:1 September 2022Planar digitalimage sensors facilitate broadapplications in a wide rangeofareas'-5,andthenumber ofpixels has scaledup rapidly in recent years26.However,theOpen accesspractical perfor mance ofimaging systems is fundamentally limited by spatiallyCheck for updatesnonuniformopticalaberrationsoriginating from imperfect lensesor environmentaldisturbances?8.Here weproposeanintegrated scanning light-field imaging sensor,termed a meta-imaging sensor,to achieve high-speedaberration-correctedthree-dimensional photography for universalapplications withoutadditionalhardware modifications.Instead ofdirectly detecting atwo-dimensionalintensityprojection,the meta-imaging sensor captures extra-fine four-dimensionallight-fielddistributions through a vibrating coded microlensarray,enabling flexible and precisesynthesis ofcomplex-field-modulated images in post-processing.Using thesensor,we achievehigh-performance photographyup to agigapixel with asinglesphericallens withouta data prior,leading toorders-of-magnitudereductionsinsystemcapacity and costs for optical imaging.Even in the presenceofdynamic atmosphereturbulence,the meta-imaging sensor enables multisite aberration correction across1,000 arcseconds onan80-centimetreground-basedtelescope withoutreducing theacquisitionspeed,paving the way for high-resolution synopticsky surveys.Moreover,high-densityaccuratedepth mapscan be retrieved simultaneously,facilitatingdiverse applicationsfrom autonomous driving to industrialinspections.Two-dimensional (2D)imaging sensors have revolutionized manyhigh-performance incoherent imaging systems with large effectivefields,including industrialinspection,mobile devices,autonomousspace-bandwidthproductsare usuallyveryexpensive andbulky,suchaslarge-aperturetelescopes2andmesoscopes4.Metalens andfree-formBenefiting from the rapid development of semiconductor industry,optics maypotentially alleviatethis problem by fabricating optimizedthe number of pixels in digital sensors has grown quickly in the pastlens surfaces whengiven sufficient machiningprecisionacross a largescale.Image deblurringalgorithms mayimprove the imagecontrasttems has reached a bottleneck set by optics instead ofelectronics.viaaccurateestimationsofthe pointspreadfunction(PSF)1.PSFengi-For example,given a gigapixel sensor,the effective pixel number ofneering with acodedaperturepreservesmoreinformation byreducingan imaging system is usually limited to the megapixellevel,owing tooptical aberrations originatingfrom imperfect lenses or environmen-retrieve thehigh-frequency information lost byalow modulation transtal disturbances,which cause light emitted from one point to spreadferfunction(MTF),and these approachesusually require specific datapriors andprecise PSF estimations,especially for spatiallynonuniformthree-dimensional(3D)scenes to a 2D plane incurs the loss of variousaberrations".Moreover,all thesemethodsarestill sensitive todynamicfreedoms of the light field,such as depth and local coherence.Asaenvironmental aberrations with small depths of field.result,it has long been achallenge to obtainhigh-density depthmapsAdaptive optics achieves active aberration corrections with awith an integrated sensor.deformable mirror array or spatiallight modulator to direct the raysemitted from one point to the same position on the sensor at differ-ing perfect imaging systems for aberration correction with multipleprecision-engineered lenses in a sequential mode.However,the dif.and a wavefront sensor or by iterative updates according to specificficulty of optical design and fabrication increases exponentially withevaluation metrics2.Adaptive opticshasexhibitedgreatsuccess inthe space-bandwidth product,which describes the total number ofboth astronomy and microscopy and has contributed to importantdegreesof freedomsforanoptical system and setsan upper boundaryon the effective pixel number due to diffraction limits".In this case,the current adaptive optics approach is very small,owing to spatiallySchool Tsinghua University.Shenzhen,China.These authors contributed equally:Jiamin Wu,YuduoGuo,Chao Deng.e-mail:fanglu@tsinghua.edu.cn:ghdai@tsinghua.edu.cnNature www.nature.com 1Articlenonuniform aberrations?.Particularly for ground-based telescopes,through frequencyaliasing(Extended Data Fig.1a,b).By increasing thethe aberration caused by atmospheric turbulence limits the FOV ofsize ofmicrolensaperture,thediffraction effectwill graduallydecrease,adaptive optics to approximately 40 arcsecondsindiameter.which isleading to reduction of spatial resolution(Extended Data Fig.1c).Different froma previous scanning light-field technique,we furthercurrent adaptive optics systems are usually complicated,bulky andcoat achromium film withacircular pattem on eachmicrolens toblockexpensive,which makes the development of lightweight systems orthe light passing through the surrounding corners of the square foot-portable devices difficult.print ofeach microlens(see Methods and Fig 1a).Such acircular aperHere we propose anintegrated scanning light-field imaging frame.ture reduces the nulls intheoptical transfer functions for artefact-freework including both hardware and software,termed a meta-imagingreconstruction,whichisessential foruniversalimaging scenarios with-sensor,to achieve aberration-corrected 3D imaging with a largeout the sparse prior applied in fluorescence imaging(Extended Dataspace-bandwidth product at low cost.Akin to metasurfaces forFig.1d-g).Then we use high-speed periodic drifting of the microlensarray to increase the spatial sampling density limited by the physicalthemeta-imaging sensor facilitatesmeasurements and synthesesof thesize of the microlens,which can unmixthe frequency aliasing of ISAlight field in the spatial-angular domain at high speedwitha vibrating(Extended Data Fig.2a-f).After assembling the pixels with the samecoded microlens array,whichare muchmore precise thantraditionalangleaccordingtothemicrolens positionsinphase space(Fig.1b),welight-fieldtechniques,decoupling the optical modulation process fromapply a deconvolution-based ISAalgorithm to obtaina full-resolutiondata acquisition.We thenachievehigh-performance 3D imagingwithfocal stack or all-in-focus images with the extended depth of fieldmultisite aberration correction via wave-optics-based digital adaptiveand diffraction-limited resolution ofthe imaging lens(see Methodsoptics(DAO)on a single integrated sensor.By exploiting spatiotem-and Extended Data Fig.3a-f).Ahigh-density depthmapcan beretrievedporal continuity,wedevelop anoptical-flow-basedmotioncorrectionsimultaneously based on multiview stereo(see Methods).The meta-algorithmto preventmotion artefacts andmaintain theimagingspeedimaging sensor can directly replace conventionalimaging sensors(up to the camera frame rate).without additional hardware modifications(Fig.1c).Toestablishthe capabilities ofthemeta-imagingsensor,we conductMore importantly,with high-resolution plenoptic light distribu-quantitative analysiswith diverse applications in photography,autono.tions,we can generate complex-field-modulated images precisely inmous driving,industrial inspection,videosurveillance andastronomypost-processing without additional optical devices,which is the keySpecifically,we obtain high-performance all-in-focus images up toadvantage of the meta-imaging sensor over conventional 2D sensors.gigapixel with a single lens,indicating a three-orders-of-magnitudeA typical example is aberration correction,whichis a fundamentalreduction in system costs and capacity.Especially under condi-problem in optics.Imperfect lenses or turbulence disturb the raystions with severe nonuniform aberrations,the meta-imaging sensoremitted from the same point,leading to severe blurs in 2D sensors.achieves an over tenfold improvement in resolution.Moreover,theTheseblurs aredifficult to correct without anactivedeviceor accuratemeta-imagingsensor facilitates multisiteaberration corrections overaberration measurements(Fig.1d).By contrast,the meta-imaging1,000 arcseconds in diameter on an 80-cm-aperture telescope,pav-sensor is more robust to aberrations with reduced crosstalk betweening the way towards high-resolution ground-based synoptic surveys.angular components,which can be recombined into a perfect focusMegapixeldepth maps can beobtained simultaneously at millisecondduringpost-processing:thisis theDAOframework.Hereweextendthescale with better accuracy and resolution than traditional light-fieldconceptof DAOfrom geometricoptics towave optics,providing a morecameras for diverse industrial applications.accurate description of the imaging model(see Methods).The aber-rated phasecan beestimatedfrom therelativelateral shiftsof the samestructurein differentviews,which canbe used togenerateaccurate4DPrinciple of the meta-imaging sensorTraditional 2D sensorspose agreat challenge onopticsas aresult oftheperformance,we conduct numerical simulationswith different aberra-long-established design philosophy for the human retina-"what yoution levels(ExtendedDataFig.4a,b).Wave-optics-based DAOachievessee is what youget".Light-fieldimagingor plenopticimagingprovidesanapproximately10-dB peak signal-to-noise ratio improvementoveranothersolutionfrom machine visionby detectingfour-dimensionalgeometric-optics-basedDAO,withmore accurateestimationsof aber-(4D)spatial-angular informationas building blocks and synthesizingrated wavefronts.Compared to a hardware adaptive optics approachimages with arbitrary modulations during post-processing:272 thisbased ona deformable mirror array with15x 15 tip-tilt actuators,DAOapproachhasshowngreatpotentialin 3Dvision2,aberrationcorrectionobtains similar performance(Extended Data Fig.4c-e).Experimen-tal results on a resolution test chart further verify thesuperiority ofthe wave-optics model(Extended Data Fig 4f).In addition,the DAOsynthetic aperture (ISA),restricting their practical applications inframework exhibitsstrong noise robustness withconsistent aberrationestimation performance,because the correlation process utilizes allsmall aperture of each microlens,scanning light-field microscopysignals contained in a small region(Extended Data Fig.5a-g).bypasses the trade-off between spatial and angular resolutions withBy integrating the adaptive optics capability into the sensor,meta-imaging sensor can achieve multisite aberration correctionstype of microscopy requires additionaloptical systems and cannotwithout reducing the dataacquisition speed(Fig.leand Supplementarybe used for universalimaging.Tablel).Owingto the high nonuniformity ofaberrationdistributions,To address these problems,we propose a framework for thethe effectiveFOVs of traditional hardwareadaptiveoptics systems aremeta-imaging sensor by integrating a coded microlens array in ausually very small,leading to a considerable waste ofsensor pixels.high-speedpiezostage,whichisthen bonded to aconventional imagingsensorwith the photosensitive area located at the back focal plane ofcorrections forevery local regionacross awide FOV,whichisalsohardthe microlens array(Fig.1a).Each microlensfocusesthelight from dif-to obtain with traditional light cameras(Fig.1f).ferent angles onto different sensor pixels for angular sampling (Fig.1b,corresponding to 15 x 15 angular pixels).The aperture size of eachmicrolensis only about ten times of the diffraction limit of the imageRobust photography witha singlelensplane,whichintroduces adiffraction effect tothe incoherent light fieldThecost andsize of cameralensesgrow rapidlywiththe increaseof theand preservesthe high-frequency information during angle separationeffectivespace-bandwidth product,as they usuallyrequiremultiple well2 Nature www.nature.comPeriodic circular patternMeta sensorMicrolens arrayCMOSPixel realignment4D spatial-angular measurementsIncoherentMicrolens arayRaw datasynthetic apertureAAAAAAAAWithout aberrat ion2D sensor2D sensorMeta sensorA B CD E FAAG H IMeta sensoradaptive opticsADAOD E FG H IWith aberration2D sensorMeta sensorDigital adaptive opticsBD2D sensorTraditional light fieldMeta sensor DAO offMeta sensor DAO onFig.1|Principle of the integrated meta-imagingsensor.a,The metainside,as used in all the experiments.d,Optical aber ration disturbs the lightimagingsensor(meta sensor)integrates fourmainelements:a periodicemitted fromthe same point,resultinginblur ry 2D images orlateral shifts ofcircular pat tern with acircular intensitymask oneachmicrolens,a microlensdifferent angular com ponents.Althoughall the angularcomponents are mixedcoherently ina 2Dsensor they are separatedin themeta-imagingsensor andstageto vibratethe microlens ar ray periodicallyandincrease the spatialcanbe realigned duringpost-processingto recover the aberration-correctedsamplingdensityfor stateunmixing.anda conventional complementaryhigh-reso lutionimage:this technique is called digital adaptiveoptics(DAO).metal-oxide-semiconductor(CMOS)sensorplacedat the back focal plane ofe.For spatially nonunifor maber rations,a hardware adaptive optics systemthemicrolens array forhigh-throughput photondetection.b.The imagingwitha deformable mirror arraycanonly cor recta small FOV,whereas DAO canprincipleof themeta-imagingsensor.Light fromdifferentangles(labelledwithachieve multisite aberr ationcorrections simultaneouslywithout influencingdifferent colours)is focusedontodifferent pixels aftereach microlens.the data acquisition speed.f.Comparisons of the results yieldedby a singleHigh-resolution4D spatial-angularmeasurements areobtainedby combiningplastic lens witha 2D sensor,a traditionallight-field sensor and thepixels with the sameanglebased onthe microlens positions,whichcanbe usedmeta-imaging sensorwithandwithout DAO.The same type of CMOS chipwastogenerate complex-field-modulatedimages up tothe diffractionlimit of theused ineach test.imaginglens with ISA.c.The prototype camera withthe meta-imagingsensorNature www.nature.com 3