Optical Versus Video See-Through Head-Mounted Displays

SchoolofOptics/CREOL,and

SchoolofElectricalEngineeringandComputerScience

UniversityofCentralFloridaOrlando,FL32816–2700HenryFuchs

DepartmentofComputerScienceUniversityofNorthCarolinaChapelHill,NC27599-3175

Presence,Vol.9,No.3,June2000,287–309

r2000bytheMassachusettsInstituteofTechnology

OpticalVersusVideoSee-ThroughHead-MountedDisplaysinMedicalVisualization

Abstract

Wecomparetwotechnologicalapproachestoaugmentedrealityfor3-Dmedicalvisualization:opticalandvideosee-throughdevices.Weprovideacontexttodiscussthetechnologybyreviewingseveralmedicalapplicationsofaugmented-realityre-searcheffortsdrivenbyrealneedsinthemedical󰀷eld,bothintheUnitedStatesandinEurope.Wethendiscusstheissuesforeachapproach,opticalversusvideo,frombothatechnologyandhuman-factorpointofview.Finally,wepointtopotentiallypromisingfuturedevelopmentsofsuchdevicesincludingeyetrackingandmultifocusplanescapabilities,aswellashybridoptical/videotechnology.

1Introduction

Oneofthemostpromisingandchallengingfutureusesofhead-mounteddisplays(HMDs)isinapplicationsinwhichvirtualenvironmentsenhanceratherthanreplacerealenvironments.Thisisreferredtoasaugmentedreality(Bajura,Fuchs,&Ohbuchi,1992).Toobtainanenhancedviewoftherealen-vironment,userswearsee-throughHMDstosee3-Dcomputer-generatedob-jectssuperimposedontheirreal-worldview.Thissee-throughcapabilitycanbeaccomplishedusingeitheranopticalHMD,asshowninfigure1,oravideosee-throughHMD,asshowninfigure2.Weshalldiscussthetradeoffsbetweenopticalandvideosee-throughHMDswithrespecttotechnologicalandhu-man-factorissues,anddiscussourexperiencedesigning,building,andtestingtheseHMDsinmedicalvisualization.

Withopticalsee-throughHMDs,therealworldisseenthroughhalf-trans-parentmirrorsplacedinfrontoftheuser’seyes,asshowninfigure1.Thesemirrorsarealsousedtoreflectthecomputer-generatedimagesintotheuser’seyes,therebyopticallycombiningthereal-andvirtual-worldviews.Withavideosee-throughHMD,thereal-worldviewiscapturedwithtwominiaturevideocamerasmountedontheheadgear,asshowninfigure2,andthecom-puter-generatedimagesareelectronicallycombinedwiththevideorepresenta-tionoftherealworld(Edwards,Rolland,&Keller,1993;Stateetal.,1994).See-throughHMDswerefirstdevelopedinthe1960s.IvanSutherland’s1965and1968opticalsee-throughandstereoHMDswerethefirstcomputergraphics-basedHMDsthatusedminiatureCRTsfordisplaydevices,ame-chanicaltrackertoprovideheadpositionandorientationinrealtime,andahand-trackingdevice(Sutherland,1965,1968).Whilemostofthedevelop-mentsinsee-throughHMDsaimedatmilitaryapplications(Buchroeder,

RollandandFuchs287

288PRESENCE:VOLUME9,NUMBER3

Figure1.Opticalsee-throughhead-mounteddisplay(PhotocourtesyofKaiserElectro-Optics).

Figure2.Acustomopticsvideosee-throughhead-mounteddisplaydevelopedatUNC-CH.Edwardsetal.(1993)designedtheminiaturevideocameras.TheviewerwasalargeFOVopaqueHMDfromVirtualResearch.

Seeley,&Vukobratovich,1981;Furness,1986;

Droessler&Rotier,1990;Barrette,1992;Kandebo,1988;Desplat,1997),developmentsin3-Dscientificandmedicalvisualizationwereinitiatedinthe1980sattheUniversityofNorthCarolinaatChapelHill(Brooks,1992).

Inthispaper,weshallfirstreviewseveralmedicalvisu-alizationapplicationsdevelopedusingopticalandvideosee-throughtechnologies.Weshallthendiscusstechno-logicalandhuman-factorsandperceptualissuesrelatedtosee-throughdevices,someofwhichareemployedin

thevariousapplicationssurveyed.Finally,weshalldis-cusswhatthetechnologymayevolvetobecome.

2

SomePastandCurrentApplicationsofOpticalandVideoSee-ThroughHMDs

Theneedforaccuratevisualizationanddiagnosisinhealthcareiscrucial.Oneofthemaindevelopmentsofmedicalcarehasbeenimaging.SincethediscoveryofX-raysin15byWilhelmRoentgen,andthefirstX-rayclinicalapplicationayearlaterbytwoBirmingham(UK)doctors,X-rayimagingandothermedicalimagingmo-dalities(suchasCT,ultrasound,andNMR)haveemerged.Medicalimagingallowsdoctorstoviewas-pectsoftheinteriorarchitectureoflivingbeingsthatwereunseenbefore.Withtheadventofimagingtech-nologies,opportunitiesforminimallyinvasivesurgicalprocedureshavearisen.Imagingandvisualizationcanbeusedtoguideneedlebiopsy,laparoscopic,endoscopic,andcatheterprocedures.Suchproceduresdorequireadditionaltrainingbecausethephysicianscannotseethenaturalstructuresthatarevisibleinopensurgery.Forexample,thenaturaleye-handcoordinationisnotavail-ableduringlaparoscopicsurgery.Visualizationtech-niquesassociatedwithsee-throughHMDspromisetohelprestoresomeofthelostbenefitsofopensurgery(forexample,byprojectingavirtualimagedirectlyonthepatient,eliminatingtheneedforaremotemonitor).Thefollowingparagraphsbrieflydiscussexamplesofrecentandcurrentresearchconductedwithopticalsee-throughHMDsattheUniversityofNorthCarolinaatChapelHill(UNC-CH),theUniversityofCentralFlorida(UCF),andtheUnitedMedicalandDentalSchoolsofGuy’sandSaintThomas’sHospitalsinEn-gland,video-see-throughatUNC-CH,andhybridopti-cal-videosee-throughattheUniversityofBlaisePascalinFrance.

Arigorouserror-analysisforanopticalsee-throughHMDtargetedtowardtheapplicationofopticalsee-throughHMDtocraniofacialreconstructionwascon-ductedatUNC-CH(Holloway,1995).Thesuperimpo-sitionofCTskulldataontotheheadoftherealpatientwouldgivethesurgeons‘‘X-rayvision.’’Thepremiseof

Figure3.(a)TheVRDAtoolwillallowsuperimpositionofvirtualanatomyonamodelpatient.(b)AnillustrationoftheviewoftheHMDuser(CourtesyofAndreiState).(c)Arenderedframeoftheknee-jointbonestructuresanimatedbasedonakinematicmodelofmotiondevelopedbyBaillotandRollandthatwillbeintegratedinthetool(1998).

thatsystemwasthatviewingthedatainsituallowssur-geonstomakebettersurgicalplansbecausetheywillbeabletoseethecomplexrelationshipsbetweentheboneandsofttissuemoreclearly.Hollowayfoundthatthelargestregistrationerrorbetweenrealandvirtualobjectsinopticalsee-throughHMDswascausedbydelaysinpresentingupdatedinformationassociatedwithtrack-ing.ExtensiveresearchintrackinghasbeenpursuedsinceatUNC-CH(Welch&Bishop,1997).

Oneoftheauthorsandcolleaguesarecurrentlydevel-opinganaugmented-realitytoolforthevisualizationofhumananatomicaljointsinmotion(Wrightetal.,1995;Kancherlaetal.,1995;Rolland&Arthur,1997;Parsons&Rolland,1998;Baillot&Rolland,1998;Baillotetal.,1999).Anillustrationofthetoolusinganopticalsee-throughHMDforvisualizationofanatomyisshowninfigure3.Inthefirstprototype,wehaveconcentratedonRollandandFuchs2

Figure4.Firstdemonstrationofthesuperimpositionofagraphicalknee-jointsuperimposedonalegmodelforuseintheVRDAtool:(a)apictureofthebenchprototypesetup;asnapshotofthesuperimpositionthroughonelensofthesetupin(b)adiagonalviewand(c)asideview(1999).

thepositioningofthelegaroundthekneejoint.Thejointisaccuratelytrackedopticallybyusingthreeinfra-redvideocamerastolocateactiveinfraredmarkersplacedaroundthejoint.Figure4showstheresultsoftheopticalsuperimpositionofthegraphicalkneejointonalegmodel,seenthroughoneofthelensesofourstereoscopicbenchprototypedisplay.

Anopticalsee-throughHMDcoupledwithopticaltrackingdevicespositionedalongthekneejointofamodelpatientisusedtovisualizethe3-Dcomputer-renderedanatomydirectlysuperimposedontherealleginmotion.Theusermayfurthermanipulatethejointandinvestigatethejointmotions.Fromatechnologicalaspect,thefieldofview(FOV)oftheHMDshouldbesufficienttocapturetheknee-jointregion,andthetrack-ingdevicesandimage-generationsystemmustbefastenoughtotracktypicalknee-jointmotionsduringma-

290PRESENCE:VOLUME9,NUMBER3

nipulationatinteractivespeed.Thechallengeofcaptur-ingaccurateknee-jointmotionusingopticalmarkerslocatedontheexternalsurfaceofthejointwasaddressedbyRollandandArthur(1997).Theapplicationaimsatdevelopingamoreadvancedtoolforteachingdynamicanatomy(advancedinthesensethatthetoolallowscombinationofthesensesoftouchandvision).Weaimthistooltospecificallyimpartbetterunderstandingofbonemotionsduringradiographicpositioningfortheradiologicalscience(Wrightetal.,1995).

TosupporttheneedforaccuratemotionsofthekneejointintheVirtualRealityDynamicAnatomy(VRDA)tool,anaccuratekinematicmodelofjointmotionbasedonthegeometryofthebonesandcollisiondetectionalgorithmswasdeveloped(Baillot&Rolland,1998;Baillotetal.,1999).Thiscomponentoftheresearchisdescribedinanotherpaperofthisspecialissue(Baillotetal.,2000).Thedynamicregistrationofthelegwiththesimulatedbonesisreportedelsewhere(Outtersetal.,1999).High-accuracyopticaltrackingmethods,care-fullydesignedandcalibratedHMDtechnology,andap-propriatecomputergraphicsmodelsforstereopairgen-erationplayanimportantroleinachievingaccurateregistration(VaissieandRolland,2000;Rollandetal.,2000).

AttheUnitedMedicalandDentalSchoolsofGuy’sandSaintThomas’sHospitalsinEngland,researchersareprojectingsimpleimagefeaturesderivedfrompreop-erativemagneticresonanceandcomputer-tomographyimagesintothelightpathofastereooperatingmicro-scope,withthegoalofeventuallyallowingsurgeonstovisualizeunderlyingstructuresduringsurgery.Thefirstprototypeusedlow-contrastcolordisplays(Edwardsetal.,1995).Thecurrentprototypeuseshigh-contrastmonochromedisplays.Themicroscopeistrackedintra-operatively,andtheopticsarecalibrated(includingzoomandfocus)usingapinholecameramodel.Theintraoperativecoordinateframeisregisteredusingana-tomicalfeaturesandfiducialmarkers.Theimagefeaturesusedinthedisplayarecurrentlysegmentedbyhand.Theseincludetheoutlineofalesion,thetrackofkeynervesandbloodvessels,andbonelandmarks.Thiscomputer-guidedsurgerysystemcanbesaidtobe

Figure5.Real-timeacquisitionandsuperimpositionofultrasoundsliceimagesonapregnantwoman(1992).

equivalenttoanopticalsee-throughsystemoperatingonamicroscopicscale.Inthiscase,therealsceneisnowseenthroughmagnifyingoptics,buttheeyeoftheob-serverisstillthedirectdetectingdeviceasinopticalsee-through.

OneoftheauthorsandcolleaguesattheUNC-CHarecurrentlydevelopingtechniquesthatmergevideoandgraphicalimagesforaugmentedreality.Thegoalistodevelopasystemdisplayinglive,real-time,ultrasounddataproperlyregisteredin3-Dspaceonascannedsub-ject.Thiswouldbeapowerfulandintuitivevisualizationtoolaswell.Thefirstapplicationdevelopedwasthevisu-alizationofahumanfetusduringultrasoundechogra-phy.Figure5showsthereal-timeultrasoundimageswhichappeartobepastedinfrontofthepatient’sbody,ratherthanfixedwithinit(Bajuraetal.,1992).Real-timeimagingandvisualizationremainsachallenge.Fig-ure6showsamorerecent,non-real-timeimplementa-tionofthevisualizationinwhichthefetusisrenderedmoreconvincinglywithinthebody(Stateetal.,1994).Recently,knowledgefromthisvideoandultrasoundtechnologyhasalsobeenappliedtodevelopingavisual-izationmethodforultrasound-guidedbiopsiesofbreastlesionsthatweredetectedduringmammographyscreen-ingprocedures(Figure7)(Stateetal.,1996).Thisap-plicationwasmotivatedfromthechallengesweobservedduringabiopsyprocedurewhilecollaboratingonre-searchwithEttaPisano,headoftheMammographyRe-searchGroupatUNC-CH.Thegoalwastobeabletolocateanytumorwithinthebreastasquicklyandaccu-ratelyaspossible.Thetechnologyofvideosee-through

Figure6.Improvedrenderingoffetusinsidetheabdomen(1994).

Figure7.Ultrasoundguidedbiopsy(a)LaboratorysetupduringevaluationofthetechnologywithEttaPisanoandHenryFuchs(b)AviewthroughtheHMD(1996).

alreadydevelopedwasthusappliedtothisproblem.Theconventionalapproachtobiopsyistofollowtheinser-tionofaneedleinthebreasttissuewitharemotemoni-tordisplayingreal-time,2-D,ultrasounddepthimages.Suchaproceduretypicallyrequiresfiveinsertionsofthe

RollandandFuchs291

Figure8.Laboratoryprototypeofthehybridoptical/videosee-throughARtoolforguidedscoliosissurgerydevelopedbyPeuchotattheUniversityofBlaisePascal,France(1995).

needletomaximizethechancesofbiopsyofthelesion.Inthecaseinwhichthelesionislocatedfairlydeepinthebreasttissue,theprocedureisdifficultandcanbelengthy(onetotwohoursisnotatypicalfordeeple-sions).Severalchallengesremaintobeovercomebeforethetechnologydevelopedcanactuallybetestedintheclinic,includingaccurateandprecisetrackingandatechnicallyreliableHMD.Thetechnologymayhaveapplicationsinguidinglaparoscopy,endoscopy,orcath-eterizationaswell.

AttheUniversityofBlaisePascalinClermontFer-rand,France,researchersdevelopedseveralaugmented-realityvisualizationtoolsbasedonhybridopticalandvideosee-throughtoassistinsurgerytocorrectscoliosis(abnormalcurvatureofthespinecolumn)(Peuchot,Tanguy,&Eude,1994,1995).Thisapplicationwasde-velopedincollaborationwithasurgeonofinfantilesco-liosis.Thevisualizationsystemshowninfigure8isfromanopticspointofview,thesimplestsee-throughsystemonemayconceive.Itisfirstofallfixedonastand,anditisdesignedasaviewboxpositionedabovethepatient.

292PRESENCE:VOLUME9,NUMBER3

Figure9.Graphicsillustrationofcurrentandfutureuseofcomputer-guidedsurgeryaccordingtoBernardPeuchot.

Thesurgeonpositionshimselfabovetheviewboxtoseethepatient,andthegraphicalinformationissuperim-posedonthepatientasillustratedinfigure9.Thesys-temincludesalargemonitorwhereastereopairofim-agesisdisplayed,aswellashalf-silveredmirrorsthatallowthesuperimpositionoftherealandvirtualobjects.Themonitorisopticallyimagedonaplanethroughthesemi-transparentmirrors,andthespineundersurgeryislocatedwithinasmallvolumearoundthatplane.Anop-ticallayoutofthesystemisshowninfigure10.

Intheabovehybridoptical-videosystem,vertebraearelocatedinspacebyautomaticanalysisoftheperspec-tiveviewfromasinglevideocameraofthevertebrae.Astandardalgorithmsuchastheinverseperspectivealgo-rithmisusedtoextractthe3-Dinformationfromtheprojectionsobservedinthedetectorplane(Dhomeetal.,19).Themethodreliesheavilyonaccuratevideotrackingofvertebraldisplacements.High-accuracyalgo-rithmsweredevelopedtosupporttheapplicationinclud-ingdevelopmentofsubpixeldetectorsandcalibrationtechniques.Themethodhasbeenvalidatedonvertebral

Figure10.Opticalschemeofthehybrid

optical/videosee-throughARtoolshowninFig.8.

specimensandaccuracyofsubmillimetersindepthhasbeendemonstrated(Peuchot,1993,1994).

Thesuccessofthemethodcanbeattributedtothefinecalibrationofthesystem,which,contrarytomostsystems,doesnotassumeapinholecameramodelforthevideocamera.Moreover,havingafixedviewerwithnoopticalmagnification(contrarytotypicalHMDs)andaconstantaverageplaneofsurgicaloperationre-ducesthecomplexityofproblemssuchasregistrationandvisualization.Itcanbeshown,forexample,thatren-dereddeptherrorsareminimizedwhenthevirtualim-ageplanesthroughtheoptics(asimplesemi-transparentmirrorinPeuchot’scase)islocatedintheaverageplaneofthe3-Dvirtualobjectvisualized(Rollandetal.,

1995).Furthermore,thesystemavoidsthechallengingproblemsoftracking,opticaldistortioncompensation,andconflictsofaccommodationandconvergencere-latedtoHMDs(Robinett&Rolland,1992;Rolland&Hopkins,1993).Sometrackinganddistortionissueswillbefurtherdiscussedinsections3.1and3.2,respec-tively.However,goodregistrationofrealandvirtualobjectsinastaticframeworkisafirststeptogoodcali-brationinadynamicframework,andPeuchot’sresultsarestateoftheartinthisregard.

Itisimportanttonotethatthemethoddevelopedforthisapplicationemploysahybridoptical-videotechnol-ogy.Inthiscase,videoisessentiallyusedtolocalizerealobjectsinthesurgicalfield,andopticalsee-throughisusedasthevisualizationtoolforthesurgeon.Whilethefirstsystemdevelopedusedonevideocamera,themeth-odshavebeenextendedtoincludemultiplecameras

RollandandFuchs293

Figure11.Outlineofsections3.1and3.2ofthispaper.

withdemonstratedaccuracyandprecisionof0.01mm(Peuchot,1998).Peuchotchosethehybridsystemoveravideosee-throughapproachbecause‘‘itallowstheop-eratortoworkinhisrealenvironmentwithaperceptionspacethatisreal.’’Peuchotjudgedthispointtobecriti-calinamedicalapplicationlikesurgery.

3

AComparisonofOpticalandVideoSee-ThroughTechnology

Assuggestedinthedescriptionoftheapplicationsdescribed,themaingoalofaugmented-realitysystemsistomergevirtualobjectsintotheviewoftherealscenesothattheuser’svisualsystemsuspendsdisbeliefintoper-ceivingthevirtualobjectsaspartoftherealenviron-ment.Currentsystemsarefarfromperfect,andsystemdesignerstypicallyendupmakinganumberofapplica-tion-dependenttradeoffs.Weshalllistanddiscussthesetradeoffsinordertoguidethechoiceoftechnologyde-pendinguponthetypeofapplicationconsidered.Bothsystems,opticalandvideo,havetwoimagesources:therealworldandthecomputer-generatedworld.Theseimagesourcesaretobemerged.Opticalsee-throughHMDstakewhatmightbecalleda‘‘mini-mallyobtrusive’’approach;thatis,theyleavetheviewoftherealworldnearlyintactandattempttoaugmentitby

mergingareflectedimageofthecomputer-generatedsceneintotheviewoftherealworld.Videosee-throughHMDsaretypicallymoreobtrusiveinthesensethattheyblockoutthereal-worldviewinexchangefortheabilitytomergethetwoviewsmoreconvincingly.Inrecentdevelopments,narrowfieldsofviewinvideosee-throughHMDshavereplacedlargefield-of-viewHMDs,thusreducingtheareawheretherealworld(capturedthroughvideo)andthecomputer-generatedimagesaremergedintoasmallpartofthevisualscene.Inanycase,afundamentalconsiderationiswhethertheadditionalfeaturesaffordedbyvideosee-throughHMDsjustifythelossoftheunobstructedreal-worldview.

Ourexperienceindicatesthattherearemanytradeoffsbetweenopticalandvideosee-throughHMDswithre-specttotechnologicalandhuman-factorsissuesthataf-fectdesigning,building,andassessingtheseHMDs.Thespecificissuesarelaidoutinfigure11.Whilemostoftheseissuescouldbediscussedfrombothatechnologi-calandhuman-factors-standpoint(becausethetwoarecloselyinterrelatedinHMDsystems),wehavechosentoclassifyeachissuewhereitismostadequatelyaddressedatthistime,giventhepresentstateofthetechnology.Forexample,delaysinHMDsystemsareaddressedun-dertechnologybecausetechnologicalimprovementsareactivelybeingpursuedtominimizedelays.Delaysalso

294PRESENCE:VOLUME9,NUMBER3

certainlyhaveimpactonvarioushuman-factorissues(suchastheperceivedlocationofobjectsindepthanduseracceptance).Therefore,themultiplearrowsshowninfigure11indicatethatthetechnologicalandhuman-factor-categoriesarehighlyinterrelated.

3.1TechnologicalIssues

ThetechnologicalissuesforHMDsincludelatencyofthesystem,resolutionanddistortionoftherealscene,fieldofview(FOV),eyepointmatchingofthesee-throughdevice,andengineeringandcostfactors.Whileweshalldiscusspropertiesofbothopticalandvideosee-throughHMDs,itmustbenotedthat,contrarytoopti-calsee-throughHMDs,therearenocommerciallyavail-ableproductsforvideosee-throughHMDs.Therefore,discussionsofsuchsystemsshouldbeconsideredcare-fullyasfindingsmaybeparticulartoonlyafewcurrentsystems.Nevertheless,weshallprovideasmuchinsightaspossibleintowhatwehavelearnedwithsuchsystemsaswell.

3.1.1SystemLatency.Anessentialcomponentofsee-throughHMDsisthecapacitytoproperlyregisterauser’ssurroundingsandthesyntheticspace.Ageomet-riccalibrationbetweenthetrackingdevicesandtheHMDopticsmustbeperformed.Themajorimpedi-menttoachievingregistrationisthegapintime,re-ferredtoaslag,betweenthemomentwhentheHMDpositionismeasuredandthemomentwhenthesyn-theticimageforthatpositionisfullyrenderedandpre-sentedtotheuser.

LagisthelargestsourceofregistrationerrorinmostcurrentHMDsystems(Holloway,1995).Thislagintypicalsystemsisbetween60msand180ms.Theheadofausercanmoveduringsuchaperiodoftime,andthediscrepancyinperceivedsceneandsuperimposedscenecandestroytheillusionofthesyntheticobjectsbeingfixedintheenvironment.Thesyntheticobjectscan

‘‘swim’’aroundsignificantlyinsuchawaythattheymaynotevenseemtobepartoftherealobjecttowhichtheybelong.Forexample,inthecaseofultrasound-guidedbiopsy,thecomputer-generatedtumormayappeartobelocatedoutsidethebreastwhiletrackingtheheadoftheuser.ThisswimmingeffecthasbeendemonstratedandminimizedbypredictingHMDpositioninsteadofsim-plymeasuringpositions(Azuma&Bishop,1994).CurrentHMDsystemsarelaglimitedasaconse-quenceoftrackerlag,thecomplexityofrendering,anddisplayingtheimages.Trackerlagisoftennotthelimit-ingfactorinperformance.Ifdisplayingtheimageisthelimitingfactor,noveldisplayarchitecturessupportingframelessrenderingcanhelpsolvetheproblem(Bishopetal.,1994).Framelessrenderingisaprocedureforcon-tinuouslyupdatingadisplayedimage,asinformationbecomesavailableinsteadofupdatingentireframesatatime.Thetradeoffsbetweenlagandimagequalityarecurrentlybeinginvestigated(Scher-Zagier,1997).Ifweassumethatwearelimitedbythespeedofrenderinganimage,eye-trackingcapabilitymaybeusefultoquicklyupdateinformationonlyaroundthegazepointoftheuser(Thomasetal.,19;Rolland,Yoshida,etal.,1998;Vaissie&Rolland,1999).

Oneofthemajoradvantagesofvideosee-throughHMDsisthepotentialcapabilityofreducingtherelativelatenciesbetweenthe2-Drealandsyntheticimagesasaconsequenceofbothtypesofimagesbeingdigital(Ja-cobsetal.,1997).Manipulationoftheimagesinspaceandintimeisappliedtoregisterthem.Three-dimen-sionalregistrationiscomputationallyintensive,ifatallrobust,andchallengingforinteractivespeed.Thespatialapproachtoforcingregistrationinvideosee-throughsystemsistocorrectregistrationerrorsbyimagingland-markpointsintherealworldandregisteringvirtualob-jectswithrespecttothem(Stateetal.,1996).Oneap-proachtoeliminatingtemporaldelaysbetweentherealandcomputer-generatedimagesinsuchacaseistocap-tureavideoimageanddrawthegraphicsontopofthevideoimage.Thenthebufferisswapped,andthecom-binedimageispresentedtotheHMDuser.Insuchaconfiguration,nodelayapparentlyexistsbetweentherealandcomputer-generatedimages.Iftheactualla-tencyofthecomputer-generatedimageislargewithre-specttothevideoimage,however,itmaycausesensoryconflictsbetweenvisionandproprioceptionbecausethevideoimagesnolongercorrespondtothereal-worldscene.Anymanualinteractionswithrealobjectscouldsufferasaresult.

Anotherapproachtominimizingdelaysinvideosee-throughHMDsistodelaythevideoimageuntilthecomputer-generatedimageisrendered.BajuraandNeu-mann(1995)appliedchromakeying,forexample,todynamicallyimageapairofredLEDsplacedontworealobjects(onestream)andthenregisteredtwovirtualob-jectswithrespecttothem(secondstream).Bytrackingmorelandmarks,betterregistrationofrealandvirtualobjectsmaybeachieved(TomasiandKanade,1991).Thelimitationoftheapproachtakenistheattempttoregister3-Dscenesusing2-Dconstraints.Iftheuserrotateshisheadrapidlyorifareal-worldobjectmoves,theremaybeno‘‘correct’’transformationforthevirtualsceneimage.Toalignallthelandmarks,onemusteitherallowerrorsinregistrationofsomeofthelandmarksorperformanonlinearwarpingofthevirtualscenethatmaycreateundesirabledistortionsofthevirtualobjects.Thenontrivialsolutiontothisproblemistoincreasethespeedofthesystemuntilscenechangesbetweenframesaresmallandcanbeapproximatedwithsimple2-Dtransformations.

Inasimilarvein,itisalsoimportanttonotethatthevideoviewoftherealscenewillnormallyhavesomelagduetothetimeittakestoacquireanddisplaythevideoimages.Thus,theimageinavideosee-throughHMDwillnormallybeslightlydelayedwithrespecttotherealworld,evenwithoutaddingdelaytomatchthesyntheticimages.Thisdelaymayincreaseifanimage-processingstepisappliedtoeitherenforceregistrationorperformocclusion.Thekeyissueiswhetherthedelayinthesys-temistoogreatfortheusertoadapttoit(Held&Durlach,1987).

Systemsusingopticalsee-throughHMDshavenomeansofintroducingartificialdelaysintotherealscene.Therefore,thesystemmayneedtobeoptimizedforlowlatency,perhapslessthan60ms,wherepredictivetrack-ingcanbeeffective(Azuma&Bishop,1994).Foranyremaininglag,theusermayhavetolimithisactionstoslowheadmotions.Applicationsinwhichspeedofmovementcanbereadilycontrolled,suchasintheVRDAtooldescribedearlier,canbenefitfromopticalsee-throughtechnology(Rolland&Arthur,1997).Theadvantageofhavingnoartificialdelaysisthatrealob-

RollandandFuchs295

jectswillalwaysbewheretheyareperceivedtobe,andthismaybecrucialforabroadrangeofapplications.3.1.2Real-SceneResolutionandDistortion.Ifreal-sceneresolutionreferstotheresolutionofthereal-sceneobject,thebestreal-sceneresolutionthatasee-throughdevicecanprovideisthatperceivedwiththenakedeyeunderunitmagnificationoftherealscene.CertainlyundermicroscopicobservationasdescribedbyHill(Edwardsetal.,1995),thebestsceneresolutiongoesbeyondthatobtainedwithanakedeye.Itisalsoassumedthatthesee-throughdevicehasnoimage-pro-cessingcapability.

Aresolutionextremelyclosetothatobtainedwiththenakedeyeiseasilyachievedwithanonmicroscopicopti-calsee-throughHMD,becausetheopticalinterfacetotherealworldissimplyathinparallelplate(suchasaglassplate)positionedbetweentheeyesandtherealscene.Suchaninterfacetypicallyintroducesonlyverysmallamountsofopticalaberrationstotherealscene:Forexample,forareal-pointobjectseenthrougha2mmplanarparallelplateplacedinfrontofa4mmdia.eyepupil,thediffusionspotduetosphericalaberrationwouldsubtenda2107arc-minutevisualangleforapointobjectlocated500mmaway.Sphericalaberrationisoneofthemostcommonandsimpleaberrationsinopticalsystemsthatleadtoblurringoftheimages.Suchadegradationofimagequalityisnegligiblecomparedtotheabilityofthehumaneyetoresolveavisualangleof1minuteofarc.Similarly,planarplatesintroducelowdis-tortionoftherealscene,typicallybelow1%.Thereisnodistortiononlyforthechiefraysthatpasstheplateparal-leltoitsnormal.1

Inthecaseofavideosee-throughHMD,real-sceneimagesaredigitizedbyminiaturecameras(Edwardsetal.,1993)andconvertedintoananalogsignalthatisfedtotheHMD.TheimagesarethenviewedthroughtheHMDviewingopticsthattypicallyuseaneyepiecede-sign.TheperceivedresolutionoftherealscenecanthusbelimitedbytheresolutionofthevideocamerasortheHMDviewingoptics.Currentlyavailableminiature

1.AchiefrayisdefinedasaraythatemanatesfromapointintheFOVandpassesthroughthecenterofthepupilsofthesystem.TheexitpupilinanHMDistheentrancepupilofthehumaneye.

296PRESENCE:VOLUME9,NUMBER3

videocamerastypicallyhavearesolutionof0480,whichisalsoneartheresolutionlimitoftheminiaturedisplayscurrentlyusedinHMDs.2Dependinguponthemagnificationandthefieldofviewoftheviewingoptics,variouseffectivevisualresolutionsmaybereached.

Whiletheminiaturedisplaysandthevideocamerasseemtocurrentlylimittheresolutionofmostsystems,suchperformancemayimprovewithhigher-resolutiondetec-torsanddisplays.

Inassessingvideosee-throughsystems,onemustdis-tinguishbetweennarrowandwideFOVdevices.Large-FOV(50deg.)eyepiecedesignsareknowntobeex-tremelylimitedinopticalqualityasaconsequenceoffactorssuchasopticalaberrationsthataccompanylargeFOVs,pixelizationthatmaybecomemoreapparentun-derlargemagnification,andtheexitpupilsizethatmustaccommodatethesizeofthepupilsofaperson’seyes.Thus,evenwithhigher-resolutioncamerasanddisplays,videosee-throughHMDsmayremainlimitedintheirabilitytoprovideareal-sceneviewofhighresolutionifconventionaleyepiecedesignscontinuetobeused.InthecaseofsmalltomoderateFOV(10deg.to20deg.)videosee-throughHMDs,theresolutionisstilltypicallymuchlessthantheresolvingpowerofthehumaneye.Anewtechnology,referredtoastiling,mayovercomesomeofthecurrentlimitationsofconventionaleyepiecedesignforlargeFOVs(Kaiser,1994).Theideaistousemultiplenarrow-FOVeyepiecescoupledwithminiaturedisplaystocompletelycover(ortile)theuser’sFOV.BecausetheindividualeyepieceshaveafairlynarrowFOV,higherresolution(neverthelesscurrentlylessthanthehumanvisualsystem)canbeachieved.Oneofthefewdemonstrationsofhigh-resolution,large-FOVdis-playsisthetileddisplays.Achallengeistheminimizationofseamsinassemblingthetiles,andtherenderingofmultipleimagesatinteractivespeed.Thetileddisplayscertainlybringnewpracticalandcomputationalchal-lengesthatneedtobeconfronted.Ifasee-throughca-pabilityisdesired(forexample,todisplayvirtualfurni-tureinanemptyroom),itiscurrentlyunclearwhether

2.Thenumberofphysicalelementsistypically0480.Onecanusesignalprocessingtointerpolatebetweenlinestogethigherresolu-tions.

thetechnicalproblemsassociatedwithprovidingoverlaycanbesolved.

Theoretically,distortionisnotaprobleminvideosee-throughsystemsbecausethecamerascanbedesignedtocompensateforthedistortionoftheopticalviewer,asdemonstratedbyEdwardsetal.(1993).However,ifthegoalistomergerealandvirtualinformation,asinultra-soundechography,havingawarpedrealscenesignifi-cantlyincreasesthecomplexityofthesynthetic-imagegeneration(Stateetal.,1994).Real-timevideocorrec-tioncanbeusedattheexpenseofanadditionaldelayintheimage-generationsequence.Analternativeistouselow-distortionvideocamerasattheexpenseofanar-rowerFOV,mergeunprocessedrealsceneswithvirtualscenes,andwarpthemergedimages.Warpingcanbedoneusing(forexample)real-timetexturemappingtocompensateforthedistortionoftheHMDviewingop-ticsasalaststep(Rolland&Hopkins,1993;Watson&Hodges,1995).

Theneedforhigh,real-sceneresolutionishighlytaskdependent.Demandingtaskssuchassurgeryorengi-neeringtraining,forexample,maynotbeabletotoler-atemuchlossinreal-sceneresolution.Becausethelarge-FOVvideosee-throughsystemsthatwehave

experiencedareseriouslylimitedintermsofresolution,narrow-FOVvideosee-throughHMDsarecurrentlypreferred.Independentlyofresolution,anadditionalcriticalissueinaimingtowardsnarrow-FOVvideosee-throughHMDsistheneedtomatchtheviewpointofthevideocameraswiththeviewpointoftheuser.Match-ingischallengingwithlarge-FOVsystems.Also,meth-odsformatchingvideoandrealscenesforlarge-FOVtileddisplaysmustbedeveloped.Atthistime,consider-ingthegrowingavailabilityofhigh-resolutionflat-paneldisplays,weforeseethattheresolutionofsee-thoughHMDscouldgraduallyincreaseforbothsmall-andlarge-FOVsystems.Thedevelopmentandmarketingofminiaturehigh-resolutiontechnologymustbeunder-takentoachieveresolutionsthatmatchthatofhumanvision.

3.1.3FieldofView.AgenerallychallengingissueofHMDsisprovidingtheuserwithanadequateFOVforagivenapplication.Formostapplications,increasing

thebinocularFOVmeansthatfewerheadmovementsarerequiredtoperceiveanequivalentlylargescene.WebelievethatalargeFOVisespeciallyimportantfortasksthatrequiregrabbingandmovingobjectsandthatitprovidesincreasedsituationawarenesswhencomparedtonarrow-FOVdevices(Slater&Wilbur,1997).Thesituationwithsee-throughdevicesissomewhatdifferentfromthatoffullyopaqueHMDsinthattheaimofusingthetechnologyisdifferentfromthatofimmersingtheuserinavirtualenvironment.

3.1.3.1OverlayandPeripheralFOV.ThetermoverlayFOVisdefinedastheregionoftheFOVwheregraphicalinformationandrealinformationaresuperim-posed.TheperipheralFOVisthereal-worldFOVbe-yondtheoverlayFOV.ForimmersiveopaqueHMDs,nosuchdistinctionismade;onereferssimplytotheFOV.ItisimportanttonotethattheoverlayFOVmayneedtobenarrowonlyforcertainaugmented-realityapplications.Forexample,inavisualizationtoolsuchastheVRDAtool,onlytheknee-jointregionisneededintheoverlayFOV.InthecaseofvideoHMD-guided

breastbiopsy,theoverlayFOVcouldbeasnarrowasthesynthesizedtumor.Therealsceneneednotnecessarilybesynthesized.TheavailableperipheralFOV,however,iscriticalforsituationawarenessandismostoftenre-quiredforvariousapplicationswhetheritisprovidedaspartoftheoverlayoraroundtheoverlay.Ifprovidedaroundtheoverlay,thetransitionfromrealtovirtualimagerymustbemadeasseamlessaspossible.Thisisaninvestigationthathasnotyetbeenaddressedinvideosee-throughHMDs.

Opticalsee-throughHMDstypicallyprovidefrom20deg.to60deg.overlayFOVviathehalf-transparentmirrorsplacedinfrontoftheeyes,acharacteristicthatmayseemsomewhatlimitedbutpromisingforavarietyofmedicalapplicationswhoseworkingvisualizationdis-tanceiswithinarmreach.LargerFOVshavebeenob-tained,upto82.567deg.,attheexpenseofreducedbrightness,increasedcomplexity,andmassive,expensivetechnology(Welch&Shenker,1984).SuchFOVsmayhavebeenrequiredforperformingnavigationtasksinrealandvirtualenvironmentsbutarelikelynotrequiredinmostaugmented-realityapplications.Opticalsee-

RollandandFuchs297

throughHMDs,however,whetherornottheyhavealargeoverlayFOV,havebeentypicallydesignedopenenoughthatuserscanusetheirperipheralvisionaroundthedevice,thusincreasingthetotalreal-worldFOVtocloselymatchone’snaturalFOV.Anannulusofobstruc-tionusuallyresultsfromthemountsofthethinsee-throughmirrorsimilartothewaythatourvisionmaybepartiallyoccludedbyaframewhenwearingeyeglasses.Inthedesignofvideosee-throughHMDs,adifficultengineeringtaskismatchingthefrustumoftheeyewiththatofthecamera(asweshalldiscussinsection3.1.4).Whilesuchmatchingisnotsocriticalforfar-fieldview-ing,itisimportantfornear-fieldvisualizationasinvari-ousmedicalvisualizations.Thisdifficultmatchingprob-lemhasleadtotheconsiderationofnarrower-FOVsystems.Acompact,4030deg.FOVdesign,de-signedforopticalsee-throughHMDbutadaptabletovideosee-through,wasproposedbyManhart,Malcolm,&Frazee(1993).Videosee-throughHMDs,ontheotherhand,canprovide(intermsofasee-throughFOV)theFOVdisplayedwiththeopaquetypeviewingopticsthattypicallyrangefrom20deg.to90deg.InsuchsystemswheretheperipheralFOVoftheuserisoccluded,theeffectivereal-worldFOVisoftensmallerthaninopticalsee-throughsystems.Whenusingavideosee-throughHMDinahand-eyecoordinationtask,wefoundinarecenthuman-factorstudythatusersneededtoperformlargerheadmovementstoscananactivefieldofvisionthanwhenperformingthetaskwiththeun-aidedeye(Biocca&Rolland,1998).Wepredictthattheneedtomakelargerheadmovementswouldnotariseasmuchwithsee-throughHMDswithequivalentoverlayFOVsbutlargerperipheralFOVs,becauseusersarepro-videdwithincreasedperipheralvision,andthusaddi-tionalinformation,tomorenaturallyperformthetask.3.1.3.2IncreasingPeripheralFOVinVideo

See-ThroughHMDs.AnincreaseinperipheralFOVinvideosee-throughsystemscanbeaccomplishedintwoways:inafoldedopticaldesign,asusedforopticalsee-throughHMDs,howeverwithanopaquemirrorinsteadofahalf-transparentmirror,orinanonfoldeddesignbutwithnonenclosedmounts.Thelattercallsforinno-vativeoptomechanicaldesignbecauseheavieroptics

298PRESENCE:VOLUME9,NUMBER3

havetobesupportedthanineitheropticalorfoldedvideosee-through.Foldedsystemsrequireonlyathinmirrorinfrontoftheeyes,andtheheavieropticalcom-ponentsareplacedaroundthehead.However,the

tradeoffwithfoldedsystemsisasignificantreductionintheoverlayFOV.

3.1.3.3TradeoffResolutionandFOV.WhiletheresolutionofadisplayinanHMDisdefinedinthegraphicscommunitybythenumberofpixels,therel-evantmeasureofresolutionisthenumberofpixelsperangularFOV,whichisreferredtoasangularresolution.Indeed,whatisofimportanceforusabilityistheangularsubtendsofapixelattheeyeoftheHMDuser.Mostcurrenthigh-resolutionHMDsachievehigherresolu-tionattheexpenseofareducedFOV.Thatis,theyusethesameminiature,high-resolutionCRTsbutwithop-ticsoflessmagnificationinordertoachievehigheran-gularresolution.ThisresultsinaFOVthatisoftennar-row.Theapproachthatemployslargehigh-resolutiondisplays,orlightvalves,andtransportsthehigh-resolu-tionimagestotheeyesbyimagingopticscoupledtoabundleofopticalfibersachieveshighresolutionatfairlylargeFOVs(Thomasetal.,19).Thecurrentpro-posedsolutionsthatimproveresolutionwithouttradingFOVareeithertilingtechniques,high-resolutioninsetdisplays(Fernie,1995;Rolland,Yoshida,etal.,1998),orprojectionHMDs(Huaetal.,2000).

ProjectiveHMDsdifferfromconventionalHMDsinthatprojectionopticsareusedinsteadofeyepieceopticstoprojectrealimagesofminiaturedisplaysintheenvi-ronment.Ascreenplacedintheenvironmentreflectstheimagesbacktotheeyesoftheuser.ProjectiveHMDshavebeendesignedanddemonstrated,forexample,byKijimaandOjika(1997)andParsonsandRolland

(1998).Kijimausedaconventionalprojectionscreeninhisprototype.ParsonsandRollanddevelopedafirst-prototypeprojectionHMDsystemtodemonstratethatanundistortedvirtual3-Dimagecouldberenderedwhenprojectingastereopairofimagesonabentsheetofmicroretroreflectorcubes.Thefirstproof-of-conceptsystemisshowninfigure12.Acomprehensiveinvestiga-tionoftheopticalcharacteristicsofprojectiveHMDsisgivenbyHuaetal.(2000).Wearealsodevelopingthe

Figure12.Proofofconceptprototypeofaprojectionhead-mounteddisplaywithmicrore󰁁ectorsheeting(1998).

next-generationprototypesofthetechnologyusingcus-tom-mademiniaturelightweightoptics.ThesystempresentsvariousadvantagesoverconventionalHMDs,includingdistortion-freeimages,occludedvirtualob-jectsfromreal-objectsinterposition,noimagecross-talksformultiuserparticipantsinthevirtualworld,andthepotentialforawideFOV(upto120deg.).

3.1.4ViewpointMatching.Invideosee-throughHMDs,thecameraviewpoint(thatis,theen-trancepupil)mustbematchedtotheviewpointoftheobserver(theentrancepupiloftheeye).Theviewpointofacameraoreyeisequivalenttothecenterofprojec-tionusedinthecomputergraphicsmodelthatcomputesthestereoimagesandistakenheretobethecenteroftheentrancepupiloftheeyeorcamera(Vaissie&Rolland,2000).Inearliervideosee-throughdesigns,Edwardsetal.(1993)investigatedwaystomountthecamerastominimizeerrorsinviewpointmatching.Theerrorminimizationversusexactmatchingwasaconse-quenceofworkingwithwide-FOVsystems.Iftheview-pointsofthecamerasdonotmatchtheviewpointsoftheeyes,theuserexperiencesaspatialshiftintheperceivedscenethatmayleadtoperceptualanomalies(asfurther

Figure13.A10degreeFOVvideosee-throughHMD:DglassesdevelopedatUNC-CH.Lipstickcamerasandadoublefoldmirrorarrangementwasusedtomatchtheviewpointsofthecameraanduser(1997).

discussedunderhuman-factorsissues(Biocca&Rolland,1998).Erroranalysisshouldthenbecon-ductedinsuchacasetomatchtheneedoftheapplica-tion.

ForcasesinwhichtheFOVissmall(lessthanapproxi-mately20deg.),exactmatchinginviewpointsispos-sible.Becausethecamerascannotbephysicallyplacedattheactualeyepoints,mirrorscanbeusedtofoldtheop-ticalpath(muchlikeaperiscope)tomakethecameras’viewpointscorrespondtotherealeyepointsasshowninfigure13(Edwardsetal.,1993).Whilesuchgeometrysolvestheproblemoftheshiftinviewpoint,itincreasesthelengthoftheopticalpath,whichreducesthefieldofview,forthesamereasonthatopticalsee-throughHMDstendtohavesmallerfieldsofview.Thus,videosee-throughHMDsmusteithertradetheirlargeFOVsforcorrectreal-worldviewpointsorrequiretheusertoadapttotheshiftedviewpointsasfurtherdiscussedinsection3.2.3.

RollandandFuchs299

Finally,correctlymountingthevideocamerasinavideosee-throughHMDrequiresthattheHMDhasaninterpupillarydistance(IPD)adjustment.GiventheIPDofauser,thelateralseparationofthevideocamerasmustthenbeadjustedtothatvalueinorderfortheviewsob-tainedbythevideocamerastomatchthosethatwouldhavebeenobtainedwithnakedeyes.Ifoneweretoac-countforeyemovementsinvideosee-throughHMDs,thelevelofcomplexityinslavingthecameraviewpointtotheuserviewpointwouldbehighlyincreased.Toourknowledge,suchcomplexityhasnotyetbeenconsid-ered.

3.1.5EngineeringandCostFactors.HMDdesignsoftensufferfromfairlylowresolution,limitedFOV,poorergonomicdesigns,andexcessiveweight.AgoodergonomicdesignrequiresanHMDwhoseweightissimilartoapairofeyeglasses,orwhichfoldsaroundtheuser’sheadsothedevice’scenterofgravityfallsnearthecenterofrotationofthehead(Rolland,1994).Thegoalhereismaximumcomfortandusability.ReasonablylightweightHMDdesignscurrentlysuffernarrowFOVs,ontheorderof20deg.Toourknowledge,atpresent,nolarge-FOVstereosee-throughHMDsofanytypearecomparableinweighttoapairofeyeglasses.RollandpredictsthatitcouldbeachievedwithsomeemergingtechnologyofprojectionHMDs(Rolland,Parsons,etal.,1998).However,itmustbenotedthatsuchtechnologymaynotbewellsuitedtoallvisualiza-tionschemesasitrequiresaprojectionscreensome-whereinfrontoftheuserthatisnotnecessarilyattachedtotheuser’shead.

Withopticalsee-throughHMDs,thefoldingcanbeaccomplishedwitheitheranon-axisoranoff-axisde-sign.Off-axisdesignsaremoreelegantandalsofarmoreattractivebecausetheyelimatetheghostimagesthatcurrentlyplagueusersofon-axisHMDs(Rolland,2000).Off-axisdesignsarenotcommerciallyavailablebecauseveryfewprototypeshavebeenbuilt(andthosethathavebeenbuiltareclassified)(Shenker,1998).

Moreover,off-axissystemsaredifficulttodesignandarethusexpensivetobuildasaresultofoff-axiscomponents(Shenker,1994).Anonclassified,off-axisdesignhasbeendesignedbyRolland(1994,2000).Severalfactors

300PRESENCE:VOLUME9,NUMBER3

(includingcost)havealsohinderedtheconstructionofafirstprototypeaswell.Newgenerationsofcomputer-controlledfabricationandtestingareexpectedtochangethistrend.

Sincetheirbeginning,high-resolutionHMDshavebeenCRTbased.Earlysystemswereevenmonochrome,butcolorCRTsusingcolorwheelsorframe-sequentialcolorhavebeenfabricatedandincorporatedintoHMDs(Allen,1993).Fiveyearsago,wemayhavethoughtthat,today,high-resolution,color,flat-paneldisplayswouldbethefirstchoiceforHMDs.Whilethisisslowlyhap-pening,miniatureCRTsarenotfullyobsolete.Thecur-rentoptimism,however,ispromptedbynewtechnolo-giessuchasreflectiveLCDs,microelectromechanicalsystems(MEMS)-baseddisplays,laser-baseddisplays,andnanotechnology-baseddisplays.

3.2Human-FactorandPerceptualIssues

Assumingthatmanyofthetechnologicalchal-lengesdescribedhavebeenaddressedandhigh-perfor-manceHMDscanbebuilt,akeyhuman-factorissueforsee-throughHMDsisthatofuseracceptanceandsafety,whichwillbediscussedfirst.Weshallthendiscussthetechnicalitiesofperceptioninsuchdisplays.Theulti-matesee-throughdisplayisonethatprovidesquantita-tiveandqualitativevisualrepresentationsofscenesthatconformtoapredictivemodel(forexample,conformtothatgivenbytherealworldifthatistheintention).Is-suesincludetheaccuracyandprecisionoftherenderedandperceivedlocationsofobjectsindepth,theaccuracyandprecisionoftherenderedandperceivedsizesofrealandvirtualobjectsinascene,andtheneedofanunob-structedperipheralFOV(whichisimportantformanytasksthatrequiresituationawarenessandthesimplema-nipulationofobjectsandaccessories.

3.2.1UserAcceptanceandSafety.Afairques-tionforeithertypeoftechnologyis‘‘willanyoneactuallywearoneofthesedevicesforextendedperiods?’’Theanswerwilldoubtlessbespecifictotheapplicationandthetechnologyincluded,butitwillprobablycenteruponwhethertheadvancedcapabilitiesaffordedbythe

technologyoffsettheproblemsinducedbytheencum-branceandsensoryconflictsthatareassociatedwithit.Inparticular,oneofusthinksthatvideosee-throughHMDsmaybemetwithresistanceintheworkplacebe-causetheyremovethedirect,real-worldviewinordertoaugmentit.Thisissueoftrustmaybedifficulttoover-comeforsomeusers.Ifwide-angleFOVvideosee-throughHMDsareused,thisproblemisexacerbatedinsafety-criticalapplications.Akeydifferenceinsuchappli-cationsmayturnouttobethefailuremodeofeachtechnology.Atechnologyfailureinthecaseofopticalsee-throughHMDsmayleavethesubjectwithoutanycomputer-generatedimagesbutstillwiththereal-worldview.Inthecaseofvideosee-through,itmayleavetheuserwiththecompletesuppressionofthereal-worldview,aswellasthecomputer-generatedview.

However,itmaybethattheissuehasbeengreatlylessenedbecausethevideoviewoccupiessuchasmallfraction(approximately10deg.visualangle)ofthesceneinrecentdevelopmentsofthetechnology.Itisespeciallytrueofflip-upandflip-downdevicessuchasthatdevelopedatUNC-CHandshowninfigure13.Imagequalityanditstradeoffsaredefinitelycriticalissuesrelatedtouseracceptanceforalltypesoftechnol-ogy.Inapersonalcommunication,MartinShenker,asenioropticalengineerwithmorethantwentyyearsofexperiencedesigningHMDs,pointedoutthattherearecurrentlynostandardsofimagequalityandtechnologyspecificationsforthedesign,calibration,andmainte-nanceofHMDs.Thisisacurrentconcernatatime

whenthetechnologymaybeadoptedinvariousmedicalvisualizations.

3.2.2PerceivedDepth.3.2.2.1Occlusion.Theabilitytoperformocclusioninsee-throughHMDsisanimportantissueofcomparisonbetweenopticalandvideosee-throughHMDs.Oneofthemostimportantdifferencesbetweenthesetwotechnologiesishowtheyhandlethedepthcueknownasocclusion(orinterposi-tion).Inreallife,anopaqueobjectcanblocktheviewofanotherobjectsothatpartorallofitisnotvisible.Whilethereisnoprobleminmakingcomputer-gener-atedobjectsoccludeeachotherineithersystem,itisconsiderablymoredifficulttomakerealobjectsocclude

virtualobjects(andviceversa)unlesstherealworldforanapplicationispredefinedandhasbeenmodeledinthecomputer.Eventhen,onewouldneedtoknowtheexactlocationofauserwithrespecttothatrealenvironment.Thisisnotthecaseinmostaugmented-realityapplica-tions,inwhichtherealworldisconstantlychangingandon-the-flyacquisitionisalltheinformationonewilleverhaveoftherealworld.Occlusionisastrongmonocularcuetodepthperceptionandmayberequiredincertainapplications(Cutting&Vishton,1995).

Inbothsystems,computingocclusionbetweentherealandvirtualscenesrequiresadepthmapofbothscenes.Adepthmapofthevirtualsceneisusuallyavail-able(forz-bufferedimagegenerators),butadepthmapoftherealsceneisamuchmoredifficultproblem.Whileonecouldcreateadepthmapinadvancefromastaticrealenvironment,manyapplicationsrequireon-the-flyimageacquisitionoftherealscene.Assumingthesystemhasadepthmapoftherealenvironment,videosee-throughHMDsareperfectlypositionedtotakeadvan-tageofthisinformation.Theycan,onapixel-by-pixelbasis,selectivelyblocktheviewofeithersceneorevenblendthemtominimizeedgeartifacts.Oneofthechiefadvantagesofvideosee-throughHMDsisthattheyhandlethisproblemsowell.

Thesituationforopticalsee-throughHMDscanbemorecomplex.Existingopticalsee-throughHMDsblendthetwoimageswithbeamsplitters,whichblendtherealandvirtualimagesuniformlythroughouttheFOV.Normally,theonlycontrolthedesignerhasistheamountofreflectanceversustransmittanceofthebeamsplitter,whichcanbechosentomatchthebrightnessofthedisplayswiththeexpectedlightlevelsinthereal-worldenvironment.Ifthesystemhasamodeloftherealenvironment,itispossibletohaverealobjectsoccludevirtualonesbysimplynotdrawingtheoccludedpartsofthevirtualobjects.Theonlylightwillthenbefromtherealobjects,givingtheillusionthattheyareoccludingthevirtualones.Suchaneffectrequiresadarkenedroomwithlightdirectedwhereitisneeded.Thistech-niquehasbeenusedbyCAEElectronicsintheirflightsimulator.Whenthepilotslookoutthewindow,theyseecomputer-generatedobjects.Iftheylookinsidethe

RollandandFuchs301

cockpit,however,theappropriatepixelsofthecom-puter-generatedimagearemaskedsothattheycanseetherealinstruments.Theroomiskeptfairlydarksothatthistechniquewillwork(Barrette,1992).DavidMizell(fromBoeingSeattle)andTomCaudell(UniversityofNewMexico)arealsousingthistechnique;theyrefertoitas‘‘fusedreality’’(Mizell,1998).

Whileopticalsee-throughHMDscanallowrealob-jectstooccludevirtualobjects,thereverseisevenmorechallengingbecausenormalbeamsplittershavenowayofselectivelyblockingouttherealenvironment.Thisproblemhasatleasttwopossiblepartialsolutions.Thefirstsolutionistospatiallycontrolthelightlevelsintherealenvironmentandtousedisplaysthatarebrightenoughsothatthevirtualobjectsmasktherealonesbyreasonofcontrast.(Thisapproachisusedintheflightsimulatorjustmentionedforcreatingthevirtualinstru-ments.)Thismaybeasolutionforafewapplications.Apossiblesecondsolutionwouldbetolocallyattenuatethereal-worldviewbyusinganaddressablefilterdeviceplacedonthesee-throughmirror.Itispossibletogener-atepartialocclusioninthismannerbecausetheeffectivebeamoflightenteringtheeyefromsomepointinthescenecoversonlyasmallareaofthebeamsplitter,theeyepupilbeingtypically2mmto4mminphotopicvi-sion.Aproblemwiththisapproachisthattheuserdoesnotfocusonthebeamsplitter,butrathersomewhereinthescene.Apointinthescenemapstoadiskonthebeamsplitter,andvariouspointsinthescenemaptooverlappingdisksonthebeamsplitter.Thus,anyblock-ingdoneatthebeamsplittermayoccludemoreofthescenethanexpected,whichmightleadtooddvisualef-fects.Afinalpossibilityisthatsomeapplicationsmayworkacceptablywithoutproperlyrenderedocclusioncues.Thatis,insomecases,theusermaybeabletouseotherdepthcues,suchashead-motionparallax,tore-solvetheambiguitycausedbythelackofocclusioncues.3.2.2.2RenderedLocationsofObjectsinDepth.Weshalldistinguishbetweenerrorsintherenderedandper-ceivedlocationsofobjectsindepth.Theformeryieldsthelatter.Onecanconceive,however,thaterrorsintheperceivedlocationofobjectsindepthcanalsooccur

302PRESENCE:VOLUME9,NUMBER3

evenintheabsenceoferrorsinrendereddepthsasare-sultofanincorrectcomputationalmodelforstereopairgenerationorasuboptimalpresentationofthestereoimages.Thisistruebothforopticalandvideosee-throughHMDs.Indeed,ifthetechnologyisadequatetosupportacomputationalmodel,andthemodelac-countsforrequiredtechnologyandcorrespondingpa-rameters,therenderedlocationsofobjectsindepth—aswellastheresultingperceivedlocationsofobjectsindepth—willfollowexpectations.Vaissierecentlyshowedsomelimitationsofthechoiceofastaticeyepointincomputationalmodelsforstereopairgenerationforvir-tualenvironmentsthatyielderrorsinrenderedandthusperceivedlocationofobjectsindepths(Vaissieand

Rolland,2000).Theultimategoalistoderiveacompu-tationalmodelanddeveloptherequiredtechnologythatyieldthedesiredperceivedlocationofobjectsindepth.ErrorsinrendereddepthtypicallyresultfrominaccuratedisplaycalibrationandparameterdeterminationsuchastheFOV,theframe-bufferoverscan,theeyepoints’loca-tions,conflictingornoncompatiblecuestodepth,andremainingopticalaberrationsincludingresidualopticaldistortions.

3.2.2.3FOVandFrame-BufferOverscan.Inaccu-raciesofafewdegreesinFOVareeasilymadeifnocali-brationisconducted.Suchinaccuraciescanleadtosig-nificanterrorsinrendereddepthsdependingontheimaginggeometry.Forsomemedicalandcomputer-guidedsurgeryapplications,forexample,errorsofsev-eralmillimetersarelikelytobeunacceptable.TheFOVandtheoverscanoftheframebufferthatmustbemea-suredandaccountedfortoyieldaccuraterendereddepthsarecriticalparametersforstereopairgenerationinHMDs(Rollandetal.,1995).Theseparametersmustbesetcorrectlyregardlessofwhetherthetechnologyisopticalorvideosee-through.

3.2.2.4SpecificationofEyepointLocation.Thespecificationofthelocationsoftheuser’seyepoints(whichareusedtorenderthestereoimagesfromthecorrectviewpoints)mustbespecifiedforaccurateren-dereddepth.Thisappliestobothopticalandvideosee-throughHMDs.Inaddition,forvideosee-through

HMDs,thereal-scenevideoimagesmustbeacquiredfromthecorrectviewpoint(Biocca&Rolland,1998).Forthecomputergraphics-generationcomponent,threechoicesofeyepointlocationswithinthehumaneyehavebeenproposed:thenodalpointoftheeye3(Robi-nett&Rolland,1992;Deering,1992),theentrancepupiloftheeye(Rolland,1994;Rollandetal.,1995),andthecenterofrotationoftheeye(Holloway,1995).Rolland(1995)discussesthatthechoiceofthenodalpointwouldinfactyielderrorsinrendereddepthinallcaseswhethertheeyesaretrackedornot.Foradevicewitheye-trackingcapability,theentrancepupiloftheeyeshouldbetakenastheeyepoint.Ifeyemovementsareignored,meaningthatthecomputer-graphicseyepointsarefixed,thenitwasproposedthatitisbesttoselectthecenterofrotationoftheeyeastheeyepoint(Fry,1969;Holloway,1995).Anin-depthanalysisofthisissuere-vealsthatwhilethecenterofrotationyieldshigheraccu-racyinposition,thecenteroftheentrancepupilyieldsinfacthigherangularaccuracy(Vaissie&Rolland,2000).Therefore,dependingonthetaskinvolved,andwhetherangularaccuracyorpositionaccuracyismostimportant,thecentersofrotationorthecentersoftheentrancepu-pilmaybeselectedasbesteyepointslocationinHMDs.3.2.2.5ResidualOpticalDistortions.Opticaldis-tortionisoneofthefewopticalaberrationsthatdonotaffectimagesharpness;rather,itintroduceswarpingoftheimage.Itoccursonlyforopticsthatincludelenses.Iftheopticsincludeonlyplanemirrors,therearenodis-tortions(Peuchot,1994).Warpingoftheimagesleadstoerrorsinrendereddepths.Distortionresultsfromthelocationsoftheuser’spupilsawayfromthenodalpointsoftheoptics.Moreover,itvariesasafunctionofwheretheuserlooksthroughtheoptics.However,iftheopticsarewellcalibratedtoaccountfortheuser’sIPD,distor-tionwillbefairlyconstantfortypicaleyemovementsbehindtheoptics.Prewarpingofthecomputer-gener-atedimagecanthusbeconductedtocompensateforthe

3.Nodalpointsareconjugatepointsinanopticalsystemthatsatisfyanangularmagnificationof1.Twopointsareconsideredtoconjugateofeachotheriftheyareimagesofeachother.

RollandandFuchs303

Figure14.(a)Benchprototypehead-mounteddisplaywithhead-motionparallaxdevelopedintheVGILabatUCF(1997).(b)Schematicoftheopticalimagingfromatopviewofthesetup.

opticalresidualdistortions(Robinett&Rolland,1992;Rolland&Hopkins,1993;Watson&Hodges,1995).3.2.2.6PerceivedLocationofObjectsinDepth.Oncedepthsareaccuratelyrenderedaccordingtoagivencomputationalmodelandthestereoimagesarepresentedaccordingtothecomputationalmodel,theperceivedlocationandsizeofobjectsindepthbecomeanimportantissueintheassessmentofthetechnologyandthemodel.Accuracyandprecisioncanbedefinedonlystatistically.Givenanensembleofmeasuredper-ceivedlocationofobjectsindepths,thedepthperceptwillbeaccurateifobjectsappearinaverageattheloca-tionpredictedbythecomputationalmodel.Theper-ceivedlocationofobjectsindepthwillbepreciseifob-jectsappearwithinasmallspatialzonearoundthataveragelocation.Weshalldistinguishbetweenoverlap-pingandnonoverlappingobjects.Inthecaseofnonoverlappingobjects,onemayresorttodepthcuesotherthanocclusion.Theseincludefamil-iarsizes,stereopsis,perspective,texture,andmotionparallax.Apsychophysicalinvestigationoftheperceivedlocationofobjectsindepthinanopticalsee-throughHMDusingstereopsisandperspectiveasthevisualcuestodepthisgiveninRollandetal.,(1995),andRollandetal.(1997).TheHMDshowninfigure14ismountedonabenchforcalibrationpurposeandflexibilityinvari-ousparametersettings.

Inafirstinvestigation,asystematicshiftof50mmintheperceivedlocationofobjectsindepthversuspre-dictedvalueswasfoundinthisfirstsetofstudy(Rollandetal.,1995).Moreover,theprecisionofthemeasuresvariedsignificantlyacrosssubjects.Aswelearnmoreabouttheinterfaceopticsandcomputationalmodelusedinthegenerationofthestereoimagepairsandim-proveonthetechnology,wehavedemonstratederrors

304PRESENCE:VOLUME9,NUMBER3

ontheorderof2mm.Thetechnologyisnowreadytodeployforextensivetestinginspecificapplications,andtheVRDAtoolisoneoftheapplicationswearecur-rentlypursuing.

Studiesoftheperceivedlocationofobjectsindepthforoverlappingobjectsinanopticalsee-throughHMDhavebeenconductedbyEllisandBuchler(1994).Theyshowedthattheperceivedlocationofvirtualobjectscanbeaffectedbythepresenceofanearbyopaquephysicalobject.Whenaphysicalobjectwaspositionedinfrontof(orat)theinitialperceivedlocationofa3-Dvirtualob-ject,thevirtualobjectappearedtomoveclosertotheobserver.Inthecaseinwhichtheopaquephysicalobjectwaspositionedsubstantiallyinfrontofthevirtualobject,humansubjectsoftenperceivedtheopaqueobjecttobetransparent.IncurrentinvestigationswiththeVRDAtool,theopaquelegmodelappearstransparentwhenavirtualkneemodelisprojectedonthelegasseeninfig-ure4.Thevirtualanatomysubjectivelyappearstobeinsidethelegmodel(Baillot,1999;Outtersetal.,1999;Baillotetal.,2000).

3.2.3Adaptation.Whenasystemdoesnotofferwhattheuserultimatelywants,twopathsmaybetaken:improvingonthecurrenttechnology,orfirststudyingtheabilityofthehumansystemtoadapttoanimperfecttechnologicalunitandthendevelopingadaptationtrain-ingwhenappropriate.Thisispossiblebecauseoftheastonishingabilityofthehumanvisualandpropriocep-tivesystemstoadapttonewenvironments,ashasbeenshowninstudiesonadaptation(Rock,1966,forex-ample).

BioccaandRolland(1998)conductedastudyofad-aptationtovisualdisplacementusingalarge-FOVvideosee-throughHMD.Usersseetherealworldthroughtwocamerasthatarelocated62mmhigherthanand165mmforwardfromtheirnaturaleyepointsasshowninfigure2.Subjectsshowedevidenceofperceptualad-aptationtosensorydisarrangementduringthecourseofthestudy.Thisrevealeditselfasimprovementinperfor-manceovertimewhilewearingthesee-throughHMDandasnegativeaftereffectsoncetheyremovedit.Moreprecisely,thenegativeaftereffectmanifesteditselfclearlyasalargeovershootinadepth-pointingtask,aswellas

anupwardtranslationinalateralpointingtaskafterwearingtheHMD.Moreover,someparticipantsexperi-encedsomeearlysignsofcybersickness.

Thepresenceofnegativeaftereffectshassomepoten-tiallydisturbingpracticalimplicationsforthediffusionoflarge-FOVvideosee-throughHMDs(Kennedy&Stan-ney,1997).SomeoftheintendedearlierusersoftheseHMDsaresurgeonsandotherindividualsinthemedicalprofession.Hand-eyesensoryrecalibrationforhighlyskilledusers(suchassurgeons)couldhavepotentiallydisturbingconsequencesifthesurgeonweretoentersurgerywithinsomeperiodafterusinganHMD.Itisanempiricalquestionhowlongthenegativeaftereffectsmightpersistandwhetheraprogramofgradualadapta-tion(Welch,1994)ordualadaptation(Welch,1993)mightminimizetheeffectaltogether.Inanycase,anyshiftinthecameraeyepointsneedtobeminimizedasmuchaspossibletofacilitatetheadaptationprocessthatistakingplace.Aswelearnmoreabouttheseissues,wewillbuilddeviceswithlesserrorandmoresimilaritybe-tweenusingthesesystemsandapairofeyeglasses(sothatadaptationtakeslesstimeandaftereffectsdecreaseaswell).

Aremainingissueistheconflictbetweenaccommoda-tionandconvergenceinsuchdisplays.Theissuecanbesolvedatsomecost(Rolland,etal.,2000).Forlower-endsystems,aquestiontoinvestigateishowusersadapttovarioussettingsofthetechnology.Forhigh-endsys-tems,muchresearchisstillneededtounderstandtheimportanceofperceptualconflictsandhowtobestmini-mizethem.

3.2.4PeripheralFOV.Giventhatperipheralvisioncanbeprovidedinbothopticalandvideosee-throughsystems,thenextquestioniswhetheritisusedeffectivelyforbothsystems.Inopticalsee-through,thereisalmostnotransitionordiscrepancybetweentherealscenecapturedbythesee-throughdeviceandtheperipheralvisionseenonthesideofthedevice.

Forvideosee-through,theperipheralFOVhasbeenprovidedbylettingtheuserseearoundthedevice,aswithopticalsee-through.However,itremainstobeseenwhetherthedifferenceinpresentationofthesuperim-posedrealsceneandtheperipheralrealscenewillcause

discomfortorprovideconflictingcuestotheuser.Theissueisthatthevirtualdisplayscallforadifferentaccom-modationfortheuserthantherealsceneinvariouscases.

3.2.5DepthofField.Oneimportantpropertyofopticalsystems,includingthevisualsystem,isdepthoffield.(Depthoffieldreferstotherangeofdistancesfromthedetector(suchastheeye)inwhichanobjectappearstobeinfocuswithouttheneedforachangeintheopticsfocus(suchaseyeaccommodation).Forthehumanvisualsystemexample,ifanobjectisaccuratelyfocusedmonocularly,otherobjectssomewhatnearerandfartherawayarealsoseenclearlywithoutanychangeinaccommodation.Stillnearerorfartherobjectsare

blurred.Depthoffieldreducesthenecessityforpreciseaccommodationandismarkedlyinfluencedbythediam-eterofthepupil.Thelargerthepupil,thesmallerthedepthoffield.Fora2mmand4mmpupil,thedepthsoffieldare0.06and0.03diopters,respectively.Fora4mmpupil,forexample,suchadepthoffieldtrans-latesasaclearfocusfrom0.94mto1.06mforanobject1maway,andfrom11mto33mforanobject17maway(Campbell,1957;Moses,1970).Animportantpointisthataccommodationplaysanimportantroleonlyatcloseworkingdistances,wheredepthoffieldisnarrow.

Withvideosee-throughsystems,theminiaturecam-erasthatacquirethereal-sceneimagesmustprovideadepthoffieldequivalenttotherequiredworkingdis-tanceforatask.Foralargerangeofworkingdistances,thecameramayneedtobefocusedatthemiddlework-ingdistance.Forcloserdistances,thesmalldepthoffieldmayrequireanautofocusinsteadofafixed-focuscam-era.

Withopticalsee-throughsystems,theavailabledepthoffieldfortherealsceneisessentiallythatofthehumanvisualsystem,butforalargerpupilthanwouldbeacces-siblewithunaidedeyes.Thiscanbeexplainedbythebrightnessattenuationoftherealscenebythehalf-trans-parentmirror.Asaresult,thepupilsaredilated(weas-sumeherethattherealandvirtualscenesarematchedinbrightness).Therefore,theeffectivedepthoffieldis

RollandandFuchs305

slightlylessthanwithunaidedeyes.Thisisaproblemonlyiftheuserisworkingwithnearbyobjectsandthevirtualimagesarefocusedoutsideofthedepthoffieldthatisrequiredfornearbyobjects.Forthevirtualimagesandnoautofocuscapabilityforthe2-Dvirtualimages,thedepthoffieldisimposedbythehumanvisualsystemaroundthelocationofthedisplayedvirtualimages.

Whentheretinalimagesarenotsharpfollowingsomediscrepancyinaccommodation,thevisualsystemiscon-stantlyprocessingsomewhatblurredimagesandtendstotoleratebluruptothepointatwhichessentialdetailisobscured.Thistoleranceforblurconsiderablyextendstheapparentdepthoffieldsothattheeyemaybeasmuchas0.25dioptersoutoffocuswithoutstimulat-ingaccommodativechange(Moses,1970).

3.2.6QualitativeAspects.Therepresentationofvirtualobjects,andinsomecasesofrealobjects,isalteredbysee-throughdevices.Aspectsofperceptualrepresentationincludetheshapeofobjects,theircolor,brightness,contrast,shading,texture,andlevelofdetail.Inthecaseofopticalsee-throughHMDs,foldingtheopticalpathbyusingahalf-transparentmirrorisneces-sarybecauseitistheonlyconfigurationthatleavestherealscenealmostunaltered.Athin,foldingmirrorwillintroduceasmallapparentshiftindepthofrealobjectspreciselyequaltoe(n1)/n,whereeisthethicknessoftheplateandnisitsindexofrefraction.Thisisinaddi-tiontoasmallamountofdistortion(1%)ofthesceneattheedgesofa60deg.FOV.Consequently,realob-jectsareseenbasicallyunaltered.

Virtualobjects,ontheotherhand,areformedfromthefusionofstereoimagesformedthroughmagnifyingoptics.Eachopticalvirtualimageformedofthedisplayassociatedwitheacheyeistypicallyopticallyaberrated.Forlarge-FOVopticssuchasHMDs,astigmatismandchromaticaberrationsareoftenthelimitingfactors.Cus-tom-designedHMDopticscanbeanalyzedfromavisualperformancepointofview(Shenker,1994;Rolland,2000).Suchanalysisallowsthepredictionoftheex-pectedvisualperformanceofHMDusers.

Itmustbenotedthatrealandvirtualobjectsinsuchsystemsmaybeseensharplybyaccommodatingindif-

306PRESENCE:VOLUME9,NUMBER3

ferentplanesundermostvisualizationsettings.Thisyieldsconflictsinaccommodationforrealandvirtualimagery.Forapplicationsinwhichthevirtualobjectsarepresentedinasmallworkingvolumearoundsomemeandisplaydistance(suchasarm-lengthvisualization),the2-Dopticalimagesoftheminiaturedisplayscanbelo-catedatthatsamedistancetominimizeconflictsinac-commodationandconvergencebetweenrealandvirtualobjects.AnotherapproachtominimizingconflictsinaccommodationandconvergenceismultifocalplanestechnologyasdescribedinRollandetal.,2000).

Besidebrightnessattenuationanddistortion,otheraspectsofobjectrepresentationarealteredinvideosee-throughHMDs.Theauthors’experiencewithatleastonesystemisthatthecolorandbrightnessofrealob-jectsarealteredalongwiththelossintextureandlevelsofdetailduetothelimitedresolutionoftheminiaturevideocamerasandthewide-angleopticalviewer.Thisalterationincludesspatial,luminance,andcolorresolu-tion.Thisisperhapsresolvablewithimprovedtechnol-ogy,butitcurrentlylimitstheabilityoftheHMDusertoperceiverealobjectsastheywouldappearwithun-aidedeyes.Inwide-FOVvideosee-throughHMDs,bothrealandvirtualobjectscallforthesameaccommo-dation;however,conflictsofaccommodationandcon-vergencearealsopresent.Aswithopticalsee-throughHMDs,theseconflictscanbeminimizedifobjectsareperceivedatarelativelyconstantdepthneartheplaneoftheopticalimages.Innarrow-FOVsystemsinwhichtherealsceneisseeninlargepartoutsidetheoverlayimag-ery,conflictsinaccommodationcanalsoresultbetweentherealandcomputer-generatedscene.

Forbothtechnologies,asolutiontothesevariousconflictsinaccommodationmaybetoallowautofocusofthe2-Dvirtualimagesasafunctionofthelocationoftheusergazepointinthevirtualenvironment,ortoimplementmultifocalplanes(Rollandetal.,2000).Giveneye-trackingcapability,autofocuscouldbepro-videdbecausesmalldisplacementsoftheminiaturedis-playnearthefocalplaneoftheopticswouldyieldlargeaxialdisplacementsofthe2-Dvirtualimagesinthepro-jectedvirtualspace.The2-Dvirtualimageswouldmoveindepthaccordingtotheusergazepoint.Multifocal

planesalsoallowautofocusingbutwithnoneedforeyetracking.

4Conclusion

Wehavediscussedissuesinvolvingopticalandvideosee-throughHMDs.Themostimportantissuesaresystemlatency,occlusion,thefidelityofthereal-worldview,anduseracceptance.Opticalsee-throughsystemsofferanessentiallyunhinderedviewoftherealenvironment;theyalsoprovideaninstantaneousreal-worldviewthatassuresthatvisualandproprioceptioninformationissynchronized.Videosystemsforfeittheunhinderedviewinreturnforimprovedabilitytoseerealandsyntheticimagerysimultaneously.

Someofusworkingwithopticalsee-throughdevicesstronglyfeelthatprovidingtherealscenethroughopti-calmeansisimportantforapplicationssuchasmedicalvisualizationinwhichhumanlivesareatstake.Others,workingwithvideosee-throughdevicesfeelthataflip-upviewisadequateforthesafetyofthepatient.Also,howtorenderocclusionoftherealsceneatgivenspatiallocationsmaybeimportant.Videosee-throughsystemscanalsoguaranteeregistrationoftherealandvirtualscenesattheexpenseofamismatchbetweenvi-sionandproprioception.Thismayormaynotbeper-ceivedasapenaltyifthehumanobserverisabletoadapttosuchamismatch.Hybridsolutions,suchasthatde-velopedbyPeuchot(1994),includingopticalsee-throughtechnologyforvisualizationandvideotechnol-ogyfortrackingobjectsintherealenvironment,mayplayakeyroleinfuturedevelopmentsoftechnologyfor3-Dmedicalvisualization.

Clearly,thereisno‘‘right’’systemforallapplications:Eachofthetradeoffsdiscussedinthispapermustbeex-aminedwithrespecttospecificapplicationsandavailabletechnologytodeterminewhichtypeofsystemismostappropriate.Furthermore,additionalHMDfeaturessuchasmultiplanefocusingandeyetrackingarecur-rentlyinvestigatedatvariousresearchanddevelopmentsitesandmayprovidesolutionstocurrentperceptual

conflictsinHMDs.AsharedconcernamongscientistsdevelopingfurthertechnologyisthelackofstandardsnotonlyinthedesignbutalsomostimportantlyinthecalibrationandmaintenanceofHMDsystems.

Acknowledgments

ThisreviewwasexpandedfromanearlierpaperinaSPIEpro-ceedingbyRolland,Holloway,andFuchs(1994),andtheau-thorswouldliketothankRichHollowayforhisearliercontri-butiontothiswork.WethankMyronKruegerfromArtificialRealityCorp.forstimulatingdiscussionsonvariousaspectsofthetechnology,aswellasMartinShenkerfromM.S.O.D.andBrianWelchfromCAEElectronicsfordiscussionsoncurrentopticaltechnologies.Finally,wethankBernardPeuchot,DerekHill,andAndreiStateforprovidinginformationabouttheirresearchthathassignificantlycontributedtotheimprovementofthispaper.Wedeeplythankourvarioussponsorsnotonlyfortheirfinancialsupportthathasgreatlyfacilitatedourre-searchinsee-throughdevicesbutalsoforthestimulatingdis-cussionstheyhaveprovidedovertheyears.ContractsandgrantsincludeARPADABT63-93-C-0048,NSFCooperativeAgreementASC-20219;ScienceandTechnologyCenterforComputerGraphicsandScientificVisualization,ONRN00014-86-K-0680,ONRN00014-94-1-0503,ONRN000149710654,NIH5-R24-RR-02170,NIH1-R29LM06322-O1A1,andDAAH04-96-C-0086.

References

Allen,D.(1993).Ahighresolutionfieldsequentialdisplayforhead-mountedapplications.Proc.IEEEvirtualrealityan-nualinternationalsymposium(VRAIS’93)(pp.3–370).Azuma,R.,&Bishop,G.(1994).Improvingstaticanddy-namicregistrationinanopticalsee-throughHMD.Com-puterGraphics:ProceedingsofSIGGRAPH’94(pp.197–204).

Bajura,M.,Fuchs,H.,&OhbuchiR.(1992).Mergingvirtualobjectswiththerealworld,ComputerGraphics,26,203–210.

Bajura,M.,&NewmannH.(1995).Dynamicregistrationcor-rectioninvideo-basedaugmentedrealitysystems.IEEEComputerGraphicsandApplications,15(5),52–60.

RollandandFuchs307

Baillot,Y.,&Rolland,J.P.(1998).ModelingofakneejointfortheVRDAtool.Proc.ofMedicineMeetsVirtualReality(pp.366–367).

Baillot,Y.,Rolland,J.P.,&Wright,D.(1999).Kinematicmodelingofknee-jointmotionforavirtualrealitytool.InProc.ofMedicineMeetsVirtualReality(pp.30–35).Baillot,Y.,Rolland,J.P.,Lin,K.,&Wright,D.L.(2000).Au-tomaticmodelingofknee-jointmotionfortheVirtualReal-ityDynamicAnatomy(VRDA)Tool.Presence:TeleoperatorsandVirtualEnvironments,9(3),223–235.

Baillot,Y.(1999).FirstimplementationoftheVirtualRealityDynamicAnatomy(VRDA)tool.Unpublishedmaster’sthe-sis,UniversityofCentralFlorida.

Barrette,R.E.(1992).Widefieldofview,full-color,high-resolution,helmet-mounteddisplay.Proc.ofSID’92Sympo-sium(pp.69–72).

Biocca,F.A.,&Rolland,J.P.(1998).Virtualeyescanrear-rangeyourbody:Adaptationtovirtual-eyelocationinsee-thruhead-mounteddisplays.Presence:TeleoperatorsandVir-tualEnvironments,7(3),262–277.

Bishop,G.,Fuchs,H.,McMillan,L.,&Scher-Zagier,E.J.(1994).Framelessrendering:Doublebufferingconsideredharmful.Proc.ofSIGGRAPH’94(pp.175–176).

Brooks,F.P.(1992).Walkthroughproject:Finaltechnicalre-porttoNationalScienceFoundationComputerandInfor-mationScienceandEngineering.TechnicalReportTR92-026,UniversityofNorthCarolinaatChapelHill.Buchroeder,R.A.,Seeley,G.W.,&Vukobratovich,D.

(1981).DesignofacatadioptricVCASShelmet-mounteddisplay.OpticalSciencesCenter,UniversityofArizona,un-dercontracttoU.S.AirForceArmstrongAerospaceMedi-calResearchLaboratory,Wright-PattersonAirForceBase,Dayton,Ohio,AFAMRL-TR-81-133.

Campbell,F.W.(1957).Thedepthoffieldofthehumaneye.OpticaActa,4,157–1.

Cutting,J.E.,&Vishton,P.M.(1995).Perceivingthelayoutandknowingdistances:Theintegration,relativepotency,andcontextualuseofdifferentinformationaboutdepth.InW.Epstein&S.Rogers(Eds.),PerceptionofSpaceandMo-tion(pp.69–117).AcademicPress.

Deering,M.(1992).Highresolutionvirtualreality.Proc.ofSIGGRAPH’92,ComputerGraphics,26(2),195–201.Desplat,S.(1997).CharacterizationdeselementsactualsetfuturedeloculometreMetrovision-Sextant.TechnicalRe-port,EcoleNationaleSuperieuredePhysiquedeMarseilles,November.

308PRESENCE:VOLUME9,NUMBER3

Dhome,M.,Richetin,M.,Lapreste,J.P.,&Rives,G.(19).Determinationoftheattitudeof3-Dobjectsfromasingleperspectiveview.IEEETrans.PatternAnalysisandMachineIntelligence,11(12),1265–1278.

Droessler,J.G.,&Rotier,D.J.(1990).Tiltedcathelmet-mounteddisplay.OpticalEngineering,29(8),849–854.Edwards,E.K.,Rolland,J.P.,&Keller,K.P.(1993).Videosee-throughdesignformergingofrealandvirtualenviron-ments.Proc.ofIEEEVRAIS’93(pp.223–233).

Edwards,P.J.,Hawkes,D.J.,Hill,D.L.G.,Jewell,D.,Spink,R.,Strong,A.,&Gleeson,M.(1995).Augmentationofrealityusinganoperatingmicroscopeforotolaryngologyandneurosurgicalguidance.J.ImageGuidedSurgery,1(3),172–178.

Ellis,S.R.,&Bucher,U.J.(1994).Distanceperceptionofstereoscopicallypresentedvirtualobjectsopticallysuperim-posedonphysicalobjectsinahead-mountedsee-throughdisplay.Proc.oftheHumanFactorsandErgonomicSociety,Nashville.

Fernie,A.(1995).A3.1Helmet-mountedDisplaywithDualResolution.CAEElectronics,Ltd,Montreal,Canada,SID95ApplicationsDigest,37–40.

Fernie,A.(1995).Improvementsinareaofinteresthelmet-mounteddisplays.InProc.ofSID95.

Fry,G.A.(1969).GeometricalOptics.ChiltonBookCompany.Furness,T.A.(1986).Thesupercockpitanditshumanfactorschallenges.ProceedingsoftheHumanFactorsSociety30,48–52.

Held,R.,&Durlach,N.(1987).Telepresence,timedelayandadaptation.NASAConferencepublication10032.

Holloway,R.(1995).Ananalysisofregistrationerrorsinasee-throughhead-mounteddisplaysystemforcraniofacialsur-geryplanning.Unpublisheddoctoraldissertation,UniversityofNorthCarolinaatChapelHill.

Hua,H.,Girardot,A.,Gao,C.,andRolland,J.P.(2000).En-gineeringofhead-mountedprojectivedisplays.AppliedOp-tics(inpress).

Jacobs,M.C.,Livingston,M.A.,&State,A.(1997).Manag-inglatencyincomplexaugmentedrealitysystems.Proc.of1997SymposiumonInteractive3DGraphics,ACMSIGGRAPH,235–240.

KaiserElectro-Optics.(1994).PersonalcommunicationfromFrankHepburnofKEO,Carlsbad,CA.GeneraldescriptionofKaiser’sVIMsystem(‘‘Fullimmersionhead-mounteddisplaysystem’’)isavailableviaARPA’sESTOhomepage:http://esto.sysplan.com/ESTO/.

Kancherla,A.,Rolland,J.P.,Wright,D.,&Burdea,G.(1995).Anovelvirtualrealitytoolforteachingdynamic3Danatomy.Proc.ofCVRMed’95,163–169.

Kandebo,S.W.(1988).NavytoevaluateAgileEyehelmet-mounteddisplaysystem.AviationWeek&SpaceTechnology,August15,94–99.

Kennedy,R.S.,&Stanney,K.M.(1997).Aftereffectsinvir-tualenvironmentexposure:Psychometricissues.InM.J.Smith,G.Salvendy,&R.J.Koubek(Eds.),DesignofCom-puting.

Kijima,R.,&Ojika,T.(1997).Transitionbetweenvirtualen-vironmentandworkstationenvironmentwithprojectivehead-mounteddisplay.Proc.ofVRAIS’97,130–137.

Manhart,P.K.,MalcomR.J.,&Frazee,J.G.(1993).Augeye:Acompact,solidSchmidtopticalrelayforhelmetmounteddisplays.Proc.ofIEEEVRAIS’93,234–245.Mizell,D.(1998).Personnalcommunication.

Moses,R.A.(1970).AdlersPhysiologyoftheEye.St.Louis,MO:Mosby.

Oster,P.J.,&Stern,J.A.(1980).‘‘MeasurementofEyeMovement,’’inI.Martin&P.H.Venables(Eds.),Tech-niquesofPsychophysiology,NewYork:JohnWiley&Sons.Outters,V.,Argotti,Y.,&Rolland,J.P.(1999).Kneemotioncaptureandrepresentationinaugmentedreality.TechnicalReportTR99-006,UniversityofCentralFlorida.

Parsons,J.,&Rolland,J.P.(1998).Anon-intrusivedisplaytechniqueforprovidingreal-timedatawithinasurgeonscriticalareaofinterest.Proc.ofMedicineMeetsVirtualReal-ity,246–251.

Peuchot,B.(1993).Cameravirtualequivalentmodel:0.01pixeldetectors.Specialissueon3DAdvancedImageProcess-inginMedicineinComputerizedMedicalImagingandGraphics,17(4/5),2–294.

———.(1994).Utilizationdedetecteurssubpixelsdanslamodelisationd’unecamera—verificationdel’hypothesestenope,9eCongresAFCET,reconnaissancedesformesetin-telligenceartificielle,Paris.

———.(1998).Personalcommunication.

Peuchot,B.,Tanguy,A.,&Eude,M.(1994).Dispositifop-tiquepourlavisualizationd’uneimagevirtuelletridimen-sionelleensuperimpositionavecanobjectnotammentpourdesapplicationschirurgicales.DepotCNRS#94106623,May31.

———.(1995).Virtualrealityasanoperativetoolduringsco-liosissurgery.Proc.ofCVRMed’95,549–554.

Robinett,W.,&Rolland,J.P.(1992).Acomputationalmodelforthestereoscopicopticsofaheadmounteddisplay.Pres-ence:TeleoperatorsandVirtualEnvironments,1(1),45–62.

Rock,I.(1966).Thenatureofperceptualadaptation.NewYork:BasicBooks.

Rolland,J.P.(1994).Head-mounteddisplaysforvirtualenvi-ronments:Theopticalinterface.InternationalOpticalDe-signConference94,Proc.OSA,22,329–333.

Rolland,J.P.(1998).Mounteddisplays,OpticsandPhotonicsNews,9(11),26–30.

Rolland,J.P.(2000).Wideangle,off-axis,see-throughhead-mounteddisplay.OpticalEngineering—SpecialIssueonPushingtheEnvelopinOpticalDesignSoftware,39(7),(inpress).

Rolland,J.P.,Ariely,D.,&Gibson,W.(1995).Towardsquan-tifyingdepthandsizeperceptioninvirtualenvironments.Presence:TeleoperatorsandVirtualEnvironments,4(1),24–49.

Rolland,J.P.&Arthur,K.(1997).Studyofdepthjudgmentsinasee-throughmounteddisplay,ProceedingofSPIE3058,AEROSENSE,66–75.

Rolland,J.P.,Holloway,R.L.,&Fuchs,H.(1994).Acom-parisonofopticalandvideosee-throughhead-mounteddis-plays.ProceedingsofSPIE2351,293–307.

Rolland,J.P.,&Hopkins,T.(1993).Amethodforcomputa-tionalcorrectionofopticaldistortioninhead-mounteddis-plays.TechnicalReportTR93-045,UniversityofNorthCarolinaatChapelHill.

Rolland,J.P.,Krueger,M.,andGoon,A.(2000).Multifocusplanesinhead-mounteddisplays,AppliedOptics,39(19),inpress.

Rolland,J.P.,Parsons,J.,Poizat,D.,&Hancock,D.(1998).Conformalopticsfor3Dvisualization,Proc.oftheInterna-tionalLensDesignConference,Hawaii(June),760–7.Rolland,J.P.,Yoshida,A.,Davis,L.,&Reif,J.H.(1998).High-resolutioninsethead-mounteddisplay,AppliedOptics,37(19),4183–4193.

Roscoe,S.N.(1984).Judgmentsofsizeanddistancewithim-agingdisplays,HumanFactors,26(6),617–629.

Roscoe,S.N.(1991).Theeyespreferrealimages,inPictorialCommunicationinVirtualandRealEnvironments,Ed.StephenR.Ellis,TaylorandFrancis.

Scher-Zagier,E.J.(1997).Ahuman’seyeview:motion,blur,andframelessrendering.ACMCrosswords97.

Shenker,M.(1994).Imagequalityconsiderationsforhead-mounteddisplays.Proc.oftheOSA:InternationalLensDe-signConference,22,334–338.

Shenker,M.(1998).PersonalCommunication.

Slater,M.,andWilbur,S.(1997).Aframeworkforimmersivevirtualenvironments(FIVE):Speculationsontheroleof

RollandandFuchs309

presenceinvirtualenvironments,Presence:TeleoperatorsandVirtualEnvironments,6(6),603–616.

State,A.,Chen,D.,Tector,C.,Brandt,C.,Chen,H.,Ohbu-chi,R.,Bajura,M.,andFuchs,H.(1994).Casestudy:Ob-servingavolumerenderedfetuswithinapregnantpatient,ProceedingsofVisualization’94.Washington,DC.,3–373.State,A.,Hirota,G.,Chen,D.T.,Garrett,W.E.,andLiving-ston,M.(1996).Superioraugmented-realityregistrationbyintegratinglandmarktrackingandmagnetictracking.Proc.ofSIGGRAPH1996,ACMSIGGRAPH,429–438.

Sutherland,I.(1965).Theultimatedisplay,InformationPro-cessing1965:Proc.ofIFIPCongress65,506–508.Sutherland,I.E.(1968).Ahead-mountedthree-dimensionaldisplay,FallJointComputerConference,AFIPSConferenceProceedings33,757–7.Thomas,M.L.,Siegmund,W.P.,Antos,S.E.,andRobinson,R.M.(19).Fiberopticdevelopmentforuseonthefiberoptichelmet-mounteddisplay,inHelmet-MountedDisplays,J.T.Colloro,Ed.,Proc.ofSPIE1116,90–101.Tomasi,C.,&Kanade,T.(1991).Shapeandmotionfromim-agestreams:afactorizationmethod-Part3:Detectionandtrackingofpointfeatures,CarnegieMellontechnicalreportCMU-CS-91-132.

Vaissie,L.,&Rolland,J.P.(1999).Eye-trackingintegrationinhead-mounteddisplays,PatentFiledJanuary.Vaissie,L.,Rolland,J.P.(2000).Alberlí´anerrorsinhead-mounteddisplays:choiceofeyejointslocation.

TechnicalReportTR2000-001,UniversityofCentralFlorida.

Watson,B.,&Hodges,L.F.(1995).Usingtexturemapstocorrectforopticaldistortioninhead-mounteddisplays,Proc.ofVRAIS’95,172–178.

Welch,B.,&Shenker,M.(1984).Thefiber-opticHelmet-MountedDisplay,ImageIII,345–361.

Welch,R.B.(1993).Alternatingprismexposurecausesdualadaptationandgeneralizationtoanoveldisplacement.Per-ceptionandPsychophysics,54(2),195–204.

Welch,R.B.(1994).Adaptingtovirtualenvironmentsandteleoperators,Unpublishedmanuscript,NASA-AmesRe-searchCenter,MoffettField,CA.Welch,G.,&Bishop,G.(1997).SCAAT:Incrementaltrack-ingwithincompleteinformation,Proc.ofACMSIGGRAPH,(pp.333–344).

Wright,D.L.,Rolland,J.P.,&Kancherla,A.R.(1995).Us-ingvirtualrealitytoteachradiographicpositioning,Radio-logicTechnology,66(4),167–172.