A Realstc E-Learg System based o Mxed Realty Kyusug Cho 1, Juho Lee 1, Jaem Soh 1, Juseok Lee 2, Hyu S. Yag 1 1 Departmet of Computer Scece, Korea Advaced Isttute of Scece ad Techology, 373-1 Guseog-dog, Yuseog-gu, Daejeo 305-701, Republc of Korea 2 Electrocs ad Telecommucatos Research Isttute, 138 Gajeogo, Yuseog-gu, Daejeo 305-700, Republc of Korea {qtboy, jhlee, jmsoh}@paradse.kast.ac.kr, leejs@etr.re.kr, hsyag@paradse.kast.ac.kr Abstract. I ths paper, we descrbe a E-Learg system based o the mxed realty. Mxed realty provdes users wth mmerso, presece, ad addtoal formato sutable for the curret cotext. That pot results may researches applyg mxed realty to educato feld. However, they have bee just prototype systems or expermetal systems. I cotrast wth the prevous works, our system s the total soluto for a mxed realty based E-learg system at commercal level. It has three large compoets: the clet system, the learg maagemet system ad the authorg tool. It also requres the robust marker recogto for commercalzato. I ths research, we focus o ths part. Wth ths system, we have coducted the user study wth K-4 ad K-5 studets. As a result, we could cofrm that our system ehaces the studets terest lesso ad the educatoal achevemet. Keywords: Augmeted Realty, Mxed Realty, E-Learg 1 Itroducto Ths paper descrbes a mxed realty based E-Learg system for ehacg the learg effect wth mmersve ad realstc sese. Mxed realty ca offer ew formats compared to tradtoal educatoal strateges of teachg, readg, ad hearg. It also allows studets to become a tegral part of ther ow learg process, to allow them to experece ew systems stead of smply readg or hearg, ad moreover to preted to work ad lear varous stuatos wthout rsk. Some mxed realty applcatos have already bee preseted educato feld. They have bee just prototype systems or expermetal systems. However, our system s the total soluto tegratg the clet system, the authorg system, ad the LMS (Learg Maagemet System) o the web. Fg 1 shows the cofgurato of a mxed realty based the E-Learg system. Teachers ca make teachg materals wth the authorg system ad upload them to the LMS system. The role of the LMS system s to maage teachg materals ad users accouts/actvtes by commucatg wth the clet system. The core part of the whole system s the clet system whch reders 3D models of teachg materals, teracts wth terface devces ad recogzes markers real books. These markers
are used for locatg ad detfyg the redered 3D models. Markers are classfed to two types: oe s for locatg 3D models ad the other s for hadlg 3D models. We call the former fxed marker ad the latter cotrol marker. I order to commercalze a mxed realty based system, the marker recogto process should be very robust to varous evromets ad users ufamlar wth mxed realty. Fg. 1. The cofgurato of a E-Learg system based o Mxed Realty 2 Prevous Work The Magc Book [4] s well kow for the frst replacemet of the real books by the realstc ad mmersve learg materals. The Magc Book was created by HIT Lab. at the Uversty of Caterbury. The Magc Book looks lke a ordary storybook wth colorful pages ad smple text. However, whe users look at the same pages through a HMD, 3D vrtual objects emerge from the pages. You ca stll see the real world book, as well as everythg that s gog o aroud you, just except a lttle vrtual scee whch sts o the page. Shelto ad Hedley [5] preseted the research to teach studets the relatoshp betwee the su ad the earth usg AR. I ths research, the mportat thg s to show ther relatve postos space. The AR techology allows users to uderstad deeply ad easly about ther relatoshp because AR s approprate for showg 3D relatoshp. Several years ago, researchers of Electroc Vsualzato Laboratory at the Uversty of Illos at Chcago stalled the CAVE system for vrtual realty at Abraham Lcol Elemetary School. The Roud Earth Project [6] cluded the user study whch verfes whether the vrtual realty techology s a effectve teachg tool for studets or ot. Though the project was orgazed wth the vrtual realty techology, t gave us strog possblty to apply a ew techology to the teachg method for K-12 studets. The dfferece betwee our system ad all related works s that our system has the total soluto for a E-Learg System based o Mxed Realty so that t ams at commercalzato. Our paper s orgazed as follows: I secto 3 we wll descrbe the proposed system archtecture ad desg scheme. I secto 4 we wll focus o descrbg the
marker recogto process as the core part for the mxed realty. I secto 5 we wll show the results ad fally coclude secto 6. 3 The Archtecture of the Proposed System As show Fg 1, the E-Learg system based o the mxed realty techology cossts of 3 large compoets: the clet System, the learg maagemet system, ad the authorg system. I ths secto, we descrbe the detal archtecture ad fuctos of each compoet 3.1 The Clet System The Clet System based o the mxed realty techology s the core part of the total system. It s composed of the Maker Recogzer, the Realstc Cotets Maager, ad the Vewer. Whe a sequece of mages s etered from a camera, the Marker Recogzer coducts the marker recogto process for detfyg markers ad estmatg the pose of markers (6 DOF: 3 for traslato ad 3 for rotato). The Realstc Cotets Maager maages 3D models, textures, mappg, soud, vdeo, ad so forth. It s a very mportat module for dyamc ad real tme serto ad deleto to the scee graph. The last module the Clet System s Vewer. The scee graph s represeted by VRML format ad redered wth OpeGL graphcs ege. It also reders the soud ad vdeo data as well as the graphc data. (Fg 2. left) 3.2 The LMS (Learg Maagemet System) The role of the LMS s to maage the realstc cotets ad the users accoutg ad proflg data durg learg ad teachg. Studets log the LMS usg ther IDs ad passwords, ad the the Clet System dowloads the realstc cotets from the LMS. After that, studets ca study wth the clet system ad studets actvtes are trasmtted to the LMS durg study. These data play a crucal role the evaluato of educatoal achevemet. (Fg 2. mddle) 3.3 The Authorg System The Authorg System s the tool for creatg ad edtg the teachg materals. 3D cotets are created by usg a 3D modeler such as 3D MAX or MAYA ad mported to the Authorg System by the customzed VRML mport plug-. Wth the Authorg tool, teachers ca select some requred markers ad create the lks betwee the markers ad the realstc cotets cludg 3D models, moves ad audos. Moreover, the tool supports the acto scrpt, so that varous complcated scearos appear to be possble. For example, the realstc cotets may be rotated or magfed whe a certa cotrol marker s close to them, or a certa move ad
audo may be played f the realstc cotets s moved ad located at the appoted posto. (Fg 2. rght) I the authorg system, the cosderable pot s the user terface of the tool because ovces or teachers ufamlar wth IT may hadle ths system. Clet System Realstc Marker Recogzer Cotets Maager LMS Authorg Realstc Cotets Marker Acto Marker Flterg Realstc Cotets Loader Dsplay Aalyzer Maager of Realstc Cotets Realstc Profle Cotets Marker Lk Marker Cotets Wth Dsplay Creato Cotets Iformato Marker Creator Extracto of Camera Learg Tools parameters Vewer Realstc Cotets Wdows Vewer VRML Parser Loggg & Accout Maager Meta Data Edtor VRML based Smulato Edtor VRML Vewer VRML Parser Import from Modeler Fg. 2. (left)the System Archtecture for Clet System, (mddle)lms ad (rght)authorg System 4 Marker Recogto Process The system obtas the mage cludg some markers by a camera ad detects quadragles from the mage. Those are caddates of markers. To classfy real markers amog caddates, the system recogzes patters of quadragles. The caddate cludg a patter ID s to be accepted as a real marker, ad the pass the pose estmato process. The system returs the projecto matrces ad the patter IDs of the markers as fal results. Those are used to supermpose some vrtual objects oto the mage captured by a camera. 4.1 Quadragle Detecto The represetatve marker recogto system, ARToolkt [1] performs mage barzato based o the fxed threshold to detect quadragles. Ths method, however, has a weakess for varat lghtg codtos. Due to that problem, some marker recogto systems determe the threshold adaptvely. Ufortuately, those also have a weakess for ueve lghtg codtos such as shadows. To overcome those problems, we use the edge-based method, whch s more robust to varat lghtg
codtos ad partal occluso. Fg 3 shows that the system s able to detect quadragles eve partal occluso ad dark lghtg codto. Fg. 3. Quadragle Detecto varous evromets 4.2 Patter Recogto As metoed above, the system eeds to support systematc codg of marker patters correspodg to may pages of textbooks. Moreover, t s able to recogze varous coc patters; we also defe total 256 fxed patters whch are appled to systematc codg, ad allow users to defe ther ow patters whch help users to otce the meag of markers tutvely. Fg 4 shows some fxed patters ad coc patters. Fg. 4. Fxed ad Icoc Patters Markers have a black boudary le wth a square shape, whch a patter s cluded. A fxed patter s dvded to the 9 secto. The upper left oe (the drecto bt) s most mportat amog the 9 sectos. It elmates symmetry of markers ad moreover allows the system to dstgush betwee the fxed patters ad the coc patters. The other 8 sectos represet a patter ID, ad correspod to 8 bts the bary system. If oe amog them s flled wth black, the correspodg bt s to be 1, otherwse, 0. For example, f the 5th ad the 6th bts of the fxed patter are flled, the patter represets 0011000 the bary system, whch s 48 the decmal system. To recogze patters the system fds the drecto bt the detected quadragle. If there exsts the drecto bt, the patter the quadragle s decoded to fd the
patter ID, otherwse, t s lkely to have a coc patter. To fd a ID of a coc patter, the patter s compared wth the patter database. PCA s used to reduce the dmeso of patters. If there does ot exst a smlar patter the patter database, the quadragle s elmated from caddates of markers. The mportat thg of patter recogto s to reduce the false-postve rate ad the cofuso of patters. If there are errors patter recogto, t occurs that the system supermposes udesrable cotets oto the mage. That s fatal to the commercalzg a mxed realty system. I ths research, color ad edge formato are used to reduce errors patter recogto process. 4.3 Pose Estmato I ths research, we use a object-space collearty error as the error metrc [7]. Error 4 = = 1 vˆ vˆ where Vˆ = vˆ ( I Vˆ )( Rp + t), t t vˆ 2 (1) I the error, vˆ s the observed mage pot, Vˆ s the observed le-of-sght projecto matrx, ad p s the model pot. R ad t meas rotato ad traslato of a camera respectvely. The more accurate the system estmates the pose of the camera, the less the object-space collearty error. Fg. 5. The frst pose ad the secod pose Our pose estmato algorthm s motvated by the Schweghofer ad Pz s work [2]. They poted out that there exst two local mma of equato (1). We foud that θ 1 ad θ 2 are smlar, where θ 1 represets the agle betwee the camera drecto ad the ormal of the marker s pose wth the frst local mmum, ad θ 2 represets that wth the secod oe (Fg 5). We assume therefore θ 1 θ 2. To calculate the marker s pose wth the frst local mmum, the system calculates the tal pose by usg drect lear method [3]. To mmze (1), The orthogoal terato method s used [7], where the above result s used as the tal value. By the assumpto, θ 1 θ 2, we could roughly trasform the frst pose to the secod oe.
The trasformato matrx, M s calculated by rotatg 2θ 1 roud the ormal vector of the plae cludg 1 ad z, where z meas the drecto of the camera vew. 2 M I + s(2θ1 )A + (1 cos(2θ 1 ))A, 0 r3 r2 where A = r3 0 r1, r2 r1 0 ad = z = (2) r 1 [ ] The marker s pose wth the secod local mmum s calculated smply by P 2 = MP 1, where P 1 ad P 2 are the projecto matrx of the frst pose ad the secod pose, respectvely. The object-space collearty error of the secod pose s also mmzed by the orthogoal terato method lke the frst pose. Lastly, the objectspace collearty error of the frst pose s compared wth that of the secod oe. The pose wth the less error both s to be the fal pose of the marker. r1 r2 r3 = r r, 5 Results We developed a textbook for elemetary school scece ad Eglsh for the user study, ad had some lessos for 4 ad 5 grade studets of elemetary school. The scece textbook cotas 4 books ad they clude the structure of the earth, the atmosphere, astroomy, ad so forth. The Eglsh textbook takes am at dstgushg L ad R proucatos. Fg 6. left shows two kds of textbooks. We also developed teachg materals usg the authorg tool (Fg 6. mddle). They were uploaded to the LMS, so studets could dowload. Fg 6. rght shows how the vewer of the clet system looks. Fg. 6. (left)the Eglsh textbook, (mddle)the authorg tool ad (rght)the vewer The real class work was held at the Chrsta elemetary school Suwo. Total 180 studets of 4 ad 5 grades took the class the computer lab of the school, ad made the worksheet for surveyg the attamet. As a result, almost teachers ad studets were very terested mxed realty whch s ew ad mmersed the system. The measuremet of learg was also hgh. However, sometmes people felt hard to cotrol 3D cotets wth cotrol markers because of the shortage of the tuto. Ths otced us that we eed more tutve terface better tha the cotrol marker for ext verso.
6 Cocluso We developed a commercal-level realstc E-Learg system based o mxed realty. We also suggested a total soluto cludg producto ad dstrbuto of the realstc cotets, ot just for showg. The Authorg System was developed to be easy to make cotets for ovces lke teachers who are ufamlar wth IT. The LMS s for dstrbutg the realstc cotets, aalyzg the studets actvtes, ad evaluatg the educatoal achevemet. To develop a E-Learg system based o mxed realty at commercal level, robust marker recogto s eeded. It must recogze markers users varous evromets, ad coduct the stable pose estmato of small-szed markers because users feel uatural ad ucomfortable wth the prted markers. It also eeds systematc codg for marker patters correspodg to may pages of textbooks, ad coc patters whch gve users tuto. Our system satsfes these codtos. Usg the proposed system, the user study wth may K-4 ad K-5 studets was coducted. For the user study, we made commercal level test books ad teachg materals by usg the authorg tool. As a result of the study, t s foud that the mxed realty techology helps to draw more studets terest, ad ehace the educatoal achevemet. Ackowledgmets. Ths research s supported by the ubqutous Computg ad Network (UCN) Project, the Mstry of Iformato ad Commucato (MIC) 21st Cetury Froter R&D Program Korea ad the Korea Scece ad Egeerg Foudato (KOSEF) through the Advaced Iformato Techology Research Ceter (AITrc). Refereces 1. http://www.htl.washgto.edu/artoolkt/ 2. Schweghofer, G. ad Pz, A.: Robust Pose Estmato from a Plaar Target, IEEE Trasactos o Patter Aalyss ad Mache Itellgece, vol. 28, o. 12, (2006) 2024-2030. 3. Hartley, R. ad Zsserma, A.: Multple vew geometry computer vso, Cambrdge Uversty Press, 2d edto, (2003). 4. Bllghurst, M., Kato, H., Poupyrev, I., The Magc Book: A Iterface that Moves Seamlessly Betwee Realty ad Vrtualty, IEEE Computer Graphcs ad Applcatos, (2001) 2-4. 5. Shelto, B., Hedley, N., Usg AR for teachg earth-su relatoshps to udergraduate geography studets, Frst IEEE Itl. Augmeted Realty Toolkt Workshop, Germay, (2002). 6. Johso, A., Moher, T., Ohlsso, S., Gllgham, M., The Roud Earth Project: Deep Learg a Collaboratve Vrtual World, Proc. of IEEE VR99, Housto TX, (1999) 164-171. 7. Lu, C., Hager, G., ad Mjolsess, E., Fast ad Globally Coverget Pose Estmato from Vdeo Images, IEEE Trasactos o Patter Aalyss ad Mache Itellgece, vol. 22, o. 6, (2000) 601-622.