Training of Elman networks and dynamic system modelling

Transcription

1 l teratioal Joural of Systets Sciece, i996, volume 27, umber 2, pages 221,226 Traiig of Elma etworks ad dyamic system modellig D. T. Pset ad X. Llut A dyamic backpropagatio (DBP) algorithm is preseted to trai the Elma etwork to model dyamic systems. The relatioship betwee the Elma etwork traied by the DBP algorithm ad the modified Elma etwork preuiously proposed by the authors is clarified. The paper shows that the modifed Elta etwork is a approxfiate realizatio of the Elma etwork traied by the DBP algorithr. t is the selffeedback iks of the cotext uits of the rodified Elma etwork which prouide a dyamic trace of the gradiets i the parameter space ad eable the etwork to model dyamic systems of orders higher tha oe. The paper first giues the results of modellig q secodorder liear plat ad a thirdorder liear plat. N either plat could be modelled usig Elma etworks traied by the stadard backpropagatio algorithm, but both were successfully modelled by DBPtraied Elma etworks as they had bee i preuious studies by modifed Elma etworks. Fially, the paper reports o the applicatio of the DBPtraied Elma et to model a bechmark oliear process. 1. troductio The Elma etwork (Elma 1990) is a type of recurret etwork. There has bee much research iterest i this etwork (Pham ad Liu 1,992, Hertz et a. 1991, Kua 1989, Kua et al. 1989) ad it has bee built ito the MATLAB software (The Math Works c. 1989) ad applied to dyamic system idetificatio (Pham ad Liu i992) ad fiacial predictio (Kamijo ad Taigawa 1990). A modified Elma etwork has bee proposed by Pham ad Liu (1992) because it was foud that the basic Eima etwork traied by the stadard backpropagatio (BP) aigorithm (Rumelhart ad McClellad 1986) was abie to model oly firstorder dyamic systems. This paper describes the dyamic BP (DBP) algorithm which is proper for traiig the basic Elma etwork ad shows that the modified Eima etwork is a approximatio of the Elma etwork traied by DBP. This further clarifies why the modified Elma etwork ca model higherorder dyamic systems. 2. The Elma etwork Elma (1990) itroduced a simpie recurret eural etwork, as show i Fig. 1 (the etwork is assumed to have Received 16 August 1995, Accepted 22 August t telliget Systems Research Laboratory, School of Egieerig, Uiversity of Wales, Cardiff, UK. oly oe iput uit ad oe output uit i this case). This etwork is very similar to the etwork proposed by Robiso ad Fallside (1987). The latter possesses a structure whose coectios are defied i the same way as that employed by Werbos (1990). By forcig some weights to zero, the etworks give by Elma (1990) ad Robiso ad Fallside (1987) have the same structure. The work of this paper follows the feedforward layered structure defied by Elma (1990). From Fig. 1 it ca be see that i a basic Elma etwork, i additio to the iput uit, the hidde uits Cotext uits Output uit ' 'fa L o f) Hidde uits f) l.,o Figure 1. Basic Elma etwork. tput uit 0020'7121i96 Sl:.00 e 1996 Taylor & Fracis Ltd

2 222 D. T. Pham ad X. Liu ad the output uit, there are also cotext uits. The iput ad output uits iteract with the outside eviromet, whereas the hidde ad cotext uits do ot. The iput uit is oly a buffer uit which passes the sigals without chagig them. The output uit is a iiear uit which sums the sigals fed to it. The hidde uits ca have liear or oiiear activatio fuctios. The cotext uits are used oly to memorise the previous activatios of the hidde uits ad ca be cosidered to luctio as oestep time delays. The feedforward coectios (dotted lies) are modifiable; the recurret coectios (solid lies) are fixed. Because the recurret coectios are fixed, the Elma etwork is sometimes cailed a partially recurret etwork. At a specific time k, the previous activatios of the hidde uits (at time k 1) ad the curret iput (at time /<) are used as iputs to the etwork. At this stage, the etwork acts as a feedforward etwork ad propagates these iputs forward to produce the output. The stadard backpropagatio learig rule (Rumeihart ad McClellad 1986) ca the be errployed to trai the etwork. After tlris traiig step the activatios of the hidde uits at time k are set back through the recurret liks to the cotext uits ad saved there for the ext traiig step (time k + l). At the begiig of the traiig process, the activatios ofthe hidde uits are ukow. Usuaily, they are set to oehalf of their maximum rage. For a sigmoidal activatio fuctio the iitial values ca be set to 0'5. For a hyperbolic taget activatio fuctio they ca be equated to 0.0. Fig. 1 the exterai iput to the etwork is represeted by u(k) ad the etwork output by. The total iput to the lth hidde uit is deoted as u,(k). The output of the ith hidde uit is deoted as x,(k). The output olthe jth cotext uit is xj(ft). The followig equatios hold: u,(k) :,i;(k j= t' x,(k): f(u,), xj(k):x/k1), : witk l)r,(k). i= 1 l)x,r(k) + w'i(k l)u(k), (a) (1 b) (1 c) (t d) where uri('),,i;( ) ad wi( ), t, j:1,2,...,, are the weights ol the liks, respectively, betwee the iput uit ad the hidde layer, betwee the cotext layer ad the hidde layer, ad betwee the hidde layer ad the output uit. / is a sigioidal activatio fuctio. particuiar, if iput a(k) is delayed by oe time step before it is set to the iput uit, xj(k) is replaced by 1(,k 1), ad the hidde uits are assumed to be liear. the above equatios become il u,(k) : wir(/< j=t x;1k): u;(k), l(a): wit/< l;.r,rk). l)x;(k 1) + 14,;'(k l)u(k 1). (2,,) (2h) (2c) Equatios (2a)(2c) are the statespace descriptio q1 a thorder iiear dyamic system, where ri is llis dimesio of t(ft) : {.",(/,)}, that is the umber of hidde/ cotext uits. The order ol the model depeds o the umber ol states, which is also the umber o[ hidde uits. Equatios (2 a)(2c) ca be expaded ito the foliowig: : AJ(k 1) + Ary(k 2) + t Bru(k 1) + Bru(k 2) + f A,y(k ) 1 B,u(k ). (3) Therefore, theoretically a Elma etwork is abie to model a thorder dyamic system if it ca be traied to do so. To model a system represeted by (3) usig iputoutput data, 2 iput uits would be eeded if a feedlorward eurai etwork is used. For a Elma etwork the iput uit umber is oe or + if the cotext uits are regarded as iput uits. Thus a Elma etwork will be sigificatly smaller i structure tha a feedforward etwork whe is large. 3. Traiig of the Elma etwork The leedback vector r"(k): {.*i(ft)} is x(k 1) which is a fuctio of w'(k 2)x(k 2) + w'(k 2)u(k 2). Therefore, x'(k) depeds o the weights of previous time istats. Whe the BP method is applied, the depedece olx'(k) o the weights should also be take ito accout. A BP algorithm that takes care of this depedece is the dyamic BP algorithm (Kua 1989). To obtai the dyamic BP algorithm, cosider (1 a)(1 d). Let the traiig data set be (u(k), yo(,t)), k: 7,2,..., N, where yo(k) is the desired output of the etwork. Whe a iputoutput data pair is preseted to the etwork at time ft, the squared error at the etwork output is defied as Eu:+(yo(k) )'. (:4) Whe patterbased learig is adopted, the weights are modified at each time step k. For wi(k 1), the error gradiet is AEu dr,vi(ft 1)

3 Elta rtt' i tttorks a.d dyati r: s r st em m o d el i q JJ For y,i(/< 1) 21d i1,;'r(ft 1)?E, ;r'j'(k 1) aer li,ft tl _ ler e),qt 1"(l] du,(/r) dl (A),.\i(k) er,iij F'l1L r (yo(/<) )wl(k dr,(ft) tx,(k) _ill it'(k) d.r,(k),'r;r1k t1 (yo(/,) )wi(k * 1) 1).f, ir(/<). (6) Ax,(k) x_ (1) dwirlk 1y', (6) f, deotes the derivative ol f with respect to u,. As discussed above, the iterai leedback x(k 1) is depedet o w'(k 2). Therefore, from (1a) ad (lb) Ax,(k) _ 6x,(k) Au,(k) 1wi,t(k * 1) Au,$) Awi.,(k 1) :l, Au,(k) "'d*1'1k l; ( :.f,,\x,tk _ t) t + 'i.,rt l) '*4:l) l t='t r''' ' 1wi.,& l)) f the weight chages are assumed to be small i iteratio, the (8) ca be approximately writte recursive form as Artkt ( tol3l',: /"{x;tk t) +, f rvr,(k r, at4l) l,,lr''i,l\ t, ar;,tr _ Z1t (8) each ia Equatio (9) keeps a dyamic trace of the gradiet. This is equivalet to applyrg BP through time (Werbos 1 e90). The geerai weight modificatio i the gradiet descet method is ae, A,w: ", 'Aw The dyamic BP algorithm for traiig a Elma etwork ca thus be summarized as foiiows N ettuork u,(k) : wi;(k 1)x;(k 1) + wi'(k l)u(k), j=t r,(k) : /(u,), )(ft): rri(k1)r,(k). (e) (10) 3.2. Al.cloritlu Arrj(k) :'?(l,j(k) y(fr)),x,(k), A,j'(/<) : 4("yd(ft).1'(/,.))r,ul'(ft * Aw;.;(lc) : 4(,ro(k) )u'i(/r f r Lr c/\ i \KJ 1wi.,(k 1) where!4ll ( f,,f x,(k +i ^ df :L oui Awi./k 2\ "',2, 1) 1).f,. u(k), l). =' f,r,r A r'ru;',(a l) wilr(k"rhh+l :(1 f'(r,)), (1ta) (11b) (l1c) (ll r/) f i wi..&_ 2,d't\, t+i. ow;r(/< 3)' ' r t' f the depedece of x(k ) o x,"(k 2) is igored, the above algorithm degeerates to the stadard Bp algorithm, ad (11 d) ad (1 c) become _ar{! ;,wi'1@ D: f'''xfk1\' (11e) Awi;(/r) : qjd(k) )wl(k l)f,,x,(k t). (lt f) 4. Relatio with the modified Elma etwork t has bee discovered through simulatios that a liear Elma etwork traied by the stadard Bp algorithm ca oly model firstorder liear systems (Pham ad Liu 1992). t ca be see from (1 1 e) that the gradiet oly traces back oe time step i stadard BP. However, the gradiet i (11 d) traces back idefiitely. Followig the theory of Robiso ad traliside (1987) it ca be deduced that a Eima etwork traied by the stadard Bp algorithm ca oly represet a firstorder fiite impulse respose (hece firstorder dyamic systems). However, 11(a)11(d) ca trai a Elma etwork to model a ifiite impulse respose (thus higherorder dyamic systems). Pham ad Liu (1992) itroduced selffeedback liks with fixed gais to the cotext uits to eable the Elma etwork to represet higherorder systems (see Fig. 2). The idea of employig selffeedback liks was borrowed from the Jorda etwork (Jorda 1986). The modified Elma etwork of Pham ad Liu (1992) ca be described by the lollowig equatios.

4 224 D, T, Pham ad X. Liu 4.1. N etwork il u,(k) : ri;(k 1)xj(k) + wi(k )u(k), jt x;(k) : /(u,), xi(k) : xi(k 1) + axf(k i), : L *i(k l)x'(k) Algorithm (stadard BP) Awi(k) : 40 d(k) )x;(k), Awi(k) : 0 a(k) )wi(k )f",u(k), Awi;(k) :,1(yd$) )w/(k \//,\ l),awf,,(k_l), " :+9 Substitutig (13d) ito (12c) gives (t2 a) (12b) (12 c) (t2 d) dx,(k) : f..x,(k_ 1) + o 4",(!:1i. (14) Jvt, dwi.;(/<l) Awi.t(k2) t ca be see that (14) is similar i structure to (9). Although (14) does ot provide exactly the same search directio as (9) (ote that a(1x,(k \lawf,i(k 2)) appears i (14), but (9) has f,, L *f,,(k D Ax,(k \law{.t(k 2) l= (13 a) (13 b) (13 c) (13 d) i it), it ca provide a ifiite impulse respose. This is the reaso why the modified Elma etwork of Pham ad Liu (1992) was able to model higherorder dyamic systems, eve whe traied with the stadard BP algorithm. 5. Simulatio results 5.7. Liear systems idetfficatio The dyamic backpropagatio algorithm lias first bee used to idetify a secodorder liear system ad a thirdorder liear system. For those liear syster5, Eha etworks with liear hidde uits were employed. The secodorder iiear system model is ts discrete lorm is OJ G(s) : ;.. (s+a)'+@' : AJ(k l)+ary(k2)+bru(kt) * Bru(k ( 1s; 2). (16) With the samplig period 7: 0'1 s ad the parameters ad r.'t:2112'5, the coefficiets of (16) are a:l'0 At: l'752821, A2: 0' , Bt: 0' ad 82: 0.0t0942. A liear etwork with oe iput uit, six hidde/cotext uits ad oe output uit was used to idetify the system represeted by (16) (the umber of hidde/cotext uits, which should at least be equal to the order of the system to be modelled, was take as six to eable the etwork to model r4ost practical systems). A traiig set of 100 data poits was produced by sedig a uiformly radom bouded sequece lu(k)l + af)la(:2'911160) to the system model with zero iitial coditios ad recordig the output data. After traiig, the etwork was tested usig a step iput sigal u(k) : (a2 * cltz)lut. The system described by (16) could ot be idetified by the origial Elma etwork traied by the stadard BP algorithm (Pham ad Liu 1995). For compariso, the results of Pham ad Liu (1995) are show i Fig. 3. Usig the dyamic backpropagatio algorithm described above, the resposes of the system ad the traied Output uit Hidde uits 0.6 o./a Cotext uits lput uit u(k) Selffeedback liks Figure 2. Modified Elma etrvork. o o 20.{o Y"t(k) Y(k) Figure 3. Respose of Elma etwork traied by stadard BP (secodorder liear system) i

5 Elma etworks ad dyamic system modellig 22s :St ild 1S, t,.l 1.4 '1 o.8 r5) i6) ers ars td 0 y*(k) Figure 4. Respose of Elma etwork traied by dyamic BP (secodorder liear system). o.6 o.4 o O too y,*g) Figure 6. Respose of Elma etwork traied by stadard Bp (thirdorder liear system). ext em rits,.em 0rk 100 om rto,rd WAS fied.ard io, bed ed 0 y"(k) Figure 5. Respose of modified Elma etwork with a = 0.6 (secodorder liear system). etwork are show i Fig. 4. The respose of a modified Elma etwork with the same umber of hidde ad cotext uits traied by stadard BP is plotted i Fig. 5. The thirdorder system employed i the simulatios has oe real pole ad two complex poies: ts discrete lorm is G(s): (s+b)[(s+a)2+@2f (r7) 0.4 o Y.t(k) Figure 7. Respose of Elma etwork traied by dyamic Bp (thirdorder liear system). was tested with a step iput u(k) : b(a2 + o2;. The resposes of the Elma etwork traied by the stadard BP algorithm, the Elma etwork traied by the dyamic BP algorithm ad the modified Elma etwork traied by the stadard BP algorithm are show i Figs 68. All three etworks had six hidde/cotext uits. The traiig times for the dyamic BPtraied Elma etwork ad the BPtraied modified Elma etwork were approximately the same i both simulatio examples. 100 : At(k 1) + Ar!(k 2) + Ly& 3) t Bru(k 1) + Bru(k 2) + \u(k 3). (18) With the samplig period f : 0'08 s, ad the characterlstrc parameters a :1.0, b :2.5 ad a :212.5, the coefficiets of (18) are A, : , A2: , At : 0' , Br: 0' , 82: ad Bs:0' Agai a uiformly radom bouded sequece la(/c)i < b(a2 + co2) was applied to the system with zero iitial coditios ad the output of the system was recorded to produce a traiig data file. The etwork 5.2. N oliear system i.dettficatio The dyamic backpropagatio algorithm was also tested o the problem of idetifyig a oliear system. A Elma etwork possessig eight hidde uits with hyperbolic activatio fuctios was employed. The oliear system model is (Che ad Billigs 1994): y(ft) : ( exp (y'(k * l)))y(k 1) (0'3 + 0'9 exp (y'(k 1)))y(k 2) r 0.1 si (3' y(k 1)) + e(k), (19)

6 226 Elmcm etv,orks ad tl 1tcrtic srsrei 1odeiq show tlrat the rodificd Elma etwork is ll ppr.oxi_ m:rte realizatio of the Elma eti'vork tr.licd bv rl^ DBP algorith. This is bore out i t. ritout,,ti,l studies reported i the paper. Both the DBPtraie,i Elma uetwork ad the BPtraied lllodiliecl [1qo etwork could model systems with orders highcl tha i whereas the BPtraied origial Elma etwork could ot. The oliear system modellig ability ol the Elma etwork traied with the DBP algorithm has also bec illustrated i the paper for a bechmalk oliear system. v(k) Y"t(k) Figure 8. Respose of modified Elma etwork with a = 0'? (thirdorder liear systems)',* vm" i Figure 9. Respose of oliear process (see (19))' lr wlrere e(/t) was a gaussia white oise sequece with zero mea ad variace equal to 0'01. A database ol 1000 data poits was created usig (19) (iitial coditio: y(0) : y(1):01). The first 900 data were employed as traiig data. The last 100 data were reserved as ew data to test the traied model. Tlre etwork was traied usig oise e(/<) as the iput ad y(ft) as the desired output. The DBP algorithm was employed as the traiig algorithm. The respose of the traied etwork to the ew data is show i Fig Coclusio This paper has preseted a dyamic backpropagatio (DBP) algorithm for traiig the Elma etwork' t ltas ivestigated tlie relatioship betwee the Elma etwork traied by the DBP algorithm ad the modified Elma etrvork ploposed by Pham ad Liu (1992) ad has Ackowledgmet The authors would like to thak the Europea Colrmissio for its fiacial support uder the BRTE EURAM Focused Fudametal Research programmc (PSYCHO Project, Cotract No' BRE 2 CT94 09i6). Refereces CrN, S., acl Btluxcs, S. A., 1994, Neurzrl etworks ior modellig ad idetificatio. Aduaces i ltelliglet Cotrol. edited by C. J. Harris (Lodo, U.K.: Taylor & Fracis), pp Ertl,tN, J. L., 1990, Fidig structure i time CoqitiLte Scict:c, 14, Hurrz, J., Kocu, A., ad P,tLtut, R. G.' 1991, Rccltrret cural etworks. trodltctio to the Tlrcor1, ol Netoctl Cotputrttio (Czii' loria: AddisoWesley), Chapter 7. Joo,rN, M.., 1986, Attractor dyamics ad parallelisr i a coectioist scquetial machie. Proceedfutls ol the Bth Aual Coferece d the Cogitiue Sciece Socier,t" pp ' K,rvtto, K., ad TaNi;r,we, T, 1990, Stock price patter rcc()gitioa recurret eural etwork approach. teraliool J oit Colcrcru:e o Net'ul Neltt'or/<s, Vol. 1, pp KuzrN. C.M., 1989, Estimatio of euretl etwork modcls PhD' thesis, Uiversity of Califoria, Sa Diego. Ku,rN, C.M., HoxNtc, K., ad Wstrl, H , Soe covergccc results for learig i recurret eural etworks Disctrssio prpcr' Departmet of Ecoo.rics, Uiversity of Caliloria' Sarr Dicgo' PHrrrl. D. T., ad Lru, X., 1992, Dyamic system modellig usig partially recurret eural etwork s. J ourcl o/ Sy'sterrrs Eqieerittll' 2, P'ru. D. T., ad Ltu. X., 1995, Neutul Networks.fitr lleti.ficatio' Pred[ctio cd Cottol (Lodo, U.K.: SprigcrVerlag)' RosrNsoN, A. J., ad Fa.rrstlr, F., 1987, The utility drivcu dyamtc error propagatio etwork CUEDFNFENT/TR 1(1987)' Egieerig Departmet, Cambridge Uiversity, U.K Rutsrr, D., ad McClrl,quo' :.. iqso, Ptu'ttllel Disttibctl Processigl: Expktratos trt tlte Nict'ostt'uctttre o.f CorTiritirrir' Vol l (Cambridge, Mass.: MT Press) Tsr M.qrH Wors lxc ' lrrtroducig Versio 20 of thc Ncurrl Network Toolbox. EEE Crtu'ol Si'sfetris Ml1ct:ie' 14,73' WERBos. P.i., 1990, Backpropagatio through time: rvl.rat is it arrd how to do it'l Prttteetliqs ol'the EEE,78' '