Coded Strobing Photography: Compressive Sensing of High-speed Periodic Events

Transcription

1 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 1 Coded Srobing hoography: Compressive Sensing of High-speed eriodic Evens Ashok Veeraraghavan, Member, IEEE, Dikpal Reddy, Suden Member, IEEE, and Ramesh Raskar, Member, IEEE Absrac We show ha, via emporal modulaion, one can observe a high-speed periodic even well beyond he abiliies of a low-frame rae camera. By srobing he exposure wih unique sequences wihin he inegraion ime of each frame, we ake coded projecions of dynamic evens. From a sequence of such frames, we reconsruc a high-speed video of he high frequency periodic process. Srobing is used in enerainmen, medical imaging and indusrial inspecion o generae lower bea frequencies. Bu his is limied o scenes wih a deecable single dominan frequency and requires high-inensiy lighing. In his paper, we address he problem of sub-nyquis sampling of periodic signals and show designs o capure and reconsruc such signals. The key resul is ha for such signals he Nyquis rae consrain can be imposed on srobe-rae raher han he sensorrae. The echnique is based on inenional aliasing of he frequency componens of he periodic signal while he reconsrucion algorihm explois recen advances in sparse represenaions and compressive sensing. We exploi he sparsiy of periodic signals in Fourier domain o develop reconsrucion algorihms ha are inspired by compressive sensing. Index Terms Compuaional imaging, High-speed imaging, Compressive sensing, Sroboscopy I.INTRODUCTION eriodic signals are all around us. Several human and animal biological processes such as hear-bea, breahing, several cellular processes, indusrial auomaion processes and everyday objecs such as hand-mixer and blender all generae periodic processes. Neverheless, we are mosly unaware of he inner workings of some of hese high-speed processes because hey occur a a far greaer speed han can be perceived by he human eye. Here, we show a simple bu effecive echnique ha can urn an off-he-shelf video camera ino a powerful high-speed video camera for observing periodic evens. Srobing is ofen used in enerainmen, medical imaging and indusrial applicaions o visualize and capure high-speed visual phenomena. Acive srobing involves illuminaing he scene wih a rapid sequence of flashes wihin a frame ime. The classic example is Edgeron s Raparon o capure a golf swing [13]. In modern sensors, i is achieved passively by mulipleexposures wihin a frame ime [37][28] or fluering [29]. We use he erm srobing o indicae boh acive illuminaion and passive sensor mehods. In case of periodic phenomenon, srobing is commonly used o achieve aliasing and generae lower bea frequencies. While srobing performs effecively when he scene consiss of a single frequency wih a narrow sideband, i is difficul o visualize muliple or a wider band of frequencies simulaneously. Ashok Veeraraghavan and Dikpal Reddy conribued equally o his work. Time eriodic phenomenon wih unknown period (say 16 ms) = 16ms Video camera w/ frame rae f s = 25fps Coded Srobing Schemaic Capure M = 125 frames in 5s Every frame is modulaed U = 8 imes wih a unique binary code by opening & closing he shuer Coded Srobing: Time Domain A each pixel, he periodic signal is emporally modulaed wih a binary code T Frame = Frame Duraion = 4ms Coded Srobing: Frequency Domain Compuaionally reconsruc N = 1 frames Srucured sparse recovery A a pixel, he M observed inensiy values are linear combinaions of he periodic signal s sparse Fourier coefficiens - f Max 4f 4f f -2f -f f =1/ 2f Max Measure Linear Combinaions Time Binary code of lengh U=8 Srucured Sparsiy Enforcing Reconsrucion Algorihm Fig. 1: Coded srobing camera (CSC): A fas periodic visual phenomenon is recorded by a normal video camera (25 fps) by randomly opening and closing he shuer a high speed (2 Hz). The phenomenon is accuraely reconsruced from he capured frames a he high-speed shuer rae (2 fps). Insead of direc observaion of bea frequencies, we exploi a compuaional camera approach based on differen sampling sequences. The key idea is o measure appropriae linear combinaions of he periodic signal and hen decode he signal by exploiing he sparsiy of he signal in Fourier domain. We observe ha by coding during he exposure duraion of a low-frame-rae (e.g., 25 fps) video camera, we can ake appropriae projecions of he signal needed o reconsruc a high-frame-rae (e.g., 2 fps) video. During each frame, we srobe and capure a coded projecion of he dynamic even and sore he inegraed frame. Afer capuring several frames, we compuaionally recover he signal independenly a each pixel by exploiing he Fourier sparsiy of periodic signals. Our mehod of coded exposure for sampling periodic signals is ermed coded srobing and we call our camera he coded srobing camera (CSC). Figure 1 illusraes he funcioning of CSC. A. Conribuions We show ha sub-nyquis sampling of periodic visual signals is possible and ha such signals can be capured and recovered using a coded srobing compuaional camera. We develop a sparsiy-exploiing reconsrucion algorihm and expose connecions o Compressive Sensing. We show ha he primary benefi of our approach over radiional srobing is, increased ligh-hroughpu and f

2 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 2 abiliy o ackle muliple frequencies simulaneously poscapure. B. Benefis and limiaions The main consrain for recording a high-speed even is ligh hroughpu. We overcome his consrain for periodic signals via sufficien exposure duraion (in each frame) and exended observaion window (muliple frames). For wellli non-periodic evens, high-speed cameras are ideal. For a saic snapsho, a shor exposure phoo (or single frame of he high-speed camera) is sufficien. In boh cases, ligh hroughpu is limied bu unavoidable. eriodic signals can also be capured wih high-speed camera. Bu one will need a well-li scene or mus illuminae i wih unrealisic brigh lighs. For example, if we use a 2 fps camera for vocal cord analysis insead of srobing using a laryngoscope, we would need a significanly brigher illuminaion source and his creaes he risk of burn injuries o he hroa. A safer opion would be 25 fps camera wih srobed ligh source and hen exploi he periodiciy of vocal fold movemen. Here, we show ha an even beer opion in erms of ligh-hroughpu is a compuaional camera approach. Furher, he need o know frequency of he signal a capure-ime is also avoided. Moreover, he compuaional recovery algorihm can ackle he presence of muliple fundamenal frequencies in a scene, which poses a challenge o radiional srobing. C. Relaed work High-speed imaging hardware: Capuring high-speed evens wih fas, high-frame rae cameras require imagers wih high phooresponsiviy a shor inegraion imes, synchronous exposure and high-speed parallel readou due o he necessary bandwidh. In addiion, hey suffer from challenging sorage problems. A high-speed camera also fails o exploi he inerframe coherence, while our echnique akes advanage of a simplified model of moion. Edgeron and ohers have shown visually sunning resuls for high-speed objecs using exremely narrow-duraion flash [13]. These snapshos capure an insan of he acion bu fail o indicae he general movemen in he scene. Muliple low-frame rae cameras can be combined o creae high-speed sensing. Using a saggered exposure approach, Shechman e al. [34] used frames capured by muliple co-locaed cameras wih overlapped exposure ime. This saggered exposure approach also assised a novel reconfigurable muli-camera array [38]. Alhough here are very few mehods o super-resolve a video emporally [15], numerous super-resoluion echniques have been proposed o increase he spaial resoluion of images. In [17], superresoluion echnique o reconsruc a high-resoluion image from a sequence of low-resoluion images was proposed by backprojecion mehod. A mehod o do super-resoluion on a low qualiy image of a moving objec by firs racking i, esimaing moion and deblurring he moion blur and creaing a high qualiy image was proposed in [4]. Freeman e al. [14] proposed a learning based echnique for superresoluion from one image where he high frequency componens like edges of an image are filled by paches obained from examples wih similar low resoluion properies. Finally, fundamenal limis on super-resoluion for reconsrucion based algorihms have been explored in [1][22]. Sroboscopy and periodic moion: Sroboscopes (from Greek word στ ρωβωσ for whirling ) play an imporan role in scienific research, o sudy machinery in moion, in enerainmen and medical imaging. Muybridge in his pioneering work used muliple riggered cameras o capure high-speed moion of animals [25] and proved ha all four of a horse s hooves lef he ground a he same ime during a gallop. Edgeron also used flashing lamp o sudy machine pars in moion [13]. The mos common approaches for freezing or slowing down he movemen are based on emporal aliasing. In medicine, sroboscopes are used o view he vocal cords for diagnosis. The paien hums or speaks ino a microphone which in urn acivaes he sroboscope a eiher he same or a slighly lower frequency [2],[31]. However, in all healhy humans, vocalfold vibraions are aperiodic o a greaer or lesser degree. Therefore, srobolaryngoscopy does no capure he fine deail of each individual vibraory cycle; raher, i shows a paern averaged over many successive nonidenical cycles [24][33]. Modern srobocopes for machine inspecion [11] are designed for observing fas repeaed moions and for deermining RM. The idea can also be used o improve spaial resoluion by inroducing high-frequency illuminaion [16]. rocessing: In compuer vision, periodic moion of humans has received significan aenion. Seiz e al. [32] inroduced a novel moion represenaion, called he period race, ha provides a complee descripion of emporal variaions in a cyclic moion, which can be used o deec moion rends and irregulariies. A echnique o repair videos wih large saic background or cyclic moion was presened in [18]. Lapev e al. [19] presened a mehod o deec and segmen periodic moion based on sequence alignmen wihou he need for camera sabilizaion and racking. [5] exploied periodiciy of moving objecs o perform 3D reconsrucion by reaing frames wih same phase o be of same pose observed from differen views. In [35], he auhors showed a srobe based approach for capuring high-speed moion using muliexposure images obained wihin a single frame of a camera. The images of a baseball appear as disinc non-overlapping posiions in he image. High emporal and spaial resoluion can be obained via a hybrid imaging device which consiss of a high spaial resoluion digial camera in conjuncion wih a high frame-rae bu low resoluion video camera [6]. In cases where he moion can be modeled as linear, here have been several ineresing mehods o engineer he moion blur poin spread funcion so ha he blur induced by he imaging device is inverible. These include coding he exposure [3] and moving he sensor during he exposure duraion [21]. The mehod presened in his paper ackles a differen bu broadly relaed problem of reconsrucing periodic signals from very lowspeed images acquired via a convenional video camera (albei enhanced wih coded exposure). Comparison wih fluer shuer: In [3], he auhors showed ha by opening and closing he shuer according o an opimized coded paern during he exposure duraion of a

3 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 3 phoograph, one can preserve high-frequency spaial deails in he blurred capured image. The image can be hen de-blurred using a manually specified poin-spread funcion. Similarly, we open and close he shuer according o a coded paern and his code is opimized for capure. Neverheless, here are significan differences in moion models and reconsrucion procedures of boh hese mehods. In fluer shuer (FS), a consan velociy linear moion model was assumed and deblurring was done in blurred pixels along he moion direcion. On he oher hand, CSC works even on very complicaed moion models as long as he moion is periodic. In CSC each of he capured frames is he resul of modulaion wih a differen binary sequence whereas in FS a single frame is modulaed wih a all-pass code. Furher, our mehod conrass fundamenally wih FS in reconsrucion of he frames. In FS he sysem of equaions is no under-deermined whereas in CSC we have a severely under-deermined sysem. We overcome his problem by l 1 -norm regularizaion, appropriae for enforcing sparsiy of periodic moion in ime. In FS a single sysem of equaions is solved for enire image whereas in CSC a each pixel we emporally reconsruc he periodic signal by solving an under-deermined sysem. D. Capure and reconsrucion procedure The sequence of seps involved in capure and reconsrucion of a high-speed periodic phenomenon wih ypical physical values are lised below wih references o appropriae secions for deailed discussion. Goal: Using a 25 fps camera and a shuer which can open and close a 2 Hz, capure a high-speed periodic phenomenon of unknown period by observing for 5s. The lengh of he binary code needed is N = 2 5 = 1. For an upsampling facor of U = 2/25 = 8, find he opimal pseudo random code of lengh N (Secion III-A). Capure M = 25 5 = 125 frames by fluering he shuer according o he opimal code. Each capured frame is an inegraion of he incoming visual signal modulaed wih a corresponding subsequence of binary values of lengh U = 8 (Secion II-C). Esimae he fundamenal frequency of he periodic signal (Secion II-D3). Using he esimaed fundamenal frequency, a each pixel reconsruc he periodic signal of lengh N = 1 from M = 125 values by recovering he signal s sparse Fourier coefficiens (Secion II-D). II.STROBING AND LIGHT MODULATION A. Tradiional sampling echniques Sampling is he process of convering a coninuous domain signal ino a se of discree samples in a manner ha allows approximae or exac reconsrucion of he coninuous domain signal from jus he discree samples. The mos fundamenal resul in sampling is ha of Nyquis-Shannon sampling heorem. Figure 2 provides a graphical illusraion of radiional sampling echniques applied o periodic signals. Nyquis sampling: Nyquis-Shannon sampling saes ha when a coninuous domain signal is band-limied o [, f ] Hz, one can exacly reconsruc he band-limied signal, by jus observing discree samples of he signal a a sampling rae f s greaer han 2f [27]. When he signal has frequency componens ha are higher han he prescribed band-limi, hen during he reconsrucion, he higher frequencies ge aliased as lower frequencies making he reconsrucion erroneous (see Figure 2(Righ)(c)). If he goal is o capure a signal whose maximum frequency f Max is 1 Hz, hen one needs a highspeed camera capable of 2 fps in order o acquire he signal. Such high-speed video cameras are ligh limied and expensive. Band-pass sampling (srobing): If he signal is periodic as shown in Figure 2(Lef), hen we can inenionally alias he periodic signal by sampling a a frequency very close o he fundamenal frequency of he signal as shown in Figure 2(Lef)(e). This inenional aliasing allows us o measure he periodic signal. This echnique is commonly used for vocal fold visualizaion [24][33]. However, radiional srobing suffers from he following limiaions. The frequency of he original signal mus be known a capure-ime so ha one may perform srobing a he righ frequency. Secondly, he srobe signal mus be ON for a very shor duraion so ha he observed high-speed signal is no smoohed ou and his makes radiional srobing ligh-inefficien. Despie his handicap, radiional srobing is an exremely ineresing and useful visualizaion ool (and has found several applicaions in varying fields). Non-uniform sampling: Wih periodic sampling, aliasing occurs when he sampling rae is no adequae because, all frequencies of he formf 1 +k f s (k an ineger) lead o idenical samples. One mehod o couner his problem is o employ non-uniform or random sampling [7][23]. The key idea in nonuniform sampling [7][23] is o ensure a se of sampling insans such ha he observaion sequence for any wo frequencies are differen a leas in one sampling insan. This scheme has never found widespread pracical applicabiliy because of is noise sensiiviy and ligh inefficiency. B. eriodic signals Since, he focus of his paper is on high-speed video capure of periodic signals, we firs sudy he properies of such signals. 1) Fourier domain properies of periodic signals: Consider a signal x(), which has a period = 1/f and a bandlimi f Max. Since he signal is periodic, we can express i as, j=q x() = x DC + a j cos(2πjf )+b j sin(2πjf ) (1) j=1 Therefore, he Fourier ransform of he signal x() conains energy only in he frequencies corresponding o jf, where j { Q, (Q 1),...,1,...,Q}. Thus, a periodic signal has a maximum of (K = 2Q+1) non-zero Fourier coefficiens. Therefore, periodic signals by definiion, have a very sparse represenaion in he Fourier domain. Recen advances in he field of compressed sensing (CS) [12][9][2][8][36] have developed reliable recovery algorihms for inferring sparse represenaions if one can measure arbirary linear combinaions

4 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 4 eriodic Signal (x()) wih period eriodic signal wih period, band-limied o f Max. Fourier domain conains only erms ha are muliples of f =1/ Box filer of duraion T Box applied o he periodic signal - f Max -2f -f f =1/ 2f f Max Sinc response of he box filer aenuaes he harmonics near i s zeros (green) and he high frequencies (yellow) T Box = T Frame (c) (d) Normal Video Camera: Frames of he normal camera can be reaed as he samples (every T Frame ) of he oupu of he above box filer. T Frame = Frame Duraion High-speed Video Camera: Nyquis Sampling of (x()) Since each period of x has very fine high frequency variaions Nyquis sampling rae is very high. (c) (d) - f Max f Max f B =1/T Box Normal Camera causes aliasing of high frequency informaion (blue, green, yellow) Aliasing - f Max High-speed Video Camera wases he bandwidh and is inefficien since periodic signal is sparse in he Fourier domain f S = 1/T Frame = Sampling Frequency f Max (e) T s = 1/(2*f Max ) Tradiional Srobing: Sampling rae of camera is low, bu he ligh hroughpu is also very low in order o avoid blurring he feaures during he srobe ime. (e) - f Max f Max Tradiional Srobing requires he period of he signal o be known before capure f Srobe = 1/T Srobe ~ f (f) T Srobe = Srobe Duraion T Srobe ~ Coded Srobing: In every exposure duraion differen linear combinaions of he underlying periodic signal are observed (f) - f Max f Max Coded Srobing: Measure linear combinaions of a periodic signal s harmonics. Coded srobing is independen of he frequency f =1/. Ligh hroughpu is on an average 5% which is significanly greaer han radiional srobing. Time Domain - f Max 4f -f -2f f =1/ 2f 4f f Max Measure Linear Combinaions Frequency Domain Sparsiy Enforcing Reconsrucion Fig. 2: Time domain (Lef) and he corresponding frequency domain (Righ) characerisics of various sampling echniques as applicable o periodic signals. Noe ha capuring high-speed visual signals using normal camera can resul in aenuaion of high frequencies (b & c) whereas a high-speed camera demands large bandwidh (d) and radiional srobing is ligh-inefficien (e). Coded srobing is shown in (f). To illsurae sampling only wo replicas have been shown and noe ha colors used in ime domain and frequency domain are unrelaed. f of he signals. Here, we propose and describe a mehod for measuring such linear combinaions and use he reconsrucion algorihms inspired by CS o recover he underlying periodic signal from is low-frame-rae observaions. 2)Effec of visual exure on periodic moion: Visual exure on surfaces exhibiing periodic moion inroduces high frequency variaions in he observed signal (Figure 3(d)). As a very simple insrucive example consider he fan shown in Figure 3. The fan roaes a a relaively slow rae of 8.33 Hz. This would seem o indicae ha in order o capure he spinning fan one only needs a fps camera. During exposure ime of 6 ms of a Hz camera, he figure 1 wrien on he fan blade complees abou half a revoluion blurring i ou (Figure 3). Shown in Figure 3(c) is he ime profile of he inensiy of a single pixel using a high-speed video camera. Noe ha he sudden drop in inensiy due o he dark number 1 appearing on he blades persiss only for abou 1 millisecond. Therefore, we need a 1 fps highspeed camera o observe he 1 wihou any blur. In shor, he highes emporal frequency observed a a pixel is a produc of he highes frequency of he periodic even in ime and he highes frequency of he spaial paern on he objecs across he direcion of moion. This makes he capure of high-speed periodic signals wih exure more challenging. 3) Quasi-periodic signals: Mos real world periodic signals are no exacly so, bu almos; here are small changes in he period of he signal over ime. We refer o such broader class of signals as quasi-periodic. For example, he Cres oohbrush we use in our experimens exhibis a quasi-periodic moion wih fundamenal frequency ha varies beween Hz. Figure 4 shows few periods of a quasi-periodic signal a a pixel of a vibraing ooh brush. Variaion in fundamenal frequency f, beween 63 and 64 Hz, over ime can be seen in. Variaion in f of a quasi-periodic signal is refleced in is Fourier ransform which conains energy no jus a muliples jf bu in small band around jf. Neverheless, like periodic signals, he Fourier coefficiens are concenraed a jf (Figure 4(c)) and are sparse in he frequency domain. The coefficiens are disribued in a band [jf j f, jf +j f ]. For example, f =.75 Hz in Figure 4(d). C. Coded exposure sampling (or Coded srobing) The key idea is o measure appropriae linear combinaions of he periodic signal and hen recover he signal by exploiing he sparsiy of he signal in Fourier domain (Figure 5). Observe ha by coding he incoming signal during he exposure duraion, we ake appropriae projecions of he desired signal. 1) Camera observaion model: Consider a luminance signal x(). If he signal is band-limied o [ f Max, f Max ], hen in order o accuraely represen and recover he signal, we only need o measure samples of he signal ha areδ = 1/(2f Max ) apar where δ represens he emporal resoluion wih which we wish o reconsruc he signal. If he oal ime of observing

5 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 5 A frame from high speed video A frame from fps video Inensiy of pixel (115,245) (c) Signal a a pixel over few periods eriod = 119 ms 1 ms noch due o '1' Time in ms Magniude of FT in log scale 1 4 Frequency specrum of he signal a pixel (115,245) (d) Fundamenal frequency f = 8.33H z F req u en cy in H z Fig. 3: Video of a fan from a high-speed camera A Hz camera blurs ou he 1 in he image (c) Few periods of he signal a a pixel where he figure 1 passes. Noe he noch of duraion 1 ms in he inensiy profile. (d) Fourier ransform of he signal in (c). Noice he higher frequency componens in a signal wih low fundamenal frequency f. Fourier Coefficiens Quasi-periodic signal Time in ms x 1 4 (c) Fourier coefficiens Frequency in Hz Fourier Coefficiens Frequency in Hz Time in ms x Time varying fund. freq. f (d) Energy around fund. freq Frequency in Hz Fig. 4: Six periods of a N = ms long quasi-periodic signal a a pixel of a scene capured by 1 fps high-speed camera. Fundamenal frequency f varying wih ime. (c) Fourier coefficiens of he quasi-periodic signal shown in. (d) On zoom we noice ha he signal energy is concenraed in a band around he fundamenal frequency f and is harmonics. he signal is Nδ, hen he N samples can be represened in a N dimensional vecor x. In a normal camera, he radiance a a single pixel is inegraed during he exposure ime, and he sum is recorded as he observed inensiy a a pixel. Insead of inegraing during he enire frame duraion, we perform ampliude modulaion of he incoming radiance values, before inegraion. Then he observed inensiy values y a a given pixel can be represened as y = Cx+η, (2) where he M N marix C performs boh he modulaion and inegraion for frame duraion, and η represens he observaion noise. Figure 5 shows he srucure of marix C. If he camera observes a frame every T s seconds, he oal number of frames/observaions would be M = Nδ/T s and so y is a M 1 vecor. The camera sampling ime T s is far larger han he ime resoluion we would like o achieve (δ), herefore M << N. The upsampling facor (or decimaion raio) of CSC can be defined as, Upsampling facor = U = N M = 2f Max f s. (3) For example, in he experimen shown in Figure 15, f Max = 1 Hz, and f s = 25 fps. Therefore, he upsampling facor achieved is 8, i.e., he frame-rae of CSC is eighy imes smaller han ha of an equivalen high-speed video camera. Even hough, he modulaion funcion can be arbirary, in pracice i is usually resriced o be binary (open or close shuer). Effecive modulaion can be achieved wih codes ha have a 5% ransmission, i.e., he shuer is in ON posiion for 5% of he oal ime, hereby limiing ligh-loss a capureime o jus 5%. 2) Signal model: If x, he luminance a a pixel is bandlimied i can be represened as, x = Bs, (4) where, he columns of B conain Fourier basis elemens. Moreover, since he signal x() is assumed o be periodic, we know ha he basis coefficien vecor s is sparse as shown in Figure 5. uing ogeher he signal and observaion model, he inensiies in he observed frames are relaed o he basis coefficiens as, y = Cx+η = CBs+η = As+η, (5) where A is he effecive mixing marix of he forward process. Recovery of he high-speed periodic moion x amouns o solving he linear sysem of equaions (5). D. Reconsrucion algorihms To reconsruc he high-speed periodic signal x, i suffices o reconsruc is Fourier coefficiens s from modulaed inensiy observaions y of he scene. Unknowns, measuremens and sparsiy: In (5), he number of unknowns exceeds he number of known variables by a facor U (ypically 8) and hence he sysem of equaions (5) is severely under-deermined (M << N). To obain robus soluions, furher knowledge abou he signal mus be used. Since he Fourier coefficiens s, of a periodic signal x, are sparse, a reconsrucion echnique enforcing sparsiy of s could sill hope o recover he periodic signal x. We presen wo reconsrucion algorihms, one which enforces he sparsiy of he Fourier coefficiens and is inspired by compressive sensing and oher which addiionally enforces he srucure of he sparse Fourier coefficiens.

6 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 6 Frame 1 Frame M Observed Inensiy y = Observaion Model Coded Srobing Frame Inegraion eriod T S C Observed low-frame rae video y eriodic Signal x eriod () N unknowns = C B s = Mixing marix A s Signal Model Fourier coeff. of he periodic signal ( Sparse ) Very few (K) non-zero elemens eriodic Signal b1 b2 bn Fourier Basis Sparse Basis Coeff x = B s Fig. 5: Observaion model shows he capure process of he CSC where differen colors correspond o differen frames and he binary shuer sequence is depiced using he presence or absence of color. Noe ha each frame uses a differen binary sub-sequence. The signal model illusraes he sparsiy in he frequency specrum of a periodic signal. 1) Sparsiy enforcing reconsrucion: Esimaing a sparse vecor s (wih K non-zero enries) ha saisfies y = As+η, can be formulaed as an l opimizaion problem: () : min s s. y As 2 ǫ. (6) Alhough for general s his is a N-hard problem, for K sufficienly small he equivalence beween l and l 1 -norm [8] allows us o reformulae he problem as one of l 1 -norm minimizaion, which is a convex program wih very efficien algorihms [12][8][2]. (1) : min s 1 s. y As 2 ǫ (7) M frames from coded srobing camera Esimae fundamenal frequency f Consruc se S fh of possible Fourier coeff. Sparsiy enforcing reconsrucion Srucured sparse reconsrucion Recover Fourier coeff. s for every pixel by solving BDN Recover Fourier coeff. s for every pixel by solving BDN O r ig in a l s ig n a l S r u c u r e d s p a r s iy S p a r s iy e n fo r c in g The parameer ǫ allows for he variaion in he modeling of signal s sparsiy and/or noise in he observed frames. In pracice, i is se o a fracion of capured signal energy (e.g., ǫ =.3 y 2 ) and is dicaed by he prior knowledge abou camera noise in general and he exen of periodiciy of he capured phenomenon. An inerior poin implemenaion (BDN) of (1) is used o accuraely solve for s. Insead, in mos experimens in his paper, a he cos of minor degradaion in performance we use CoSaM [26], a faser greedy algorihm o solve (). Boh () and (1) don ake ino accoun he srucure in he sparse coefficiens of he periodic signal. By addiionally enforcing srucure of he sparse coefficiens s, we achieve robusness in recovery of he periodic signal. 2) Srucured sparse reconsrucion: We recall ha periodic/quasi-periodic signals are sparse in he Fourier basis and if he period is = 1/f, he only frequency conen he signal has is in he small bands a he harmonics jf, j an ineger. Ofen, he period is no known a priori. If he period is known or can be esimaed from he daa y, hen for a hypohesized fundamenal frequency f H, we can consruc a se S fh wih basis elemens [jf H f H, jf H + f H ], for j { Q,...,1,...,Q} such ha all he sparse Fourier coefficiens will lie in his smaller se. Now he problem () can insead be reformulaed as T im e Fig. 6: Overview of srucured sparse and sparsiy enforcing reconsrucion algorihms Five periods of a noisy (SNR=35 db) periodic signal x ( = 14 unis). Signal recovered by srucured and normal sparsiy enforcing reconsrucion are also shown. ( Srucured ) : min s s. (8) y As 2 ǫ and nonzero(s) S fh for some f H [, f Max ]. where nonzero(s) is a se conaining all he non-zero elemens in he reconsruced s. Since he exen of quasiperiodiciy is no known a priori, he band f H is chosen safely large and he non-zero coefficiens coninue o remain sparse in he se S fh. Inuiively, problem Srucured gives a beer sparse soluion compared o () since he non-zero coefficiens are searched over a smaller se S fh. An example of a periodic signal and is recovery using sparsiy enforcing (1) and srucured sparsiy are shown in Figure 6. The

7 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 7 recovery using Srucured is exac whereas () fails o recover he high-frequency componens. The resaemen of he problem provides wo significan advanages. Firsly, i reduces he problem search space of he original l formulaion. To solve he original l formulaion, one has o search over N C K ses. For example, if we observe a signal for 5 seconds a 1 ms resoluion, hen N is 5 and N C K is prohibiively large (1 212 for K = = 1). Secondly, his formulaion implicily enforces he quasiperiodiciy of he recovered signal and his exra consrain allows us o solve for he unknown quasi-periodic signal wih far fewer measuremens han would oherwise be possible. The ype of algorihms which exploi furher saisical srucure in he suppor of he sparse coefficiens come under model-based compressive sensing [3]. 3) Knowledge of fundamenal frequency: Srucured sparse reconsrucion performs beer over a larger range of upsampling facors and since he srucure of non-zero coefficiens is dependen on fundamenal frequency f, we esimae i firs. Idenificaion of fundamenal frequency: For boh periodic and quasi-periodic signals we solve a sequence of leas-square problems o idenify he fundamenal frequency f. For a hypohesized fundamenal frequency f H, we build a se S fh wih only he frequencies jf H (for boh periodic and quasiperiodic signals). Truncaed marix A fh is consruced by reaining only he columns wih indices in S fh. Non-zero coefficiens ŝ fh are hen esimaed by solving he equaion y = A fh s fh in a leas-squares sense. We are ineresed in f H which has a small reconsrucion error y ŷ fh (or larges oupu SNR) where ŷ fh = A fh ŝ fh. If f is he fundamenal frequency, hen all he ses S fh, where f H is a facor of f, will provide a good fi o he observed signal y. Hence, he plo of oupu SNR has muliple peaks corresponding o he good fis. From hese peaks we pick he one wih largesf H. In Figure 7, we show resuls of experimens on synheic daases, under wo scenarios: noisy signal and quasi-periodiciy. We noe ha even when he signal is noisy and when he quasi-periodiciy of he signal increases, he las peak in he SNR plo occurs a fundamenal frequency f. We generae quasi-periodic signals from periodic signals by warping he ime variable. Noe ha, solving a leas squares problem for a hypohesized fundamenal frequency f H is equivalen o solving srucured wih f H =. Seing f H = eases he process of finding he fundamenal frequency by avoiding he need o se he parameer f H appropriae for boh he capured signal and f H. This is especially useful for quasiperiodic signals where a priori knowledge of quasi-periodiciy is no available. III.DESIGN ANALYSIS In his secion, we analyze imporan design issues and gain a beer undersanding of he performance of coded srobing mehod hrough experimens on synheic examples. A. Opimal code for coded srobing Theoreically opimal code: The opimizaion problems (6) and (7) give unique and exac soluions provided he underdeermined marix A saisfies he resriced isomery propery SNR in db of coded srobing signal y True fund. freq. SNR = 5dB SNR = 4dB SNR = 3dB SNR = 2dB N=231, =14 U= Hypohesized fundamenal frequency f H SNR in db of coded srobing signal y True fund. freq. Increasing quasiperiodiciy N=231,=14 U=22 SNR=3dB Hypohesized fundamenal frequency f H Fig. 7: Idenifying he fundamenal frequency f. Oupu SNR y / y ŷ fh in db is ploed agains hypohesized fundamenal frequency f H. lo of SNR as he noise in y is varied. Noe ha he las peak occurs a f H = 165 (= N ). lo of SNR wih varying level of quasi-periodiciy. (RI) [1]. Since he locaion of he K non-zeros of he sparse vecor s which generaes he observaion y is no known a priori, RI demands ha all sub-marices of A wih 2K columns have a low condiion number. In oher words, every possible resricion of 2K columns are nearly orhonormal and hence isomeric. Evaluaing RI for a marix is a combinaorial problem since i involves checking he condiion number of all N C 2K submarices. Alernaely, marix A saisfies RI if every row of C is incoheren wih every column of B. In oher words, no row of C can be sparsely represened by columns of B. Tropp e al. [36] showed in a general seing ha if he code marix C is drawn from a IID Rademacher disribuion, he resuling mixing marix A saisfies RI wih a high probabiliy. I mus be noed ha a modulaion marix C wih enries +1, -1 is implemenable bu would involve using a beam splier and wo cameras in place of one. Due o ease of implemenaion (deails in secion IV), for modulaion we use a binary 1, code marix C as described in secion II-C1. For a given signal lengh N and an upsampling facor U we would like o pick a binary 1, code which resuls in mixing marix A, opimal in he sense of RI. Noe ha he sparsiy of quasi-periodic signals is srucured and he non-zero elemens occur a regular inervals. Hence, unlike he general seing, RI should be saisfied and evaluaed over only a selec subse of columns. Since he fundamenal frequency f of he signal is no known a priori, i suffices if he isomery is evaluaed over a sequence of marices Ā corresponding o hypohesized fundamenal frequency f H. Hence, for a given N and U, a code marix C which resuls in smalles condiion number over all he sequence of marices Ā is desired. In pracice, such a C is sub-opimally found by randomly generaing he binary codes ens of housand imes and picking he bes one. Compared o a normal camera, CSC blocks half he ligh bu capures all he frequency conen of he periodic signal. The sinc response of he box filer of a normal camera aenuaes he harmonics near is zeros as well as he higher frequencies as shown in Figure 2. To avoid he aenuaion of harmonics, he frame duraion of he camera has o be changed appropriaely. Bu, his is undesirable since mos cameras come wih a discree se of frame raes. Moreover, i is hard o have a priori knowledge of he signal s period. This

8 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 8 Sinusoid (x()) wih period Sinusoid of frequency f Modulae he sinusoid wih a random binary code (r()) x - f Max -f * f =1/ f Max Random binary code has frequency componens hroughou he specrum f (c) Box filer applied o he modulaed sinusoid T Box = T Frame (c) - f Max f Max Sinusoid f has a unique signaure from he code s specrum in he sinc s passband urple = Red + Blue f (d) Modulaed and filered sinusoid sampled every T Frame. T Box = T Frame (d) - f Max f Max Unique signaure of he sinusoid (purple) is aenuaed by he sinc response and hen shifed due o sampling every T Frame. Due o modulaion, no sinusoid is compleely aenuaed. f (e) Coded Srobing In every exposure duraion differen linear combinaions of he underlying periodic signal are observed Coded srobing is independen of he frequency of he underlying periodic signal. Ligh hroughpu is on an average 5% which is significanly greaer han radiional srobing. (e) - f Max f Max Coded Srobing: Measure linear combinaions of a periodic signal s harmonics. Each harmonic s unique signaure helps in recovering hem by enforcing sparsiy. - f Max 4f -2f -f f =1/ 2f 4f f Max Measure Linear Combinaions Sparsiy Enforcing Reconsrucion f f Fig. 8: Time domain (Lef) and corresponding frequency domain (Righ) undersanding of CSC. Shown in is a single sinusoid.,(c) & (d) show he effec of coded srobing capure on he sinusoid. (e) coded srobing capure of muliple sinusoids is simply a linear combinaion of he sinusoids. problem is enirely avoided by modulaing he incoming signal wih a pseudo-random binary sequence. Shown in Figure 8 is he emporal and frequency domain visualizaion of he effec of CSC on a single harmonic. Modulaion wih a pseudorandom binary code spreads he harmonic across he specrum. Thus, every harmonic irrespecive of is posiion avoids he aenuaion, he sinc response causes. We perform numerical experimens o show he effeciveness of CSC (binary code) over he normal camera (all 1 code). Shown in Table I are he comparison of he larges and smalles condiion numbers of he marix Ā arising in CSC and normal camera. Consider he second column (U = 25). For a given signal lengh N = 5 and upsampling facor U = 25, he larges condiion number ( ) of mixing marix Ā of a normal camera occurs for signal of period = 75. Similarly, he smalles condiion number occurs for = 67. On he oher hand, he mixing marix Ā of CSC has significanly lower maximum (a = 9) and minimum (a = 67) condiion numbers. erformance evaluaion: We perform simulaions on periodic signals o compare he performance of sparsiy enforcing and srucured sparse reconsrucion algorihms on CSC frames, srucured sparse reconsrucion on normal camera frames and radiional srobing. SNR plos of he reconsruced signal using he four approaches for varying period, upsampling facor U and noise level in y are shown in Figure 9. The signal lengh is fixed o N = 2 unis. The advanage of srucured sparse reconsrucion is apparen from comparing blue and red plos. The advanage of CSC over normal camera can be seen by comparing blue and black plos. Noe ha he normal camera performs poorly when he upsampling facor U is a muliple of he period. B. Experimens on a synheic animaion We perform experimens on a synheic animaion of a fracal o show he efficacy of our approach. We also analyze he performance of he algorihm under various noisy scenarios. We assume ha a every δ = 1 ms, a frame of he animaion is being observed and ha he animaion is repeiive wih = 25 ms (25 disinc images in he fracal). Three such frames are shown in Figure 1. A normal camera running a f s = 25 fps will inegrae 4 frames of he animaion ino a single frame, resuling in blurred images. Three images from a 25 fps video are shown in. By performing ampliude modulaion a he shuer, as described in II-C1, he CSC obains frames a he same rae as ha of he normal camera (25 fps) bu wih he images encoding he emporal movemen occurring during he inegraion process of he camera sensor. Three frames from he CSC are shown in (c). Noe ha in images & (c) and also images in oher experimens we rescaled he inensiies appropriaely for beer display. For our experimen, we observe he animaion for 5 seconds (N = 5) resuling in M = 125 frames. From hese 125 frames we recover frequency conen of he periodic signal being observed by enforcing sparsiy in reconsrucion as described in II-D. We compare srucured sparse reconsrucion on normal camera frames, normal sparse and srucured sparse reconsrucion on CSC frames and he resuls are shown in (d),(e) and (f) respecively. I is imporan o modulae he scene wih a code o capure all frequencies and enforcing boh sparsiy and srucure in reconsrucion ensures ha he periodic signal is recovered accuraely. Noise analysis and influence of upsampling facor: We perform saisical analysis on he impac of wo mos common sources of noise in CSC and also analyze he influence of

9 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 9 Condiion Number κ / eriod U = 25 U = 4 U = 47 U = 55 U = 63 U = 91 NC: larges / / / / / /7 CSC: larges / / / / / /7 NC: smalles / / / / / /55 CSC: smalles 16.5/ /94 1.1/98 9.7/9 1.9/ /53 TABLE I: Table comparing he larges and smalles condiion numbers of mixing marix Ā corresponding o normal (NC) and coded srobing exposure (CSC). SNR in db of reconsruced signal Srucured sparsiy Sparsiy enforcing Norm. Cam.: Sruc sparse Tradiional srobing N = 2 U = 25 SNR = 35dB eriod SNR in db of reconsruced signal Srucured sparsiy Sparsiy enforcing Norm. Cam.: Sruc sparse Tradiional srobing N = 2 = 15 SNR = 35dB Upsampling facor U SNR in db of reconsruced signal Srucured sparsiy Sparsiy enforcing Norm. Cam.: Sruc sparse Tradiional srobing N=2 U = 25 = SNR in db of he inpu periodic signal Fig. 9: erformance analysis of srucured and normal sparsiy enforcing reconsrucion for CSC and srucured sparsiy enforcing reconsrucion for normal camera: Reconsrucion SNR as he period increases. Reconsrucion SNR as upsampling facor U increases. (c) Reconsrucion SNR as he noise in y is varied. upsampling facor on reconsrucion. We recover he signal using srucured sparsiy enforcing reconsrucion. Firs, we sudy he impac of sensor noise. Figure 11 shows he performance of our reconsrucion wih increasing noise level η. We fixed he upsampling facor a U = 4 in hese simulaions. The reconsrucion SNR varies linearly wih he SNR of he inpu signal in accordance wih compressive sensing heory. The second mos significan source of errors in a CSC are errors in he implemenaion of he code due o lack of synchronizaion beween he shuer and he camera. These errors are modeled as bi-flips in he code. Figure 11 shows he resilience of he coded srobing mehod o such bi-flip errors. The upsampling facor is again fixed a 4. Finally, we are ineresed in an undersanding of how far he upsampling facor can be pushed wihou compromising on he reconsrucion qualiy. Figure 11(c) shows he reconsrucion SNR as he upsampling facor increases. This indicaes ha by using srucured sparsiy enforcing reconsrucion algorihm, we can achieve large upsampling facors wih a reasonable fideliy of reconsrucion. Using he procedure described in previous secion we esimae he fundamenal frequency as f p = 4 Hz (Figure 11(d)). IV.EXERIMENTAL ROTOTYES A. Hi-speed video camera In order o sudy he feasibiliy and robusness of he proposed camera, we firs esed he approach using a high-speed video camera. We used an expensive 1 fps video camera, and capured high-speed video. We had o use srong illuminaion sources o ligh he scene and capure reasonably noise-free high-speed frames. We hen added several of hese frames (according o he srobe code) in sofware o simulae low speed coded srobing camera frames. The simulaed CSC frames were used o reconsruc he high-speed video. Some resuls of such experimens are repored in Figure 12. B. Sensor inegraion mechanism We implemen CSC for our experimens using an off-he-shelf Dragonfly2 camera from oingrey Research [28], wihou SNR in db of reconsruced signal SNR in db of reconsruced signal SNR of recons. signal vs inpu periodic signal SNR in db of he inpu periodic signal SNR of recons. signal vs upsampling facor (c) Upsampling facor U SNR in db of reconsruced signal SNR in db of coded srobing signal y SNR of recons. signal vs % of error in fluer ercenage of bis flipped in he fluer sequence Esimaing he fund. freq. of a quasi-periodic signal (d) Hypohesized fundamenal freqequency f in Hz H Fig. 11: erformance analysis of CSC: Reconsrucion SNR as he observaion noise increases. Impac of bi-flips in binary exposure sequence. (c) Coded srobing camera capures he scene accuraely upo an upsampling facor U = 5. (d) y / y ŷ agains varying hypohesized fundamenal frequency f H. modificaions. The camera allows a riggering mode (Muliple Exposure ulse Widh Mode- Mode 5) in which he sensor inegraes he incoming ligh when he rigger is 1 and is inacive when he rigger is. The rigger allows us exposure conrol a a emporal resoluion of δ = 1 ms. For every frame we use a unique riggering sequence corresponding o a unique code. The camera oupus he inegraed sensor readings as a frame afer a specified number of inegraion periods. Also, each inegraion period includes a is end a period of abou 3 ms during which he camera processes he inegraed sensor readings ino a frame. The huge benefi of his seup is ha i allows us o use an off-he-shelf camera o slow down high-speed evens around us. On he oher hand, he hardware boleneck in he camera resrics us o operae a an effecive frame rae of 1 fps (1 ms) and a srobe rae

10 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 1 Original frames (d) Srucured sparse recovery: CSC Recon SNR 17.8dB Normal camera capure (e) Srucured sparse recovery: Normal camera Recon SNR 7.2dB (c) Coded srobing capure (f) Sparsiy enforcing recovery: CSC Recon SNR 7.5dB Fig. 1: Original frames of he fracal sequence which repea every = 25 ms. Frames capured by a normal 25 fps camera. (c) Frames capured by a CSC running a 25 fps. (d) Frames reconsruced by enforcing srucured sparsiy on CSC frames. (e) Frames reconsruced by enforcing srucured sparsiy on normal camera frames. (f) Frames reconsruced by enforcing simple sparsiy on CSC frames. Overall 5 seconds (N = 5) of he sequence was observed o reconsruc i back fully. Upsampling facor was se a U = 4 (M = 125) corresponding o δ = 1 ms. Noe ha image inensiies in and (c) have been rescaled appropriaely for beer display. of 1 srobes/second (δ = 1 ms). C. Ferro-elecric shuer The oingrey Dragonfly2 provides exposure conrol wih a ime resoluion of 1 ms. Hence, i allows us a emporal resoluion of δ = 1 ms a recovery ime. However, when he maximum linear velociy of he objec is greaer han 1 pixel per ms, he reconsruced frames have moion blur. One can avoid his problem wih finer conrol over he exposure ime. For example, a DisplayTech ferro-elecric liquid crysal shuer provides an ON/OFF conras raio of abou 1 : 1, while simulaneously providing very fas swiching ime of abou 25µs. We buil a prooype where he Dragonfly2 capures he frames a usual 25 fps and also riggers a IC conroller afer every frame which in urn fluers he ferro-elecric shuer wih a new code a a specified emporal frequency. In our experimen we se he emporal resoluion a 5µs i.e. 2 srobes/second. D. Rerofiing commercial sroboscopes Anoher exciing alernaive o implemen CSC is o rerofi commercial sroboscopes. Commercial sroboscopes used in laryngoscopy usually allow he srobe ligh o be riggered via a rigger inpu. Sroboscopes ha allow such an exernal rigger for he srobe can be easily rerofied o be used as a CSC. The IC conroller used o rigger he ferro-elecric shuer can insead be used o synchronously rigger he srobe ligh of he sroboscope, hus convering a radiional sroboscope o a coded sroboscope. V.EXERIMENTAL RESULTS To validae our design we conduc wo kinds of experimens. In he firs experimen, we capure high-speed videos and hen generae CSC frames by appropriaely adding frames of he high-speed video. In he second se of experimens we capured videos of fas moving objecs wih a low-frame-rae CSC implemened using a Dragonfly2 video camera. A. High-speed video of oohbrush We capure a high-speed (1 fps) video of a pulsaing Cres oohbrush wih quasi-periodic linear and oscillaory moions a abou 63 Hz. Figure 4 shows he frequency of he oohbrush as a funcion of ime. Noice ha even wihin a shor window of 3 seconds, here are significan changes in frequency. We render a 1 fps, 2 fps, 1 fps CSC (i.e., a frame duraion of 1 ms, 5 ms, 1 ms respecively) by adding appropriae high-speed video frames, bu reconsruc he moving oohbrush images a a resoluion of 1 ms as shown in Figures 12(c)-(e) respecively. Frames of he CSC operaing a 1, 2 and 1 fps (U = 1, 5 and 1 respecively) are shown in Figure 12. The fine brisles of he oohbrush add high frequency componens because of exure variaions. The brisles on he circular head moved almos 6 pixels wihin 1 ms. Thus he capured images from he high-speed camera hemselves exhibied blur of abou 6 pixels which can be seen in he recovered images. Noice ha conrary o wha i seems o he naked eye, he circular head of he oohbrush does no acually complee a roaion. I jus exhibis oscillaory moion

11 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 11 Hi-speed capure a 1fps Coded srobing capure U=1 U=5 U=1 (c) Srucured sparse recovery U=1 U=5 U= Recon SNR = 2.8dB Recon SNR = 16.4dB Recon SNR = 13.6dB Fig. 12: Reconsrucion resuls of an oscillaing oohbrush under hree differen capure parameers (U): Images for simulaion capured by a 1 fps high-speed camera a ime insances 1, 2 and 3 are shown in. The second row shows a frame each from he coded srobing capure (simulaed from frames in ) a upsampling facors U = 1, 5, and 1 respecively. Reconsrucion a ime insances 1, 2 and 3 from he frames capured a U = 1 are shown in firs column of (c). of 45 degrees and we are able o see i from he high-speed reconsrucion. Recon SNR = 2.8dB Recon SNR = 13.2dB Fig. 13: Reconsrucion resuls of oohbrush wih upsampling facor U = 1 wihou and wih 15 db noise in and respecively. Srucured sparse recovery Coded Srobing Camera Srucured sparse recovery Normal Camera Recon SNR = 16.4dB Recon SNR = 13.dB Fig. 14: Reconsrucion resuls of oohbrush wih upsampling facor U = 5 using srucured sparse reconsrucion and sparsiy promoing super-resoluion. To es he robusness of coded srobing capure and recovery on he visual qualiy of images, we corrup he observed images y wih whie noise having SNR = 15 db. The resuls of he recovery wihou and wih noise are shown in Figure 13. We compare frames recovered from CSC o hose recovered from a normal camera (by enforcing srucured sparsiy) o illusrae he effeciveness of modulaing he frames. Normal camera doesn capure he moion in he brisles as well (Figure 14) and is sauraed. B. Mill-ool resuls using ferro-elecric shuer We use a Dragonfly2 camera wih a ferro-elecric shuer and capure images of a ool roaing in a mill. Since he ool can roae a speeds as high as 12 rpm (2 Hz), o preven blur in reconsruced images we use he ferro-elecric shuer for modulaion wih a emporal resoluion of.5 ms. The CSC runs a 25 fps (4 ms frame lengh) wih he ferroelecric shuer fluering a 2 srobes/second. Shown in Figure 15 are he reconsrucions a 2 fps (δ =.5 ms) of a ool roaing a 3, 6, 9 and 12 rpm. Wihou a priori knowledge of scene frequencies, we use he same srobed coding and he same sofware decoding procedure for he mill ool roaing a differen rpm. This shows ha we can capure any sequence of periodic moion wih unknown period

12 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 12 3 RM 6 RM (c) 9 RM (d) 12 RM Fig. 15: Tool bi roaing a differen rpm capured using coded srobing: Top row shows he coded images acquired by a GR Dragonfly2 a 25 fps, wih an exernal FLC shuer fluering a 2 Hz. -(d) Reconsrucion resuls, a 2 fps (emporal resoluion δ = 5µs), of a ool bi roaing a 3,6,9 and 12 rpm respecively. For beer visualizaion, he ool was pained wih color prior o he capure. Frame from 1 fps camera Frames reconsruced from a 1 fps Dragonfly2 coded srobing camera ( U = 1 ) Fig. 16: Demonsraion of CSC a upsampling facor U = 1 using Dragonfly2. Capured image from a 1 fps CSC (Dragonfly2). Two reconsruced frames. While he CSC capured an image frame every 1 ms, we obain reconsrucions wih a emporal resoluion of 1 ms. wih a single pre-deermined code. In conras, in radiional srobing prior knowledge of he period is necessary o srobe a he appropriae frequency. Noice ha he reconsruced image of he ool roaing a 3 rpm is crisp and he images blur progressively as he rpm increases. Since he emporal resoluion of Dragonfly2 srobe is.5 ms, he feaures on he ool begin o blur a speeds as fas as 12 rpm. In fac, he linear velociy of he ool across he image plane is abou 33 pixels per ms (for 12 rpm), while he widh of he ool is abou 45 pixels. Therefore, he recovered ool is blurred o abou one-hird is widh in.5 ms. C. Toohbrush using Dragonfly2 camera We use a Dragonfly2 camera operaing in Trigger Mode 5 o capure a coded sequence of he Cres oohbrush oscillaing. The camera operaed a 1 fps, bu we reconsruc video of he oohbrush a 1 fps (U = 1) as shown in Figure 16. Even hough he camera acquires a frame every 1 ms, he reconsrucion is a a emporal resoluion of 1 ms. If we assume ha here arelphoons per ms, hen each frame of he camera would acquire around.5 1 L phoons. In comparison, each frame of a high-speed camera would accumulae L phoons, while radiional srobing camera would accumulae L f /f s = 6.3L phoons per frame. D. High-speed video of a jog Using frames from a high-speed (25 fps) video of a person jogging-in-place we simulae in compuer he capure of he scene using a normal camera and he CSC a upsampling facors of U = 25,5 and 75. The coded frames from CSC are used o reconsruc back he original high-speed frames by enforcing srucured sparsiy. The resuls of he reconsrucion using frames from he CSC a differen upsampling facors are conrased wih frames capured using a normal camera in Figure 17. A any given pixel, he signal is highly quasiperiodic since i is no a mechanically driven moion bu our algorihm performs reasonably well in capuring he scene. In Figure 17 we conras he reconsrucion a wo differen pixels, one where he moion is fas and he oher where i is relaively slower. Noe ha he pixels corresponding o faser moion have high frequency variaions due o exure and are harder o reconsruc. VI.BENEFITS AND LIMITATIONS A. Benefis and advanages Coded srobing allows hree key advanages over radiional srobing: (i) signal o noise raio (SNR) improvemens due o ligh-efficiency, (ii) no necessiy for prior knowledge of dominan frequency, and (iii) he abiliy o capure scenes wih muliple periodic phenomena wih differen fundamenal frequencies. Ligh hroughpu: Ligh efficiency plays an imporan role if one canno increase he brighness of exernal ligh sources. Le us consider he linear noise model (scene independen) where he SNR of he capured image is given by LT Exposure /σ gray, where L is he average ligh inensiy a a pixel and σ gray is a signal independen noise level which includes effecs of dark curren, amplifier noise and A/D converer noise. For boh radiional and coded srobing cameras, he duraion of he shores exposure ime should a

13 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE Normal camera capure Srucured sparse recovery Original signal Reconsruced signal: U=25 Reconsruced signal: U=5 Reconsruced signal: U= Fig. 17: Fronal scene of a person jogging-in-place. A frame capured by a normal camera and one of he frames recovered from coded srobing capure a U = 25. lo in ime of he pixel (yellow) of he original signal and signal reconsruced from coded srobing capure a U = 25,5 and 75. Noe ha he low frequency pars of he signal are recovered well compared o he high-frequency spikes. mos be δ = 1/(2f Max ). In radiional srobing, his shor exposure δ is repeaed once every period of he signal, and herefore he oal exposure ime in every frame is given by T Srobing = (1/2f Max )(f /f s ). Since he oal exposure ime wihin a frame can be as large as 5% of he oal frame duraion for CSC, T Coded = 1/2f s. The decoding process in coded srobing inroduces addiional noise, and his decoding noise facor is d = race((a T A) 1 )/M. Therefore, he SNR gain of CSC as compared o radiional srobing is given by SNR Gain = SNR Coded SNR Srobing = (LT Coded)/(dσ) (LT Srobing )/(σ) = f Max df (9) For example, in he case of he ool spinning a 3 rpm (or 5 Hz), his gain is 2log(1/(2 5)) = 2dB since f Max = 1 Hz for srobe rae 2 srobes/second. So coded srobing is a grea alernaive for ligh-limied scenarios such as medical inspecion in laryngoscopy (where paien issue burn is a concern) and long range imaging. Knowledge of fundamenal frequency: Unlike radiional srobing, coded srobing can deermine signal frequency in pos-capure, sofware only process. This allows for ineresing applicaions such as simulaneous capure of muliple signals wih very differen fundamenal frequencies. Since he processing is independen for each pixel, we can suppor scenes wih several independenly periodic signals and capure hem wihou a-priori knowledge of he frequency bands as shown in Figure 18. Shown, in Figure 15 are he reconsrucions obained for he ool which was roaing a 3,45,6 and 12 rpm. In all hese cases, he same coded shuer sequence was used a capure-ime. Also, he reconsrucion algorihm can also eminenly handle boh periodic and quasiperiodic signals using he same framework. Muliple periodic signals: Unlike radiional srobing, coded srobing allows us o capure and recover scenes wih muliple periodic moions wih differen fundamenal frequencies. The capure in coded srobing doesn rely on frequency of he periodic moion being observed and he recovery of he signal a each pixel is independen of he oher. This makes i possible o capure a scene wih periodic moions wih differen fundamenal frequency all a he same ime using he same hardware seings. The differen moions are reconsruced independenly by firs esimaing he respecive fundamenal frequencies and hen reconsrucing by enforcing srucured sparsiy. We perform experimens on an animaion wih wo periodic moions wih differen fundamenal frequencies. Shown in Figure 18 are few frames of he animaion wih a roaing globe on he lef and a horse galloping on he righ. The animaion was creaed using frames of a roaing globe which repeas every 24 frames and frames of he classic galloping horse which repeas every 15 frames. For simulaion, we assume ha a new frame of he animaion is being observed a a resoluion of δ = 1 ms and we observe he animaion for a oal ime of 4.8 seconds (N = 48). This makes he period of he globe 24 ms (f = Hz) and ha of horse 15 ms (f = Hz). The scene is capured using a 25 fps (U = 4) camera and few of he capured CSC frames are shown in. The reconsruced frames obained by enforcing srucured sparsiy are shown in (c). rior o he reconsrucion of he scene a each pixel, fundamenal frequencies of he differen moions were esimaed. For one pixel on horse (marked blue in Figure 18) and one pixel on he globe (marked red), he oupu SNR y / y ŷ is shown as a funcion of hypohesized fundamenal frequency f H in Figure 18(d). The fundamenal frequency are accuraely esimaed as Hz for he horse and Hz for he globe. Ease of implemenaion: The previous benefis assume significance because modern cameras, such as oingrey DragonFly 2, allow coded srobing exposure and hence here is no need for expensive hardware modificaions. We ransform his offhe-shelf camera insanly ino a 2 fps high-speed camera using our sampling scheme. On he oher hand, radiional srobing has been exremely popular and successful because of is direc-view capabiliy. Since our reconsrucion algorihm is no ye real-ime, we can only provide a delayed viewing of he signal. Table II liss he mos imporan characerisics of he various sampling mehodologies presened. B. Arifacs and limiaions We address he hree mos dominan arifacs in our reconsrucions: Blur in he reconsruced images due o

14 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 14 Esimaing he fund. freq. of muliple quasi-periodic signals Original frames Coded srobing, U=4 (c) Reconsruced frames SNR in db of coded srobing signal y True fund. freq. Globe Horse Hypohesized fundamenal freqequency f in Hz H Fig. 18: Recovery of muliple periodic moion in a scene. shows periodic evens wih differen periods. The scene as capured by CSC is shown in. The recovered frames are shown in (c). Shown in (d) is he esimaed fundamenal frequency of globe and horse a poins marked red and blue. Noe ha he las peak in boh globe and horse corresponds o he respecive fundamenal frequency of Hz and Hz. Mehod Sampling Rae Bes Scenario Benefis Limiaions High-speed (Nyquis) 2 f Scene wihin f Robus Cosly Srobing (band-pass) Lower han f eriodic and Brighly li Direc-view Linear search Non-uniform Lower han f Brighly li No aliasing No robus o noise Coded Srobing Lower han f eriodic Ligh-efficien No direc-view TABLE II: Table showing relaive benefis and appropriae sampling for presened mehods. ime resoluion, emporal ringing inroduced during deconvoluion process, and (c) sauraion due o speculariy. Blur: As shown in Fig 19, we observe blur in he reconsruced images when he higher spaio-emporal frequency of he moion is no capured by he shores exposure ime of.5 ms. Noice ha he blur when δ =.5 ms is less compared o when δ = 1 ms. The widh of he ool is abou 45 pixels and he linear velociy of he ool across he image plane is 33 pixels per millisecond. Hence, here is a blur of abou 16 pixels in he reconsruced image when δ =.5 ms and 33 pixels when δ = 1 ms. Noe ha his blur is no a resul of he reconsrucion process and is dependen on he smalles emporal resoluion. I mus also be noed here ha while 12 rpm (corresponding o 2 Hz) is significanly less compared o he 2 Hz emporal resoluion offered by coded srobing, he blur is a resul of visual exure on he ool. Temporal ringing: Temporal ringing is inroduced in he reconsruced images during he reconsrucion (deconvoluion) process. For simpliciy, we presened resuls wihou any regularizaion in he reconsrucion process (Figure 12(d),(e)). Noe ha in our algorihm reconsrucion is per pixel and he ringing is over ime. Figure 2 shows emporal ringing a wo spaially close pixels. Since he waveforms a hese wo pixels are relaed (ypically phase shifed), he emporal ringing appears as spaial ringing in he reconsruced images (Figure 16). Eiher daa independen Tikhonov regularizaion or daa dependen regularizaion (like priors) can be used o improve he visual qualiy of he reconsruced videos. Sauraion: Sauraion in he capured signaly resuls in sharp edges which in urn leads o ringing arifacs in he reconsruced signal. In Figure 2 we can see ha he periodic.5 ms 1 ms Fig. 19: Coded srobing reconsrucions exhibi blur when he emporal resoluion δ is no small enough. Shown in and are he same mill ool roaing a 12 rpm and capured by a srobe wih δ =.5 ms and δ = 1 ms respecively. The reconsrucions shown in he second and hird column show ha δ = 1 ms srobe rae is insufficien and leads o blur in he reconsrucions Original Signal:ixel 1 Recovered Signal:ixel 1 Original Signal:ixel 2 Recovered Signal:ixel Time Original signal Recovered signal: Unsauraed Recovered signal: Sauraed Time Fig. 2: Ringing arifacs (in ime) in he reconsruced signal a wo pixels separaed by 8 unis in Fig 12(f). Also shown are he inpu signals. Noe ha arifacs in reconsrucion (in ime) manifess iself as arifacs in space in he reconsruced image. Arifacs in he reconsruced signal due o sauraion in he observed signal y. signal recovered from sauraed y has emporal ringing. Since reconsrucion is independen for each pixel, he effec of sauraion is local and does no affec he res of he pixels in he image. Typical cause of sauraion in he capured image is due o speculariies in he observed scene. Speculariies (ha are no sauraed) do no pose a problem and are reconsruced as well as oher regions.

15 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE Signal 11 Signal 13 Signal 31 Signal 33 ACKNOWLEDGEMENTS The auhors would like o hank rof. Chellappa for his encouragemen and suppor, John Barnwell for his help wih he hardware and Brandon Taylor for making a superb video for he projec. Thanks o Ami Agrawal, Jay Thornon and numerous members of Misubishi Elecric Research Labs for engaging discussions abou he projec. The work of Dikpal Reddy was suppored by he ONR Gran N Time Fig. 21: The waveforms in a neighborhood are highly similar and hence he informaion is redundan. Shown are he waveforms of 4 pixels a he corners of a 3 3 neighborhood. The waveforms are displaced verically for beer visualizaion. VII.EXTENSIONS AND CONCLUSIONS A. Spaial redundancy In his paper, we discussed a mehod called coded srobing ha explois he emporal redundancy of periodic signals and in paricular, heir sparsiy in he Fourier domain in order o capure high-speed periodic and quasi-periodic signals using a low frame-rae CSC. The analysis and he reconsrucion algorihms presened considered he daa a every pixel as independen. In realiy, adjacen pixels have emporal profiles ha are very similar. In paricular (see Figure 21), he emporal profiles of adjacen pixels are relaed o each oher via a phase shif which depends upon he local speed and direcion of moion of scene feaures. This redundancy is currenly no being exploied in our curren framework. We are currenly exploring exensions of he CSC, ha explicily model his relaionship and use hese consrains during he recovery process. B. Spaio-emporal resoluion rade-off The focus of his paper, was on he class of periodic and quasiperiodic signals. One ineresing and exciing avenue for fuure work is o exend he applicaion of he CSC o a wider class of high-speed videos such as high-speed videos of saisically regular dynamical evens (e.g., waerfall, fluid dynamics ec) and finally o arbirary high-speed evens such as bursing balloons ec. One alernaive we are pursuing in his regard is considering a scenario which allows for spaio-emporal resoluion rade-offs, i.e., use a higher resoluion CSC in order o reconsruc lower resoluion high-speed videos of arbirary scenes. The spaio-emporal regulariy and redundancy available in such videos needs o be efficienly exploied in order o achieve his end. C. Conclusions In his paper, we presen a simple, ye powerful sampling scheme and reconsrucion algorihm ha urns a normal video camera ino a high-speed video camera for periodic signals. We show ha he curren design has many benefis over radiional approaches and show a working prooype ha is able o urn an off-he-shelf 25 fps oingrey Dragonfly2 camera ino a 2 fps high-speed camera. REFERENCES [1] S. Baker and T. Kanade. Limis on super-resoluion and how o break hem. IEEE Trans. aern Anal. Mach. Inell., 24(9): , 22. [2] R. Baraniuk. Compressive Sensing. Signal rocessing Magazine, IEEE, 24(4): , 27. [3] R. G. Baraniuk, V. Cevher, M. F. Duare, and C. Hegde. Model-based compressive sensing. o appear in IEEE Transacions in Informaion Theory, 21. [4] B. Bascle, A. Blake, and A. Zisserman. Moion deblurring and superresoluion from an image sequence. In ECCV 96: roceedings of he 4h European Conference on Compuer Vision-Volume II, pages , London, UK, Springer-Verlag. [5] S. Belongie and J. Wills. Srucure from periodic moion. In Workshop on Spaial Coherence for Visual Moion Analysis (SCVMA), rague, Czech Republic, 24. Springer Verlag, Springer Verlag. [6] M. Ben-Ezra and S. Nayar. Moion-based moion deblurring. IEEE Transacions on aern Analysis and Machine Inelligence, pages , 24. [7] I. Bilinskis and A. Mikelson. Randomized Signal rocessing. renice- Hall, Inc. Upper Saddle River, NJ, USA, [8] E. Candes. Compressive sampling. In roceedings of he Inernaional Congress of Mahemaicians, Madrid, Spain, volume 3, pages , 26. [9] E. Candes and J. Romberg. Sparsiy and incoherence in compressive sampling. INVERSE ROBLEMS, 23(3):969, 27. [1] E. Candes and T. Tao. Decoding by linear programming. Informaion Theory, IEEE Transacions on, 51(12): , Dec. 25. [11] Checkline. Indusrial sroboscopes. hp:// 28. [12] D. Donoho. Compressed Sensing. Informaion Theory, IEEE Transacions on, 52(4): , 26. [13] H. Edgeron. Raparonic hoographs. hp://simplehinking.com/home/raparonic phoographs.hm, [14] W. Freeman, T. Jones, and E. aszor. Example-based super-resoluion. Compuer Graphics and Applicaions, IEEE, 22(2):56 65, Mar/Apr 22. [15] H. Greenspan, S. eled, G. Oz, and N. Kiryai. Mri iner-slice reconsrucion using super-resoluion. In MICCAI 1: roceedings of he 4h Inernaional Conference on Medical Image Compuing and Compuer-Assised Inervenion, pages , London, UK, 21. Springer-Verlag. [16] M. G. L. Gusafsson. Nonlinear srucured-illuminaion microscopy: Wide-field fluorescence imaging wih heoreically unlimied resoluion. roceedings of he Naional Academy of Sciences of he Unied Saes of America, 12(37): , 25. [17] M. Irani and S. eleg. Improving resoluion by image regisraion. CVGI: Graph. Models Image rocess., 53(3): , [18] J. Jia, Y.-W. Tai, T.-. Wu, and C.-K. Tang. Video repairing under variable illuminaion using cyclic moions. aern Analysis and Machine Inelligence, IEEE Transacions on, 28(5): , May 26.

16 IEEE TRANSACTIONS ON ATTERN ANALYSIS AND MACHINE INTELLIGENCE 16 [19] I. Lapev, S. J. Belongie,. erez, and J. Wills. eriodic moion deecion and segmenaion via approximae sequence alignmen. In ICCV 5: roceedings of he Tenh IEEE Inernaional Conference on Compuer Vision (ICCV 5) Volume 1, pages , Washingon, DC, USA, 25. IEEE Compuer Sociey. [2] H. Larsson, S. Heregård, and B. Hammarberg. Vocal Fold Vibraions: High-Speed Imaging, Kymography, and Acousic Analysis: A reliminary Repor. The Laryngoscope, 11(12):2117, 2. [21] A. Levin,. Sand, T. Cho, F. Durand, and W. Freeman. Moion-invarian phoography. In roc. ACM SIGGRAH, volume 8, 28. [22] Z. Lin and H.-Y. Shum. Fundamenal limis of reconsrucion-based superresoluion algorihms under local ranslaion. IEEE Trans. aern Anal. Mach. Inell., 26(1):83 97, 24. [23] F. Marvasi. Nonuniform Sampling: Theory and racice. Kluwer Academic/lenum ublishers, 21. [24]. Mergell, H. Herzel, and I. Tize. Irregular vocal-fold vibraionhighspeed observaion and modeling. The Journal of he Acousical Sociey of America, 18:2996, 2. [25] E. Muybridge. Animals in Moion. Dover ublicaions, Firs ed., Chapman and Hall London [26] D. Needell and J. Tropp. Cosamp: Ieraive signal recovery from incomplee and inaccurae samples. Applied and Compuaional Harmonic Analysis, 26(3):31 321, 29. [27] A. V. Oppenheim, R. W. Schafer, and J. R. Buck. Discree-Time Signal rocessing. renice-hall, [28] oingrey Research. GR IEEE-1394 Digial Camera Regiser Reference. 26. [29] R. Raskar, A. Agrawal, and J. Tumblin. Coded exposure phoography: moion deblurring using fluered shuer. ACM Trans. Graph., 25(3):795 84, 26. [3] R. Raskar, A. Agrawal, and J. Tumblin. Coded exposure phoography: moion deblurring using fluered shuer. In SIGGRAH 6: ACM SIGGRAH 26 apers, pages , New York, NY, USA, 26. ACM. [31] G. Schade, M. Hess, F. Mller, T. Kirchhoff, M. Ludwigs, R. Hillman, and J. Kobler. hysical and echnical elemens of shor-inerval, colorfilered double srobe flash-sroboscopy. HNO, 5(12): , Dec 22. [32] S. Seiz and C. Dyer. View-Invarian Analysis of Cyclic Moion. Inernaional Journal of Compuer Vision, 25(3): , [33] H. S. Shaw and D. D. Deliyski. Mucosal wave: A normophonic sudy across visualizaion echniques. Journal of Voice, 22(1):23 33, January 28. [34] E. Shechman, Y. Caspi, and M. Irani. Increasing space-ime resoluion in video. In ECCV 2: roceedings of he 7h European Conference on Compuer Vision-ar I, pages , London, UK, 22. Springer- Verlag. [35] C. Theobal, I. Albrech, J. Haber, M. Magnor, and H.-. Seidel. iching a baseball: racking high-speed moion wih muli-exposure images. In SIGGRAH 4: ACM SIGGRAH 24 apers, pages , New York, NY, USA, 24. ACM. [36] J. A. Tropp, J. N. Laska, M. F. Duare, J. K. Romberg, and R. G. Baraniuk. Beyond nyquis: Efficien sampling of sparse bandlimied signals. IEEE Transacions in Informaion Theory, 56(1):52 544, Jan. 21. [37] B. Wandell,. Carysse, J. DiCarlo, D. Yang, and A. E. Gamal. Muliple capure single image archiecure wih a cmos sensor. In he Inernaional Symposium on Mulispecral Imaging and Color Reproducion for Digial Archives, pages Sociey of Mulispecral Imaging of Japan, [38] B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Anunez, A. Barh, A. Adams, M. Horowiz, and M. Levoy. High performance imaging using large camera arrays. In SIGGRAH 5: ACM SIGGRAH 25 apers, pages , New York, NY, USA, 25. ACM. Ashok Veeraraghavan is currenly a Research Scienis a Misubishi Elecric Research Labs in Cambridge, MA. His research ineress are broadly in he areas of compuaional imaging, compuer vision and roboics. He received his B.Tech in Elecrical Engineering from he Indian Insiue of Technology, Madras in 22 and M.S and hd. from he Deparmen of Elecrical and Compuer Engineering a he Universiy of Maryland, College ark in 24 and 28 respecively. His hesis received he Docoral Disseraion award from he Deparmen of Elecrical and Compuer Engineering a he Universiy of Maryland. Dikpal Reddy received his B.Tech in Elecrical Engineering from he Indian Insiue of Technology, Kanpur in 25. He is currenly a hd candidae in he Deparmen of Elecrical and Compuer Engineering a he Universiy of Maryland a College ark. His research ineress are in signal, image and video processing, compuer vision and paern recogniion. Ramesh Raskar joined he Media Lab from Misubishi Elecric Research Laboraories in 28 as head of he Labs Camera Culure research group. The group focuses on developing ools o help us capure and share he visual experience. This research involves developing novel cameras wih unusual opical elemens, programmable illuminaion, digial wavelengh conrol, and femosecond analysis of ligh ranspor, as well as ools o decompose pixels ino percepually meaningful componens. Raskar s research also involves creaing a universal plaform for he sharing and consumpion of visual media. Raskar received his hd from he Universiy of Norh Carolina a Chapel Hill, where he inroduced Shader Lamps, a novel mehod for seamlessly merging synheic elemens ino he real world using projecor-camera based spaial augmened realiy. In 24, Raskar received he TR1 Award from Technology Review, which recognizes op young innovaors under he age of 35, and in 23, he Global Indus Technovaor Award, insiued a MIT o recognize he op 2 Indian echnology innovaors worldwide. In 29, he was awarded a Sloan Research Fellowship. He holds 3 US paens and has received hree Misubishi Elecric Invenion Awards. He is currenly coauhoring a book on compuaional phoography.