CARDIOWATCH: A SOLUTION FOR MONITORING THE HEART RATE ON A MOBILE DEVICE

U.P.B. Sci. Bull., Series C, Vol. 78, Iss. 3, 2016 ISSN 2286-3540 CARDIOWATCH: A SOLUTION FOR MONITORING THE HEART RATE ON A MOBILE DEVICE Andreea Lavinia Popescu 1, Radu Tudor Ionescu 2, Dan Popescu 3 This work proposes an algorithm for monitoring the heart rate on a mobile device using data captured by the camera while pressing a fingertip against the embedded camera. The method proposed in this article computes the heart rate using the mean of pixels in the RGB color space on a frame by frame basis, giving a higher weight to the Red and the Green channels in order to increase accuracy. The values are computed in real-time at 30 frames per second. To reduce the variation induced by the noisy readings, two moving average procedures are sequentially applied to obtain a smooth signal. The next step is to compute the temporal frequency of the signal by counting down the peaks within a window of 10 seconds. Finally, the computed frequency is used to determine the number of heart beats per minute. The proposed algorithm is robust and fast enough to work in real-time on a mobile device with limited computational power. To demonstrate its usefulness, the algorithm is embedded in a mobile application called CardioWatch, which is available in the Apple App Store at http://appstore.com/cardiowatch. Keywords: signal processing; moving average; photoplethysmogram; PPG; health monitoring device; heart rate monitor; heart rate device; mobile devices. 1. Introduction In nowadays, people have become more and more preoccupied about closely monitoring their health, stimulated by the recent development of sophisticated and expensive health monitoring devices. Indeed, people no longer have to go to the doctor in order to monitor their heart rate, their blood pressure, or their glucose level. Wearable devices provide the means to constantly monitor one s health, even during physical exercises or sleep. One the most popular kind of monitoring provided by health monitoring devices is that of the 1 Faculty of Automatic Control and Computer Science, Politehnica University of Bucharest. 313 Splaiul Independentei Street, Bucharest, Romania. E-mail: andreea.lavinia@ymail.com 2 Department of Computer Science, University of Bucharest. 14 Academiei Street, Bucharest, Romania. E-mail: raducu.ionescu@gmail.com 3 Faculty of Automatic Control and Computer Science, Politehnica University of Bucharest. 313 Splaiul Independentei Street, Bucharest, Romania. E-mail: dan popescu 2002@yahoo.com 63

64 Andreea Lavinia Popescu, Radu Tudor Ionescu, Dan Popescu heart rate. Companies that provides wearable devices with sensors for heart rate monitoring have developed algorithms to reduce the noise generated by humans movements in order to provide accurate results. Perhaps the first and most used kind of device is the chest strap which measures the hearts electrical activity. This strap usually communicate via Bluetooth with mobile devices such as smart phones. Other wearable devices used for heart monitoring are bracelets and, more recently, smart watches. Despite the growing popularity of wearable devices, simple and far more cheap solutions should not be disregarded. This work presents a reliable solution for heart rate monitoring directly on a smart mobile device, completely eliminating the need to purchase an extra (wearable) device that could perform this task. More precisely, the mobile device serves to capture video recordings while pressing a fingertip against its camera. The proposed method is to compute the heart rate using the mean of pixels in the RGB color space, frame by frame. A higher weight is associated to the Red channel, naturally because the blood color is red. The Green channel is also given a higher weight, since it seems to be the most informative channel under motion artifact conditions according to [1]. Even so, the readings captured at 30 frames per second are noisy. To reduce the noise in the data, two moving average procedures are sequentially applied to smooth out the signal. Next, the temporal frequency of the signal is computed by counting down the peaks within a short window of 10 seconds. Finally, the computed frequency is used to determine the number of heart beats per minute (bpm). The proposed algorithm is embedded in a mobile application called CardioWatch, which is already available in the Apple App Store at http://appstore.com/cardiowatch. Almost every person in nowadays already has a smart phone in his or her pocket, which can be immediately used as a heart rate monitor by simply installing an application, such as CardioWatch. This a much cheaper yet reliable solution for heart rate monitoring than wearable devices. The paper is organized as follows. Related work about heart rate monitoring and control is discussed in Section 2. The algorithm proposed in this work is described in Section 3. Some empirical results are presented in Section 4. The particularities of the heart rate monitoring application for mobile devices and its architecture are described in Section 5. Finally, the conclusions are drawn in Section 6. 2. Related Work There are many solutions for monitoring the heart rate of a person. They can be roughly divided into three or four categories. The first category is represented by professional medical devices such as electrocardiogram (ECG) devices, stethoscopes, and so on. These are usually more accurate than any other kind of devices. Moving away from professional medical devices, there is the category of wearable devices such as chest straps, bracelets, and smart

CardioWatch: A solution for monitoring the heart rate on a mobile device 65 watches. The wearable devices also provide very accurate readings. The third category is represented by software algorithms designed for detecting the heart rate right on a mobile device, by measuring the pixel intensity changes generated by placing a fingertip on top of the optical sensor (camera). This basically transforms the mobile device into a device capable of capturing a photoplethysmogram (PPG) [2, 3]. These are much cheaper than a wearable device or a visit to the doctor, but they sometimes provide less reliable results, depending on the illumination conditions. However, many studies that have compared PPG and ECG measurements of heart rate reached to the conclusion that the measurements are highly correlated [4]. Furthermore, similar high correlations between PPG and ECG measurements were observed when smart phones are used to record the PPG signal [5, 6, 7, 8]. The fourth category is comprised of approaches that do not require any kind of contact with the monitored subject [9, 10, 11, 12]. This kind of solutions are very appealing to control and security applications, which aim to monitor subjects from a distance, without the subject s knowledge or consent. Most of the research efforts from the recent years went into developing solutions that fall in one of the last two categories [5, 7, 8, 9, 10, 11, 13]. It comes to no surprise that the algorithm proposed in this work belongs to the third category. Unlike other approaches from the same category [5, 13], the proposed algorithm is designed to work in strict illumination conditions in order to always provide reliable results. More specifically, the camera flashlight is turned on while the user s places one of his fingertips on the camera. A control image is also displayed on the screen to let the user know if the finger is correctly placed in order to obtain the optimal accuracy. Although PPG approaches to measure heart rate have been available from a long time ago [2], PPG measurements have been used until recently only in clinical settings to monitor patients [3]. In the recent years, several PPG approaches for mobile devices emerged [5, 6, 7, 8, 14]. The authors of [5] and [6] provide statistical evidence that the PPG signal measured on a mobile device (ios or Android) is correlated with ECG measurements. Furthermore, the work of [7] shows that a mobile phone can serve as an accurate monitor for several physiological variables such as breathing rate, cardiac RR intervals, and blood oxygen saturation. Their method is based on the mobile phone s ability to record and analyze the varying color signals of a fingertip placed in contact with its camera. The goal of [14] is to demonstrate how smart phones, and more specifically PPG applications that use the optical sensor of a smart phone, can be used to measure differences in heart rate as a function of relived emotional experiences such as happiness and anger. The work of [13] describes a method for monitoring the heart rate using a low-end video camera. Their measurement technique is based on extracting beat-to-beat intervals by passing the color intensity average through a processing pipeline comprised of six stages. Similar

66 Andreea Lavinia Popescu, Radu Tudor Ionescu, Dan Popescu to the pipeline presented in this work, they also include a moving average filter. However, their pipeline is designed specifically for personal computers and thus, it consumes more resources than the algorithm proposed in this work. The authors of [8] try to go further and determine the heart rate variability by finding tiny fluctuations in the time intervals between heartbeats in the PPG signal. The recordings come from a mobile device, but their analysis is done offline on a desktop computer. The approaches in the fourth category have impressive results considering that they are able to remotely determine the heart rate. However, these kind of approaches are usually prone to inaccurate results due to illumination and motion variations. Eulerian Video Magnification (EVM) is proposed in [9] to reveal temporal variations in videos that are impossible to observe with the naked eye and display them in an indicative manner. EVM takes a video recording as input, and applies spatial decomposition, followed by temporal filtering to the frames. The resulting signal is then amplified to reveal hidden information such as the flow of blood in the face. The authors of [10] developed FaceBEAT, an iphone application for remote heart rate measurement based on facial video. To extract more accurate cardiac pulse signal, they applied Independent Component Analysis to the raw trace signal. The authors of [11] go deeper on this path and determine that the forehead and the cheek face regions are more relevant for computing the heart rate. Their findings lead to an optimized face scanning method, faster processing times and better pulse detection results. However, the authors of [12] note that such methods work well on stationary subjects under well controlled conditions, but their performance significantly degrades when the videos are recorded under more challenging conditions, specifically when subjects motions and illumination variations are involved. In [12], a framework which utilizes face tracking and Normalized Least Mean Square adaptive filtering methods to counter the influences of such variations is proposed. The framework obtains considerably better results than the state of the art approaches. 3. Method An algorithm that computes the heart rate on a mobile device in reliable and efficient manner is presented in this section. More precisely, the algorithm transforms the mobile device into a photoplethysmograph. A photoplethysmogram (PPG) is an optically obtained plethysmogram, which is defined as a volumetric measurement of an organ usually resulting from fluctuations in the amount of blood or air it contains. In order to determine the heart rate, the change in volume caused by the pressure pulse is detected by illuminating the fingertip with the built-in LED flashlight of the mobile device. Then, the amount of light reflected to the camera sensor is recorded in a video sequence for about 15 seconds. There are 30 frames per second recorded, but each frame is processed in real-time to extract the mean of the pixels in the RGB color

CardioWatch: A solution for monitoring the heart rate on a mobile device 67 FIG. 1. The fast processing pipeline for real-time computing on a mobile device. The mean of pixels in the RGB color space on a frame by frame basis is computed first. To remove the noise, two moving average procedures are sequentially applied to obtain a smooth signal. Finally, the peaks are detected and the average time interval between consecutive peaks is computed.

68 Andreea Lavinia Popescu, Radu Tudor Ionescu, Dan Popescu space. Since the mean values are computed at the acquisition time, it is no longer necessary to store the video sequence for subsequent processing. It is important to mention that when the mean is computed, only one pixel in every 2 2 patch is considered. In other words, the mean is rather estimated based on 25% of the total amount of pixels in each frame. This enables the algorithm to run in real-time on a mobile device with limited computational resources. Increasing the number of pixels will cause the frame rate to drop and variate under 30 frames per second, which is very prohibitive for accurately computing the heart rate. Another important remark is that the RGB channels do not have even contributions to the mean. Indeed, a higher weight of 0.4 is associated to the Red channel, naturally because the blood color is red and the main color in each captured frame is bright red. The Green channel is also given a weight of 0.4, specifically because in [1] it was found to be the most informative channel under motion artifact conditions. The remaining Blue channel has a weight of 0.2. Put together, the mean pixels values computed at a frame by frame basis produce a signal in which each cardiac cycle appears as a peak. At this point however, the signal contains a lot of noise which has to be removed by further processing. Before any other processing steps, the first 5 seconds of the recorded signal are removed due to very large variations in the computed mean values caused by the delay of the auto-calibration of the LED flashlight and the camera focus. The remaining 10 seconds of the signal enter the denoising phase. In this phase, the noise is removed by applying a moving average procedure, which is commonly used with time series to smooth out short-term fluctuations and highlight longer-term trends or cycles. In the recorded signal, the shortterm fluctuations correspond to noise and the longer-term cycles correspond to the heart beat cycles. The moving average procedure considers the unweighted (simple) mean of the previous 6 data points. Empirical results showed that this step removes a great amount of noise, but occasionally, some fake peaks determined by noise still appear on the graph. This phenomenon seems to occur more often if the heart rate is above 100 beats per minute. Consequently, another simple moving average procedure based the previous 3 data points is applied to further smooth out the signal. Finally, the peaks are detected and the time elapsed between each two consecutive beats is computed. Note that, a timestamp is stored along with each mean pixel value. When two peaks are detected, the precise amount of time between them can be determined by subtracting their timestamps. An alternative solution would have been to rely on the fact that the frames are generated at 30 frames per second, but actually the number of frames per second depends on the CPU load at a certain time, and thus, it can fluctuate uncontrollably. Considering this fact, using a precise timestamp is much more reliable. The average time between consecutive peaks is computed to obtain an accurate estimation of the heart rate. The entire processing pipeline is illustrated in Figure 1. Empirical observations indicate

CardioWatch: A solution for monitoring the heart rate on a mobile device 69 FIG. 2. Raw input signal and smoothed signal measured on a rested person. The signal computed by the mean pixel values is colored in red. The smoothed signal after applying two moving average procedures (with 6 and 3 steps, respectively) is colored in blue. The time of the recorded values is represented on the horizontal axis and the mean values of the pixels in each frame are represented on the vertical axis. The detected peaks are represented by purple dots. that the heart rate value produced by the proposed algorithm based on the 10 seconds window is less than ±4 bpm off, compared to an ECG. Obviously, this accuracy level is obtained only if the fingertip is correctly placed on the optical sensor during the entire 10 seconds recording. 4. Empirical Results The proposed method was tested in different illumination conditions. The algorithm works fine even if the mobile device does not have a builtin flashlight, but, in this case, the finger must be illuminated by an external lighting source such a desktop lamp, for example. The approach was evaluated on 8 subjects of different ages. There are 5 PPG measurements per subject. Two of the subjects were also asked to perform physical exercises to make sure that the algorithm is properly tuned for a broad range of heart rate frequencies. Another 5 PPG measurements were taken right after 5 or 10 minutes physical exercises. In the end, the algorithm was tested on 50 recorded PPG measurements and the resulted heart rates range from 54 bpm up to 132 bpm. All the computed heart rates were compared to ECG measurements and the error rate was less than ±4 bpm. Some interesting examples are presented next. The raw signal illustrated in Figure 2 is recorded immediately after the optical sensor and the flashlight is turned on. In the first 50 frames the optical sensor and the flashlight are automatically calibrated, and for this reason, the signal shows significant light variations that prevent the detection of the heart pulse. Even if there are two peaks detected in the first 50 frames, it can be observed that the time interval between the first two peaks is shorter than the

70 Andreea Lavinia Popescu, Radu Tudor Ionescu, Dan Popescu FIG. 3. Raw input signal and smoothed signal measured on a person right after intensive physical exercises. The signal computed by the mean pixel values is colored in red. The smoothed signal after applying two moving average procedures (with 6 and 3 steps, respectively) is colored in blue. The time of the recorded values is represented on the horizontal axis and the mean values of the pixels in each frame are represented on the vertical axis. The detected peaks are represented by purple dots. FIG. 4. Raw input signal and smoothed signal measured on a rested person when the LED flashlight is turned off and there is no other artificial light source nearby. The signal computed by the mean pixel values is colored in red. The smoothed signal after applying two moving average procedures (with 6 and 3 steps, respectively) is colored in blue. The time of the recorded values is represented on the horizontal axis and the mean values of the pixels in each frame are represented on the vertical axis. The detected peaks are represented by purple dots. time intervals between the other consecutive peaks. This indicates that the first peak is not accurately determined and this has an impact on the accuracy of the computed heart rate. Such patterns were observed in several recordings from different subjects. To prevent any negative impact on the accuracy, the first 5 seconds of the recorded signal are simply removed. Figure 2 shows that the denoising procedure outputs a smooth signal in which the peaks can

CardioWatch: A solution for monitoring the heart rate on a mobile device 71 easily be detected. The heart rate computed through the proposed pipeline is 75 bpm and the error from an ECG measurement is almost two bpm. A normal healthy person should have a heart rate between 60 and 85 bpm, so the detected heart rate in the case illustrated in Figure 2 is well within the normal values. However, the proposed approach was also tested on a rested professional athlete. While resting, professional athletes usually have a lower heart beat rate than normal persons. Indeed, the heart rate of the professional athlete computed through the proposed pipeline was around 54 or 56 bpm and the errors were lower than one bpm compared to ECG measurements. It can easily be observed in the diagram illustrated in Figure 3 that the detected peaks are more frequent compared to Figure 2. Certainly, the heart rate is much higher because the subject was asked to perform physical exercises for 5 minutes before recording the heart rate. Once again, the first 50 frames had to be eliminated to prevent any negative influence on the results. The computed heart rate in this case is 108 bpm and the error is roughly three bpm compared to an ECG measurement. When the results of the approach proposed in this paper were compared to ECG measurements, the built-in flashlight of the mobile device was turned on. Nevertheless, the framework was also tested with the flashlight turned off and without any external artificial light. The raw signal illustrated in Figure 4 was recorded in broad daylight. The raw signal has a lot of noise, but the denoising procedure seems to work well enough even in this case. Most of the peaks are correctly detected, but the time interval between consecutive peaks seems to variate. Although this variation can be demoted by computing an average over all the time intervals as in the last step of the algorithm, it can still have a significant impact on the error rate. Indeed, several measurements were made without using the flashlight, and the error rate can go up to 10 or 15 bpm. This result highlights the importance of using the flashlight in order to obtain accurate measurements with the proposed method. On the positive side, the heart rate can be accurately predicted if there is an artificial light source in a very close range (10 to 15 cm), such that the fingertip color appears to be bright red. 5. Mobile Device Application CardioWatch is an ios application available on the Apple App Store at http://appstore.com/cardiowatch. The processing pipeline presented in the current work is embedded in the mobile application. The heart rate is computed from the frames captured by the backside facing optical sensor of the iphone. The application uses the flashlight to remove some of the noise during the image acquisition process. However, the application can also run on devices that do not have a built-in flashlight (such as the ipad), given that there is an external light source available in a very close range. In order to determine the heart rate, the user must place a fingertip to fully cover the backside

72 Andreea Lavinia Popescu, Radu Tudor Ionescu, Dan Popescu FIG. 5. Two screen captures of the CardioWatch ios application. The screen capture on the left-hand side shows the message displayed when the predominant color of the input video does not match the color inside the control circle. The screen capture on the right-hand side shows that signal is being recorded when the fingertip is gently placed on the backside facing camera. camera of the device. The fingertip must rest on top of the camera for 15 seconds without sudden movements. The time required by the method to determine the heart rate is about 10 seconds. The first 5 seconds are removed to eliminate the signal perturbation caused by the automated calibration of the flashlight and the optical sensor. The user has to trigger the image acquisition process by tapping on the START button. As illustrated in Figure 5, the application can automatically detect if the fingertip is not properly placed on the backside optical sensor. In this case, the previously recorded data is removed. CardioWatch automatically restarts the image registration process if the mean of the pixels in the RGB color space is between 70 and 130. A preview of the source video is displayed on the screen in the upper circle illustrated in both screen captures presented in Figure 5. The predominant color in the recorded video should match the color in the bottom control circle. The first 5 seconds window gives the user enough time to adjust the position of the fingertip such that the colors inside the two circles match perfectly, which ensures an accurate measurement of the heart rate.

CardioWatch: A solution for monitoring the heart rate on a mobile device 73 6. Conclusion An algorithm for monitoring the heart rate on a mobile device using data captured by pressing a fingertip against the built-in optical sensor was proposed in this work. The proposed framework is based on computing the mean of pixels in the RGB color space on a frame by frame basis, giving a higher weight to the Red and the Green channels. A denoising procedure is then applied to smooth out the raw input signal. Empirical results showed that the algorithm produces fast and accurate heart rate measurements for healthy persons. However, the algorithm was not tested on people with cardiac disorders such as arrhythmia. In order to compute the heart beat frequency for a person with arrhythmia a different method should probably be applied, more specifically, a method that is not based on averaging the time intervals between consecutive detected heart beats. Remarkably, since the blood flow to the skin can be modulated by multiple other physiological systems, the PPG measurements can also be used to monitor breathing or other circulatory conditions. In future work, the aim is to extend the ios application by including algorithms to detect other circulatory conditions besides the heart rate in order to provide a more complete assessment of the user s health. Acknowledgments Andreea-Lavinia Popescu has been funded by the Sectoral Operational Programme Human Resources Development 2007-2013 of the Ministry of European Funds through the Financial Agreement POSDRU/159/1.5/S/134398. R E F E R E N C E S [1] K. Matsumura, P. Rolfe, J. Lee, and T. Yamakoshi, iphone 4s Photoplethysmography: Which Light Color Yields the Most Accurate Heart Rate and Normalized Pulse Volume Using the iphysiometer Application in the Presence of Motion Artifact? PLoS ONE, vol. 9, no. 3, p. e91205, 03 2014. [2] A. V. J. Challoner, Photoelectric plethysmography for estimating cutaneous blood flow, Non-Invasive Physiological Measurements, vol. 1, pp. 125 151, 1979. [3] J. Allen, Photoplethysmography and its application in clinical physiological measurement, Physiological Measurement, vol. 28, no. 3, pp. R1 R39, 2007. [4] T. Kageyama, M. Kabuto, T. Kaneko, and N. Nishikido, Accuracy of pulse rate variability parameters obtained from finger plethysmogram: A comparison with heart rate variability parameters obtained from ECG, Journal of Occupational Health, vol. 39, pp. 154 155, 1997. [5] J. Bolkhovsky, C. Scully, and K. Chon, Statistical analysis of heart rate and heart rate variability monitoring through the use of smart phone

74 Andreea Lavinia Popescu, Radu Tudor Ionescu, Dan Popescu cameras, in Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, Aug 2012, pp. 1610 1613. [6] M. J. Gregoski, M. Mueller, A. Vertegel, A. Shaporev, B. B. Jackson, R. M. Frenzel, S. M. Sprehn, and F. A. Treiber, Development and Validation of a Smartphone Heart Rate Acquisition Application for Health Promotion and Wellness Telehealth Applications, International Journal of Telemedicine and Applications, vol. 7, 2012. [7] C. Scully, J. Lee, J. Meyer, A. Gorbach, D. Granquist-Fraser, Y. Mendelson, and K. Chon, Physiological Parameter Monitoring from Optical Recordings With a Mobile Phone, IEEE Transactions on Biomedical Engineering, vol. 59, no. 2, pp. 303 306, Feb 2012. [8] R. Peng, X. Zhou, W. Lin, and Y. Zhang, Extraction of Heart Rate Variability from Smartphone Photoplethysmograms, Computational and Mathematical Methods in Medicine, vol. 2015, 2015. [9] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. Freeman, Eulerian Video Magnification for Revealing Subtle Changes in the World, ACM Transactions on Graphics, vol. 31, no. 4, pp. 65:1 65:8, Jul. 2012. [10] S. Kwon, H. Kim, and K. S. Park, Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone, in Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, Aug 2012, pp. 2174 2177. [11] D. Datcu, M.-A. Cidotă, S. Lukosch, and L. J. M. Rothkrantz, Noncontact automatic heart rate analysis in visible spectrum by specific face regions, in Computer Systems and Technologies, CompSysTech 13, Ruse, Bulgaria, B. Rachev and A. Smrikarov, Eds. ACM, 2013, pp. 120 127. [Online]. Available: http://dl.acm.org/citation.cfm?id=2516775 [12] X. Li, J. Chen, G. Zhao, and M. Pietikainen, Remote Heart Rate Measurement from Face Videos under Realistic Situations, in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, June 2014, pp. 4264 4271. [13] L. Pestritu, A. Todiruta, M. Goga, and N. Goga, Method for measuring the heart rate through fingertip using a low-end video camera and its application in self care, in Bioinformatics and Bioengineering (BIBE), 2013 IEEE 13th International Conference on, Nov 2013, pp. 1 4. [14] D. Lakens, Using a Smartphone to Measure Heart Rate Changes during Relived Happiness and Anger, IEEE Transactions on Affective Computing, vol. 4, no. 2, pp. 238 241, April 2013.