OPTIMIZING VIDEO SCALERS USING REAL-TIME VERIFICATION TECHNIQUES

Transcription

1 OPTIMIZING VIDEO SCALERS USING REAL-TIME VERIFICATION TECHNIQUES Paritosh Gupta Department of Electrical Engineering and Computer Science, University of Michigan Valeria Bertacco Department of Computer Science and Engineering, University of Michigan ABTRACT Video hardware design is slowly becoming top money grosser for many semiconductor companies. As per statistics the video industry involved in content broadcasting and format change alone has become almost a $1 Billion dollar industry in the United States. And now with the advent of 3D televisions, video gaming etc. this industry is bound to grow even more. Video based designs almost completely rely on algorithms for post/pre processing, asynchronous bridges etc. With so many applications and devices to view this content on, video scaling is an integral aspect of the complete video pipeline. All video consumer devices will have a scaler at their I/O interface. The quality of scaled input will determine performance of the complete video engine. Then why have a video scaler? Why not simply use the full frame instead? This would indeed be an obvious choice in such scenarios. Though keeping optimization in mind, a certain tradeoff needs to be met in between performance and memory requirements. In this project, we present a novel scheme of using a verification-based checker module to automate this tradeoff while also maintaining low power consumption. Keywords Polyphase filters, Autofocus, Video scaler 1. INTRODUCTION Video scalers are essentially used for converting the size of an input frame, which is part of some video sequence. In parallel, a scaler is also used for outputting a frame by transforming its size to that of the destination device. Due to this generic function, one would expect to find a video scaler in almost every consumer device. Currently many video standards exist under the SD, HD, WXGA etc. Figure 1 shows the most common standards used for defining a video frame. For example one of the most common is the 1920x1080, often called the 1080I. With so many standards existing, designing a scaler that is compatible for all standards become all the trickier. The design of a scaler is most critical for a video pipeline. At the input interface, the scaler will convert the source into a desired input, which will then be used in the rest of the pipeline. Hence any significant quality drop in the input source will have an accumulative effect throughout the pipeline. Similarly at the output interface, data will be scaled to desired output Figure 1: Types of Video Displays configuration. Any error in the output will immediately have an apparent visible effect. This would make it obvious to the reader to always use the full input video stream for the internal pipeline. Then the next obvious question would be as to why do we need an input video scaler if we plan to use the full frame in our pipeline? The answer is data bandwidth. Suppose you have an input video stream in 1080I. That means that you would need at least a 16MB memory slot dedicated just for storing a single video frame. If this pipeline also supports motion estimation sort of other features, then an additional three frames need to be stored. Hence at least 16*4 = 64MB memory needs to be allocated for frames. This would need to be on-board memory otherwise we are liable to have latency issues. Having a 64MB memory dedicated for just frame storage is too costly and indeed quite a waste of hardware resources. Hence we need an input video scaler, to downscale the input frames so as to use the hardware resources more efficiently. In this project we propose a design scheme for integrating the scaler and autofocus model along with a checker model. The checker model will compare the output from autofocus to a golden reference. A feedback signal is generated based on the comparison made by the checker model. The feedback signal generated is sent back to the scaler, which then alters the downscaling, based upon the

2 kind of feedback received. Hence in this scheme, the autofocus stats generated on full frame serve as a golden reference. The comparison between this golden reference and the autofocus stats generated for downscaled video frame is used as an error parameter. The checker model generates this error parameter. It then compares this error with the threshold provided by the user. This comparison is sent as a feedback to the video scaler. If the feedback shows that error is beyond the threshold, then downscaling is lowered. If error is well within the threshold, then downscaled is increased. 2. VIDEO OVERVIEW Every image seen in digital format is composed of pixels. An image is shown in figure 2 depicting the grid of pixels in the magnified window. comparing to the first two images. This leads us to the polyphase filter-bank implementation. Such filters are simply a combination of horizontal and vertical tap filtering simultaneously being done on the input source image so as at use the hardware more efficiently (i.e. in lesser number of clock cycles). One of the most common schemes in doing horizontal plus vertical polyphase filter implementation is shown in figure 4. A four x four grid of pixels in original frame is downscaled to just one pixel. Hence its downscaled by 4 on x and y. Its corresponding visualization is shown in figure 4. Figure 4: 4x4 Pixel Grid downscaled to a single pixel Figure 2: Pixel Representation The expanded window shows a grid of about 20x20 pixels. Any cheap video scaler will resize the image by simply decimating it. This means selecting alternate pixels or dropping pixels at regular frequency. Many such decimation schemes exist. In figure 3, some of the common decimation and interpolation schemes are shown. Actual engineering goes into developing such schemes that do not simply select pixels but recreate them (for e.g. interpolation). In the right most image in figure 3, we can see that the quality of rendered image is much better when As we can see from figure 4, the 4x4 grid of pixels is first vertically downscaled to 1x4 grid of pixels. Then they are horizontally downscaled to a single pixel. In this whole procedure none of the final pixels obtained were selected from amongst the original pixels. They were all created using a combination of input pixels. This is a representative of a good scaler scheme. Hardware circuit implementation of the above algorithm is shown in figure 5. The circuit shown in figure 5 is for an eight-order filter. The final pixel y(n) is generated using x(k-4) to x(k+3) pixels. The vertical/horizontal tap coefficients are pre-stored in the Coefficient Table. The question now would be as to how should we decide on what kind of a downscaler to use. A Video scaler cannot decide on what type of downscaling should be done. This depends on the kind of Interphase c(-4) c(-3) c(-2) Coefficient table c(-1) c(0) c(1) c(2) c(3) x(k-4) x(k-3) x(k-2) x(k-1) x(k) x(k+1) x(k+2) x(k+3) New input sample request y(n) Figure 3: Left and middle images depict Decimation schemes. Right image shows Interpolation being done. Figure 5: Video downscaler implementation

3 application that this video scaler is supporting. For our purpose, the video scaler is driving an input for the Autofocusing algorithm. The autofocus algorithm outputs a focused image from the input stream that it is provided by the Video scaler as an output. The video scaler generates this output by using the full frame video input. 3. DESIGN MODEL We now have the video scaler and the autofocus model in place. We now propose the scheme in which we integrate the checker model with the previous two blocks. Essentially we want to demonstrate that runtime verification techniques can be used to quantify the performance of autofocus. Based on the output score, a feedback will be generated to determine the kind of downscaling that needs to be done. This is quite novel, since verification has mostly been related to finding bugs or quantifying errors. Here we also use verification for similar purposes but with a slight difference. We use the comparison between the golden reference and the output AF score as a form of error. Our golden here is the AF score, which was computed using a patch on the original frame. If the difference between the golden and output AF score is beyond a certain threshold then downscaling is reduced. This will deteriorate the memory requirements but the quality of video output will increase. If the AF output score is well within the reference golden, then we downscale the frame further to have an even higher memory performance without suffering a significant fall in quality of image. This helps in using the hardware resource much more efficiently. This checker scheme helps in automating the whole process of video scaling. If such a scheme is implemented in consumer devices, then the user does not have to bother about manually doing the video scaling. The algorithm will take this decision. This full scheme described is shown in figure 6. Here the blocks in blue are the functional blocks. The video scaler downscales an image. Autofocus model generates the score. It sort of quantifies the performance of the video scaler. The blocks in yellow form the verification pipeline. Here the checker block selects a small macro block from the original frame to have an initial golden estimate for the input frame. This serves as golden reference. This golden score is compared with the score generated by the AF block. Based on the comparison, a feedback (shown in red) is provided to the scaler block. This feedback can be either of the three: to reduce downscaling, to increase downscaling or to do nothing. This is the basic scheme that I developed in MatLab. In this scheme, the novelty is the way we use the concept of run-time verification, not in detecting bugs, but in selecting the performance of the Video Scaler. The proposed model was fully implemented in MatLab. The results are shown in experiments section ANALYZING CHECKER OUTPUT The dry run of the proposed scheme is shown in figure 7. The stats generated by the autofocus model are the outputs of various kernel functions that were used on the downscaled input frame. These stats are basically plots of score functions versus time frame. What is critical are the peaks detected in the stats. They should essentially be sharp enough for the hill-climbing algorithm to be able to detect them. The hill-climbing algorithm basically tells us the direction in which to move, to acquire the peak of the given functions. Hence if the peaks are not that strong and have a weak differential value, then the performance of the hillclimbing algorithm will suffer. Therefore when we mention that the checker model compares the golden reference versus the stats from downscaled frame, the checker model is essentially taking the difference Feedback selects Scaling VIDEO STREAM MACROBLOCK CHECKER BLOCK CHECKER FEEDBACK VIDEO SCALER USE FEEDBACK TO DECIDE OPTIMUM SCALING AUTO FOCUS MODULE COMPUTES STATS AS A PARAMETER TO QUANTIFY SCALING Figure 6: Proposed design model DECISION MODEL FEEDBACK Figure 7: Feedback from checker selects the downscaling parameter between these peaks and checking whether or not they are within a certain specified threshold level or not. It accordingly generates the feedback signal. Hence as we can see in figure 7, the stats generated for original frame have sharper peaks compared to the stats generated for the

4 downscaled frame. This is evident since for a full frame video, all the information for that frame is available. The surprising aspect of this work was that even when the frame was downscaled by up to 2 or even 4 levels, then also the autofocus performance evaluated by checker model was quite impressive. The full design was implemented in MatLab. The design implementation was split into five blocks. The video scaler was made up of two blocks. The autofocus model was split into three blocks for better modularity. In current design implementation about 200 flops would be required. In consumer-based models, about 400 to 600 flops would be required considering the fact that for better auto-focusing, we would need information from previous frames also. The checker model implemented would need about 50 to 70 flops. Since the checker model is actually a comparator with the threshold being provided as an input by the user. This input is used to compare the difference in autofocus scores for the golden and downscaled video frame. What is interesting is that the design model proposed is fully scalable, since the checker model will always be the same size irrespective of the input frame being provided to it. Hence for larger input frame sizes, the proportion of flops used by the checker model will be lesser when compared to the flops required by rest of the design. 5. EXPERIMENTS Several experiments were performed to test the robustness and quality of the proposed scheme. The checker-based scheme does have a good performance. The checker model keeps a tab on the focus quality and keeps changing the scaler formats so as to have the best focus possible. deciding on the kind of downscaling to be done. To get an idea of the comparisons being made, the statistics are first generated on full frames. Certain windows are marked out, on which autofocus scores are plotted. The plots shown in figure 9 are basically autofocus scores versus time. So suppose for a specific colored window we see a peak in Figure 8: Comparing outputs; Left: Original frame; Right: Downscaled frame In figure 8, the downscaled frame is compared to its corresponding original frame. The difference in quality can clearly be observed. This is based on the checker model Figure 9: Experimental results. Top: Video Frame showing the windows marked; Middle: autofocus stats on original full frame; Bottom: autofocus stats on input frame downscaled by 2

5 autofocus score at time frame t. That means that that window was best focused for time frame t. Ideally, this is what all scalers would generate, but having full input frame buffered requires one to have the ability to handle high traffic. Hence full frame stats are not used. These are initially used just once to get an initial estimate of the reference parameters, which will be used, as golden. Once the initial frame is processed, the golden reference is redefined in every iteration. In the figure 9, we use the initial frame. The windows are the ones that are being tracked across time frames by the checker model. The autofocus model accordingly generates stats for these windows so as to compute their score. The two plots shown in figure 9 show the stats that are generated. The middle plot in figure 9 shows the stats computed using full input frame. Hence it is evident to have such strong peaks since we have the complete information available for those specific windows in the full scale frame. In the second stats figure, results are generated for an input frame, which is downscaled by 2. On comparing both these stats we can easily see that the peaks in second set of results are much softer when compared to the former. Though the peaks are still quite well pronounced to have the hill climbing algorithm be successfully used. As confirmed by the experiments, the autofocus algorithm does a good job in detecting the peaks on a frame downscaled by 2 and 4 respectively. In figure 10, the performance of auto-focusing can clearly be appreciated. Image on left is the input frame in full scale. Image on right is the output from the autofocus model. The checker model here converged on suppose we have three sets to choose from when generating the autofocus stats. If we select the kernel functions in autofocus model used for generating the third stats set, then having a downscale by 2 would indeed be a problem. Since on comparing we can see that that for the third set of stats generated by the autofocus model, those generated for downscaled by 2 are not that good. Detecting a peak on the downscaled image using the third set of stats is simply not possible. Hence in such scenarios we will need to use the original frame. Though if the kernels are selected from amongst the ones used for generating the first or second set Figure 10: Left: Original window frame; Right: Video frame downscaled by 2. Downscaled image is visibly sharper downscaling by 2. The difference in quality focus is clearly evident in figure 10. Further results are presented below in figure 11, where the autofocus model computes three sets of stats using different kernel functions for each of these three stats. The window is shown in red in both the frames. The first set of stats is for the original frame, while the second set of stats is for the frame downscaled by 2. The important aspect to note here is the jitter introduced in the peaks because of the downscaling in the second set. So Figure 11: Top: Three sets of autofocus stats generated on input video frame downscaled by 2; Bottom: Same set of autofocus stats generated on full- scale input frame of stats, then having a downscaled by 2 input video frame is good enough and we would not need to use the original frame. So we also get to see here the benefits of working on a patch-type model. The output of autofocus from every

6 patch can then be combined to finally generate a full output image. Another advantage of having an automated decision model such as the checker block is when there is excessive motion in the video sequence. In this case having a fixed downscaled scheme or having a scheme where the user has to select the downscale factor is not a good scheme. Since when there is excessive motion, the user will have to keep changing the threshold of error. Or if the algorithm is not good enough it will simply keep falling back to full-scale input. We show in figure 12, the results that we got using our checker-based scheme. The input frames of a video sequence are artificially rotated in MatLab. It is true that such a discontinuous motion would probably not be possible in real-life scenarios, but this benchmark does help in testing the robustness of our approach. The face of the person in the photograph is targeted within the red window. The first set is having the original frame. In the second set we artificially rotate the frame by 90 degrees and then again generate the stats. Downscaling the input frame by 2 generates the stats shown. The first set of stats is good. A peak can easily be detected. The second of stats are completely random. Here it is impossible for any algorithm to detect any kind of a peak in such highly random stats. When detecting, the algorithm would most probably be trapped within a local maxima. In such a case, the autofocus score would fall much below the threshold, hence the checker model would automatically send a feedback to the scaler, to fall back onto the full frame definition. simplicity in its implementation lowers the probability of having any kind of bugs in the design. Lastly, I present Table 1, which shows a comparative analysis in terms of the number of clock cycles required. Higher the number of clock cycles required, higher will be the power consumed by component. Here we compare the complete pipeline of generating stats on full frame versus generating them on frame down scaled by four. The full frame is first read as input. Then the data is read in terms of parallel packets. The number of taps used in the polyphase filter determines the packet size. Then convolution is performed using the input frames. Finally we assume that the internal logic is having around 500 flip-flops. These calculations are shown in Table 1. The table shows that using a frame down-scaled by 4 clearly is about 16 times faster than when using full frame. Hence we end up using lower number of clock cycles. This essentially means we end up consuming lower power. Besides the improvement in power consumption, one can clearly observe that we also have a performance improvement. The full frame pipeline that initially took 8032M clock cycles, now only takes 502M clock cycles with the checker model being used in design. This is an improvement by almost 16 times. Full Frame Full Frame 1920x1080pix = 16MB Frame is down scaled by /4x1080/ 4pix= 1 MB Polyphase Input of Order 8 16 M clocks 1M clocks Convolution with Kernels functions 32 M clocks 2 M clocks Assume Design requires 500 flops 16M*500 = 8000M clocks 1M*500 = 500M clocks TOTAL 8032M clocks 502M clocks Figure 12: Top: Two sets of autofocus stats generated on frame downscaled by 2; Bottom: Same set of autofocus stats generated on input frame downscaled by 2 and rotated by 90 degrees Table 1: Power Consumption levels are shown in terms of clock cycles. Lower the clock cycles consumed, lower is the power consumption With all the results shown above, it clearly proves that our proposed model does indeed give a good performance. The novelties of this approach are that it is very effective, highly robust and also very simple to integrate. The 6. CONCLUSIONS In this project, we propose a design model integrating the video scaler and autofocus models along with a checker block. Up till now, checker blocks have always been used

7 for typical verification purposes such as detecting bugs, finding mismatches etc. In this project, we quantify the quality of focusing being done. This quantity is generated from the autofocus model. It is then compared with the golden reference. For our design, golden reference is the output of the autofocus module using the full frame video as an input. These two values are then compared and their difference is used as a form of error. The checker model that has been put in place compares this error with the threshold value that has been set. If the error goes beyond a certain threshold, then a feedback is sent to the scaler to reverse the downscaling on input video stream. This will help in maintaining the quality of focus on output image. If the error is well within the threshold, then a feedback is sent to the scaler to increase the downscaling. This will help in optimizing memory usage. As our experiments have clearly proven, this scheme works very well. Downscaling by 2 or 4 factors leads to a memory reduction up to 4 and 16 times respectively. Another added advantage of this scheme is in power consumption. It also helps in improving the performance of design. Reducing the memory requirements has an immediate impact of reducing the number of clock cycles required for full-pipeline. This means that with a downscaled image we can get an output in sufficiently lesser number of clocks than what we would have gotten had we been using the full video frame. When considering the reduction in clock cycles over a complete video sequence, the power conserved is indeed significant. Any amount of power conserved gives the device an added advantage of greater portability. In terms of future work, this checker-based methodology can be used in other parts of a video pipeline. The special characteristic of a video pipeline that differentiates it from other sub-systems is that we get qualitative errors in design. These errors will not generate wrong results in terms of incorrect data being inputted. Though these errors will have a visible impact on the output. Hence we need automated checking mechanisms to monitor certain key outputs so as to automatically change configuration of design. The checker- based model that was integrated in this pipeline can be made to monitor many aspects of the design besides the focusing content. An interesting way to use this checker model would be in estimating the maximum clock frequency at which the design can operate. It will be an interesting problem and indeed a most challenging one. 7. REFERENCES [1] Austin, T., Bertacco, V., Blaauw, D., and Mudge, T Opportunities and challenges for better than worst-case design. In Proceedings of the 2005 Asia and South Pacific Design Automation Conference (Shanghai, China, January 18-21, 2005). ASP-DAC '05. ACM, New York, NY [2] Dong Wang; Nanning Zheng; Pengju Ren; Kaifang Lu, "A new ringing detection based adaptive video scaler with parallel memory architecture," Industrial Electronics and Applications, ICIEA rd IEEE Conference, vol., no., pp , 3-5 June 2008 [3] Ramaswamy, A.; Nijim, Y.; Mikhael, W.B., "Polyphase implementation of a video scalar," Signals, Systems & Computers, Conference Record of the Thirty-First Asilomar Conference, vol.2, no., pp vol.2, 2-5 Nov 1997vol.2, no., 2-5 Nov 1997 [4] Baoxin Li; Chellappa, R.; Qinfen Zheng; Der, S.Z., "Modelbased temporal object verification using video," Image Processing, IEEE Transactions on, vol.10, no.6, pp , Jun 2001