ECE3296 Digital Image and Video Processing Lab experiment 2 Digital Video Processing using MATLAB Objective i. To learn a simple method of video standards conversion. ii. To calculate and show frame difference which is widely used in video compression iii. To detect motion in video sequences. Equipment MATLAB software version R2009b. Introduction This lab experiment consists of three parts. In part 1, you will learn a method in video standards conversion which is called frame rate up-conversion. Video standards conversion is actually the task of converting digital video from one standard to another. This task is necessary as various digital video systems, ranging from all-digital-high-definition TV to videophone, have different spatio-temporal resolution requirements. This leads to the emergence of different format standards to store, transmit, and display digital video signals. Frame rate up-conversion increase the temporal sampling rate in progressive video. Examples of frame rate conversion include conversion from a digitized motion picture recorded with a temporal rate of 24 frames/sec to NTSC format, which requires 60 fields/sec. Example of product that uses frame rate up conversion is the Samsung 46" high-definition LCD TV with auto motion plus 120Hz technology. It virtually eliminates any motion blur and creates smooth transitions between frames. It basically calculates the image between frames and inserts it, creating a non-repetitive transition from one to the next. The result is a clearer picture especially during a slow motion video. In part 2 of the lab, you will calculate frame difference between the current frame and the previous frame of a video sequence. The frame difference can indicate the amount of motion present in the video. In the frame difference, the lighter pixels are those whose intensity has changed with respect to the previous frame due to motion. The frame difference is usually computed in the video compression using the modified original previous frame. The video compression methods usually employ a two step procedure to compress the video sequence. The first step involves prediction of the frame to be coded by using a motion compensation algorithm on the previous frame(s) which has been already coded (and is therefore available to the decoder). After subtracting the motion predicted frame from the frame to be coded, we are left with a residual image which is termed Displaced Frame Difference (DFD). The DFD is usually coded using a transform based technique. In part 3, you will use the Video and Image Processing Blockset to detect motion in video sequences using a method derived from the frame difference technique in part 2. The objective is to detect whether the current video shots have large motion or little motion. One application example is in a video surveillance camera system. The system will initiate alarm signal if awkward motion (such as thief snatching and running) is detected. Revised: January 2010 (Hezerul Abdul Karim), February 2011 (Sarina Mansor) 1
Lab Answer Sheet Name:.. Student ID:.. Major:.. Part 1: Frame rate up conversion a) Download image sequence foreman.qcif from the MATLAB server. b) Download file YUVviewer from the MATLAB server. c) Play the sequence using the YUVviewer. Frame size is QCIF. d) Type the command below at MATLAB editor or in notepad and save as readimg.m: Nx=176; %Vertical size of image% Ny=144; %Horizontal size of image% Nt=50; %No of frames in image sequence% A = zeros(ny,nx); fid=fopen('c:\foreman.qcif','r','b'); %Input image sequence% figure; for t=1:nt, A(:,:)=transpose(fread(fid,[Nx,Ny],'uint8')); %Reading the Y (luminance) information% tmp=fread(fid,[nx/2,ny/2],'uint8'); %Reading the U (chrominance) information% tmp=fread(fid,[nx/2,ny/2],'uint8'); %Reading the V (chrominance) information% B = A / max(a(:)); imshow(b); imwrite(b,sprintf('c:\\foreman%03d.jpg',t),'jpg','quality',100); %Writing 1 frame (the luminance information) to a file% end fclose(fid); The commands above will write the luminance information of the images in the video sequence to files. e) Read three frames of the sequence into MATLAB array for example frame 20, 21 and 22. Choose frames that show large motion. f) Perform frame interpolation between frame 20 and frame 22 using repeat previous frame method and pixel averaging method. So you will have frame 20, frame 20 (repeated frame) and frame 22 for repeat previous frame method; and frame 20, (frame 20 + frame 22)/2 and frame 22 for pixel averaging method. g) Calculate PSNR between the original frame 21 and the repeated frame db h) Calculate PSNR between the original frame 21 and the averaged frame. db i) Compare both PSNRs, spatial and temporal picture quality and suggest which method is better. Revised: January 2010 (Hezerul Abdul Karim), February 2011 (Sarina Mansor) 2
Student ID: j) How can you relate this experiment to frame rate up conversion? (Hints: Assume the original foreman image sequence is recorded at 30 frames/sec) Part 2: Frame difference a) You may use the same image sequence from part 1 of the lab for this experiment. b) You may use some of the source code developed in part 1 for part 2. c) Choose two frames from the image sequence. You may name it current and previous frame (e.g. frame 10 is the previous frame and frame 11 is the current frame) d) Calculate the absolute value of the frame difference between current and previous frame. e) Multiply the difference by a constant value, k (e.g. k=3), and show the image of the frame difference. f) Discuss with your lab partner how this frame difference can be minimized. Part 3: Detect Motion in Video Seqences a) From the Matlab main window, select File->New->Model. A new simulink window will appear. Save this model in the hard disk. b) On the new simulink window, select Help->Blockset->Video and Image Processing Blockset. The Simulink Block Data Type Support for Video and Image Processing Blockset window will appear. This window contains the libraries (block diagram) needed to perform your simulation. Click on the Block Matching to see the examples of the block diagram. Double clicking on the block diagram will reveal the details to be filled in for that block. c) The blocks that you need for your simulation are shown in Figure 1. Revised: January 2010 (Hezerul Abdul Karim), February 2011 (Sarina Mansor) 3
Student ID: Figure 1: Block sets needed to detect motion d) Search the block sets shown in Figure 1 from the Video and Image Processing Blockset and the Simulink Block Data Type Support for all libraries and drag them to your simulink window. Arrange the block sets appropriately. e) The idea of this experiment is to detect large motion on video sequences. If there is large motion, a text Big motion is to be inserted in the video frame. The large motion can be detected using the sum of absolute differences (SAD) between current frame and previous frame. If the SAD is larger than a threshold value, the current frame can be considered to have large motion. f) Download long_interview_qcif.yuv sequence from the MATLAB server. g) Connect the block diagrams appropriately and filled in the necessary parameters for each block diagram. You may leave the parameters as default values. h) Run the simulink model by clicking Simulation->Start or the Start simulation button. i) You may use other block sets that are not shown in Figure 1 in your experiment. j) Discuss your observation:.... k) Identify the suitable threshold value to identify the big motion. Threshold value for long_interview_qcif.yuv: l) Repeat your simulation with foreman.qcif sequence. Evaluate your observation and compare with the previous video sequence..... Threshold value for foreman.qcif: End of experiment Revised: January 2010 (Hezerul Abdul Karim), February 2011 (Sarina Mansor) 4
Marking Scheme You are required to write down your answers in the lab sheet. The distributed marks are as follows: 1. Part 1 10% 2. Part 2 5% 3. Part 3 10% Revised: January 2010 (Hezerul Abdul Karim), February 2011 (Sarina Mansor) 5