Lecture Fundamentals of Video technology Gerald Schuller Gerald. schuller@tu-ilmenau.de Assistent: Oleg Golokolenko, ilmenau.de Oleg.Golokolenko@tu- Website www.tu-ilmenau.de/mt -> lectures -> Bachelor for MT -> Grundlagen der Videotechnik We also have a Moodle2 website: https://moodle2.tu-ilmenau.de/course/view.php?id=206
For this lecture, every participant needs to register in Moodle2, to access learning materials (Lectures), seminars and quizzes. Moodle2 link: https://moodle2.tu-ilmenau.de Seminars: The Seminars are based on Raspberry Pi microcontroller, with Raspbian Linux as operating system. There will be biweekly homework assignments. Seminars count 30% of the final grade, and final exam would count remaining 70%. The first homework assignment will help with practicing or revising the Python concepts. Everyone should have a Raspberry Pi or a computer with Linux distribution installed. Ubuntu is relatively easy to install and much more
user-friendly when it comes for programming rather than Windows. Also, Windows causes too many software problems. Ubuntu is the most popular Linux distribution available and can be installed next to Windows (Dual-boot). Instructions for installing Ubuntu: http://www.ubuntu.com/download/desktop/install-ubuntudesktop or: http://www.everydaylinuxuser.com/2015/11/how-to-installubuntu-linux-alongside.html or: http://itsfoss.com/install-ubuntu-1404-dual-boot-modewindows-8-81-uefi/
Programming homework assignments will be uploaded on Moodle2. There will be short quizzes in Moodle2, which counts 25% for homework (you should register in Moodle2) and will be available weekly after every lecture and should be done in one week. These homework tasks are an integral part of this course. This is the only way to get the concepts to acquire the required skills for building video systems. The course has total of 4 LP (credit points). Each LP is about 2-3 hours of work per week, so altogether about 8-12 hours per week.
Note: For every week, around 3 hours will be for lecture, therefore the remaining 5-9 hours will be for homework i.e., about 1 working day per week. Note: The scripts in Copyshop are not up to date. The most up-to-date slides will be available in Moodle2 (and possibly in the website). Note: The lecture slides are not the replacement of course book, they are just a guide for self-study, not as a complete source. For self-study, the slides contain the references and to books and websites. The slides are deliberately kept in the editing mode because they are supposed to function like a blackboard. In this way, we can respond flexibly to questions from students and enter
the answers at the same time. One of the main aims of the lecture is to answer questions. The latest versions of the slides will then be uploaded on Moodle2.
Goal of this lecture: - Learn about the techniques and principles of wireless transmission of video streams and audio streams. - The example of Raspberry Pi: Transmission of an image using sound.
Content overview: - Introduction to Python - Python basics - Psycho-physics (properties of eyes: vision; ear: hearing) - Digital Filters - Digital transmission
- Stone age: cave painting -> Illustration of 3 dimensions on a 2-dimensional surface. - Brain interpolates the original 3-dimensional scenery - Ancient Rome: Wall newspapers, in public places: mass media - Book printing: more distribution. - Modern time: Pinhole camera
Lochblende Objekt - Schirm für Abbildung Unschärfe Nachzeichnen oder Fotoplatte Illustration from 3D to 2D Apparatus, not by hand. - Later: perforated plate replaced by lens for more light.
Photozelle Stromquelle Glühlampe -> 2500 decomposition of the image into lots of pixels, which are transferred individually. -> brain/eye interpolates every individual pixel into connected image.
Nipkow Aufnahme Stromquelle Wiedergabe Bild blende Photozelle Blende für abzutastendes Bild. Je nur ein Loch streift über das Bild Glühlampe, leuchtet entsprechend dem eingefangenem Licht -> but impractical, too complex/costly. Dreht synchron zur Aufnehme Scheibe
-> Image is scanned and rebuilt line by line. -> Principle used Not only local interpolation but also temporal interpolation! The rapid flickering is perceived as constant light. -> Therefore: Reduction of complexity/cost. Limiting this mechanical system: a few lines and a few frames per second, so low quality.
Transmission: High carrier frequency required because of the required bandwidth. TV analog ca. 5,5 Mhz Bandwidth UHF: ca. 470-690 MHz, corr. ca. 200 MHz, ca. 40 TV channels B/W Television, USA: 30, 40s Germany widely established in 50s years
Color TV -> Used: Eye has only receptors for 3 different primary colors (RGB: Red, Green, Blue) Empfindl Wellenlänge -> Just the 3 primary colors need to be transferred
-> Spatial and temporal resolution of color perception is less than in case of brightness perception. -> Required bandwidth for color information is less than for brightness information. -> TV: the lower bandwidth makes it possible to embed the color information into the gaps of B/W-Signal. -> Color TV is backwards compatible with B/W TV. -> Very important for introduction of a new System.
Digital TV: DVB: Digital Video Broadcast: Not backward compatible - But: it uses the same frequencies and channels as the analog TV - Set-top Box for conversion into analog TV is relatively cheap. Thüringen: Nov. 2005, Berlin was 2002 - Uses the higher information density of digital transmission: 1 analog programme is supported by 3-5 digital programmes.
Programme Resolution of analog TV approximately like VGA exactly: 50 Hz countries: 625 lines, 25 frames per sec. 60 Hz countries: 525 lines, 30 frames per sec. Number of Connected to the lines per mains frequency to seconds avoid artifacts in the is similar picture. Digital TV, HDTV -> essential for High Definition TV the transmissionbandwidth - An analog channel is not replaced by 3-5 digital channels Power suply 220 V, 50 Hz 110 V, 60 Hz
of the same resolution, but by one digital channel of higher resolution. - Resolution approximately like modern Computer monitors. HDTV televisions already available (see e.g. last Radio exhibition) Operation : official in USA, Germany: Sky (ehem. Premiere). Satellite programmes: various digital offers. Technique for recording, transmission, playback available. Monitors, color mixing: Example, close-up of a LCD Monitor or TV screen (recorded with a USB Microscope):
1 Pixel Further magnification:
Note: Every Pixel ( Picture Element ) consists of 3 short strips in colors Blue, Green, Red. The colors of the pixels are created by color mixing, depending on the proportion of the three colors, and each pixel represents a small color point of our image. The principle of color mixing is also found in book
printing. Example of a microscope image of a printed color image, the same principle: Brief Introduction to Python with Numpy, Scipy
Note: This is not a course to introduce in Python, but a course on algorithms that are implemented using Python. Knowledge of Python is therefore required. The following is a brief introduction for those who need a quick introduction. Python is a powerful, Object-oriented programming language. At the same time, it does not need to be compiled, but can be interpreted at runtime. It also has a convenient command line interface, making it easy to try out a few program lines. In addition, there is an enormous variety of program packages or libraries for it, and everything in OpenSource! With the command line interpreter "ipython" and the packages "Numpy", "Scipy" and "matplotlib" the operation is very similar to Matlab. Matlab, however, is a commercial, paid program. In addition, Python is more
powerful; Audio and video streams in real-time, which is not in Matlab. Python Numpy and Scipy are packages that are very similar to Matlab, which are generally used in digital signal processing, and therefore also for digital video technology, for the calculation, simulation and implementation of algorithms. Because of the wide distribution and the importance of the ability to program in Matlab, or here in Numpy and Scipy, it is also used for our homework. Octave is the Open Source version of Matlab, however slower and a bit less comfortable. For (non-real-time and non-
video) computation, it is still very comfortable, but because of its limitations, we will use Python. Numpy and Scipy are the packages, that run under Python, they are OpenSource, and can be installed directly on the Raspberry Pi from the Software Center, as python-numpy and python-scipy. Since they run under Python, they also have the extended capabilities of Python, such as video streaming for recording and playback, which we want to use. The most comfortable and to Matlab most similar environment is obtained with ipython and the option --pylab, which automatically loads numpy and matplotlib, and runs under the Raspberry Pi.
First, the installation of ipython in the Linux console window with: sudo apt-get install ipython Start from console window with: ipython --pylab The "--pylab" option is equivalent to calling: ipython and then within ipython, import the relevant libraries : from numpy import * from matplotlib.pyplot import * This also shows a kind of importing of libraries that directly imports all the functions of the library. Most interesting
mathematical functions are in the library "Numpy". We can also use them in python with: import numpy as np The "as np" means that we can refer to the "numpy" library as a "np" in abbreviated form, which simply means simplification in writing. This referencing of the library has the advantage that we can distinguish functions of the same name from different libraries. If we want to access all the functions of Numpy without referencing (if no duplicates from other libraries are there), we can use instead: from numpy import *
Python Basics: On Ubuntu start a shell / console via menu or with the shortcut Alt-Ctrl-T, then enter python followed by the return key. Commands are generally completed with the return key. Alternatively, we type ipython, which is basically the same but contains more comfort features like auto-completion. This opens the interactive mode of Python, where we can now directly enter Python commands, for example: help(<function-name>): help function for command Example:
help(sin) Assignment of a value to a variable in memory with =, e.g. : A = 1 + 2 Display of value of the variable in memory: print(a) Testing for equality with ==, e.g. : if A == B: print(a)
In Python, with Library OpenCV, an image is represented as an array, one Pixel consists of 3 values (1 value per primary color). Example for input of an array (for a Black/White image) in Python: C = array([[1,2],[3,4]]) Or, as the array is a type from the numpy library, so it can be used as: C = numpy.array([[1,2], [3,4]]) For a color image, not only a brightness value appears for each pixel, but an array of 3 intensities, one per basic color. Example:
C = numpy.array([[[1,2,3],[2,3,4]],[[3,4,5], [4,5,6]]]) The intensities of the base colors of the pixel at the position [0,0] (upper left) are then obtained with: C[0,0] array([1, 2, 3]) for pixel of position [0,0] we can obtain the intensity of the primary color Red with: C[0,0,2] 3 Note: Python distinguishes between arrays and matrices. Example for entering a 2x2 matrix in Python: A=matrix([[1, 2],[3, 4]]) Or as it is a part of library numpy, we can define it as :
A=numpy.matrix([[1, 2],[3, 4]]) A matrix([[1, 2], [3, 4]]) B=matrix([[5, 6],[7, 8]]) B matrix([[5, 6], [7, 8]]) Addition of two matrices: A+B
matrix([[ 6, 8], [10, 12]]) Multiplication: A*B matrix([[19, 22], [43, 50]]) Element by element multiplication: multiply(a,b) matrix([[ 5, 12], [21, 32]]) Inverse of a matrix: inv(a)
matrix([[-2., 1. ], [ 1.5, -0.5]]) Alternative representation for the matrix inversion: A.I We can display the dimension of an array or a matrix with the function "shape": shape(a) or A.shape Example of a simple Python script: The function y = 2 * a + 3 * b is to be programmed as a script. Create a file with the extension.py eg: testscript.py
This requires a so-called text editor. A text editor differs from a word processor such as "Word" or "Writer" by storing only the pure text entered, no formatting information. Each programming language only understands files created by editors! A common recommended editor, which runs also on Raspberry Pi, is gedit.if necessary, it can be installed with: sudo apt-get install gedit and we type there the following content: #command: this is a Test script a=4 b=5 y=2*a+3*b
print y This is then stored (with "save as") under the name "testscript.py". The presence of the file can be checked in the terminal window with the command ls (list). It shows all files in the current folder. With "cd foldername" we switch to a subfolder, with "cd.." to the higher folder. The contents of the file can be viewed for checking in the console window with the command: more testscript.py In the console/shell window we then type python testscript.py The output is then: >>23
Alternatively, we can also define a function within Python. To do this we write a file with the name "testfunktion.py", which contains our functions (it can also be several), with the content: def myfunction(a,b): """This is the help text for myfunction...""" y=2*a+3*b return y Note the indentation below the function declaration: The has in python the function of parentheses. Then we start ipython (or we have already started it) with:
ipython In ipython, we can import our function(s) with: import testfunction Now our function is a part of our own Library "testfunction". We then call our function in ipython with parameters (by a = 7 and b = 8): testfunction.myfunction(7,8) Out[6]: 38 Note the prefix of "testfunction". This has the advantage that we can distinguish several similar functions from several files ("libraries"). We obtain the help text of our function with:
help(testfunction.myfunction) >>>Help on function myfunction in the module testfunction: myfunction(a, b) This is the help text for myfunction... In order to avoid the library name "testfunction", if we are sure there are no name conflicts, we can import: from testfunction import * Then we can easily access our function with: myfunction(7,8)
>>>38