Tvheadend - Bug #2470 CPU usage slowly increasing to 100% while watching 2014-11-11 16:15 - Milan vn Status: Fixed Start date: 2014-11-11 Priority: Normal Due date: Assignee: % Done: 100% Category: General Estimated time: 0.00 hour Target version: Found in version: 3.9.2066~ge0f8ee4 Affected Versions: Description Hi all, I recently set up tvheadend on my home server to and I noticed that after watching a channel for some time it starts to stutter and buffer continuously. I started investigating and found that when I start playing tvheadend uses <10% cpu, but while watching a channel it will steadily increase to 100% and that's when the problem starts. If I stop watching it will go back to normal. If I change channel it will drop to <10% again, but will start increasing again immediately. The problem occurs on both SD and HD channels, but is a lot more noticeable on HD channels because the cpu usage increases more quickly on those. It happens when I stream a channel using XBMC and also while recording a channel in tvheadend. I'm using IPTV with a softcam to provide the keys for descrambling (AES). The logs (--debug --trace) show only the following warning: 2014-11-11 16:03:00.713 [WARNING] parser: The timediff for TELETEXT is big (921431024), using current dts I'm running Tvheadend 3.9.2066~ge0f8ee4 on Debian 7.7. I'm quite sure you'll need some more info, so please let me know what you need to know and I'll be happy to provide it. Associated revisions Revision fc32a92f - 2016-03-18 16:09 - Jaroslav Kysela IPTV: udp:// - don't allow to receive non-raw UDP data, fixes #2470 Revision 4c75447a - 2016-03-18 16:42 - Jaroslav Kysela mpegts input: mpegts_input_recv_packets - add retry, fixes #2470 History #1-2014-11-11 22:48 - Jaroslav Kysela Use gprof or any other tool to determine, where the tvh spends the time. 1) add -pg to the Makefile after -g -O2 2) run tvh from cmdline (sources) -./build.linux/tvheadend <YOUR_ARGS> 3) gprof./build.linux/tvheadend > analysis.txt 2017-11-30 1/7
#2-2014-11-11 23:49 - Milan vn - File profiler.txt added The requested file is attached here. Unfortunately, all times show 0.00 so only calls and functions are listed. #3-2014-11-12 11:11 - Milan vn Some more info: The problem does not seem to happen with unencrypted channels. Also, while it occurs during the recording of an encrypted channel, playback of the recording is without any problems (until the point where the CPU maxed out during the actual recording). Seems like a descrambler issue to me. #4-2014-11-12 16:49 - Jaroslav Kysela Do you use DVBCSA library? If yes, could you try the internal DES decrambler? #5-2014-11-12 20:54 - Milan vn I checked my.config.mk and it said CONFIG_TVHCSA = yes... CONFIG_DVBCSA = no so I should be using the internal descrambler. I configured and built with --enable-dvbcsa and tried again, but unfortunately I got the same result. Is there any way to get more output than with --debug --trace? #6-2014-11-14 19:36 - Jaroslav Kysela Could you also try the "time" test? On cmd-line "time <tvh_binary>" and show me the result. I just want to check the sys time compared to the real time. #7-2014-11-14 20:21 - Milan vn Here you go: real user sys 7m59.273s 2m55.211s 0m11.637s These timings were taken as follows: -set the cpufreq governor to powersave to reach 100% cpu a bit faster -started time tvheadend -watched an HD stream in VLC until the stream started stuttering and top showed tvheadend using 100% cpu 2017-11-30 2/7
-stopped tvheadend CPU is an Intel Celeron G1820 running at 2x 800MHz for this test. #8-2014-11-14 23:26 - Jaroslav Kysela OK. Thanks. So it's not a kernel issue. Could you repeat the gprof test, but use -O0 (zero) instaed -O2 in the CFLAGS in Makefile (with -pg) and keep the vlc running when it start stutter for longer time (say 2 minutes or more) to collect statistics in which routines tvh stays most of time? #9-2014-11-15 20:20 - Milan vn - File profiler-o0.txt added Ok, I did as you said and let it run for a few minutes after the CPU hit 100%. In the end, VLC showed an input bitrate varying from 20-300kb/s (stream should be ~12Mb/s H264) and took forever to fill the buffer each time. The result from gprof is attached here. #10-2014-11-26 18:03 - Milan vn Any progress on this? I just rebuilt and can confirm it's still present in 3.9.2167~g39a4db4 #11-2014-11-26 21:12 - Jaroslav Kysela No idea, the gprof output is not much helpful.. Does 'top -H' show which thread(s) consume(s) the all CPU time? #12-2014-11-27 10:18 - Milan vn PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23329 hts 20 0 469m 107m 2608 R 99,9 0,9 9:11.31 iptv_input_thre That's what top -H gives me after playing a stream for a while. #13-2014-11-27 15:22 - Jaroslav Kysela OK, so it's the first thread which grabs data from the network. iptv_input_thread.. Do you use UDP or HTTP url? #14-2014-11-27 18:21 - Milan vn All streams I use are multicast UDP streams. #15-2014-11-27 18:41 - Milan vn Your last response gave me an idea for a workaround: I set up udpxy to get the streams via http instead. This seems to work without any problems! 2017-11-30 3/7
Now I just need to reconfigure all my muxes to use udpxy as a source and I can watch TV again. If you still want more info to squash this bug, please let me know. I'm still happy to help out where I can. #16-2014-11-27 19:32 - Jaroslav Kysela If you can do a little C programming, could you look to iptv_udp_read() function in src/input/mpegts/iptv/iptv_udp.c? Perphaps some printf() might explain this behaviour. It's unusual. #17-2014-11-28 13:35 - Milan vn Unfortunately, my knowledge of C is close to zero. I tried to have a look, but I was unable to see what's going on or where it goes wrong. #18-2014-11-28 17:24 - Jaroslav Kysela diff --git a/src/input/mpegts/iptv/iptv_udp.c b/src/input/mpegts/iptv/iptv_udp.c index 42ba3ef..e0b60d4 100644 --- a/src/input/mpegts/iptv/iptv_udp.c +++ b/src/input/mpegts/iptv/iptv_udp.c @@ -81,10 +81,12 @@ iptv_udp_read ( iptv_mux_t *im ) ssize_t res = 0; n = udp_multirecv_read(um, im->mm_iptv_fd, IPTV_PKTS, &iovec); + printf("multirecv: %i\n", n); if (n < 0) return -1; for (i = 0; i < n; i++, iovec++) { + printf("multirecv[%i]: %li\n", i, iovec->iov_len); sbuf_append(&im->mm_iptv_buffer, iovec->iov_base, iovec->iov_len); res += iovec->iov_len; } What values are printed in 100% CPU usage? Note that your should run tvh directly on command-line (no daemon mode).. #19-2014-11-28 18:25 - Milan vn Ok, that gives me a constant stream of to following: multirecv: 1 multirecv[0]: 1328 From time to time it returns 2 or 3 at a time, eg: multirecv: 2 multirecv[0]: 1328 multirecv[1]: 1328 2017-11-30 4/7
But most of the time it's just 1. However, when the CPU usage reaches 100% it will increase the number up to 32 and stay there all the time, repeating the following over and over: multirecv: 32 multirecv[0]: 1328 multirecv[1]: 1328 multirecv[2]: 1328 multirecv[3]: 1328 multirecv[4]: 1328 multirecv[5]: 1328 multirecv[6]: 1328 multirecv[7]: 1328 multirecv[8]: 1328 multirecv[9]: 1328 multirecv[10]: 1328 multirecv[11]: 1328 multirecv[12]: 1328 multirecv[13]: 1328 multirecv[14]: 1328 multirecv[15]: 1328 multirecv[16]: 1328 multirecv[17]: 1328 multirecv[18]: 1328 multirecv[19]: 1328 multirecv[20]: 1328 multirecv[21]: 1328 multirecv[22]: 1328 multirecv[23]: 1328 multirecv[24]: 1328 multirecv[25]: 1328 multirecv[26]: 1328 multirecv[27]: 1328 multirecv[28]: 1328 multirecv[29]: 1328 multirecv[30]: 1328 multirecv[31]: 1328 #20-2014-11-28 19:17 - Jaroslav Kysela What;s network bandwith in this situation? #21-2014-11-28 22:15 - Milan vn The web interface (status>stream) initially shows just over 12000kb/s. When the CPU hits 100% this decreases to around 150kb/s. #22-2015-06-24 16:04 - Jan Saris Milan vn wrote: 2017-11-30 5/7
The web interface (status>stream) initially shows just over 12000kb/s. When the CPU hits 100% this decreases to around 150kb/s. I have exactly the same problem. Did you find any solution for this issue? #23-2016-03-17 20:16 - Milan vn I realise this is quite an old thread, but I wanted to provide an update as I was looking into this issue again (I want to get rid of udpxy). In the latest version (4.1-1698~g7adea4a) this bug is still present. I did notice though, that it only occurs when I add the streams with the udp protocol specified in the url. When I use rtp://<address>:<port> instead, there seems to be no problem at all so far. I will test this further over the next few days to confirm that this definitely works and that even on the longer term there are no issues with this method. #24-2016-03-17 20:41 - Jaroslav Kysela Could you run strace on thread with 100% CPU usage? #25-2016-03-17 21:48 - Milan vn - File 100cpu.trace added Here you go, output of strace -p <pid> on the tvh:iptv thread with ~100% CPU usage. I ran the trace for approximately 10 seconds before stopping it again. Note that the exact same channel with rtp:// instead of udp:// in the mux url works without any problems. Also - and I'm not sure if it was the same before - the rise in CPU usage seems to be exponential, increasing the rate as it gets closer to 100%. #26-2016-03-18 09:39 - Jaroslav Kysela Thanks. I forgot to note that I need the timing and statistics 'strace -ttt...' 'strace -wc...'. It would be also nice to run this at the start to compare 'working - less than 100% CPU' and 'busy - 100% CPU' behaviour, too. #27-2016-03-18 12:10 - Milan vn - File juststartedwc.trace added - File juststartedttt.trace added - File 100cpuwc.trace added - File 100cputtt.trace added Both commands ran for about 10 seconds in both conditions, so a total of 4 files are attached. If I'm not mistaken, mremap could indicate a memory (re)allocation problem, but the memory usage on my system is not showing anything out of the ordinary. There's still plenty of RAM available and tvheadend's usage is not showing any abnormal increase either. #28-2016-03-18 15:58 - Jaroslav Kysela From logs: The behaviour seems fine. When started, about 1 network packet is processed / 1000 iterations, when load increases 5 packets / 250 2017-11-30 6/7
iterations and under full load, there are about 12 packets / 80 iterations. So it's fine. Thinking more - do you know, the you are trying to pass RTP packets as the raw UDP input? In this case, TVH tries to synchronize (looking for start of MPEG-TS packets - searching for 0x47 bytes with 188 offsets). You're receiving packets with RTP headers (no 0x47 byte as first - it's on offset 12). I think that the behaviour is caused when the TVH buffers more packets (CPU is doing something else) and the massive resync is activated. More buffering - more resync - more CPU usage. As you said, when you switched to RTP, everything is fine (because the RTP header is skipped before any further MPEG-TS processing). OK, it seems that I need to add some checks to the code to refuse RTP input for the raw UDP to avoid these reports :-) #29-2016-03-18 16:42 - Jaroslav Kysela - Status changed from New to Fixed - % Done changed from 0 to 100 Applied in changeset commit:tvheadend fc32a92f818dfabcbc10b2da04af65067511ecbc. #30-2016-03-18 16:48 - Jaroslav Kysela I added raw UDP check for udp:// support and also I think that I found a little bug which explain the remaping (reallocations). There was missing one loop when the MPEG-TS packets were processed and the resync occurs. This caused the 100% CPU usage. Fixed in v4.1-1708-g4c75447. #31-2016-03-18 16:49 - Jaroslav Kysela Also fixed in v4.0.9-1-g7f3fb87. #32-2016-03-19 00:38 - Milan vn Confirmed fixed in 4.1.1708-g4c75447! The log now shows "iptv: receving non-raw UDP data for <mux> in <network>!" upon scanning the mux and will fail the scan thus preventing the problem from occurring in the first place. Adding the mux as rtp:// instead still works as it should. Thanks for working this out and of course for the great project in general! Files profiler.txt 52.4 KB 2014-11-11 Milan vn profiler-o0.txt 623 KB 2014-11-15 Milan vn 100cpu.trace 3.38 MB 2016-03-17 Milan vn juststartedwc.trace 594 Bytes 2016-03-18 Milan vn 100cpuwc.trace 648 Bytes 2016-03-18 Milan vn juststartedttt.trace 7.1 MB 2016-03-18 Milan vn 100cputtt.trace 3.97 MB 2016-03-18 Milan vn 2017-11-30 7/7