Tvheadend - Bug #3134 configuration save/load - improve throughput - save space (was: big starting time) 2015-10-06 12:45 - C vh Status: New Start date: 2015-10-06 Priority: Normal Due date: Assignee: % Done: 0% Category: General Estimated time: 0.00 hour Target version: 4.4 Found in version: 4.0.x Affected Versions: Description Setup: RPi2 with OpenElec + Tvh latest stable/master (no difference) + "quick" SD card + Sat>IP (network is ready before tvh starts) At start time tvh takes a "break" for 30sec to 2min. I got no idea where this comes from. I thought that it tries to load the whole epg.db (~100mb) but even with 0 size db it didn't makes an difference. Trace, debug... shows nothing (maybe wrong values?). I disable all what I can think of (initial epg scan...) at gui but no change. 12:19:36.297 [ INFO] main: Log started 12:19:36.356 [ INFO] config: loaded 12:19:40.840 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 12:19:40.840 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 12:19:40.840 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 12:19:40.840 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 12:21:03.893 [ INFO] dvr: Creating new configuration '' 12:21:03.902 [ INFO] cwc: Attemping to connect to 192.168.0.244:24241 12:21:03.902 [ INFO] descrambler: adding CAID 0963 as quick ECM (Sky UK) 12:21:03.902 [ INFO] descrambler: adding CAID 098C as quick ECM (Sky DE) 12:21:03.902 [ INFO] descrambler: adding CAID 09C4 as quick ECM (Sky DE) 12:21:03.902 [ INFO] descrambler: adding CAID 09CD as quick ECM (Sky IT) 12:21:03.902 [ INFO] epggrab: module eit created 12:21:03.903 [ INFO] epggrab: module uk_freesat created 12:21:03.903 [ INFO] epggrab: module uk_freeview created 12:21:03.903 [ INFO] epggrab: module viasat_baltic created 12:21:03.903 [ INFO] epggrab: module Bulsatcom_39E created 12:21:03.903 [ INFO] cwc: Connected to 192.168.0.244:24241 what is he doing between scanfile: ATSC - loaded 2 regions with 14 networksanddvr: Creating new configuration '' History #1-2015-10-06 12:52 - C vh At an 1 month old (oldest I have) log I saw a value of 20sec "break" - that was even slow in comparison to the expected (other RPi Tvh users) values of about 1-2sec. #2-2015-10-06 13:08 - Jaroslav Kysela Hmm. It's on my laptop: 2017-11-28 1/5
2015-10-06 13:05:41.950 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 2015-10-06 13:05:41.950 [ INFO] mpegts: Loading input/dvb/networks 2015-10-06 13:05:43.785 [ INFO] mpegts:... done 2015-10-06 13:05:43.871 [ INFO] linuxdvb: adapter added /dev/dvb/adapter1 The <cfg>/input/dvb/networks files (all) are parsed and networks, muxes and services are created in that time. iff --git a/src/input/mpegts/mpegts_network_dvb.c b/src/input/mpegts/mpegts_network_dvb.c index 47007c3..f65ba8b 100644 --- a/src/input/mpegts/mpegts_network_dvb.c +++ b/src/input/mpegts/mpegts_network_dvb.c @@ -658,6 +658,7 @@ void dvb_network_init ( void ) dvb_network_builder); /* Load settings */ + tvhinfo("mpegts", "Loading input/dvb/networks"); if (!(c = hts_settings_load_r(1, "input/dvb/networks"))) return; @@ -673,6 +674,7 @@ void dvb_network_init ( void ) htsmsg_destroy(c); + tvhinfo("mpegts", "... done"); void dvb_network_done ( void ) #3-2015-10-06 13:57 - C vh 13:54:53.513 [ INFO] main: Log started 13:54:53.576 [ INFO] config: loaded 13:54:58.214 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 13:54:58.214 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 13:54:58.214 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 13:54:58.214 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 13:54:58.214 [ INFO] mpegts: Loading input/dvb/networks 13:56:05.405 [ INFO] mpegts:... done #4-2015-10-06 21:18 - Jaroslav Kysela Cached? (You need to clear page cache to simulate the boot process: "echo 1 > /proc/sys/vm/drop_caches") #5-2015-10-06 21:18 - Jaroslav Kysela - Subject changed from huge timeouts at starting time to big starting time #6-2015-10-06 21:19 - C vh 2017-11-28 2/5
a reboot should do it? #7-2015-10-06 21:27 - Jaroslav Kysela Yeah. I overlooked difference between 54/56 (minutes). I though that the timediff is only 7 seconds. You may compare, if the start time is lower when the files are cached (start tvh multiple times after boot). Just for a curiosity to check the i/o delays at boot. #8-2015-10-06 22:08 - C vh I had tested this at the mean time, looks like it works "quick - 20sec" if it is cached (new profile is a tvh profile from an x64 htpc) my old very slow (1-2min) profile has ~14000 files at tvh/input/dvb/networks/id the other one from my x64 (20-30sec) has only ~6000 but both profiles have same network, same channels and same config O_o Maybe it is purely overwhelmed by the number of files that the i/o is breaking down (some cache is full?)? I don't know if Tvh has to open every file at input/dvb/... or if it is just on demand? The pure read speed should not be a problem, but to look into 14000 files is a problem ;) If it has to open every file (design fault?) it would explain it. Is there no sort of cache which prevents to load the settings at every start up? For example throw the settings at runtime into a settings.xmlz (or at shut-down) and let the ~1mb xmlz been read instead of the ~k files at startup? cached new profile 20:06:14.049 [ INFO] main: Log started 20:06:14.099 [ INFO] config: loaded 20:06:15.040 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 20:06:15.040 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 20:06:15.040 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 20:06:15.040 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 20:06:28.769 [ INFO] dvr: Creating new configuration '' 20:09:32.826 [ INFO] main: Log started 20:09:32.878 [ INFO] config: loaded 20:09:33.785 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 20:09:33.785 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 20:09:33.785 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 20:09:33.785 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 20:09:47.196 [ INFO] dvr: Creating new configuration '' 20:15:55.918 [ INFO] main: Log started 20:15:55.965 [ INFO] config: loaded 20:15:56.878 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 20:15:56.878 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 20:15:56.878 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 20:15:56.878 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 20:16:10.481 [ INFO] dvr: Creating new configuration '' old profile 20:19:20.629 [ INFO] main: Log started 20:19:20.680 [ INFO] config: loaded 2017-11-28 3/5
20:19:22.269 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 20:19:22.270 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 20:19:22.270 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 20:19:22.270 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 20:19:39.562 [ INFO] dvr: Creating new configuration '' 20:33:57.306 [ INFO] main: Log started 20:33:57.355 [ INFO] config: loaded 20:33:58.276 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 20:33:58.276 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 20:33:58.276 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 20:33:58.276 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 20:34:12.061 [ INFO] dvr: Creating new configuration '' 21:04:53.319 [ INFO] main: Log started 21:04:53.366 [ INFO] config: loaded 21:04:54.283 [ INFO] scanfile: DVB-T - loaded 43 regions with 729 networks 21:04:54.284 [ INFO] scanfile: DVB-S - loaded 1 regions with 92 networks 21:04:54.284 [ INFO] scanfile: DVB-C - loaded 17 regions with 52 networks 21:04:54.284 [ INFO] scanfile: ATSC - loaded 2 regions with 14 networks 21:05:08.165 [ INFO] dvr: Creating new configuration '' #9-2015-10-07 09:45 - Jaroslav Kysela - Subject changed from big starting time to configuration save/load - improve throughput - save space (was: big starting time) - Target version set to 4.4 You should probably clean the dead services / muxes in the config with more files. They're not remove automatically. Anyway, the configuration storage is not optimal. I'm playing with some ideas like to create "snapshots" to a compressed file (like EPG) in a binary format (instead text one). But it's something for 4.4. #10-2015-10-07 13:42 - C vh Maybe some kind of xml file? I get a ~4mb file which could be gziped (fastest) to 300kb. Compression + file size shouldn't be a problem for every platform. In the mean time tx for your quick help (as always) /plugin_assets/redmine_wiki_extensions/images/smile.png #11-2015-10-07 15:40 - C K C vh wrote: Maybe some kind of xml file? I get a ~4mb file which could be gziped (fastest) to 300kb. Compression + file size shouldn't be a problem for every platform. In the mean time tx for your quick help (as always) /plugin_assets/redmine_wiki_extensions/images/smile.png What about using a database like mysql or postgresql for all the storage? this could allow - in a case of a re-installation - to keep your channel 2017-11-28 4/5
nummbers :-) #12-2015-10-07 18:14 - C vh sqlite/libsqlite would do it also (no need for heavy mysql server), would fix the i/o problem also including simple import/backup possibility also sounds like easy implementation downside, it would break the config again :/ -> easy downgrade isn't possible #13-2016-02-11 23:04 - Jaroslav Kysela There are two changes in latest master to reduce disk I/O in the start phase: 1) EPG data can be gziped now (it's on by default - global config) 2) mux config / service config data are merged together and gziped, too (forced - updated with the new data save) It should reduce significantly the loading times when the disk I/O is an issue. Note: You should rescan all networks/muxes to trigger the new save functions. Otherwise, the old configs are not touched. #14-2016-02-12 07:21 - Volker Schmidt Just for the record: Savings at my installation was approx. 10 MBytes. Feeds: Astra 19.2 E plus a local DVB-C cable provider. 2017-11-28 5/5