Page 1 of 2
Spontaneous reboots
Posted: Sun Feb 04, 2018 8:17 pm
by Rudolf
I rooted my Toon fairly recently and I haven't put it to use yet, but I've been noticing some strange behaviour lately; spontaneous restarts.
I noticed because I have to start x11vnc manually and sometimes I can't login because the device has been rebooted. Manually restarting x11vnc is not an issue, but I'd like to solve the underlying issue before I start using Toon as my thermostat.
I just added a small script that logs every reboot with a timestamp in a file using @reboot in cron.
I think it must be a memory issue of one of the apps I installed... But those are nothing out of the ordinary; toonstore, boilerstatus, trash, traffic and buienradar.
So should I log the results of 'free' in a file using cron?
Or can I see the reason for the reboot somewhere on the filesystem, in a file?
Any other stats I should look into? Please advise.
Re: Spontaneous reboots
Posted: Sun Feb 04, 2018 8:54 pm
by Toonz
Typically Toon restarts itself when it runs out of memory.
What you could do is disable some apps you don't really need and see if that resolves the issues.
Personally I don't have all Eneco apps running, many of them only make sense if you have a subscription anyway.
Re: Spontaneous reboots
Posted: Sun Feb 04, 2018 10:21 pm
by michel30
You can also check with top if x11vnc is start up one time.
I had the same problem in the beginning when I root the TOON x11vnc was starting up many times and that was my issue.
Re: Spontaneous reboots
Posted: Mon Feb 05, 2018 8:55 am
by TheHogNL
Could we try compiling a new kernel with swap support enabled and then use a small (but preferably fast) usb stick as a small swap disk? I guess this will offload some less used memory pages to swap to free up some memory.
Re: Spontaneous reboots
Posted: Mon Feb 05, 2018 9:44 am
by marcelr
Quite possible.
Re: Spontaneous reboots
Posted: Mon Feb 05, 2018 8:09 pm
by TheHogNL
I would really try to get into building it myself. I has been a while when I last compiled my own kernels but for sure I'm not into the embedded stuff. Really need to crash course it first
If anyone can provide a few hints and where to start, I would be on my way.
Re: Spontaneous reboots
Posted: Fri Feb 09, 2018 9:41 am
by Rudolf
I've been logging the memory consumption of qt-gui but it looks allright. Had a few reboots in the past week though, it last approx. 30 hours.
Here's the result of 'top', followed by shift + M
to sort by memory consumption.
Code: Select all
Mem: 122320K used, 3876K free, 8384K shrd, 0K buff, 30284K cached
CPU: 39% usr 15% sys 0% nic 44% idle 0% io 0% irq 0% sirq
Load average: 0.56 0.45 0.36 3/141 6403
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
710 1 root S 333m 270% 1% {hcb_comm} HCBv2 hcb_comm [OK] [20 clients] [S1|N1]
708 1 root S 153m 125% 0% /HCBv2/sbin/qt-gui -platform linuxfb -plugin Tslib
731 1 root S 38340 30% 3% {hdrv_zwave} HCBv2 hdrv_zwave [OK] [EBD931A9 SIS 3.67] [S1|N1]
721 1 root S 22316 18% 0% {hcb_config} HCBv2 hcb_config [OK] [S1|N1]
725 1 root S 22276 18% 0% {hdrv_hue} HCBv2 hdrv_hue [OK] [S1|N1]
716 1 root S 21396 17% 0% {happ_thermstat} HCBv2 happ_thermstat [OK] [S20|N1]
713 1 root S 21064 17% 0% {happ_pwrusage} HCBv2 happ_pwrusage [OK] [S30|N1]
723 1 root R 20880 17% 46% {hcb_rrd} HCBv2 hcb_rrd [OK] [S1|N1]
712 1 root S 20316 16% 0% {happ_kpi} HCBv2 happ_kpi [OK] [S1|N1]
724 1 root S 20016 16% 0% {hcb_watchdog} HCBv2 hcb_watchdog [OK] [S20|N1]
709 1 root S 19524 15% 0% {hcb_netcon} HCBv2 hcb_netcon [OK] [S1|N1]
730 1 root S 19296 15% 0% {hdrv_p1} HCBv2 hdrv_p1 [OK] [C:OK E:OP G:OP ] [S1|N1]
715 1 root S 19208 15% 0% {happ_smartplug} HCBv2 happ_smartplug [OK] [S1|N1]
718 1 root S 19136 15% 0% {happ_usermsg} HCBv2 happ_usermsg [OK] [S1|N1]
714 1 root S 19052 15% 0% {happ_scsync} HCBv2 happ_scsync [OK] [S1|N1]
719 1 root S 13784 11% 0% {happ_weather} HCBv2 happ_weather [OK] [S20|N1]
720 1 root S 13552 11% 0% {hcb_bxtproxy} HCBv2 hcb_bxtproxy [OK] [S:0:6:2020 R:0:0 Q:0 (PRD)] [S1|N1]
735 1 root S 12016 10% 0% /HCBv2/bin/lighttpd -D -m /HCBv2/lib/lighttpd -f /HCBv2/etc/lighttpd/lighttpd.conf
711 1 root S 11516 9% 0% {happ_eventmgr} HCBv2 happ_eventmgr [OK] [S1|N1]
722 1 root S 10836 9% 0% {hcb_log} HCBv2 hcb_log [OK] [S1|N1]
544 1 root S 4064 3% 0% /usr/sbin/wpa_supplicant -B -P /var/run/wpa_supplicant.wlan0.pid -i wlan0 -c /etc/network/wpa_supplicant.wlan0.conf -D wext
1143 1 root S 3472 3% 0% /usr/sbin/nmbd -s /etc/samba/smb.conf
6395 5256 root R 3280 3% 1% top
5256 5242 root S 3280 3% 0% -sh
648 1 root S 3168 3% 0% /sbin/syslogd -n -C64 -m 20
613 1 root S 3104 2% 0% udhcpc -b -R -i eth0 -A 5 -p /var/run/udhcpc.eth0.pid -H eneco-001-037311
567 1 root S 3104 2% 0% udhcpc -R -n -p /var/run/udhcpc.wlan0.pid -i wlan0 -x hostname:eneco-001-037311 -t 2
651 1 root S 3104 2% 0% /sbin/klogd -n
1109 1 root S 2616 2% 0% /usr/sbin/chronyd -r
5242 643 root S 2336 2% 0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
643 1 root S 2164 2% 0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
653 1 root S 1772 1% 0% /usr/sbin/cron
1 0 root S 1600 1% 0% init [5]
3 2 root SW 0 0% 0% [ksoftirqd/0]
518 2 root SW 0 0% 0% [flush-ubifs_0_0]
5 2 root SW 0 0% 0% [kworker/u:0]
6 2 root SW 0 0% 0% [watchdog/0]
358 2 root SW 0 0% 0% [ubifs_bgt0_0]
309 2 root SW 0 0% 0% [ubi_bgt0d]
541 2 root SW 0 0% 0% [RTKTHREAD]
3926 2 root SW 0 0% 0% [kworker/0:0]
244 2 root SW 0 0% 0% [kswapd0]
5858 2 root SW 0 0% 0% [kworker/0:1]
136 2 root SW 0 0% 0% [khubd]
5434 2 root SW 0 0% 0% [kworker/0:3]
2 0 root SW 0 0% 0% [kthreadd]
7 2 root SW< 0 0% 0% [khelper]
8 2 root SW 0 0% 0% [kworker/u:1]
11 2 root SW< 0 0% 0% [pm]
119 2 root SW 0 0% 0% [sync_supers]
121 2 root SW 0 0% 0% [bdi-default]
123 2 root SW< 0 0% 0% [kblockd]
235 2 root SW< 0 0% 0% [rpciod]
243 2 root SW 0 0% 0% [khungtaskd]
245 2 root SW 0 0% 0% [fsnotify_mark]
246 2 root SW< 0 0% 0% [nfsiod]
247 2 root SW< 0 0% 0% [crypto]
281 2 root SW 0 0% 0% [mtdblock0]
286 2 root SW 0 0% 0% [mtdblock1]
Code: Select all
eneco-001-037311:~# uptime
07:11:29 up 1 day, 9:08, load average: 0.34, 0.38, 0.35
Code: Select all
eneco-001-037311:~# cat reboot.log
Toon was restarted.
Mon Feb 5 16:45:33 UTC 2018
Toon was restarted.
Tue Feb 6 15:26:07 UTC 2018
Toon was restarted.
Wed Feb 7 22:03:10 UTC 2018
No reboot after wednesday yet, that's quite peculiar.
I disabled all custom apps, to see if the problem goes away. Will report back later.
Re: Spontaneous reboots
Posted: Sun Apr 29, 2018 12:20 pm
by Rudolf
I thought the issue was gone, but I was proven wrong.
After re-enabling all custom apps one by one the issue didn't reappear, until I configured HASS to start polling the boiler-status again using this component:
https://github.com/cyberjunky/home-assi ... rstatus.py
Code: Select all
- platform: toon_boilerstatus
host: toon
port: 10080
scan_interval: 10
resources:
- boilersetpoint
- boilerintemp
- boilerouttemp
- boilerpressure
- boilermodulationlevel
- roomtemp
- roomtempsetpoint
Maybe the http-requests with this small interval introduce a problematic memory-leak?
Does it make sense to monitor lighttpd's memory consumption over time? Any other suggestions?
Re: Spontaneous reboots
Posted: Sun Apr 29, 2018 10:19 pm
by TheHogNL
Reboots are often caused by a lack of memory. Hammering the lighttpd could indeed cause this so following the memory usage of that process and the free memory of the Toon would be a good method to find out of that is indeed the case. Good luck!
Re: Spontaneous reboots
Posted: Mon Apr 30, 2018 9:53 am
by Rudolf
It seems the folder '/var/volatile/tmp/www-cache/boilerstatus' is slowly growing in size.
The system is up for 20 hours, and this folder contains over 1200 files. Only 4,7MiB, but this might continue to grow and reach critical values within days.
I'll try to monitor this folder for a bit longer.
Is there a more elegant solution then flushing this folder from a cronjob?
https://bash.cyberciti.biz/file-managem ... he-script/
I don't see an option for lighttpd to handle this.
Am I the only one experiencing this issue with the boilerstatus-app? Is it because I'm polling the server too often?
Re: Spontaneous reboots
Posted: Mon Apr 30, 2018 10:20 am
by TheHogNL
Looks like it is being caused by mod_compress as a module from lighttpd. Also check
https://redmine.lighttpd.net/projects/1 ... odcompress
Please check /qmf/etc/lighttpd/lighttpd.conf and change cache_dir to "" (empty). According to the documentation this will cause the lighttpd not to cache anymore
Re: Spontaneous reboots
Posted: Mon Apr 30, 2018 11:30 am
by Rudolf
Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.
So for now I implemented this crontab:
Code: Select all
*/5 * * * * find /var/volatile/tmp/www-cache/boilerstatus -type f -mmin +5 -delete
This runs every 5 minutes and deletes the cache that's over 5 minutes old.
Re: Spontaneous reboots
Posted: Sun May 06, 2018 5:32 am
by Fietspomp
Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now
But after this sensor don't work anymore in HASS
Code: Select all
2018-05-06 10:34:27: (../../../src/log.c.168) server started
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir No such file or directory
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down.
from file on Toon /var/volatile/tmp/lighttpd_error.log
So Ill try the cron instead.
Re: Spontaneous reboots
Posted: Sun May 06, 2018 7:09 am
by TheHogNL
Rudolf wrote:Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.
Gzipping the content of an html page is a normal job for a webserver. The only way for lighttpd to do this is to compress it just before sending and caching the result so that a next hit on the same page will not require a zip process again. However this strategy will only work for static pages. Not dynamic like the boilerstatus. It will create a boilerstatus zip cache file everytime it is changed.
Gzipping (and therefore the cache) can be turned off without any negative effect. I think your home network has enough bandwith available
Re: Spontaneous reboots
Posted: Mon May 07, 2018 8:41 am
by Rudolf
Fietspomp wrote:Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now
But after this sensor don't work anymore in HASS
Code: Select all
2018-05-06 10:34:27: (../../../src/log.c.168) server started
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir No such file or directory
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down.
from file on Toon /var/volatile/tmp/lighttpd_error.log
So Ill try the cron instead.
Glad it helped you, I started to think I was the only one running in to this issue.
lighttpd.conf loads all modules with the following directive:
include_shell "/HCBv2/etc/lighttpd/genModules"
and this loads all modules from the <lighttpd>/modules/ folder; maybe you want to try removing mod_compress from that folder.
But, the cronjob is working great for me.