Spontaneous reboots
Moderators: marcelr, TheHogNL, Toonz
Spontaneous reboots
I rooted my Toon fairly recently and I haven't put it to use yet, but I've been noticing some strange behaviour lately; spontaneous restarts.
I noticed because I have to start x11vnc manually and sometimes I can't login because the device has been rebooted. Manually restarting x11vnc is not an issue, but I'd like to solve the underlying issue before I start using Toon as my thermostat.
I just added a small script that logs every reboot with a timestamp in a file using @reboot in cron.
I think it must be a memory issue of one of the apps I installed... But those are nothing out of the ordinary; toonstore, boilerstatus, trash, traffic and buienradar.
So should I log the results of 'free' in a file using cron?
Or can I see the reason for the reboot somewhere on the filesystem, in a file?
Any other stats I should look into? Please advise.
I noticed because I have to start x11vnc manually and sometimes I can't login because the device has been rebooted. Manually restarting x11vnc is not an issue, but I'd like to solve the underlying issue before I start using Toon as my thermostat.
I just added a small script that logs every reboot with a timestamp in a file using @reboot in cron.
I think it must be a memory issue of one of the apps I installed... But those are nothing out of the ordinary; toonstore, boilerstatus, trash, traffic and buienradar.
So should I log the results of 'free' in a file using cron?
Or can I see the reason for the reboot somewhere on the filesystem, in a file?
Any other stats I should look into? Please advise.
Re: Spontaneous reboots
Typically Toon restarts itself when it runs out of memory.
What you could do is disable some apps you don't really need and see if that resolves the issues.
Personally I don't have all Eneco apps running, many of them only make sense if you have a subscription anyway.
What you could do is disable some apps you don't really need and see if that resolves the issues.
Personally I don't have all Eneco apps running, many of them only make sense if you have a subscription anyway.
member of the Toon Software Collective
Re: Spontaneous reboots
You can also check with top if x11vnc is start up one time.
I had the same problem in the beginning when I root the TOON x11vnc was starting up many times and that was my issue.
I had the same problem in the beginning when I root the TOON x11vnc was starting up many times and that was my issue.
Re: Spontaneous reboots
Could we try compiling a new kernel with swap support enabled and then use a small (but preferably fast) usb stick as a small swap disk? I guess this will offload some less used memory pages to swap to free up some memory.
Member of the Toon Software Collective
Re: Spontaneous reboots
Quite possible.
Re: Spontaneous reboots
I would really try to get into building it myself. I has been a while when I last compiled my own kernels but for sure I'm not into the embedded stuff. Really need to crash course it first
If anyone can provide a few hints and where to start, I would be on my way.
If anyone can provide a few hints and where to start, I would be on my way.
Member of the Toon Software Collective
Re: Spontaneous reboots
I've been logging the memory consumption of qt-gui but it looks allright. Had a few reboots in the past week though, it last approx. 30 hours.
Here's the result of 'top', followed by shift + M
to sort by memory consumption.
No reboot after wednesday yet, that's quite peculiar.
I disabled all custom apps, to see if the problem goes away. Will report back later.
Here's the result of 'top', followed by shift + M
to sort by memory consumption.
Code: Select all
Mem: 122320K used, 3876K free, 8384K shrd, 0K buff, 30284K cached
CPU: 39% usr 15% sys 0% nic 44% idle 0% io 0% irq 0% sirq
Load average: 0.56 0.45 0.36 3/141 6403
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
710 1 root S 333m 270% 1% {hcb_comm} HCBv2 hcb_comm [OK] [20 clients] [S1|N1]
708 1 root S 153m 125% 0% /HCBv2/sbin/qt-gui -platform linuxfb -plugin Tslib
731 1 root S 38340 30% 3% {hdrv_zwave} HCBv2 hdrv_zwave [OK] [EBD931A9 SIS 3.67] [S1|N1]
721 1 root S 22316 18% 0% {hcb_config} HCBv2 hcb_config [OK] [S1|N1]
725 1 root S 22276 18% 0% {hdrv_hue} HCBv2 hdrv_hue [OK] [S1|N1]
716 1 root S 21396 17% 0% {happ_thermstat} HCBv2 happ_thermstat [OK] [S20|N1]
713 1 root S 21064 17% 0% {happ_pwrusage} HCBv2 happ_pwrusage [OK] [S30|N1]
723 1 root R 20880 17% 46% {hcb_rrd} HCBv2 hcb_rrd [OK] [S1|N1]
712 1 root S 20316 16% 0% {happ_kpi} HCBv2 happ_kpi [OK] [S1|N1]
724 1 root S 20016 16% 0% {hcb_watchdog} HCBv2 hcb_watchdog [OK] [S20|N1]
709 1 root S 19524 15% 0% {hcb_netcon} HCBv2 hcb_netcon [OK] [S1|N1]
730 1 root S 19296 15% 0% {hdrv_p1} HCBv2 hdrv_p1 [OK] [C:OK E:OP G:OP ] [S1|N1]
715 1 root S 19208 15% 0% {happ_smartplug} HCBv2 happ_smartplug [OK] [S1|N1]
718 1 root S 19136 15% 0% {happ_usermsg} HCBv2 happ_usermsg [OK] [S1|N1]
714 1 root S 19052 15% 0% {happ_scsync} HCBv2 happ_scsync [OK] [S1|N1]
719 1 root S 13784 11% 0% {happ_weather} HCBv2 happ_weather [OK] [S20|N1]
720 1 root S 13552 11% 0% {hcb_bxtproxy} HCBv2 hcb_bxtproxy [OK] [S:0:6:2020 R:0:0 Q:0 (PRD)] [S1|N1]
735 1 root S 12016 10% 0% /HCBv2/bin/lighttpd -D -m /HCBv2/lib/lighttpd -f /HCBv2/etc/lighttpd/lighttpd.conf
711 1 root S 11516 9% 0% {happ_eventmgr} HCBv2 happ_eventmgr [OK] [S1|N1]
722 1 root S 10836 9% 0% {hcb_log} HCBv2 hcb_log [OK] [S1|N1]
544 1 root S 4064 3% 0% /usr/sbin/wpa_supplicant -B -P /var/run/wpa_supplicant.wlan0.pid -i wlan0 -c /etc/network/wpa_supplicant.wlan0.conf -D wext
1143 1 root S 3472 3% 0% /usr/sbin/nmbd -s /etc/samba/smb.conf
6395 5256 root R 3280 3% 1% top
5256 5242 root S 3280 3% 0% -sh
648 1 root S 3168 3% 0% /sbin/syslogd -n -C64 -m 20
613 1 root S 3104 2% 0% udhcpc -b -R -i eth0 -A 5 -p /var/run/udhcpc.eth0.pid -H eneco-001-037311
567 1 root S 3104 2% 0% udhcpc -R -n -p /var/run/udhcpc.wlan0.pid -i wlan0 -x hostname:eneco-001-037311 -t 2
651 1 root S 3104 2% 0% /sbin/klogd -n
1109 1 root S 2616 2% 0% /usr/sbin/chronyd -r
5242 643 root S 2336 2% 0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
643 1 root S 2164 2% 0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
653 1 root S 1772 1% 0% /usr/sbin/cron
1 0 root S 1600 1% 0% init [5]
3 2 root SW 0 0% 0% [ksoftirqd/0]
518 2 root SW 0 0% 0% [flush-ubifs_0_0]
5 2 root SW 0 0% 0% [kworker/u:0]
6 2 root SW 0 0% 0% [watchdog/0]
358 2 root SW 0 0% 0% [ubifs_bgt0_0]
309 2 root SW 0 0% 0% [ubi_bgt0d]
541 2 root SW 0 0% 0% [RTKTHREAD]
3926 2 root SW 0 0% 0% [kworker/0:0]
244 2 root SW 0 0% 0% [kswapd0]
5858 2 root SW 0 0% 0% [kworker/0:1]
136 2 root SW 0 0% 0% [khubd]
5434 2 root SW 0 0% 0% [kworker/0:3]
2 0 root SW 0 0% 0% [kthreadd]
7 2 root SW< 0 0% 0% [khelper]
8 2 root SW 0 0% 0% [kworker/u:1]
11 2 root SW< 0 0% 0% [pm]
119 2 root SW 0 0% 0% [sync_supers]
121 2 root SW 0 0% 0% [bdi-default]
123 2 root SW< 0 0% 0% [kblockd]
235 2 root SW< 0 0% 0% [rpciod]
243 2 root SW 0 0% 0% [khungtaskd]
245 2 root SW 0 0% 0% [fsnotify_mark]
246 2 root SW< 0 0% 0% [nfsiod]
247 2 root SW< 0 0% 0% [crypto]
281 2 root SW 0 0% 0% [mtdblock0]
286 2 root SW 0 0% 0% [mtdblock1]
Code: Select all
eneco-001-037311:~# uptime
07:11:29 up 1 day, 9:08, load average: 0.34, 0.38, 0.35
Code: Select all
eneco-001-037311:~# cat reboot.log
Toon was restarted.
Mon Feb 5 16:45:33 UTC 2018
Toon was restarted.
Tue Feb 6 15:26:07 UTC 2018
Toon was restarted.
Wed Feb 7 22:03:10 UTC 2018
I disabled all custom apps, to see if the problem goes away. Will report back later.
Re: Spontaneous reboots
I thought the issue was gone, but I was proven wrong.
After re-enabling all custom apps one by one the issue didn't reappear, until I configured HASS to start polling the boiler-status again using this component:
https://github.com/cyberjunky/home-assi ... rstatus.py
Maybe the http-requests with this small interval introduce a problematic memory-leak?
Does it make sense to monitor lighttpd's memory consumption over time? Any other suggestions?
After re-enabling all custom apps one by one the issue didn't reappear, until I configured HASS to start polling the boiler-status again using this component:
https://github.com/cyberjunky/home-assi ... rstatus.py
Code: Select all
- platform: toon_boilerstatus
host: toon
port: 10080
scan_interval: 10
resources:
- boilersetpoint
- boilerintemp
- boilerouttemp
- boilerpressure
- boilermodulationlevel
- roomtemp
- roomtempsetpoint
Does it make sense to monitor lighttpd's memory consumption over time? Any other suggestions?
Re: Spontaneous reboots
Reboots are often caused by a lack of memory. Hammering the lighttpd could indeed cause this so following the memory usage of that process and the free memory of the Toon would be a good method to find out of that is indeed the case. Good luck!
Member of the Toon Software Collective
Re: Spontaneous reboots
It seems the folder '/var/volatile/tmp/www-cache/boilerstatus' is slowly growing in size.
The system is up for 20 hours, and this folder contains over 1200 files. Only 4,7MiB, but this might continue to grow and reach critical values within days.
I'll try to monitor this folder for a bit longer.
Is there a more elegant solution then flushing this folder from a cronjob?
https://bash.cyberciti.biz/file-managem ... he-script/
I don't see an option for lighttpd to handle this.
Am I the only one experiencing this issue with the boilerstatus-app? Is it because I'm polling the server too often?
The system is up for 20 hours, and this folder contains over 1200 files. Only 4,7MiB, but this might continue to grow and reach critical values within days.
I'll try to monitor this folder for a bit longer.
Is there a more elegant solution then flushing this folder from a cronjob?
https://bash.cyberciti.biz/file-managem ... he-script/
I don't see an option for lighttpd to handle this.
Am I the only one experiencing this issue with the boilerstatus-app? Is it because I'm polling the server too often?
Re: Spontaneous reboots
Looks like it is being caused by mod_compress as a module from lighttpd. Also check https://redmine.lighttpd.net/projects/1 ... odcompress
Please check /qmf/etc/lighttpd/lighttpd.conf and change cache_dir to "" (empty). According to the documentation this will cause the lighttpd not to cache anymore
Please check /qmf/etc/lighttpd/lighttpd.conf and change cache_dir to "" (empty). According to the documentation this will cause the lighttpd not to cache anymore
Last edited by TheHogNL on Sun May 06, 2018 7:04 am, edited 1 time in total.
Member of the Toon Software Collective
Re: Spontaneous reboots
Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.
So for now I implemented this crontab:
This runs every 5 minutes and deletes the cache that's over 5 minutes old.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.
So for now I implemented this crontab:
Code: Select all
*/5 * * * * find /var/volatile/tmp/www-cache/boilerstatus -type f -mmin +5 -delete
Re: Spontaneous reboots
Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now
But after this sensor don't work anymore in HASS
from file on Toon /var/volatile/tmp/lighttpd_error.log
So Ill try the cron instead.
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now
But after this sensor don't work anymore in HASS
Code: Select all
2018-05-06 10:34:27: (../../../src/log.c.168) server started
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir No such file or directory
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down.
So Ill try the cron instead.
Last edited by Fietspomp on Sun May 06, 2018 10:41 am, edited 1 time in total.
Re: Spontaneous reboots
Gzipping the content of an html page is a normal job for a webserver. The only way for lighttpd to do this is to compress it just before sending and caching the result so that a next hit on the same page will not require a zip process again. However this strategy will only work for static pages. Not dynamic like the boilerstatus. It will create a boilerstatus zip cache file everytime it is changed.Rudolf wrote:Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.
Gzipping (and therefore the cache) can be turned off without any negative effect. I think your home network has enough bandwith available
Member of the Toon Software Collective
Re: Spontaneous reboots
Glad it helped you, I started to think I was the only one running in to this issue.Fietspomp wrote:Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now
But after this sensor don't work anymore in HASS
from file on Toon /var/volatile/tmp/lighttpd_error.logCode: Select all
2018-05-06 10:34:27: (../../../src/log.c.168) server started 2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir No such file or directory 2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down.
So Ill try the cron instead.
lighttpd.conf loads all modules with the following directive:
include_shell "/HCBv2/etc/lighttpd/genModules"
and this loads all modules from the <lighttpd>/modules/ folder; maybe you want to try removing mod_compress from that folder.
But, the cronjob is working great for me.