Spontaneous reboots

If your (rooted) Toon doesn't do what it's supposed to, post here. Dutch allowed / Posten in 't Nederlands toegestaan.

Moderators: marcelr, TheHogNL, Toonz

Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Spontaneous reboots

Post by Rudolf »

I rooted my Toon fairly recently and I haven't put it to use yet, but I've been noticing some strange behaviour lately; spontaneous restarts.
I noticed because I have to start x11vnc manually and sometimes I can't login because the device has been rebooted. Manually restarting x11vnc is not an issue, but I'd like to solve the underlying issue before I start using Toon as my thermostat.

I just added a small script that logs every reboot with a timestamp in a file using @reboot in cron.
I think it must be a memory issue of one of the apps I installed... But those are nothing out of the ordinary; toonstore, boilerstatus, trash, traffic and buienradar.

So should I log the results of 'free' in a file using cron?
Or can I see the reason for the reboot somewhere on the filesystem, in a file?

Any other stats I should look into? Please advise.
Toonz
Forum Moderator
Forum Moderator
Posts: 1873
Joined: Mon Dec 19, 2016 1:58 pm

Re: Spontaneous reboots

Post by Toonz »

Typically Toon restarts itself when it runs out of memory.
What you could do is disable some apps you don't really need and see if that resolves the issues.
Personally I don't have all Eneco apps running, many of them only make sense if you have a subscription anyway.
member of the Toon Software Collective
michel30
Member
Member
Posts: 286
Joined: Fri Aug 25, 2017 4:42 pm

Re: Spontaneous reboots

Post by michel30 »

You can also check with top if x11vnc is start up one time.
I had the same problem in the beginning when I root the TOON x11vnc was starting up many times and that was my issue.
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Spontaneous reboots

Post by TheHogNL »

Could we try compiling a new kernel with swap support enabled and then use a small (but preferably fast) usb stick as a small swap disk? I guess this will offload some less used memory pages to swap to free up some memory.
Member of the Toon Software Collective
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Spontaneous reboots

Post by marcelr »

Quite possible.
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Spontaneous reboots

Post by TheHogNL »

I would really try to get into building it myself. I has been a while when I last compiled my own kernels but for sure I'm not into the embedded stuff. Really need to crash course it first :-)
If anyone can provide a few hints and where to start, I would be on my way.
Member of the Toon Software Collective
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Spontaneous reboots

Post by Rudolf »

I've been logging the memory consumption of qt-gui but it looks allright. Had a few reboots in the past week though, it last approx. 30 hours.

Here's the result of 'top', followed by shift + M
to sort by memory consumption.

Code: Select all

Mem: 122320K used, 3876K free, 8384K shrd, 0K buff, 30284K cached
CPU:  39% usr  15% sys   0% nic  44% idle   0% io   0% irq   0% sirq
Load average: 0.56 0.45 0.36 3/141 6403
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  710     1 root     S     333m 270%   1% {hcb_comm} HCBv2 hcb_comm       [OK] [20 clients] [S1|N1]
  708     1 root     S     153m 125%   0% /HCBv2/sbin/qt-gui -platform linuxfb -plugin Tslib
  731     1 root     S    38340  30%   3% {hdrv_zwave} HCBv2 hdrv_zwave     [OK] [EBD931A9 SIS 3.67] [S1|N1]
  721     1 root     S    22316  18%   0% {hcb_config} HCBv2 hcb_config     [OK] [S1|N1]
  725     1 root     S    22276  18%   0% {hdrv_hue} HCBv2 hdrv_hue       [OK] [S1|N1]
  716     1 root     S    21396  17%   0% {happ_thermstat} HCBv2 happ_thermstat [OK] [S20|N1]
  713     1 root     S    21064  17%   0% {happ_pwrusage} HCBv2 happ_pwrusage  [OK] [S30|N1]
  723     1 root     R    20880  17%  46% {hcb_rrd} HCBv2 hcb_rrd        [OK] [S1|N1]
  712     1 root     S    20316  16%   0% {happ_kpi} HCBv2 happ_kpi       [OK] [S1|N1]
  724     1 root     S    20016  16%   0% {hcb_watchdog} HCBv2 hcb_watchdog   [OK] [S20|N1]
  709     1 root     S    19524  15%   0% {hcb_netcon} HCBv2 hcb_netcon     [OK] [S1|N1]
  730     1 root     S    19296  15%   0% {hdrv_p1} HCBv2 hdrv_p1        [OK] [C:OK E:OP G:OP ] [S1|N1]
  715     1 root     S    19208  15%   0% {happ_smartplug} HCBv2 happ_smartplug [OK] [S1|N1]
  718     1 root     S    19136  15%   0% {happ_usermsg} HCBv2 happ_usermsg   [OK] [S1|N1]
  714     1 root     S    19052  15%   0% {happ_scsync} HCBv2 happ_scsync    [OK] [S1|N1]
  719     1 root     S    13784  11%   0% {happ_weather} HCBv2 happ_weather   [OK] [S20|N1]
  720     1 root     S    13552  11%   0% {hcb_bxtproxy} HCBv2 hcb_bxtproxy   [OK] [S:0:6:2020 R:0:0 Q:0 (PRD)] [S1|N1]
  735     1 root     S    12016  10%   0% /HCBv2/bin/lighttpd -D -m /HCBv2/lib/lighttpd -f /HCBv2/etc/lighttpd/lighttpd.conf
  711     1 root     S    11516   9%   0% {happ_eventmgr} HCBv2 happ_eventmgr  [OK] [S1|N1]
  722     1 root     S    10836   9%   0% {hcb_log} HCBv2 hcb_log        [OK] [S1|N1]
  544     1 root     S     4064   3%   0% /usr/sbin/wpa_supplicant -B -P /var/run/wpa_supplicant.wlan0.pid -i wlan0 -c /etc/network/wpa_supplicant.wlan0.conf -D wext
 1143     1 root     S     3472   3%   0% /usr/sbin/nmbd -s /etc/samba/smb.conf
 6395  5256 root     R     3280   3%   1% top
 5256  5242 root     S     3280   3%   0% -sh
  648     1 root     S     3168   3%   0% /sbin/syslogd -n -C64 -m 20
  613     1 root     S     3104   2%   0% udhcpc -b -R -i eth0 -A 5 -p /var/run/udhcpc.eth0.pid -H eneco-001-037311
  567     1 root     S     3104   2%   0% udhcpc -R -n -p /var/run/udhcpc.wlan0.pid -i wlan0 -x hostname:eneco-001-037311 -t 2
  651     1 root     S     3104   2%   0% /sbin/klogd -n
 1109     1 root     S     2616   2%   0% /usr/sbin/chronyd -r
 5242   643 root     S     2336   2%   0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
  643     1 root     S     2164   2%   0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
  653     1 root     S     1772   1%   0% /usr/sbin/cron
    1     0 root     S     1600   1%   0% init [5]
    3     2 root     SW       0   0%   0% [ksoftirqd/0]
  518     2 root     SW       0   0%   0% [flush-ubifs_0_0]
    5     2 root     SW       0   0%   0% [kworker/u:0]
    6     2 root     SW       0   0%   0% [watchdog/0]
  358     2 root     SW       0   0%   0% [ubifs_bgt0_0]
  309     2 root     SW       0   0%   0% [ubi_bgt0d]
  541     2 root     SW       0   0%   0% [RTKTHREAD]
 3926     2 root     SW       0   0%   0% [kworker/0:0]
  244     2 root     SW       0   0%   0% [kswapd0]
 5858     2 root     SW       0   0%   0% [kworker/0:1]
  136     2 root     SW       0   0%   0% [khubd]
 5434     2 root     SW       0   0%   0% [kworker/0:3]
    2     0 root     SW       0   0%   0% [kthreadd]
    7     2 root     SW<      0   0%   0% [khelper]
    8     2 root     SW       0   0%   0% [kworker/u:1]
   11     2 root     SW<      0   0%   0% [pm]
  119     2 root     SW       0   0%   0% [sync_supers]
  121     2 root     SW       0   0%   0% [bdi-default]
  123     2 root     SW<      0   0%   0% [kblockd]
  235     2 root     SW<      0   0%   0% [rpciod]
  243     2 root     SW       0   0%   0% [khungtaskd]
  245     2 root     SW       0   0%   0% [fsnotify_mark]
  246     2 root     SW<      0   0%   0% [nfsiod]
  247     2 root     SW<      0   0%   0% [crypto]
  281     2 root     SW       0   0%   0% [mtdblock0]
  286     2 root     SW       0   0%   0% [mtdblock1]

Code: Select all

eneco-001-037311:~# uptime
 07:11:29 up 1 day,  9:08,  load average: 0.34, 0.38, 0.35

Code: Select all

eneco-001-037311:~# cat reboot.log
Toon was restarted.
Mon Feb  5 16:45:33 UTC 2018
Toon was restarted.
Tue Feb  6 15:26:07 UTC 2018
Toon was restarted.
Wed Feb  7 22:03:10 UTC 2018
No reboot after wednesday yet, that's quite peculiar.

I disabled all custom apps, to see if the problem goes away. Will report back later.
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Spontaneous reboots

Post by Rudolf »

I thought the issue was gone, but I was proven wrong.

After re-enabling all custom apps one by one the issue didn't reappear, until I configured HASS to start polling the boiler-status again using this component:
https://github.com/cyberjunky/home-assi ... rstatus.py

Code: Select all

- platform: toon_boilerstatus
  host: toon
  port: 10080
  scan_interval: 10
  resources:
    - boilersetpoint
    - boilerintemp
    - boilerouttemp
    - boilerpressure
    - boilermodulationlevel
    - roomtemp
    - roomtempsetpoint
Maybe the http-requests with this small interval introduce a problematic memory-leak?
Does it make sense to monitor lighttpd's memory consumption over time? Any other suggestions?
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Spontaneous reboots

Post by TheHogNL »

Reboots are often caused by a lack of memory. Hammering the lighttpd could indeed cause this so following the memory usage of that process and the free memory of the Toon would be a good method to find out of that is indeed the case. Good luck!
Member of the Toon Software Collective
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Spontaneous reboots

Post by Rudolf »

It seems the folder '/var/volatile/tmp/www-cache/boilerstatus' is slowly growing in size.

The system is up for 20 hours, and this folder contains over 1200 files. Only 4,7MiB, but this might continue to grow and reach critical values within days.
I'll try to monitor this folder for a bit longer.

Is there a more elegant solution then flushing this folder from a cronjob?
https://bash.cyberciti.biz/file-managem ... he-script/

I don't see an option for lighttpd to handle this.

Am I the only one experiencing this issue with the boilerstatus-app? Is it because I'm polling the server too often?
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Spontaneous reboots

Post by TheHogNL »

Looks like it is being caused by mod_compress as a module from lighttpd. Also check https://redmine.lighttpd.net/projects/1 ... odcompress

Please check /qmf/etc/lighttpd/lighttpd.conf and change cache_dir to "" (empty). According to the documentation this will cause the lighttpd not to cache anymore
Last edited by TheHogNL on Sun May 06, 2018 7:04 am, edited 1 time in total.
Member of the Toon Software Collective
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Spontaneous reboots

Post by Rudolf »

Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.


So for now I implemented this crontab:

Code: Select all

*/5 * * * * find /var/volatile/tmp/www-cache/boilerstatus -type f -mmin +5 -delete
This runs every 5 minutes and deletes the cache that's over 5 minutes old.
Fietspomp
Starting Member
Starting Member
Posts: 27
Joined: Tue Jan 30, 2018 12:09 pm

Re: Spontaneous reboots

Post by Fietspomp »

Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now :D

But after this sensor don't work anymore in HASS :cry:

Code: Select all

2018-05-06 10:34:27: (../../../src/log.c.168) server started 
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir  No such file or directory 
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down. 
from file on Toon /var/volatile/tmp/lighttpd_error.log

So Ill try the cron instead.
Last edited by Fietspomp on Sun May 06, 2018 10:41 am, edited 1 time in total.
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Spontaneous reboots

Post by TheHogNL »

Rudolf wrote:Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.
Gzipping the content of an html page is a normal job for a webserver. The only way for lighttpd to do this is to compress it just before sending and caching the result so that a next hit on the same page will not require a zip process again. However this strategy will only work for static pages. Not dynamic like the boilerstatus. It will create a boilerstatus zip cache file everytime it is changed.

Gzipping (and therefore the cache) can be turned off without any negative effect. I think your home network has enough bandwith available :)
Member of the Toon Software Collective
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Spontaneous reboots

Post by Rudolf »

Fietspomp wrote:Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now :D

But after this sensor don't work anymore in HASS :cry:

Code: Select all

2018-05-06 10:34:27: (../../../src/log.c.168) server started 
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir  No such file or directory 
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down. 
from file on Toon /var/volatile/tmp/lighttpd_error.log

So Ill try the cron instead.
Glad it helped you, I started to think I was the only one running in to this issue.

lighttpd.conf loads all modules with the following directive:
include_shell "/HCBv2/etc/lighttpd/genModules"
and this loads all modules from the <lighttpd>/modules/ folder; maybe you want to try removing mod_compress from that folder.

But, the cronjob is working great for me.
Post Reply

Return to “Toon issues and support (Nederlands toegestaan)”