Spontaneous reboots

If your (rooted) Toon doesn't do what it's supposed to, post here. Dutch allowed / Posten in 't Nederlands toegestaan.

Moderator: marcelr

Spontaneous reboots

Postby Rudolf » Sun Feb 04, 2018 7:17 pm

I rooted my Toon fairly recently and I haven't put it to use yet, but I've been noticing some strange behaviour lately; spontaneous restarts.
I noticed because I have to start x11vnc manually and sometimes I can't login because the device has been rebooted. Manually restarting x11vnc is not an issue, but I'd like to solve the underlying issue before I start using Toon as my thermostat.

I just added a small script that logs every reboot with a timestamp in a file using @reboot in cron.
I think it must be a memory issue of one of the apps I installed... But those are nothing out of the ordinary; toonstore, boilerstatus, trash, traffic and buienradar.

So should I log the results of 'free' in a file using cron?
Or can I see the reason for the reboot somewhere on the filesystem, in a file?

Any other stats I should look into? Please advise.
Rudolf
Member
Member
 
Posts: 94
Joined: December 2017

Re: Spontaneous reboots

Postby Toonz » Sun Feb 04, 2018 7:54 pm

Typically Toon restarts itself when it runs out of memory.
What you could do is disable some apps you don't really need and see if that resolves the issues.
Personally I don't have all Eneco apps running, many of them only make sense if you have a subscription anyway.
Toonz
Advanced Member
Advanced Member
 
Posts: 716
Joined: December 2016

Re: Spontaneous reboots

Postby michel30 » Sun Feb 04, 2018 9:21 pm

You can also check with top if x11vnc is start up one time.
I had the same problem in the beginning when I root the TOON x11vnc was starting up many times and that was my issue.
michel30
Member
Member
 
Posts: 168
Joined: August 2017

Re: Spontaneous reboots

Postby TheHogNL » Mon Feb 05, 2018 7:55 am

Could we try compiling a new kernel with swap support enabled and then use a small (but preferably fast) usb stick as a small swap disk? I guess this will offload some less used memory pages to swap to free up some memory.
TheHogNL
Member
Member
 
Posts: 441
Joined: August 2017

Re: Spontaneous reboots

Postby marcelr » Mon Feb 05, 2018 8:44 am

Quite possible.
marcelr
Advanced Member
Advanced Member
 
Posts: 925
Joined: May 2012
Location: Ehv

Re: Spontaneous reboots

Postby TheHogNL » Mon Feb 05, 2018 7:09 pm

I would really try to get into building it myself. I has been a while when I last compiled my own kernels but for sure I'm not into the embedded stuff. Really need to crash course it first :-)
If anyone can provide a few hints and where to start, I would be on my way.
TheHogNL
Member
Member
 
Posts: 441
Joined: August 2017

Re: Spontaneous reboots

Postby Rudolf » Fri Feb 09, 2018 8:41 am

I've been logging the memory consumption of qt-gui but it looks allright. Had a few reboots in the past week though, it last approx. 30 hours.

Here's the result of 'top', followed by shift + M
to sort by memory consumption.

Code: Select all
Mem: 122320K used, 3876K free, 8384K shrd, 0K buff, 30284K cached
CPU:  39% usr  15% sys   0% nic  44% idle   0% io   0% irq   0% sirq
Load average: 0.56 0.45 0.36 3/141 6403
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  710     1 root     S     333m 270%   1% {hcb_comm} HCBv2 hcb_comm       [OK] [20 clients] [S1|N1]
  708     1 root     S     153m 125%   0% /HCBv2/sbin/qt-gui -platform linuxfb -plugin Tslib
  731     1 root     S    38340  30%   3% {hdrv_zwave} HCBv2 hdrv_zwave     [OK] [EBD931A9 SIS 3.67] [S1|N1]
  721     1 root     S    22316  18%   0% {hcb_config} HCBv2 hcb_config     [OK] [S1|N1]
  725     1 root     S    22276  18%   0% {hdrv_hue} HCBv2 hdrv_hue       [OK] [S1|N1]
  716     1 root     S    21396  17%   0% {happ_thermstat} HCBv2 happ_thermstat [OK] [S20|N1]
  713     1 root     S    21064  17%   0% {happ_pwrusage} HCBv2 happ_pwrusage  [OK] [S30|N1]
  723     1 root     R    20880  17%  46% {hcb_rrd} HCBv2 hcb_rrd        [OK] [S1|N1]
  712     1 root     S    20316  16%   0% {happ_kpi} HCBv2 happ_kpi       [OK] [S1|N1]
  724     1 root     S    20016  16%   0% {hcb_watchdog} HCBv2 hcb_watchdog   [OK] [S20|N1]
  709     1 root     S    19524  15%   0% {hcb_netcon} HCBv2 hcb_netcon     [OK] [S1|N1]
  730     1 root     S    19296  15%   0% {hdrv_p1} HCBv2 hdrv_p1        [OK] [C:OK E:OP G:OP ] [S1|N1]
  715     1 root     S    19208  15%   0% {happ_smartplug} HCBv2 happ_smartplug [OK] [S1|N1]
  718     1 root     S    19136  15%   0% {happ_usermsg} HCBv2 happ_usermsg   [OK] [S1|N1]
  714     1 root     S    19052  15%   0% {happ_scsync} HCBv2 happ_scsync    [OK] [S1|N1]
  719     1 root     S    13784  11%   0% {happ_weather} HCBv2 happ_weather   [OK] [S20|N1]
  720     1 root     S    13552  11%   0% {hcb_bxtproxy} HCBv2 hcb_bxtproxy   [OK] [S:0:6:2020 R:0:0 Q:0 (PRD)] [S1|N1]
  735     1 root     S    12016  10%   0% /HCBv2/bin/lighttpd -D -m /HCBv2/lib/lighttpd -f /HCBv2/etc/lighttpd/lighttpd.conf
  711     1 root     S    11516   9%   0% {happ_eventmgr} HCBv2 happ_eventmgr  [OK] [S1|N1]
  722     1 root     S    10836   9%   0% {hcb_log} HCBv2 hcb_log        [OK] [S1|N1]
  544     1 root     S     4064   3%   0% /usr/sbin/wpa_supplicant -B -P /var/run/wpa_supplicant.wlan0.pid -i wlan0 -c /etc/network/wpa_supplicant.wlan0.conf -D wext
1143     1 root     S     3472   3%   0% /usr/sbin/nmbd -s /etc/samba/smb.conf
6395  5256 root     R     3280   3%   1% top
5256  5242 root     S     3280   3%   0% -sh
  648     1 root     S     3168   3%   0% /sbin/syslogd -n -C64 -m 20
  613     1 root     S     3104   2%   0% udhcpc -b -R -i eth0 -A 5 -p /var/run/udhcpc.eth0.pid -H eneco-001-037311
  567     1 root     S     3104   2%   0% udhcpc -R -n -p /var/run/udhcpc.wlan0.pid -i wlan0 -x hostname:eneco-001-037311 -t 2
  651     1 root     S     3104   2%   0% /sbin/klogd -n
1109     1 root     S     2616   2%   0% /usr/sbin/chronyd -r
5242   643 root     S     2336   2%   0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
  643     1 root     S     2164   2%   0% /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22
  653     1 root     S     1772   1%   0% /usr/sbin/cron
    1     0 root     S     1600   1%   0% init [5]
    3     2 root     SW       0   0%   0% [ksoftirqd/0]
  518     2 root     SW       0   0%   0% [flush-ubifs_0_0]
    5     2 root     SW       0   0%   0% [kworker/u:0]
    6     2 root     SW       0   0%   0% [watchdog/0]
  358     2 root     SW       0   0%   0% [ubifs_bgt0_0]
  309     2 root     SW       0   0%   0% [ubi_bgt0d]
  541     2 root     SW       0   0%   0% [RTKTHREAD]
3926     2 root     SW       0   0%   0% [kworker/0:0]
  244     2 root     SW       0   0%   0% [kswapd0]
5858     2 root     SW       0   0%   0% [kworker/0:1]
  136     2 root     SW       0   0%   0% [khubd]
5434     2 root     SW       0   0%   0% [kworker/0:3]
    2     0 root     SW       0   0%   0% [kthreadd]
    7     2 root     SW<      0   0%   0% [khelper]
    8     2 root     SW       0   0%   0% [kworker/u:1]
   11     2 root     SW<      0   0%   0% [pm]
  119     2 root     SW       0   0%   0% [sync_supers]
  121     2 root     SW       0   0%   0% [bdi-default]
  123     2 root     SW<      0   0%   0% [kblockd]
  235     2 root     SW<      0   0%   0% [rpciod]
  243     2 root     SW       0   0%   0% [khungtaskd]
  245     2 root     SW       0   0%   0% [fsnotify_mark]
  246     2 root     SW<      0   0%   0% [nfsiod]
  247     2 root     SW<      0   0%   0% [crypto]
  281     2 root     SW       0   0%   0% [mtdblock0]
  286     2 root     SW       0   0%   0% [mtdblock1]



Code: Select all
eneco-001-037311:~# uptime
07:11:29 up 1 day,  9:08,  load average: 0.34, 0.38, 0.35


Code: Select all
eneco-001-037311:~# cat reboot.log
Toon was restarted.
Mon Feb  5 16:45:33 UTC 2018
Toon was restarted.
Tue Feb  6 15:26:07 UTC 2018
Toon was restarted.
Wed Feb  7 22:03:10 UTC 2018


No reboot after wednesday yet, that's quite peculiar.

I disabled all custom apps, to see if the problem goes away. Will report back later.
Rudolf
Member
Member
 
Posts: 94
Joined: December 2017

Re: Spontaneous reboots

Postby Rudolf » Sun Apr 29, 2018 11:20 am

I thought the issue was gone, but I was proven wrong.

After re-enabling all custom apps one by one the issue didn't reappear, until I configured HASS to start polling the boiler-status again using this component:
https://github.com/cyberjunky/home-assi ... rstatus.py
Code: Select all
- platform: toon_boilerstatus
  host: toon
  port: 10080
  scan_interval: 10
  resources:
    - boilersetpoint
    - boilerintemp
    - boilerouttemp
    - boilerpressure
    - boilermodulationlevel
    - roomtemp
    - roomtempsetpoint


Maybe the http-requests with this small interval introduce a problematic memory-leak?
Does it make sense to monitor lighttpd's memory consumption over time? Any other suggestions?
Rudolf
Member
Member
 
Posts: 94
Joined: December 2017

Re: Spontaneous reboots

Postby TheHogNL » Sun Apr 29, 2018 9:19 pm

Reboots are often caused by a lack of memory. Hammering the lighttpd could indeed cause this so following the memory usage of that process and the free memory of the Toon would be a good method to find out of that is indeed the case. Good luck!
TheHogNL
Member
Member
 
Posts: 441
Joined: August 2017

Re: Spontaneous reboots

Postby Rudolf » Mon Apr 30, 2018 8:53 am

It seems the folder '/var/volatile/tmp/www-cache/boilerstatus' is slowly growing in size.

The system is up for 20 hours, and this folder contains over 1200 files. Only 4,7MiB, but this might continue to grow and reach critical values within days.
I'll try to monitor this folder for a bit longer.

Is there a more elegant solution then flushing this folder from a cronjob?
https://bash.cyberciti.biz/file-managem ... he-script/

I don't see an option for lighttpd to handle this.

Am I the only one experiencing this issue with the boilerstatus-app? Is it because I'm polling the server too often?
Rudolf
Member
Member
 
Posts: 94
Joined: December 2017

Re: Spontaneous reboots

Postby TheHogNL » Mon Apr 30, 2018 9:20 am

Looks like it is being caused by mod_compress as a module from lighttpd. Also check https://redmine.lighttpd.net/projects/1 ... odcompress

Please check /qmf/etc/lighttpd/lighttpd.conf and change cache_dir to "" (empty). According to the documentation this will cause the lighttpd not to cache anymore
Last edited by TheHogNL on Sun May 06, 2018 6:04 am, edited 1 time in total.
TheHogNL
Member
Member
 
Posts: 441
Joined: August 2017

Re: Spontaneous reboots

Postby Rudolf » Mon Apr 30, 2018 10:30 am

Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.


So for now I implemented this crontab:
Code: Select all
*/5 * * * * find /var/volatile/tmp/www-cache/boilerstatus -type f -mmin +5 -delete


This runs every 5 minutes and deletes the cache that's over 5 minutes old.
Rudolf
Member
Member
 
Posts: 94
Joined: December 2017

Re: Spontaneous reboots

Postby Fietspomp » Sun May 06, 2018 4:32 am

Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now :D

But after this sensor don't work anymore in HASS :cry:

Code: Select all
2018-05-06 10:34:27: (../../../src/log.c.168) server started
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir  No such file or directory
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down.


from file on Toon /var/volatile/tmp/lighttpd_error.log

So Ill try the cron instead.
Last edited by Fietspomp on Sun May 06, 2018 9:41 am, edited 1 time in total.
Fietspomp
Starting Member
Starting Member
 
Posts: 15
Joined: January 2018

Re: Spontaneous reboots

Postby TheHogNL » Sun May 06, 2018 6:09 am

Rudolf wrote:Thanks, I missed that remark in the lighttpd docs.
(Although I think that the cache might serve a purpose and - edit: Caching seems hardly useful, but compressing the cache even less...) I couldn't find a way to restart lighttpd to test this new config.


Gzipping the content of an html page is a normal job for a webserver. The only way for lighttpd to do this is to compress it just before sending and caching the result so that a next hit on the same page will not require a zip process again. However this strategy will only work for static pages. Not dynamic like the boilerstatus. It will create a boilerstatus zip cache file everytime it is changed.

Gzipping (and therefore the cache) can be turned off without any negative effect. I think your home network has enough bandwith available :)
TheHogNL
Member
Member
 
Posts: 441
Joined: August 2017

Re: Spontaneous reboots

Postby Rudolf » Mon May 07, 2018 7:41 am

Fietspomp wrote:Thanks for this solution!
Was already wondering why my Toon was quite sluggish after a while and rebooted itself several times.
Just edited the /qmf/etc/lighttpd/lighttpd.conf and rebooted, seems to stop caching now :D

But after this sensor don't work anymore in HASS :cry:

Code: Select all
2018-05-06 10:34:27: (../../../src/log.c.168) server started
2018-05-06 10:34:27: (../../../src/mod_compress.c.229) can't stat compress.cache-dir  No such file or directory
2018-05-06 10:34:27: (../../../src/server.c.1009) Configuration of plugins failed. Going down.


from file on Toon /var/volatile/tmp/lighttpd_error.log

So Ill try the cron instead.

Glad it helped you, I started to think I was the only one running in to this issue.

lighttpd.conf loads all modules with the following directive:
include_shell "/HCBv2/etc/lighttpd/genModules"
and this loads all modules from the <lighttpd>/modules/ folder; maybe you want to try removing mod_compress from that folder.

But, the cronjob is working great for me.
Rudolf
Member
Member
 
Posts: 94
Joined: December 2017


Return to Toon issues and support (Nederlands toegestaan)

Who is online

Users browsing this forum: No registered users and 1 guest