Toon firmware: issues (and solutions)

[marcelr]

Please post the first section of your config_happ_scsync.xml. Should be in there.

[flamingbob]

Thanks! Would be interesting to know. Hope when 4.9 arrives we can get the boiler back. Noticed every time scs connects it goes back to this so its quite annoying.

This should be it:

Code: Select all

 <Config packageRevision="246">
  <sysConfig>
   <locale>nl_NL</locale>
   <timezone>Europe/Amsterdam</timezone>
   <currency>EUR</currency></sysConfig>
    <device>
     <package>happ_scsync</package>
     <uuid>neco-001-XXXXX:happ_scsync_XXXXXXX</uuid>
     <type>agreementDetail</type>
     <internalAddress>agreementDetail</internalAddress>
     <visibility>0</visibility><StartDate>1372377600</StartDate><EndDate>-1</EndDate><Status>IN_SUPPLY</Status><ProductVariant>Toon</ProductVariant><activated>1</activated><wizardDone>1</wizardDone><SoftwareUpdates>1</SoftwareUpdates><BoilerManagement activated="0">0</BoilerManagement><ElectricityDisplay>1</ElectricityDisplay><GasDisplay>1</GasDisplay><HeatDisplay>0</HeatDisplay><SolarDisplay>0</SolarDisplay><SolarActivated>0</SolarActivated><OtherProviderElec>1</OtherProviderElec><OtherProviderGas>0</OtherProviderGas><SME>0</SME><HeatWinner>0</HeatWinner><ContentApps>1</ContentApps><TelmiEnabled>0</TelmiEnabled><mobileAccess>1</mobileAccess><supportEnabled>0</supportEnabled><supportEnabledStart>0</supportEnabledStart><researchEnabled>0</researchEnabled><doSolarWhatsnew>0</doSolarWhatsnew><latestWhatsnewVersion>qt-gui - 1.6755-ene-release-ene-3.7</latestWhatsnewVersion><statusUsageFirstUse>0</statusUsageFirstUse><scStatusFlags>4</scStatusFlags><commissionState>0</commissionState></device>

[Timeless]

Hi,

I've been using the DomoticzBoard plugin for some time now and I noticed this evening that Toon was rebooting (I thought) but then I saw in the /var/log/messages file that it seems to have killed the qt-gui processes to free up some memory:

Code: Select all

Nov  4 01:26:05 toon user.warn kernel: hcb_netcon invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
Nov  4 01:26:06 toon user.warn kernel: [<c00293d0>] (unwind_backtrace+0x0/0xf0) from [<c0071774>] (dump_header.clone.14+0x70/0x190)
Nov  4 01:26:06 toon user.warn kernel: [<c0071774>] (dump_header.clone.14+0x70/0x190) from [<c0071a90>] (oom_kill_process.clone.16+0x7c/0x1d8)
Nov  4 01:26:06 toon user.warn kernel: [<c0071a90>] (oom_kill_process.clone.16+0x7c/0x1d8) from [<c0071ec4>] (out_of_memory+0x148/0x30c)
Nov  4 01:26:06 toon user.warn kernel: [<c0071ec4>] (out_of_memory+0x148/0x30c) from [<c00755f8>] (__alloc_pages_nodemask+0x540/0x56c)
Nov  4 01:26:06 toon user.warn kernel: [<c00755f8>] (__alloc_pages_nodemask+0x540/0x56c) from [<c0075634>] (__get_free_pages+0x10/0x3c)
Nov  4 01:26:06 toon user.warn kernel: [<c0075634>] (__get_free_pages+0x10/0x3c) from [<c002b3d8>] (get_pgd_slow+0x14/0xe0)
Nov  4 01:26:06 toon user.warn kernel: [<c002b3d8>] (get_pgd_slow+0x14/0xe0) from [<c00347cc>] (mm_init.clone.57+0x90/0xd4)
Nov  4 01:26:06 toon user.warn kernel: [<c00347cc>] (mm_init.clone.57+0x90/0xd4) from [<c0034a4c>] (dup_mm+0x6c/0x4f8)
Nov  4 01:26:06 toon user.warn kernel: [<c0034a4c>] (dup_mm+0x6c/0x4f8) from [<c00354fc>] (copy_process+0x5ec/0xc6c)
Nov  4 01:26:06 toon user.warn kernel: [<c00354fc>] (copy_process+0x5ec/0xc6c) from [<c0035c00>] (do_fork+0x60/0x364)
Nov  4 01:26:06 toon user.warn kernel: [<c0035c00>] (do_fork+0x60/0x364) from [<c0024ea0>] (ret_fast_syscall+0x0/0x2c)
Nov  4 01:26:06 toon user.warn kernel: Mem-info:
Nov  4 01:26:06 toon user.warn kernel: Normal per-cpu:
Nov  4 01:26:06 toon user.warn kernel: CPU    0: hi:   42, btch:   7 usd:  18
Nov  4 01:26:06 toon user.warn kernel: active_anon:13388 inactive_anon:13445 isolated_anon:0
Nov  4 01:26:06 toon user.warn kernel:  active_file:0 inactive_file:14 isolated_file:28
Nov  4 01:26:06 toon user.warn kernel:  unevictable:12 dirty:23 writeback:1 unstable:0
Nov  4 01:26:06 toon user.warn kernel:  free:1397 slab_reclaimable:168 slab_unreclaimable:640
Nov  4 01:26:06 toon user.warn kernel:  mapped:17 shmem:91 pagetables:317 bounce:0
Nov  4 01:26:06 toon user.warn kernel: Normal free:5588kB min:1440kB low:1800kB high:2160kB active_anon:53552kB inactive_anon:53780kB active_file:0kB inactive_file:56kB unevictable:48kB isolated(anon):0kB isolated(file):112kB present:130048kB mlocked:0kB dirty:92kB
Nov  4 01:26:06 toon user.warn kernel: lowmem_reserve[]: 0 0
Nov  4 01:26:06 toon user.warn kernel: Normal: 1091*4kB 111*8kB 17*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5588kB
Nov  4 01:26:06 toon user.warn kernel: 149 total pagecache pages
Nov  4 01:26:06 toon user.warn kernel: 32768 pages of RAM
Nov  4 01:26:06 toon user.warn kernel: 1644 free pages
Nov  4 01:26:06 toon user.warn kernel: 1356 reserved pages
Nov  4 01:26:06 toon user.warn kernel: 808 slab pages
Nov  4 01:26:06 toon user.warn kernel: 66 pages shared
Nov  4 01:26:06 toon user.warn kernel: 0 pages swap cached
Nov  4 01:26:06 toon user.info kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Nov  4 01:26:06 toon user.info kernel: [  516]     0   516      718       18   0       0             0 udhcpc
Nov  4 01:26:06 toon user.info kernel: [  612]     0   612      541       23   0       0             0 dropbear
Nov  4 01:26:06 toon user.info kernel: [  615]     0   615      718       15   0       0             0 syslogd
Nov  4 01:26:06 toon user.info kernel: [  617]     0   617      718       18   0       0             0 klogd
Nov  4 01:26:06 toon user.info kernel: [  657]     0   657    44983    23973   0       0             0 qt-gui
Nov  4 01:26:06 toon user.info kernel: [  658]     0   658      719       17   0       0             0 getty
Nov  4 01:26:06 toon user.info kernel: [  659]     0   659     5163       90   0       0             0 hcb_netcon
Nov  4 01:26:06 toon user.info kernel: [  660]     0   660    89763      419   0       0             0 hcb_comm
Nov  4 01:26:06 toon user.info kernel: [  661]     0   661     3239       88   0       0             0 happ_eventmgr
Nov  4 01:26:06 toon user.info kernel: [  662]     0   662     5415      137   0       0             0 happ_kpi
Nov  4 01:26:06 toon user.info kernel: [  663]     0   663     5461      112   0       0             0 happ_pwrusage
Nov  4 01:26:06 toon user.info kernel: [  664]     0   664     5127       89   0       0             0 happ_scsync
Nov  4 01:26:06 toon user.info kernel: [  665]     0   665     5159      100   0       0             0 happ_smartplug
Nov  4 01:26:06 toon user.info kernel: [  666]     0   666     5640      157   0       0             0 happ_thermstat
Nov  4 01:26:06 toon user.info kernel: [  667]     0   667     5111       85   0       0             0 happ_usermsg
Nov  4 01:26:06 toon user.info kernel: [  668]     0   668     3609       96   0       0             0 happ_weather
Nov  4 01:26:06 toon user.info kernel: [  669]     0   669     3240      129   0       0             0 hcb_bxtproxy
Nov  4 01:26:06 toon user.info kernel: [  670]     0   670     5493      163   0       0             0 hcb_config
Nov  4 01:26:06 toon user.info kernel: [  671]     0   671     3070       98   0       0             0 hcb_log
Nov  4 01:26:06 toon user.info kernel: [  677]     0   677     5251      126   0       0             0 hcb_rrd
Nov  4 01:26:06 toon user.info kernel: [  681]     0   681     3505      136   0       0             0 hcb_upnp
Nov  4 01:26:06 toon user.info kernel: [  682]     0   682     5366       96   0       0             0 hcb_watchdog
Nov  4 01:26:06 toon user.info kernel: [  683]     0   683     5680       98   0       0             0 hdrv_hue
Nov  4 01:26:06 toon user.info kernel: [  684]     0   684     5408      100   0       0             0 hdrv_p1
Nov  4 01:26:06 toon user.info kernel: [  694]     0   694     9671      128   0       0             0 hdrv_zwave
Nov  4 01:26:06 toon user.info kernel: [  695]     0   695     3378      157   0       0             0 lighttpd
Nov  4 01:26:06 toon user.info kernel: [ 1028]     0  1028      595       28   0       0             0 chronyd
Nov  4 01:26:06 toon user.info kernel: [ 1041]     0  1041      868       57   0       0             0 nmbd
Nov  4 01:26:06 toon user.err kernel: Out of memory: Kill process 657 (qt-gui) score 733 or sacrifice child
Nov  4 01:26:06 toon user.err kernel: Killed process 657 (qt-gui) total-vm:179932kB, anon-rss:95892kB, file-rss:0kB

I don't know if this happened before I started using DomoticzBoard, or if my "customized" version of DomoticzBoard is to blame. Anyone seen this out of memory exception before?
here can my customized version of DomoticzBoard be found, maybe I have a memory leak somewhere? Would be nice if some one could take a look. I can't find any.

[Toonz]

This is standard behaviour of Toon. It restarts the gui automatically when running out of memory.
If it happens a lot you could try to disable/remove the apps from Globals.qml which you don't really need.

[Artificial]

Hi All,

I have updated my rooted Toon to 4.8.25 today:). Got most stuff to work (thank you for all the tutorials).
I have also added the extra lines to InternetSettingsApp.qml to get rid of the activation messages, but this didn't completely work for me. I still get a pop-up message on the homescreen saying there is no connection to the service centre. When i press this message it opens the internet tab which has all green:S . Does anybody know how to fix this? thanks in advance!

[michel30]

Hello,

Did you also check here: viewtopic.php?f=87&t=11810 Notifications bar troppic.

Regards,
Michel

[Artificial]

Thanks! that's it. Deactivated all notifications

[Timeless]

This is standard behaviour of Toon. It restarts the gui automatically when running out of memory.
If it happens a lot you could try to disable/remove the apps from Globals.qml which you don't really need.

Yeah this is also standard Linux behavior I think.
But is it also normal for Toon that it runs out of memory after +/-9 days? I checked my ram usage after the kill (which was 20MB free) and today it only has 10MB free. So I guess it's going to run out of memory pretty soon.

I also checked TOP:

Code: Select all

Mem: 114992K used, 10656K free, 0K shrd, 0K buff, 23844K cached
CPU:   2% usr   4% sys   0% nic  92% idle   0% io   0% irq   0% sirq
Load average: 1.32 0.68 0.46 1/142 7361
  PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND
  694     1 root     S    38684  31%   2% HCBv2 hdrv_zwave     [OK] [---- SIS Z-Wave 3.67] [S1|N1]
  666     1 root     S    22560  18%   1% HCBv2 happ_thermstat [OK] [ON/OFF] [S20|N1]
 7317  7152 root     R     3048   2%   1% top
  659     1 root     S    20652  16%   1% HCBv2 hcb_netcon     [OK] [S1|N1]
    3     2 root     SW       0   0%   0% [ksoftirqd/0]
  660     1 root     S     350m 286%   0% HCBv2 hcb_comm       [OK] [21 clients] [S1|N1]
  681     1 root     S    14020  11%   0% HCBv2 hcb_upnp       [OK] [S20|N1]
  669     1 root     S    12960  10%   0% HCBv2 hcb_bxtproxy   [OK] [x:xx:xxx R:0:0 Q:0 (PRD)] [S1|N1]
27720     1 root     S     146m 119%   0% /HCBv2/sbin/qt-gui -platform linuxfb -plugin Tslib
  683     1 root     S    22720  18%   0% HCBv2 hdrv_hue       [OK] [S1|N1]
  670     1 root     S    21972  17%   0% HCBv2 hcb_config     [OK] [S1|N1]
  663     1 root     S    21844  17%   0% HCBv2 happ_pwrusage  [OK] [S30|N1]
  662     1 root     S    21788  17%   0% HCBv2 happ_kpi       [OK] [S1|N1]
  684     1 root     S    21632  17%   0% HCBv2 hdrv_p1        [OK] [!connected] [S1|N1]
  682     1 root     S    21464  17%   0% HCBv2 hcb_watchdog   [OK] [S20|N1]
  677     1 root     S    21004  17%   0% HCBv2 hcb_rrd        [OK] [S1|N1]
  665     1 root     S    20636  16%   0% HCBv2 happ_smartplug [OK] [S1|N1]
  664     1 root     S    20508  16%   0% HCBv2 happ_scsync    [OK] [S1|N1]
  667     1 root     S    20444  16%   0% HCBv2 happ_usermsg   [OK] [S1|N1]
  668     1 root     S    14436  11%   0% HCBv2 happ_weather   [OK] [S20|N1]
  695     1 root     S    13512  11%   0% /HCBv2/bin/lighttpd -D -m /HCBv2/lib/lighttpd -f /HCBv2/etc/lighttpd/lighttpd.conf
  661     1 root     S    12956  10%   0% HCBv2 happ_eventmgr  [OK] [S1|N1]
  671     1 root     S    12276  10%   0% HCBv2 hcb_log        [OK] [S1|N1]
... and more

The MEM% seem strange to me? how can a process take up 286% of the memory? What is the purpose of hcb_comm anyway, probably for json requests? it seems to have 21 clients or something.

Other than DomoticzBoard I have no non-factory apps in Globals:

Code: Select all

                                // Base set of apps that is available on all displays
                                var appsToLoad = [
                                                        "homescreen",
                                                        "systray",
                                                        "thermostat",
                                                        "clock",
                                                        "systemSettings",
                                                        "thermostatSettings",
                                                        "internetSettings",
                                                        "eMetersSettings",
                                                        "settings",
                                                        "graph",
                                                        "DomoticzBoard"]

I don't use the Zwave/Hue/eMeters but I guess these can't be disabled?

[marcelr]

But is it also normal for Toon that it runs out of memory after +/-9 days? I checked my ram usage after the kill (which was 20MB free) and today it only has 10MB free. So I guess it's going to run out of memory pretty soon.

So you have a leak somewhere. Toon's own code is absolutely leak-free, trust me. I would go find it somewhere in the added stuff. On another note: The root fs is partly mapped to memory. /var, /tmp and the likes. Not sure how quickly that fills up. Any process can write there.

The MEM% seem strange to me? how can a process take up 286% of the memory? What is the purpose of hcb_comm anyway, probably for json requests? it seems to have 21 clients or something.

It's virtual memory, not physical. Most of the flash is mapped to memory. The Freescale iMX27 has an address space of up to 4 GB, so virtual memory can become large.

I don't use the Zwave/Hue/eMeters but I guess these can't be disabled?

Not really, unless you rip a lot of the qml code apart.

[Timeless]

Not sure how quickly that fills up. Any process can write there.

Yeah I noticed that Toon's factory software is also continuously logging to /tmp because the OpenVPN is disabled and such. It ain't much but still after a while it will grow.

So you have a leak somewhere. Toon's own code is absolutely leak-free, trust me. I would go find it somewhere in the added stuff. On another note: The root fs is partly mapped to memory. /var, /tmp and the likes. Not sure how quickly that fills up. Any process can write there.

Mmh thats strange. I'm still running 3.4.4 which is fine I suppose. but even after disabling DomoticzBoard and rebooting Toon I still only have 16MB of RAM free. I did notice that /var /tmp directors are mapped as a tmpFS but unless there are files copied over from FLASH back to RAM after a reboot I see no reason why my RAM is so bloated. A reboot should normally clear the tmpFS. If I may ask, what is your average RAM usage marcel?
Because other than my customized version of DomoticzBoard I do not have any custom software running. I know that Linux treads memory as "not used RAM = lose of resources" but still having 16MB free of 128MB is quite low, and because it's killing processes it is probably really low on RAM.

I don't use the Zwave/Hue/eMeters but I guess these can't be disabled?

Not really, unless you rip a lot of the qml code apart.

Mmh, like i feared . I hoped that a easy setting like setting "ContentApps" to 0 disabled stuff like Hue but unfortunately it did not.

Maybe I can add a USB flash drive and map it as swap. Don't know how long USB NAND flash will last though.

[marcelr]

The later 3.x.x versions had issues with RAM filling up. After the gui had been revamped and a lot of software rewritten, memory usage dropped by some 10MB. Still, 10MB free RAM is not so bad.
I'm not sure what's eating all this memory, during bootup, before the gui is started, free memory is some 60MB. The apps only eat all the rest.

Why is this? You tell me. For the tiny bits of data that the apps process, 50 MB of RAM is a ridiculous amount of memory.

[flamingbob]

See 4.9 also is rolling out today. Lets hope we can then get the boiler information back. Probably just need to reopen ssh again and see how it works. Interesting that a lot of persons are complaing about bugs also.
https://forum.toon.nl/software-updates- ... ie-4-9-871
Also have a private beta with the new toon app. Maybe also intersting to look more into that part and see what can be reused for us.

[Ierlandfan]

Fast observations running 4.9, looking through project.xml and stracing happ_thermstat:
The good News: We can query the important OT options through bxt now. It depends on your CV which options are available. (Or in case of a rooted Toon, probably just edit in the options you want to query)
For example:

Code: Select all

CurrentBoilerTemperature            (Already in 4.8 and succesfully queried)
CurrentBoilerReturnTemperature  (Already in 4.8 and succesfully queried)
BoilerBurnerHours (New)
BoilerDhwBurnerHours (New)
BoilerFailedBurnerStarts (New)
BoilerSuccesfulBurnerStarts(New)
BoilerPumpStarts (New)
BoilerDhwBurnerStarts (New)
BoilerChPressure (New)

The last one ((Waterpressure) is a nice one on my Agpo and the whole reason I actually started the whole rooting investigation. Waterpressure in my case always defaults to 1.4 (1.39 rounded actually) (and not showing the actual pressure) and simply throws error to Toon if too low. (Whatever value that may be) But since Toon is downstairs and my shower not...I wanted an message before jumping around in cold water Rooting saved me also a lot of time since that damn thing kept failing especially during wintertimes.

Back on topic :

As stated probably more values exist depending on your boilertype.
It' s probably a pre-populated list of values based on boiler manufacturer and Boiler (Beta) user input.
I can also see in the strace from happ_thermstat that Toon is actively checking for exisiting ID's and throws a "Not supported" message otherwise.

The (not so bad) News is that they (Quby) completely removed the logic to update the PrintTableInfo (was still there in 4.8 but not updating the values)
but we can easely come up with a script that can query those ID's now through bxt. Only need to extend it now.

Code: Select all

./bxt -c -q BoilerChPressure -d 7d8ef81a-b3b6-4c0c-a0f9-272973bd7497

Code: Select all

<query class="response" uuid="7d8ef81a-b3b6-4c0c-a0f9-272973bd7497" destuuid="bxt.c-7169" serviceid="urn:hcb-hae-com:serviceId:specific1">
    <u:QueryStateVariableResponse xmlns:u="urn:hcb-hae-com:service:specific1:1"><BoilerChPressure>1.39</BoilerChPressure>
    </u:QueryStateVariableResponse>
</query>

It also seems to do something with the outside temperature (Or maybe to be used in future updates)

Code: Select all

[happ_thermstat]handleOutdoorTempUpdate (Query): 8.90

That's outside temp my CV does not has but they changed all (really all) of the internals last year so I am not very sure about that anymore)

(And why the forum writes the word "News" with a Capital while I do not wrote it that way is beyond my understanding.)

[marcelr]

Hi Ierlandfan,

Again, a nice find.

Looking at the reply to the ./bxt query, it shouldn't be too hard to squeeze the queries into a piece of qml code. If that works, getting all data back, as in the old http query, shouldn't be too hard.

[Toonz]

Nice find indeed. Will have a look to get this into a pice of qml (via bxt)