Toon app: Zon op Toon using PVoutput

[Ierlandfan]

Yes, I think bxt is the way to go. I was thinking the same. I also wrote my own script to dump the values from a Steca 1500 Solar converter that dumps it to PVoutput.
I don\t know which package is responsible for the Solar writing part. IF we know the package I can create the bxt part.
Since my laptop decided to retire I only have a tablet with physical keyboard to investigate atm so it takes a little longer now.
Then it's just a matter of writing some simple bash script that runs every x minutes and updates the values.

[marcelr]

Hi Ierlandfan,

I think it's the hcb_rrd executable that does all that work. From /HCBv2/config/config_hcb_rrd.xml at least, it looks like that. Not sure how (yet), but maybe I can squeeze more info from that executable with strace, similar to our fishing expedition in the OT protocol implementation. No guarantees, though.

What exactly are we looking for? The bxt protocol, or the shape of the data as extracted from the zwave interface, or something else?

[bwbboot]

Do you have strace available as an ipk? Possibly tcpdump as wel?

[marcelr]

Yes, you can find them here:

http://files.domoticaforum.eu/uploads/T ... r1_qb2.ipk
http://files.domoticaforum.eu/uploads/T ... r1_qb2.ipk

tcpdump depends on this one:

http://files.domoticaforum.eu/uploads/T ... r1_qb2.ipk

Neither are the latest versions, they were built from the openembedded tree for toon.

[bwbboot]

Many thanks, works without any issues so far!

[marcelr]

You're welcome. Just my curiosity: What are you trying to do with it?

[bwbboot]

Well, i figured since TOON can be updated from the mothership (like activate subscription) i want to run tcpdump on the TAP0 interface for a while to see whats coming in and out.
Interestingly my first findings confirm that communication is done by using the /bxtproxy. I even wrote a quick/dirty script to test if I could have it accept data. And it seems to work. Now trying to figure out if it can be used to update other stuff as well.

Code: Select all

#!/bin/bash
#
/usr/bin/curl -v -H "X-BoxTalk-Class: JS-Pavel" -H "X-BoxTalk-UUID: BoxtalkGatewayServlet" -H "Cache-Control: no-cache" -H "Pragma: no-cache" -H "Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2" -H "Connection: keep-alive" -H "Content-type: application/x-www-form-urlencoded" -A "User-Agent: Java/1.8.0_131" -X POST -d '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<action class="invoke" uuid="a0b6a2690-f343-11e1-a21f-0800200c9a66" destuuid="eneco-001-XXXXXX:happ_thermstat" serviceid="urn:hcb-hae-com:serviceId:specific1" commonname="eneco-001-XXXXXX" xmlns:ns5="urn:hcb-hae-com:service:ConfigProvider:1" xmlns:u="urn:hcb-hae-com:service:specific1:1" xmlns:ns4="urn:hcb-hae-com:service:SwitchPower:1" xmlns:ns3="urn:hcb-hae-com:service:Benchmark:1">
    <u:UpdateDataSetSubscription>
        <updateAction>extend</updateAction>
        <dataSet>thermostatInfo thermostatStates</dataSet>
    </u:UpdateDataSetSubscription>
</action>' http://192.168.2.180/bxtproxy

echo -e "\n"

The result:

Code: Select all

*   Trying 192.168.2.180...
* Connected to 192.168.2.180 (192.168.2.180) port 80 (#0)
> POST /bxtproxy HTTP/1.1
> Host: 192.168.2.180
> User-Agent: Java/1.8.0_131
> X-BoxTalk-Class: JS-Pavel
> X-BoxTalk-UUID: BoxtalkGatewayServlet
> Cache-Control: no-cache
> Pragma: no-cache
> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
> Connection: keep-alive
> Content-type: application/x-www-form-urlencoded
> Content-Length: 624
> 
* upload completely sent off: 624 out of 624 bytes
< HTTP/1.1 200 OK
< Content-Length: 12
< Date: Tue, 08 Aug 2017 12:28:22 GMT
< Server: lighttpd/1.4.33-devel-17
< 
* Connection #0 to host 192.168.2.180 left intact
message sent

The /HCVv2/sbin/hcb_bxtproxy -v process responds with the following message:

Code: Select all

[hbxt]\--------------------------------/
[hcb_bxtproxy:../src/hcb_bxtproxy.c@http_post():444]ERROR: Not a valid BoxTalk message, not sent to hcb_comm!
[hcb_bxtproxy:../src/hcb_bxtproxy.c@updateMsgStatuses():908]Messages  SENT Ok:23 Retries:0 Error:0  RECV Ok:1 Err:22  QUEUE:0 env:PRD
[hcom]Received HBXT_ACTION_RESPONSE from eneco-001-xxxxxx:happ_thermstat to a0b6a2690-f343-11e1-a21f-0800200c9a66/specific1: n=UpdateDataSetSubscriptionResponse result=ok
[hcb_bxtproxy:../src/hcb_bxtproxy.c@handler():1051]ERROR: Retry this msg 3 times | send anonymous: 0
[hcb_bxtproxy:../src/hcb_bxtproxy.c@enqueuePacket_l():112]enqueing for uuid: eneco-001-XXXXXX:happ_thermstat to url: http://feed.hae.int:31080/toonMobileBackendWeb/display/receiveXml data: <action class="response" uuid="eneco-001-xxxxxx:happ_thermstat" destuuid="a0b6a2690-f343-11e1-a21f-0800200c9a66" serviceid="urn:hcb-hae-com:serviceId:specific1" commonname="eneco-001-XXXXXX">
    <u:UpdateDataSetSubscriptionResponse xmlns:u="urn:hcb-hae-com:service:specific1:1"><result>ok</result>
    </u:UpdateDataSetSubscriptionResponse>
</action> retries: 3
[hcb_bxtproxy:../src/hcb_bxtproxy.c@transmitMsgQueueHead_l():926]Sending to url: http://feed.hae.int:31080/toonMobileBackendWeb/display/receiveXml
[hcb_bxtproxy:../src/hcb_bxtproxy.c@http_post():304]ERROR: overrideUrl: http://172.18.96.1:31080/toonMobileBackendWeb/display/receiveXml
[hbxt]/--------------------------------\

Think we can safely ignore the first line since it seems to appear frequently without me interfering:

Code: Select all

"[hcb_bxtproxy:../src/hcb_bxtproxy.c@http_post():444]ERROR: Not a valid BoxTalk message, not sent to hcb_comm!"

Now onto constructing an update message and see what happens.

[marcelr]

Nice, I think ...

Not sure yet what's possible with this, but please carry on!

The master thesis of Jurgen Braam (developer at Quby) describes the boxtalk protocol al little. Maybe it's helpful:
http://files.domoticaforum.eu/uploads/H ... _Braam.pdf

[Ierlandfan]

You can use strace or just invoke the binary with -vvvvvvv
Then you'll see the commands that are being send. Maybe faster than strace. You want to look for the xml request that updates the solar values so I guess a ./hcb_rrd -vvvvvvv |grep solar " or something similar will do. So I would say as long as the z-wave registers a solar value (any value will do) hcb_rrd will try to update that value by an xml request to Boxtalk.


Edit...try to send the commands to localhost:1337, (or open the port in in the iptables for remote access) you're sending things to or through the Quby servers now.

[bwbboot]

no luck so far, any constructed XML posts to /bxtproxy are accepted but totally ignored by the hcb_bxtproxy which seems responsible for processing incoming requests.

@Ierlandfan, i don't have solar generation data so the rrd graphs will never have to process any updates there.

[TerrorSource]

In what format does the data needs to be delivered? Maybe i can give you some information as i do have 3yrs of history data from my solar panels.

[bwbboot]

This is one of the last tests that i did. A strace on for example HCBv2 happ_pwrusage reveils the following messages being sent:

Code: Select all

read(3, "<action class=\"response\" uuid=\"eneco-002-XXXXXX:hcb_config_61127789807\" destuuid=\"eneco-002-XXXXXX:happ_pwrusage\" serviceid=\"urn:hcb-hae-com:serviceId:specific1\" requestid=\"1295-1266\">\n    <u:GetRraDataResponse xmlns:u=\"urn:hcb-hae-com:service:specific1:1\"><requestId>1295-1266</requestId>\n   <rraData>{ \"1502218800\": NaN}</rraData>\n   <firstSample>1344430800</firstSample>\n   <lastSample>1502218800</lastSample>\n   <tzOffset>2</tzOffset>\n    </u:GetRraDataResponse>\n</action>\0", 16384) = 477
gettimeofday({1502218412, 425384}, NULL) = 0
read(3, "<notify uuid=\"5f195db2-8ae7-4e75-a7ba-d227f556c166\" serviceid=\"urn:hcb-hae-com:serviceId:ElectricityQuantityMeter\">\n    <CurrentElectricityQuantity>10798938</CurrentElectricityQuantity>\n</notify>\0", 16384) = 196
read(3, "<notify uuid=\"713aa639-1f74-42fd-a978-4558f50d1159\" serviceid=\"urn:hcb-hae-com:serviceId:ElectricityQuantityMeter\">\n    <CurrentElectricityQuantity>10857074</CurrentElectricityQuantity>\n</notify>\0", 16384) = 196
read(3, "<notify uuid=\"5f195db2-8ae7-4e75-a7ba-d227f556c166\" serviceid=\"urn:hcb-hae-com:serviceId:ElectricityFlowMeter\">\n    <CurrentElectricityFlow>890</CurrentElectricityFlow>\n</notify>\0", 16384) = 179
write(2, "\33[01;34m[\33[01;36mhdrv\33[0m\33[01;34m]\33[0m\33[01;35mDS: \33[0mdatasetUpdate powerUsage\n", 79) = 79
write(2, "\33[01;34m[\33[01;36mhdrv\33[0m\33[01;34m]\33[0m\33[01;35mDS: \33[0mUpdate for dataSet \33[01;35mpowerUsage\33[0m\n", 96) = 96
write(2, "\33[01;34m[\33[01;36mhdrv\33[0m\33[01;34m]\33[0m\33[01;35mDS: \33[0mSend full \33[01;35mpowerUsage\33[0m update to \33[01;35meneco-002-XXXXXX:qt-gui\33[0m\n", 133) = 133
write(3, "<action class=\"invoke\" uuid=\"eneco-002-XXXXXX:happ_pwrusage\" destuuid=\"eneco-002-XXXXXX:qt-gui\" serviceid=\"urn:hcb-hae-com:serviceId:specific1\">\n    <u:UpdateDataSet xmlns:u=\"urn:hcb-hae-com:service:specific1:1\"><powerUsage><value>890</value>\n    <dayCost>4.70</dayCost>\n    <valueProduced>0</valueProduced>\n    <dayCostProduced>0.00</dayCostProduced>\n    <valueSolar>NaN</valueSolar>\n    <maxSolar>NaN</maxSolar>\n    <dayCostSolar>NaN</dayCostSolar>\n    <avgSolarValue>NaN</avgSolarValue>\n    <avgValue>34.61</avgValue>\n    <avgDayValue>830.57</avgDayValue>\n    <avgProduValue>296.56</avgProduValue>\n    <avgDayProduValue>7117.43</avgDayProduValue>\n    <dayUsage>5605</dayUsage>\n    <dayLowUsage>3391</dayLowUsage>\n    <meterReading>10857074</meterReading>\n    <meterReadingLow>10960915</meterReadingLow>\n    <meterReadingProdu>6669844</meterReadingProdu>\n    <meterReadingLowProdu>2570472</meterReadingLowProdu>\n    <isSmart>1</isSmart>\n    <lowestDayValue>0</lowestDayValue>\n    <solarProducedToday>NaN</solarProducedToda"..., 1073) = 1073
write(2, "\33[01;34m[\33[01;36mhdrv\33[0m\33[01;34m]\33[0m\33[01;35mDS: \33[0mSend full \33[01;35mpowerUsage\33[0m update to \33[01;35ma0b6a2690-f343-11e1-a21f-0800200c9a66\33[0m\n", 147) = 147
write(3, "<action class=\"invoke\" uuid=\"eneco-002-XXXXXX:happ_pwrusage\" destuuid=\"a0b6a2690-f343-11e1-a21f-0800200c9a66\" serviceid=\"urn:hcb-hae-com:serviceId:specific1\">\n    <u:UpdateDataSet xmlns:u=\"urn:hcb-hae-com:service:specific1:1\"><powerUsage><value>890</value>\n    <dayCost>4.70</dayCost>\n    <valueProduced>0</valueProduced>\n    <dayCostProduced>0.00</dayCostProduced>\n    <valueSolar></valueSolar>\n    <maxSolar></maxSolar>\n    <dayCostSolar></dayCostSolar>\n    <avgSolarValue></avgSolarValue>\n    <avgValue>34.61</avgValue>\n    <avgDayValue>830.57</avgDayValue>\n    <avgProduValue>296.56</avgProduValue>\n    <avgDayProduValue>7117.43</avgDayProduValue>\n    <dayUsage>5605</dayUsage>\n    <dayLowUsage>3391</dayLowUsage>\n    <meterReading>10857074</meterReading>\n    <meterReadingLow>10960915</meterReadingLow>\n    <meterReadingProdu>6669844</meterReadingProdu>\n    <meterReadingLowProdu>2570472</meterReadingLowProdu>\n    <isSmart>1</isSmart>\n    <lowestDayValue>0</lowestDayValue>\n    <solarProducedToday></solarProducedToday"..., 1072) = 1072

write(3, "<action class=\"invoke\" uuid=\"eneco-002-XXXXXX:happ_pwrusage\" destuuid=\"a0b6a2690-f343-11e1-a21f-0800200c9a66\" serviceid=\"urn:hcb-hae-com:serviceId:specific1\">\n <u:UpdateDataSet xmlns:u=\"urn:hcb-hae-com:service:specific1:1\"><powerUsage><value>890</value>\n

Based on that last write line, i constructed a XML that would update a "known" dataset and destination uuid. Something in the lines of:

Code: Select all

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<action class="invoke" uuid="eneco-002-XXXXXX:happ_pwrusage" destuuid="a0b6a2690-f343-11e1-a21f-0800200c9a66" serviceid="urn:hcb-hae-com:serviceId:specific1" commonname="eneco-002-XXXXXX" xmlns:u="urn:hcb-hae-com:service:specific1:1" 
    <u:UpdateDataSet>
    	<powerUsage>
        <Value>8000</Value>
        </powerUsage>
    </u:UpdateDataSet>

[RDNZL]

While having a look too, I found a (bit offtopic) url which returns alot of data:

Code: Select all

http://x.x.x.x:10080/hcb_config?action=getPackageConfig.json&package=happ_pwrusage
http://x.x.x.x:10080/hcb_config?action=getPackageConfig.json&package=happ_thermstat

etc
Your port can be different.
I didn't post the results, because it seems to contains a lot of personal data.
The resulting json contains items like:

Code: Select all

{daysCountBoilerWasActive":"28"}
{"realCost":"106.46"},{"realLowCost":"81.29"},{"benchmarkResult":"0"},{"visitedMonth":"1"},{"package":"happ_pwrusage"},{"uuid":"eco-001-xxxx:happ_pwrusage_xxxx"},{"type":"elec"},{"internalAddress":"elec_11-116"},{"visibility":"0"},{"mostExpensiveDay":"11"},{"mostExpensiveDayUsage":"24809"},{"worstCaseMonthCost":"172.55"}

Whats the deal with all that dojo stuff in the html code, it seems not complete?

Code: Select all

<script type="text/javascript" src="/inc/dojo-new/dojo/dojo.js" djConfig="parseOnLoad: true, usePlainJson: true"></script>
<script type="text/javascript" src="/inc/dojo-new/dojo/dojo-hcb.js"></script>            
                                                                                                         
<script type="text/javascript">                                                                                             
        dojo.require("hcb.utils");                                           
        dojo.require("hcb.translate");  

[Ierlandfan]

Thanx! The xml revealed the name of the value to be updated! Try it with /hcbv2/bin/bxt - c - d *UUD* -a value produced -v some-value. I hate not having the exact command but tablet is empty so I only have my phone now bxt will give you some output being ok or error on that. I am not sure on the -v but invoke bxt with no options and correct them if necessary.

Edit: we have to construct the xml...out of my head that's another option. At least we have the xml and value name so not a real problem. Well done!

Question is: is Toon subtracting the values from the meter readings...that would be interesting since mine is reversing while producing Energy..I have a non-smart electricity meter...and intend to keep it that way...no alu-head just economically.

[TheHogNL]

On a side note. I recently did something else to get my solar output visible on the Toon.

I have two seperate solar installations which are impossible view togheter using the normal Toon solar meter adapter. They are not on the same kWh meter.

I created a script on a raspberry pi which calculates the total generated solar Watts (which it gets from the seperate kWh meters per each installation, but the source can ofcourse be anything which reports the current Watts of each installation) and then flashes a red LED in just the right amount of interval. The Toon solar sensor is directed towards this red LED and now the Toon is showing the right amount of generated Watts.

A bit crude but it works