Toon app: Zon op Toon using PVoutput

[bwbboot]

Hi Everyone,

I was hoping to source some assistance in getting "Zon op Toon" working without the need for an expensive engineer to install extra hardware.
Most solar inverters have the ability to extract its usage already and export its data to external sites or locations. In my situation i am uploading all data to https://pvoutput.org/list.jsp?userid=15171 every 5 minutes and it looks like this:

[/url]

PVOutput has a very rich API system which can be used to upload and retrieve data as well. Now would it not be awesome to run a script that runs every 5 minutes and updates Toon's internals with the solar generation details as if it would using the official zwave meter that is installed by the engineer.

I have created a simple script that can be used to extract the information from PVOutput but lack the skills to get the information in Toon's (RRD) databases. And this is where i would need the community assistance. This is the script that i have created which runs locally on Toon:

Code: Select all

#!/bin/sh

CURL_OPT1=/tmp/pvoutput1
SYSTEM_ID1=13033
PVOUTPUT_APIKEY=1234567890
TIMEZONE=Europe/Amsterdam

echo "-d \"d=`env TZ=$TIMEZONE date +"%Y%m%d"`\"" > $CURL_OPT1
echo "-d \"t=`env TZ=$TIMEZONE date +"%H:%M"`\"" >> $CURL_OPT1
echo "--header \"X-Pvoutput-Apikey: $PVOUTPUT_APIKEY\" " >> $CURL_OPT1
echo "--header \"X-Pvoutput-SystemId: $SYSTEM_ID1\" " >> $CURL_OPT1
echo "--url https://pvoutput.org/service/r2/getstatus.jsp" >> $CURL_OPT1

curl -q -s -K $CURL_OPT1
echo -e "\n"

Its output would be as simple as:

Date,Time,Energy,Power,Energy Used,Power Used,???,Temperature,Voltage
20170805,22:45,11930,0,9700,850,NaN,15.6,0.0

The fields would translate to:

Energy = Wh generated so far (11,930 kWh)
Power = Instant power generated (0 W)
EnergyUsed = Wh consumed so far (9,700 kWh)
PowerUsed = Instant power consumed (850 W)
NaN = Don't know this is will have to look it up
Temperature = This is taken from WeatherUnderground by PVOutput
Voltage = This is Inverter output Voltage to the power-net

Since i also upload my P1 meter information to PVOutput consumption fields are populated. This information is not so relevant here since it can be taken from Toon locally.

Anyone here that would give it a try to get this information injected in to the Toon local system so that solar power (Zon op Toon) information is populated?

Cheers!

[Toonz]

You might want to look into tools like this (haven't fiddled around with the rrd databases myself):
https://oss.oetiker.ch/rrdtool/doc/rrdupdate.en.html

[bwbboot]

Hey Toonz,

Thanks for the suggestion. Looked at it indeed however as a second though this might not be the correct way to handle this. By updating the RRD databases directly you would miss out on all the logic that Toon applies. For example all calculations to graph consumption and generation correctly: generation-consumption=net delivery. What do you think?

Assume the best approach would be to ingest data as if it would come from the meter installed by the engineer. Would need to somehow fake one in the system and then update its values. Unfortunately i don't have this meter installed and therefor the UUID is not generated .

/qmf/config/config_hcb_rrd.xml

Code: Select all

 <rrdLogger>                                                                                                                                                                                                                  
     <package>hcb_rrd</package>                                                                                                                                                                                                  
     <uuid>eneco-001-XXXXXX:hcb_config_67295XXXXXX</uuid>                                                                                                                                                                        
     <name>elec_solar_flow</name>                                                                                                                                                                                                
     <internalAddress>elec_solar_flow</internalAddress>                                                                                                                                                                          
     <visibility>0</visibility><deviceUuid>eneco-001-XXXXXX:hdrv_p1_6767CXXXXXX</deviceUuid><deviceVar>CurrentElectricityFlow</deviceVar><deviceSvc>ElectricityFlowMeter</deviceSvc><sampleType>double</sampleType><rra>         
    <binLength>300</binLength>                                                                                                                                                                                                   
    <bins>8928</bins>                                                                                                                                                                                                            
    <futureBins>0</futureBins>                                                                                                                                                                                                   
    <consolidator>avg</consolidator>                                                                                                                                                                                             
    <name>5min</name>                                                                                                                                                                                                            
    <minSamplesPerBin>1</minSamplesPerBin>                                                                                                                                                                                       
    <saveInterval>1</saveInterval></rra></rrdLogger>                                                                                                                                                                             
    <rrdLogger>

There must be a way to simply ingest the elec_solar_flow directly into Toon's internals by using a 'fake' flow meter.

Any ideas on the approach would be welcome!

[marcelr]

The problem is that toon doesn't use proper rrd databases. The data are stored in combinations of .rra and .dat files. (in /mnt/data/qmf/var/hcb_rrd/*.rra and /mnt/data/qmf/var/hcb_rrd/*.dat)
The actual meaning of these files is stored in /mnt/data/qmf/config/config_hcb_rrd.xml.

Some time ago I made a php script to read these files and convert them to readable .csv files with sensible names. It's stored here:
viewtopic.php?f=87&t=8743&start=75#p77504

It should give at least an idea about the file formats, and how to work with them.

[bwbboot]

Possibly there is a way to update values by using the /qmf/bin/bxt binary. Trying various options to see if I can make it work.

Code: Select all

eneco-001-XXXXXX:/qmf/bin# ./bxt 
[hxml] libhcb_xml          v0.1 r95    (Sep 16 2016 14:47:44) [master#5f72a810bf] (ezxml 0.8.6)
[hbxt] libhcb_boxtalk      v0.1 r108   (Sep 16 2016 14:48:00) [master#3304abef08]
[hcom] libhcb_com          v0.1 r109   (Sep 16 2016 14:48:13) [master#10e07c208f] (OpenSSL 0.9.8m)
[hbas] libhcb_base         v0.1 r259   (Feb  8 2017 16:43:05) [master#0090ab8ae9]
[hdrv] libhcb_drv          v0.1 r315   (Apr  5 2017 12:17:51) [master#4021121776]
[bxt.c] bxt.c              v r315      (Apr  5 2017 12:17:53) [master#4021121776] pid(4771)
--------------------------------------------------------------------------------------------
valid options for './bxt':

	-h host        host to connect to. Default: 'localhost'
	-p port        port to connect to. Default: '1337'
	-s serviceid   service id of message. Default: 'specific1'
	-u uuid        fake source UUID to send msg from, eg. 'hdrv_null'
	-d dest_uuid   destination UUID to send msg to, start with ':' to auto-prefix CN
	-r [uuid]      subscribe to uuid & serviceid '*'. Default: uuid of msg
	               note: -r needs to be specified *after* -d!
	-w secs        wait for incoming msgs before exit; -1 = 1 msg, 0 = EoU. Default: '-1'
	-a argument    name of an argument, eg. 'mask'
	-v value       value of last given argument, eg. '11'
	               the option combo '-a x -v y' can be repeated!
	-t             test: do not really send the message
	-c             condensed output: hide timestamp/srcfile/line/level lines
	-b             print all received BoxTalk messages to console
  HBXT_ACTION_INVOKE/RESPONSE messages:
	-n name        action name to set in BoxTalk message, eg. 'setLedMask[Response]'
	-x xml         action arguments xml, eg. '<arg><subarg>val</subarg></arg>'
  HBXT_QUERY_INVOKE/RESPONSE messages:
	-q variable    variable name to query for, eg. 'CurrentState'
  HBXT_NOTIFY messages:
	-N             make this a notify message from -u
  Quick 'n dirty shortcuts: 
	-Q             '-n quit'
	-P             '-n ping'
	-E xSTATE      '-d hdrv_hcbhw  -n RawCommand -i hcb-hw-leds -a lednr -v x -a state -v STATE'
	-O msg         '-d hcb_log     -n log -a type -v 1 -a text -v msg'
	-D dim         '-n SetLoadLevelTarget -a NewLoadLevelTarget -v dim'
	-S val         '-n SetTarget -a NewTargetValue -v val'
	-B lvl         '-n SetStdoutLvl -s Debug1 -r -a ALL -v lvl'
	-L lvl         '-n SetNotifyLvl -s Debug1 -r -w MAX_INT -a ALL -v lvl'

--------------------------------------------------------------------------------------------

[TerrorSource]

I like the idea of importing the Smart meter data or converter data directly into Toon, importing via Domoticz can also be an option.
Domoticz already has a method to read data from PVOutput or SBFspot build-in.

[marcelr]

The smart meter data are already there, stored in a bunch of .rra files.
If we take the path to custom kernels, we could also build SFBspot for toon an run everything locally. The custom kernel would be required to add bluetooth support to toon.

[bwbboot]

Marcel, would you be able to create a script that updates the solar part of the .rra files using the PVOutput CURL method i posted before?

[marcelr]

In which programming language?
Toon has /bin/sh, qml (a sort of javascript), and that's about it. Not really fit for the job, either of them.

Writing valid files is quite different from reading files and converting them to a format of your choosing. Data need to be synchronized with toon's clock, resampled to 5 minute intervals, and then written.
You will need to parse a chunk of xml, write doubles to a file, and hope for the best. With a little luck (not!) there's a checksum stashed away somewhere (Quby are very fond of checksums), so probably that needs calculation ...
(when I was writing the serial IO between toon and boiler module, I spent most of the time in finding the package checksum calculation methods and seeds).
The UUID of each of the files changes with every restart, so that's a bit of thing too.

Before you know it, you're knee deep in a swamp filled with all kinds of libraries, timing issues and unexpected features that need sorting, and then write it up in C or something similar.
Might not be as simple as it looks.

[bwbboot]

Thanks Marcel. Ok, so basically the best approach is by somehow updating the solar piece by ingesting in for example the P1 module level. This way all the regular mechanics will apply and data will be processed as if a real solar meter is attached. Also this method will most likely continue to work after updates. One million dollar question is how to get this done. Each of my tries using the BXT binary failed...

[marcelr]

What you could do is feed S0 pulses from the inverter directly into the meter adapter, and then activate zon op toon, no extra kWh meter required. I can get you the wiring specifics if needed. This requires an inverter that generates those pulses.

[bwbboot]

Yes interested. This works by using the existing P1 meter? My inverter does not support s0 pulses but can try to convert the data to s0 pulses using my RPI.

[marcelr]

You will need a recent meteradapter. PN: 6599-1500-0201, or later. (mine is from 2015, SN starts with 15).
I will try and dig up the schematics (nothing fancy, but it's somewhere in the attic, may take some time to find).

[bwbboot]

I did some googling and its going to be too much hassle to get it to work using s0 pulses (in my case, since my inverter does not have a s0 pulse output). Looks like i need to keep investigating options to update values directly into toon.

[marcelr]

I think it's possible to fill the databases through bxt calls. Someone will need to dig deeper into that to get it to work. However, it's not very high on my priority list right now, I'm afraid.