Copy Historical data (rra databases) between Toons

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, TerrorSource, Toonz

Copy Historical data (rra databases) between Toons

Postby Toonz » Fri Sep 21, 2018 8:00 pm

Hi all,

I would like to share my latest (small) activity: copy historical data from one Toon to another one.
Source and target Toons can either be a Toon 1 or a Toon 2.
It was my first Python script ever, so things can probably be done smarter, but hey, it works....

This script looks for matching topics in the rra database config files from both Toons and copy matching items from source to target with the target uuid's in the filename.
Items copied from one Toon to another: electricity and gas usage, electricity produced, OpenTherm boiler data and thermostat data (actual temperature, target temperature, program state)

Short 'manual':

- on your pc create three folders rraSource, rraTarget and rraOutput
- copy the contents of folder /qmf/var/hcb_rrd from the source Toon into the folder rraSource (typically about 80 files or more if you have smart switches)
- copy the contents of folder /qmf/var/hcb_rrd from the target Toon into the folder rraTarget (typically about 80 files or more if you have smart switches)
- copy the contents of folder /qmf/var/hcb_rrd from the target Toon into the folder rraOutput as well
- copy the contents of file /qmf/config/config_hcb_rrd.xml from the source Toon into the folder rraSource (contains the description of the files in rraSource)
- copy the contents of file /qmf/config/config_hcb_rrd.xml from the target Toon into the folder rraTarget (contains the description of the files in rraTarget)

- download the python script below and edit the first lines with actual folder names
- execute the Python script on your pc (e.g.: py CopyDataToon.py)

once executed, all matching data has been copied from the rraSource folder into the rraOutput folder (but renamed to the new uuid's of the target Toon)

- copy all files from rraOutput back to the folder /qmf/var/hcb_rrd on the target Toon
- sync and reboot Toon. That's all folks.

Have fun, comments are welcome of course.

Kind regardz,

Toonz
Attachments
CopyDataToon.zip
(817 Bytes) Downloaded 158 times
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1106
Joined: December 2016

Re: Copy Historical data (rra databases) between Toons

Postby spattex » Fri Sep 21, 2018 9:41 pm

Top!
spattex
Starting Member
Starting Member
 
Posts: 20
Joined: July 2009
Location: Netherlands

Re: Copy Historical data (rra databases) between Toons

Postby TheHogNL » Sat Sep 22, 2018 7:19 am

Great work. Does not contain the code to move the solar generated electricity yet.

Didn't knew it was so easy. Could be converted to a bash script which you run on one of both toons and which collect the necessary data itself.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1022
Joined: August 2017

Re: Copy Historical data (rra databases) between Toons

Postby marcelr » Sat Sep 22, 2018 8:09 am

Probably it can be done in qml/javascript as well. Then it should be possible to transform it into a toonstore app.
marcelr
Global Moderator
Global Moderator
 
Posts: 1061
Joined: May 2012
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Postby Toonz » Sat Sep 22, 2018 9:17 am

By all means, be my quest :)
I used Python to learn it a bit, this was more of a 'demo' project for me
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1106
Joined: December 2016

Re: Copy Historical data (rra databases) between Toons

Postby Toonz » Sat Sep 22, 2018 12:18 pm

Solar data is copied as well but only if the target Toon also has a solar rra database.
If we want to copy 'new' rra databases to a Toon you also need to add that section in the config xml file. Haven't played with that option. Might work, haven't tried....
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1106
Joined: December 2016

Re: Copy Historical data (rra databases) between Toons

Postby Rudolf » Thu Sep 27, 2018 9:53 am

Do I have to pair Toon2 / rraTarget with a meteradapter first, or is that totally unrelated to the uuid's in the files rewritten in this script?
Rudolf
Member
Member
 
Posts: 116
Joined: December 2017

Re: Copy Historical data (rra databases) between Toons

Postby Toonz » Thu Sep 27, 2018 10:13 am

Rudolf wrote:Do I have to pair Toon2 / rraTarget with a meteradapter first, or is that totally unrelated to the uuid's in the files rewritten in this script?

good question, depends when the consumption rra databases are being created. Before or after the pairing with the meteradapter.
I guess the latter but am not 100% sure.
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1106
Joined: December 2016

Re: Copy Historical data (rra databases) between Toons

Postby marcelr » Sat Mar 23, 2019 9:50 pm

I'm working on a piece of code that transfers data from an old toon into a new one, typically through the export function, but also through raw data transfer from an old toon. Everything seems to work, but I bump into issues when the old toon has single-tariff electricity and the new one has dual tariff (low/normal). This code eventually should find its way into the TSC menu.

So, I would like to get a dataset from someone who has had a toon for quite some time, and during that period changed his contract from single to dual tariff, or the other way around.
Specifically, I will need /HCBv2/config/config_hcb_rrd.xml and the entire contents of /qml/var/hcb_rrd/ (or /HCBv2/data/hcb_rrd/ in really old firmwares).

If you are ready to hand me those data, please contact me through PM or mail.
marcelr
Global Moderator
Global Moderator
 
Posts: 1061
Joined: May 2012
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Postby glsf91 » Sat Mar 30, 2019 9:56 am

marcelr wrote:I'm working on a piece of code that transfers data from an old toon into a new one, typically through the export function, but also through raw data transfer from an old toon. Everything seems to work, but I bump into issues when the old toon has single-tariff electricity and the new one has dual tariff (low/normal). This code eventually should find its way into the TSC menu.

So, I would like to get a dataset from someone who has had a toon for quite some time, and during that period changed his contract from single to dual tariff, or the other way around.
Specifically, I will need /HCBv2/config/config_hcb_rrd.xml and the entire contents of /qml/var/hcb_rrd/ (or /HCBv2/data/hcb_rrd/ in really old firmwares).

If you are ready to hand me those data, please contact me through PM or mail.


Are you sure that with a single tariff not elec_quantity_lt and elec_quantity_nt are both filled at the right time?
I had single tariff (but not from Eneco) and analog sensor last year and both are filled.
Where can I check where the tariff is defined?
(maybe that was defined wrong)
glsf91
Member
Member
 
Posts: 77
Joined: September 2017

Re: Copy Historical data (rra databases) between Toons

Postby marcelr » Sat Mar 30, 2019 9:14 pm

Are you sure that with a single tariff not elec_quantity_lt and elec_quantity_nt are both filled at the right time?


Probably.

My production toon has single tariff, and my test specimen has dual tariff. No idea where that choice is set. It has elec_quantity_nt completely filled, but that's only shown on the test specimen in the times where normal tariff applies.
Maybe both files are filled normally , but for some reason my production toon does not have that info.

I had single tariff (but not from Eneco) and analog sensor last year and both are filled.
Where can I check where the tariff is defined?
(maybe that was defined wrong)


See previous. No idea.
marcelr
Global Moderator
Global Moderator
 
Posts: 1061
Joined: May 2012
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Postby hafeez » Wed May 08, 2019 9:50 am

I also want to copy my historical electricity and gas usage, solar electricity produced data from my old Toon to my new Toon2. Both are rooted.
Any idea when the app to accomplish this will be released?
Else I might start to copy it via the instructions in this topic but would like to take easy step if available already.
hafeez
Starting Member
Starting Member
 
Posts: 8
Joined: December 2017

Re: Copy Historical data (rra databases) between Toons

Postby marcelr » Wed May 08, 2019 11:37 am

It's in the testing phase right now. Give it another week and there should be a more mature version. If you can't wait, there's an alpha version available from the ftp server, for toon type 2 only. It's called transfer-logs-<long version-number>.ipk. Be sure to use the -b option (creating back-ups), before you begin. xml output is functional, but not very good-looking, will be fixed in later versions. The rra databases should be fine.
To use it, just open a shell and type transfer-logs. This will give you some instructions on how to use the code.
marcelr
Global Moderator
Global Moderator
 
Posts: 1061
Joined: May 2012
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Postby marcelr » Thu May 09, 2019 8:46 pm

Version 0.0.3 is now available for toon2. Prettyprint function added to get a proper xml layout in the output file for monthly usage.

And remember:

THOU SHALT MAKE BACK-UPS!

You will also need the libcurl5 package. Available from the same directory.
marcelr
Global Moderator
Global Moderator
 
Posts: 1061
Joined: May 2012
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Postby marcelr » Sat May 11, 2019 4:47 pm

Just removed the transfer-logs package. some things need fixing first.
marcelr
Global Moderator
Global Moderator
 
Posts: 1061
Joined: May 2012
Location: Ehv


Return to Toon software development

Who is online

Users browsing this forum: No registered users and 1 guest