Pachube issue, graphing/polling intervals and energy cost

[infernix]

Hi,

Installed myself a Plugwise system yesterday, and after toying with Source in Wine (it installed but couldn't connect to the internet so couldn't activate it) and then inside a Qemu VM just to get the stuff initialized, I had enough of that and installed Domotiga. In one word: awesome

I'm wondering about a few things though:

- Pachube data isn't being submitted correctly when using non-English languages. Debug log shows that it simply does not send anything as <value>, but the rest of the xml seems fine:

Code: Select all

<data id='0'>
<tag>Server power usage</tag>
<value></value>
<unit symbol='Watt' type='basicSI'></unit>

The RRDtool graphs were working fine so I suspected a bug in the Pachube code. I'm running DomotigaServer so I changed the Pachube.module code slightly but that didn't help. I then looked at the mysql table and then it hit me:

Code: Select all

+----+--------------+--------------------+-----------------------------+----------+-------------+------------------+-------+----------+---------+
| id | datastreamid | tags               | devicename                  | deviceid | devicelabel | devicelabelshort | units | unittype | value   |
+----+--------------+--------------------+-----------------------------+----------+-------------+------------------+-------+----------+---------+
|  1 | 0            | Server power usage | Serverruimte                |        3 | Watt        | NULL             |       | basicSI  | Waarde2 

I set Domotiga to dutch, and then it'll start filling in Waarde2 in the value column. But the Pachube.module code looks like this:

Code: Select all

     SELECT rResult!value
        CASE "Value"
          sValue = Devices.GetCurrentValueForDevice(rResult!deviceid)
        CASE "Value2"
          sValue = Devices.GetCurrentValue2ForDevice(rResult!deviceid)
        CASE "Value3"
          sValue = Devices.GetCurrentValue3ForDevice(rResult!deviceid)
        CASE "Value4"
          sValue = Devices.GetCurrentValue3ForDevice(rResult!deviceid)
      END SELECT

That's just never going to work Switching to English and reconfiguring Pachube fixes this.

- Polling and graphing intervals have me confused somewhat. I've set the Plugwise polltime to 300 because RRDTool is also set to that, but if I change them both to, say, 1 minute, will the RRDTool graphs still work? Or does that need adjustments for the many more data points?

- Energy cost is something I couldn't find. It would basically require the user to enter his energy pricing scheme, and use that together with polled data to calculate costs. I'm not sure if the rrdtool dataset is usable for this; is the data also stored in mysql? If it's not there, I could have a go at it; unfortunately, the last time I used Basic (apart from some obligatory braindead stuff at school) was on an Acorn Electron, so saying my Basic skills are rusty is quite the understatement

Thanks for any hints

[RDNZL]

Since tonight we have a bigger Pachube problem, they updated their software and broke the API.
Well an specific post mechanism, and DomotiGa happens to use that...

See http://community.pachube.com/node/358

I never got sending those headers working, so I use the _method=post workaround.
I will try to come up with a solution soon.

There is energy, water and gas data stored in mysql table, but only for RFXMeters.
The rest of the sensors store data only in rrdtool.

I'm investigating a way to store all sensor data in mysql.

I will fix the Pachube Value and translation issue too.

[infernix]

Since tonight we have a bigger Pachube problem, they updated their software and broke the API.
Well an specific post mechanism, and DomotiGa happens to use that...
See http://community.pachube.com/node/358

Looks like they found out because i tweeted them about it, that's my example API url

I tried to patch gb.net.curl because I think the hPost.Headers object parameter isn't being used at all. Unfortunately I couldn't get it to work. I found your post on the mailing list, perhaps you can follow up on that? Because I too believe it's a gambas bug.

Meanwhile I'll try to make automatic feeds work by hacking on the webclient; I'm more comfortable with PHP than C

[Merdeka]

Ah, ok.

I thought it was me. I had just added two more sensors and pachube stopped working. But it was a coincidence.
Have you tried rddtool on a NSLU2 Ron? I couldn't install it with apt-get.

[RDNZL]

I fixed the problems with Value when non-English GUI language was used, both for Pachube and Bwired module.
You have to check/adjust your Bwired and Pachube devices for correct Value settings after the update.

The RRDTool databases are created with the RRDTool polltime as the -s step parameter, (the rate at which data is added to the database) so if you change that, you have to recreate your rrd db files (just delete them from Database Admin tool)

I don't know if the RRA:AVARAGE values are correct for 1 minute update though, maybe they need to be adjusted too not to loose to much details. I'm not a RRDTool wizzkid...
You can find them in the RRDTool.module in the CreateRRDs() routine.

They are hardcoded, maybe I should make them configurable?
Let me know if you want that.

@Timo:
I never installed RRDTool on NLSLUG, but since there are some weather related projects for NSLUGs it kinda suprised me that installing rrdtool is not working.
Maybe the dependencies are wrong? it needs some ttf libraries for example.

@infernix:
Yeah, I believe the headers are not working because of a Gambas bug too, didn't get a reply on my cry for help on that one some time ago
Tried a bit to get the curl workaround working, but didn't succeed yet, I hope they fix the API soon.

Thanks for your bugreport.

[infernix]

As far as I know, it's a limitation for the rails version they are using now after they upgraded, so I don't think it can be fixed easily at their end.

Can we skip Curl and just use TCP instead? It might be a bit more work but even if gambas gets fixed, that won't help anyone with an older version.

[j.hoekstra]

Just received a warning from pachube that my polling-interval is too high and they limited it.
Switched pachube off for now, more users having these problems?

[infernix]

OK, I wrote a small patch to get Pachube auto feeds working with DomotiGa.

The patch changes CXMLRPC so that it accept a new xmlrpc call, changes Pachube so that the CreateData function is accessible and adds a new view to the webclient that simply outputs the values from the xmlrpc call.

To use it, you have to enable the php webclient and access it with ?setview=pachube, like this:

Code: Select all

http://your.server.com/domotiga/?setview=pachube

Patch attached, it's not ideal due to the .htaccess restrictions on the webclient but it works for me; one could just copy the web client to some randomly named directory and use that for the Pachube automatic feed URL.

Enjoy!

Code: Select all

--- old/webclient/index.php	2010-01-15 01:38:46.000000000 +0100
+++ new/webclient/index.php	2010-01-17 18:03:53.000000000 +0100
@@ -44,6 +44,20 @@
 if (isset($r_reload)) unset($_SESSION['lastget']);
 if (!isset($_SESSION['refresh'])) $_SESSION['refresh']=$defaultrefresh;
 if (!isset($r_debug)) $r_debug=0;
+
+// Pachube view only outputs the data from Pachube.CreatePachubeData()
+if ($_SESSION['view']=="pachube") {
+   $request = xmlrpc_encode_request("pachube.list",null);
+   $response = do_xmlrpc($request);
+   if (xmlrpc_is_fault($response)) {
+       trigger_error("xmlrpc: $response[faultString] ($response[faultCode])");
+   } else {
+     header("Content-type: application/xml");
+     print($response);
+   }
+   exit;
+}
+
 ?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
--- old/DomotiGa/CXMLRPC.class	2010-01-15 01:38:44.000000000 +0100
+++ new/DomotiGa/CXMLRPC.class	2010-01-17 17:48:17.000000000 +0100
@@ -101,6 +101,8 @@
   hXMLRPC.Register(hRpcFunc)
   hRpcFunc = NEW RpcFunction("voicetext.speak", [XmlRpc.xString, XmlRpc.xString], XmlRpc.xBoolean)
   hRpcFunc.Help = ("speak text param1 with voice param2")
+  hRpcFunc = NEW RpcFunction("pachube.list", NULL, XmlRpc.xString)
+  hRpcFunc.Help = ("get pachube device list in eeml")
   hXMLRPC.Register(hRpcFunc)
   TRY hXMLRPC.Listen(iHTTPPort, iMaxConn)
 
@@ -443,6 +445,8 @@
       PlaySound(sData)
     CASE "voicetext.speak"
       VoiceTextSpeak(sData)
+    CASE "pachube.list"
+      hXMLRPC.SetReply(Pachube.CreatePachubeData())
     DEFAULT
       RETURN
   END SELECT
--- old/DomotiGa/Pachube.module	2010-01-15 01:38:44.000000000 +0100
+++ new/DomotiGa/Pachube.module	2010-01-17 17:34:02.000000000 +0100
@@ -68,7 +69,7 @@
 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 ' scan pachube devices table and create xml document
 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-PRIVATE FUNCTION CreatePachubeData() AS String
+PUBLIC SUB CreatePachubeData() AS String
 
   DIM sXml, sValue, sTag AS String
   DIM rResult AS Result
--- old/DomotiGaServer/CXMLRPC.class	2010-01-15 01:38:44.000000000 +0100
+++ new/DomotiGaServer/CXMLRPC.class	2010-01-17 17:48:17.000000000 +0100
@@ -101,6 +101,8 @@
   hXMLRPC.Register(hRpcFunc)
   hRpcFunc = NEW RpcFunction("voicetext.speak", [XmlRpc.xString, XmlRpc.xString], XmlRpc.xBoolean)
   hRpcFunc.Help = ("speak text param1 with voice param2")
+  hRpcFunc = NEW RpcFunction("pachube.list", NULL, XmlRpc.xString)
+  hRpcFunc.Help = ("get pachube device list in eeml")
   hXMLRPC.Register(hRpcFunc)
   TRY hXMLRPC.Listen(iHTTPPort, iMaxConn)
 
@@ -443,6 +445,8 @@
       PlaySound(sData)
     CASE "voicetext.speak"
       VoiceTextSpeak(sData)
+    CASE "pachube.list"
+      hXMLRPC.SetReply(Pachube.CreatePachubeData())
     DEFAULT
       RETURN
   END SELECT
--- old/DomotiGaServer/Pachube.module	2010-01-15 01:38:44.000000000 +0100
+++ new/DomotiGaServer/Pachube.module	2010-01-17 17:34:02.000000000 +0100
@@ -68,7 +69,7 @@
 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 ' scan pachube devices table and create xml document
 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-PRIVATE FUNCTION CreatePachubeData() AS String
+PUBLIC SUB CreatePachubeData() AS String
 
   DIM sXml, sValue, sTag AS String
   DIM rResult AS Result

[RDNZL]

I just came home, and enabled my Pachube module, it seems to work ok again, they fixed the broken _method=put option it seems.
I post every 10 minutes and didn't get a warning yet, haven't searched there website, if a new policy is enforced.

@infernix:
Nice work, didn't had time to found out how Pachube's auto feeds worked.
Will look into it a bit better soon.

For the last 3-4 weeks, the number of people who sent me new modules, bug-fixes/reports, code snippets and ideas is growing rapidly, I like that!
Please continue to do that, it makes me happy if others use the software I started to write with that goal in mind.

[RDNZL]

@infernix: I have merged your code, it's online now.

One can now enter a PushTime of 0 to enable automatic feeds, so DomotiGa will not Push data, but have Pachube get it by pulling the URL. You have to change your feeds to enable this. I also updated the wiki with this info.

[j.hoekstra]

Hmmm, it looks like it's the app I started on my phone to view the the feed I created(unfortunately it doesn't really work ).

As soon as I start the app it seems to reload the feed 2 quickly and after half an hour I get the warning.

Sorry for the fuss.......

[RDNZL]

Which phone app? IPhone?
I use the Android app, it works.

Keep in mind that the application needs to supply the apikey with its request to get the data more often than once per 15 minutes.
In other words; You need it to get realtime data.

[j.hoekstra]

I'm using the android app as well, it can use other feeds, but just not mine.
It keeps hanging hon loading feed.
I use tha API-key to be able to pull data and added a feed coming from domotiga. However I somehow got the feeling I'm doing something wrong an probably has to do with the type of feed.

[RDNZL]

Mail the author of the app, and supply your feed id, I also had some problems, he fixed them and updated his application.
I guess you are running the latest version?

[infernix]

guys, just don't put in your API key. If you do it will refresh every 10 seconds. Which is useless anyway, because it only shows historical data which is showing only one datapoint every 15 minutes.