Using NodeJS as MQTT client for Toon

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, TerrorSource, Toonz

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Wed Feb 21, 2018 9:06 pm

You don't need mosquitto stuff. But you indeed do need to install mqtt via npm.

Are there different versions of the device? I have installed NodeJS packages a couple of times. I haven't observed the behaviour during the installs though, but the installations succeeded without any intervention. Could also be that I don't have any other stuff running on to maybe?
HansS
Starting Member
Starting Member
 
Posts: 22
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Wed Feb 21, 2018 9:51 pm

No hardware specs differences as what I am aware of.

The NPM install took about 5 minutes and at least a disabled qt-gui to have enough memory. I do have some extra tiles/apps on the qt-gui running so yeah maybe I was just having a bit too much.

But also running the test script with node took at least 10 seconds to run and took also a big chunk of memory. Keep sending any interesting data towards any mqtt broker will then probably just take too much memory/cpu.

Isn't there a more lightweight mqtt client which can be installed/compiled? Even a small esp8266 can run a mqtt client so there should be something less memory eating than using node.js
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Wed Feb 21, 2018 10:54 pm

Never mind. I'm running an install of mqtt now, and indeed it eats up all the memory. I also got disconnected from ssh. Also, it seems that mqtt was only partially installed. Just running npm --help eats up all the memory.
I just searched a bit for a lightweight version of npm, but did not find any. Thing is, for a specific project, we could probably do without npm. It's likely that once mqtt is installed, it could be distributed with the package, so you don't have to install it yourself. I've activated npm/nodejs packages before simply by copying the folder from node_modules. In this case, it might need some additional work, like creating symlinks. I re-ran the install of mqtt, and now it finished. So it will install eventually, but it's not very clean, and it is probably disruptive. Is it maybe possible to install an npm package using the oe-toolchain on another host, and create an mqtt add-on to nodejs that way?

I have sort of a proof of concept nodejs server running at the moment. It does connect to mqtt, but doesn't send anything yet. It uses 30% of memory according to top, net I'd say about 10MB. A couple of those MB's is probably mqtt. Without it was more like 3MB I think. So it's not very efficient as far as resources go. Since it doesn't do anything for most of the time, cpu utilization seems to not be a problem at the moment. For now, it reads the latest value from the database, with an interval of one minute. I haven't implemented the integer part yet. I do not use the xml config, because that seems to require another module. So it's not very clean, but it seems to do the trick.

Anyway, I'm not giving up just yet ;)
HansS
Starting Member
Starting Member
 
Posts: 22
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Thu Feb 22, 2018 12:09 am

As far as proof of concept goes, see attachment ;)
Attachments
GrafanaPoC.png
GrafanaPoC.png (55.54 KiB) Viewed 4889 times
HansS
Starting Member
Starting Member
 
Posts: 22
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Thu Feb 22, 2018 8:12 pm

Can you share your proof of concept code?
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Thu Feb 22, 2018 9:13 pm

Sure, see attachment.

It makes a list of all the dat files in the rra directory, puts the 30days ones in an array, and iterates over the array every minute, getting the latest value, and pushing it out over mqtt.

As people have stated earlier on in this post, the NodeJS/NPM/MQTT combination does seem quite 'heavy'. Also, not all data is written to the database, so the boiler modulation for example will still have to be retrieved in a different manner. Since the database data is not real time, I'm starting to wonder what the added value of mqtt is here, especially because of the effort it takes to get it on the device, and the resources it needs.
So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?
Attachments
follow.js.zip
(2.85 KiB) Downloaded 110 times
HansS
Starting Member
Starting Member
 
Posts: 22
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Thu Feb 22, 2018 10:30 pm

Just noticed that Toon already has mosquitto-clients (mosquitto-pub and -sub) available for download via opkg (when connected to Eneco).
Very lightweight but ofcourse only commandline so only usable in a bash script or something like that.

But running a bash script from cron, which pulls some data from the rrd (using a curl http to localhost) and then using the mosquitto-pub to publish that (as mqtt messages) to a broker is probably consuming a less memory then the node-js solution.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Fri Feb 23, 2018 12:43 am

Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
HansS
Starting Member
Starting Member
 
Posts: 22
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby Toonz » Fri Feb 23, 2018 9:42 am

HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.

Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1087
Joined: December 2016

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Fri Feb 23, 2018 5:36 pm

Toonz wrote:
HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.

Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?


Just read the follow.js.zip and saw that HansS is reading the data directly from the RRD (so not using a http call to localhost) which ofcourse is a very nice job!

You can install the original Toon mosquitto from their feeds servers (you must have the openvpn running):

Code: Select all
opkg install http://feed.hae.int/feeds/qb2/oe/2.6.36-base-r10-qmf0/libmosquitto_1.4.9-r0_qb2.ipk
opkg install http://feed.hae.int/feeds/qb2/oe/2.6.36-base-r10-qmf0/mosquitto-clients_1.4.9-r0_qb2.ipk
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Fri Feb 23, 2018 6:05 pm

HansS wrote:So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?


One url is providing a lot of information already (except for example water temperature and boiler pressure): http://toon-ip/happ_thermstat?action=getThermostatInfo
That is one call. I'm polling that using a python script currently, which then feeds a domoticz server.

This information is probably also available using the /qmf/bin/bxt command. So doing this in a bash script and then using the mosquitto_pub for the mqtt output could be a lightweight solution.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Fri Feb 23, 2018 7:04 pm

Adding to my own post,.. just figured out the correct bxt command to get the current thermostat info:

Code: Select all
/qmf/bin/bxt -d :happ_thermstat -s specific1 -n webRequest -a action -v getThermostatInfo -c 2>&1 | grep u:webRequestResponse


Need some more bash parsing.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Sat Feb 24, 2018 7:12 pm

Toonz wrote:
HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.

Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?


Yeah, that's right. And as TheHogNL stated, I read the rrd files directly. But I use NodeJS for that, so that can be considered overhead as well. MQTT is more valuable if you can push out data the moment it changes. Which might still be possible. Not sure how it works yet, but I saw that the builtin thermostat app reacts to boiler info changes somehow. Don't know if that's also bound to minute interval polls. But maybe it's not. I believe that the thermostat displays the flame icon this way, and it doesn't take a minute to determine that it needs to be on.

But, http for now. I do use the url that you mention, but indeed the boiler temperature values are not in there. Most of the values are not that interesting to me though. So for now I just get the boiler temperatures, so three http calls in total.

I tried installing the mosquito stuff, but I seem to be missing some libraries:

* satisfy_dependencies_for: Cannot satisfy the following dependencies for libmosquitto:
* libssl1.0.0 (>= 1.0.2h) * libcrypto1.0.0 (>= 1.0.2h) *
HansS
Starting Member
Starting Member
 
Posts: 22
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Sat Feb 24, 2018 7:23 pm

What's your FW version again?
marcelr
Global Moderator
Global Moderator
 
Posts: 1050
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Sat Feb 24, 2018 8:50 pm

HansS wrote:I tried installing the mosquito stuff, but I seem to be missing some libraries:

* satisfy_dependencies_for: Cannot satisfy the following dependencies for libmosquitto:
* libssl1.0.0 (>= 1.0.2h) * libcrypto1.0.0 (>= 1.0.2h) *


Avaiable from the same source, filenames are:
Code: Select all
libssl1.0.0_1.0.2h-r2_qb2.ipk
libcrypto1.0.0_1.0.2h-r2_qb2.ipk
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 962
Joined: August 2017

PreviousNext

Return to Toon software development

Who is online

Users browsing this forum: No registered users and 1 guest

cron