Using NodeJS as MQTT client for Toon

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, Toonz

HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

You don't need mosquitto stuff. But you indeed do need to install mqtt via npm.

Are there different versions of the device? I have installed NodeJS packages a couple of times. I haven't observed the behaviour during the installs though, but the installations succeeded without any intervention. Could also be that I don't have any other stuff running on to maybe?
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

No hardware specs differences as what I am aware of.

The NPM install took about 5 minutes and at least a disabled qt-gui to have enough memory. I do have some extra tiles/apps on the qt-gui running so yeah maybe I was just having a bit too much.

But also running the test script with node took at least 10 seconds to run and took also a big chunk of memory. Keep sending any interesting data towards any mqtt broker will then probably just take too much memory/cpu.

Isn't there a more lightweight mqtt client which can be installed/compiled? Even a small esp8266 can run a mqtt client so there should be something less memory eating than using node.js
Member of the Toon Software Collective
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

Never mind. I'm running an install of mqtt now, and indeed it eats up all the memory. I also got disconnected from ssh. Also, it seems that mqtt was only partially installed. Just running npm --help eats up all the memory.
I just searched a bit for a lightweight version of npm, but did not find any. Thing is, for a specific project, we could probably do without npm. It's likely that once mqtt is installed, it could be distributed with the package, so you don't have to install it yourself. I've activated npm/nodejs packages before simply by copying the folder from node_modules. In this case, it might need some additional work, like creating symlinks. I re-ran the install of mqtt, and now it finished. So it will install eventually, but it's not very clean, and it is probably disruptive. Is it maybe possible to install an npm package using the oe-toolchain on another host, and create an mqtt add-on to nodejs that way?

I have sort of a proof of concept nodejs server running at the moment. It does connect to mqtt, but doesn't send anything yet. It uses 30% of memory according to top, net I'd say about 10MB. A couple of those MB's is probably mqtt. Without it was more like 3MB I think. So it's not very efficient as far as resources go. Since it doesn't do anything for most of the time, cpu utilization seems to not be a problem at the moment. For now, it reads the latest value from the database, with an interval of one minute. I haven't implemented the integer part yet. I do not use the xml config, because that seems to require another module. So it's not very clean, but it seems to do the trick.

Anyway, I'm not giving up just yet ;)
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

As far as proof of concept goes, see attachment ;)
Attachments
GrafanaPoC.png
GrafanaPoC.png (55.54 KiB) Viewed 11744 times
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

Can you share your proof of concept code?
Member of the Toon Software Collective
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

Sure, see attachment.

It makes a list of all the dat files in the rra directory, puts the 30days ones in an array, and iterates over the array every minute, getting the latest value, and pushing it out over mqtt.

As people have stated earlier on in this post, the NodeJS/NPM/MQTT combination does seem quite 'heavy'. Also, not all data is written to the database, so the boiler modulation for example will still have to be retrieved in a different manner. Since the database data is not real time, I'm starting to wonder what the added value of mqtt is here, especially because of the effort it takes to get it on the device, and the resources it needs.
So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?
Attachments
follow.js.zip
(2.85 KiB) Downloaded 571 times
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

Just noticed that Toon already has mosquitto-clients (mosquitto-pub and -sub) available for download via opkg (when connected to Eneco).
Very lightweight but ofcourse only commandline so only usable in a bash script or something like that.

But running a bash script from cron, which pulls some data from the rrd (using a curl http to localhost) and then using the mosquitto-pub to publish that (as mqtt messages) to a broker is probably consuming a less memory then the node-js solution.
Member of the Toon Software Collective
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
Toonz
Forum Moderator
Forum Moderator
Posts: 1873
Joined: Mon Dec 19, 2016 1:58 pm

Re: Using NodeJS as MQTT client for Toon

Post by Toonz »

HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?
member of the Toon Software Collective
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

Toonz wrote:
HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?
Just read the follow.js.zip and saw that HansS is reading the data directly from the RRD (so not using a http call to localhost) which ofcourse is a very nice job!

You can install the original Toon mosquitto from their feeds servers (you must have the openvpn running):

Code: Select all

 opkg install http://feed.hae.int/feeds/qb2/oe/2.6.36-base-r10-qmf0/libmosquitto_1.4.9-r0_qb2.ipk
 opkg install http://feed.hae.int/feeds/qb2/oe/2.6.36-base-r10-qmf0/mosquitto-clients_1.4.9-r0_qb2.ipk
Member of the Toon Software Collective
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

HansS wrote: So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?
One url is providing a lot of information already (except for example water temperature and boiler pressure): http://toon-ip/happ_thermstat?action=getThermostatInfo
That is one call. I'm polling that using a python script currently, which then feeds a domoticz server.

This information is probably also available using the /qmf/bin/bxt command. So doing this in a bash script and then using the mosquitto_pub for the mqtt output could be a lightweight solution.
Member of the Toon Software Collective
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

Adding to my own post,.. just figured out the correct bxt command to get the current thermostat info:

Code: Select all

 /qmf/bin/bxt -d :happ_thermstat -s specific1 -n webRequest -a action -v getThermostatInfo -c 2>&1 | grep u:webRequestResponse
Need some more bash parsing.
Member of the Toon Software Collective
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

Toonz wrote:
HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?
Yeah, that's right. And as TheHogNL stated, I read the rrd files directly. But I use NodeJS for that, so that can be considered overhead as well. MQTT is more valuable if you can push out data the moment it changes. Which might still be possible. Not sure how it works yet, but I saw that the builtin thermostat app reacts to boiler info changes somehow. Don't know if that's also bound to minute interval polls. But maybe it's not. I believe that the thermostat displays the flame icon this way, and it doesn't take a minute to determine that it needs to be on.

But, http for now. I do use the url that you mention, but indeed the boiler temperature values are not in there. Most of the values are not that interesting to me though. So for now I just get the boiler temperatures, so three http calls in total.

I tried installing the mosquito stuff, but I seem to be missing some libraries:

* satisfy_dependencies_for: Cannot satisfy the following dependencies for libmosquitto:
* libssl1.0.0 (>= 1.0.2h) * libcrypto1.0.0 (>= 1.0.2h) *
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

What's your FW version again?
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

HansS wrote: I tried installing the mosquito stuff, but I seem to be missing some libraries:

* satisfy_dependencies_for: Cannot satisfy the following dependencies for libmosquitto:
* libssl1.0.0 (>= 1.0.2h) * libcrypto1.0.0 (>= 1.0.2h) *
Avaiable from the same source, filenames are:

Code: Select all

libssl1.0.0_1.0.2h-r2_qb2.ipk
libcrypto1.0.0_1.0.2h-r2_qb2.ipk
Member of the Toon Software Collective
Post Reply

Return to “Toon software development”