Using NodeJS as MQTT client for Toon

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, Toonz

HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Using NodeJS as MQTT client for Toon

Post by HansS »

Hello all, new poster here. First of all, good job overall, very nice project you've got going here!

One tip: I reassembled my device and the reset button was stuck inside the casing, which caused it not to start. Freaked me out, but re-seating the board worked for me.

Anyway, I don't like to query the device using http, but would prefer pushing out information via MQTT. This way, I can have my Telegraf /InfluxDB / Grafana setup register and display the (historical) information. I browsed and searched the forum, but did not find a solution for running an MQTT client. If it does exist, please correct me now ;)

So I compiled NodeJS to run on AMRv5. It was a bit of a struggle, but in the end it seems I have it working. I was able to run node and connect to mqtt broker test.mosquitto.org, subscribe and got a message.

Thing is, I haven't really looked into where all the data (boiler status, set temperature, all that stuff) is stored, and how I can go about getting the data (with a node script) and push it out. Would love to push out information the moment it changes. Do you know what's the best way to get the information?

If you're interested in running NodeJS: I haven't got a packaged version. If you have the same processor as I do, my version will probably run, but it will require placing some files and directories manually. I can also post the instruction on how I got the ting compiled.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

If you could post the compiling instructions (and people are interested) we could try and make an installation package out of it, and distribute that through the forum file server.
glsf91
Member
Member
Posts: 184
Joined: Fri Sep 15, 2017 9:25 pm

Re: Using NodeJS as MQTT client for Toon

Post by glsf91 »

HansS wrote: Thing is, I haven't really looked into where all the data (boiler status, set temperature, all that stuff) is stored, and how I can go about getting the data (with a node script) and push it out. Would love to push out information the moment it changes. Do you know what's the best way to get the information?
A lot of information you can get also from RRD. Like this: http://192.168.1.234/hcb_rrd/ReadRRD.html
Here an example: http://192.168.1.234/hcb_rrd?action=get ... 18%2014:00

Here you can find what is in RRD and how to access it: /mnt/data/qmf/config/config_hcb_rrd.xml
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

I used a Docker container to build NodeJS, information : https://github.com/dockcross/dockcross
You cannot run NodeJS versions higher than 0.10.*. Older versions of NodeJS can be downloaded: https://nodejs.org/en/download/releases/. I'm currently using Node.js 0.10.4, not the latest version, but I used an older version because of compile issues. The latest 0.10 version will probably work though.

I eventually used the second procedure as described on https://techsparx.com/software-developm ... mpile.html to perform the compilation of NodeJS. It is similar to the one on the original Crossdock site though. It doesn't work out of the box. Tweaks are contained in the procedure below.

I use a Synology DS716+II for running the container. Create a separate folder for this. This requires command line shell and docker.

Put the NodeJS version 0.10 download in the same folder, and untar (tar -xvf <file.tar>)
cd to the newly created directory (node-v0.10.something)
In that folder, run the steps from the site mentioned above:
docker run --rm dockcross/linux-armv5 >dockcross-linux-armv5
chmod +x dockcross-linux-armv5</nowiki>
I ran the whole thing as root, so the last command was not necessary in my case
The site mentions moving the produced file, but I don't see the need for that for a temporary project.

If you have already used the Dockcross container, and want to compile a new project (different version of NodeJS for example), you need to do a make clean first:
./dockcross-linux-armv5 bash -c '.make clean'

Run the configure command:
./dockcross-linux-armv5 bash -c './configure --without-snapshot --dest-cpu=arm --dest-os=linux'
I disabled ssl at first because I do not use it, so wanted to keep the installation smaller. But when re-executing my own procedure, npm gave me troubles because it wants to use https to connect to the package repository. So I removed the --without-ssl option in the above command for that to work.

Now, an additional step is necessary, as described on https://blog.cepharum.de/en/post/instal ... heezy.html
edit (vi) deps/v8/src/arm/macro-assembler-arm.cc
look for lines reading like this

// We always generate arm code, never thumb code, even if V8 is compiled to
// thumb, so we require inter-working support
#if defined(__thumb__) && !defined(USE_THUMB_INTERWORK)
#error "flag -mthumb-interwork missing"
#endif

// We do not support thumb inter-working with an arm architecture not supporting
// the blx instruction (below v5t). If you know what CPU you are compiling for
// you can use -march=armv7 or similar.
#if defined(USE_THUMB_INTERWORK) && !defined(CAN_USE_THUMB_INSTRUCTIONS)
# error "For thumb inter-working we require an architecture which supports blx"
#endif

Adjust it to read like this:

// We always generate arm code, never thumb code, even if V8 is compiled to
// thumb, so we require inter-working support
#if defined(__thumb__) && !defined(USE_THUMB_INTERWORK)
#error "flag -mthumb-interwork missing"
#endif

#if !defined(CAN_USE_THUMB_INSTRUCTIONS)
# define CAN_USE_THUMB_INSTRUCTIONS 1
#endif

// We do not support thumb inter-working with an arm architecture not supporting
// the blx instruction (below v5t). If you know what CPU you are compiling for
// you can use -march=armv7 or similar.
#if defined(USE_THUMB_INTERWORK) && !defined(CAN_USE_THUMB_INSTRUCTIONS)
# error "For thumb inter-working we require an architecture which supports blx"
#endif

Now you can build node, using:
./dockcross-linux-armv5 bash -c 'make -j4'
My Synology is not the fastest, so compiling does take some time. The j4 option makes the compiler use 4 threads, so depending on the number of cpu/cores on the host machine this can be changed to speed up things.
The build will fail in the end, with the following error:
/usr/bin/arm-linux-gnueabi-ld: bad -rpath option

This can be fixed by changing the linker in out/Makefile. Originally reads:
LINK.target ?= /usr/bin/arm-linux-gnueabi-ld
Needs to be:
LINK.target ?= /usr/bin/arm-linux-gnueabi-g++

Another build can now be started, and just the linker part will be executed again, so it will be done quite fast
./dockcross-linux-armv5 bash -c 'make -j4'

I tried to change this earlier on, but that didn't stick somehow. The option is set in the container's ENV. I tried overriding the ENV, but that also did not work. So had to go with the above workaround.
I haven't packaged the result, so I moved the files manually via SFTP.

Note: Not all directories needed already exist, so you need to create some yourself (/usr/local/ for example)<br />
Copy out/Release/node to /usr/local/bin
Copy deps/npm directory to /usr/local/lib/node_modules/
cd /usr/local/bin
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js npm

Before you can run npm, you need to set an environment variable:<br />
export NODE_PATH='/usr/local/lib/node_modules'

Might be an option to set this in your startup script, else commands will fail when you login again.

The current MQTT V2 removed support for older NodeJS versions, so the latest release you can use is 1.14.1:
npm install -g mqtt@1.14.1

Now you can test with a simple NodeJS script from https://github.com/mqttjs/MQTT.js/tree/v1.14.1#connect
var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://test.mosquitto.org');

client.on('connect', function () {
client.subscribe('presence');
client.publish('presence', 'Hello mqtt');
});

client.on('message', function (topic, message) {
// message is Buffer
console.log(message.toString());
client.end();
});

It should say "hello MQTT".

==Todo==
Check security. Node runs as root, can that be a different user?

Thanks for the info on the rrd, will have a look at that. Still requires https calls though, it seems. Also, there's nothing in my mnt folder. Seems that my folders are setup differently.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

Thanks for the description, will look into it. It should be possible to build this on toon's oe toolchain.
As for the https calls: I have built a lighttpd version with https support. It's currently being tested, so far, I haven't heard any complaints, apart form not being able to use RSA key certificates in conjunction with it.
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Using NodeJS as MQTT client for Toon

Post by Rudolf »

Interesting project, thanks for your efforts and sharing your work.

Pushing changes using MQTT sounds efficient and fast, which I like.
glsf91
Member
Member
Posts: 184
Joined: Fri Sep 15, 2017 9:25 pm

Re: Using NodeJS as MQTT client for Toon

Post by glsf91 »

HansS wrote: Thanks for the info on the rrd, will have a look at that. Still requires https calls though, it seems. Also, there's nothing in my mnt folder. Seems that my folders are setup differently.
It is the same dir as /qmf/config or /HCBv2/config. Otherwise try: find . -name config_hcb_rrd.xml -print
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

I had a go at node-v0.10.48, used the oe-toolchain for toon, on an i7 at 3.4GHz ;-), pretty fast.
HansS wrote: Now, an additional step is necessary, as described on https://blog.cepharum.de/en/post/instal ... heezy.html
edit (vi) deps/v8/src/arm/macro-assembler-arm.cc
look for lines reading like this

... snip ...

# error "For thumb inter-working we require an architecture which supports blx"
#endif
I managed to build node without this edit. What does it do, and is it necessary to run node on toon?
HansS wrote: Now you can build node, using:
./dockcross-linux-armv5 bash -c 'make -j4'
My Synology is not the fastest, so compiling does take some time. The j4 option makes the compiler use 4 threads, so depending on the number of cpu/cores on the host machine this can be changed to speed up things.
The build will fail in the end, with the following error:
/usr/bin/arm-linux-gnueabi-ld: bad -rpath option
Didn't get this error, possibly my toolchain has the compiler set as linker driver by default.
HansS wrote: Note: Not all directories needed already exist, so you need to create some yourself (/usr/local/ for example)<br />
Copy out/Release/node to /usr/local/bin
Copy deps/npm directory to /usr/local/lib/node_modules/
cd /usr/local/bin
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js npm

Before you can run npm, you need to set an environment variable:<br />
export NODE_PATH='/usr/local/lib/node_modules'

Might be an option to set this in your startup script, else commands will fail when you login again.
OK, this needs to go into the installer, no problem.
HansS wrote: The current MQTT V2 removed support for older NodeJS versions, so the latest release you can use is 1.14.1:
npm install -g mqtt@1.14.1

Now you can test with a simple NodeJS script from https://github.com/mqttjs/MQTT.js/tree/v1.14.1#connect
var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://test.mosquitto.org');

client.on('connect', function () {
client.subscribe('presence');
client.publish('presence', 'Hello mqtt');
});

client.on('message', function (topic, message) {
// message is Buffer
console.log(message.toString());
client.end();
});

It should say "hello MQTT".
Will test that later this week.
HansS wrote:==Todo==
Check security. Node runs as root, can that be a different user?
Not by default yet, there's only one user on toon. But it's easy to add one. (adduser comes to mind ;-) ).

Two remarks:
1: node is f*cking huge. 8.9MB for the executable alone. It's statically linked against a few libraries, openssl (which is available on toon) and some others.

EDIT: The first 1.9MB have been shaved off: linked against native libz , libcrypto and libssl :-).

2: The npm_modules also eat up a LOT of space:

Code: Select all

eneco-001-xxxxxx:/usr/local/lib/npm_modules# du -ks
20644	.
Simple question: do you need ALL of it? I can imagine to link node against existing libraries, so as to save some space (and memory later on), the npm_modules, however, take most of the space. Is all of the contents in that directory required?
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

marcelr wrote:
HansS wrote: ... snip ...

# error "For thumb inter-working we require an architecture which supports blx"
#endif
I managed to build node without this edit. What does it do, and is it necessary to run node on toon?
I don't know why this was needed in my setup. But without the modification, I got the "For thumb inter-working we require an architecture which supports blx" error. The modification seems to work around that.
marcelr wrote:
HansS wrote: The build will fail in the end, with the following error:
/usr/bin/arm-linux-gnueabi-ld: bad -rpath option
Didn't get this error, possibly my toolchain has the compiler set as linker driver by default.
Could be. All I know is that this linker was set in the Docker image, and that it is not the one to be used. I have no in-depth knowledge about what I was doing, just managed to bypass some problems using Google ;)
marcelr wrote: Two remarks:
1: node is f*cking huge. 8.9MB for the executable alone. It's statically linked against a few libraries, openssl (which is available on toon) and some others.

EDIT: The first 1.9MB have been shaved off: linked against native libz , libcrypto and libssl :-).
Oh, ok. As I said, I just started looking onto this. I don't know much about the available resources on the device. I read somewhere that you can do without openssl altogether, but you have to configure somewhere that packages may be downloaded over plain http.
marcelr wrote: 2: The npm_modules also eat up a LOT of space:

Code: Select all

eneco-001-xxxxxx:/usr/local/lib/npm_modules# du -ks
20644	.
Simple question: do you need ALL of it? I can imagine to link node against existing libraries, so as to save some space (and memory later on), the npm_modules, however, take most of the space. Is all of the contents in that directory required?
I don't know. Never built node or npm before. Always used Docker images for that, space was no issue until now. Will check if there's build options for embedded devices.
Also, I'm not sure how it will perform. Sending out data via MQTT does sound efficient, unless NodeJS is using a lot of resources to get the messages out. Since only very old versions of node and mqtt can be used, a lot of efficiency benefits of newer versions won't be available.

Didn't know about the oe-toolchain, have to look into that as well.

I looked into the database that stuff is written to, and it doesn't seem to be a standard environment like rrdtool, but some adaptation of that. Trying to see what's a good way to stream the database contents, convert it as it comes in and push it out to mqtt.

Thanks for your help!
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

The static libraries linked into node were built with debugging symbols switched on. Will try again tonight without those. Should make a _BIG_ difference in size.
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

I just compiled NodeJS 0.10.48. This comes with a newer version of npm, which in turn is required to install some packages that might be needed. The older npm version has a bug that causes it not to install dependencies. NodeJS version 0.10.48 seems to work so far.

I've been looking at the most direct way to get the information out of the database. Seems there's no object that can be used directly to query the database. I'm not comfortable with querying through http yet. On the one hand it seems to add resource overhead (http, which in turn needs some mechanism to get the data from the database). On the other hand, it does interpret the data. For now, I'm going to try reading the database files directly from within NodeJS. That does mean interpreting the data in javascript as well.

Will keep you posted.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

HansS wrote:For now, I'm going to try reading the database files directly from within NodeJS. That does mean interpreting the data in javascript as well.

Will keep you posted.
I trust you found the php scripts for database interpretation that I posted quite some time ago.

BTW, I'm down to 6.8 MB for node now ;-). Still a few libs to go.
HansS
Starting Member
Starting Member
Posts: 34
Joined: Thu Feb 15, 2018 11:54 pm

Re: Using NodeJS as MQTT client for Toon

Post by HansS »

Nice job reducing the footprint!

Found the php scripts yeah. I just cannot find something in NodeJS that will read the file and continue reading when it reaches EOF, so present me with data as it is appended to the database file. Even the shell's tail -f won't show new data. Oh, I just realised. The database is round robin, so the files don't grow. Need to change tactics. 8)
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Post by marcelr »

Down to 5.1MB, and packaging completed. There's still more weight to be lost, although the packaged npm-files now take up 15MB instead of the 20MB posted earlier (no docs installed). The environment is not yet set.

I tested this on one of my toons, it's fairly slow, I'm afraid, and I don't have the mosquitto stuff on my test toon, so the example fails.
For people who want to give it a spin, it's here:

http://files.domoticaforum.eu/uploads/T ... r0_qb2.ipk
http://files.domoticaforum.eu/uploads/T ... r0_qb2.ipk

And remember to set the npm environment:

Code: Select all

export NODE_PATH='/usr/local/lib/node_modules'
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Using NodeJS as MQTT client for Toon

Post by TheHogNL »

I have had a succesful test (Hello mqtt printed).
However I did take A LOT OF memory. First to install the mqtt 1.14.1 using npm took so much memory I had to stop qt-gui (and keep it from restarting) to free up enough memory to let npm run.
Second, when running the test script it took also some amount of memory. But it worked.

Don't think it would be fast enough to rough native on the Toon. As there would be some kind of script to first read the values from rrd (or something else) and then present it to mqtt. This will overload the Toon I guess. To bad, I did like the idea.
Member of the Toon Software Collective
Post Reply

Return to “Toon software development”