Using NodeJS as MQTT client for Toon

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, TerrorSource, Toonz

Using NodeJS as MQTT client for Toon

Postby HansS » Sun Feb 18, 2018 1:51 pm

Hello all, new poster here. First of all, good job overall, very nice project you've got going here!

One tip: I reassembled my device and the reset button was stuck inside the casing, which caused it not to start. Freaked me out, but re-seating the board worked for me.

Anyway, I don't like to query the device using http, but would prefer pushing out information via MQTT. This way, I can have my Telegraf /InfluxDB / Grafana setup register and display the (historical) information. I browsed and searched the forum, but did not find a solution for running an MQTT client. If it does exist, please correct me now ;)

So I compiled NodeJS to run on AMRv5. It was a bit of a struggle, but in the end it seems I have it working. I was able to run node and connect to mqtt broker test.mosquitto.org, subscribe and got a message.

Thing is, I haven't really looked into where all the data (boiler status, set temperature, all that stuff) is stored, and how I can go about getting the data (with a node script) and push it out. Would love to push out information the moment it changes. Do you know what's the best way to get the information?

If you're interested in running NodeJS: I haven't got a packaged version. If you have the same processor as I do, my version will probably run, but it will require placing some files and directories manually. I can also post the instruction on how I got the ting compiled.
HansS
Starting Member
Starting Member
 
Posts: 20
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Sun Feb 18, 2018 2:42 pm

If you could post the compiling instructions (and people are interested) we could try and make an installation package out of it, and distribute that through the forum file server.
marcelr
Global Moderator
Global Moderator
 
Posts: 1012
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby glsf91 » Sun Feb 18, 2018 8:31 pm

HansS wrote:Thing is, I haven't really looked into where all the data (boiler status, set temperature, all that stuff) is stored, and how I can go about getting the data (with a node script) and push it out. Would love to push out information the moment it changes. Do you know what's the best way to get the information?


A lot of information you can get also from RRD. Like this: http://192.168.1.234/hcb_rrd/ReadRRD.html
Here an example: http://192.168.1.234/hcb_rrd?action=get ... 18%2014:00

Here you can find what is in RRD and how to access it: /mnt/data/qmf/config/config_hcb_rrd.xml
glsf91
Member
Member
 
Posts: 59
Joined: September 2017

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Sun Feb 18, 2018 10:55 pm

I used a Docker container to build NodeJS, information : https://github.com/dockcross/dockcross
You cannot run NodeJS versions higher than 0.10.*. Older versions of NodeJS can be downloaded: https://nodejs.org/en/download/releases/. I'm currently using Node.js 0.10.4, not the latest version, but I used an older version because of compile issues. The latest 0.10 version will probably work though.

I eventually used the second procedure as described on https://techsparx.com/software-developm ... mpile.html to perform the compilation of NodeJS. It is similar to the one on the original Crossdock site though. It doesn't work out of the box. Tweaks are contained in the procedure below.

I use a Synology DS716+II for running the container. Create a separate folder for this. This requires command line shell and docker.

Put the NodeJS version 0.10 download in the same folder, and untar (tar -xvf <file.tar>)
cd to the newly created directory (node-v0.10.something)
In that folder, run the steps from the site mentioned above:
docker run --rm dockcross/linux-armv5 >dockcross-linux-armv5
chmod +x dockcross-linux-armv5</nowiki>
I ran the whole thing as root, so the last command was not necessary in my case
The site mentions moving the produced file, but I don't see the need for that for a temporary project.

If you have already used the Dockcross container, and want to compile a new project (different version of NodeJS for example), you need to do a make clean first:
./dockcross-linux-armv5 bash -c '.make clean'

Run the configure command:
./dockcross-linux-armv5 bash -c './configure --without-snapshot --dest-cpu=arm --dest-os=linux'
I disabled ssl at first because I do not use it, so wanted to keep the installation smaller. But when re-executing my own procedure, npm gave me troubles because it wants to use https to connect to the package repository. So I removed the --without-ssl option in the above command for that to work.

Now, an additional step is necessary, as described on https://blog.cepharum.de/en/post/instal ... heezy.html
edit (vi) deps/v8/src/arm/macro-assembler-arm.cc
look for lines reading like this

// We always generate arm code, never thumb code, even if V8 is compiled to
// thumb, so we require inter-working support
#if defined(__thumb__) && !defined(USE_THUMB_INTERWORK)
#error "flag -mthumb-interwork missing"
#endif

// We do not support thumb inter-working with an arm architecture not supporting
// the blx instruction (below v5t). If you know what CPU you are compiling for
// you can use -march=armv7 or similar.
#if defined(USE_THUMB_INTERWORK) && !defined(CAN_USE_THUMB_INSTRUCTIONS)
# error "For thumb inter-working we require an architecture which supports blx"
#endif

Adjust it to read like this:

// We always generate arm code, never thumb code, even if V8 is compiled to
// thumb, so we require inter-working support
#if defined(__thumb__) && !defined(USE_THUMB_INTERWORK)
#error "flag -mthumb-interwork missing"
#endif

#if !defined(CAN_USE_THUMB_INSTRUCTIONS)
# define CAN_USE_THUMB_INSTRUCTIONS 1
#endif

// We do not support thumb inter-working with an arm architecture not supporting
// the blx instruction (below v5t). If you know what CPU you are compiling for
// you can use -march=armv7 or similar.
#if defined(USE_THUMB_INTERWORK) && !defined(CAN_USE_THUMB_INSTRUCTIONS)
# error "For thumb inter-working we require an architecture which supports blx"
#endif

Now you can build node, using:
./dockcross-linux-armv5 bash -c 'make -j4'
My Synology is not the fastest, so compiling does take some time. The j4 option makes the compiler use 4 threads, so depending on the number of cpu/cores on the host machine this can be changed to speed up things.
The build will fail in the end, with the following error:
/usr/bin/arm-linux-gnueabi-ld: bad -rpath option

This can be fixed by changing the linker in out/Makefile. Originally reads:
LINK.target ?= /usr/bin/arm-linux-gnueabi-ld
Needs to be:
LINK.target ?= /usr/bin/arm-linux-gnueabi-g++

Another build can now be started, and just the linker part will be executed again, so it will be done quite fast
./dockcross-linux-armv5 bash -c 'make -j4'

I tried to change this earlier on, but that didn't stick somehow. The option is set in the container's ENV. I tried overriding the ENV, but that also did not work. So had to go with the above workaround.
I haven't packaged the result, so I moved the files manually via SFTP.

Note: Not all directories needed already exist, so you need to create some yourself (/usr/local/ for example)<br />
Copy out/Release/node to /usr/local/bin
Copy deps/npm directory to /usr/local/lib/node_modules/
cd /usr/local/bin
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js npm

Before you can run npm, you need to set an environment variable:<br />
export NODE_PATH='/usr/local/lib/node_modules'

Might be an option to set this in your startup script, else commands will fail when you login again.

The current MQTT V2 removed support for older NodeJS versions, so the latest release you can use is 1.14.1:
npm install -g mqtt@1.14.1

Now you can test with a simple NodeJS script from https://github.com/mqttjs/MQTT.js/tree/v1.14.1#connect
var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://test.mosquitto.org');

client.on('connect', function () {
client.subscribe('presence');
client.publish('presence', 'Hello mqtt');
});

client.on('message', function (topic, message) {
// message is Buffer
console.log(message.toString());
client.end();
});

It should say "hello MQTT".

==Todo==
Check security. Node runs as root, can that be a different user?

Thanks for the info on the rrd, will have a look at that. Still requires https calls though, it seems. Also, there's nothing in my mnt folder. Seems that my folders are setup differently.
HansS
Starting Member
Starting Member
 
Posts: 20
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Mon Feb 19, 2018 9:08 am

Thanks for the description, will look into it. It should be possible to build this on toon's oe toolchain.
As for the https calls: I have built a lighttpd version with https support. It's currently being tested, so far, I haven't heard any complaints, apart form not being able to use RSA key certificates in conjunction with it.
marcelr
Global Moderator
Global Moderator
 
Posts: 1012
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby Rudolf » Mon Feb 19, 2018 9:44 am

Interesting project, thanks for your efforts and sharing your work.

Pushing changes using MQTT sounds efficient and fast, which I like.
Rudolf
Member
Member
 
Posts: 114
Joined: December 2017

Re: Using NodeJS as MQTT client for Toon

Postby glsf91 » Mon Feb 19, 2018 7:08 pm

HansS wrote:Thanks for the info on the rrd, will have a look at that. Still requires https calls though, it seems. Also, there's nothing in my mnt folder. Seems that my folders are setup differently.


It is the same dir as /qmf/config or /HCBv2/config. Otherwise try: find . -name config_hcb_rrd.xml -print
glsf91
Member
Member
 
Posts: 59
Joined: September 2017

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Mon Feb 19, 2018 8:14 pm

I had a go at node-v0.10.48, used the oe-toolchain for toon, on an i7 at 3.4GHz ;-), pretty fast.

HansS wrote:Now, an additional step is necessary, as described on https://blog.cepharum.de/en/post/instal ... heezy.html
edit (vi) deps/v8/src/arm/macro-assembler-arm.cc
look for lines reading like this

... snip ...

# error "For thumb inter-working we require an architecture which supports blx"
#endif



I managed to build node without this edit. What does it do, and is it necessary to run node on toon?

HansS wrote:Now you can build node, using:
./dockcross-linux-armv5 bash -c 'make -j4'
My Synology is not the fastest, so compiling does take some time. The j4 option makes the compiler use 4 threads, so depending on the number of cpu/cores on the host machine this can be changed to speed up things.
The build will fail in the end, with the following error:
/usr/bin/arm-linux-gnueabi-ld: bad -rpath option


Didn't get this error, possibly my toolchain has the compiler set as linker driver by default.

HansS wrote:Note: Not all directories needed already exist, so you need to create some yourself (/usr/local/ for example)<br />
Copy out/Release/node to /usr/local/bin
Copy deps/npm directory to /usr/local/lib/node_modules/
cd /usr/local/bin
ln -s /usr/local/lib/node_modules/npm/bin/npm-cli.js npm

Before you can run npm, you need to set an environment variable:<br />
export NODE_PATH='/usr/local/lib/node_modules'

Might be an option to set this in your startup script, else commands will fail when you login again.

OK, this needs to go into the installer, no problem.
HansS wrote:The current MQTT V2 removed support for older NodeJS versions, so the latest release you can use is 1.14.1:
npm install -g mqtt@1.14.1

Now you can test with a simple NodeJS script from https://github.com/mqttjs/MQTT.js/tree/v1.14.1#connect
var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://test.mosquitto.org');

client.on('connect', function () {
client.subscribe('presence');
client.publish('presence', 'Hello mqtt');
});

client.on('message', function (topic, message) {
// message is Buffer
console.log(message.toString());
client.end();
});

It should say "hello MQTT".

Will test that later this week.
HansS wrote:==Todo==
Check security. Node runs as root, can that be a different user?

Not by default yet, there's only one user on toon. But it's easy to add one. (adduser comes to mind ;-) ).

Two remarks:
1: node is f*cking huge. 8.9MB for the executable alone. It's statically linked against a few libraries, openssl (which is available on toon) and some others.

EDIT: The first 1.9MB have been shaved off: linked against native libz , libcrypto and libssl :-).

2: The npm_modules also eat up a LOT of space:
Code: Select all
eneco-001-xxxxxx:/usr/local/lib/npm_modules# du -ks
20644   .


Simple question: do you need ALL of it? I can imagine to link node against existing libraries, so as to save some space (and memory later on), the npm_modules, however, take most of the space. Is all of the contents in that directory required?
marcelr
Global Moderator
Global Moderator
 
Posts: 1012
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Mon Feb 19, 2018 11:46 pm

marcelr wrote:
HansS wrote:... snip ...

# error "For thumb inter-working we require an architecture which supports blx"
#endif


I managed to build node without this edit. What does it do, and is it necessary to run node on toon?

I don't know why this was needed in my setup. But without the modification, I got the "For thumb inter-working we require an architecture which supports blx" error. The modification seems to work around that.

marcelr wrote:
HansS wrote:The build will fail in the end, with the following error:
/usr/bin/arm-linux-gnueabi-ld: bad -rpath option


Didn't get this error, possibly my toolchain has the compiler set as linker driver by default.

Could be. All I know is that this linker was set in the Docker image, and that it is not the one to be used. I have no in-depth knowledge about what I was doing, just managed to bypass some problems using Google ;)

marcelr wrote:Two remarks:
1: node is f*cking huge. 8.9MB for the executable alone. It's statically linked against a few libraries, openssl (which is available on toon) and some others.

EDIT: The first 1.9MB have been shaved off: linked against native libz , libcrypto and libssl :-).

Oh, ok. As I said, I just started looking onto this. I don't know much about the available resources on the device. I read somewhere that you can do without openssl altogether, but you have to configure somewhere that packages may be downloaded over plain http.

marcelr wrote:2: The npm_modules also eat up a LOT of space:
Code: Select all
eneco-001-xxxxxx:/usr/local/lib/npm_modules# du -ks
20644   .


Simple question: do you need ALL of it? I can imagine to link node against existing libraries, so as to save some space (and memory later on), the npm_modules, however, take most of the space. Is all of the contents in that directory required?

I don't know. Never built node or npm before. Always used Docker images for that, space was no issue until now. Will check if there's build options for embedded devices.
Also, I'm not sure how it will perform. Sending out data via MQTT does sound efficient, unless NodeJS is using a lot of resources to get the messages out. Since only very old versions of node and mqtt can be used, a lot of efficiency benefits of newer versions won't be available.

Didn't know about the oe-toolchain, have to look into that as well.

I looked into the database that stuff is written to, and it doesn't seem to be a standard environment like rrdtool, but some adaptation of that. Trying to see what's a good way to stream the database contents, convert it as it comes in and push it out to mqtt.

Thanks for your help!
HansS
Starting Member
Starting Member
 
Posts: 20
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Tue Feb 20, 2018 8:44 am

The static libraries linked into node were built with debugging symbols switched on. Will try again tonight without those. Should make a _BIG_ difference in size.
marcelr
Global Moderator
Global Moderator
 
Posts: 1012
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Tue Feb 20, 2018 8:31 pm

I just compiled NodeJS 0.10.48. This comes with a newer version of npm, which in turn is required to install some packages that might be needed. The older npm version has a bug that causes it not to install dependencies. NodeJS version 0.10.48 seems to work so far.

I've been looking at the most direct way to get the information out of the database. Seems there's no object that can be used directly to query the database. I'm not comfortable with querying through http yet. On the one hand it seems to add resource overhead (http, which in turn needs some mechanism to get the data from the database). On the other hand, it does interpret the data. For now, I'm going to try reading the database files directly from within NodeJS. That does mean interpreting the data in javascript as well.

Will keep you posted.
HansS
Starting Member
Starting Member
 
Posts: 20
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Tue Feb 20, 2018 8:42 pm

HansS wrote:For now, I'm going to try reading the database files directly from within NodeJS. That does mean interpreting the data in javascript as well.

Will keep you posted.
I trust you found the php scripts for database interpretation that I posted quite some time ago.

BTW, I'm down to 6.8 MB for node now ;-). Still a few libs to go.
marcelr
Global Moderator
Global Moderator
 
Posts: 1012
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby HansS » Tue Feb 20, 2018 9:38 pm

Nice job reducing the footprint!

Found the php scripts yeah. I just cannot find something in NodeJS that will read the file and continue reading when it reaches EOF, so present me with data as it is appended to the database file. Even the shell's tail -f won't show new data. Oh, I just realised. The database is round robin, so the files don't grow. Need to change tactics. 8)
HansS
Starting Member
Starting Member
 
Posts: 20
Joined: February 2018

Re: Using NodeJS as MQTT client for Toon

Postby marcelr » Tue Feb 20, 2018 9:47 pm

Down to 5.1MB, and packaging completed. There's still more weight to be lost, although the packaged npm-files now take up 15MB instead of the 20MB posted earlier (no docs installed). The environment is not yet set.

I tested this on one of my toons, it's fairly slow, I'm afraid, and I don't have the mosquitto stuff on my test toon, so the example fails.
For people who want to give it a spin, it's here:

http://files.domoticaforum.eu/uploads/T ... r0_qb2.ipk
http://files.domoticaforum.eu/uploads/T ... r0_qb2.ipk

And remember to set the npm environment:
Code: Select all
export NODE_PATH='/usr/local/lib/node_modules'
marcelr
Global Moderator
Global Moderator
 
Posts: 1012
Joined: May 2012
Location: Ehv

Re: Using NodeJS as MQTT client for Toon

Postby TheHogNL » Wed Feb 21, 2018 7:27 pm

I have had a succesful test (Hello mqtt printed).
However I did take A LOT OF memory. First to install the mqtt 1.14.1 using npm took so much memory I had to stop qt-gui (and keep it from restarting) to free up enough memory to let npm run.
Second, when running the test script it took also some amount of memory. But it worked.

Don't think it would be fast enough to rough native on the Toon. As there would be some kind of script to first read the values from rrd (or something else) and then present it to mqtt. This will overload the Toon I guess. To bad, I did like the idea.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 896
Joined: August 2017

Next

Return to Toon software development

Who is online

Users browsing this forum: No registered users and 1 guest