MQTT discussion topic

Forum about MQTT, machine-to-machine (M2M), "Internet of Things" and Node.js

Moderators: Digit, Rene, Willem4ever, Bwired

MQTT discussion topic

Postby MarnixT » Sun Dec 29, 2013 11:28 am

Hi

since the discussion in this topic (domoticaforum.eu/viewtopic.php?f=66& ... =30#p67074) became a little offtopic, I have created this one.

Personally, I think home automation as of now still feels like a large unexplored territory. Standardization feels very much absent. There's a huge amount of software available, but they all feel somewhat incomplete and/or still take lots of effort from people using it. Home automation is not a 1 click install thing, the combinations of hardware/protocols (eg. kaku, x10, zwave, plugwise, elv max, ip-based things like network audio/streaming, roomba, telephony, camera's etc.) is huge and everyone has bought different stuff.

There's a number of programs on the market, but they tend to be huge and monolithic. Programs become huge because they have to embed a lot of functionality. On the one hand they should be able to interface with all this different hardware and map information on the hardware level to some internal structure. This is often done with the use of a plugin mechanism, but still every program has its own plugins. Then there's usually some functionality to handle events and perform actions because of them. So that takes some scripting/if-then-else kind of stuff. And then we have the GUI side of it. People these days want a slick web interface, but also access from android/ios devices.
I tried some programs, but for example home seer feels too much 1980's to me. Domotiga is very nice in the hardware support department but for me the GUI side is trailing. Domoticz is easy to install and the GUI is nice, but it's still young and I'm not sure in what direction it is going. Still enjoy using it though. Some people I admire are able to completely write their own software but that's a little too much for me.

Anyway, also because of Robert's excellent blog I got interested in MQTT (blog.hekkers.net/2012/10/13/realtime-da ... socket-io/ and blog.hekkers.net/2012/06/24/ready-for-t ... he-future/ for example). Since I don't want to also create my own webpages I thought about using Domoticz as well. It already has a nice GUI and a lot of core functionality. It's pretty open because using JSON it is possible to interface with it (http://www.domoticz.com/wiki/Domoticz_API/JSON_URL's). It can also execute LUA and/or bash scripts whenever something happens inside, so in those scripts you can interface with the outside world.
So I was able to do just that: interfacing to and from Domoticz with MQTT. Now I should start set up the messaging, for which this https://github.com/maartendamen/DomoMQTT/wiki seems an excellent start. I will also start rereading all blogs by Robert, now that I know a little more.

My first question here is: who else is interested and/or working with MQTT?
MarnixT
Starting Member
Starting Member
 
Posts: 42
Joined: November 2012

Re: MQTT discussion topic

Postby Rene » Sun Dec 29, 2013 11:07 pm

I am ;-) All software I have coded the last year communicates using MQTT.
Rene.
User avatar
Rene
Global Moderator
Global Moderator
 
Posts: 1689
Joined: October 2008
Location: Netherlands

Re: MQTT discussion topic

Postby Bwired » Sun Dec 29, 2013 11:55 pm

busy too, transforming current (lagecy) bwired software to node.js/mqtt
User avatar
Bwired
Administrator
Administrator
 
Posts: 5304
Joined: March 2006
Location: Netherlands

Re: MQTT discussion topic

Postby Digit » Mon Dec 30, 2013 12:14 am

For the record: so am I :wink:

I chose MQTT because the protocol is: relatively easy, lightweight, available from Arduino to desktop/server, libraries are available for many programming languages, ...
Need I go on? :D
User avatar
Digit
Global Moderator
Global Moderator
 
Posts: 3388
Joined: March 2006
Location: Netherlands

Re: MQTT discussion topic

Postby MarnixT » Mon Dec 30, 2013 1:09 am

No surprises yet :-) I must say, for now especially nodejs proves to be very powerful as well. Maybe I should add it to the topic title as well.
edit: only now I found out that this topic is now in a subforum which also has Node.js in the name. Excellent!
MarnixT
Starting Member
Starting Member
 
Posts: 42
Joined: November 2012

Re: MQTT discussion topic

Postby pbrand » Mon Dec 30, 2013 7:51 pm

I can relate to your opening post to a large extent, Marnix.

I too am searching for some mature software which still seems hard to find in home automation land. Which of course, like many others, results in writing your own software. The down side of writing it yourself is that it's not that easy to let it interface with existing packages which perhaps have a nice gui.

I would not dare to say that the multitude in solutions would entirely disappear when some uniformity occurs in home automation land. This because, home automation is often a hobby of developers of some sort. And for every 5 developers there usually exist probably 6 visions/opinions about what is the best solution :mrgreen:

I personally don't see any fun in node.js. I really don't like javascript (which is a bummer being a .net web developer :D) and why I should write server side javascript, untyped, unreadable, is beyond me. But we all have our own tastes in what we like.

As far as MQTT is concerned however, I will eagerly follow this topic to learn and perhaps make some small contributions here and there if I can. Although it is only a protocol, it provides us with a nice mechanism to let software communicate which each other in a hardware, and software, agnostic manner.

Perhaps someone will point out some good software to us which integrates in a MQTT way :)
pbrand
Member
Member
 
Posts: 100
Joined: October 2008
Location: Netherlands

Re: MQTT discussion topic

Postby Digit » Mon Dec 30, 2013 9:05 pm

Facebook (Messenger, IIRC) for instance, so maybe you're already an MQTT user
But don't know if that's good software.. :wink:
User avatar
Digit
Global Moderator
Global Moderator
 
Posts: 3388
Joined: March 2006
Location: Netherlands

Re: MQTT discussion topic

Postby MarnixT » Mon Dec 30, 2013 10:48 pm

pbrand wrote:I can relate to your opening post to a large extent, Marnix.

I too am searching for some mature software which still seems hard to find in home automation land. Which of course, like many others, results in writing your own software. The down side of writing it yourself is that it's not that easy to let it interface with existing packages which perhaps have a nice gui.

I would not dare to say that the multitude in solutions would entirely disappear when some uniformity occurs in home automation land. This because, home automation is often a hobby of developers of some sort. And for every 5 developers there usually exist probably 6 visions/opinions about what is the best solution :mrgreen:

Yes, it's a funny paradox. The more we're searching for mature standardized software, the more we end up developing. MQTT has long term perspective but stil has a long way to go (in the home automation context at least). In 2011 I thought Google and it's android@home would change things, but nobody heard anything after it's initial presentation.
And now I've got lot's of mqqt messages flying around but nothing happens with them :-)
MarnixT
Starting Member
Starting Member
 
Posts: 42
Joined: November 2012

Re: MQTT discussion topic

Postby Digit » Tue Dec 31, 2013 12:06 am

I had the same in the beginning. MQTT might look like one big chat room at first, where everyone (or, in this case, every thing) can contribute in it's own way. And if you're not careful enough, every[one|thing] in that chat room will receive all the chatter, even the stuff they're not interested in.

So the first thing to do is create some structure - on IRC, you have channels for that - topics are related to channels in some way, but are more flexible (wildcards and stuff).

In fact, it's all about the different types of information streams in your application. Identify them and give them a name.
For example, I've got 'raw' sensor information travelling from here to there over my network, as well as human readable values (those are mostly for the UI's).
And I've also got commands/actions going from A to B - for example from a rules engine (A) to a hardware driver (B), those commands make the roller shutters go up @ sunrise.
For now, I can live with those 3 top-level topics. Oh and there are 2 "technical" topics - 1 for system settings and another one for logging.

Just give "it" structure, think about what you really need. It's not that much different from normalization what you'd do when creating a new database, imho.

What more can I say, or do. I've got some ideas about that, but maybe others do too.
User avatar
Digit
Global Moderator
Global Moderator
 
Posts: 3388
Joined: March 2006
Location: Netherlands

Re: MQTT discussion topic

Postby pbrand » Tue Dec 31, 2013 12:37 am

What I find the greatest challenge to be so far is to determine a topic structure with a hierarchy which makes clever use of the wild card possibilities when subscribing to topics.

A client could for example take an interest in all the temperatures of all available sensors, including both temperature only sensors and the temperature readings on all thermostats.
Whilst another client would perhaps only take an interest in the temperatures of all thermostats.

Or the state of a(ll) (shutter)contact(s) in a single room, an entire floor, an entire building.

Foreseeing what possible requests one can expect and match the topic hierarchy to that is not yet all to clear to me.

I'm not quite seeing the analogy with database normalisation (yet :) ).
pbrand
Member
Member
 
Posts: 100
Joined: October 2008
Location: Netherlands

Re: MQTT discussion topic

Postby Digit » Tue Dec 31, 2013 9:06 am

What I was talking about is just the first step - minimize redundancy and dependency. I've used those objectives before in another work area :)
I don't want static metadata in my topics nor in their payloads - at least, not in those few essential ones.

The things you are talking about can be done with republishing - presenting the same information in a derived hierarchy for a specific purpose.
Cause you'll never succeed in creating a single topic hierarchy that will fit all your future needs - wishful thinking.
At least that was my conclusion at that time.

A client could for example take an interest in all the temperatures of all available sensors

Most of the times examples are too simple compared to reality: how about the CPU temperature sensors of the Rapsberry Pi's, how do you keep those out when the client is only interested in the room temperatures...
Trying to think of everything that might be interesting in the future is impossible unless you have a crystal ball and I ain't got one :wink:

I wrote about this (republishing) some time ago : http://blog.hekkers.net/2012/09/18/mqtt ... lean-code/
User avatar
Digit
Global Moderator
Global Moderator
 
Posts: 3388
Joined: March 2006
Location: Netherlands

Re: MQTT discussion topic

Postby mlommers » Tue Dec 31, 2013 11:50 am

Agree with Robert, keep the devices as and topic structure as clean and stupid as possible. While making another one responsible for publishing on a more informative structure. Keep this responsibility on 1 place, makes it easy to maintain and keeps a good seperation of concern.
mlommers
Starting Member
Starting Member
 
Posts: 12
Joined: July 2010

Re: MQTT discussion topic

Postby MarnixT » Tue Dec 31, 2013 12:35 pm

Interesting discussions. I'm not yet far enough to figure out how exactly I'm going to setup the topics. It will come by itself I guess.
For now I'm starting with this
// https://github.com/maartendamen/DomoMQTT/wiki
// actions/device_id/property
// events/device_type/device_id/property
// config/id
// logging/id

so for instance /events/domoticz/"name of device in Domoticz"/temperature.
In the above link it is suggested to use a JSON object for the payload. But why? It doesn't make sense to me because at the lowest level it's easy and straightforward to just create extra topics. (e.g. /events/domoticz/"name of device in Domoticz"/humidity). Or am I missing something here?

And regarding the comparison to databases: MQTT is more like a hierarchical database model instead of a relational model. And republishing makes to overcome some of the limitations the hierarchical setup I guess.
MarnixT
Starting Member
Starting Member
 
Posts: 42
Joined: November 2012

Re: MQTT discussion topic

Postby Digit » Tue Dec 31, 2013 2:12 pm

I also create a separate topic for every device value, for example an Oregon Scientific sensor can have 4 of them: */temp, */hum, */batt and */comfort. Cause if you want all the info of that sensor, you simply do a */#.
Using an object notation for the payload does keep it more flexible, e.g. adding a timestamp later on, so you know when that value in the payload was 'sensed'.

So that you can store information like when the AV receiver was powered on (*/powerstate = {value:on, changed: "13:44:00"}), but also when the last time was that the volume level has changed (*/volume = {value:35, changed: "14:00:23"}) In a way, you keep all options open for the future, without having to change a single line of code other than for adding that new 'feature'. I don't use JSON in the payload.
User avatar
Digit
Global Moderator
Global Moderator
 
Posts: 3388
Joined: March 2006
Location: Netherlands

Re: MQTT discussion topic

Postby MarnixT » Fri Jan 03, 2014 8:07 am

I'm starting to see the advantages of JSON, since it allows to use different kinds of payloads for the same topic.
See code example in this topic :
http://www.domoticz.com/forum/viewtopic ... 7434#p7434

Sent from my GT-I9505 using Tapatalk
MarnixT
Starting Member
Starting Member
 
Posts: 42
Joined: November 2012

Next

Return to MQTT & Node.js

Who is online

Users browsing this forum: No registered users and 1 guest

cron