My home automation project / software

[airox]

Some News about the project so far. The iPad application is finished. It is looking great now thanks to HTML5 effects etc.
I also ordered the Aviosys 9100 video to LAN server to which I connected three cheap (dealextreme) cameras. One on which I added a fisheye lens.
I now have a total view over my garden.

I have a total of 1.5 milion readings from all sort of devices collected in a single database. When I reached around 700000 I optimised the software a bit but it's running smooth now. Off course the size of the database is noticable, but it never takes more then 20 seconds to load a page (the server is a an intel atom low profile cpu).

The biggest advantage I have in the system is being able to monitor my energy usage very closely. Check this link:
http://fotoboek.vandulmen.net/domotica/perdag.png

I have these graphs for every day totalling the costs per day. Another calculation will show me the estimate for the current month based on the last 7 days. I'm looking forward to the winter which will show me how much gas we will be using. So far only 43 euro's a month But this is mostly electricity.

Another advantage is being able to monitor who enters the garden, rings the bell, brings the paper mail and enters the home. Using apple's push mechanism is really helpfull. When at work it's nice to check the house when something happens.

I also implemented automated lights on and off in the living room. With a bit of tuning I am now not turning on the lights myself anymore. This happens automagically. Turning them off isn't necessary because turning on the alarm when going to bed will turn everything off in the living room.

But ... I also have a lot of experiences now with the concepts embeded in the software. Still not really satisfied. The thing I still need to do is to turn all those device readings in real helpfull information like which I have done with the energy costs.

What reports are nice to have? Those need to give some insight in my living pattern. Any ideas?

[nightcrawler]

some ideas for nice reports:

- walking routes at your home can be monitored by the loggin of your PIR sensors. witch sensor has been activated at what time? make an simple count table from that logging and display it in a chart or an image of your floorplan.
- reports with time-between-actions. examples: times opening an door, times of power on an device (tv,pc), time of starting up lightning @ livingroom etc. and if you want to make this fancy take the sunrise and sunset times within this report.

[airox]

Thanks for the reports. I'm gonna implement them in the upcoming weeks.

An update on the project. I have redone the architecture of the system. I added queuing at points where it made sense. The Zwave, Rfxcom, Bluetooth drivers all put events on a queue. Another process looks at this queue, saves the events and value changes to the database and checks if those events trigger any actions. If there are any actions to be triggered this event processing daemon will put actions in another queue. I then have several other daemons (one for each action type like tweeting, sending email, playing sounds, etc) which process this action queue.

The result? Everything got almost instant. When I press a button on my KIKA KCT-150 (rfxcom) the Zwave lights are turned off parallel to playing the sound of enabled alarm. At the same time I get a notification on my iphone that the alarm has been enabled. This all happens parallel and direct after the event. Previously this all was done after each other and so this took a bit longer.

At this time I have about 1.8 million readings in the database. This took it's toll on performance, especially the generation of graphs. So I investigated how to make stuff faster. I first added database indexes where they were appropriate. This resulted in < 0.05ms for the queries on the database. So every webpage displaying stuff is only taking up a few seconds. Displaying graphs is a whole other matter.
I checked why those graphs took so long. It turned out the queries weren't the issue (even when they had difficult aggregration in them). The time it took to create the graph from the data returned from the query was by far the largest. So I looked for a solution and found FlotGraph (http://code.google.com/p/flot/). This is a graph drawing library which draws graph at the clientside (javascript). I tried how much this would help and ..... it now only takes 3 seconds to refresh the page and get a graph visible for the user!!! This means aggregrations over 1.8 million rows only takes up to three seconds now!!! I'm really glad with the results of these improvements.

The next steps now are:
- Easy defining your actions on certain events.
- More graphs
- More device support
- Starting and stopping drivers and refreshing settings from the webapp.
- Releasing it as commercial software!?!?! Would anybody be interested?

In the next few weeks I will post some screenshots of the system.

[dali]

I think you've definitely got the right approach. A modular system with separate daemons is brilliant.

I guess I'd probably pay for such a software, but I'd much rather contribute with code in an open source project.
Have you considered releasing your software as open source (or semi open)?

Php with web gui makes this a potential replacement for e.g. Misterhouse.

[martijn]

I also would be interested in such a project.
Like Dali, I would rather work on an open source project then buying a PHP project ( let's face it, building something is half the fun )

But if you're going commercial, and the price is ok, I'd probably buy it just to see how you handle certain parts of the program.


For now, I'll leave with 1 single question:
Since PHP doesnt have any multi threading; How do you constantly read from the Serial port while also writing to it?

I've been thinking about this part for a few days now but I cant seem to find the right way (this is also because my current zwave controller has problems with being read as a serial device and this is keeping me from testing it )


Greets,
Martijn

[airox]

I haven't considered open sourcing the software. Have to think about this.

My zwave driver has an internal queue of messages (just a php array). This is the queue from which the driver picks up messages, sends them out and waits for the acknowledgements. When it receives the acknowledgements it deletes the message from the queue and sends out the next message. Before sending out a message it always check if there isn't something received by the driver. It handles this and puts this event in a System V IPC message queue (interprocess communication with a serverDaemon which receives events from all the drivers, bluetooth, zwave and rfxcom and others). When nothing is received and the message queue is empty it checks if there is something send to the zwave driver (another system V IPC message queue) and puts this onto the internal queue. This is basically how it works.

Another thing I've been working on is integrating the Motion opensource project into the home automation software. When motion is detected on a camera it sends a message to the home automation software where events can be added to this signal to do some stuff. I also integrated the possibility for the home automation system to put cameras in active or pause mode. This all results in the capability to put the cameras in active mode when turning on the security of the house. It then starts recording motions and sends signals of motion to the home automation software. This then activates the alarm mode which stores the movie and sends out push notifications to my mobile.

So instead of only pictures logged from the cameras I now got movies!

[airox]

Hi guys,

As promised, some new screenshots of the system.

http://fotoboek.vandulmen.net/domotica/v2/

You will see the new graph creation library being used and see what information I am getting out of the system right now. It also shows the motion camera support it currently has as well an overview of all the devices I currently have in my house.

I'm currently busy with:
- Making drivers start and stop through the web interface (as you can see it's already done for rfxcom)
- Creating dashboards with graphs through the web interface. So besides the already available graphs you can define your own graphs and add them to a dashboard.
- Scenario based eventing (capable of triggering when action a, b, c have happened in the last ten minutes right after each other)
- Google latitude support for monitoring my position and acting upon it (and creating a history of where I have been).
- Finishing the support for bluetooth devices.

I will post screenshots of the dashboard creation interface in the following weeks as well as google latitude support. The scenario based eventing is still in the design phase.. not really sure howto implement this without having it have a huge impact on performance.

[RDNZL]

Familiar icons

What is that, when I was looking for descent HA there was none, and after I started DomotiGa, everyone seems started building their own..

[airox]

Yeah, the famfamfam icon set really rocks. I use it quite a lot in projects. I did looked at Domotiga when I was diving into the home automation, but decided I want a more web based approach. That's why my software doesn't have a GUI desktop application. And the fun is also in the creation of the software

Maybe we should talk sometime about the concepts which are the same overall. I notice that a lot of software is concentrated on integrating a lot of protocols, but not focused on integrating all those components into getting something which is really helpfull in the daily life. When would our software be usefull for non-technical people.

Do you have a instant messenger address on which I can contact you?

I see you added Google Latitude support in the last few days btw, nice! Didn't know you had plans for integrating it.

[RDNZL]

I started coding the protocols first to test my skills and gambas possibilities.

Many people like the current GUI for simplicity, if you just want to setup / log something.
But a nice html frontend would be nice... still on todo list.

You should have known about my Latitude plans domoticaforum.eu/viewtopic.php?f=47&amp ... ude#p41494
At first I wanted to code the real latitude API, so you don't have to expose your badge, but native oauth support is not easy yet. So I went the easy route for now.

I will sent you a PM with my gtalk address later.

[airox]

I incorporated the support for Latitude including the OAuth support. So you don't have to expose the badge.
I am fetching the location every 20 seconds (Google has a limit of 1 million requests a day, so this should be fine).

Also, geonames.org has a nice database you can use (or their webservices) to do reverse geocoding. I took the database and imported it into MySQL.
Now I can just query the database to get the closest city. Trackers now have two values on which triggers and conditions can be set.
Location, Closest city. You can define a trigger when when the tracker is near a certain specified location.

Some other stuff done:
- Live following a tracker on a Google Maps graph.
- Manual trackers which do not use Google Latitude (will be used for a public available tracker which publishes the current location to twitter)
- You have to point on a google map to tell the system where you are
- Or you can use the w3c geolocation API (makes it possible to hail a link in your phone to update the public tracker)
- History overview of previous locations

Screenshots coming soon ...

@RDNZL:
checkout this demo for more info about the OAuth process: https://github.com/billglover/Google-Latitude-Demo
and this link: http://forum.geonames.org/gforum/posts/list/30/732.page
for information about importing the geonames database and modifying the data so you can use MySQL's spatial extensions.

[airox]

Here are the screenshots:

http://fotoboek.vandulmen.net/domotica/v2_update/

In which you can see the possibility of creating graphs on the fly. I also perfected my XML import and export possibility. I can now define my events, flags and scenes in an XML file and import it. Creating events is something like this in XML:

Code: Select all

<conditionBasedEvent id="OVERLOOP_LIGHTS_WHEN_NECESSARY" name="Zet lichten overloop aan wanneer nodig">
	<conditions>
		<equal>
			<flagValue id="SECURITY">uit</flagValue>
			<flagValue id="PRESENCE">ja</flagValue>
			<motion id="BEWEGINGSENSOR_OVERLOOP">1</motion>
		</equal>
		<conditions logical="AND">
			<greater>
				<time>19:00</time>
			</greater>
			<less>
				<time>23:59</time>
			</less>
		</conditions>
	</conditions>
	<actionsWhenTrue>
		<switchOn device="OVERLOOPLAMP" />
	</actionsWhenTrue>
	<actionsWhenFalse>
		<switchOff device="OVERLOOPLAMP" />
	</actionsWhenFalse>
</conditionBasedEvent>

One other thing I have done is the possibility to use device values (from all types of devices) in actions. The use case here would be for example email me the temperature or current usage when asked for. Previously this wasn't possible because actions couldn't use values in their definitions. This would also make it possible to tweet my current location (city) when the city changes.

The definition in XML is like this for a scene which can be called.

Code: Select all

<scene id="WHERE_IS_GIJS" name="Where is Gijs?">
<actions>
<speak text="Gijs is in the city called ${device.40.city}" />
</actions>
</scene>

This gives a lot of flexibility. A list of other small things I added:

It's now possible to categorize devices into the following (dutch) categories: Verlichting, Klimaat, Energie, Apparatuur, Beveiliging and Overig. Devices could already be allocated to a location (room). All this is needed for the extension of the iPad application I'm thinking about. I needed a basic and simple categorization of devices for navigating in the iPad application.

What's next?
- iPad application version 2
- A complete revision for the events I defined for my home. Adding a lot more events based on location, energy usage.
- Installation of my iPad to the wall so it can show the application fulltime.
- Dashboard creation by defining some graphs.
- MySQL partitioning support.... or archiving old values somehow (currently the device values table is 2.3 million rows, but still fast!

[lxx_33]

Very slick design. You make it sounds very easy, while I'm having lots of problems make a led blink with a arduino.

One word for you RESPECT.

[airox]

Well I am avoiding all the hardware development. This is where my experience is lacking. So I use a lot of standard stuff which you can buy (z-wave and klikaanklikuit devices).
If I had to design hardware components and need to do some soldering then this would be hard for me too.

One other thing I'm doing for the graph dashboards is making it possible to show the time it took between the door open and door close event. This concept would also help in measuring the time between light and dark switches, movement and no movement. This would create possibilities to see how long you are in a certain room in the house on average. Some stuff which is nice to measure, not really sure what it's worth though

[mishoboss]

airox, respect for this effort! I'm a PHP developer too and had something similar in my mind. However I'm still stuck on the hardware part. Could you point me to the devices you use for lights, shutters and heating? It's not a problem for me to use wired solutions too, if they are cheaper.

Some words on the database rows. I guess these 2.3 million rows are just sensor data used for statistics. Why not just archive the data by months or days? For example in the end of the day run a cron that puts summarized data of a sensor in another DB table and delete all the records. This another table may have something like these fields: id, sensor name, date, data0, data1, data2, data3... data23, dataday. data0 to data23 is the average sensor data for every hour of that day and data day is the calculated average data for the day. One record per day per sensor... nothing more is needed. I did something similar for an invoicing software 6 years ago and still works fine and fast!