[WIP/Released] P1 Data injector

[timkoers]

I don't think I need to say anything!

SUCCESS.png (45.6 KiB) Viewed 14853 times

The gas usage however, is from the previous owner. How do I remove that from the readouts?
Also the current usage is not working as expected yet.

EDIT: the smart meter readout was stopped without me noticing it. It sent the retained MQTT message to the Toon, which was good, but not up to date.
I restarted the readout and the current usage popped up!

SUCCESS 2.png (39.75 KiB) Viewed 14852 times

[TheHogNL]

Great job! Now a summary please

[timkoers]

Great job! Now a summary please

I'll make an installation guide, complete with code and schematics!

Will take a while though to make it.

[timkoers]

Great job! Now a summary please

I've edited the first post with the details of the project!

[TheHogNL]

Important notice!
Please do not use this piece of software to fake your energy bills.

This isn't possible The toon data (which now can be manipulated using your simulator) is just for your viewing pleasure and not for billing. For billing they only trust your real meter.

[timkoers]

Important notice!
Please do not use this piece of software to fake your energy bills.

This isn't possible The toon data (which now can be manipulated using your simulator) is just for your viewing pleasure and not for billing. For billing they only trust your real meter.

Pfew, that is a big relief for me hahaha

[warp]

Hi all,

this thread is dead for 1 year, but maybe you can help me.
I'm trying to achieve more or less the same as what timkoers did : inject data on the P1 port.
The reason for doing so is that I have a smartmeter, with P1 port, but here in Luxembourg the data is encrypted.
So the toon would be unable to read it.
I have a RPi reading the data, unencrypting it, and sending to another RPi running domoticz.

First of all the USB-to-serial cable is a specific one that inverts the level.
It works fine for read, and I assume I can use the same for wirting, connecting it to the zwave appliance.
-> maybe this first assumption is wrong, but I don't know how to check if this cable works for both read and write.
I bought it here : https://webshop.cedel.nl/Slimme-meter-kabel-P1-naar-USB

I configured the toon to work with a P1 smartmeter.
Nothing specific here, I have the green flag on the meter.

Then I used the code provided by timkoers. (but adapted for now in order to not wait for anything from mqtt. I just hardcoded a copy of a data received from my own smartmeter)
It seems I never receive the rts from the toon.
Or actually maybe the opposite : it seems rts is always high since the my software sends the data every seconds. There's no wait for the toon.

Code: Select all

#!/usr/bin/env python
#
import re
import os
import serial
from serial import Serial
import time
import datetime
from time import sleep # import the time function from the sleep library
import crcmod.predefined
from parse import parse


# Program variables
# The true telegram ends with an exclamation mark after a CR/LF
pattern = re.compile(b'\r\n(?=!)')
#pattern = re.compile(b'\n(?=!)')
# According to the DSMR spec, we need to check a CRC16
crc16 = crcmod.predefined.mkPredefinedCrcFun('crc16')
# Create an empty telegram
checksum_found = False
good_checksum = False
checksum = 0


# define ttyUSB0 as serial
#Serial port configuration
ser = serial.Serial()
ser.baudrate = 115200
ser.bytesize = serial.EIGHTBITS
ser.parity = serial.PARITY_NONE
ser.stopbits = serial.STOPBITS_ONE
ser.xonxoff = 0
ser.rtscts = 0
ser.timeout = 1
ser.write_timeout = 1.0
ser.port = "/dev/ttyUSB0"
ser.open()

def calculateCRC(injectedTelegram):
    calculated_checksum = 0
    for m in pattern.finditer(injectedTelegram):
      # Remove the exclamation mark from the checksum,
      # and make an integer out of it.
      # given_checksum = "0x" + int(telegram[m.end() + 1:].decode('ascii'), 16)
      # The exclamation mark is also part of the text to be CRC16'd
      calculated_checksum = crc16(injectedTelegram[:m.end() + 1])
    return calculated_checksum


def fillTelegram() :
    timeString = "{:0>2d}".format(datetime.datetime.now().year-2000) + "{:0>2d}".format(datetime.datetime.now().month) + "{:0>2d}".format(datetime.datetime.now().day) + "{:0>2d}".format(datetime.datetime.now().hour) + "{:0>2d}".format(datetime.datetime.now().minute) + "{:0>2d}".format(datetime.datetime.now().second) + "S" if time.localtime().tm_isdst is 1 else "W"
    injectedTelegram =  "/Lux5\253663629_D\r\n\r\n\
1-3:0.2.8(42)\r\n\
0-0:1.0.0(190916204632S)\r\n\
0-0:42.0.0(53414731303330373030333035353037)\r\n\
1-0:1.8.0(001157.757*kWh)\r\n\
1-0:2.8.0(000000.003*kWh)\r\n\
1-0:3.8.0(000005.312*kvarh)\r\n\
1-0:4.8.0(000420.219*kvarh)\r\n\
1-0:1.7.0(01.190*kW)\r\n\
1-0:2.7.0(00.000*kW)\r\n\
1-0:3.7.0(00.154)\r\n\
1-0:4.7.0(00.057)\r\n\
0-0:17.0.0(77.376)\r\n\
0-0:96.3.10(1)\r\n\
0-0:96.7.21(00013)\r\n\
1-0:32.32.0(00002)\r\n\
1-0:52.32.0(00002)\r\n\
1-0:72.32.0(00002)\r\n\
1-0:32.36.0(00000)\r\n\
1-0:52.36.0(00000)\r\n\
1-0:72.36.0(00000)\r\n\
0-0:96.13.0()\r\n\
0-0:96.13.2()\r\n\
0-0:96.13.3()\r\n\
0-0:96.13.4()\r\n\
0-0:96.13.5()\r\n\
1-0:31.7.0(000*A)\r\n\
1-0:51.7.0(000*A)\r\n\
1-0:71.7.0(004*A)\r\n\
!71B3"

#    checksum = calculateCRC(injectedTelegram)
#    injectedTelegram += format(checksum,"04X") + "\r\n"
    return injectedTelegram


######################
# *** Start here *** #
######################
receivedTelegram = False
lastTelegram = datetime.datetime.now()
shouldWait = True
msg = ""

while True:
    if shouldWait:
        # Wait for the Toon to stop requesting data
        shouldWait = not ser.rts
        msg = fillTelegram()
        print "Waiting..."
    else:
        if ser.rts:
            msg = fillTelegram()
            if abs(datetime.datetime.now().second - lastTelegram.second) >= 1:
               if True:
                   print "Sending " + msg
                   pass
               try:
                   ser.write(msg)
                   ser.flush()
               except:
                   print("Exception!")
                   pass

               lastTelegram = datetime.datetime.now()
               shouldWait = True
            else:
                pass

To start hdrv_p1 -vvvv , I first had to edit inittab since that process is respawning autoöatically.
So I was not able to kill it. I guess having 2 hdrv_p1 processes at the same time is not a good idea.
In the end it works, and only one process is running, the one I start manually.
From another ssh session I can see the following :

Code: Select all

eneco-001-151219:~# ps | grep hdrv
  693 root     24424 S    {hdrv_hue} HCBv2 hdrv_hue       [OK] [S1|N1]
  698 root     39996 S    {hdrv_zwave} HCBv2 hdrv_zwave     [OK] [F5BDD5D5 SIS 3.67] [S1|N1]
 5127 root     22336 S    {hdrv_p1} HCBv2 hdrv_p1        [OK] [C:OK E:COM (P1) ] [S99|N0]
 5171 root      3236 S    grep hdrv

The only difference between the one I start from the command line, and the, let's say, original one, it that in the end I have S99|N0 instead of S1|N1
Here is the "original" one :

Code: Select all

eneco-001-151219:~# ps | grep hdrv
  693 root     24424 S    {hdrv_hue} HCBv2 hdrv_hue       [OK] [S1|N1]
  698 root     39996 S    {hdrv_zwave} HCBv2 hdrv_zwave     [OK] [F5BDD5D5 SIS 3.67] [S1|N1]
 3772 root     22336 S    {hdrv_p1} HCBv2 hdrv_p1        [OK] [C:OK E:COM (P1) ] [S1|N1]
 3839 root      3236 S    grep hdrv

From the logs of hdrv_p1 I can see something that looks strange (but maybe not an issue ...) :

Code: Select all

[hdrv_p1:/tmp/sonar-scanner/jenkins/workspace/qmf_hdrv_p1-Pipeline_master/application/hdrv_p1/src/hdrv_p1.c@initVars():1328]ERROR: No profile 'Metering' found
[hdrv_p1]initting type:usageInfo, intAddr:elec_delivered_total.304d8722-58c9-49b9-8342-2214372ef544, zombie:0, orphan:0
[hdrv_p1:/tmp/sonar-scanner/jenkins/workspace/qmf_hdrv_p1-Pipeline_master/application/hdrv_p1/src/hdrv_p1.c@initVars():1328]ERROR: No profile 'Metering' found
[hdrv_p1]initting type:usageInfo, intAddr:elec_delivered_nt.304d8722-58c9-49b9-8342-2214372ef544, zombie:0, orphan:0
[hdrv_p1:/tmp/sonar-scanner/jenkins/workspace/qmf_hdrv_p1-Pipeline_master/application/hdrv_p1/src/hdrv_p1.c@initVars():1328]ERROR: No profile 'Metering' found
[hdrv_p1]initting type:usageInfo, intAddr:elec_received_nt.304d8722-58c9-49b9-8342-2214372ef544, zombie:0, orphan:0
[hdrv_p1:/tmp/sonar-scanner/jenkins/workspace/qmf_hdrv_p1-Pipeline_master/application/hdrv_p1/src/hdrv_p1.c@initVars():1328]ERROR: No profile 'Metering' found
[hdrv_p1]initting type:usageInfo, intAddr:elec_delivered_lt.304d8722-58c9-49b9-8342-2214372ef544, zombie:0, orphan:0
[hdrv_p1:/tmp/sonar-scanner/jenkins/workspace/qmf_hdrv_p1-Pipeline_master/application/hdrv_p1/src/hdrv_p1.c@initVars():1328]ERROR: No profile 'Metering' found
[hdrv_p1]initting type:usageInfo, intAddr:elec_received_lt.304d8722-58c9-49b9-8342-2214372ef544, zombie:0, orphan:0
[hdrv_p1:/tmp/sonar-scanner/jenkins/workspace/qmf_hdrv_p1-Pipeline_master/application/hdrv_p1/src/hdrv_p1.c@initVars():1328]ERROR: No profile 'Metering' found

In the end it behaves like if the cable is not connected at all.....

[warp]

It took some time for my post to be approved (not criticizing, just explaining why I have more info), and I continued to search.
I focused on the cable itself. Let's call it the "smartmeter P1 cable".
I'm now sure it works with read only. Only 3 wires are connected : GND, RTS and RX. No wire for TX. So obviously my previous test could not work.

I found another serial-to-USB cable, and connected both together.
With miniterm I'm now able to validate that both cables can communicate.
I simply connected one to the other, opened 2 sessions on each USB port, and when writing on one session, I see the characters on the other session.

So I use now the second cable, that is able to write. (let's call it the "write cable")
But the result is strictly the same.
Now I'm wondering if it doesn't come from the inverted signals.
The "smartmeter P1 cable" has the inverted signals (since it works with the smartmeter). So the first guess would be that the "write cable" has the inverted signals as well.
But I've seen that some cables are able to automatically adapt the signal.
So the second guess is that the "smartmeter P1 cable" detects and inverts when connected to the smartmeter, but doesn't inverts when connected to the "write cable".
.....don't know if all this makes sense....

Finally, how do I know what the RJ11 connector on the zwave applicance is waiting for ?
Does it wait for inverted signals ? Is it able to adapt ?
What about the pinout ?
I connected 3 pins : RTS (hardware connected to CTS), GND, and TX.
I guess it shoud work.....

[TheHogNL]

Finally, how do I know what the RJ11 connector on the zwave applicance is waiting for ?
Does it wait for inverted signals ? Is it able to adapt ?
What about the pinout ?
I connected 3 pins : RTS (hardware connected to CTS), GND, and TX.
I guess it shoud work.....

It will most probably not be able to invert itself. But if you look at a official toon p1 cable you notice that it is a flat cable. That means a pin-rollover is occuring. You may be better of with normal rj11 cable extensions plug (https://www.netwerkkabelshop.nl/telefoo ... lstuk-rj11). Put the official toon p1 cable in one end and your solution in the other end.

[warp]

Finally, how do I know what the RJ11 connector on the zwave applicance is waiting for ?
Does it wait for inverted signals ? Is it able to adapt ?
What about the pinout ?
I connected 3 pins : RTS (hardware connected to CTS), GND, and TX.
I guess it shoud work.....

It will most probably not be able to invert itself. But if you look at a official toon p1 cable you notice that it is a flat cable. That means a pin-rollover is occuring. You may be better of with normal rj11 cable extensions plug (https://www.netwerkkabelshop.nl/telefoo ... lstuk-rj11). Put the official toon p1 cable in one end and your solution in the other end.

Hi TheHogNL,
the cable I have in the box is not a flat cable, but a standard phone cable (pin1 connected to pin1, pin2 to pin2 ....), or straight cable.
My "write cable" is kind of homemade, with a female RJ45 in the end. I can easily re-wire it.
So if this is indeed a flat cable, it will take some minutes to make a new test.
I'll post an update on that.

[warp]

It seems the previous wiring was correct (or at least "less wrong"),
because I wired according to the use of a flat cable, but I got an error in the counters sections saying "No sensor found".
Then I re-wired it back to what I first did, and the message is now : "Sensors are calibrated" (These are my traductions since I put my Toon in french) with a green flag on the counter.

I could also read the following in the hdrv_p1 logs some seconds after :

Code: Select all

[hdrv_p1]Got burnerInfo 0
[hcom]Received HBXT_ACTION_INVOKE from eneco-001-151219:hcb_watchdog to eneco-001-151219:hdrv_p1/specific1: n=ping
[hdrv]Received PING from 'eneco-001-151219:hcb_watchdog' as 'eneco-001-151219:hdrv_p1', sending PONG back...
[hcom]Sending HBXT_ACTION_RESPONSE from eneco-001-151219:hdrv_p1 to eneco-001-151219:hcb_watchdog/specific1: n=pingResponse version=
[hcom]Received HBXT_ACTION_INVOKE from eneco-001-151219:qt-gui to eneco-001-151219:hdrv_p1/specific1: n=GetSensorConfiguration
[hcom]Sending HBXT_ACTION_RESPONSE from eneco-001-151219:hdrv_p1 to eneco-001-151219:qt-gui/specific1: n=GetSensorConfigurationResponse device=
[hcom]Received HBXT_ACTION_INVOKE from eneco-001-151219:qt-gui to eneco-001-151219:hdrv_p1/specific1: n=GetMeterConfiguration
[hcom]Sending HBXT_ACTION_RESPONSE from eneco-001-151219:hdrv_p1 to eneco-001-151219:qt-gui/specific1: n=GetMeterConfigurationResponse device=
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to 018860cb-cafb-4703-b30f-52f0dd7ea679/MeterTableMonitor: n=StatusDepthGet requestId=6284-66 timeout=10
[hcom]Received NOTIFY from 018860cb-cafb-4703-b30f-52f0dd7ea679/SensorStatusMeter: CurrentSensorStatus=COMMISSIONING
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:hcb_log/Logger: n=LogMessage facility=1 timestampGenerated=2019-09-19 13:06:16 +0000 timestampExpire=2019-10-19 13:06:16 +0000 categoryId=10 entryType=1 appId=24 objectType=8 objectId=hdrv_p1 logText=setMeterStatus for elec_delivered_nt.304d8722-58c9-49b9-8342-2214372ef544 update DISABLED -> COMMISSIONING logTextF=
[hcom]Sending NOTIFY from 300da888-2131-4dcc-b85e-a1fb5357749c/Metering: SensorStatus=COMMISSIONING
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:hcb_log/Logger: n=LogMessage facility=1 timestampGenerated=2019-09-19 13:06:16 +0000 timestampExpire=2019-10-19 13:06:16 +0000 categoryId=10 entryType=1 appId=24 objectType=8 objectId=hdrv_p1 logText=setMeterStatus for elec_delivered_lt.304d8722-58c9-49b9-8342-2214372ef544 update DISABLED -> COMMISSIONING logTextF=
[hcom]Sending NOTIFY from 77fa4607-1cac-4d65-859f-5bf6ad03213d/Metering: SensorStatus=COMMISSIONING
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:hcb_log/Logger: n=LogMessage facility=1 timestampGenerated=2019-09-19 13:06:17 +0000 timestampExpire=2019-10-19 13:06:17 +0000 categoryId=10 entryType=1 appId=24 objectType=8 objectId=hdrv_p1 logText=setMeterStatus for elec_received_lt.304d8722-58c9-49b9-8342-2214372ef544 update DISABLED -> COMMISSIONING logTextF=
[hcom]Sending NOTIFY from a46d60b5-afc9-458c-82d6-73b530bc6a56/Metering: SensorStatus=COMMISSIONING
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:hcb_log/Logger: n=LogMessage facility=1 timestampGenerated=2019-09-19 13:06:17 +0000 timestampExpire=2019-10-19 13:06:17 +0000 categoryId=10 entryType=1 appId=24 objectType=8 objectId=hdrv_p1 logText=setMeterStatus for elec_received_nt.304d8722-58c9-49b9-8342-2214372ef544 update DISABLED -> COMMISSIONING logTextF=
[hcom]Sending NOTIFY from 669ea8c2-1f2a-4048-b130-52ed56087038/Metering: SensorStatus=COMMISSIONING
[hdrv]DS: datasetUpdate connectedInfo
[hdrv]DS: Update for dataSet connectedInfo
[hdrv]DS: Send full connectedInfo update to eneco-001-151219:happ_pwrusage
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:happ_pwrusage/specific1: n=UpdateDataSet connectedInfo=
[hdrv]DS: Send full connectedInfo update to eneco-001-151219:qt-gui
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:qt-gui/specific1: n=UpdateDataSet connectedInfo=
[hdrv]DS: datasetUpdate usageDevicesInfo
[hdrv_p1]Clearing notification for error in meter module sensor
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:happ_usermsg/Notification: n=DeleteNotification type=error subType=energymeter
[hdrv]DS: Update for dataSet usageDevicesInfo
[hdrv]DS: Send full usageDevicesInfo update to eneco-001-151219:happ_kpi
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:happ_kpi/specific1: n=UpdateDataSet usageDevicesInfo=
[hdrv]DS: Send full usageDevicesInfo update to eneco-001-151219:happ_pwrusage
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:happ_pwrusage/specific1: n=UpdateDataSet usageDevicesInfo=
[hdrv]DS: Send full usageDevicesInfo update to eneco-001-151219:qt-gui
[hcom]Sending HBXT_ACTION_INVOKE from eneco-001-151219:hdrv_p1 to eneco-001-151219:qt-gui/specific1: n=UpdateDataSet usageDevicesInfo=
[hcom]Received HBXT_ACTION_RESPONSE from 018860cb-cafb-4703-b30f-52f0dd7ea679 to eneco-001-151219:hdrv_p1/MeterTableMonitor: n=StatusDepthGetResponse requestId=6284-66 repToFollow=0x00 opStatus1=0x00 opStatus2=0x00 opStatus3=0x10
[hcom]Received HBXT_ACTION_RESPONSE from eneco-001-151219:happ_usermsg to eneco-001-151219:hdrv_p1/Notification: n=DeleteNotificationResponse response=404 reason=no notification found
[hcom]Received HBXT_ACTION_INVOKE from eneco-001-151219:happ_thermstat to eneco-001-151219:hdrv_p1/specific1: n=UpdateDataSet thermostatInfo=

But except from that, nothing more happens.

[timkoers]

Then I used the code provided by timkoers. (but adapted for now in order to not wait for anything from mqtt. I just hardcoded a copy of a data received from my own smartmeter)

Just for reference

You can find the code here: https://gitlab.com/timkoers/p1-simulator

I could also read the following in the hdrv_p1 logs some seconds after :

But except from that, nothing more happens.

My hdrv_p1 logs showed the following on the first succesfull transmission:

https://pastebin.com/VrSGF78X

[warp]

Hi Timkoers,

I’be seen that git repo.
I just missed the schematics.

I have ordered a ftdi cable so that I will be able to revert TX.

By the way : the link you posted on pastebin seems to be a private one.
It’s not publicly accessible.

[timkoers]

Hi Timkoers,
By the way : the link you posted on pastebin seems to be a private one.
It’s not publicly accessible.

I've made it public (unlisted), so you can now see it!

Hi Timkoers,

I have ordered a ftdi cable so that I will be able to revert TX.

Please be aware that the ftdi cable does not revert TX and that you don't want to revert TX.

You need a hex inverter to invert theTX signal as you'd want to have the signal inverted (by the DSMR standard).

[warp]

Hi,
the link is now indeed public. I will take time to read it.

Concerning reverting or inverting data, what I wanted to say is indeed "invert".
This is something provided by the ftdi chip itself.
On windows this can be changed using the software ft_prog.

Now the question is : is it really what is needed to send data to the toon ?
I guess yes. This can be seen on several websites on the internet related to reading P1 data.
So I can imagine writing must be done the same way.
Furthermore my read cable (that is plugged to the smartmeter), which works correctly,
has this same configuration (ft_prog shows the signal is inverted) for the RX pin.

....anyway, it's not 10€ for the cable that will ruin me in case it doesn't work.
Let' see in 3-4 days.
Thanks for your remarks.