Long Term Visualization of Heating Behavior?

[tzieg]

Hi,

Now, as OTGW for my Brötje WGB 2 is operating stable, I can start to analyze the collected data with the goal to optimize the parameterization of the heating system.
What's the best way to visualize the behavior for e.g. a week or even a month?
The otmonitor-Graph is about 2h only. Are there tools which take the otlog-yyyymmdd.txt files as input for offline analysis?

Thanks,
Thomas

[marcelr]

Are there tools which take the otlog-yyyymmdd.txt files as input for offline analysis?

I've written a bunch of scripts for matlab, and a quick and dirty parser for the log files, in C.

The C-code rips the logs apart and stores them per OT ID in a .csv file like this:
<timestamp>,<ID>,<value>
so, e.g.,:

1452466808.102322,1,6.00

The matlab scripts read these files and turn them into graphs, like
https://domoticaforum.eu/viewtopic.php? ... 664#p77798
Although over time I've added a lot more info.
Not sure if that's what you are looking for.

[tzieg]

Hi,

yes, this is exactly what I'm looking for.

Is it possible to share the parser and the matlab scripts?

Thanks,
Thomas

[marcelr]

Hmm.. forgot to mention: The c-code is specific for Linux (Darwin on MacOSX may work as well).
Need to weed out the m-scripts before sending them anywhere. Typical quick and dirty works-for-me code.
Do you still want it?

[hvxl]

If you have OTmonitor running, you can configure the datalog to save a record of the items you are interested in at regular intervals. Then the only task left is to make a graph out of that.

[Bororo]

If cloud is not an issue I suggest to use built in Thingspeak support.

[tzieg]

Hmm.. forgot to mention: The c-code is specific for Linux (Darwin on MacOSX may work as well).
Need to weed out the m-scripts before sending them anywhere. Typical quick and dirty works-for-me code.
Do you still want it?

Sorry for the late response, I'm on a biz trip: Linux is absolutely fine, so the answer is yes!

Thanks,
Thomas

[marcelr]

Ran some test with the matlab code yesterday. Dead slow.
Will need to look into that (and is quite useless without a --bloody expensive-- matlab license).

About the c-code, here you go.

Build with

Code: Select all

gcc parse_otlog.c -o parse_otlog

The translation from f8.8 to double may be a bit off, looking at the code now. Give it a shot, and please report any errors.
If you need any other OT IDs you can just add them in the right part of the switch{} according to the type they have.

Code: Select all

/*
 * parse_otlog.c
 *
 * function to read and convert otmonitor data to UTC timestamped OpenTherm 
 * data.
 *  
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#include <sys/stat.h>
#include <sys/types.h>
#include <sys/errno.h>

#define MAX_LINE_LENGTH  127
#define ARRAY_LENGTH     128*1024


void   parse_otlog( char* );
void   usage( void );
double OT_float_to_double( unsigned short );

int main( int argc, char **argv )
{

  char *filename;
  
  if (argc <= 1)
    {
      usage();
      exit(0);
    }
  
  filename = argv[1];
  
  parse_otlog( filename );

  return 0;
}



void usage( void )
{
  printf("usage:\nparse_otlog <filename>\n" );
  
}



void parse_otlog( char* file_in )  
{
  FILE* fp;
  char *line;
  int max_len = MAX_LINE_LENGTH;
  int cnt;
  int valid;
  int year, mon, day, hour, min, sec, usec;
  int dum, id, val;

  int ids[256] = {0};
  int i, j, k;

  struct tm* tm;
  time_t utc;
  double timestamp;

  double ot_t[ARRAY_LENGTH];
  int ot_id[ARRAY_LENGTH];
  unsigned short ot_val[ARRAY_LENGTH];

  /* parsed data */
  
  double         dval;
  unsigned char  bitval[16];
  unsigned short usval;
  char           cval[2];
  unsigned char  ucval[2];
  unsigned short OT_payload;
  char dir_out[128];
  
  char file_out[280];
  FILE* fp_out;
  int status;
  
  fp = fopen( file_in, "r" );

  /* read date from filename */

  sscanf( file_in, "otlog-%4d%2d%2d.txt", &year, &mon, &day );
  //  printf("year: %d, mon: %d, day: %d\n", year, mon, day );

  
  tm = malloc( sizeof( struct tm ) );

  /* partly fill time struct */

  tm->tm_year = year-1900;
  tm->tm_mon  = mon-1;
  tm->tm_mday = day;
 
  line = malloc( (MAX_LINE_LENGTH + 1) * sizeof( char ) );

  cnt = 0;
  valid = 0;

  /* read the file, line by line, and store in a series of arrays */

  while ( line = fgets( line, max_len, fp ) )
    {
      cnt++;
      
      if ( strstr( line, "Unk-DataId" ) || strstr( line, "Read-Ack" ) ||
	   strstr( line, "Write-Ack" ) )
	{
	  /* parse, complete time struct, transform to UTC timestamp */

	  sscanf( line, "%02d:%02d:%02d.%06d  %03X%02X%04X", &hour, &min, &sec,
		  &usec, &dum, &id, &val );
	  tm->tm_hour = hour;
	  tm->tm_min  = min;
	  tm->tm_sec  = sec;

	  utc = mktime( tm );
	  //	  printf("isdst  : %2d\n", tm->tm_isdst);
	  timestamp     = (double)utc + 1e-6 * (double)usec;

	  ot_t[valid]   = timestamp;
	  ot_id[valid]  = id;
	  ot_val[valid] = (unsigned short)val;

	  ids[id] += 1;

	  //	  printf("%17.6f, %3d, %04x, %s", timestamp, id, val, line );
	  //printf("%17.6f, %3d, %04x\n", timestamp, id, val);

	  valid++;
	}
    }

  
  for ( i = 0; i < 256; i++ )
    {
      if ( ids[i] != 0 )
	{
	  // printf("ids[%3d] = %5d\n", i, ids[i] );
	  sprintf( dir_out, "./%04d/", year );
	  status = mkdir( dir_out, S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH );
		 if ( (status != 0) & (errno != EEXIST) )
		   {
		   printf( "ERROR: failed to create directory %s\n", dir_out );
		   exit(-1);
		   }
	  sprintf( dir_out, "./%04d/OT%03d/", year, i );
	  
	  sprintf( file_out, "%sOT%03d-%04d%02d%02d.csv", dir_out, i, year,
		   mon, day );
	  status = mkdir( dir_out, S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH );
		 if ( (status != 0) & (errno != EEXIST) )
		   {
		   printf( "ERROR: failed to create directory %s\n", dir_out );
		   exit(-1);
		   }
	  
	  fp_out = fopen( file_out, "w" );

	  for ( j = 0; j < valid; j++ )
	    {
	      if ( ot_id[j] == i )
		{
		  OT_payload = ot_val[j];
		  
		  switch ( ot_id[j] )
		    {
		      /* bit fields */
		      
		    case 0: /* status byte: high byte: master, low byte: slave */
		    case 2: /* high byte: master config, low byte: master ID   */
		    case 3: /* high byte: slave config, low byte: slave ID     */
		    case 5: /* high byte: app-specific fault flags, 
			       low byte: OEM fault code  */
		    case 6: /* high byte: remote par enable flag, 
			       low byte: remote par r/w flag */
		      
		      bitval[ 0] =   OT_payload & 0x0001;
		      bitval[ 1] = ( OT_payload & 0x0002 ) >> 1;
		      bitval[ 2] = ( OT_payload & 0x0004 ) >> 2;
		      bitval[ 3] = ( OT_payload & 0x0008 ) >> 3;
		      bitval[ 4] = ( OT_payload & 0x0010 ) >> 4;
		      bitval[ 5] = ( OT_payload & 0x0020 ) >> 5;
		      bitval[ 6] = ( OT_payload & 0x0040 ) >> 6;
		      bitval[ 7] = ( OT_payload & 0x0080 ) >> 7;
		      bitval[ 8] = ( OT_payload & 0x0100 ) >> 8;
		      bitval[ 9] = ( OT_payload & 0x0200 ) >> 9;
		      bitval[10] = ( OT_payload & 0x0400 ) >> 10;
		      bitval[11] = ( OT_payload & 0x0800 ) >> 11;
		      bitval[12] = ( OT_payload & 0x1000 ) >> 12;
		      bitval[13] = ( OT_payload & 0x2000 ) >> 13;
		      bitval[14] = ( OT_payload & 0x4000 ) >> 14;
		      bitval[15] = ( OT_payload & 0x8000 ) >> 15;
		      
		      fprintf( fp_out, "%17.6f,%d,", ot_t[j], ot_id[j] );
		      for ( k = 15; k >= 0; k-- )
			{
			  fprintf(fp_out, "%d", bitval[k] );
			  if ( k > 0 )
			    fprintf( fp_out, "," );
			}
		      fprintf( fp_out, "\n" );
      		      break;
		      
		    case 1:  /* boiler control setpoint [deg C]       */
		    case 9:  /* remote override room setpoint [deg C] */
		    case 14: /* maximum modulation level setting [%]  */ 
		    case 15: /* maximum boiler power [kW]             */
		    case 16: /* room temperature setpoint [deg C]     */
		    case 17: /* relative modulation level [0-100%]    */
		    case 18: /* boiler water pressure [bar]           */
		    case 24: /* room temperature [deg C]              */
		    case 25: /* boiler water temperature [deg C]      */
		    case 26: /* DHW temperature [deg C]               */
		    case 27: /* Outside temperature [deg C]           */
		    case 28: /* return water temperature [deg C]      */
		    case 29: /* solar storage temperature [deg C]     */
		    case 56: /* max DHW setpoint [deg C]              */
		    case 57: /* max CH water setpoint [deg C]         */
		      
		      dval = OT_float_to_double( OT_payload );
		      fprintf( fp_out, "%17.6f,%d,", ot_t[j], ot_id[j] );
		      fprintf( fp_out, "%.2f\n", dval );
		      break;
		      
		      /* two signed characters */
		      
		    case 35: /* boiler fan speed and setpoint                           */
		    case 48: /* high byte: DHW temp high bound, low: DHW temp low bound */
		      
		      cval[0] = (char)(OT_payload & 0x00ff);
		      cval[1] = (char) ((OT_payload & 0xff00) >> 8 );
		      fprintf( fp_out, "%17.6f,%d,", ot_t[j], ot_id[j] );
		      fprintf( fp_out, "%d,%d\n", cval[1], cval[0] );
		      break;
		      
		      /* unsigned short */
		      
		    case 73:  /* OEM-specific code for ventilation / heat recovery 
				 (unknown data format, stored as unsigned short for now */
		    case 113: /* number of unsuccessful burner starts     */
		    case 114: /* number of times flame signal was too low */
		    case 115: /* OEM diagnostic code                      */
		    case 116: /* number of burner starts                  */
		    case 117: /* number of CH pump starts                 */
		    case 118: /* number of DHW pump/valve starts          */
		    case 119: /* number of burner starts for DHW          */
		    case 120: /* burner operation hours                   */
		    case 121: /* CH pump hours                            */
		    case 122: /* DHW pump/valve hours                     */
		    case 123: /* DHW burner hours                         */
		      
		      usval = OT_payload;
		      fprintf( fp_out, "%17.6f,%d,", ot_t[j], ot_id[j] );
		      fprintf( fp_out, "%d\n", usval );
		      break;
		      
		      /* two unsigned characters */
		      
		    case 127: /*slave number and type */
		      
		      ucval[0] = (unsigned char)(OT_payload & 0x00ff);
		      ucval[1] = (unsigned char) ((OT_payload & 0xff00) >> 8 );
		      fprintf( fp_out, "%17.6f,%d,", ot_t[j], ot_id[j] );
		      fprintf(fp_out, "%d,%d\n", ucval[1], ucval[0] );
		      break;
		      
		    default:
		      break;
		    }
		}  
	    }
	  
	  fclose( fp_out );
	}
    }

  printf("%s: %6d lines read, %6d lines parsed\n", file_in, cnt, valid );

  /* clean up */

  free( line );
  free( tm );
  fclose( fp ); 
}


double OT_float_to_double( unsigned short val )
{
  /* 
   * convert unsigned short (f8.8 float really) to double according to the 
   * OT protocol. 
   */
  short h, l;
  h = (short)( val >> 8 );
  h = h << 8;
  l = (short)( val & 0x00ff );

  return ( (double) ( h | l ) ) / 256.0;
}

[tzieg]

If you have OTmonitor running, you can configure the datalog to save a record of the items you are interested in at regular intervals. Then the only task left is to make a graph out of that.

Hi Schelte,

I gave it a try: the resulting .csv file is an excellent starting point for visualization: the log file with different intervals for different data is much more difficult to parse.

I have some difficulties to write the datalog in daemon mode though. Using the gui, I can enter the data fields, the interval and the file and the data log works as expected. If I stop the gui and restart otmonitor in daemon mode, the datalog does not get appended --there are no additional entries.

The config file seems to be ok:

datalog {
append true
file /home/pi/otdata.txt
enable true
itemlist {
outside
flame
dhwmode
chmode
dhwenable
diag
setpoint
fault
temperature
timestamp
otcstate
boilertemp
modulation
returntemp
controlsp
dhwsetpoint
chwsetpoint
maxmod
}
interval 300000
}


Best,
Thomas

[hvxl]

Works for me.

Do you use the same config file when running otmonitor in gui mode and daemon mode? You can check the configuration via the web server.

Does the user that runs otmonitor in daemon mode have the right permissions to write to /home/pi/otdata.txt?

[tzieg]

Hi Schelte,
Good to hear it works for you --so something must be different on my side.

I did some more experiments:

• I realized: the behavior for gui and daemon mode is the same:
  - after entering interval, data fields, and filename for the first time using the gui, all works as expected
  - if restarting in gui- or the daemon-mode, the datalog does not work
• The user for daemon mode and gui mode is the same.
• Checking the configuration via the web server shows the check mark for the datalog and the append mode (see attachment)
• If I use this command line
  

  Code: Select all

  /usr/local/bin/otmonitor --daemon --datafile=/home/pi/otdata.txt --append

  the datalog works as expected.

So, it seems, the settings in .config/otmonitor/otmonitor.conf file are somehow ignored in my system (raspberry-3, debian jessie, otmonitor with the 'dash-patch' I've sent a few weeks back).

I can live with this solution, but it's strange that the system does not behave as it should.

I had a quick look into otmonitor.tcl
I found where the command line options are parsed and eventually the command line options overwrite the settings from the config file

Code: Select all

# Now override with the collected command line options
foreach n [array names optcfg] {
    set cfg($n) $optcfg($n)
}

I could not find the parsing of the config file itself --any hints?


Best,
Thomas

[hvxl]

I could not find the parsing of the config file itself --any hints?

It's in lib/tcl8/8.6/settings-1.1.tm. Unfortunately that library is a bit hard to read due to windows complications. You'll be most interested in settings::loadfile.

[umtauscher]

Hi Schelte,

first of all I wanted to thank you very much for your great project and support. I have built one of your gateways myself a few days ago and it's working great so far.
I was very impressed how well the project is dokumented.

Adding to this topic, I can totally verify Thomas' findigs. I am running OTMonitor in deamon mode on a RaspberryPi3 and it behaves exactly the same. The data file is only written, when the filename is given in the command line. The config file is ignorded for that matter which means everthing you change in the config page on the webserver is ignored.

Another nice feature would be to write a header with the chosen fields to the data file as that would mean one could directly load it into an excel sheet.

I know this has been a project of yours for years and I don't know, if you are still working on it, but I wanted to make the suggestion anyway.
Thanks again for your great contributions.
Cheers
Wilhelm

[rpav]

Adding to this topic, I can totally verify Thomas' findigs. I am running OTMonitor in deamon mode on a RaspberryPi3 and it behaves exactly the same. The data file is only written, when the filename is given in the command line. The config file is ignorded for that matter which means everthing you change in the config page on the webserver is ignored.

I confirm this. otmonitor starts to write data to datafile only if it is specify on command line. However it reads config file for parameters of data file - which data to log and how often.

Regards,
Roman

[rpav]

Hi,

Now, as OTGW for my Brötje WGB 2 is operating stable, I can start to analyze the collected data with the goal to optimize the parameterization of the heating system.
What's the best way to visualize the behavior for e.g. a week or even a month?

What about rrdtool & cacti:http://www.cacti.net/