Toon app: Afvalwijzer

[Toonz]

QtWebsockets isn't really for this purpose, but it allows tools like domoticz to interact directly with specific QML apps like sending realtime updated device info to Toon.
Let's start with wget

[marcelr]

OK, done, see Downloads thread.

[Toonz]

That's quick. Working great, just need to use `--no-check-certificate' to skip validating the webserver certificate issuer authority.
Now need to start reading the cron wiki (never used it ) and need to find out the exact url for a specific waste zipcode in Eindhoven.
The parsing bit is easy.....

[marcelr]

cron is easy to use .
I think we just need to make a small shell script that takes care of the downloads (through wget) and then repeat that download every week or so, through a crontab entry.

[Toonz]

@marcelr: I'm struggling to get to your waste data via wget. First page asks you to enter postcode / housenr and then 'POST's the form data. As far as I know you cannot submit such form data via an URL (this works for forms with 'GET' method but not for 'POST'). Any tips? Could this be scripted otherwise?

[Toonz]

Hi all,

I enhanced the app with the waste provider 'cure-afvalbeheer.nl'.
As this provider is using 'https', a separate method of retrieving the data needs to be installed (will be posted separately by Marcelr).
This version is reading the file '/root/waste/waste_calendar.ics' to get the waste collection dates for Cure Afvalbeheer.
You therefore do not need to enter the postcode/housenr in the WastecollectionApp.qml app for this provider.

Configuration:
You need to edit WastecollectionApp.qml at the top to select one of the three supported waste providers
And edit your zipcode/house nr (not needed for Cure Avalbeheer).
This version is fully compatible with the thermostat_mod V1.2 posted earlier.

Currently supported:
"mijnafvalwijzer.nl"
"deafvalapp.nl"
"cure-afvalbeheer.nl"

Let me know if you want support for additional providers.

Regardsz,

Toonz

[QuasaR]

Can you implemente support for Cyclus ?

https://afvalkalender.cyclusnv.nl/form (2841 SP / 44)

[marcelr]

Can you implemente support for Cyclus ?

https://afvalkalender.cyclusnv.nl/form (2841 SP / 44)

Should be easy; the site is an almost exact copy of the one of cure afvalbeheer; will post the download scripts and cron install manual later today.

[marcelr]

Installation of wget and cron, for waste data download.

This is the manual:

Code: Select all

###############################################################################

How to download "afvalwijzer" data from a https website, using wget and cron

20170513, marcelr, first draft

###############################################################################

Recently, Toonz created an app which displays the waste collection agenda for 
a specific address on toon.

From version 1.3 onwards, it also supports cure-afvalbeheer (active in the 
Eindhoven area).

The waste collection agenda provided by this company is available as an .ics 
file, from their website. The website is served as a https service, offered 
through service port 443 (secure http or https).

## getting acccess to files served through https (port 443) ##
 
Toon has no support for this protocol, by default. So, to gain access to this 
site, the following is required:

1: (Access to) a rooted toon, with working ssh and (preferably) sftp servers.

2: Install package for wget, with ssl support. Available from the Downloads 
   thread on this forum.

3: Some proficiency in using vi. 

What to do:

1: Upload wget with ssl support (wget_1.12-r8.2_qb2.ipk) to toon and install:
   opkg install wget_1.12-r8.2_qb2.ipk

2: Open port 443 in the firewall:
   Locate the file  /etc/default/iptables.conf and add the following line:
   
   -A HCB-INPUT -p tcp -m tcp --dport 443 --tcp-flags SYN,RST,ACK SYN -j ACCEPT

3: Restart iptables by issueing the command:

   /etc/init.d/iptables restart

## Find the right URL for your waste agenda and adapt the download script ##

1: The agenda for the waste collection is given in the cure website:
   
   Fill in your address details (post code, house number) and click 
   NAAR MAAND- EN JAARKALENDER > -> ZET INZAMELDATA IN UW AGENDA

   Locate the link to your agenda and copy it into the "get_data_from_cure.sh" 
   script (it has the following shape):

   https://afvalkalender.cure-afvalbeheer.nl/ical/0yyyy000000xxxxx


#! /bin/sh
#
# get_data_from_cure.sh
#
# script to download calendar data from waste management company Cure 
# (Eindhoven area, NL)
#
# call: get_data_from_cure.sh <address code>
#
# The address code is found at the cure website. 
# fill in your address details (post code, house number) and click 
# NAAR MAAND- EN JAARKALENDER > -> ZET INZAMELDATA IN UW AGENDA
#
# At this page, copy the link 
# https://afvalkalender.cure-afvalbeheer.nl/ical/<whatever-your-number-is 
# to the command line after the call to this script:
# get_data_from_cure.sh <address code>
#
# Hit <enter> and your personal calender is downloaded to the directory 
# defined in $DIR_OUT.
# The address code is linked to your address, not to a time, and thus remains 
# the same, so you can repeat downloads from the same URL, and get updates for 
# your calendar.
#
# another option is to paste the URL into this script, replacing the first 
# argument ($1) of this script, in the definition of $URL below.

WGET=/usr/bin/wget
FILE_OUT=waste_calendar.ics
DIR_OUT=~/waste/
#URL=$1
URL=<paste URL here>

mkdir -p $DIR_OUT

$WGET --no-check-certificate -O /$DIR_OUT/$FILE_OUT $URL &> /dev/null 

# end of script


2: On toon, create the directory /root/bin:

   mkdir -p /root/bin

3: Upload the "get_data_from_cure.sh" script to toon, put it in /root/bin, and 
   make sure it is executable:

   chmod 755 /root/bin/get_data_from_cure.sh

4: Test by issueing the command /root/bin/get_data_from_cure.sh
   This should give you a file called waste_calendar.ics, in the directory
   /root/waste

## Automation of the calendar download ##

To automate regular execution of programs, unix systems (like toon) use a 
utility called cron. Cron is a daemon process which runs in the background, 
and checks every minute whether something from the crontab file needs to be 
done. The crontab file is a file which sets the rules for cron, what to 
execute, and when. It is edited with the program crontab.

To automate waste agenda download, do the following:

1: Download and install cron_3.0pl1-r8_qb2.ipk

2: Edit the crontab for user root, by issueing the following command:

   crontab -e 
 
 This opens an editor (typically vi), paste the following into the file:

# crontab for root
#      The time and date fields are:
#
#             field          allowed values
#             -----          --------------
#             minute         0-59
#             hour           0-23
#             day of month   1-31
#             month          1-12 (or names, see below)
#             day of week    0-7 (0 or 7 is Sunday, or use names)
#
SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:                            
15 4 3 * * /root/bin/get_data_from_cure.sh

and save. You may omit the lines starting with a hash mark (#).

3: Start the cron daemon, by issueing:

   /etc/init.d/cron start

## Getting the information on toon ##

1: Install the app wastecollection v1.3 or later.

2: Edit the file WastecollectionApp.qml, to have the property wasteCollector
   reflect your waste collector:

	property string wasteCollector : "cure-afvalbeheer.nl"
   
   and save.

3: Add the wastecollection app to the file /HCBv2/qml/qb/base/Globals.qml.

4: Restart qt with killall qt-gui

5: All done, you should now be able to install a tile with your waste 
   collection data from toon's tile selection menu.

and these are the scripts you need:

get_data_from_cure.sh:

Code: Select all

#! /bin/sh
#
# get_data_from_cure.sh
#
# script to download calendar data from waste management company Cure 
# (Eindhoven area, NL)
#
# call: get_data_from_cure.sh <address code>
#
# The address code is found at the cure website. 
# fill in your address details (post code, house number) and click 
# NAAR MAAND- EN JAARKALENDER > -> ZET INZAMELDATA IN UW AGENDA
#
# At this page, copy the link 
# https://afvalkalender.cure-afvalbeheer.nl/ical/<whatever-your-number-is 
# to the command line after the call to this script:
# get_data_from_cure.sh <address code>
#
# Hit <enter> and your personal calender is downloaded to the directory 
# defined in $DIR_OUT.
# The address code is linked to your address, not to a time, and thus remains 
# the same, so you can repeat downloads from the same URL, and get updates for 
# your calendar.
#
# another option is to paste the URL into this script, replacing the first 
# argument ($1) of this script, in the definition of $URL below.

WGET=/usr/bin/wget
FILE_OUT=waste_calendar.ics
DIR_OUT=~/waste/
#URL=$1
URL=<paste URL here>

mkdir -p $DIR_OUT

$WGET --no-check-certificate -O /$DIR_OUT/$FILE_OUT $URL &> /dev/null 

# end of script

crontab for root:

Code: Select all

# crontab for root
#      The time and date fields are:
#
#             field          allowed values
#             -----          --------------
#             minute         0-59
#             hour           0-23
#             day of month   1-31
#             month          1-12 (or names, see below)
#             day of week    0-7 (0 or 7 is Sunday, or use names)
#
SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:                            
15 4 3 * * /root/bin/get_data_from_cure.sh

Will post everything in Downloads as well, but that's for later, gotta run ...

[Toonz]

almost the same, collection of paper is called differently ('Oud papier' instead of 'Papier') and plastic is collected separately. Implementation will be easy. will do later today.

[Toonz]

Hi all,

I enhanced the app with the waste provider 'cyclusnv.nl'.
This site is almost a copy of cureafvalbeheer.nl.
As this provider is using 'https', a separate method of retrieving the data needs to be installed as posted by Marcelr earlier.
This version is reading the file '/root/waste/waste_calendar.ics' to get the waste collection dates for Cyclus NV.

Configuration:
You need to edit WastecollectionApp.qml at the top to select one of the four supported waste providers
And edit your zipcode/house nr (not needed for Cure Avalbeheer and Cyclus NV).
This version is fully compatible with the thermostat_mod V1.2 posted earlier.

Currently supported:
"mijnafvalwijzer.nl"
"deafvalapp.nl"
"cure-afvalbeheer.nl"
"cyclusnv.nl"

Let me know if you want support for additional providers.

Regardsz,

Toonz

[Toonz]

SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:
15 4 3 * * /root/bin/get_data_from_cure.sh

Hi QuasaR,

I noticed your waste calender contains only data for the next 4 weeks (CureAfvalbeheer gives more entries in the future).
You might consider retrieving the file every 2 weeks instead of once a month.

Regardz,

Toonz

[QuasaR]

SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:
15 4 3 * * /root/bin/get_data_from_cure.sh

Hi QuasaR,

I noticed your waste calender contains only data for the next 4 weeks (CureAfvalbeheer gives more entries in the future).
You might consider retrieving the file every 2 weeks instead of once a month.

Regardz,

Toonz

Tnx Toonz.

So my entry for the crontab would be like this ?

15 4 3,17 * * /root/bin/get_data_from_cure.sh

[marcelr]

Yes, only the name of the script in the zip-file has changed (in the manuals/tutorials thread).
(get_waste_calendar_data.sh)

[dennis071nl]

Hi guys,

Where can i find a list of waste collection providers?? I live in Katwijk, actually Valkenburg gemeente Katwijk, but i can't find my area on the given sites...

Can anyone confirm this??

Thanks, Dennis