Just had a look at a newer toon and found some changes in the rootfs image (UBIfs parameters have changed).
Code: Select all
20160402 marcelr
added ubifs image construction info for newer toons
20160313 marcelr
first release
######################################################################
#
# 1: Repairing a corrupted root filesystem on toon.
#
######################################################################
Some time ago I woke up in a cold house. The boiler that should have
started up at 6.00 in the morning wasn't running. I checked the
thermostat (toon) and it turned out to be dead.
It was a friday morning and I had to go to work, so I replaced it with
another toon that I had lying around (flash version, not the fancy qt
gui). Problem solved, at least for the moment.
That evening I opened up the dead toon, hooked it up to a
serial-to-USB adapter and booted up. The boot loader got stuck at
loading the kernel, and posted a warning that the splash screen could
not be loaded. The flash partitions holding the splash image and the
kernel (and probably the others too) were most likely
corrupted. Badly. I had no backup of these partitions. Stupid me.
I dumped the kernel partition of my still working toon and the dead
one to look for differences. There was no resemblance between the two,
from the corrupted kernel it was clear that it was bad. Normally, the
kernel version is stated in human-readable text in the first few bytes
of the kernel image, in this case it was only gibberish. This toon was
dead as brown bread.
What to do? Years ago I built my own router from an old headless and
diskless 486 with a network adapter with boot ROM. The thing booted
over the network, getting all its information, kernel, initrd, ramdisk
image etc., from another machine. Toon's boot loader, U-Boot,
supports network booting as well. That could be my way out. The plan
was simple: build a kernel capable of mounting a root filesystem over
NFS, load this kernel into toon through TFTP, boot, repair the flash
partitions and flash everything onto toon.
Problem solved. Easy-peasy. Yeah, right ...
######################################################################
#
# 2: Prerequisites
#
######################################################################
To repair a corrupted toon, with a working bootloader, this is what
you need:
A computer running a DHCP server, an NFS server, a TFTP server, and a
serial client. Typically, you will find all of these services in a
linux box. You will need root access to this machine. Furthermore, you
will need to have the mtd-utils package installed and working.
A serial-to-USB converter with 3.3V signal level. If you need to buy
one, go for a version with separate female header connectors at the
serial side. These are the easiest for connecting with toon.
A wired (ethernet) router to connect toon to your computer.
The quby openembedded tree for toon. Not just the tarball, but a
working installation of it. You will need to be able to build a
working, modified linux kernel for your toon.
A copy of a (recent) root filesystem for toon. Dump your own rootfs
while you still can. You may need it later. See the script
'dump_rootfs.sh' on domoticaforum.eu
Working knowledge of:
1: linux kernel configuration.
2: dhcp, nfs, tftp configuration, including firewalls to pass the
services through.
3: entering commands through a CLI.
4: toon rooting. If you have no clue how to open toon and attach a
serial interface to it, stop here, go and do something else.
######################################################################
#
# 3: Configuration of services
#
######################################################################
The following description is not carved in stone, there are many ways
to properly configure a DHCP, NFS and TFTP server. This is just how I
did it. My linux host is a CentOS 6 machine, with the old school
SysVinit startup system. Newer machines may have systemd. Can't help
you there, if yours is one like that. The location of scripts may vary
from one distro to another. Nevertheless, everything will be located
somewhere in /etc or its subdirectories.
## TFTP ##
TFTP (trivial file transfer protocol) is a simple protocol to
automatically upload files from one computer to another, typically at
boot time. In this case we need it to load a kernel image directly
into toon's memory.
On my linux box, the TFTP server requires a simple configuration
script, and a running xinetd. This is the configuration script for my
TFTP server, it resides in /etc/xinet.d/tftp:
# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -p -svv /tftpboot
per_source = 11
cps = 100 2
flags = IPv4
}
Important are the disable flag (no) and the server args. /tftpboot is
the directory from which clients can download their stuff.
The tftp server is started by running xinetd:
service xinetd start
or similar.
## DHCP ##
DHCP (dynamic host configuration protocol) configures computers on a
network, based on their MAC address alone. U-Boot supports host
configuration via DHCP, so a DHCP server is needed to assign an IP
address to toon. Simultaneously, a DHCP server can be configured to
enable automatic uploading of stuff to a remote host. In cheap routers
for residential use, this option is generally not available. So we
need to set up our own. Since for home use, computers hardly run DHCP
servers. That's normally handled by your router. Because of the
network boot of toon, you will need your own DHCP service. I
configured my (separate) router to hand out one IP address only (for
my CentOS box) and let the linux machine do all the rest.
This is the config script I used for dhcpd:
# dhcpd.conf
#
# global parameters...
option domain-name "toontest.org";
ddns-update-style none;
not authoritative;
default-lease-time 600;
max-lease-time 7200;
option broadcast-address 10.1.0.255;
option routers 10.1.0.100;
subnet 10.1.0.0 netmask 255.255.255.0 {
# subnet-specific parameters...
range 10.1.0.202 10.1.0.210;
}
host toon {
hardware ethernet 00:11:22:33:44:55;
fixed-address 10.1.0.201;
next-server 10.1.0.10;
filename "/tftpboot/toon/uImage-nfs";
}
# end of dhcpd.conf
This way, toon gets a static IP, and has an idea where to find its kernel.
Start the service (as root) as follows:
dhcpd -cf dhcpd.conf
## NFS ##
NFS (network file system) is a protocol to share block devices over a
network. Typically, an NFS server can give other hosts access to
(parts of) its disk space. NFS clients can mount NFS exports from a
server, and then use these disks as if they were part of the native
disk space of the client. When configured properly, a diskless host
can mount ALL of its disk space via NFS.
To be able to mount an NFS volume, it needs to be exported to your
client by the server. This is taken care of in the exports file:
/etc/exports: (two lines)
/tftpboot/toon/rootfs 10.1.0.201(rw,sync,no_root_squash,no_subtree_check)
/tftpboot/toon 10.1.0.201(rw,sync,no_root_squash,no_subtree_check)
Edit this file according to your needs, and (re)start the nfs server:
service nfsd restart
This command starts a host of services (rpcbind, mountd, nfsd and some
others), which need to be able to penetrate the firewall of your
server.
## IPTABLES ##
On modern linux boxes, the firewall is called iptables, and is
switched on by default. This firewall has a configuration script,
/etc/sysconfig/iptables, and you need to edit this script to hold the
following lines (edit IP addresses according to your needs):
-A INPUT -s 10.1.0.0/24 -p udp -m state --state NEW -m multiport --dports 111,892,2049,32769 -j ACCEPT
-A INPUT -s 10.1.0.0/24 -p tcp -m state --state NEW -m multiport --dports 111,892,2049,32803 -j ACCEPT
This opens all ports required to export NFS mounts.
Similarly, for TFTP, port 69 needs to be opened:
-A INPUT -p udp -m state --state NEW -m udp --dport 69 -j ACCEPT
Edit again, and restart iptables. You may have to restart nfsd as
well. A second option is to switch off your firewall, and don't
bother with the ports. It's up to you.
######################################################################
#
# 4: Booting toon over the network
#
######################################################################
When you are lucky enough that only the kernel image partition is
corrupted, you can now boot your toon over the network. (You even
don't need NFS support for that).
Put the kernel image you want to boot (the original kernel image made
in the open embedded tree will do) in /tftpboot/toon/, on your tftp
server.
If you haven't done so already, connect the serial I/O interface to
toon, and open a terminal. Serial port settings should be 115200 baud,
8N1.
Verify that your U-Boot hangs; it will present the U-Boot prompt: U-Boot>
This is what my toon says at start-up:
U-Boot 2010.09-R6 (Mar 14 2012 - 11:15:10)
CPU: Freescale i.MX27 at 400.168 MHz
Prodrive B.V. ED2.0
DRAM: 128 MiB
NAND: 128 MiB
LCD: Initializing LCD frambuffer at a1400000
LCD: 800x480, pbb 4
LCD: Drawing the logo...
In: serial
Out: serial
Err: serial
Error: no valid bmp image at a1d214a8 (signature 0xbe 0xda)
Net: FEC
Warning: FEC MAC addresses don't match:
Address in SROM is 1f:56:1d:17:ee:22
Address in environment is 00:0f:11:01:9b:49
Enter password - autoboot in 2 sec...
AND read: device 0 offset 0x300000, size 0x300000
3145728 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
U-Boot>
Make sure to have your toon wired to an ethernet router, connected to
your dhcp server, supplying your toon with a static IP.
At the U-Boot prompt, sequentially type the following lines:
dhcp
tftpboot 0xA1000000 toon/uImage-xxxx
bootm
with uImage-xxxx the actual name of your kernel image.
This connects toon to your server, with a static IP, loads the kernel
into toon's memory, and boots it. The default U-Boot environment
available on toon is used. If that partition is corrupted, you will
find out fairly quickly.
Essentially, three things can happen:
First option: After bootm, nothing seems to happen, the terminal no
longer responds. In this case, the U-Boot environment is probably
corrupted. Check by typing: printenv at the U-Boot prompt. You should
see something like this:
bootdelay=2
loadaddr=0xA1000000
bootdelay=2
mtdids=nand0=mxc_nand
mtdparts=mtdparts=mxc_nand:1M(u-boot)ro,512K(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs)
mtdparts_kernel=mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs)
mem=128M
autoload=no
backlight_brightness=50
baudrate=115200
console=ttymxc0
addtty=setenv bootargs ${bootargs} console=${console},${baudrate}
addmtd=setenv bootargs ${bootargs} ${mtdparts_kernel}
nandargs=setenv bootargs ubi.mtd=4 root=ubi0:rootfs rw rootfstype=ubifs
boot_nand=run nandargs addmtd addtty addmisc; nand read ${loadaddr} kernel; bootm ${loadaddr}
boot_nand_backup=run nandargs addmtd addtty addmisc; nand read ${loadaddr} kernel-backup; bootm ${loadaddr}
bootcmd=run boot_nand
splashimage=0x180000
ethact=FEC
baudrate=9600
sn=12-37-023-641
pn=6500-1102-0301
software_compatibility=0
manufacture_date=2012/10
ethaddr=00:0F:11:01:9B:49
bootargs=ubi.mtd=4 root=ubi0:rootfs rw rootfstype=ubifs mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs) console=ttymxc0,115200 mem=128M init=/bin/sh
partition=nand0,0
mtddevnum=0
mtddevname=u-boot
addmisc=setenv bootargs ${bootargs} mem=${mem}
name=u-boot
If you don't see such information, you will have to rebuild the U-Boot
environment with mkenvimage (or ask me for an image of it).
Second option: The kernel boots, and when it tries to mount the root
filesystem, the kernel panics:
.....
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
NET: Registered protocol family 10
ip6_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 17
lib80211: common routines for IEEE802.11 drivers
rtc-isl1208 1-006f: setting system clock to 2016-02-20 20:01:37 UTC (1455998497)
UBIFS error (pid 1): ubifs_get_sb: cannot open "ubi0:rootfs", error -19
VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0)
Please append a correct "root=" boot option; here are the available partitions:
1f00 512 mtdblock0 (driver?)
1f01 1536 mtdblock1 (driver?)
1f02 3072 mtdblock2 (driver?)
1f03 3072 mtdblock3 (driver?)
1f04 121856 mtdblock4 (driver?)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[<c00293d0>] (unwind_backtrace+0x0/0xf0) from [<c0326da4>] (panic+0x60/0x190)
[<c0326da4>] (panic+0x60/0x190) from [<c0008ea0>] (mount_block_root+0x15c/0x20c)
[<c0008ea0>] (mount_block_root+0x15c/0x20c) from [<c00090d0>] (prepare_namespac)
[<c00090d0>] (prepare_namespace+0x8c/0x178) from [<c0008b50>] (kernel_init+0x10)
[<c0008b50>] (kernel_init+0x10c/0x14c) from [<c00258ec>] (kernel_thread_exit+0x)
This means that also the rootfs partition is corrupted. You will have
to mount another rootfs, over NFS. You will also need a kernel with
rootfs on NFS support.
Third option: the kernel boots, mounts the root filesystem, and you
end up with a login prompt. You're lucky, just the kernel partition is
corrupted.
######################################################################
#
# 5: Building a kernel with NFS support
#
######################################################################
To configure the kernel, I extracted the kernel source from the quby
openembedded tree and unpacked it in a separate directory, outside the
openembedded tree. The top-level kernel source directory is called
linux_r07/.
In this directory, I put the following script:
#! /bin/sh
make CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm menuconfig
This starts kernel configuration, using the correct compiler (I happen
to have arm-linux-gnueabi- around), for the right processor type
(arm). You can also use the cross-compiler from the openembedded tree,
just make sure it's in your search path.
Run the script, and edit the kernel configuration (the default .config
for toon is loaded by starting the script).
Under Filesystems, enter the submenu "Network file systems", tick the
boxes "root file system on NFS" and all NFS client options. Make sure
the tick boxes are marked with a * (built-in) and not an M
(module). Under "Enable loadable module support" untick the box
"Module versioning support".
Save the kernel configuration to a separate file (default is .config).
Copy the configuration file to the openembedded tree, in the subdirectory
~/oe/homeautomationeurope/recipes/linux/linux-quby2/
(~/ is the directory where you unpacked the openembedded tree)
Name it defconfig-2.6.36-R07-h11, this replaces the original
configuration file by quby. If you want to keep the old configuration,
back it up before you overwrite with the new kernel config file.
Now got to ~/oe/qb2, set the bitbake environment:
. ../bb_env.sh
and build the kernel:
bitbake linux-quby2
If you have built the linux kernel before, you may need to clean up first:
bitbake -c clean linux-quby2
The freshly built kernel ends up in
~/oe/qb2/tmp/deploy/images/uImage-qb2-hae-2.6.36-R07-h11-yyyymmddHHMMss.bin
with yyyymmddHHMMss the manufacturing timestamp.
Copy this image to /tftpboot/toon/uImage-nfs.
Your kernel is all set for booting toon over the network.
######################################################################
#
# 6: Preparing the root file system
#
######################################################################
If you have previously backed up your root filesystem (as a tarball),
now is a good time to unpack it. Put the full contents of the rootfs
in /tftpboot/toon/rootfs.
If you have not made a back-up of your rootfs, you can use the
barebones rootfs that's made when the quby openembedded tree is built
according to the README file in that tree. This rootfs resides in
~/oe/qb2/tmp/rootfs/image-base-qb2. Copy the full contents of this
directory to /tftpboot/toon/rootfs, including links etc.
In the case that you need to flash the kernel, rootfs etc. onto toon
(typically images made with dd, or brand new ubifs images), place them
somewhere in the root filesystem where you can find them. /root is not
a bad place.
######################################################################
#
# 7: Booting toon over the network, with NFS root file system
#
######################################################################
At the U-Boot prompt, sequentially type the following lines: (I had
them ready in a script, copied them one by one to the U-Boot terminal)
dhcp
tftpboot 0xa1000000 toon/uImage-nfs
setenv bootargs root=/dev/nfs rw nfsroot=10.1.0.10:/tftpboot/toon/rootfs,nolock,tcp console=ttymxc0,115200 loglevel=8
setenv bootargs ${bootargs} mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs)
setenv bootargs ${bootargs} ip=10.1.0.201:10.1.0.10:10.1.0.100:255.255.255.0:toon::off panic=1
bootm
This, sequentially, gets toon an ip adress, loads the kernel from the
TFTP server into memory, at address 0xA1000000, defines the root
filesystem as an NFS mount, read-write enabled, sets the URL of
the root file system, configures a terminal, configures mtd devices on
toon, and sets the network IP adress, NFS server, gateway, netmask,
hostname, and automated reboot when something fails. You need all of
this ;-).
The final line (bootm) starts the actual booting process.
If all goes well, you end up with a login prompt on toon. You may now
login as root.
######################################################################
#
# 8: Recovery of the mtd partitions.
#
######################################################################
## splash screen ##
If you feel a bit uncomfortable flashing your toon, you can start with
the splash screen. The worst that can happen is that toon no longer
presents one. Screwing up the splash screen has no further
implications.
Before you can write to flash, old stuff has to be erased first. Just
start with erasing /dev/mtd1 (that's where the splash image resides):
flash_eraseall /dev/mtd1
and then, find a nice image, having the _EXACT_ size of 800x480
pixels, 24 bit colourdepth, and uncompressed windows .BMP format.
Flash it onto /dev/mtd1 with:
nandwrite -a -p /dev/mtd1 <whatever_its_name_is.bmp>
Reboot and enjoy your personalised toon splash screen.
## kernel ##
If only the kernel, u-boot environment, or splash image partitions are
corrupted, and you have managed to boot toon through tftpboot, you can
copy the images for the respective partitions to toon, through e.g.,
wget. If your images were ripped off the very same toon you now want
to put them back on, this is the way to do it:
(this will in general not work for a root filesystem image, will get
to that later).
Erase the partition you want to recover, in this example
/dev/mtd2. At the shell prompt, type:
flash_eraseall /dev/mtd2
then, if you made the rip with dd, dd is the tool to put it back:
dd if=ripped_partition_image_of_mtd2.img of=/dev/mtd2
This writes the kernel image back to the exact position where it came
from, including bad blocks.
If you only have an image file, made elsewhere, flash it onto the nand
by:
nandwrite -a -p /dev/mtd2 /path/to/image/file
This writes the image straight to the flash, skipping bad blocks if
they exist. In this example /dev/mtd2 was used, this method can be
applied to any partition (except the root filesystem).
CAUTION: write the kernel image ONLY after you have verified that all
the rest of your toon is in working order. Once the kernel is flashed
to NAND flash, you will have to go through the whole rooting process
again to stop it from booting (and subsequent panicking). If your root
filesystem is still corrupted, your toon will have kernel panics
during booting, and will be quite useless. It can always be fixed, but
it's no fun to do.
SO: TEST THE QUALITY OF YOUR ACTIONS EVERY STEP OF THE WAY.
And remember the second rule of sudo:
Think before you type.
######################################################################
#
# 9: Installation of the root filesystem
#
######################################################################
Toon uses ubifs for its root file system. UBI stands for Unsorted
Block Image, specifically designed for flash memory, and is quite
different from other file systems. One of the differences is that it
is not connected to a block device, and therefore, ubi images cannot
be loopback mounted. Quite annoying if you need to change things, or
have a corrupted filesystem. Even if you made an image of your rootfs,
there's no simple way to mount that image on another computer and work
with it. So, to install a UBIFS filesystem on your toon, you will need
the contents of that filesystem, on another computer, with a
filesystem that you can read and write. Contents means: all contents,
so this includes symlinks, pipes, and normal files. Device special
files aren't necessary, since these are made on the fly by udev.
UBIFS images typically are made from (sub) directory trees,
and subsequently written to flash.
If you already unpacked a rootfs tarball (section 6), tested it as an
nfs-mount, and are happy with it, to serve as toon's native root
filesystem, you can convert it to a ubi image, and reformat toon's
rootfs partition with it. If not, go back to section 6 and start
there.
## creating the ubifs image ##
Creating the image is done in two steps. Depending on the firmware
version, the ubifs parameters have different values.
## step 1 ##
When toon boots, it spits out information on the flash parameters of
the ubifs image. In earlier toons it looks like this:
...
0x000000600000-0x000000900000 : "kernel-backup"
0x000000900000-0x000008000000 : "rootfs"
UBI: attaching mtd4 to ubi0
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 129024 bytes
UBI: smallest flash I/O unit: 2048
UBI: sub-page size: 512
UBI: VID header offset: 512 (aligned 512)
UBI: data offset: 2048
...
in later versions, this has changed into:
...
0x000000600000-0x000000900000 : "kernel-backup"
0x000000900000-0x000008000000 : "rootfs"
UBI: attaching mtd4 to ubi0
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 126976 bytes
UBI: smallest flash I/O unit: 2048
UBI: VID header offset: 2048 (aligned 2048)
UBI: data offset: 4096
...
As you can see, the LEB (logical eraseblock) size, VID header offset, and data
offset have changed. This influences the way the rootfs image should be made.
The ubi filesystem is made with mkfs.ubifs.
This program has (among others) the minimum sub page size (-m
option), and LEB size (-e option) as arguments.
For the older versions (approximately first number in the serial
number < 15, check with bootstrap data) this is:
mkfs.ubifs -r ./rootfs/ -m 512 -e 129024 -c 954 -o toon_rootfs_ubifs.img
Newer versions should use:
mkfs.ubifs -r ./rootfs/ -m 2048 -e 126976 -c 954 -o toon_rootfs_ubifs.img
This creates a ubifs image from the contents of ./rootfs/ (without
rootfs/ as the top-level directory). The image is stored in the
current directory, with the name toon_rootfs_ubifs.img.
The arguments for mkfs.ubifs were copied from toon bootstrap data. The
numbers are connected to the physical flash chip, and should be kept
as they are, for your version of toon, otherwise the flashed rootfs
will not be bootable.
## step 2 ##
The second step of the image building involves the preparation of the
raw flash image. this is done with the ubinize command. Again,
depending on your toons version, two variants exist:
older:
ubinize -vv -o rootfs.ubi -m 2048 -s 512 -p 128KiB ubinize.ini
newer (SN 15- and (possibly) later):
ubinize -vv -o rootfs.ubi -m 2048 -p 128KiB ubinize.ini
Ubinize adds physical flash chip information to the image (physical
eraseblock size and some others). Besides some cli arguments, most
settings are done in a configuration file, having .ini style
formatting. For this example, the file is called ubinize.ini, and is
the same for the two variants of toon:
Contents of ubinize.ini:
[ubifs]
mode=ubi
image=toon_rootfs_ubifs.img
vol_id=0
vol_type=dynamic
vol_name=rootfs
vol_flags=autoresize
Again, keep the arguments as they are, you can change names, but do
not change volume id, eraseblock sizes etc. After completion of the
ubinize command, the file rootfs.ubi contains a rootfs image, ready to
be written to flash.
## writing the image to flash ##
Transfer the ubi image to the nfs-mounted root filesystem of your
toon. Then, flash the image to the rootfs device with ubiformat:
ubiformat /dev/mtd4 -f rootfs.ubi
Try the new filesystem, by rebooting according to section 4. If all is
well, your toon should boot normally.
TETS THOROUGHLY IF EVERYTHING WORKS AS PLANNED, and then, as a last
step, flash the kernel image of choice to /dev/mtd2.
If all went well, you now have a working toon again.
######################################################################
#
# 10: What if the bootloader is corrupted?
#
######################################################################
Then you have a different problem. Without a working boot loader, the
only access to your toon is through JTAG. JTAG is quite another kettle
of fish than the stuff presented here. I have accessed my toon through
JTAG, so if you end up in these parts, you can contact me through
domoticaforum.eu. Not sure if I can help you, though.
marcelr, march 2016.
And as attachment again. Also found the solution for the toon zon issue. You really need a newer meteradapter (with double icons for the analogue gas/heat meter), and fw 0.15/0.11. Exchanged my old adapter with such an adapter from one of the forum members, now it works. Not sure if it's calibrated, the pulses do not come from the recommended kWh meter (inepro PRO380 or inepro PRO1), but straight from my inverter, using the analog sensors. It's too early to tell, will keep you posted.