Toon Recovery

Forum about Toon hardware, both versions 1 and 2 of Toon

Moderators: marcelr, TheHogNL, TerrorSource, Toonz

Toon Recovery

Postby TheHogNL » Mon Jun 17, 2019 5:41 pm

Hi all,

I am proud to announce a new TSC tool, called Toon Recovery. https://github.com/ToonSoftwareCollective/ToonRecovery

This is posted in the Toon Hardware section as this tool allows you to recover your Toon if it is not booting anymore. Only a working uboot is necessary to recover your Toon.
It will load a default factory image to your Toon as if it was fresh from the box. With all those broken Toon due to faulty firmware updates from Eneco I decided it was time to release this wonderful piece of engineering.

It has not been tested a lot. But then again, if your Toon gets broken you can always recover it with a newer version of this software if you provide me with the details why your Toon got bricked using this software :lol:

And as with the update script, if you like my software and want to support me to allow me to develop more you can always donate on https://paypal.me/pools/c/8bU3eQp1Jt
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby jozg » Mon Jul 08, 2019 4:06 pm

Hello TheHog,

I'll try tomorrow, for now i prepared some things:

I just installed a rpi, and made the same rpi an nfs server.
So now i need to connect the serial wires 3st, and i don't need the jtag, because i have an old loader.
Both the rpi and the toon needs to be connected to the wired ethernet switch.

Give it a try tomorrow.

Thanks for helping me out!.

Regards,
jozg
Starting Member
Starting Member
 
Posts: 47
Joined: November 2017

Re: Toon Recovery

Postby TheHogNL » Mon Jul 08, 2019 4:19 pm

Yes and make sure you have enabled NFSv2 (btw: I moved your message to the topic for recovery)
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby jozg » Tue Jul 09, 2019 12:35 pm

Hello theHog,

You saved my Toon and my life ;-)
I use one single raspberry.
After the reset, i connected my usb uart connection, but i wan't to use minicom on the same raspberry pi, but i seemed not to find the correct port to connect to.
If that worked, i should be able to recover with a single pi, instead of using my windows pc with putty and the uart. But it worked ;-)

Then i backupped the VPN config, and then format and recover to 4.20.7, and after that i recover the VPN config again.
Everything went fine!, thanks. Maybe nice to notice, the root password has changed to 'toon' in stead of my own.

Thanks for the good work!.

I used these steps (on my pi3):

sudo apt-get install nfs-kernel-server portmap nfs-common
sudo nano /etc/exports

add this:
/srv/nfs *(rw,no_subtree_check,async,no_root_squash)

sudo /etc/init.d/nfs-kernel-server restart
mkdir -p /srv/nfs
cd /srv/nfs
curl -Nks "http://qutility.nl/toon-recovery-nfs-server-image.tar.gz" | tar zxvf -

To enable NFS v2 i hinted from: https://ubuntuforums.org/showthread.php?t=2381512

edit /etc/default/nfs-kernel-server

Change:

RPCNFSDCOUNT=8

to:

RPCNFSDCOUNT="-V 2 8"

And change:

RPCMOUNTDOPTS="--manage-gids"

to:

RPCMOUNTDOPTS="-V 2 --manage-gids"

then issue:

sudo service nfs-kernel-server restart
jozg
Starting Member
Starting Member
 
Posts: 47
Joined: November 2017

Re: Toon Recovery

Postby TheHogNL » Tue Jul 09, 2019 1:03 pm

Minicom should be able to use /dev/serial (the one which the toon recovery uses also, per default).
I'm using a usb uart myself on the Pi and not using the Pi's serial port itself. But again, it should work. Unknown why it didn't work for you.

But great to hear that you managed to repair your toon using the tool and providing some steps also! And yes, 'toon' is the root password after recovery. I will put that in the README
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby CrumNL » Tue Jul 16, 2019 6:22 pm

The script looks good and very useful for sure.
I'm just not getting it any further to boot from nfs for some reason.
I'm using a rpi3, tried nfs with only v2, v2&v3, all versions, but the script won't bite despite me being able to mount the same path manually.
After "Loading kernel into memory from server. Should not take more than 60 seconds ..."
It stops with Error loading from NFS server. I tried with nfs on the rpi and with a NAS but that gives the same result.
I'm uncertain if it is something in the script or something I'm messing up, any suggestions are welcome :)
CrumNL
Starting Member
Starting Member
 
Posts: 17
Joined: March 2019

Re: Toon Recovery

Postby TheHogNL » Tue Jul 16, 2019 7:08 pm

This is most certain because the NFS isn't providing NFS versions 2. The uboot kernel loading over NFS works only over NFS.

So check with "cat /proc/fs/nfsd/versions" and follow the README

And also check the server log files to see if your Toon tries to connect.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby CrumNL » Tue Jul 16, 2019 11:03 pm

I did a reinstall of raspbian stretch and followed the rpi3 instructions and readme but it doesn't seem to coorporate or I just don't get it.
When I look at the versions file, that doesn't seem to change no matter what I edit in nfs-kernel-server or the exports file.
In it, all versions are listed with +
Also when I juggle with the parameters I had it with only +2 and the rest -.
But, no connection.

Until suddenly I now get it at a timeout to quit. :
Code: Select all
pi@raspberrypi:~/ToonRecovery $ ls
assets   __main__.py  recovery.py   toon-recovery-nfs-server-image.tar.gz
LICENSE  README.md    recovery.pyc
pi@raspberrypi:~/ToonRecovery $ sudo python . --jtag-available --gatewayip 192.168.2.254
INFO:__main__:Starting up...
INFO:__main__:Detected JTAG hardware 'rpi3'
INFO:__main__:Setting server ip to 192.168.2.87
INFO:recovery:Waiting for Toon to restart
INFO:recovery:Toon has U-Boot version 2010.09-R8
INFO:recovery:Using password to log in
INFO:recovery:Logging in to U-Boot
INFO:recovery:Patching U-Boot
INFO:recovery:Requesting network details using DHCP...
INFO:recovery:Received valid IP address, netmask and gateway using DHCP.
INFO:recovery:Loading kernel into memory from server. Should not take more than 60 seconds ...
INFO:recovery:Timeout loading from NFS server
pi@raspberrypi:~/ToonRecovery $ ls /srv/nfs/
dumps/      toon/       uImage-nfs


I see an authenticated connection that gets logged every minute or so
Code: Select all
Jul 16 21:43:41 raspberrypi rpc.mountd[473]: authenticated mount request from 192.168.2.3:1000 for /srv/nfs/toon/boot (/srv/nfs)
Jul 16 21:44:54 raspberrypi rpc.mountd[473]: authenticated mount request from 192.168.2.3:1000 for /srv/nfs/toon/boot (/srv/nfs)
Jul 16 21:46:06 raspberrypi rpc.mountd[473]: authenticated mount request from 192.168.2.3:1000 for /srv/nfs/toon/boot (/srv/nfs)
Jul 16 21:47:19 raspberrypi rpc.mountd[473]: authenticated mount request from 192.168.2.3:1000 for /srv/nfs/toon/boot (/srv/nfs)
Jul 16 21:48:36 raspberrypi rpc.mountd[473]: authenticated mount request from 192.168.2.3:1000 for /srv/nfs/toon/boot (/srv/nfs)


And it looks to me like it does run v2 :
Code: Select all
Jul 16 22:58:14 raspberrypi systemd[1]: Reached target Host and Network Name Lookups.
Jul 16 22:58:14 raspberrypi systemd[1]: Starting Preprocess NFS configuration...
Jul 16 22:58:14 raspberrypi systemd[1]: Started Preprocess NFS configuration.
Jul 16 22:58:14 raspberrypi systemd[1]: Starting Notify NFS peers of a restart...
Jul 16 22:58:14 raspberrypi systemd[1]: Starting NFS status monitor for NFSv2/3 locking....
Jul 16 22:58:15 raspberrypi sm-notify[1470]: Version 1.3.3 starting
Jul 16 22:58:15 raspberrypi systemd[1]: Started Notify NFS peers of a restart.
Jul 16 22:58:15 raspberrypi rpc.statd[1473]: Version 1.3.3 starting
Jul 16 22:58:15 raspberrypi rpc.statd[1473]: Flags: TI-RPC
Jul 16 22:58:15 raspberrypi rpc.statd[1473]: Failed to read /var/lib/nfs/state: Success
Jul 16 22:58:15 raspberrypi rpc.statd[1473]: Initializing NSM state
Jul 16 22:58:15 raspberrypi systemd[1]: Started NFS status monitor for NFSv2/3 locking..

I guess I'll have to find out more about forcing only nfs v2.
CrumNL
Starting Member
Starting Member
 
Posts: 17
Joined: March 2019

Re: Toon Recovery

Postby TheHogNL » Wed Jul 17, 2019 11:06 am

It does not need to be only v2. Try the script with debug and send the log here.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby TheHogNL » Fri Jul 19, 2019 7:53 pm

Just updated the recovery image because I had a bug in the menu script (which runs after the toon recovery has booted the toon). So download the image file again and unpack it. Only /menu.sh is updated so you can also only unpack that from the recovery image .tar.gz
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby CrumNL » Sun Aug 04, 2019 10:55 am

I've used the latest version from github, it looks like the nfs mount works but it keeps stopping with an error.
Error loading from NFS server

I'm not sure how to get any further debug output or log file besides daemon log or syslog output.

When I use an external nfs share :
Image

When I use the rpi3's nfs share :
syslog:
Image

daemon log :
Image

Manually from uboot gives some additional output :
Image
CrumNL
Starting Member
Starting Member
 
Posts: 17
Joined: March 2019

Re: Toon Recovery

Postby TheHogNL » Sun Aug 04, 2019 4:56 pm

Check with tcpdump on your server if the packets are correctly sent. Could be packet corruption or something like that.
And you are sure the uImage-nfs file is in that dir?
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby CrumNL » Sat Sep 07, 2019 3:27 pm

Been a little while that I had time to work on it again, but yes; it is there :
Code: Select all
root@raspberrypi:/var/log# ls /srv/nfs/toon/boot/ -lah
total 4.7M
drwxrwxr-x  2 root root 4.0K Jan 25  2019 .
drwxr-xr-x 16 root root 4.0K Jun 16 17:48 ..
-rw-r--r--  1 root root 2.3M Jul  5  2015 uImage-2.6.36-R07-h11
-rw-r--r--  1 root root 2.4M Jan 25  2019 uImage-nfs
root@raspberrypi:/var/log#

I can mount the nfs from kali or the rpi itself without a problem and browse the directories.

When I start the recovery script :
The fourth line below is strange I think,
DEBUG:recovery:serveripp= 192.168.2.87
also that a little further below changes to serverip=192.168.2.1
Code: Select all
########### Pi toonrecovery debug output ################
root@raspberrypi:/home/pi/Downloads/ToonRecovery# python . --jtag-available --serverip 192.168.2.87 --gatewayip 192.168.2.254 --output-level DEBUG
[....]
DEBUG:recovery:manufacture_date=2014/09
DEBUG:recovery:ethaddr=00:0F:11:03:A8:D4
DEBUG:recovery:addmisc=setenv bootargs ${bootargs} mem=${mem} lpj=999424
DEBUG:recovery:serveripp= 192.168.2.87    <<<<<<<<<<<<
DEBUG:recovery:bootargs=root=/dev/nfs rw nfsroot=192.168.2.87:/srv/nfs/toon,nfsvers=3,nolock,tcp console=ttymxc0,115200 loglevel=8 mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs) ip=192.168.2.3:192.168.2.87:192.168.2.254:255.255.255.0:toon::off panic=0
DEBUG:recovery:gatewayip=192.168.2.254
DEBUG:recovery:netmask=255.255.255.0
DEBUG:recovery:ipaddr=192.168.2.3
DEBUG:recovery:serverip=192.168.2.1   <<<<<<<<<<<<
DEBUG:recovery:U-Boot>
INFO:recovery:Received valid IP address, netmask and gateway using DHCP.
INFO:recovery:Loading kernel into memory from server. Should not take more than 60 seconds ...
DEBUG:recovery: setenv serverip 192.168.2.87
U-Boot>
DEBUG:recovery: setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:/srv/nfs/toon,nfsvers=3,nolock,tcp console=ttymxc0,115200 loglevel=8 mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs) ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:toon::off panic=0
U-Boot>
DEBUG:recovery:nfs 0xa1000000 /srv/nfs/toon/boot/uImage-nfs;bootm
DEBUG:recovery:Using FEC device
DEBUG:recovery:File transfer via NFS from server 192.168.2.87; our IP address is 192.168.2.3
DEBUG:recovery:Filename '/srv/nfs/toon/boot/uImage-nfs'.
DEBUG:recovery:Load address: 0xa1000000
DEBUG:recovery:Loading: *** ERROR: Cannot mount
INFO:recovery:Error loading from NFS server


This is locally on the toon when I stop the T T T T T T on serial console :
I doubt that the two occurences of the serverip should be there (notice that locally it is .87, despite .1 above in the python output)
Code: Select all
########### toon uboot printenv ################
printenv
bootdelay=2
baudrate=115200
loadaddr=0xA1000000
bootdelay=2
mtdids=nand0=mxc_nand
mtdparts=mtdparts=mxc_nand:1M(u-boot)ro,512K(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs)
mtdparts_kernel=mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs)
mem=128M
autoload=no
backlight_brightness=50
baudrate=115200
console=ttymxc0
addtty=setenv bootargs ${bootargs} console=${console},${baudrate}
addmtd=setenv bootargs ${bootargs} ${mtdparts_kernel}
nandargs=setenv bootargs ubi.mtd=4 root=ubi0:rootfs rw rootfstype=ubifs
boot_nand=run nandargs addmtd addtty addmisc; nand read ${loadaddr} kernel; bootm ${loadaddr}
boot_nand_backup=run nandargs addmtd addtty addmisc; nand read ${loadaddr} kernel-backup; bootm ${loadaddr}
bootcmd=run boot_nand
splashimage=0x180000
ethact=FEC
sn=14-36-001-627
pn=6500-1400-2001
software_compatibility=0
manufacture_date=2014/09
ethaddr=00:0F:11:03:A8:D4
addmisc=setenv bootargs ${bootargs} mem=${mem} lpj=999424
serverip=192.168.2.87     <<<<<<<<<<<<
gatewayip=192.168.2.254
netmask=255.255.255.0
ipaddr=192.168.2.3
[b]serverip=192.168.2.87   <<<<<<<<<<<<
[/b]bootargs=root=/dev/nfs rw nfsroot=192.168.2.87:/srv/nfs/toon,nfsvers=3,nolock,tcp console=ttymxc0,115200 loglevel=8 mtdparts=mxc_nand:512K@0x00100000(u-boot-env)ro,1536K(splash-image),3M(kernel),3M(kernel-backup),119M(rootfs) ip=192.168.2.3:192.168.2.87:192.168.2.254:255.255.255.0:toon::off panic=0
CrumNL
Starting Member
Starting Member
 
Posts: 17
Joined: March 2019

Re: Toon Recovery

Postby TheHogNL » Sat Sep 07, 2019 7:09 pm

The .1 is probably being sent to the toon after the dhcp from the router. And then overwritten with .87 because you specified that from commandline.
If you just type everything manually, so without the script. Does it then work? Just check the DEBUG what the script is trying to let the Toon to do. It is just a few uboot commands what you need to copy paste.
Member of the Toon Software Collective
User avatar
TheHogNL
Forum Moderator
Forum Moderator
 
Posts: 1164
Joined: August 2017

Re: Toon Recovery

Postby CrumNL » Sun Sep 08, 2019 12:41 am

The reason for me to check ToonRecovery was because I was unable to succesfully upgrade toon's firmware. This kept failing with a message that busybox.password could not symlink to /etc/passwd because it existed.

I wasn't able to get the nfs mount working, also not when trying it manually even though other devices had no problem with the same nfs share. So I noticed a new firmware and downloaded the latest update script. It failed updating up to a certain point, again with the /etc/passwd and another error.
I removed busybox.passwd from the system, did a update -f and noticed the error was gone.

So for now it looks like the reason for me to try the script is no longer applicable :) and for now won't test it further unless something gets messy again.
Still a useful tool and I learned a bit along the troubleshooting way.
Thanks for the nudges.
CrumNL
Starting Member
Starting Member
 
Posts: 17
Joined: March 2019

Next

Return to Toon Hardware

Who is online

Users browsing this forum: Google [Bot] and 1 guest