Software: Open Embedded archive and extensions

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, Toonz

RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Software: Open Embedded archive and extensions

Post by RomMon »

Hi marcelr,

For setting up a VM (planning for 50GB) for the Quby toolchain, I'm looking at:

- CentOS 7 64bit, minimal ISO (http://isoredirect.centos.org/centos/7/ ... l-1511.iso)
- suspect python 2.5 is the better choice, but don't know if rpm packages are available.
For the (other) required packages (just uded yum provide <package>):
cvs-1.11.23-35.el7.x86_64
subversion-1.7.14-10.el7.x86_64
git-1.8.3.1-5.el7.x86_64
texi2html-1.82-10.el7.noarch
docbook-utils-0.6.14-36.el7.noarch
didn't find a package for python-pysqlite2
help2man-1.41.1-3.el7.noarch
didn't find a package for build-essential
chrpath-0.13-14.el7.x86_64

For the optional packages:
didn't find a package for libxml2-utils
xmlto-0.0.25-7.el7.x86_64
didn't find a package for python-psyco
apr-1.4.8-3.el7.x86_64
didn't find a package for gcc-multilib
didn't find a package for libsdl1.2-dev

Any advice to use a different version?
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Toon as a domotica controller?

Post by marcelr »

Not sure, my laptop is a native CentOS 7 machine, I have all possible development tools installed (well, maybe not the ADA compiler).
Typically, you will need some EPEL packages to get things going (install the EPEL repo package, activate it, install the required packages, and you're good to go).

The versions I have are (all the rest of the listed packages in your post are absent on my machine):

cvs-1.11.23-35.el7.x86_64
subversion-1.7.14-10.el7.x86_64
git-1.8.3.1-6.el7_2.1.x86_64
texi2html-1.82-10.el7.noarch
docbook-utils-0.6.14-36.el7.noarch
chrpath-0.13-14.el7.x86_64

python-2.7.5-38.el7_2.x86_64
apr-1.4.8-3.el7.x86_64

Naturally, you will need a minimal native development toolchain:

binutils, gcc, glibc, glibc-devel, kernel headers.

Then, I had to edit the bb_env.sh script a little, comment out the path to the OE-version of python, to use the native one:

Code: Select all

...
#echo $PATH | grep -q Python ||
#	PATH="$OEBASE/Python-2.6.6:$PATH"
...
Then follow the README file and build the rootfs image and kernel image. You won't need either of them, but this step is necessary to build the cross-compilation toolchain, cross-compiled base libraries etc. And the README file is not exaggerating: it takes a long time, with lots of time lost in downloading stuff. A fast network connection is therefore preferable (wired).

After that, disk space requirements are something like this:
[toon]$ du -ks
31 857 376 .

(30GB or thereabouts, although I added some packages to the original tree).

Some packages are possibly outdated, when bitbake can't find them, you can try and find them manually, drop them into <build_root>/oe/sources, and bitbake will find them there.
If all else fails, just ask, and I will try and get them to you, since I have a complete, working build system (not sure if the forum server will take them, though). I had to find 3 or 4 source packages manually, IIRC. And that was more than a year ago, so probably more have been removed in the meantime.

Maybe I should squeeze this into some manual, some day ..


good luck,

marcelr
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Toon as a domotica controller?

Post by RomMon »

Hi marcelr,


Do you remember issues with compiling package m4-native-1.4.14-r0.1?

I get:

Code: Select all

NOTE: package m4-native-1.4.14-r0.1: task do_compile: Failed
Looking in the log:

Code: Select all

./stdio.h:143:1: error: 'gets' undeclared here (not in a function)
Suspect is caused by 'm4-native-1.4.14-r0.1/m4-1.4.14/lib/stdio.h' itself

Code: Select all

# find / -name stdio.h
/usr/include/bits/stdio.h
/usr/include/stdio.h
/usr/include/c++/4.8.2/tr1/stdio.h
~/toon/oe/qb2/tmp/work/x86_64-linux/linux-libc-headers-native-2.6.30-r2/linux-2.6.30/arch/powerpc/boot/stdio.h
~/toon/oe/qb2/tmp/work/x86_64-linux/perl-native-5.10.1-r10/perl-5.10.1/mint/stdio.h
~/toon/oe/qb2/tmp/work/x86_64-linux/m4-native-1.4.14-r0.1/m4-1.4.14/lib/stdio.h

Code: Select all

/* It is very rare that the developer ever has full control of stdin,
   so any use of gets warrants an unconditional warning.  Assume it is
   always declared, since it is required by C89.  */
#undef gets
_GL_WARN_ON_USE (gets, "gets is a security hole - use fgets instead");
can you verify if you changed above part of the file <your-build-dir>/oe/qb2/tmp/work/x86_64-linux/m4-native-1.4.14-r0.1/m4-1.4.14/lib/stdio.h to get this working?
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Toon as a domotica controller?

Post by marcelr »

Just moved the posts on the OpenEmbedded archive to a seperate thread.

In case people don't know, it's about this file:

http://quby.com/static/openembedded-qb2 ... r1.tar.bz2

Now, to answer your question:

No, I didn't.
I have to admit that most of my oe-tree was built on a CentOS6 box, and transplanted to my new machine just before the summer holidays. It runs flawlessly so far, although I haven't built it from scratch on that machine.

Could it be that you are missing some headers?
Otherwise, gets is being deprecated, and possibly that's already the case in your version distro.

From the man page:

Code: Select all

...
CONFORMING TO
       C89, C99, POSIX.1-2001.

       LSB deprecates gets().  POSIX.1-2008 marks gets() obsolescent.  ISO C11
       removes the specification of gets() from the C language, and since ver‐
       sion  2.16, glibc header files don't expose the function declaration if
       the _ISOC11_SOURCE feature test macro is defined.

BUGS
       Never use gets().  Because it is impossible to tell without knowing the
       data  in  advance  how  many  characters  gets() will read, and because
       gets() will continue to store characters past the end of the buffer, it
       is  extremely  dangerous  to  use.   It has been used to break computer
       security.  Use fgets() instead.
...
Not sure if _ISOC11_SOURCE is defined, otherwise you may want to #undef it somewhere in the m4 recipe file, by passing extra arguments to oe_runmake().

best,

marcelr
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Software: Open Embedded archive and extensions

Post by RomMon »

Hi Marcelr,

I just removed the part containing 'gets'
This issue occurred 5 times so far.
For a next occurrence I will check if adding #undef _ISOC11_SOURCE just below the header works.

Now at task 267 of 1431, but now after every few task there is an issue due to deprecated/obsolete syntax.
This is going to take a while...
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Software: Open Embedded archive and extensions

Post by marcelr »

Hmm ...

At the risk of discouraging you completely, maybe it's better to start again, with a CentOS 6 VM. As far as I can see, you are having issues with the build of the cross-compilation toolchain and everything that goes with it.

As said, I did the initial build on a CentOS 6 machine, the only issues I had were the the need to use the native python installation rather than the one shipped with the archive, and a few downloads that were no longer available. The completed build was copied to a CentOS 7 machine, into exactly the same directory. That's what I'm working with now. Had no issues so far. So, probably, once the toolchain is ready for use, you can use it on a wider range of machines/distros than just the one it was built on.
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Software: Open Embedded archive and extensions

Post by RomMon »

Managed to finish setting up an OpenEmbedded toolchain in an VM on my Win7 pc.

On advice from marcelr I used CentOS-6.8. (I stared with CentOS-7.0, but ran into too many issues due to depreciated/obsolete syntax.)
With CentOS-6.8 I only encounter some VM related issues, and the expected unavailable downloads for some packages.

You can use the minimal CentOS ISO for this.
Obtain the ISO from http://mirror.widexs.nl/ftp/pub/os/Linu ... inimal.iso
  1. Defragment disk
    Advice to defragment the disk to prevent VM crashes (see note at the end).
    It is probably also advisable to have more free space available than the disk size you plan to configure, specifically if you are linking to use the dynamically allocated size option.
  2. Create a new VM e.g. VirtualBox:
    2GB memory (tried with 768MB first but that quickly turned out to be too low)
    30GB or more disk size. (I used 80GB, with a dynamically allocated size)

    While installing make sure to assign sufficient space to /home.
    (I changed the partitions to have all space assigned to the root (/))

    After installation of CentOS-6.8 finished:
  3. ssh access
    I advice to configure your terminal program (e.g. PuTTY, SecureCRT) to capture the session output in a log file.
    (you can configure a filename based on hostname and timestamp e.g. for Putty: &H__&Y&M&D__&T.log, and for SecureCRT you can even add the session name e.g.: %S__%H__%Y-%M-%D--%h-%m-%s.log )

    (starting as root)
  4. Install additional packages

    Code: Select all

    yum install wget tree
    
  5. Install the necessary packets for OpenEmbedded
    The needed packages are described here
    (In below list I also use yum provides, just to get a little more info where the package will come from, and which version/arch.)
    1. First the required packages:

      Code: Select all

      yum provides cvs
      yum provides subversion
      yum provides git-core
      yum provides texi2html
      yum provides docbook-utils
      yum provides help2man
      yum provides chrpath
      
      yum install cvs subversion git-core texi2html docbook-utils help2man chrpath
      
    2. The next two are optional according the list on OpenEmbedded:

      Code: Select all

      yum provides xmlto
      yum provides apr
      yum install xmlto apr
      
    3. I found that the following are also needed:

      Code: Select all

      yum provides texinfo
      yum install texinfo
      
      yum provides glibc-static
      yum install glibc-static
      
      yum provides perl-ExtUtils-MakeMaker
      yum install perl-ExtUtils-MakeMaker
      
      The following for 'make menuconfig' to work:
      yum provides  ncurses-devel
      yum install  ncurses-devel
      
    4. A larger required one that I prefer performing at the end

      Code: Select all

      yum groupinstall "Development Tools"
      
    5. marcelr also advised to install the EPEL repository:

      Code: Select all

      wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
      rpm -ivh epel-release-5-4.noarch.rpm
      
      To verify:

      Code: Select all

      yum repolist
      cat /etc/yum.repos.d/epel.repo
      
      To install packages from this repo the following commands are handy:

      Code: Select all

      yum --enablerepo=epel info <package>
      yum --enablerepo=epel install <package>
      
  6. Create a non-root user
    Before installing the OpenEmbedded toolchain create an user:

    Code: Select all

    useradd <user>
    passwd <user>
    
  7. Follow the Quby readme
    The next steps are mostly from the readme:

    Code: Select all

    su - <user>
    mkdir toon
    cd toon
    wget http://quby.com/static/openembedded-qb2-toon-2012r1.tar.bz2
    tar jxf ~/toon/openembedded-qb2-toon-2012r1.tar.bz2
    cd ~/toon/oe/
    vi bb_env.sh
    comment out the following two lines:
           echo $PATH | grep -q Python ||
           PATH="$OEBASE/Python-2.6.6:$PATH"
    cd ~/toon/oe/qb2
    . ../bb_env.sh
    cd ~/toon/oe/sources
    wget https://s3.amazonaws.com/net.energyhub.assets/public/sources/git_git.savannah.gnu.org.config.git_e35217687ee5f39b428119fe31c7e954f6de64f0.tar.gz
    wget https://s3.amazonaws.com/net.energyhub.assets/public/sources/git_git.savannah.gnu.org.config.git_e35217687ee5f39b428119fe31c7e954f6de64f0.tar.gz.md5
    wget http://ftp.netfilter.org/pub/libnfnetlink/libnfnetlink-1.0.0.tar.bz2
    wget http://www.mucross.com/downloads/tonga-linux/sources/GPL/netbase/netbase-4.21.tar.gz
    wget http://www.mucross.com/downloads/tonga-linux/sources/GPL/netbase/netbase-4.21.tar.gz.md5
    wget http://developer.digitalstrom.org/download/oe-sources-mirror/trunk_opkg.googlecode.com_.svn_635_.tar.gz
    echo 5695fca9d9e6a0aa55403f72ce245aa3 > trunk_opkg.googlecode.com_.svn_633_.tar.gz.md5
    wget http://developer.digitalstrom.org/download/oe-sources-mirror/trunk_opkg.googlecode.com_.svn_633_.tar.gz
    echo 94f839250d383f788aaa50e21ed5aa41 > trunk_opkg.googlecode.com_.svn_635_.tar.gz.md5
    
    cd ~/toon/oe/qb2
    bitbake -v image-base-qb2
    
  8. Resulting image files
    If you are lucky bitbake will finish without errors and you have a rootfs image, and kernel image:

    Code: Select all

    ls –lh /toon/oe/qb2/tmp/deploy/images
    
  • Some of the challenges I ran into:
  • VM crash due to BLKCACHE_IOERR:
    A number of times my VM crashed with Error ID: BLKCACHE_IOERR. According VirtualBox this is due to a heavily fragmented disk. (source)
    I ran into this a number of times until I freed up 300GB (YMMV).
    According the error message this is ‘non-fatal’ but once I had to start over again because the yumdb got corrupted, which according google is a xfs filesystem issue, but could not be repaired by xfs_repair.
    So defragment initially, and after each time you observe this issue.
  • Bitbake Compile and/or Configure task failed:
    My VM also crashed once (appeared also disk related) while bitbake was at task 577 (of 1431) causing the compile and/or configure step to fail.
    Now the session log is handy to have.
    In the session log search for the related tasks. In my case is suspect it was the compile step of package ‘libpng’.

    Code: Select all

    Running task 473 of 1431 (ID: 433, ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb, do_setscene)
    Running task 474 of 1431 (ID: 432, ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb, do_fetch)
    Running task 475 of 1431 (ID: 426, ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb, do_unpack)
    Running task 575 of 1431 (ID: 427, ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb, do_patch)
    Running task 576 of 1431 (ID: 434, ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb, do_configure)
    Running task 577 of 1431 (ID: 435, ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb, do_compile)
    
    Try to get an idea which step introduced the failure:
    1. Verify the download directory ~/toon/oe/sources/ for a correct download (e.g. using md5sum)
    2. Verify the unpacked archive ~/toon/oe/qb2/tmp/work/qb2-hae-linux-gnueabi/<packagename>
    3. Easiest next step is to move (or delete) the unpacked archive and re-perform the unpack, patch, configure and compile steps (added parameters to get more debug output)
    4. From below list, start with the failed step, and continue till all tasks have been completed successfully:

      Code: Select all

      bitbake -v -DDD -b ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb -c setscene
      bitbake -v -DDD -f -b ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb -c fetch
      bitbake -v -DDD -f -b ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb -c unpack
      bitbake -v -DDD -f -b ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb -c patch
      bitbake -v -DDD -f -b ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb -c configure
      bitbake -v -DDD -f -b ~/toon/oe/openembedded/recipes/libpng/libpng_1.2.44.bb -c compile
      

Next would be to cross-compile a program myself, and look how to create an ipk…
Last edited by RomMon on Thu Sep 29, 2016 9:34 pm, edited 1 time in total.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Software: Open Embedded archive and extensions

Post by marcelr »

Good tutorial, thanks.
The oe tree for toon is slightly outdated, and relevant documentation is hard to come by. The whole openembedded project suffers a little from poor documentation supply, at least for absolute beginners in openembedded (like me).
Anyway, after some trial and error, I found a way to build .ipk packages.

.ipk files are made through bitbake recipes. In the toon oe tree, these reside in

<oe root>/oe/openembedded/recipes/<package_name>/<package_name>_<package_version>.bb
and
<oe root>/oe/homeautomationeurope/recipes/<package_name>/<package_name>_<package_version>.bb

In recipe file <package_name>_<package_version>.bb, the underscore is significant, and the only allowed separator between <package_name> and <package_version>.bb. (That took some time to find out)
To start building packages, as explained in the README file that comes with the openembedded tree, go to

<oe root>/oe/qb2/

and execute

. ../bb_env.sh

This sets some environment variables for the toolchain, after that, you're ready to go.

So, for example, the x11vnc package was built using
<oe root>/oe/homeautomationeurope/recipes/x11vnc/x11vnc_0.9.13.bb

and then simply built by issuing:

bitbake x11vnc
while in <oe root>/oe/qb2/

bitbake looks for files in

<oe root>/oe/homeautomationeurope/recipes/<package_name>/files/

if inside the recipe, you state that the package can be found as a file rather than a download.

E.g., the x11vnc recipe looks like this:

Code: Select all

SUMMARY = "x11vnc"
HOMEPAGE = "http://www.karlrunge.com/x11vnc/"
LICENCE = "GPLV2"
SECTION = "console/utils"
SRC_URI = "file://x11vnc-0.9.13.tar.gz \
	   file://x11vnc-0.9.13/x11vnc.sh"


do_configure (){
	     ./configure --without-x  --without-v4l \
	     --without-macosx-native --without-avahi \
	     --host x86_64
}

do_compile () {
	   make CFLAGS+='-DSMALL_FOOTPRINT=3 -DNOGUI'
}


do_install() {
	install -d ${D}${bindir}
	install -m 0755 x11vnc/x11vnc ${D}${bindir}/x11vnc-bin
	install -m 0755 x11vnc.sh ${D}${bindir}/x11vnc
	
}
I added x11vnc to the homeautomationeurope/ directory, since this one has the least entries, so your own stuff is easier to find. You can add extra recipe directories to openembedded, but I haven't found the right way to do that (didn't try very hard).
So, in this case, x11vnc-0.9.13.tar.gz can be found in
<oe root>/oe/homeautomationeurope/recipes/x11vnc/files/x11vnc-0.9.13.tar.gz
and the script to start up x11vnc resides in
oe root>/oe/homeautomationeurope/recipes/x11vnc/files/x11vnc-0.9.13/x11vnc.sh

AFAIK the minimal information that needs to be set in a recipe (besides the header) is:

do_install(){ ... }
I always have the header of the recipes like in the example, I'm not sure what part of the header is really mandatory. This seems to work OK, so far.

do_install() is required to tell bitbake which files to include in the .ipk package, and where they should be installed during installation. Without it, everything is made, but nothing is packed into an .ipk.

do_configure() and do_compile() are only needed if you have a non-default configuration or want to add extra flags to make.

If you want to give it a shot, just download x11vnc-0.9.13.tar.gz, put it in the right spot in the openembedded tree, copy the recipe above to x11vnc_0.9.13.bb, put it in the right spot inside the tree as well, set the environment as stated above, run bitbake as stated, and if all goes well, you get your own freshly made version of x11vnc, specific for toon.

When bitbake fails, the best way to clean up is by issuing:

bitbake -c clean <package_name>

and start over again (after fixing the error, of course).

have fun,

marcelr
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Software: Open Embedded archive and extensions

Post by RomMon »

Hi marcelr,

Thanks, that will make it much easier than the approach I was taking.

Yesterday I created the .ipk's for getting USB for FAT working by creating a whole new oe buildtree.
(e.g. 1. making changes with your example as guide, 2. bitbake image-base-qb2 3. wait forever)
It worked, but well... takes forever

Just now made, and instlled a .ipk for tree (using the existing bitbake script without changes) in just minutes.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Software: Open Embedded archive and extensions

Post by marcelr »

To build kernel modules, you don't have to build all images again.

What I did was the following: extract the kernel package to a separate directory,
execute: make menuconfig

(for this step, probably your cross compilation toolchain needs to be in your $PATH)

configure the kernel the way you want it. Copy the .config file to the oe tree ( <oe_root>/oe/homeautomationeurope/recipes/linux/linux-quby2/ with the right name: defconfig-2.6.36-<whatever version of the kernel you need> )

edit linux-quby2-2.6.36-R07.bb and its included files to point to your config file AND reflect the kernel version of your toon.

bitbake linux-quby2

and a new kernel + modules are built and packaged, a lot faster than rebuilding everything from scratch.

For most modules or extra features, you will need a new kernel, since toon by default has module versioning switched on. This means that you can only add modules that are part of the original kernel configuration of toon. For features outside this configuration, switch module versioning off, configure your kernel according to your needs, build it, and install. If you give it the same version number as the current kernel version, there's a strong possibility that you can use the newer zwave and wifi drivers as supplied by quby, with your homebrew kernel (haven't tried this yet).

good luck,

marcelr
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Software: Open Embedded archive and extensions

Post by RomMon »

Thanks.

Was able to run 'make menuconfig' (after installing ncurses-devel).

Did you enable any other features in your kernel (other than cp437)?

(not planning to replace the kernel...yet.)
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Software: Open Embedded archive and extensions

Post by marcelr »

Yes, support for a lot of file systems, codepage 437 support to get vfat working properly, automount support, whatever.
As long as you don't disable fs support for the rootfs (UBIFS in this case) and all other stuff you need to boot, you should be good. So basically, leave everything that's built in, built in, and just add modules according to your needs.
(and switch off module versioning).

best,

marcelr
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Software: Open Embedded archive and extensions

Post by RomMon »

Hi Marcelr,

Just curious if you tried the x11vnc bitbake script part of the OpenEmbedded download?
Compiled this one (more or less by accident), which took veeeryyy long (lots of dependencies), and resulted in a 1.3MB file.

Also 'replayed' your creation of x11vnc, and this works great.
(I have not found a vncviewer for iPhone yet, and on Win7 the tigervnc (1.7) doesn't seem to update its screens.)
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Software: Open Embedded archive and extensions

Post by marcelr »

Nope, never noticed that one.
Funny, now you mention it, the dependencies indicate full use of an X11 server plus some other stuff that's not available on toon. Does it work at all?

best,

marcelr
RomMon
Starting Member
Starting Member
Posts: 44
Joined: Mon Aug 29, 2016 9:05 pm

Re: Software: Open Embedded archive and extensions

Post by RomMon »

Haven't tried...

Btw, I did find a vncviewer for Windows: http://www.karlrunge.com/x11vnc/ssvnc.html
(from the same developer as the server)
Post Reply

Return to “Toon software development”