Porting Kivy, a multi-touch UI Framework, to Colibri T20

Posted by Vijaykumar | Fri, 03 Jul 2020 | Tagged: linux, yocto, kivy

Linux is the most popular operating system in embedded devices, it is being used in various products including hand-helds, medical and Industrial HMIs devices. When it comes to natural user interface, people tend to choose the Android framework, but Android, is too heavy for buiding just a HMI. A popular alternative is the Python based Kivy.

Introduction

Kivy is a cross-platform library for development of touchscreen based GUI appliactions, written in Python. Kivy uses every trick in the book to speed up code execution and graphics rendering, and hence despite being written in Python, is extremely fast.

While there are other GUI frameworks like GTK+ and Qt, we decided to use Kivy, due to the following reasons:

  • Unlike GTK+, Kivy was designed ground up for use with touchscreen based systems. For example, Kivy has built-in support for on-screen keyboards. List scrolling is tuned for touch screen inputs, for example. And many other widgets can work with multi-touch as well.

  • Qt needs a commercial license if you are building a closed source product. Kivy does not have the licensing issues, that Qt has.

  • Kivy is a Python native framework, unlike GTK+ and Qt which are wrappers over underlying C / C++ libraries. And is easy to setup a development and testing environment across operating systems, using Python’s development ecosystem.

Again, building it for a embedded platform is not straight forward, and we need a deeper knowledge about Kivy to port it. In this article, we will discuss how we managed to port Kivy to Yocto, for a system based on the Colibri T20 module. (NVIDIA Tegra 2 SoC)

/static/images/kivy-featured.jpg

Picture Courtesy: Kivy Project

Toradex, Yocto BSP

Yocto is a collection of open source projects for creating embedded Linux distributions. There are two popular distributions Poky and Angstrom. In this article we assume that you are familiar with creating recipes for Yocto.

Colibri T20 is a module from Toradex, based on NVIDIA Tegra 2 SoC. Toradex provides a Angstrom v2017.12 based distribution for the Colibri T20. This distribution currently does not provide Kivy out of the box, nor does the latest version of Angstrom or Poky, as of this writing.

/static/images/colibri-t20.jpg

Picture Courtesy: Toradex

While installing and using Kivy on a desktop or mobile, is pretty straight-forward, porting Kivy to Angstrom, is non-trivial. Porting and running Kivy on Angstrom, is complicated thanks to Kivy’s exotic dependencies. This article will explain these dependencies, and how we managed to port Kivy to the Angstrom distribution.

Kivy Internals and Dependencies

Cython

Though Kivy is a Python based framework, it is accelerated under the hood by Cython. Cython is a superset of Python, that allows Python code to be compiled and optimized to C. Cython is also used by Kivy to wrap C libraries, and provide a high-level Pythonic API.

/static/images/cython-module.png

Released versions of both Poky 23.0.1 and Angstrom do not have have recipes for Cython. Fortunately the latest version of Poky, has added support for Cython, that we can leverage.

Another sore point with Cython, is that each version of Kivy is tested to work with a specific version of Cython. This is documented in Kivy’s Cython page. For example, the latest version of Kivy 1.11.1 is tested to work with Cython 0.29.9.

OpenGL ES 2

Kivy uses OpenGL to accelerate graphics rendering and requires a minimum of OpenGL 2.1 or OpenGL ES 2. Most people would recognize OpenGL as an API for interacting with the GPU. But there are multiple flavours of the OpenGL API, and the one designed for used in Embedded Systems is OpenGL ES 2.

NVIDIA provides an OpenGL ES 2 implementation for its Tegra 2 SoC, that we can use with Kivy. This is already part of the layers provided by Toradex, so nothing much needs to be done here.

Providers

Another key aspect of Kivy that needs to be understood, when porting to Yocto, is the concept of providers. There could be multiple implementations of a functionality that is required by Kivy. For example, for rendering font, which is called the Text Provider, there are 4 different providers.

  • pangoft2 - Pango + FreeType 2

  • pygame - Python Gaming library based on SDL2

  • pil - Python Imaging Library

  • sdl2 - SDL2 TTF based font rendering

Atleast one of these providers should be available for font rendering to work. Similiarly there are various different functionalities in Kivy that can have multiple providers. Some of the required functionalities and the available providers are listed below.

  • audio

    • android, avplayer, ffpyplayer, gstplayer, pygame, sdl2

  • image

    • sdl2, dds, ffpyplayer, imageio, pil, pygame, sdl2, tex

  • window

    • sdl2, pygame, x11, egl_rpi

Having multiple providers like these, allows Kivy to work across different platforms, which makes Kivy a truly cross-platform UI framework. If you look at the above, the sdl2 provider provides pretty much every functionality required by Kivy. This reduces the problem, to just ensuring the recipes for SDL2 is available. Fortunately for us, SDL2 and its support libraries are already availabe in Angstrom distribution.

Porting Kivy

To summarize the discussion from the previous section, the following are required by Kivy.

  1. Cython

  2. OpenGL ES 2 API

  3. SDL2

Cython Recipe

For Cython, we backport the recipes from Poky git. The backported recipe files are avaiable from /static/code/kivy-t20/python3-cython.inc[python3-cython.inc] and /static/code/kivy-t20/python3-cython.inc_0.29.10.bb[python3-cython_0.29.10.bb].

OpenGL ES 2

The OpenGL ES 2 libraries for Tegra is already available, we can enable with DISTRO_FEATURES_append = " opengl", the local.conf file.

SDL2 Recipes

We need to ensure SDL2 is built using OpenGL ES 2. This can be acheived by setting PACKAGECONFIG_GL to gles2. We write a libsdl2_2.0.5.bbappend to augment the existing recipe.

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

PACKAGECONFIG_GL = "gles2"

The Angstrom distribution also has the recipe for other SDL2 libraries, like libsdl2-image, libsdl2-ttf, libsdl2-mixer, etc, which are required for the various Kivy providers.

We hit a road block with libsdl2-image though, as the libsdl2-image version 2.0.1 available in Angstrom, did not support PNG! So we had to move to the latest version of libsdl2-image 2.0.5.

But this again lead to another issue, libsdl2-image 2.0.5 requires libsdl2 2.0.8, while the version on Angstrom was libsdl2 2.0.5. We were able to get libsdl2-image 2.0.5 to build with libsdl2 2.0.5, if pnm and svg support were disabled, in libsdl2-image. So we hacked the version check in libsdl2-image with a patch, and modified the recipes to disable the png and svg support using --disable-pnm --disable-svg option.

Kivy Recipe

The Kivy recipe is provided below. In the depends, we mention python3-cython-native, since Cython is required in the build system, and is not required at runtime.

SUMMARY = "Python3 Package Kivy"
inherit pypi setuptools3

DEPENDS = "python3-cython-native mesa libsdl2 libsdl2-image libsdl2-ttf libsdl2-mixer"

PYPI_PACKAGE = "Kivy"

LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=d273d63619c9aeaf15cdaf76422c4f87"

SRC_URI[sha256sum] = "4d0e596f74271e901b551f77661dde238df4765484fce9f5d1c72e8022984e84"

RDEPENDS_${PN} = "mesa \
                  libsdl2 \
                  libsdl2-image \
                  libsdl2-ttf \
                  libsdl2-mixer \
"

Conclusion

With software complexity increasing over time, build and integrating software for embedded Linux distributions, just gets painful by the day. We hope this article has given you the required background, and has given you an idea of what is involved in porting and using Kivy for your embedded project.

If you are interested in using Kivy for your embedded Linux projects or porting any packages to Yocto, in general, do get in touch with us at sales@zilogic.com