Danny Keller, guided by Dr. Dan Feldman
This project’s goal is to recognize a QR-Code using the GoPro camera, which the 3DR Solo drone uses. The first motivation to this idea came from another project in the RBD lab, the ParkingDrone, which dealt with finding an empty parking spot using a drone. Many times even if we did find a place to park our car, there still could be another problem afterwards – remembering where exactly that place were. The idea is to encode the license plate number of the car in QR-Code and paste it on the car’s roof, and then use image processing in order to recognize the correct car from the air. The Solo was picked for this assignment as it has an autonomous computer onboard, which allows making the calculations on the original footage, without loss of quality over transmission.
From Wikipedia: QR code (abbreviated from Quick Response Code) is the trademark for a type of matrix barcode (or two-dimensional barcode) first designed for the automotive industry in Japan. A barcode is a machine-readable optical label that contains information about the item to which it is attached. A QR code uses four standardized encoding modes (numeric, alphanumeric, byte/binary, and kanji) to efficiently store data; extensions may also be used.
The QR code system became popular outside the automotive industry due to its fast readability and greater storage capacity compared to standard UPC barcodes. Applications include product tracking, item identification, time tracking, document management, and general marketing.
A QR code consists of black squares arranged in a square grid on a white background, which can be read by an imaging device such as a camera, and processed using Reed–Solomon error correction until the image can be appropriately interpreted. The required data is then extracted from patterns that are present in both horizontal and vertical components of the image.
The Solo is a 2 kg drone designed by 3DR, which is capable of approximately 16 minutes of flights at a maximum speed of 40 km/h and a maximum altitude of 122 meters (depends on flight regulations). its communication range with the controller is about 1 km. The Solo was designed uniquely for using GoPro cameras with it, for which it has a special fitting gimbal. What is especially important for the project’s goal is the following details:
- Solo uses an onboard ARM Cortex A9 (i.MX6 Solo by Freescale) 1Ghz computer, with 1 CPU core and with VPU and GPU.
- This computer runs a 3DR Poky Linux, which is based on Yocto Project Reference Distro, Poky 1.5.1.
- Solo comes with support to Python 2.7.3
For additional details on Solo, visit https://news.3dr.com/solo-specs-just-the-facts-14480cb55722#.b4f2dv5vc.
Working with Solo
Generally speaking, 3DR has a development guide for developing using Solo, which can be found at https://dev.3dr.com/index.html. Alas, many of the “innocent looking” instructions there simply doesn’t work, some being based on outdated architecture, some without any seemly reason.
Solo comes with a command line cold “Solo CLI”, which provides tools to install packages and scripts to Solo, connect Solo to the internet and others. Sadly, as stated above some of the tools provided do not work properly, which makes it not trivial at all to achieve “working relationship” with the drone. Those issues are widely discussed over to net, and many discussions can be found using Google.
The first thing that has to be done – after establishing a working ssh connection between the computer and the drone through Solo’s wifi network – is to repartition Solo’s disk, as the ‘root’ partition arrives with only 90 MB on it, preventing almost any work to be done directly on the Solo. The CLI supposedly provides a tool for that, but in reality, it not only hangs but also makes it inevitable to factory reset the drone. Thus, the repartition should be done manually. One way to do it – which I used – is to follow the script provided by 3dr and try to execute it directly on Solo line by line.
Another major issue is accessing the video stream in Solo. The drone is designed such that it is impossible to use the video stream for scripts and the like, as long as it is being sent to the Solo app. To gain access to the video one needs first to disable the stream to the app. The CLI provides a tool to do so, but after some reverse engineering it has been proved in the forums through the net that that tool does absolutely nothing, since it had been written for an architecture that since then was vastly changed, leaving the discussed tool outdated, yet 3DR did not publish any update for it. Users eventually have found a way to manually gain access to the video stream (thus disabling streaming to the app), and I used this method in this project (see the github repo for instructions). It still remains a serious problem that one cannot simultaneously view the footage from the solo in the app, and use scripts on it. I have put a lot of effort in trying to solve this issue, and did came across experiments to split the stream pipe into two, but none of them had worked on the Solo I used.
As for script languages, Solo officially supports only Python, but using the Smart package manager (which can be installed on Solo using the CLI) a support for C programming can also be achieved, which I have done. After all this is Linux, so many of what can be done on a standard Linux machine can be done here. Further more detailed instructions can be found on the github repo.
I use the ZBar library in order to identify and decode the codes. ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code. More information can be found at http://zbar.sourceforge.net/.
Though ZBar has the ability to read codes from video streams, it was last updated in 2010, hence many if its features cannot be used on the Solo. For instance, ZBar is written to work with the v4l1 video for linux interface, while the linux Solo supports only v4l2. I have put great effort in trying to build the ZBar library from source on the Solo itself (which required building many dependencies etc. from source also), thus being able to use the ZBar utilities on the footage the second it is taken and without loss of quality, but as to this point with no luck. ZBar is moderately obsolete, and building it on the newer Solo linux is sophisticated. Having no real time support neither for ZBar nor for Solo doesn’t help too of course. Thus, as for now what I do in the project is saving a frame every two seconds from the Solo GoPro footage and using a script that downloads it (every two seconds) to the local computer, where another python script uses ZBar to detect and read the QR-Code in the frame. The script checks whether the frame contains a code and whether that code contains the requested license plate number, and if it does, the script says so and stops, otherwise keeps checking the next frame.
In the near future
The significance of this project lies in opening a whole new horizon to drone engineering, seeing as there is almost no limit to the kind of information that can be encoded in QR-Codes. One for example can encode flying directions or coordinates in a sequence of codes lying on some route, and make a drone follow this route autonomously buy “reading” the directions from the codes, and that is just the beginning. The minute this ability of decoding QR-Codes had been achieved, a large variety of applications became possible.
The Github repository: