Hello everyone, my name is Rik and I’m Bitcraze’s new intern. I took up this internship as part of my MSc studies at the faculty of Aerospace Engineering at the Delft University of Technology. Over the past year I have worked with the AI-deck as part of my thesis work at the MAVLab, and I’m looking to contribute to Bitcraze with my practical knowledge of the platform.
For my MSc thesis I have primarily worked on the design of very small optical flow CNNs, capable of running on the GAP8 SoC, using a neural architecture search (NAS). I have implemented a pipeline on the AI-deck, feeding a stream of camera images into the CNN to determine optical flow on-board. One of the last remaining goals of my thesis work is to design an application which utilizes the resulting dense optical flow. In the meantime, the NAS is ever running to find the best possible architecture.
Image 1
Image 2
Optical flow color coding
Resulting optical flow
On-board optical flow estimation on the AI-deck. A copy of The Hitchhiker’s Guide to the Galaxy, a DS-2Turbo Distortion guitar effects pedal, and a water bottle that moves in the top-left direction between image captures.
With my practical knowledge I hope to contribute to making the AI-deck an easier platform to work with. Of course, working at Bitcraze is a great learning opportunity, too. I’m already learning a lot about embedded systems and programming. After several years of studying, it’s great to get my first relevant working experience. And maybe most important of all, so far it has been a lot of fun working at Bitcraze and I expect to have a lot more of it. And yes, the falafel is really good.
The AI-decks are back in stock! Also, last week we had our quarterly meeting, where we plan our focus for the next quarter. As it is also the start of the fiscal year, we also take this opportunity to update our 1 year and 3 year plans as well. We have a big plans coming up, but one of the important focuses that we will have this year, is to get the AI-deck out of early-access!
But what would be necessary for such a task? The AI-deck is one of the most complicated boards we have worked with, so do we have to evaluate its ‘out-of-readiness’ along the same standards than any of our other products.
Mega AIdeck Tutorial
So one of our idea is to be able to achieve a state of the AIdeck in order to write a mega AIdeck tutorial series. This implies that we are able to show how somebody could go from a datasets all the way to a flying aideck-crazyflie combo. Such a series could consist of the following topics:
How to go from a dataset of images to a (Deep) Neural Network
Testing the DNN on the computer with the Image WiFi examples
Converting the neural network to Tensorflow Light (with basics on Edge AI and quantizing neural networks)
AIdeck basics (How to access the camera, how to add the network to the cluster, how to send commands)
Build and flashing the AIdeck and testing it out in the hand
Attach the AIdeck to the crazyflie, make an app-layer application to fly and react on the image input.
From the first look of it, this sounds like it should be easy to do right? Actually, there are still much to be done in order to make this tutorial possible.
Replumbing the Communication
One of the more challenging aspects of the AIdeck as it now is, is that users need to buy a JTAG-enabled programmer in order to flash the GAP8 and the NINA module of the AI-deck. That is the reason why currently the AIdeck has these 2 x 10 pin jtag connectors attached, but ideally we would want to get rid of it completely. This means is that we need to have over air flashing of the GAP8’s binary and that the intercommunication of the NINA and AIdeck will become even more important.
Moreover, the communication protocol from the GAP8 to the STM32 of the Crazyflie is currently very basic, as of right now, it is only possible to send single characters. It might work in some situations, but what if you would like to send an array of values back to the Crazyflie, like the collision probability & steering angle like in PULP platform’s implementation of Dronet? And, would we like to keep on using two UART serial ports or perhaps just relay both NINA and GAP8 communication all through one? The later will make it easier for us to maintain the crazyflie-aideck communication but can perhaps introduce communication delays.
These are just a slice of the type of re-plumbing work for the AIdeck before we can even start our dream tutorial series, but at least it will give you an idea of what we are dealing with :)
Rik the Intern
From this week we have the honor of hosting Rik Bouwmeester for a couple of months. He is currently doing his Master Thesis at the MAVlab from the faculty of Aerospace Engineering of the TU Delft. Since he has experience of working the AIdeck before, he will be able to provide us with some user perspective and help us with the above mentioned issues. You can expect a blogpost from him soon!
The AI-deck consists of the GAP8 chip developed by Greenwaves Technologies. On their website there’s an explanation of development tools where you get a general understanding of what you can use. Also their GAP SDK documentation explains how to install and try out some of their examples as well, on both a GAP8 simulator on the computer or on the GAP8 chip on the AI-deck itself.
Recently we also added the AIdeck documentation to the Bitcraze website, generated from the docuemtnation already available in the Github repository. There’s still improvements to make, so if you find any issues or any additions needed, don’t hesitate to help us improve it. On the bottom there is an ‘improve this page’ link where you can give the suggested change, or notify us by posting on the issue list of the AIdeck repository.
Also note that we have a separate AI-deck category on our forum where you can search for or add any AI-deck related questions. Remember that posting the issues that you are having will also help us to improve the platform and hopefully soon get it out of Early Access.
Workshop by PULP-platform
On the 16th of April we hosted a workshop given by PULP-platform featuring Greenwaves Technologies. In the workshop the an overview of the AI-deck and GAP8 was given as well as going through some basic hands-on exercises. About 70 people joined the workshop and we were happy it was so well received.
The workshop is a great source of information for anybody who is just getting started with the AI-deck, so have a look at the recordings on Youtube and the slides on the event page. Also make sure to check out their PULP training page for more tools that also can be used on the AI-deck. A big thanks to the PULP-platform and Greenwaves Technologies for taking part in the workshop!
Also we would like to ask if anybody who joined the workshop, to fill in this small questionnaire so that we can get some more feedback on how it went and how we can improve for the next one.
The PULP lab has done many amazing research on the field of Edge ML and were one of the collaborators in the development of the AI-deck of which their work on the Pulp Shield was the main inspiration. For more information check out their guest blogpost and be sure to read their latest work on the AI-deck!
More over, they have been working on an opensource tools that also work on the GAP8chip which are must try-outs for any AI-deck users
DORY: Deployment ORiented to memorY (Github, Paper)
NEMO: NEural Minimizer for pytOrch (Github, Paper)
All in all, since they clearly know what they are talking about, they are more than qualified to teach the rest of us how to work with all this! Also check out Luca Benini’s keynote at RISC-V or this week at the TinyML summit if you would like to learn more about PULP!
Date and content Workshop
The workshop will be tailored to those that have just started to to work with the AI-deck however, we think it will be interesting for regular users as well. Note that the tools mentioned above will not be handled this time.
These are the topics that will be discussed:
Hardware explanation (Gap8 specifics and AIdeck)
Software Preliminaries (GapSDK,, VM)
Hands-on examples
The workshop will take approximately 2 hours and will be on 16th of April in the afternoon, but the exact specifics will be given at a later date. So make sure to already block it in your calendar and to sign up for more information!
It did take much longer than anticipated, but we finally managed to get the AI-deck 1.1 back in stock! We had some difficulties with the post-production testing and Chinese new year was also a main contributor to the delay, but we are now very happy that we have a batch of AI-deck’s which are ready to ship!
There are only minor changes between the AI-deck 1.0 and 1.1 which can be read about in the “AI-deck product update” blog post. Since the AI-deck 1.1 is now using the gray-scale version of the camera module and the AI-deck 1.0 was using the Bayer RGB version, we now also offer the camera modules as stand-alone products for those that rather have the other variant.
For those of you that already have the previous AI-deck 1.0 and rather wished for the gray-scale camera module, please send us an email at contact@bitcraze.io, and we will work something out!
The AI-deck 1.1 is still a early access product, so remember to post any question or problems you have on our forum in the AIdeck group or to check out examples / documentation on the Github Repo. We also are planning to organize an online workshop / tutorial for the AI-deck in the coming months. We will hopefully have more information about that soon, so keep an eye out on the blogposts!
It has been a while since we have done an update on the AI-deck! However, as announced in the New Year post, we do have some important changes coming up for the next batch of production. These changes include updated version of the GAP8 chip and switching to a gray-scale camera. We will call this version the AI-deck 1.1, but its firmware should be compatible with the previous version. Currently we don’t have a clear time-line of when we will receive the next batch but it will be soon.
Newer version of GAP8
Together with our collaborators at Greenwaves technologies, we decided to update the GAP8 chip to the latest version available. Since this required a microchip change on the product and to prevent any confusion, we decided to call this version the AIdeck 1.1.
The new version of the GAP8 chip includes several improvements, which resolves 1. conflict problem between uart and hyperbus and 2. Cluster DMA when doing the data transfer between L2 and L1 Memory. The last issue is considered a corner case will only occur if the AI-deck 1.0 is pushed to its limit with a large deep neural network. Although we expect that both issues will not affect the current users if you are experiencing any related problems with the previous version (AI-deck 1.0), please send us an email to contact@bitcraze.io.
Gray-scale Camera
In July we have given an update where we discussed the color camera and how to process it on the GAP8 chip. Even though at the first we were pretty optimistic on the possibilities with the camera, after discussing with the community, ETH zurich and Greenwaves T., we have decided to switch back to a gray-scale camera.
This is because most examples in the gap_sdk repo mostly assume a gray-scale camera. And even though the color camera would be good for image processing with anything of color, it is less ideal for neural networks on edge machine learning devices.
For those who committed themselves to the color camera module, do not worry! We are planning to still offer the color camera module as a separate product in our store! Also we will have a limited amount of grayscale cameras available for those who are unhappy with their AI-decks 1.0’s color camera.
Firmware, Docs and Examples
Since the AI-deck is still in early release, have all the code and documentation in this github repository. This contains all the start-up guides and keeps track of all the bugs and fixes. Some months ago, we managed to update all of this to the latest GAP SDK (3.8.1), which fixed the streamer issues we are having.
However, the quickest way on how to improve this Early Release product is to get feedback by its users. So if you are having any problems at all, do not hesitate to ask a question on to forum or to open up a issue in the repository!
With the raging pandemic in the world, 2021 will most likely not be an ordinary year. Not that any year in the Bitcraze universe has been boring and without excitement so far, but it is unusually hard to make predictions about 2021. Any how, we will try to outline what we see in the crystal ball for the coming year.
Products
What products are cooking in the Bitcraze pot and what tasty new gadgets can we look forward to this year?
Lighthouse
We did hope that we would be able to release the first official version of the Lighthouse system in 2020, but unfortunately we did not make it. It has turned out to be more complex than anticipated but we do think we are fairly close now and that it will be finished soonish, including support for lighthouse V2.
Once the official version has been achieved, we are planning to assemble an full lighthouse bundle, which includes everything you need to start flying in the lighthouse positioning system. This will also include the Basestations V2 as developed by Valve corporation, so stay tuned!
New platforms and improvements
We released the AI-deck last year in early release, but the AIdeck will be soon upgraded with the latest version of the GAP8 chip. For most users this will not change much but for those that really push the deep learning to the edge will be quite happy with this improvement. More over, we are planning to by standard equipped the gray-scale camera instead of the RGB Bayer filter version, due to feedback of the community. We are still planning to offer the color camera on the side as a separate product for those that do value the color information for their application.
Also we noticed the released of several upgraded versions of sensors for the decks that we already are offering today. Pixart and ST have released a new TOF and motion sensor so we will start experimenting with those soon which hopefully lead to a new Multiranger or Flowdeck. Also we are aware of the new DWM3000 chip which would be a nice upgrade to the LPS system, so we will start exploring that as well, however we are not sure if we will be able to release the new version of LPS in 2021 already.
One of the field that we have wanted to improve for a while but have not gotten to so far is the communication with the Crazyflie. The Crazyradio is using a quite old chip and the communication protocol has hardly been touched in years. There now exists a much more powerful nRF52 radio chip with USB port so it can give us the opportunity to make a new Crazyradio and, at the same time, rework the communication protocols to make them more reliable, easier to use and to expand.
People and Collaborations
Last year we have started several collaborations with show drone orientated business, which we are definitely moving forward with in 2021. For shows stability and performance is very important so with the feedback of those that work with that on a regular basis will be crucial for the further development and reliability of our products.
Moreover we would like to continue our close collaboration with researchers at institutes and universities, to help them out with achieving their goals and contribute their work to our opensource firmware and software. Here we want to encourage the community to make their contributions easy to use by others, therefore increasing the reproducibility of the implementations, which is a crucial aspect of research. Also we are planning to have more of our online tutorial like the one we had in November.
We also will be working with closely together with one of our very active community members, Wolfgang Hönig! He has done a lot of great work for the Crazyswarm project from his time at University of Southern California (USC) and has spend the last few years at Caltech. He will be working together with us for a couple of months in the spring so we will be very happy to have him. Moreover, we will also have 2 master students from LTH working with us on the topic hardware simulation in the spring. We are making sure that we can all work together in the current situation, either sparsely at the office or fully online.
In 2021, we will also keep our eyes open for new potential Bitcrazers! We believe that everybody can add her/his own unique addition to the team and therefore it is important for us to keep growing and get new/fresh ideas or approaches to our problem. Usually we would meet new people at conferences but we will try new virtual ways to get to know our community and hopefully will meet somebody that can enhance our crazy group.
Working from home
Due to the pandemic we are currently mainly working from home and from the looks of it, this will continue a while. Even though we think we have managed to find a way to work remotely that is fairly efficient, it is still not at the same level as meeting in real-life, so there is always room for improvement. Further more the lack of access to electronics lab, flight lab and other facilities when working from home, does not speed work up. We will try to do our best though under the current circumstances and are looking forward to an awesome 2021!
2020 was not an ordinary year, and through its different roller coaster motions, we’re proud of what we’ve accomplished. As 2021 grows nearer, it’s time to look back at this year and appreciate our accomplishments.
Community
The Crazyflie is still gaining a lot of interests in universities, and we’re always happy to discover what awesome things our users are doing with our products. One of our proudest moment was that a paper published in Science magazine used a Crazyflie in their research framework!
We actually made a montage of the different research videos where the Crazyflie, and we feel blessed to be part of those amazing feats. We were also quite surprised to see how the Crazyflie is implemented in different fields. Here is the video if you want to know more:
We’ve had stellar blogposts from guests this year, that you can read if you missed:
Of course, “community” this year means something different. We were sorry to not participate in the conventions and fairs where we usually meet a lot of interesting people, and had to find ways to do things differently. We now have a Discord server where people can join and discuss all things Bitcraze related!
Hardware
In the beginning of the year, we started right by releasing the Active Marker deck. Thanks to our collaboration with Qualisys, this new motion capture sensor can estimate the full body pose of the Crazyflie without unique marker positions or known starting positions.
A great deal of work was dedicated to the release of the AI deck in early access. Thanks to our newest deck, super-edge-computing is now possible on your Crazyflie !
Software
We improved our app API. We have seen it used more and more. Being able to add code to the Crazyflie without having to fork the code has proven to be quite useful and popular.
At the beginning of the year we finally managed to decode and use the Lighthouse V2 signals in order to get the Crazyflie to fly autonomously using lighthouse V2 basestations. Our focus for the end of year was to improve the Lighthouse V2, and thanks to some hard work, we managed to improve the calibration compensation for the newest version of Valve’s base station.
The client got a new facelift, as well as the python lib, and we released no less than 4 different new releases.
Documentation
We tried as much as we can to improve our documentation, and working from home proved how much it is needed. Kimberly wrote some step-by-step guides that help you discover more hands-on how to use our python library and the motion commander.
We also tried a new way to connect with you, while improving our existing tutorials, by having an online tutorial. It proved quite successful and it was a great way to talk about all the possibilities (and difficulties!) of a swarm.
The way we work
The biggest change was, as many of you, to learn to work from home. We managed quite well, learning to adapt our process. We now are experts in flying at home, and even though our flying arena is deserted, our kitchens are now more alive than ever.
It has been about a month since the AI-deck became available in Early Access. Since then there are now quite a few of you that own an AI-deck yourself. A new development we would like to share: we thought before that we had selected a gray-scale image sensor. However, it came to our attention that the camera actually contains a color image sensor, which on second viewing of the video presented in this blogpost is pretty obvious in hindsight (thanks PULP project ETH Zurich for letting us know!).
A color image from the AI-deck
This came as a little surprise, but a color camera can also add some new possibilities, like making the Crazyflie follow a orange ball, or also train the CNNs incorporate color in their classification training as well. The only thing is that it will require an extra preprocessing task in order to retrieve the color image, which will be explained in the next section.
Demosaicing
Essentially all CMOS image sensors are gray-scale by definition. In order to retrieve color from a scene, manufacturers add a Bayer filter on top of the image sensor, so it filters out the red, green and blue on each pixels. This Color filter array does not need to be RGB, but all kinds of colors, but we will only talk about the Bayer filter. If the pattern of the filter is known, the pixels that related to a certain color will be interpolated with each-other in order to fill in the gaps in between. This process is called demosaicing and it creates the RGB channels that are converted to a color image.
Process of demosaicing with a Bayer filter
Currently we only implemented a simple nearest-neighbor interpolation scheme for demosaicing, which is fine for demonstration purposes, however is not the best technique out there. Such a simple interpolation is not very ‘edge and detail’ aware and can therefore cause artifacts, like these Moiré effects seen here below. Anyway, we are still experimenting how to get a better image and how to translate that to all the examples of the AI-deck example repository (see this issue if you would like to follow or take part in the discussion).
Moiré effect
So technically, once we have the color image, this can be converted to a gray-scale images which can be used for the examples as is. However, there is a reduction in quality since the full pixel resolution is not used for obtaining the full scale image. We are currently discussing if it would be useful to get the gray-scale version of this camera and make this available as well, so let us know if you would be interested!
Feedback and Early Access
Like we said before, there now quite a few of you out there that have an AI-deck in their procession. As it is in Early Access, the software part is still in full development. However, since we have not received any negative feedback of you, we believe that everything is fine and peachy!
Just kidding ;) we know that the AI-deck is quite a challenging deck to work with and we know for sure that many of you probably have questions or have something to say about working with it. Buying an Early Access product also comes with a little bit of responsibility. The more feedback we get from you guys, the more we can tailor the software and support to help you and others, thereby advancing the product forward and getting it out of the early access phase.
So please, let us know if you are having any trouble starting up by posting a thread on the forum (we have a special AI-deck group!). If there are any issues with the examples or the documentation of the AI-deck repo. We and also our collaborators at Greenwaves Technologies (from the GAP8 chip) are more than happy to help out. That is what we are here for :)
The AI-deck is now available in our online store! Super-edge-computing is now possible on your Crazyflie thanks to the GAP8 IoT application processor from GreenWaves Technologies. GAP8 delivers over 10 GOPS of compute power at exceptionally low power consumption enabling complex tasks such as path-finding and target following on the Crazyflie, consuming less than 0,1% of the total energy.
The AI-deck can host artificial intelligence-based workloads like Convolutional Neural Networks onboard. This will open up many research areas focusing on fully onboard autonomous navigation of tiny MAVs, like ETH Zurich’s PULP-Dronet. Moreover, there is also ESP32 WiFi connectivity with the possibility to stream the images to your personal computer.
We are happy that we managed to get everything ready so soon after our last update. Crazyflie AI-deck is in early access, which means the hardware design is now finalized and full support for building, running and debugging applications (including GreenWaves’ GAPflow tools for porting neural networks to GAP from TensorFlow) is available, however, limited examples of specific AI-deck applications have been developed so far. Read more about early access here. Even though there is still some work to be done, there are already some examples you can try out which we will explain in this blog post. Also, we aim to have all AI-decks pre-flashed with the WiFi streamer example so that you check out right away if your AI-deck is working.
Beware that you need an JTAG-enabled programmer/debugger in order to develop for the AI-deck!
Technical specifications
The AI-deck will come with two elongated male pin-headers, which enables the user to connect it to the Crazyflie with an additional deck. There are two 10 pin JTAG connectors soldered which enables connection with a JTAG-enabled programmer. This will be the main way to program the GAP8 chip and the ESP-based NINA module while it is still in early access.
Getting started
When you first receive your AI-deck, it should be flashed with a WiFi streamer example of the camera image stream. Once the AI-deck is powered up by the Crazyflie, it will automatically create a hotspot called ‘Bitcraze AI-deck Example’. In this repo in the folder named ‘NINA’ you will find a file called viewer.py. If you run this with python (preferably version 3), you will be able to see the camera image stream on your computer. This will confirm that your AI-deck is working.
Next step is to go to the docs folder of the AI-deck examples repository. Try out the WiFi demo and set up your development program with the getting-started guide. This guide contains links to the GAP-SDK documentation from GreenWaves Technologies. You can read more about the face detector example that we demonstrated in this blog post.
