AI-deck Sees in Color

2020-07-13 | Kimberly McGuire | Leave a comment

It has been about a month since the AI-deck became available in Early Access. Since then there are now quite a few of you that own an AI-deck yourself. A new development we would like to share: we thought before that we had selected a gray-scale image sensor. However, it came to our attention that the camera actually contains a color image sensor, which on second viewing of the video presented in this blogpost is pretty obvious in hindsight (thanks PULP project ETH Zurich for letting us know!).

This came as a little surprise, but a color camera can also add some new possibilities, like making the Crazyflie follow a orange ball, or also train the CNNs incorporate color in their classification training as well. The only thing is that it will require an extra preprocessing task in order to retrieve the color image, which will be explained in the next section.

Demosaicing

Essentially all CMOS image sensors are gray-scale by definition. In order to retrieve color from a scene, manufacturers add a Bayer filter on top of the image sensor, so it filters out the red, green and blue on each pixels. This Color filter array does not need to be RGB, but all kinds of colors, but we will only talk about the Bayer filter. If the pattern of the filter is known, the pixels that related to a certain color will be interpolated with each-other in order to fill in the gaps in between. This process is called demosaicing and it creates the RGB channels that are converted to a color image.

Currently we only implemented a simple nearest-neighbor interpolation scheme for demosaicing, which is fine for demonstration purposes, however is not the best technique out there. Such a simple interpolation is not very ‘edge and detail’ aware and can therefore cause artifacts, like these Moiré effects seen here below. Anyway, we are still experimenting how to get a better image and how to translate that to all the examples of the AI-deck example repository (see this issue if you would like to follow or take part in the discussion).

So technically, once we have the color image, this can be converted to a gray-scale images which can be used for the examples as is. However, there is a reduction in quality since the full pixel resolution is not used for obtaining the full scale image. We are currently discussing if it would be useful to get the gray-scale version of this camera and make this available as well, so let us know if you would be interested!

Feedback and Early Access

Like we said before, there now quite a few of you out there that have an AI-deck in their procession. As it is in Early Access, the software part is still in full development. However, since we have not received any negative feedback of you, we believe that everything is fine and peachy!

Just kidding ;) we know that the AI-deck is quite a challenging deck to work with and we know for sure that many of you probably have questions or have something to say about working with it. Buying an Early Access product also comes with a little bit of responsibility. The more feedback we get from you guys, the more we can tailor the software and support to help you and others, thereby advancing the product forward and getting it out of the early access phase.

So please, let us know if you are having any trouble starting up by posting a thread on the forum (we have a special AI-deck group!). If there are any issues with the examples or the documentation of the AI-deck repo. We and also our collaborators at Greenwaves Technologies (from the GAP8 chip) are more than happy to help out. That is what we are here for :)

New ways of meeting each other?

2020-06-29 | Kimberly McGuire | 1 Comment

It has been a few months of when the Covid-19 crisis started, but it feels like almost a year ago when we all decided to stay and work from home. Considering circumstances, we managed to do to handle ourselves pretty well. We set up our home labs in our kitchens and/or living-rooms and managed to do a lot of development. Even though this situation did not come easy as you can see from our experiences here, we were able to pull ourselves through it in one peace. Now we also have to consider that Covid-19 is here to stay and we need to deal with the complications until at least the vaccine is finished and distributed. Until then, we might have to think about alternatives on how we do things, including how we go to events and meet/talk to you all!

Every year we try to go to at least two conferences, with last year being a particular busy year of us going to three big events (ICRA 2019, IMAV 2019 and IROS 2019). Before going to those conferences, we usually try to crunch and make an awesome demo. This also enables us to add new features to the firmware or fix problems that we find during this crunch. Moreover, we also really like to meet our users face-to-face, so that we can hear about how you use the Crazyflie in your research or classroom!

Since going to conferences and in-person events will be difficult to do this year and maybe the next, we were thinking about events that we can possibly organize to as an alternative. We were thinking about a couple of options on which we would like your opinion on as well. For instance, we could do an remote tutorial or lecture, like we did here for EPFL. Or maybe we can organize an online seminar we were invite users to give a talk about their work (I personally took part in a VR seminar in Mozilla hubs, which was pretty awesome). We can also consider to invite users for an online meetup to talk about the direction of the Crazyflie and its firmware. Another idea that we had recently, is to organize an online Crazyflie competition, where users can control the Crazyflie remotely or upload custom firmware, so that it can fly autonomously through an obstacle field.

We set up a poll of these ideas, so we can know what you guys like best! Also please comment below if you have further ideas about this or start a thread on the forum!

AI-deck is available in early access

2020-06-08 | Kimberly McGuire | Leave a comment

The AI-deck is now available in our online store! Super-edge-computing is now possible on your Crazyflie thanks to the GAP8 IoT application processor from GreenWaves Technologies. GAP8 delivers over 10 GOPS of compute power at exceptionally low power consumption enabling complex tasks such as path-finding and target following on the Crazyflie, consuming less than 0,1% of the total energy.

The AI-deck can host artificial intelligence-based workloads like Convolutional Neural Networks onboard. This will open up many research areas focusing on fully onboard autonomous navigation of tiny MAVs, like ETH Zurich’s PULP-Dronet. Moreover, there is also ESP32 WiFi connectivity with the possibility to stream the images to your personal computer.

We are happy that we managed to get everything ready so soon after our last update. Crazyflie AI-deck is in early access, which means the hardware design is now finalized and full support for building, running and debugging applications (including GreenWaves’ GAPflow tools for porting neural networks to GAP from TensorFlow) is available, however, limited examples of specific AI-deck applications have been developed so far. Read more about early access here. Even though there is still some work to be done, there are already some examples you can try out which we will explain in this blog post. Also, we aim to have all AI-decks pre-flashed with the WiFi streamer example so that you check out right away if your AI-deck is working.

Beware that you need an JTAG-enabled programmer/debugger in order to develop for the AI-deck!

Technical specifications

The AI-deck will come with two elongated male pin-headers, which enables the user to connect it to the Crazyflie with an additional deck. There are two 10 pin JTAG connectors soldered which enables connection with a JTAG-enabled programmer. This will be the main way to program the GAP8 chip and the ESP-based NINA module while it is still in early access.

Getting started

When you first receive your AI-deck, it should be flashed with a WiFi streamer example of the camera image stream. Once the AI-deck is powered up by the Crazyflie, it will automatically create a hotspot called ‘Bitcraze AI-deck Example’. In this repo in the folder named ‘NINA’ you will find a file called viewer.py. If you run this with python (preferably version 3), you will be able to see the camera image stream on your computer. This will confirm that your AI-deck is working.

Next step is to go to the docs folder of the AI-deck examples repository. Try out the WiFi demo and set up your development program with the getting-started guide. This guide contains links to the GAP-SDK documentation from GreenWaves Technologies. You can read more about the face detector example that we demonstrated in this blog post.

Latest Update on the AI deck

2020-05-25 | Kimberly McGuire | 2 Comments

It has been a while since we have updated you all on the AI deck. The last full blogpost was in October, with some small updates here and there. It is not that we have not focused on it at all; on the contrary… this has been a high priority project for a while now. It is just quite a complex board with a lot of bells and whistles, which can be challenging to work with sometimes so early in development, something that our previous intern can definitely agree on. So therefore we rather wanted to wait until we were able to make sufficient progress before we gave you an update… and so we have!

Together with Greenwaves technologies we have been trying to get the SDK of the GAP8 chip on the AI deck stable enough for an early release. The latest release of the SDK (version 3.4) has proved itself to work with relative ease on the AI deck after extensive testing. Currently it is possible to use OpenOCD for flashing and debugging, and it supports most commonly available debuggers with a jtag connector. In the upcoming weeks both of Bitcraze and Greenwaves will test and try out all examples of the SDK on the AI deck to make sure that everything is still compatible. Also the documentation will be extended as well. As there is so much to document, it might be difficult to catch all of it. However, if you notify us and Greenwaves on anything that is missing once the AIdeck is out, that will help us out to catch the knowledge gaps.

The AI deck also contains the ESP-based NINA module for establishing a WiFi connection. This enables the users to stream the video stream of the AI deck onto their computers, which will be quite an essential tool if they would like to generate their own image database for training the CNNs for the GAP8 (and it happens to also be quite practical for debugging by the way!). Currently it is required to set credentials of your local WiFi network and reflash the AI-deck to be able to connect and streaming the images, but we are working on turning the Nina into an access-point instead so no reflashing would be required. We hope that we will be able to implement this before the release.

We are also trying out to adjust applications to make suitable of the AI deck. For instance, we have adapted Greenwaves’ face-detector example to use the image streamer instead of the display available on the GAPuino boards. You can see a video of the result here underneath. Beware that this face-detector is not based on a CNN but on HOG descriptors, so it only works in good conditions where the face is well lit. However, it is possible to train a CNN to detect faces in Tensorflow and flash this on the AI deck with the GAPflow framework as developed by Greenwaves. At Bitcraze we haven’t managed to try that out ourselves ( we are close to that though!) but at least this example is a nice demonstration of the AI deck’s abilities together with the WiFi-streamer. This example and more testing code can be found in our experimental repo here. For examples of GAPflow, please check out the examples/NNtool section of the GAP8 SDK.

For some reason WordPress has difficulty embedding the video that was supposed to be here, so please check https://youtu.be/0sHh2V6Cq-Q

Seeing how the development has been progressing, we will be comfortable to say that the AI deck could be ready for early release somewhere in the next month, so please keep an eye out on our website! We will continue to test the GAP SDK’s stability and we are very thankful for Greenwaves Technologies with their help so far. We will also work on getting-started guides in order to get acquainted with the AI deck, supplementing the already existing documentation about the GAP8 chip.

Even-though the AI deck will soon be ready for early release, this piece of hardware is not for the faint-hearted and embedded programming experience is a must. But keep in mind that the possibilities with the AI deck are huge, as it will be mean that super-edge-computing on a 30 gram flying platform will be available for anyone. It will all be worth it when you have your Crazyflie flying autonomously while being able to recognize its surroundings :)

The Commander Framework

2020-05-11 | Kimberly McGuire | Leave a comment

Here is another blog post where we try to explain parts of the stabilizer framework of the Crazyflie. Last time, we talked about the controllers and state estimators as part of the stabilizer.c module which was introduced in this blog post back in 2016. Today we will go into the commander framework, which handles the setpoint of the desired states, which the controllers will try to steer the estimated state to.

The Commander module

The commander module handles the incoming setpoints from several sources (src/modules/src/commander.c in the firmware). A setpoint can be set directly, either through a python script using the cflib/ cfclient or the app layer (blue pathways in the figure), or by the high-level commander module (purple pathway). The High-level commander in turn, can be controlled remotely from the python library or from inside the Crazyflie.

General framework of the stabilization structure of the crazyflie with setpoint handling. * *This part is takes place on the computer through the CFlib for python, so there is also communication protocol in between. It is left out of this schematics for easier understanding*.

It is important to realize that the commander module also checks how long ago a setpoint has been received. If it has been a little while (defined by threshold COMMANDER_WDT_TIMEOUT_STABILIZE in commander.c), it will set the attitude angles to 0 on order to keep the Crazyflie stabilized. If this takes longer than COMMANDER_WDT_TIMEOUT_SHUTDOWN, a null setpoint will be given which will result in the Crazyflie shutting down its motors and fall from the sky. This won’t happen if you are using the high level commander.

Setpoint structure

In order to understand the commander module, you must be able to comprehend the setpoint structure. The specific implementation can be found in src/modules/interface/stabilizer_types.h as setpoint_t in the Crazyflie firmware.

There are 2 levels to control, which is:

Position (X, Y, Z)
Attitude (pitch, roll, yaw or in quaternions)

These can be controlled in different modes, namely:

Absolute mode (modeAbs)
Velocity mode (modeVelocity)
Disabled (modeDisable)

Setpoint structures per controller level

So if absolute position control is desired (go to point (1,0,1) in x,y,z), the controller will obey values given setpoint.position.xyz if setpoint.mode.xyz is set to modeAbs. If you rather want to control velocity (go 0.5 m/s in the x-direction), the controller will listen to the values given in setpoint.velocity.xyz if setpoint.mode.xyz is set to modeVel. All the attitude setpoint modes will be set then to disabled (modeDisabled). If only the attitude should be controlled, then all the position modes are set to modeDisabled. This happens for instance when you are controlling the crazyflie with a controller through the cfclient in attitude mode.

High level commander

As already explained before: The high level commander handles the setpoints from within the firmware based on a predefined trajectory. This was merged as part of the Crazyswarm project of the USC ACT lab (see this blogpost). The high-level commander uses a planner to generate smooth trajectories based on actions like ‘take off’, ‘go to’ or ‘land’ with 7th order polynomials. The planner generates a group of setpoints, which will be handled by the High level commander and send one by one to the commander framework.

It is also possible to upload your own custom trajectory to the memory of the Crazyflie, which you can try out with the script examples/autonomous_sequence_high_level of.py the crazyflie python library repository. Please see this blogpost to learn more.

Support in the python lib (CFLib)

There are four main ways to interact with the commander framework from the python library.

Send setpoints directly using the Commander class from the Crazyflie object, this can be seen in the autonomousSequence.py example for instance.
Use the MotionCommander class, as in motion_commander_demo.py. The MotionCommander class exposes a simplified API and sends velocity setpoints continuously based on the methods called.
Use the high level commander directly using the HighLevelCommander class on the Crazyflie object, see autonomous_sequence_high_level.py.
Use the PositionHlCommander class for a simplified API to send commands to the high level commander, see the position_commander_demo.py

Documentation

We are busy documenting the stabilizer framework in the Crazyflie firmware documentation, including the content of this blogpost. If you feel that anything is missing or not explaining clearly enough about the stabilizer framework, please drop a comment below or comment on the forum.

New release: 2020.04

2020-05-04 | Kimberly McGuire | Leave a comment

For the users that have subscribed to our github repository this does not come as an surprise, but for the rest, we have released a new version of our Crazyflie firmware (both STM and NRF) last week!

We know that it is quite close to our last release in February, but we had so many changes and contribution that we deemed it necessary to add a stamp to this current version. In this blog-post, we will give an overview on which features to expect in this update.

UART communication

With courtesy of Saarland University, it is now possible to connect the Crazyflie through its UART to a port on your raspberry pi or through an FTDI cable directly to your computer. This is an extra port for communicating with CRTP will open up new possibilities to interact with your crazyflie.

This is compatible with CFlib version 0.1.10, however there was a fix implemented in the current master (see the ticket here). Please see the ticket for the UART communication here if you are interested in the implementation details.

Lighthouse

It is now possible to get the lighthouse geometry (the position and orientation of the base stations) without SteamVR. We made a script based on the latest stable release of openCV, to calculate the base station geometry based on the received sweep angles on the lighthouse deck. Check these full instructions on how to use this new script. It is a very new and fresh implementation, so if you are experiencing any trouble, please leave an issue on this page or leave a comment on the forum.

Also, FPGA v4 is now integrated in 2020.04, which support Basestation v2. This is still in a very early phase and not yet fully integrated in the firmware, so please keep an eye on this ticket for the implementation process in the latest master of the crazyflie-firmware. There was also a blogpost a few weeks ago about the current state of the lighthouse v2 development.

Bluetooth management

We also provided an update of the bluetooth management of the Crazyflie communication by the NRF chip. Before, it was (unintentionally) possible to connect to the Crazyflie over Bluetooth while it also connected to the CFclient through the crazyradio PA. This caused a lot of unwanted elements such as package loss and unresponsiveness. Now, whenever a Crazyradio packet has been received, Bluetooth will automatically be disabled. The same goes for the peer-2-peer packet, so the NRF firmware no longer needs to be flashed without Bluetooth support. The Crazyflie needs to be restarted after connecting through the CF dongle or P2P in order to connect to it again with the Crazyflie mobile app.

General fixes and improvements

Here are the general fixes and improvements listed that has been fixed in release v2020.04:

BMI088 (IMU of the CF2.1) has an self-test now.
Fixed memory issue with the Micro SD card deck.
High-level commander improvements.
Documentation improvements.
LPS TDoA (2 and 3) improvements.

See the release notes of the crazyflie-firmware and crazyflie-nrf-firmware to see the full list of improvements and issues that were fixed in 2020.04. The zip files for the firmware for both the roadrunner (tag) and crazyflie (cf2) can be found here.

Running Projects and Status Update

2020-04-14 | Kimberly McGuire | 2 Comments

In this blog-post we wanted to give you guys an overview of our running projects and a general update of the status of things! We got settled in our home-labs and are working on many projects in parallel. There are a lot of development happening at the moment, but the general feeling is that we do miss working with each other at our office! With our daily slack Bitcraze sync meetings and virtual fikapause (Swedish for coffee breaks), we try to substitute what we can. In the mean time, we are going on a roll with finishing all our goals we have set at our latest quarterly meeting, so here you can read about those developments.

AI-deck

The last time we gave an update about the AI-deck was in this blog post and in the final post of our intern Zhouxin. Building on his work, we are now refocusing on getting the AI-deck ready for early release. The last hurdle is mostly software wise on which we are considering several approaches together with the manufacturer of the Gap8 chip Greenwaves technologies. Currently we are preparing small testing functions as examples of the different elements of the AI-deck in our repo, which are all still in a very primarily phase.

Even though we still need some time to finalize the AI-deck’s early release, we will consider sending an early version of the AI-deck if you are willing to provide feedback while working with it. Please fill in the form and we will get back to you.

Lighthouse

We have made quite some progress on the development for the lighthouse V2. Kristoffer has been working hard from his homelab to get a seamless integration of both V1 and V2 in our firmware (check out this github issue for updates). Currently it is still very untested and very much in progress, however we do have a little preview for you to enjoy.

Crazyflie with LH basestation v2

Documentation

Right now, we are also doing a lot of revamping of the large web of documentation. Unfortunately this is a lot of work! As you noticed by now, we have added overview pages to guide the reader to the right information. We also have moved the tutorials to another part of the menu to avoid clutter on our website. In general we try to go through the repository docs to see if there is any information missing or outdated, however please let us know if you have encountered an error in any description or are missing crucial elements.

Our latest task is revamping the product pages as well, by putting all the necessary information about the hardware in just one place. Also, we are planning to make (video) tutorials soon about many elements of the Crazyflie and how to work with it. More about that later!

Production and Shipment

Production at our manufacturers in China are slowly starting up again. Although it is not yet back at full force, it does enable us to already start ordering to replenish our stock and to get started with finishing our test rigs. Moreover, we are also negotiating to resolve the propeller issue we mentioned earlier, but there is no update on that so far.

As mentioned in this blogpost, we are still shipping orders about twice a week. Both DHL and Fedex are functioning as normal, but we do notice that there is a delay of a few extra days on some deliveries. Please keep that in mind when ordering at our webshop.

The Home Lab

2020-04-06 | Kimberly McGuire | Leave a comment

Many people in the world have now settled in the reality of working from home. We have also taken precautions ourselves by not go to our office as normal and only ship out packages a few times per week instead of every day (see this blogpost). This also means that we do not have full access to our lab with all our equipment and positioning systems in our big 10 x 10 meter flight lab at the office. In this blogpost we will show how we manage to keep on developing and flying, even in the current situation.

Crazyflie flying in a kitchen with the lighthouse deck

In(light)house positioning

Currently we started to use the Lighthouse positioning system to setup up the remote home lab at our houses. As of recent additions to the Crazyflie firmware, it has been made easy to get the geometry data from the base station. Now the only items we need for indoor flight are just two (or only one) lighthouse basestations V1’s and a Crazyflie, and that is it! There is no need for an HTC Vive headset or hub, or third-party software like SteamVR and the setup is finished in 2 minutes! Check out the new documentation here if you want to know more about the new setup of the lighthouse positioning system.

Also, we recently got a very primarily version of the lighthouse V2 working (see here) and we of course want to keep the momentum going! We will be working on full compatibility from our homes so stay tuned. For now, see this video of the Crazyflie flying with just a single base-station, taken from one of our team-member’s home lab.

Remote Lecture Hall and Practicals

We were invited by Dario Floreano and Fabrizio Schiano from the EPFL-LIS laboratory to do a lecture for the ‘Aerial Robotics’ Course as part of EPFL’s Master’s program in Robotics. Due to the virus, we had to cancel our trip to go there physically… but luckily we were able to do the lecture remotely anyway!

The lecture consists of two parts. In the first hour we mostly explained about the Crazyflie ecosystem, hardware and sensors. In the second hour we focused on how the stabilization module worked, including the controllers and the state estimation. During both sessions, we alternated between the theory slides with actual hands-on demos. The lighthouse positioning system was setup in a kitchens, so that we were able to show full flights and practicals with the Crazyflie. At the end there was also the push-demo with just the flowdeck and multiranger, which didn’t use any external positioning at all.

The lectures can be found below and the documentation has been updated as well with the covered material (see here). Be sure to check out the controller tuning presented in part 2 of the lecture (25:00 – Cascaded PID controller).

Other Home labs

We know that there are currently users that are moving their flight lab from their university or company to their homes to be able to continue their work. We would love to hear about your experience and your home lab! Send us an email with your story to contact@bitcraze.io, drop us a message on forum.bitcraze.io, or mention us in your Twitter, Linkedin, Facebook or Reddit post. Also, if you want to setup your own home lab and you need any advice or help, please let us know!

Out of Control

2020-02-03 | Kimberly McGuire | 2 Comments

Two weeks ago, we had a blogpost about the state estimators that are available within the Crazyflie. So once the Crazyflie knows where it is, it would need to be determined where it wants to go, by means of the high level commander (implemented as part of the crazyswarm project) or set-points given by CFclient or directly from scripts using Crazyflie python lib. But exactly how would the crazyflie get to those desired positions in the first place? The differences between the current state estimates and the desired state, will need to be transformed to inputs given to the motors. Unfortunately, quadrotors like the Crazyflie do not have easy dynamics to maintain, so if you want to learn more, see this blogpost to read more about it!

So in order use the thrust of the motors in an useful way to get the Crazyflie to do what you want to do, there are several controllers to consider, which you can see on this quick overview here underneath. It shows the different control paths that can be taken from the high level commander all the way to the power distribution of the motors. Bear in mind that these are still simple representations and that the actual implementation is of course a bit more complicated, but at least it will give you a rough idea of which paths are possible to pursue.

PID Controller

So the default settings in the Crazyflie firmware is the proportional integral derivative (PID) control for all desired state aspects. So the High Level Commander (HLC) will send desired position set-points to the PID position controller (which used to be done off-board, so outside of the Crazyflie firmware before this blogpost). These result in desired pitch and roll angles, which are sent directly to the attitude PID controller. These determine the desired angle rates which is send to the angle rate controller (which is… you guessed… also a PID controller). This is also called Cascaded PID controller. That results in the desired thrusts for the roll pitch yaw and height that will be handled by the power distribution by the motors. (Note that height is mostly handled by the position controller)

INDI Controller

So the Incremental Nonlinear Dynamic Inversion (INDI) controller is an controller that immediately deal with the angle rates to determine the trust. This is a very new addition to the Crazyflie firmware by one of our community members and is based on the implementation of this paper. Currently, the position control is still handled by the same PID controller mentioned in the last paragraph, Nevertheless for handling the angles, it should be faster than the attitude and rate PID controller combined. We have not yet fully tested this out but if you do, let us know how you like it on the Bitcraze forum!

Mellinger Controller

As part of the Crazyswarm project, the controller designed by Daniel Mellinger has been implemented in the Crazyflie firmware as well. Please see this paper about the details of the Mellinger controller. It is a sort of “all in one”: based on the desired position and velocity vectors towards those position, it will calculate right away what the desired thrusts are that need to be distributed to all the motors. This results in a much smoother controlled trajectory of the high level commander and therefore advised to use when the Crazyflie has a precise position estimate (lighthouse and mocap). However, as it is so aggressive, any position estimate of a lesser quality (flowdeck or LPS) will not be sufficient for this controller. See some examples of mellinger controlled flights here and here.

Let us know what you think!

So do you have experience working with these controllers or want to know more about them, please drop us a message on the forum! We are currently working on stabilization and documentation of multiple aspects of the Crazyflie and the controllers is one of them, so we are really interested what your experiences are!

State estimation: To be or not to be!

2020-01-20 | Kimberly McGuire | 2 Comments

How does a Crazyflie manage to fly and stay in the air in the first place? Many of us tend to take this for granted as much research tend to happen on the application level. Although we try to make the low level elements of flight as stable as possible, it might happen that whatever you are trying to implement on the application level actually effects the Crazyflie on the low level controls and estimation. We therefore would like to focus a little bit on the inner-workings of the autopilot of the Crazyflie, starting with state estimation. The state estimation is part of the stabilizer loop in the Crazyflie, an overview of is was made in a previous blog post.

State estimation is really important in quadrotors (and robotics in general). The Crazyflie needs to first of all know in which angles it is at (roll, pitch, yaw). If it would be flying at a few degrees slanted in roll, the crazyflie would accelerate into that direction. Therefore the controller need to know an good estimate of current angles’ state and compensate for it. For a step higher in autonomy, a good position estimate becomes important too, since you would like it to move reliably from A to B.

There are two types of state estimators in the crazyflie firmware, namely a Complementary Filter and an Extended Kalman Filter.

Complementary Filter

The complementary filter is consider a very lightweight and efficient filter which in general only uses the IMU input of the gyroscope (angle rate) and the accelerator. The estimator has been extended to also include input of the ToF distance measurement of the Zranger deck. The estimated output is the Crazyflie’s attitude (roll, pitch, yaw) and its altitude (in the z direction). These values can be used by the controller and are meant to be used for manual control. If you are curious how this code is implemented exactly, we encourage you to checkout the firmware in estimator_complementary.c and sensfusion6.c. The complementary filter is set as the default state estimator on the Crazyflie firmware.

Extended Kalman Filter

The (extended) Kalman filter is an step up in complexity compared to the complementary filter, as it accepts more sensor inputs of both internal and external sensors. It is an recursive filter that estimates the current state of the Crazyflie based on incoming measurements (in combination with a predicted standard deviation of the noise), the measurement model and the model of the system itself. We will not go into detail on this but we encourage people to learn more about (extended) Kalman filters by reading up some material like this.

Shortly said, because of the more state estimation possibilities, we preferred the Kalman filter in combination with several decks: Flowdeck, Loco positioning deck and the lighthouse deck. If you look in the deck driver firmware (like for instance this one), you see that we set the required estimator to be the Kalman and that is of course because we want position/velocity estimates :). Important though is that each input of the measurement effects the quality of the position, as positioning of the Lighthouse deck (mm precision) is much more accurate that the loco positioning deck (cm precision), which has all to do with the standard deviation of the measurement of those values. Please check out the content of estimator_kalman.c and kalman_core.c to know more about the implementation. Also good to know that the Kalman filter has an supervisor, which resets if the position or velocity estimate is gets out of hand.

Of course this blogpost does not show the full detailed explanation of state estimation, but we do hope that it gives some kind of overview so you know where to look if you would like to improve anything. The Kalman filter can easily be extended to accept more inputs, or the models on which the estimates are based can be improved. If you would like implement your own filter, that would be perfectly possible to do so too.

It would be great if you guys could share your thoughts and questions about the state estimation on the crazyflie on the forum!