We are excited to announce that we will be having developer meetings on first Wednesdays of every month! Additionally, we are thrilled to be present in person at ICRA 2023 in London. During the same conference, there will be half day workshop called ‘The Role of Robotics Simulators for Unmanned Aerial Vehicles’ so make sure to sign-up! Please check out our newly updated event-page !
Monthly Developer meetings
We have had some online developer meetings in the past covering various topics. While these meetings may not have been the most popular, we believe it is crucial to maintain communication with the community and have interesting discussions, and exchange of ideas. However, we used to plan them ad-hoc and we had no regularity in them, which sometimes caused some of us **cough** especially me **cough**, to create confusion about the timing and location. To remove these factors of templexia (dyslexia for time), we will just have it simply on the first Wednesday of every month.
So our first one with be on Wednesday 5th of April at 15:00 CEST and the information about the particular developer meeting will be as usual on discussions. From 15:00 – 15:30 it will be a general discussion, probably with a short presentation, about a topic to be determined. From 15:30-16:00 will address regular support question from anybody that need help with their work on the Crazyflie.
ICRA 2023 London
ICRA will be held in London this year, from May 29 – June 2nd, atthe ExCel venue. We will be located in the H11 booth in the exhibitor hall, but as the date approaches, we will share more about what awesome prototypes we will showcase and what we will demonstrate on-site. Rest assured, plenty of Crazyflies will be flown in the cage! To get an idea of what we demo-ed last year it IROS Kyoto, please check out the IROS 2022 event page. Matej from Flapper Drones will join us at our booth to showcase the Flapper drone.
We are thinking of organizing a meetup for participants on the Wednesday after the Conference Dinner, so we will share the details of that in the near future as well. Also keep an eye on our ICRA 2023 event page for updated information.
Additionally, participants can submit an extended abstract to be invited for an poster presentation during the same workshop. The submission deadline has been extended to April 3rd, so for more information about submission, schedule and speaker info, go to the workshop’s website.
This week’s guest blogpost is from Florian Goralsky from Bok o Bok about their dance piece with multiple Crazyflies. Enjoy!
Flying bodies across the fields is a contemporary dance piece for four performers and a swarm of drones, exploring the phenomenon of the disappearance of bees and the use of pollinating drones to compensate for this loss. The piece attempts to answer this crucial question in a poetical way: can the machine create life and save us from ecological disaster?
We’re super excited to talk about a performance that we’ve been working on for the past two years in collaboration with Bitcraze. It premiered at the Environmental Forum, Centre Pompidou Paris, in 2021, and we’ve had the opportunity to showcase it at different venues since then. We are happy to share our thoughts about it!
Choreographic research
Beyond symbolizing current attempts to use drones to pollinate fields, the presence of the Crazyflie drones, supports the back and forth between nature and technology. We integrate a swarm, performing complex choreographies, which refer to the functioning of a beehive, including the famous “bee dance”, discovered by Karl von Frisch, which is used to transmit information on the food sources. Far from having a spectacular performance as its only goal, the synchronization of autonomous drones highlights bio-inspired computer techniques, focused on collective intelligence.
Making a dance performance with drones needs a high accuracy and adaptability, both before and during the show. Usually, we only have a few hours, sometimes even a few minutes, to setup the system according to the space. We quickly realized we needed pre-recorded choreographies, and hybrid choreographies where the pilot could have a few degrees of freedom on pre-defined behaviors.
GUI Editor + Python Server
Taking this into account, we developed a web GUI editor, that is able to send choreographies created with any device to a Websocket Python server. The system supports any absolute positioning system (We use the Lighthouse), and then converts all the setpoints and actions to the Crazyflie API HighLevelCommander class. This system allows us to create, update, and test complex choreographies in a few minutes on various devices.
Preview position of six drones at a certain time.Early support of the CompressedTrajectories format, with Cubic bezier curves.
What is next?
We are looking forward to developing more dancers-drones interactions in the future. It will imply, in addition to the Lighthouse system, other sensors, in order to open up new possibilities: realtime path-finding, obstacle avoidance even during a recorded choreography (to allow improvisation), etc.
A big part of our work is to provide examples, getting-started guides and other documentation to get users started quickly. Documentation should be up-to-date, be understandable and detailed but, at the same time, not overwhelming. Examples should cover common applications and, most importantly, teach how to create your own projects. This is a never-ending task as our eco-system constantly evolves.
In recent weeks we have updated many parts of the AI-deck documentation and examples – this process is not finished (and will never be), but we thought giving you an overview about what we think most struggle with as well as what we updated would be interesting – especially as we see many AI-deck related questions in the discussions. We saw that many struggle with understanding the whole communication chain and the importance to update all microcontrollers in it – so we will first give an overview on how everything is connected and then dive into where to find documentation, which examples already exist and how to get started with an own project on GAP8. Note that this post is centered around the GAP8; we do not go into detail about the NINA WiFi.
Here we go:
How does the AI-deck fit into the Crazyflie Eco-System? How does it communicate?
As with all other decks, the AI-deck is connected over expansion headers. It gets power directly from the battery (VCOM), and both microcontrollers (GAP8 and the NINA WiFi module) are connected to the STM32 via UART.
AI-deck expansion header connections.
To send messages between all those microcontrollers, the CPX protocol was introduced. As the NINA and GAP8 are also connected (via SPI), we have redundant information paths – so per definition, we always route over the NINA.
Now how can we send code to the GAP8 for it to run?
GAP8 always executes code from L2 (second-level RAM), as it has no internal flash. However, it can load code into L2 over a HyperBus interface from external flash memory on startup (which it does if a fuse is blown, however this is already done on your AI-deck and out of scope here). As GAP8 has only volatile memory, it must always load code from exactly the same flash address. To make it possible to update applications easily, we implemented a bootloader, a minimal program which is the first thing to run on startup. The bootloader can either update the application code in flash or copy it into L2, and, if the code is valid, run it. Why is this easier? First, you don’t need to connect a programmer, as the bootloader can read data over other peripherals (in our case SPI from the NINA module). Second, it is safer – if the update fails (and you, for some reason, end up with random code where your application should be) the firmware code will not be valid (the hash computation will fail) and GAP8 will not jump to the corrupt application code but instead safely stay in the bootloader.
As the chain for the over-the-air update with the bootloader is rather complex, we illustrate the ways to flash GAP8 in the image below.
the blue path illustrates how you can program over JTAG – you can either write code directly into L2 to run it (this is volatile memory, the code will disappear if you power cycle) or you can write it into flash (over GAP8), such that it is loaded on startup (if you overwrite the bootloader, not recommended) or with the bootloader.
the red path is using the cfloader. Meaning it sends your code over the Crazyradio to the nRF, then further to the STM32, from there to the ESP32 (the NINA WiFi module) and from there to the GAP8. This path uses CPX messages; you can read more about it in the CPX documentation.
System Architecture of a Crazyflie with an AI-deck connected.
Where is the documentation?
We have tutorials as well as repository documentation. Tutorials guide you through all the steps needed to run a specific example, while the repository documentation aims to document the general infrastructure and examples in more detail (but without all not directly related steps such as flashing a bootloader, updating other firmware, etc.).
So when you use your AI-deck for the first time, you should start with the getting started guide. Then you are most likely interested in a more detailed explanation about the used infrastructure, such as the GAP8 including the SDK, how to flash, which examples exist and how to run them, etc. So now you should check out the repository documentation.
As we are mostly speaking about GAP8 here, we should also mention that there is of course also documentation for it outside of Bitcraze. GAP8 is produced by Greenwaves, who provides references and has a public SDK on github – meaning one can actually look up the code for all drivers, look at open issues or even contribute with pull-requests.
Which examples exist? What are they there for?What did we update?
This example is there to get you started with your own applications – it provides a minimal implementation of how to send something to the cfclient console from the GAP8 and is explained in detail in the next section of this post.
The camera test is, as it says, to test the camera – however, it sends the image over JTAG, so if you don’t have a debugger and/or don’t want to overwrite the bootloader this is not an example for you.
This example uses filters to find faces in images – be a bit careful, though, as it is very sensitive to noisy backgrounds and, for example, blonde hair. However, along with nice image processing examples, it now also implements the streaming of the images in configurable resolution, a fun feature we recently added!
The classification demo is our AI demo which recognizes parcels. Here we recently fixed the CPX initialization, so it can again send results to the console in the cfclient!
The send character over UART example was neither updated nor tested with the newest docker (yet).
How do I write my own code on gap8?
Now we’ll walk you through a minimal example of how to send Hello World from GAP9 to the cfclient console.
C Code
We start with the main file (which we called hello_world_gap8.c and is found here) by including some dependencies:
#include "pmsis.h" for the drivers
#include "bsp/bsp.h" for some configuration parameters (pad configurations for connecting to memory, camera, etc.)
#include "cpx.h" for using the CPX functions to send our hello world to the console
Then we have to write our main function:
int main(void)
{
return pmsis_kickoff((void *)start_example);
}Code language:JavaScript(javascript)
We call pmsis_kickoff() to start the scheduler and an event kernel, giving it a pointer to the function we want to execute. This function is what we write next (insert it above the main function, such that it is found in the code of the main).
First we need to initialize the pads according to our configuration (the configuration is automatically chosen with sourcing the ai_deck.sh, which is automatically done in the docker) with pi_bsp_init(). Then we need to initialize CPX (which initializes for example the SPI connection to the NINA WiFi), to be able to send CPX packets. You find more information in the CPX documentation. Now we are ready for our while loop, in which we want to send “Hello World” to the console (called LOG_TO_CRTP). To not keep the busses overly busy we only want to send it every second, so we wait before we repeat.
Makefile
The Makefile is hierarchical – meaning we have hidden files that do most of the work and need to include $(RULES_DIR)/pmsis_rules.mk in the last line of the Makefile.
We start with defining where the io should go – possibly are host or uart (this is actually not used in this example, but if you’d add a printf, this would define where it goes). Then we define the operating system we want to use – we can use pulpos or freertos. We chose this as freertos is way more advanced (we are paying for this with some overhead, but in most cases, it will be worth it).
io=uart
PMSIS_OS = freertos
In the next step we need to set the name of our application (this defines the file names of the build output), include the sources (meaning our main c file as well as the two c files CPX needs) and include the header files directory (header files in the same directory as the Makefile should automatically found, but our CPX header files are in a library directory). Make sure all the relative paths are correct for your folder structure.
As a last step, we want to set some compiler flags. Firstly, we want to compile optimized, so we add -O3. Then we add -g to embed debug information. As we use timers for CPX we also need to add two additional defines to ensure all functions we need are included: the configUSE_TIMERS=1 and the INCLUDE_xTimerPendFunctionCall=1 defines.
Where in this example, [binary] has to be replaced with examples/other/hello_world_gap8/BUILD/GAP8_V2/GCC_RISCV_FREERTOS/target.board.devices.flash.img, and the CRAZYFLIE_URI is something like radio://0/80/2M/E7E7E7E7E7.
Now you can connect to your drone with the cfclient and should see a CPX: GAP8: Hello World print every second.
Note: The LED will not blink as in most other examples, as we did not implement a task which does this.
We hope this blog post helps you get started with your own awesome applications faster!
We’re happy to announce that the 2023.02 release is available for download!
The main new features of this release are:
Out of tree controllers
We have made it easier to add a new controller to the firmware in the Crazyflie. Controllers can now be added in an app, the same way as an estimator can be added. The main advantage is that all the code is contained in the app which makes it easy to upgrade the underlying firmware when new releases are available. You can read about how to use this feature in the firmware repository documentation.
Support to configure ESCs with BLHeli Configurator
On brushless Crazyflies, ESCs can now be configured using the BLHeli Configurator. See PR #1170
A UKF (Unscented Kalman Filter) state estimator has been added
An Unscented Kalman Filter (UKF) estimator has been added based by Klaus Kefferpütz from the paper ‘Error-State Unscented Kalman-Filter for UAV Indoor Navigation‘. The estimator is still slightly experimental and does not yet support all positioning methods (see this issue). Because of this, it is not available by default, but you can try it by enabling it using kbuild! You can read about the UKF estimator in the repository documentation.
Platform filter in client flash dialog
A filter has been added to the bootloader dialog in the client to make it easier to find the correct release. Releases are now filtered based on platform to avoid the clutter of mixing releases for cf2, tag, bolt and flapper.
Stability and bug fixes
We have fixed several bugs in the firmware and client software that, but you can check the release notes for each of these for further details.
We have created a simple deprecation policy to clarify future changes of the APIs. The short version is that we from now on will mention deprecated functionality in release notes and that the deprecated functionality will remain in the code base for 6 months before it is removed. Please see the development overview for more information.
As we have talked about in previous blog post, a big work, and a big change, coming to the Crazyflie is the development of a new communication stack. We are organizing an online dev-meeting about this the Wednesday 22th of February 2023 at 15:00 CEST, if you have any feedback, opinion, ideas or just want to talk to us, you are welcome to join. More information on github discussion.
The current communication protocols used by the Crazyflie are 10 years old by now and starts to be the limiting factor for new experiments and for improving the platform. We are starting to work on it to make the Crazyflie protocol for the next 10+ years. Among the things we have been looking at, and want to work on, are:
Making a new USB radio dongle with extended capabilities: Crazyradio 2.0
Making new low level radio protocol implementing channel hopping and P2P communication making use of the new Crazyradio 2.0 capabilities.
Making a new RPC-Based communication protocol to make it easier to develop new functionality and interfacing with framework like ROS2
Defining interface with other part of the system like decks using the same RPC protocol, this would make it easier to develop new deck by limiting the number of project to modify each time a deck is developed.
It has also been pitched internally to write the Crazyflie lib in Rust with binding to Python/C++/Javascript/… unifying the host part of the ecosystem and so simplifying the development of application connecting the Crazyflie.
As you can see, this discussion spans to everything that touches communication from the Crazyflie to outside systems as well as with decks. We think there is a way to make things much better and easier to work with. If we have some time left in the hours we can also handle some general support questions.
If you are interested in the topic please join us on Wednesday and let’s talk about it! You can check the joining information on github discussion. These dev-meeting are not recorded, they are intended as a forum where we can talk together about the Crazyflie and its ecosystem. Welcome!
It’s time for a new compilation video about how the Crazyflie is used in research ! The last one featured already a lot of awesome work, but a lot happened since then, both in research and at Bitcraze.
As usual, the hardest about making those videos is choosing the works we want to feature – if every cool video of the Crazyflie was in there, it would last for hours! So it’s just a selection of the most videogenic projects we’ve seen. You can find a more extensive list of our products used in research here.
We’ve seen a lot of projects that used the modularity of the Crazyflie to create awesome new features, like a catenary robot, some wall tracking or having it land upside down. The Crazyflie board was even made into a revolving wing drone. New sensors were used, to sniff out gas leaks (the Sniffy bug as seen in this blogpost), or to allow autonomous navigation. Swarms are also a research topic where we see a lot of the Crazyflie, this time for collision avoidance, or path planning. We also see more and more of simulators, which are used for huge swarms or physics tests.
Once again, we were surprised and awed by all the awesome things that the community did with the Crazyflie. Hopefully, this will inspire others to think of new things to do as well. We hope that we can continue with helping you to make your ideas fly, and don’t hesitate to share with us the awesome projects you’re working on!
Here is a list of all the research that has been included in the video:
A common task with the Crazyflie is to add a new controller or estimator. As we get some questions on how to do this, we will outline the process in this post. We will show how to add a custom controller and estimator that runs in the Crazyflie, built as an out-of-tree build.
This post assumes some basic knowledge about the Crazyflie firmware, the C programming language, how to build the firmware and flash it to a Crazyflie. If you need some more information on these topics, please see the “Getting started with development” tutorial. For an overview of how estimators and controllers are used by the stabilizer module, please see the firmware documentation.
Overview
The Crazyflie firmware is designed to make it easy to add custom controllers and estimators, a plugin system keeps the code clean and well separated. We will look at the details later, but the basic principle is to first write your new controller or estimator and then register it in the firmware. When the code has been compiled and flashed to the Crazyflie, the new module is activated by setting a parameter from the client or a python script.
We will implement the example as an app, which is a great way to make sure you can upgrade the underlying firmware without messing up your code. An app is a piece of code that exists somewhere in you file system outside of the main firmware source code. This setup minimizes the dependencies and the main firmware source tree can be upgraded without affecting your app (in most cases). That means there is no need for merges or complex management of source trees.
Registration of modules
Let’s first look at how controllers and estimators are registered and called in the plugin framework. We will use the controllers to show how it works, but the estimators are implemented in a similar way and it should be easy to understand how it works.
Note that there has been some updates of the Crazyflie firmware source code lately and any reference to the source code will be to the latest version (as of today).
The starting point of the controller implementation can be found in the src/modules/src/controller.c file, here we can find an array called controllerFunctions that holds a list of all the controllers in the system.
We can see that there is currently four controllers in the list: the PID controller, the Mellinger controller, the INDI controller and finally the Brescianini controller. There is also an “empty” controller at the top that is not important in this context and we will simply ignore it. At the bottom we find the out-of-tree controller, we will discuss this later.
Each controller must implement three functions: an initialization function, a test function and a controller function that performs the actual controller work. Signatures for the three functions are defined in controller.c. The functions are added to the list as function pointers that can be called by the stabilizer when needed.
There is a parameter, stabilizer.controller, in the stabilizer that tells the system which controller to use in the stabilizer loop. This parameter simply contains the index in the controllerFunctions list that will be used. For example, the default value 1 will make the stabilizer loop call the controllerPid function every iteration. If the value of the stabilizer.controller parameter is changed, the initialization function for the new controller will be called and subsequent calls from the stabilizer loop will be done to the new controller function.
We will not go into details of how to implement the actual controller here, but the existing controllers can be used as examples.
Suppose you want to add a new controller. It would be possible to add a new file in the Crazyflie firmware with your new controller implementation, add the function pointers to the list in controller.c and that would work just fine. The problem with such implementation would be that it is hard to maintain, your new files would be mixed with the files in the main firmware file tree, and even worse, you would have to modify the controller.c file to add your controller. The next time there is a new awesome feature in the firmware source code and you want to upgrade to the latest version, you will run into problems as you have to handle the files you modified!
A better solution is to use an app instead as apps are built out-of-tree, that is not in the main source tree. This removes the problem of merging changes in the main source files, all you have to do is to pull in the new file tree and recompile.
But how to register your new controller in the controller list? This is what the last line in the list of controllers is for
If CONFIG_CONTROLLER_OOT is defined we add a controller with the three functions controllerOutOfTreeInit, controllerOutOfTreeTest and controllerOutOfTree. All you have to do in your app is to define CONFIG_CONTROLLER_OOT and make sure the functions in your controller are named like above. That’s it!
Example implementation
Now we will create a new app and add a new controller, step by step. We assume that you have a newly cloned firmware repository in your filesystem to work on.
We will show the linux flavor of commands, but it should be easy to convert to other platforms.
Create a new app
The easiest way is to start from an existing app to get started, let’s use the hello world app. Copy the app and move into the new directory
cp -r examples/app_hello_world examples/my_controller
cd examples/my_controller/
Let’s rename hello_world.c
mv src/hello_world.c src/my_controller.c
We have to tell kbuild that we renamed the file. Open src/Kbuild in your favorite editor and update it to
obj-y += my_controller.o
Now let’s fix the basics in my_controller.c, open it in your editor and change according to the comments bellow:
#include <string.h>#include <stdint.h>#include <stdbool.h>#include "app.h"#include "FreeRTOS.h"#include "task.h"// Edit the debug name to get nice debug prints#define DEBUG_MODULE "MYCONTROLLER"#include "debug.h"// We still need an appMain() function, but we will not really use it. Just let it quietly sleep.
void appMain() {
DEBUG_PRINT("Waiting for activation ...\n");
while(1) {
vTaskDelay(M2T(2000));
// Remove the DEBUG_PRINT.// DEBUG_PRINT("Hello World!\n");
}
}Code language:PHP(php)
Now, lets add our new controller. We will not add a real implementation here as it would be a bit too large for this post, instead we will just call into the PID controller to make sure the Crazyflie still can fly. Add this code after appMain() in my_controller.c.
// The new controller goes here --------------------------------------------// Move the includes to the the top of the file if you want to#include "controller.h"// Call the PID controller in this example to make it possible to fly. When you implement you own controller, there is// no need to include the pid controller.#include "controller_pid.h"
void controllerOutOfTreeInit() {
// Initialize your controller data here...// Call the PID controller instead in this example to make it possible to fly
controllerPidInit();
}
bool controllerOutOfTreeTest() {
// Always return truereturntrue;
}
void controllerOutOfTree(control_t *control, const setpoint_t *setpoint, const sensorData_t *sensors, const state_t *state, const uint32_t tick) {
// Implement your controller here...// Call the PID controller instead in this example to make it possible to fly
controllerPid(control, setpoint, sensors, state, tick);
}
Code language:PHP(php)
Finally we need to tell the firmware that we have implemented the out-of-tree controller and that it should be added to the list. We do this by adding CONFIG_CONTROLLER_OOT to the app-config file. When you are done it should look like this:
Start your Crazyflie and the python client. Connect the client to the Crazyflie and open the console log tab. Make sure you are running your app by looking for the line:
MYCONTROLLER: Waiting for activation ...Code language:HTTP(http)
Now let’s activate our new controller! Open the parameter tab, find the stabilizer group and the controller parameter. Set it to 5 and check the console log that the out-of-tree controller was activated:
CONTROLLER: Using OutOfTree (5) controllerCode language:HTTP(http)
That’s it! Your new controller is activated and the Crazyflie is ready to fly.
Note: In the client, the comment for the stabilizer.controller parameter will not contain the out-of-tree controller, and it will look like only values 0-4 are valid even though 5 also works.
Conclusions
In this post we have shown how to add a new controller to the Crazyflie firmware. The process for adding an estimator is very similar, and hopefully it should be easy to understand how to do it based on the example above.
As you can see, very little code (apart from the actual controller/estimator) is required to add your own controller or estimator, and we hope that it will enable you to put your energy into the actual control problem, rather than the nitty gritty details of the code.
As already announced in a previous blog post, we have been working on a replacement for the Crazyradio PA. Crazyradio is the USB dongle used to communicate with Crazyflie 2.1, Crazyflie Bolt and any other 2.4GHz radio board we are making. We are also visiting FOSDEM in Brussels at the end of the week and will organize a community dev-meeting about Crazyradio and communication end of February: more on that at the end of the post.
Crazyradio 2.0 will have the following characteristics:
Based on the nordic-semiconductor nRF52840
64MHz Cortex-M4
1024KB flash, 256KB ram
Radio supporting Nordic protocol, Bluetooth low energy as well as IEEE802.15.4
1Mbps and 2Mbsp bitrate to Crazyflie
USB full speed (12Mbps) device
Radio power amplifier providing up to +20dBm output power
‘Drag and drop’ bootloader with physical button to start in bootloader mode
Same debug port as on the Crazyflie for ease of development
One of the main changes versus the Crazyradio PA will be the available CPU power and ease of development: this will allow to experiment with and implement much more advanced communication protocol like channel hopping and peer-to-peer communication.
On the software side, there will be two modes available for Crazyradio 2.0: a compatibility mode that emulate a Crazyradio PA and should work with all our existing software as well as a new Crazyradio mode that will have a much improved USB protocol allowing for more efficient communication when controlling multiple Crazyflie as well as making it easy to support more protocols in the future.
These two modes will be available as two different firmware and the user can ‘drag and drop’ the firmware with the wanted mode.
As for the Crazyradio PA (version 1), sourcing the components for it has been a bit challenging lately. We will sell Crazyradio PA as long as we have stock for it and the software will continue to support it for the foreseeable future.
Announcements
Kimberly and I, Arnaud, will be visiting the FOSDEM conference at the end of the week in Brussels. If you are there too and want to meet us do not hesitate to drop a message in the comment there, in Github discussions or by mail. It would be great to meet fellow Crazyflie users!
We are also planning an online dev-meeting about Crazyradio 2.0 and communication the 22nd of February 2023. The information about joining will be on Github Discussions. We are interested in talking, and bouncing ideas about radio and communication protocol: with the new Crazyradio we have an opportunity to work on communication protocols to improve them and makes them more useful to modern use of the Crazyflie.
My name is Hanna, and I just started as an intern at Bitcraze. However, it is not my first time working with a drone or even the Crazyflie, so I’ll tell you a bit about how I ended up here.
The first time I used a drone, actually even a Crazyflie, was in a semester thesis at ETH Zurich in 2017, where my task was to extend a Crazyflie with a Parallel Ultra Low-Power (PULP) System-on-Chip (SoC) connected to a camera and external memory. This was the first prototype of the AI-deck you can buy here nowadays (as used here) :)
My next drone adventure was an internship at a company building tethered drones for firefighters – a much bigger system than the Crazyflie. I was in charge of the update system, so more on the firmware side this time. It was a very interesting experience, but I swore never to build a system with more than three microcontrollers in it again.
This and a liking for tiny and restricted embedded systems brought me back to the smaller drones again. I did my master thesis back at ETH about developing a PULP-based nano-drone (nano-drones are just tiny drones that fit approximately in the palm of your hand and use only around 10Watts of power, the category Crazyflies fit in) and some onboard intelligence for it. As a starting point, we used the Crazyflie, both for the hardware and the software. It turned out to be a very hard task to port the firmware to a processor with only a very basic operating system at that time. Still, eventually I knew almost every last detail of the Crazyflie firmware, and it actually flew.
However, for this to happen, I needed some more time than the master thesis – in the meantime, I started to pursue a PhD at ETH Zurich. I am working towards autonomous miniaturized drones – so besides the part with the tiny PULP-based drone I already told you about, I also work on the “autonomous” part. Contrary to many other labs our focus is not only on novel algorithms though, we also work with novel sensors and processors. Two very interesting recent developments for us are a multi-zone Time-of-Flight sensor and the novel gap9 processor, which both fit on a Crazyflie in terms of power, size and weight. This enables new possibilities in obstacle avoidance, localization, mapping and many more. Last year my colleagues and I already posted a blog post about our newest advances in obstacle avoidance (here, with Videos!). More recently, we worked on onboard localization, using novel multi-zone Time-of-Flight sensors and the very new GAP9 processor to execute Monte Carlo localization onboard a Crazyflie (arxiv).
On the left you see an example of a multi-zone Time-of-Flight image (the background is a picture from the AI-deck), from here. On the right you see our prototype for localization in action – from our DATE23 paper (arxiv).
For me, localizing with a given map is a fascinating topic and one of the reasons I ended up in Sweden. It is one of the most basic skills of robots or even humans to navigate from A to B as fast as possible, and the basis of my favourite sport. The sport is called “orienteering” and is about running as fast as possible to some checkpoints on a map, usually through a forest. It is a very common sport in Sweden, which is the reason I started learning Swedish some years ago. So when the opportunity to go to Malmö for some months to join Bitcraze presented itself, I was happy to take it – not only because I like the company philosophy, but also because I just like to run around in Swedish forests :)
Now I am looking forward to my time here, I hope to learn lots about drones, firmware, new sensors, production, testing, company organization and to meet a lot of new nice people!
Greetings from Malmö – it can be a bit cold and rainy, but the sea and landscape are beautiful!
Bats navigate using sound. As a matter of fact, the ears of a bat are so much better developed than their eyes that bats cope better with being blindfolded than they cope with their ears being covered. It was precisely this experiment that helped the discovery of echolocation, which is the principle bats use to navigate [1]. Broadly speaking, in echolocation, bats emit ultrasonic chirps and listen for their echos to perceive their surroundings. Since its discovery in the 18th century, astonishing facts about this navigation system have been revealed — for instance, bats vary chirps depending on the task at hand: a chirp that’s good for locating prey might not be good for detecting obstacles and vice versa [2]. Depending on the characteristics of their reflected echos, bats can even classify certain objects — this ability helps them find, for instance, water sources [3]. Wouldn’t it be amazing to harvest these findings in building novel navigation systems for autonomous agents such as drones or cars?
Figure 1: Meet “Crazybat”: the Crazyflie equipped with our custom audio deck including 4 microphones, a buzzer, and a microcontroller. Together, they can be used for bat-like echolocation. The design files and firmware of the audio extension deck are openly available, as is a ROS2-based software stack for audio-based navigation. We hope that fellow researchers can use this as a starting point for further pushing the limits of audio-based navigation in robotics. More details can be found in [4].
The quest for the answer to this question led us — a group of researchers from the École Polytechnique Fédérale de Lausanne (EPFL) — to design the first audio extension deck for the Crazyflie drone, effectively turning it into a “Crazybat” (Figure 1). The Crazybat has four microphones, a simple piezo buzzer, and an additional microprocessor used to extract relevant information from audio data, to be sent to the main processor. All of these additional capabilities are provided by the audio extension deck, for which both the firmware and hardware design files are openly available.1
Video 1: Proof of concept of distance/angle estimation in a semi-static setup. The drone is moved using a stepper motor. More details can be found in [4].
In our paper on the system [4], we show how to use chirps to detect nearby obstacles such as glass walls. Difficult to detect using a laser or cameras, glass walls are excellent sound reflectors and thus a good candidate for audio-based navigation. We show in a first semi-static feasibility study that we can locate the glass wall with centimeter accuracy, even in the presence of loud propeller noise (Video 1). When moving to a flying drone and different kinds of reflectors, the problem becomes significantly more challenging: motion jitter, varying propeller noise and tight real-time constraints make the problem much harder to solve. Nevertheless, first experiments suggest that sound-based wall detection and avoidance is possible (Figure and Video 2).
Video 2: The “Crazybat” drone actively avoiding obstacles based on sound. Figure 2: Qualitative results of sound-based wall localization on the flying “Crazybat” drone. More details can be found in [4].
The principle we use to make this work is sound-based interference. The sound will “bounce off” the wall, and the reflected and direct sound will interfere either constructively or destructively, depending on the frequency and distance to the wall. Using this same principle for the four microphones, both the angle and the distance of the closest wall can be estimated. This is however not the only way to navigate using sound; in fact, our software stack, available as an open-source package for ROS2, also allows the Crazybat to extract the phase differences of incoming sound at the four microphones, which can be used to determine the location of an external sound source. We believe that a truly intelligent Crazybat would be able to switch between different operating modes depending on the conditions, just like bats that change their chirps depending on the task at hand.
Note that the ROS2 software stack is not limited to the Crazybat only — we have isolated the hardware-dependent components so that the audio-based navigation algorithms can be ported to any platform. As an example, we include results on the small wheeled e-puck2 robot in [4], which shows better performance than the Crazybat thanks to the absence of propeller noise and motion jitter.
This research project has taught us many things, above all an even greater admiration for the abilities of bats! Dealing with sound is pretty hard and very different from other prevalent sensing modalities such as cameras or lasers. Nevertheless, we believe it is an interesting alternative for scenarios with poor eyesight, limited computing power or memory. We hope that other researchers will join us in the quest of exploiting audio for navigation, and we hope that the tools that we make publicly available — both the hardware and software stack — lower the entry barrier for new researchers.
1 The audio extension deck works in a “plug-and-play” fashion like all other extension decks of the Crazyflie. It has been tested in combination with the flow deck, for stable flight in the absence of a more advanced localization system. The deck performs frequency analysis on incoming raw audio data from the 4 microphones, and sends the relevant information over to the Crazyflie drone where it is converted to the CRTP protocol on a custom driver and sent to the base station for further processing in the ROS2 stack.
References
[1] Galambos, Robert. “The Avoidance of Obstacles by Flying Bats: Spallanzani’s Ideas (1794) and Later Theories.” Isis 34, no. 2 (1942): 132–40. https://doi.org/10.1086/347764.
[2] Fenton, M. Brock, Alan D. Grinnell, Arthur N. Popper, and Richard R. Fay, eds. “Bat Bioacoustics.” In Springer Handbook of Auditory Research, 1992.https://doi.org/10.1007/978-1-4939-3527-7.
[3] Greif, Stefan, and Björn M Siemers. “Innate Recognition of Water Bodies in Echolocating Bats.” Nature Communications 1, no. 106 (2010): 1–6. https://doi.org/10.1038/ncomms1110.
[4] F. Dümbgen, A. Hoffet, M. Kolundžija, A. Scholefield and M. Vetterli, “Blind as a Bat: Audible Echolocation on Small Robots,” in IEEE Robotics and Automation Letters (Early Access), 2022. https://doi.org/10.1109/LRA.2022.3194669.