The AI-decks are back in stock! Also, last week we had our quarterly meeting, where we plan our focus for the next quarter. As it is also the start of the fiscal year, we also take this opportunity to update our 1 year and 3 year plans as well. We have a big plans coming up, but one of the important focuses that we will have this year, is to get the AI-deck out of early-access!
But what would be necessary for such a task? The AI-deck is one of the most complicated boards we have worked with, so do we have to evaluate its ‘out-of-readiness’ along the same standards than any of our other products.
Mega AIdeck Tutorial
So one of our idea is to be able to achieve a state of the AIdeck in order to write a mega AIdeck tutorial series. This implies that we are able to show how somebody could go from a datasets all the way to a flying aideck-crazyflie combo. Such a series could consist of the following topics:
How to go from a dataset of images to a (Deep) Neural Network
Testing the DNN on the computer with the Image WiFi examples
Converting the neural network to Tensorflow Light (with basics on Edge AI and quantizing neural networks)
AIdeck basics (How to access the camera, how to add the network to the cluster, how to send commands)
Build and flashing the AIdeck and testing it out in the hand
Attach the AIdeck to the crazyflie, make an app-layer application to fly and react on the image input.
From the first look of it, this sounds like it should be easy to do right? Actually, there are still much to be done in order to make this tutorial possible.
Replumbing the Communication
One of the more challenging aspects of the AIdeck as it now is, is that users need to buy a JTAG-enabled programmer in order to flash the GAP8 and the NINA module of the AI-deck. That is the reason why currently the AIdeck has these 2 x 10 pin jtag connectors attached, but ideally we would want to get rid of it completely. This means is that we need to have over air flashing of the GAP8’s binary and that the intercommunication of the NINA and AIdeck will become even more important.
Moreover, the communication protocol from the GAP8 to the STM32 of the Crazyflie is currently very basic, as of right now, it is only possible to send single characters. It might work in some situations, but what if you would like to send an array of values back to the Crazyflie, like the collision probability & steering angle like in PULP platform’s implementation of Dronet? And, would we like to keep on using two UART serial ports or perhaps just relay both NINA and GAP8 communication all through one? The later will make it easier for us to maintain the crazyflie-aideck communication but can perhaps introduce communication delays.
These are just a slice of the type of re-plumbing work for the AIdeck before we can even start our dream tutorial series, but at least it will give you an idea of what we are dealing with :)
Rik the Intern
From this week we have the honor of hosting Rik Bouwmeester for a couple of months. He is currently doing his Master Thesis at the MAVlab from the faculty of Aerospace Engineering of the TU Delft. Since he has experience of working the AIdeck before, he will be able to provide us with some user perspective and help us with the above mentioned issues. You can expect a blogpost from him soon!
The Crazyflie firmware is the result of a lot of engineering (Making It Work) effort over a long period of time, the first commit in the GitHub repository is from 2013.
There are issues, inconsistencies and pain points in our firmware that have been found along the way, things that are difficult to fix because they have, in some sense, become load-bearing.
An examples of an issue like this is our inconsistency in what units are used in the Crazyflie firmware, we would like to use SI units everywhere but today that is not the case. And converting everything is proving to be tough.
We would like to fix all of these, but, the problem is that we have APIs to consider. There are programs written that assume that the data received from the Crazyflie is in a certain way. If we were to switch to the, more correct, aerospace axes these programs might break. And worse, they might break silently, meaning there will be no error message.
Our APIs constitute some kind of promise between the Crazyflie ecosystem (firmware, python library, documentation) and our users.
Examples include the parameter- and logging API which today has ideas about which way the coordinate system points and what units are used for values. As well as the Commander Framework / API, since this API has opinions regarding what way X, Y or Z grows in relation to reality.
Our APIs exists both between the Crazyflie firmware and our Python library code and between our Python library code and our users programs, as visualized in the diagram below.
If we were to implement solutions to the known issues we need to take these API into consideration. There are applications, demos, scripts and research code out there that rely on these APIs to behave a certain way.
So what can we do? What are our options? We have tried to enumerate them below.
1. Go Ahead And Change Units and Coordinates – But Do Not Update APIs
We fix as much as we can of the unit, correctness and conventions bugs in the firmware, but we leave the APIs as they are. This means that we keep the name of the parameter- and logging variables and we keep the name of all Python classes and functions. This is the less amount of work, it only needs updates to the crazyflie-firmware repository.
All examples and programs will continue to run. But they might now operate under false pretenses, perhaps with no error messages. The programs might stop working because of the implicit change of API.
We will have to communicate loud and clear that this will happen beforehand.
2. Go Ahead And Change Units and Coordinates – And Update the APIs
We fix as much as we can of the unit, correctness and conventions bugs in the firmware, and we also rework our APIs! We will gather learnings from our years with the current CRTP and Python API and we update them to be more developer ergonomic as well as update the parameter- and logging variables to be more consistent in naming and SI units.
A big benefit of this would be that we are then open to change where the boundaries for our API lies, what we provide as libraries and what is applications responsibility. And we can address pain points that are deep and structural to todays APIs.
This will break all existing programs out there, they will no longer run. They will get error messages and will need to be updated to the new APIs. We might also get weird issues and scenarios where old library code try to interface with newer firmware. We could create policys on for how long we maintain the old API and try to have them side-by-side for a bit.
3. Go Ahead And Change Units and Coordinates – Keep Old API Forever and Add a New One
We fix as much as we can of the unit, correctness and conventions bugs in the firmware, and we also rework our APIs! We will gather learnings from our years with the current CRTP and Python API and we update them to be more developer ergonomic as well as update the parameter- and logging variables to be more consistent in naming and SI units. And! We will also keep the old API as a legacy API.
It is unclear if this is possible, if we have enough memory in the firmware as well as enough bandwidth to maintain two APIs.
We could in theory keep all of the user-facing Python API and attempt to translate between the old and new one in Python code. This might take a lot of work and effort to do, but it might be possible.
But. Fixing all the correctness, convention and units in the firmware might make it very difficult to support both APIs. It might also limit how creative and how different a new API could be, while still being translatable to the old API.
We could also investigate if it would be possible to keep the API between the firmware and the libraries (CRTP and paramter- and logging variables) but translate to be more correct in regards to Units and Conventions in our library.
Even if this was deemed possible it would not help anyone developing directly against the firmware, without using a Bitcraze library.
4. Do Not Change Units and Coordinates – Keep Old API
Accept that the crazyflie-firmware will not reach correctness in respect to units and conventions and close the issues above. We can make sure all new features going in respect SI units and convention, but consider what we have as a legacy. This is by far the one that requires the less work. zero. But it will keep the pain points and inconsistencies of the Crazyflie platform.
Conclusions
There are trade offs between all solutions. We need to decide what our current APIs are worth to us. And how painful it would be for our users to handle an API break.
Maintaining multiple APIs might be to big of an burden for both us and our code base though. This is very very tricky.
It would help us a lot to hear from our users!
If you are someone who develops or have developed against the Crazyflie ecosystem please get in touch, we have created an GitHub issue here. Tell us if you have hit any of the pain-points described in this post or how an API break would affect you.
As you maybe remember, we decided to celebrate our 10 years anniversary with our very own conference. It’s going to be 3 days of online meeting, sharing and generally awesome meetups to connect, learn, and hopefully develop new knowledges and acquaintances.
We are working hard on creating the best experience possible for you ! The program is still only a skeleton with some very interesting bones, and we add more and more flesh to it as the dates grow nearer.
As you can see from the program below, each day is organized around a global theme. Let’s break them down:
The first day, Tuesday, is going to be dedicated to navigation systems. Guest speakers Klaus Kefferputz from Hochscule Auburg and Joseph La Delfa (the man behind Drone Chi). will each talk about how they use positioning systems, in two very different ways. We’ll also have a workshop about the different positioning systems we support, to get into the more nitty gritty side of things.
The next day, we’ll talk about the AI deck ! Our collaborators at PULP-platform and Greenwaves Technologies are the perfect guests to talk about our newest deck. The talks will be followed by a workshop about app layers and a general panel discussion about Bitcraze – and more particularly its future. We want to hear from you, so don’t hesitate to join us as we try to determine a way forward (for the next 10 years, we have some ideas but we’re curious what you think)!
And finally, the last day will revolve around swarms. It’s Wolfgang‘s specialization, so he is of course the first guest we turned to. Bart Duisterhof and Guido de Croon from Delft University of Technology will also talk about their autonomous swarm, (they are the team behind the sniffy bug!). A swarm workshop will follow, and we will close the conference with some fun surprises!
We’re really excited and lucky to welcome all those speakers to talk about drone development. The program is not entirely fixed yet, some surprise guests should turn up. You can also still add your input by filling out this short survey.
And last, but not least : you can register to the event here on EventBrite. You will be able to add it to your calendar and finally mark the dates ! Also checkout the BAM days event page occasionally for updates.
On September, 1st of 2011, Arnaud, Marcus and Tobias signed a paper in Lund registering their brand-new company: Bitcraze. We’re here, now, 10 years later, hardly believing this gamble paid off so greatly.
Bitcraze’s story actually begins in 2009, when the 3 met in their day job and started dreaming about what will become the Crazyflie. For a long time, they combined a 9-to-5 job with the development of the nanodrone. Monday meetings run from the end of the workday until 2am, and they were exhausted, but the hard work paid off when we know now how many Crazyflies took off after that!
Some things have been added since 2011, of course! For starters, 4 more people joined the team, each changing the face of the company, making it evolve. Lean and self-organized was the way to go, the more people there was. The product itself changed and grew: from the first batch of Crazyflies, that had to be soldered out of the box, we’re now at the 2.1 version. And as years flew by, Bitcraze added to its backpack with new decks and extensions, always wanting to test new things and generally make awesome stuff. From its first flight, we’re now at impressive autonomous swarm flights!
The Crazyflie celebrating its first year in 2012! (and yes, it flew with the candle on it)
In 10 years, we’ve also done things we would never have thought of. Did you know Marcus gave a 15 minutes speech at a conference … entirely in French ? Or that 3 different employees got a cut on their finger for the exact same reason (we’ll let you guess why!) ? We’re still amazed at how much we discovered, learned and shared in those 10 years.
For us, it’s a special moment, looking back at 10 years of awesome things, of hard work, and of passion. We plan on celebrating, of course! First with a few days’ getaway, the first occasion we’ve had in a while to be actually together and enjoy. But this milestone would not be happening without all the support of those around us: our family and friends of course, but also our contributors and our customers. That’s why we’ve decided to organize our own conference, to gather everyone for awesome days of meetup and celebrate with you!
Keep an eye out on our social media too, as this September we’ll look back a little down the road. We’ll post some fond memories (and maybe, just maybe, some embarrassing photos too!) and see how much we’ve accomplished in 10 years.
Previously we have been using off the shelf scales and other methods to measure characteristics, such as thrust or efficiency, of the Crazyflie products. We thought it was time to build something that is easier to use, more repeatable and tailored to our needs. Well, this has been on our wanted list for a long time, already back from when we did the RPM-deck. It was however first when Wolfgang visited us this winter that he nudged us over the edge so we finally allocated some time for it. We started off by buying some load cells and breakout boards to do something simple as a start, so we could at least measure thrust more easily. We actually started looking for off the shelf thrust stands but could not find anything suitable for the Crazyflie’s size. As is often the case here at Bitcraze, the project grew. Already before we had any load cells up and running I was designing a deck with RPM sensors, load cell amplifier and power meter. Now with the objective to easily do system identification. Therefor we named the deck the system-id deck.
For the RPM sensors we used the same as on the RPM-deck, the QRD1114. They are not great as they need a reflective surface, this means adding white stickers or paint to black propellers, but they work well enough. The load cell amplifiers ended up to be the NAU7802 as it has a high accuracy and sample rate. For power metering we chose the new ACS37800 power monitoring IC that can handle up to 30A, this looked exiting.
The QRD1114 we wired the same way as previously done on the RPM-deck:
The NAU7802 was configured as per the datasheet suggestion and similarly to other open designs out there:
The ACS37800 was very new so the datasheet had to be used as the main information source. A bit tricky as this chip is mainly intended to measure mains supply, and we wanted to measure low voltage DC, which it said it could do…and in the end we managed to get it working.
We also added a buck/boost DC/DC that could provide a stable 3.3V from 2-5V input, just in case, as the ACS37800 is specified for this voltage and not the 3.0V the Crazyflie can supply.
The outcome
The PCB was designed as small as possible so it could be mounted on a Crazyflie 2.X and used while flying. A bonus would be if it could be used on a Bolt as well.
Here it is mounted on a Crazyflie 2.1 together with a 3D printed stand and load cell.
The load cell would then be mounted to a desk or similar so the the Crazyflie is mounted up-side down, pushing down on the load cell.
Software
The software, as often, took most of the time to make. Three major deck driver files was created, rpm.c, acs37800.c and loadcell_nau7802.c. Aside from these there where only small changes to make, like making it work when being up-side-down. The modifications have all been pushed to the dev-systemid branch for those that are interested. As for now we are mainly using the logging framework to transfer the data to the PC, which is quick and easy to setup and use, but writing to SD-card is also possible. The scripts for this can be found in the tools/sytem_id folder.
Remaining work is to test, gather and analyse more data. When we have done so, we will post more. Until then below is a sample of what it can measure. The data is taken with a ramping PWM from 0% – 100% – 0%. The added resistance of the extra wires and connectors are not taken into account, but the estimated efficiency of 4g/W is probably not that far off.
How to handle our documentation has been always a bit of struggle. For almost 2 years (see this blogpost and this one) we have working on improving the documentation structure, with by transferring information from the wiki, putting information closer to the code and setting up automating documentation. A few months ago, we managed to have automated logging and parameter documentation (see this blogpost).
Even though we think there is some improvement already, it can always be better! We have noticed that some of our users are a bit confused of how to go through our documentation. So in this blogpost we are discussing some navigational strategies of how you can maneuver yourself through the documentation as it is presented on bitcraze.io, which can also be found here.
Ecosystem-based navigation
So more than a year ago, we also started with a Ecosystem overview page, which are meant to take first-timers by the hand through the Crazyflie ecosystem.. This type of overview pages are starting from the three main pillars: the Crazyflie Platform, the Clients and Positioning Technology. This is a type of navigation that we mostly advise to take if you are a beginner Crazyflie user who do not know the structure of the eco system fully.
For those that already have experience with the Crazyflie and its Ecosystem, the previous way of navigating through the docs might be a bit convoluted. With the Ecosystem-based navigation, it takes about 3 scrolls and clicks to reach the STM development documentation, which is a bit to much of a round way if you already know what you are looking for. We have made the repository overview page not for this purpose but we actually started using ourselves a lot within the company, as a direct pathway to the development repository per element. So this is a page that would be useful to other advanced developers as well!
So the repository overview page is split up in 4 main categories: Python-based software, C-based firmware, Other languages and bootloaders. See the navigation tree which of those repositories approximately point too. By the way, have you noticed that repository documentation has a gray header (like this one) and all the overview pages on the web have a green header (like this one)? This are meant to make you aware if you are still on a fluffy overview website page or going in the nitty gritty details of the development documentation.
Repository-based navigation tree
Feature-based navigation ?
Still a remaining problem is that the repository documentation might not be enough to get a good overview. Where do you need to look if you are interested in ‘controllers’ or ‘state estimators’, or how to make an app-layer application? Currently all of this is within the stm32 firmware documentation, as that is the exact location of where all of this is implemented. But how to document spanning features like the CRTP, where not only the STM chip but also the NRF, Crazyradio PA and the Crazyflie python library are also involved? Or how about the loco positioning system, where the Crazyflie communicates through the LPS deck with a separate LPS node?
So perhaps a good way how to present all this information, is to do it feature-based, like ‘controllers’, ‘positioning’, ‘high level commander’, where we present a structure that points to parts of the detailed documentation within the repo-docs. With ecosystem-based, or even repository-based, navigation documentation strategy, it will take for instance 4-7 clicks to come to the specific controller page, as you can verify by looking at the bread-crumb of the header. Perhaps splitting it up based on feature instead of Ecosystem elements or programming language might be a more logical structure of the current state of the Crazyflie documentation.
Feedback
One reason why it is so difficult to do this properly, is that we have a lot of repositories based on each microprocessor of all of our products, which makes our opensource projects quite unique. It is therefore difficult to find another opensource project of which we can take inspiration from. So, let us know what you would prefer for navigating through our documentation in this poll, but we are always open to other suggestions! If you know of any example of a similar opensource software project that is doing it the right way, or have any other tips, send us an email (contact_at_bitcraze.io), contact us on social media platforms or post a comment on this blogpost!
Update 2021-12-21:
The poll is closed and this is the result! Thanks all for responding!
We like to describe the Crazyflie as a versatile open source flying development platform. It is something that enables you to do cool stuff. It is not a finished, polished end product in its own.
This approach makes determining the expectations and requirements for the platform and the surrounding ecosystem a bit tricky. It is dependent on what you as a user plan to create using our products. And since the ecosystem is growing we need a, scalable, way to handle these fuzzy expectations during development and maintenance.
We think testing is big part of solving this, testing in a systematic, scalable and reproducible way. This is the reason for setting up our first physical test lab, aimed at providing stability while moving forward with new products and features.
Testing today
Continuous integration
On each change proposal to our software we run tests. For the firmware in the device we build multiple different configuration and run unit-tests. For the Crazyflie client and the Python library we make sure we can build for Linux and Windows and that the code passes our style guidelines. If any test fail we go back and update the proposed changed and re-run the tests. This is our first level of defense against defects.
Release testing
For each release we follow a checklist of procedures and tests to make sure that quality does not degrade. We make sure all the examples in the Python library are working and that the Crazyflie can fly around as expected in our flight arena, using various positioning systems.
Limitations
The way we test today makes it very difficult to determine if we regress in, for example, flight capability or in radio communication quality. We either test different software packages in isolation, without hardware in the loop, or we test by having a human trying to estimate if any degradation has happened since the last release.
We run into scalability issues as our ecosystem grows, it is near impossible to test all the different combinations of products. And it is very hard for us to detect stability or quality regressions without having a more systematic approach to testing the software on relevant hardware.
Introducing the test lab
What we have done so far are two things:
Created an infrastructure for setting up a site for testing our software on real devices
Prepared a test lab in our printer room at our office in Malmö
This new Git repository contains the building blocks we need to setup our new test lab. You can check out the repository README.md file for information on how to run it.
The repository contains a way to specify a test site, which is a collection of devices to run the test suite against. You specify your site in a TOML file which contain information about each device, such as which decks are connected and the radio:// URI. This site specification is then used by the test framework when running the tests, making sure each test is run against all devices compatible with that test.
The infrastructure also has management scripts to perform tasks like flashing all devices in a site, or recovering them if they go into boot loader mode by accident. All aimed at being able to handle testing with the least amount of human intervention possible.
This acts as a safety net for us. We will quickly know if the communication performance degrades, or if we have messed up with our parameter or logging frameworks. And we can catch silly firmware bugs as early as possible.
Future
We want to keep adding test cases and other infrastructure to our testing framework. Going forward it would be really nice to be able to test positioning systems in some way. And of course, some type of test of flight, be it free flying tests or using some kind of harness.
It might be interesting to look into adding simulations (hardware in loop or not) to our testing setup. It is all a question of bandwidth, there are a lot of cool things to work on, and a limit on time and bodies.
You can help! You might even be able to help yourself while helping us! If you contribute tests that correspond to your use-case, then you can relax knowing that those tests will run each night, and that Bitcraze engineers will be notified the minute they fail.
Or you can define your own site and run the test-suite against all your devices to make sure nothing strange is going on.
Hopefully this infrastructure and lab will help all of us to do more cool stuff using the Bitcraze ecosystem!
Ever since we released the Lighthouse deck back in 2019, we’ve wanted to offer a bundle with the deck and the base stations. There’s multiple reasons for this, but the main reason was that we wanted users to be able to buy a full swarm (like the Loco Positioning Swarm) directly from us, without having to find the base stations separately. Initially this seemed easy to do, but it turned out to be a bit tricky. This post is about how we finally managed to get the Lighthouse Swarm Bundle finished and into the E-store.
The Lighthouse swarm bundle
When the Lighthouse deck was initially released it only had support for Lighthouse V1 base stations, but Ligthouse V2 was already out. Since the V1 base stations were already in short supply, we wanted to support V2 since this was what would be available in the future. We had started looking at V2 support, but there was still ongoing efforts from us (and others) to reverse engineer the protocol. After some prototyping we had some initial support, but there was still a lot of infrastructure work to be done before it could be released.
In parallell with this work we started trying to buy the Lighthouse V2 base stations. Normally there’s two options here, either buy from local distributors or buy directly from the manufacturer. Buying from local distributors wasn’t a good option for us since these will only have local power plugs and buying directly from the manufacturer often requires very large orders. So this process quickly stalled. But after a couple of months we got an offer to buy a bulk shipment of Ligthouse V2 base stations (without box or power adapters) which we finally decided to accept. And yeah, that’s me looking really happy next to a bunch of base stations…
Marcus looking happy about the base stations
With a bunch of base stations at the office, work with sourcing a power adapter and creating a box started. Unfortunately the number of COVID-19 cases started rising again shortly after receiving the base stations, so we started working more from home again. And with only 2 persons at the office at a time, it’s hard to work with hardware. Different team-members needs access to different resources, like the electronics labs, flight arena or packing orders. So getting box/adapter samples from manufacturers, doing testing and getting input on physical objects from other team-members quickly went from days to weeks.
Finally, after a couple of months of testing, evaluating and learning lots about adapters and cardboard, we had good candidates. But then, literally as we’re ordering the power adapters, it turns out the certification was not good for all the regions we wanted. Thankfully this time around we already had other options so we quickly decided on the second best option (now the best option) and ordered.
In the meantime work was underway finalizing the implementation of Lighthouse V2, including client support, firmware updates of the Lighthouse deck and documentation/videos. Finally in the beginning of 2021 we got documentation and the full implementation (although only for 2 base stations) in place (blog post).
After a bit more than a month of waiting, the power adapters and boxes finally showed up at our office. With all the supplies in place, we started preparing for the packing. Since you can buy base stations for multiple sources, we wanted to keep track of the base stations that we were sending out to be able to debug issues users might have with these units. Also, even though the base stations had already been factory tested, we wanted to quickly test them before shipping them out. So our flight arena was turned into a makeshift assembly line and we had some outside help come in to do the packing.
Finally, the end result! We’re really excited to be able to offer yet another swarm bundle, the Lighthouse swarm bundle. And we’re pretty happy about how the packaging turned out :-)
This week we have a guest blog post from Bart Duisterhof and Prof. Guido de Croon from the MAVlab, Faculty of Aerospace Engineering from the Delft University of Technology. Enjoy!
Tiny drones are ideal candidates for fully autonomous jobs that are too dangerous or time-consuming for humans. A commonly shared dream would be to have swarms of such drones help in search-and-rescue scenarios, for instance to localize gas leaks without endangering human lives. Drones like the CrazyFlie are ideal for such tasks, since they are small enough to navigate in narrow spaces, safe, agile, and very inexpensive. However, their small footprint also makes the design of an autonomous swarm extremely challenging, both from a software and hardware perspective.
From a software perspective, it is really challenging to come up with an algorithm capable of autonomous and collaborative navigation within such tight resource constraints. State-of-the-art solutions like SLAM require too much memory and processing power. A promising line of work is to use bug algorithms [1], which can be implemented as computationally efficient finite state machines (FSMs), and can navigate around obstacles without requiring a map.
A downside of using FSMs is that the resulting behavior can be very sensitive to their hyperparameters, and therefore may not generalize outside of the tested environments. This is especially true for the problem of gas source localization (GSL), as wind conditions and obstacle configurations drastically change the problem. In this blog post, we show how we tackled the complex problem of swarm GSL in cluttered environments by using a simple bug algorithm with evolved parameters, and then tested it onboard a fully autonomous swarm of CrazyFlies. We will focus on the problems that were encountered along the way, and the design choices we made as a result. At the end of this post, we will also add a short discussion about the future of nano drones.
Why gas source localization?
Overall we are interested in finding novel ways to enable autonomy on constrained devices, like CrazyFlies. Two years ago, we showed that a swarm of CrazyFlie drones was able to explore unknown, cluttered environments and come back to the base station. Since then, we have been working on an even more complex task: using such a swarm for Gas Source Localization (GSL).
There has been a lot of research focussing on autonomous GSL in robotics, since it is an important but very hard problem. The difficulty of the task comes from the complexity of how odor can spread in an environment. In an empty room without wind, a gas will slowly diffuse from the source. This can allow a robot to find it by moving up gradient, just like small bacteria like E. Coli do. However, if the environment becomes larger with many obstacles and walls, and wind comes into play, the spreading of gas is much less regular. Large parts of the environment may have no gas or wind at all, while at the same time there may be pockets of gas away from the source. Moreover, chemical sensors for robots are much less capable than the smelling organs of animals. Available chemical sensors for robots are typically less sensitive, noisier, and much slower.
Due to these difficulties, most work in the GSL field has focused on a single robot that has to find a gas source in environments that are relatively small and without obstacles. Relatively recently, there have been studies in which groups of robots solve this task in a collaborative fashion, for example with Particle Swarm Optimization (PSO). This allows robots to find the source and escape local maxima when present. Until now this concept has been shown in simulation [2] and on large outdoor drones equipped with LiDAR and GPS [3], but never before on tiny drones in complex, GPS-denied, indoor environments.
Required Infrastructure
In our project, we introduce a new bug algorithm, Sniffy Bug, which uses PSO for gas source localization. In order to tune the FSM of Sniffy Bug, we used an artificial evolution. For time reasons, evolution typically takes place in simulation. However, early in the project, we realized that this would be a challenge, as no end-to-end gas modeling pipeline existed yet. It is important to have an easy-to-use pipeline that does not require any aerodynamics domain knowledge, such that as many researchers as possible can generate environments to test their algorithms. It would also make it easier to compare contributions and to better understand in which conditions certain algorithms work or don’t work. The GADEN ROS package [4] is a great open-source tool for modeling gas distribution when you have an environment and flow field, but for our objective, we needed a fully automated tool that could generate a great variety of random environments on-demand with just a few parameters. Below is an overview of our simulation pipeline: AutoGDM.
AutoGDM, a fully automated gas dispersion modeling (GDM) simulation pipeline.
First, we use a procedural environment generator proposed in [5] to generate random walls and obstacles inside of the environment. An important next step is to generate a 3D flowfield by means of computational fluid dynamics (CFD). A hard requirement for us was that AutoGDM needed to be free to use, so we chose to use the open-source CFD tool OpenFOAM. It’s used for cutting-edge aerodynamics research, and also the tool suggested by the authors of GADEN. Usually, using OpenFOAM isn’t trivial, as a large number of parameters need to be selected that require field expertise, resulting in a complicated process. Next, we integrate GADEN into our pipeline, to go from environment definition (CAD files) and a flow field to a gas concentration field. Other parts that needed to be automated were the random selection of boundary conditions, which has a large impact on the actual flow field, and source placement, which has an equally large impact on the concentration field.
After we built this pipeline, we started looking for a robot simulator to couple it to. Since we weren’t planning on using a camera, our main requirement was for the simulator to be efficient (preferably in 2D) so that evolutions would take relatively little time. We decided to use Swarmulator [6], a lightweight C++ robot simulator designed for swarming and we plugged in our gas data.
Algorithm Design
Roughly speaking, we considered two categories of algorithms for controlling the drones: 1) a neural network, and 2) an FSM that included PSO, with evolved parameters. Since we used a tiny neural network for light seeking with a CrazyFlie in our previous work, we first evolved neural networks in simulation. One of the first experiments is shown below.
A single agent in simulation seeking a light source using a tiny neural network.
While it worked pretty well in simple environments with few obstacles, it seemed challenging to make this work in real life with complex obstacles and multiple agents that need to collaborate. Given the time constraints of the project, we have opted for evolving the FSM. This also facilitated crossing the reality gap, as the simulated evolution could build on basic behaviors that we developed and validated on the real platform, including obstacle avoidance with four tiny laser rangers, while communicating with and avoiding other drones. An additional advantage of PSO with respect to the reality gap is that it only needs gas concentration and no gradient of the gas concentration or wind direction (which many algorithms in literature use). On a real robot at this scale, estimating the gas concentration gradient or the direction of a light breeze is hard if not impossible.
Hardware
Our CrazyFlie needs to be able to avoid obstacles, execute velocity commands, sense gas, and estimate the other agent’s position in its own frame. For navigation, we added the flow deck and laser rangers, whereas for gas sensing we used a TGS8100 gas sensor that was used on a CrazyFlie before in previous work [7]. The sensor is lightweight and inexpensive, but accurately estimating gas concentrations can be difficult because of its size. It tends to drift and needs time to recover after a spike in concentration is observed. Another thing we noticed is that it is possible to break them, a crash can definitely destroy the sensor.
To estimate the relative position between agents, we use a Decawave Ultra-Wideband (UWB) module and communicate states, as proposed in [8]. We also use the UWB module to communicate gas information between agents and collaboratively seek the source. The complete configuration is visible below.
A 37.5 g nano quadcopter, capable of fully autonomous waypoint tracking, obstacle avoidance, relative localization, communication and gas sensing.
Evaluation in Simulation
After we optimized the parameters of our model using Swarmulator and AutoGDM, and of course trying many different versions of our algorithm, we ended up with the final Sniffy Bug algorithm. Below is a video that shows evolved Sniffy Bug evaluated in six different environments. The red dots are an agent’s personal target waypoint, whereas the yellow dot is the best-known position for the swarm.
Sniffy Bug evaluated in Swarmulator environments.
Simulation showed that Sniffy Bug is effective at locating the gas source in randomly generated environments. The drones successfully collaborate by means of PSO.
Real Flight Testing
After observing Sniffy Bug in simulation we were optimistic, but unsure about performance in real life. First, inspired by previous works, we disperse alcohol through the air by placing liquid alcohol into a can which is then dispersed using a computer fan.
Dispersion of liquid alcohol in flight tests.
We test Sniffy Bug in our flight arena of size 10 x 10 meters with large obstacles that are shaped like walls and orange poles. The image below shows four flight tests of Sniffy Bug in cluttered environments, flying fully autonomously, i.e., without the help from any external infrastructure.
Time-lapse images of real-world experiments in our flight arena. Sniffy was evaluated on four distinct environments, 10 x 10 meters in size, seeking a real isopropyl alcohol source. The trajectories of the nano quadcopters are clearly visible due to their blue lights.
In the total of 24 runs we executed, we compared Sniffy Bug with manually selected and evolved parameters. The figure below shows that the evolved parameters are more efficient in locating the source as compared to the manual parameters.
Maximum recorded gas reading by the swarm, for each time step for each run.
This does not only show that our system can successfully locate a gas source in challenging environments, but it also demonstrates the usefulness of the simulation pipeline. The parameters that were learned in simulation yield a high-performance model, validating the environment generation, randomization, and gas modeling parts of our pipeline.
Conclusion and Discussion
With this work, we believe we have made an important step towards swarms of gas-seeking drones. The proposed solution is shown to work in real flight tests with obstacles, and without any external systems to help in localization or communication. We believe this methodology can be extended to larger environments or even to 3 dimensions, since PSO is a robust, multi-dimensional heuristic search method. Moreover, we hope that AutoGDM will help the community to better compare gas seeking algorithms, and to more easily learn parameters or models in simulation, and deploy them in the real world.
To improve Sniffy Bug’s performance, adding more laser rangers will definitely help. When working with only four laser rangers you realize how little information it actually provides. If one of the rangers senses a low value it is unclear if a slim pole or a massive wall is detected, adding inefficiency to the algorithm. Adding more laser rangers or using other sensor modalities like vision will help to avoid also more complex obstacles than walls and poles in a reliable manner.
Another interesting discussion can be held on the hardware required for real deployment. When working with 40 grams of maximum take-off weight, the sensors and actuators that can be selected are limited. For example, the low-power and lightweight flow deck works great but fails in low-light scenarios or with smoke. Future work exploring novel sensors for highly constrained nano robots could really help increase the Technological Readiness Level (TRL) of these systems.
Finally, this has been a really fun project to work on for us and we can’t wait to hear your thoughts on Sniffy Bug!
It feels like the day of yesterday, when Arnaud, Tobias and Marcus came together in a backyard shed and decided to go fully in developing the Crazyflie 1.0 and start a business. The result that came from that was Bitcraze, and after 10 years we are still here! Also in the course of the next months, we will be releasing more information about the history of Bitcraze on the blog or social media.
But more important we would like you to leave the 19th-21th of October open in your agenda for the grand party, because we will be organizing our own multi day convention: BAM days a.k.a. Bitcraze Awesome Meetup days.
Of-course we did everything we could to come to abbreviation BAM as you can see :)
This event will be fully online and will be filled with (guest) talks, workshops but most importantly: Fika time, which is coffee breaks in Swedish. We really want to put the emphasize on networking and the coffee breaks as we considered the most important part of any conference, seminar or convention. So in between the talks, which will be in a video chat format like Zoom or Google Meet, there will be equally long coffee breaks in spatial chat format as like Gathertown, Mozilla Hubs or MiBo. We are currently browsing to several online event portal alternatives to accommodate all of this to make sure that everything goes smoothly!
We are currently still building up the program and inviting speakers to give talks and workshops. Moreover, we probably will prepare workshops and demos ourselves as well! So please fill in this interest form if you want to receive more information the event, and give us some pointers of the content. Also check out the event page for any current information.
And in the mean time, make sure to keep the 19th to the 21th of October free in your agenda so that you can come and celebrate our 10 year anniversary with us!
