The python client is based on Qt and we have been using version 5 so far. Qt5 has been replaced by Qt6 quite some time ago and as Qt5 is not really maintained any more, we have been looking at switching to Qt6 for a long time. Finally we have taken the step, this blog post will outline what has changed.
The switch to PyQt6 is not that complicated for the majority of the client code base, apart from some minor changes in various classes, the biggest update is that enumerations are handled differently. If you check out the changes in the pull request you will see that imports have changed for obvious reasons
The main blocker for doing this change has been that the Qualisys tab was using a library that used to be available in PyQt5 but that has not been added to PyQt6, we ended up removing the Qualisys tab to be able to move on. The Qualisys tab was originally contributed by Qualisys for demo purposes and it had two nice areas of functionality that we would like to re-implement:
Motion capture positioning. It was very easy to connect a Qualisys mocap system and feed the position information to the Crazyflie for automated flight.
Trajectory support. It was possible to fly some simple trajectories, for instance a circle which is nice for testing and demos.
The idea is to re-implement the mocap connectivity using the libmotioncapture library from Crazyswarm2, as an extra bonus this would support all major mocap systems. For the trajectory part, we would like to add this as a new tab that can be used by any positioning systems, mocap, Loco, Lighthouse or flow deck. If you are interested in helping out with this, let us know!
The state of the code
The changes for PyQt6 have been merged into the master branch. It seems to work but we have not tested all functionality yet, please let us know if you run into any problems or weird behavior.
When you have pulled in the code from github you will also have to re-install all dependencies by running
Today, Vivek Adajania from Learning Systems and Robotics lab write about a project for a safe motion planning of Crazyflie swarm that was published at ICRA 2023. Enjoy!
Motivation
Quadrotor swarms offer significant potential in applications like search and rescue, environmental mapping, and payload transport due to their flexibility and robustness compared to single quadrotors. The core challenge in these applications is collision-free and kinematically feasible trajectory planning. As the quadrotors share space, they must safely manoeuvre around each other and avoid collisions with static obstacles. Existing solutions [1] [2], while effective for generating collision-free trajectories, often struggle in densely cluttered scenarios due to simplifying approximations.
Background
There are two literature groups in the domain of optimization-based quadrotor swarm motion planning: centralized and distributed approaches. In a centralized setup, a central computer solves a joint optimization problem that computes trajectories for all quadrotors at once. These approaches have broad solution space but quickly become computationally intractable as the number of quadrotors increases. On the other hand, the distributed approach involves each quadrotor independently solving its optimization problem and incorporating trajectories shared by the neighbouring quadrotors. This strategy offers improved scalability, yet existing distributed approaches struggle in cluttered environments.
Fig. Centralized and distributed planning approach to quadrotor swarm motion planning. The arrows indicate the flow of communication.
In this work, we adopt a distributed planning strategy. The independent optimization problem that needs to be solved by each of the quadrotors in the distributed setup is a non-convex quadratically constrained quadratic program (QCQP). This nature of the problem stems from non-convex and quadratic collision avoidance constraints and kinematic constraints.
Existing distributed approaches rely on sequential convex programming (SCP) that performs conservative approximations to obtain a quadratic program (QP). First, linearization of the collision avoidance constraints to obtain affine hyperplane constraints. Second, axis-wise decoupling of the kinematic constraints to obtain affine box constraints. We obtain a QP but with small feasible sets.
Fig. Conservative approximations made by Sequential Convex Programming (SCP) based approaches.
Proposed Approach
In contrast, our proposed approach obtains a QP without relying on the previously mentioned approximations. The first ingredient is the polar reformulation of collision avoidance and kinematic constraints. An example of the 2D polar reformulation of collision avoidance constraints is shown below:
Fig. Example illustration of polar reformulation of 2D collision avoidance constraints.
The second ingredient is to relax the reformulated constraints as l-2 penalties into the cost function and apply Alternating Minimization. Alternating Minimization results in subproblems that are convex QPs, and some have closed-form solutions, thus obtaining a QP form without relying on linearization; further details can be found in our paper [3]. We can also use and reformulate alternative collision avoidance constraints, barrier function (BF) constraints
where hij is the Euclidean distance between quadrotor i and quadrotor j, and the parameter γ controls how fast the quadrotor i is allowed to approach the boundary of quadrotor j.
Results
We experimentally demonstrate our approach on a 12 Crazyflie 2.0 swarm testbed in challenging scenes: obstacle-free, obstacle-rich, shared workspace with a human. The experimental video is provided below:
In the simulation, we compare our approach against two SCP approaches: SCP (Continuous) [2] enforces constraints across the entire horizon, while SCP (On-demand) [1] enforces only on the first predicted collision. Our (Axiswise) includes box kinematic constraints, while Our (Quadratic) preserves the original quadratic constraints.
From our simulation results, we see that SCP (On-demand) has a lower compute time than SCP (Continuous), as SCP (On-demand) enforces fewer constraints. But, this compute time trend comes at the expense of success rate. On the contrary, our approaches achieve a high success rate with low compute times. Ours (Quadratic) has a slightly higher success rate than Ours (Axiswise) as it has access to large kinematic bounds.
Fig. Simulation results from 100 start-goal configurations with swarm sizes ranging from 10 to 50 in a cluttered environment with 16 cylindrical static obstacles.
Fig. Simulation results from 100 start-goal configurations with swarm sizes ranging from 10 to 50 and three different γvalues in a cluttered environment with 16 cylindrical static obstacles.
On average, our approaches achieved a 72% success rate improvement, a 36% reduction in mission time, and 42x faster per-agent computation time—our approach trades-off mission time with inter-agent clearance and distance to obstacles via BF constraints.
Outlook
In this work, we presented an online and scalable trajectory planning algorithm for quadrotor swarms in cluttered environments that do not rely on the linearization of collision avoidance constraints and axis-wise decoupling of kinematic constraints. We do so by reformulating the quadratic constraints to a polar form and applying alternating minimization to the resulting problem. Consequently, our planner achieves high scalability and low computation times than existing approaches. We also show that we can reformulate barrier function constraints to introduce safety behaviours in the swarm. One of the future works is to extend the approach to navigate the swarm in a complex 3D environment.
References
[1] Luis, Carlos E., Marijan Vukosavljev, and Angela P. Schoellig. “Online trajectory generation with distributed model predictive control for multi-robot motion planning.” IEEE Robotics and Automation Letters 5.2 (2020): 604-611.
[2] E. Soria, F. Schiano and D. Floreano, “Distributed Predictive Drone Swarms in Cluttered Environments,” in IEEE Robotics and Automation Letters, vol. 7, no. 1, pp. 73-80, Jan. 2022, doi: 10.1109/LRA.2021.3118091.
[3] V. K. Adajania, S. Zhou, A. K. Singh and A. P. Schoellig, “AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms in Cluttered Environments,” 2023 IEEE International Conference on Robotics and Automation (ICRA), London, United Kingdom, 2023, pp. 1421-1427, doi: 10.1109/ICRA48891.2023.10161063.
The authors are with the Learning Systems and Robotics Lab at the University of Toronto and the Technical University of Munich. The authors are also affiliated with the Vector Institute for Artificial Intelligence and the University of Toronto Robotics Institute (RI) in Canada and the Munich Institute of Robotics and Machine Intelligence (MIRMI) in Germany.
Feel free to contact us with any questions or ideas: vivek.adajania@robotics.utias.utoronto.ca. Please cite this as:
@INPROCEEDINGS{
adajania2023amswarm,
author={Adajania, Vivek K. and Zhou, Siqi and Singh, Arun Kumar and Schoellig, Angela P.},
booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
title={AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms in Cluttered Environments},
year={2023},
pages={1421-1427},
doi={10.1109/ICRA48891.2023.10161063}
}
When the Crazyflie was created the intended use case was manual flight with one drone. Over the years we have added support for positioning, swarms, autonomous flight and all sorts of nice features, and it has all been built on top of the original code base. Some of the original code is actually untouched after 7-8 years and needless to say, there is a slight worry that we might have taken design decisions back then that will come back and bite us in new use cases. This blog post will outline some of the work we have been doing to handle this problem by setting up an autonomous system where a Crazyflie is continuously flying – the infinite flight project.
The original design essentially assumed one Crazyflie that is controlled by one computer with one Crazyradio. On the computer the user was running the python client and controlling the flight manually with a game pad. The user might have restarted the Crazyflie before each flight, or at least when changing battery.
Now fast forward to the current situation where a swarm of Crazyflies might be controlled by multiple radios, each connecting to multiple Crazyflies. The Crayflies are flying autonomously, perhaps getting their current position from the Lighthouse system, or via radio based on information from a mocap system. Maybe telemetry data is sent back to the ground while commands for trajectories to the Crazyflies go in the other direction. In some systems the Crazyflies use wireless charging to be able to run continuously.
Obviously the current situation is very different from the original design with new or changed requirements. One is the extended use of the radio, and this is something that we have been talking about in some previous blog posts and we will not discuss that here. This blog post will instead be about one other important topic: long term stability.
In the original design, the Crazyflie was restarted often, maybe before each flight. This means that the code did not run for a very long time, so what happens if we use wireless charging and keep the firmware running for days? Will there be a problem? We decided to find out by starting an internal project we called “Infinite flight”. The idea was to set up a system with a Crazyflie with a Qi-charger for wireless charging and a Lighthouse deck for positioning. An app in the Crazyflie takes off, flies a trajectory and lands for recharging when the battery is out, the cycle is then repeated for as many times as possible. By doing this, we hoped to find any software problems in the firmware that might show up after some time, finding hardware that is worn out over time or other finding other issues. Spoiler alert: we have not reached infinity yet, but we have got a bit closer :-)
The setup is fairly straight forward, the firmware is based on the app used in the demo we used at IROS and ICRA, with some modifications. We have a ground station computer that collects data, it tries to continuously maintain a connection and re-establish it if it is lost. We log as much as possible to be able to analyse problems and understand what happened. We also added some tools to make it easier to visualize and dig in all the log data. The usual work flow has been to
Start the Crazyflie and run the app
Wait for something to go wrong (sometimes days)
Analyze what happened and figure out if something needs to be changed
Update and start over again
A surprising number of runs failed fairly quickly, only after a few flights. The reason has usually been some sort of handling error or problems with the test software, but some have been of more general interest or bugs.
Stopped logging
We had a problem where the logs from the Crazyflie in the ground computer stopped without any apparent reason. It turned out to be related to the session-less nature of the CRTP protocol, there is no good way to determine if a session is alive or not, other than using a timeout. It turned out that there is a timeout in the Crazyflie firmware (that was not fresh in our memories) that stops logging if no packets are received for a while. The rationale is to avoid having old logs running if a client is disconnected. In our case we lost communication for a short period of time and the firmware simply stopped the logging. The python lib on the other hand had a longer timeout and did not have the view that the connection was lost. The solution we used is to set up the logs again if we don’t receive logging for a while (we fixed an issue in the python lib related to this)
In the future com stack we plan to have proper session handling which should remove this problem.
Controller tuning
We have been using various flavors of Crazyflies in the tests, including some prototypes. We had some issues with one prototype that we did not understand, it had a hard time hitting the landing pad when landing. It turned out that the STM that was used in the prototype was reused from an old Crazyflie and it had some weird PID controller settings stored as persistent parameters. With the persistent parameters cleared, it worked as expected.
We have played a bit with tuning the controller, but the default settings are fairly OK and we used them most of the time.
Some of the prototypes we used had some glitches or irregularities, it is very hard to hand solder PCBs. These problems are only related to a specific hardware individual and can cause some unexpected behavior which takes time to figure out.
Landing pad edge
The landing/charging pad we use (also used in our demos) generally works fine, it has a “slope” towards the center which helps the Crazyflie slide to the correct position. If the Crazyflie miss the landing too much though, it will end up on the edge with one leg on the ground and an angle away from the center. In this case it sometimes fails to take off properly and crash. We solved this by adding a foam pad around the landing pad to “raise” the surrounding floor the the same level as the landing pad and thus reduce angle.
Lighthouse bug
There is a known bug in the lighthouse deck that prevents it from receiving data at certain angles. We had some cases where our landing pad was located in a spot where we lost tracking of both the active base stations for some yaw angles. If the Crazyflie happened to land in that exact yaw angle, it lost the position while charging and it did not know its current position when it should take off again. This was solved by moving the landing pad to a different position.
Not yet investigated possible problems
One possible known problem is the system tick counter in FreeRTOS. The counter is a 32-bit unsigned word and it is increased every millisecond. The counter is used in the firmware as the internal “clock” to determine for instance how long ago the estimator was updated or to determine when to execute some piece of code the next time. This counter will wrap after 2^32 ms, that is around 7 weeks, and we don’t really know what will happen.
Results and conclusions
The longest we managed to keep the system running was 5 days. We had a slow charging cycle and only flew 57 times. In this session, the flight time was very stable between 5:30 and 5:45 for all flights. This test was done with a standard Crazyflie 2.1 with a motor upgrade kit.
The second longest session was 3 days, in this case we used a brushless prototype and pushed the charging very hard. We ended up doing 276 flights but the battery was pushed beyond specs and being too warm and charged too fast, it degraded over time and the flight time was reduced to only 2-3 minutes at the end.
We believe we have fixed most of the long term stability issues, but it is hard to know. There might be bugs lurking in the firmware that only show up under very special conditions. What we do know is that it is possible to fly for 5 days!
Today, Lennart Bult from Emergent Swarns presents us with this project of a 24/7 swarming demo. Enjoy!
Over the last few months our team has been working on creating a 24/7 swarming demo. Initially tasked by Guido de Croon and Chris Verhoeven from TU Delft MAVLab and the TU Delft Robotics Institute, we set out to find our way within the Crazyflie ecosystem to gradually increase the size and capabilities of the swarm. In this article we will first talk about some of the work and methods that we used. After that, we will introduce the TU Delft Science Centre Swarming Lab and talk about some applications of swarming drones.
Developing the 24/7 swarm
The project started in February with the goal of creating a physical swarm capable of real-time collision avoidance with drones and static obstacles. We started out with three drones equipped with the Flow Deck, and by setting them up in a clever way we could perform the first collision avoidance and landing tests. We were impressed with the performance we got out of the Flow Deck, however, eventually, it is mostly a battle against the drift of the position estimate, that is, we could increase some of the margins on the collision avoidance only so far before we would either fly out of the test zone or collide with another drone. Luckily with short test flights, we were able to see some of the flaws in our algorithms and correct them before testing with the new setup.
Setup after the first expansion to eight drones.
After a few weeks of testing we got approved for the first swarm expansion, five more drones and a Lighthouse positioning setup. This is when we could do our first real tests with the collision avoidance algorithm, which, much to our own surprise, worked on the first try. This is also when we first posted a project update on LinkedIn. There were however a lot of bugs that still needed to be worked out, and a lot of system experience still to be gained. After flying for a bit longer we noticed that some of the drones would flip quite often, which is when we discovered that we needed the thrust upgrade to control the additional weight of the larger battery and charging deck.
For the charging setup we took inspiration from the Bitcraze IROS 2022 demo; we 3D printed sloped landing pads that we tape onto a wireless charger. After a few iterations we landed on a design that uses minimal printer resources and allows the Crazyflie to land a bit off-center. This last feature turned out to be quite useful considering the large amount of destabilizing airflow that is generated by 40 drones. After receiving the last order of drones we also expanded the charging setup, which at this point takes up quite a bit of floor space. There are some ideas to create a vertical landing pad stack, which would bring the additional challenge of missing the landing pad not being an option.
All 40 drones recharging before their next flight.
After prototyping the charging setup and building confidence with the initial setup, we were confident enough in our system capabilities to expand it to the point where a continuous demo of 5-8 drones is possible. Although the system integration of the previous expansion went without much trouble, we did encounter a few issues when expanding to 40 drones. The first issue of which was radio communication, we noticed that a delay in the radio communication would be present if we increased the update rate above a certain level for a specific number of drones per radio. The second issue we encountered were performance drops related to the violation of certain bounds in the collision avoidance algorithm. These two issues were very difficult to debug since it was not immediately obvious where the source of the issue was.
The third and last major issue was the increase in destabilizing airflow of 40 drones compared to 8. With 40 drones there is a noticeable breeze when you stand next to the drone cage, which is nice for summertime, but not so nice when drones need to land in a tight-packed configuration. To combat this issue there is a limit to the amount of drones that can land at the same time. There is also a minimum separation distance between two active landing pads, which reduced the severity of the induced turbulence. There are still ongoing efforts to increase the landing success rate, which is currently affected by drones running out of power during the landing procedure.
To control and monitor the swarm we designed a custom GUI, an impression of which you can see below. Although some of the buttons are still a work in progress, there are a lot of features that have already proven very useful, especially when testing a new feature.
V1 of the graphical user interface developed for the 24/7 swarm.
The code base that we created for the swarm will be largely open-sourced (only the collision avoidance will not be open-source) to provide researchers all around the world with the possibility to setup their own Crazyflie swarm for research. You can find the repository through this link. Note that the documentation and code base are still under development and might contain bugs/errors.
Human interaction
After creating all functionality to provide a continuously operating swarm demo, it was time to work on some of our stretch goals: 1. walking through the swarm whilst it is operating and 2. controlling the swarm using our arms. In the image below you can see an impression of precisely this functionality. The drones are following the operator’s gesture commands whilst performing live collision avoidance with an operator.
Team member Seppe directing the 40 drone swarm, see the full video here.
This demo requires multiple techniques and hardware elements working together to create a relatively low-latency, human-controlled swarm. We used a Kinect-like 3D sensor to perform human pose estimation, we subsequently used this data to create a dynamic obstacle in our collision avoidance software. An important element to consider here is the synchronization of the Lighthouse- and 3D sensor coordinate frames, i.e. without proper calibration the human will not be correctly positioned with respect to the drones and the drones will crash into the human. The interaction between the swarm control software and the human gesture commands also requires careful consideration, proper tuning is required to ensure a responsive system that is reliable and not too aggressive.
TUD Science Centre Swarming Lab
The next step in this project will be to set up the swarm at its new location, the TU Delft Science Centre. Here, the swarm will first and foremost be visible as a public demo, showcasing the capabilities of TU Delft state-of-the-art swarming research. There will also be a focus on developing the swarm as a research platform. This will allow TU Delft students and researchers to extend swarm functionalities and test their theory on a physical swarming system. Besides demos and academic research, there will also be worked on developing educational applications across the full educational board (primary school, high school and applied education). If you are interested in working on, or collaborating with the swarming lab on any of the above-mentioned tasks, feel free to email the lab management at operations.swarminglab@tudelft.nl.
The TU Delft Swarming Lab setup with 40 drones and charging pads for continuous operations and research.
Applications of Swarming
There are a lot of potential use-cases for fully autonomous drone swarms, ranging from indoor applications such as warehouse monitoring and factory inspection to outdoor applications such as search and rescue and surveillance. In our opinion, the true potential of drone swarms lies in applications where there is a significant need for a scalable system with a lot of built-in redundancy. A lot of additional use cases open up when we consider fully onboard autonomous systems, where the full benefits of decentralized swarming can be utilized. Currently, the size of drones needed to achieve such feats is quite large, though maybe in a few years, we could see more and more being done on drone platforms such as the Crazyflie.
A swarm inspection of an F-16 Fighting Falcon at Deltion College in Zwolle, the Netherlands.
An interesting area of application for drone swarms could be in the inspection of aircraft. Drone swarms provide a scalable and flexible means to perform a fast inspection of aircraft across an entire airfield or military base. To showcase that this can be done with any size of drone, we went to Deltion College in Zwolle to perform a mock inspection of an F-16 fighter jet. Above you can see an impression of the inspection. Another area of application is search and rescue, where there is a need for systems that can find people or objects of interest in unknown and cluttered environments. Furthermore, the area that needs to be searched is usually very large and sometimes difficult to travel on foot. A drone swarm could provide fast and reliable coverage of the area of interest, whilst providing full data traceability. Seppe and Lennart will work on creating drone swarms for these use cases with the start-up Emergent Swarms.
We’re happy to announce that there is a new release of the software for the Crazyflie ecosystem! The new release is called 2023.06 and is available for download on github or through the python client.
Major changes
The main addition is an extended supervisor framework and updated arming functionality.
Extended supervisor framework
The purpose of the supervisor is (will be) to keep an eye on the Crazyflie and make sure that everything is fine. If it detects a problem it can take action to hopefully handle the situation in a way that is better for the Crazyflie as well as people close by. The supervisor taps into the stabilizer loop and has the power to take control of the motors when needed.
The current version actually behaves very much like the previous version, but the underlying framework has been re-written to enable better handling in the future. There are now well defined states that the Crazyflie goes through for preflight checks, when flying and after landing.
Arming
Basic arming functionality has been added, mainly intended for larger platforms with brushless motors. A manual action is required after preflight checks have passed, to let the Crayflie know that a human is in control. If the system is not armed, it is not possible to fly.
Arming is required by default for the Bolt platform. For the Crazyflie 2.X, there is an auto-arming feature that immediately arms the platform when the preflight checks have passed, that is it works like it used to do.
If you use a BigQuad deck, auto-arming will also be enabled by default (as it uses the Crazyflie 2.X platform) and the firmware should be rebuilt with the MOTORS_REQUIRE_ARMING kbuild config flag set to enable manual arming.
The arming functionality is built on top of the supervisor.
Updates to the python client
An arming button has been added to the flight tab in the client to support the new arming functionality.
An emergency stop button has also been added to the top of the client window that shuts down the motors immediately.
Updates to the python library
A new CRTP message has been added to arm/disarm the system. The CRTP version has been updated to version 6.
Note, if you are controlling a Bolt from a script (or any other platform with arming enabled) you have to send an arming message to the platform before you can fly.
Release details
The following versions were released. See each release for details.
It is easy to forget that the reason why it is nice to develop for the Crazyflie is because it weighs only about 30 grams. In case something goes wrong with your script or there is a fly-away, you can simply pick it up from the air without worrying about the propellers hitting you. Moreover, when the Crazyflie crashes, it usually only requires a brush off and a potential replacement of a motor-mount or propeller. The risk of damage to yourself, other people, indoor furniture, or the vehicle itself is extremely low. However, things become very different if you’ve built a larger platform with the Bolt or BQ deck with large brushless motors (like with this blogpost), where the risk of injury to people or to the vehicle itself increases significantly. That is one of the major reasons why the BQ deck and the Bolt are still in early access and have been for a while. In our efforts to get it out of early access, it’s time to start thinking about safety features.
In this blog post, we’ll be discussing how other open-source autopilot programs are implementing safety features, followed by a discussion on current efforts for Crazyflie, along with an announcement of the developer meeting scheduled for May 3rd (see below for more info).
Catching the Crazyflie with a net
Safety in other Autopilots
We are a bit late to the game in terms of safety compared to other autopilot programs such as PX4, ArduPilot, Betaflight and Paparazzi UAV, which have been thinking about safety for quite some time. It makes a lot of sense when you consider the types of platforms that run these autopilots, such as large fixed VTOL or fixed-wing vehicles or 10-kilo quadcopters with cinematic cameras, or the degree of outdoor flight regulation. Flying a UAV autonomously or by yourself has become much more challenging as the US, EU, and many other countries have made it more restrictive. In most cases, you are not even allowed to fly if fail-safes are not implemented, such as what to do if your vehicle loses GPS signal. These types of measures can be separated into pre-flight checks and during-flight checks.
Pre-flight checks
Before a vehicle is allowed to fly, or even before the motors are allowed to spin, which is called ‘arming’, several conditions must be met. First, it needs to be checked if all internal sensors, such as the IMU, barometer, and magnetometer, are calibrated and functional, so they don’t give values outside of their normal operating range. Then, the vehicle must receive a GPS signal, and the internal state estimator (usually an extended Kalman filter) should converge to a position based on that information. It should also be determined if an external remote control is connecting to the vehicle and if there is any datalink to a ground station for telemetry. Feasibility checks can also be implemented, such as ensuring that the mission loaded to the UAV is not outside its mission parameters or that the start location is not too far away from its take-off position (assuming the EKF is functional). Additionally, the battery should not be low, and the vehicle should not still be in an error state from a previous flight or crash.
All of these features have the potential to be turned off or made less restrictive, depending on your situation. However, keep in mind that changing any of these may require recertification of the drone or make it fall outside what is required for outdoor flight regulation. Therefore, these should only be changed if you know what you are doing.
Now that the pre-flight checks have passed, the UAV is armed and you have given it the takeoff command. However, there is so much more that can go wrong during a UAV flight, and takeoff is one of the most dangerous moments where everything could go wrong. Therefore, there are many more safety features, aka failsafes, during the flight than for the pre-flight checks. These can also be separated into ‘triggers’ and ‘behaviors,’ so that the developer can choose what the UAV should do in case of a failure, such as ‘GPS loss’ to ‘land safely’ and so on.
Thus, there are triggers that can enable the autopilot’s failsafe mechanics:
No connection with the remote control
No connection with the Ground station or Datalink
Low Battery
Position estimate diverges or full GPS loss
Waypoint going beyond geofence or Mission is not feasible
Other vehicles are nearby.
Also, sometimes the support of an external Automatic Trigger system is required, which is a box that monitors the conditions where the UAV should take action in case there is no GPS, other aerial vehicles are nearby, or the UAV is crossing a geofence determined by outdoor flight restrictions. Note that all of these triggers usually have a couple of conditions attached, such as the level of the ‘low battery’ or the number of seconds of ‘GPS loss’ deemed acceptable.
Fail-safe behavior
If any of the conditions mentioned above are triggered, most autopilot suites have some failsafe behaviors linked to those set by default. These behaviors can include the following:
No action at all
Warning on the console or remote control display
Continue the mission autonomously
Stay still at the same position or go to a home position
Fly to a lower altitude
Land based on position or safely land by reducing thrust
No input to motors or completely disarming the motors
Usually, these actions are set in regulation, but per trigger, it is possible to give a different behavior than the default. One can decide to completely disarm the vehicle, but then the chances of the UAV crashing are pretty high, which can result in damage to the vehicle or cause harm to people or objects. By the way: disarming is the opposite act of arming, which is not allowing the motors to spin, no matter if it is receiving an input. If you decide to never do anything and force the drone to finish the mission autonomously, then in a case of GPS or position loss, you risk losing your vehicle or that it will end up in areas where it is absolutely not allowed, such as airports. Again, changing these default behaviors should be done by someone who knows what they are doing, and it should be done with careful consideration.
Fail-safes are measures that ensure safe flight. However, there will always be a chance that an emergency will occur, which will require an immediate action as well. If the vehicle has crashed during any of its phases or has flipped, or if the hardware breaks, such as the motors, arms, or perhaps even the autopilot board itself, what should be done then?
The standard default behavior for this is to completely disarm the vehicle so that it won’t react to any input to the motors itself. Of course, it’s difficult to do if the autopilot program is on, but at least it won’t try to take off and finish its mission while laying on its side. It might be that a backup system is connected to the ESCs that will take over in case the autopilot is not responding anymore, perhaps using a different channel of communication.
Also, the most important safety feature of all is the pilot itself. Each remote control should have a special button or switch that can put the drone in a different mode, make it land, or disarm it so that the pilot can act upon what they see. In case the motors are still spinning, have a net or towel available to throw over them, disconnect the battery as soon as possible, and make sure to have sand or a special fire retardant in case the LiPo batteries are pierced.
All of the autopilots have some tips to deal with such situations, but make sure to do some good research yourself on how to handle spinning parts or potential LiPo battery fires. I’m just giving a compilation of tips given in the documentation above here, but please make sure to read up in detail!
Safety in the Crazyflie Firmware
So how about the Crazyflie-firmware ? We have some safety features build in here and there but it is all over the code base. Since the Crazyflie is so safe, there was no immediate need for this and we felt it is more up to the developer to integrate it themselves. But with the Bolt and BQ deck coming out of early access, we want to at least do something. As we started already started looking into how other autopilot softwares are doing it, we can get some ideas, however we did notice that many of these are mostly meant for outdoor flight. The Crazyflie and the Crazyflie Bolt have been designed for indoor use and perhaps deal with different issues as well.
Current safety features
This is a collection of safety features currently in the firmware at the time of writing this blogpost. Most safety features in the Crazyflie are up for the developer to double check before and during flight, but these are some automatic once that are scattered around the firmware:
Watchdog, hard faults and asserts scattered throughout the firmware.
We might find more on the way…
However, if for instance your Crazyflie or Bolt platform loses its positioning in air, or doesn’t have a flowdeck attached before takeoff, there are no default safety systems in check. You either need to catch it, make it land or use an self-made emergency stop button using one of the emergency stop services above.
Safety features in works
As mentioned earlier, we have safety features spread throughout the code base of the Crazyflie firmware. Our current effort is to collect all of these emergency stops and triggers in the supervisor module to have them all in one place.
In addition, since indoor positioning is critical, we want to be notified when it fails. For instance, if the lighthouse geometry is incorrect, we need to see if the position diverges. This check was done outside of the Crazyflie firmware in a cflib script, but it has not been implemented inside the firmware. We also want to provide some options in terms of behavior for these triggers. Currently, we are working on two options: ‘turn the motors off’ or ‘safe land,’ with ‘safe land’ decreasing the thrust while keeping the drone level in attitude.
Furthermore, we want to integrate these features into the cfclient as well. For example, we want to add more emergency safety features to our remote control through the cfclient, and show users how to arm and disarm the vehicle.
These are the elements we are currently working on, but there might be more to come!
Developer meeting May 3rd
You probably already guessed it… the topic about the next developer meeting will be about the safety features in the Crazyflie and the Bolt! We will present the current safety features in the Crazyflie and what we are currently working on to make it better. In this sense, we really want to have your feedback on what you think is important for brushless versions of the Crazyflie for indoor flight!
The Dev meeting will be on Wednesday May the 3rd at 3 PM CEST. Please keep an eye on the discussion forum in the developer meeting thread.
In this blog post we will take a look at the new Loco positioning TDoA outlier filter, but first a couple of announcements.
Announcements
Crazyradio PA out of stock
Some of you may have noticed that there are a lot bundles out of stock in our store, the reason is the transition from Crazyradio PA to the new Crazyradio 2.0. Most bundles contain a radio and even though the production of the new Crazyradio 2.0 is in progress, the demand for the old Crazyradio PA was a bit higher than anticipated and we ran out too early. Sorry about that! We don’t have a final delivery date for the Crazyradio 2.0 yet, but our best guess at this time is that it will be available in about 4 weeks.
Developer meeting
The next developer meeting is on Wednesday, April 5 15:00 CEST, the topic will be the Loco positioning system. We’ll start out with around 30 minutes about the Loco Positioning system, split into a presentation and Q&A. If you have any specific Loco topics/questions you want us to talk about in the presentation, please let us know in the discussions link above.
The second 30 minutes of the meeting with be for general support questions (not only the Loco system).
The outlier filter
When we did The Big Loco Test Show in December, we found some issues with the TDoA outlier filter and had to do a bit of emergency fixing to get the show off the ground. We have now analyzed the data and implemented a new outlier filter which we will try to describe in the following sections.
Why outlier rejection
In the Loco System, there are a fair amount of packets that are corrupt in one way or the other, and that should not be part of the position estimation process. There are a number of reasons for errors, including packet collisions, interference from other radio systems, reflections, obstacles and more. There are several levels of protection in the path from when an Ultra Wide Band packet is received in the Loco Deck radio to the state estimator, that aims at removing bad packets. It works in many cases, but a few bad measurements still get all the way through to the estimator, and the TDoA outlier filter is the last protection. The result of an outlier getting all the way through to the estimator is usually a “jump” in the estimated position, and in worst case a flip or crash. Obviously we want to catch as many outliers as possible to get a good and reliable position estimate and smooth flight.
The problem(s)
The general problem of outlier rejection is to decide what is a “good” measurement and what is an outlier. The good data is passed on to the state estimator to be used for estimating the current position, while the outliers are discarded. To decide if a measurement is good or an outlier, it can be compared to the current position, if it is “too far away” it is probably an outlier and is rejected. The major complication is that the only knowledge we have about the current position is the estimated position from the state estimator. If we let outliers through, the estimated position will be distorted and we may reject good data in the future. On the other hand if we are too restrictive, we may discard “good” measurements which can lead to the estimator loosing tracking and the estimated position drift away (due to noise in other sensors). It is a fine balance as we use the estimated position to determine the quality of a measurement, at the same time as the output of the filter affects the estimated position.
Another group of parameters to take into account is related to the system the Crazyflie and Loco deck are used in. The over all packet rate in a TDoA3 system is changed dynamically by the anchors, the Crazyflie may be located in a place where some anchors are hidden, or the system may use the Long Range mode that uses a lower packet rate. All these factors change the packet rate and his means that the outlier filter should not make assumptions about the system packet rate. Other factors that depend on the system is the physical layout and size, as well as the noise level in the measurements, and this must be handled by the outlier filter.
In a TDoA system, the packet rate is around 400 packets/s which also puts a requirement on resource usage. Each packet will be examined by the outlier filter, why it should be fairly light weight when it comes to computations.
Finally there are also some extra requirements, apart from stable tracking, that are “nice to have”. As a user you probably expect the Crazyflie to find its position if you put it somewhere on the ground, without having to tell the system the approximate position, that is a basic discovery functionality. Similarly if the system looses position tracking, you might expect it to recover as soon as possible, making it more robust.
The solution
The new TDoA outlier filter is implemented in outlierFilterTdoa.c. It is only around 100 lines of code, so it is not that complex. The general idea is that the filter can open and close dynamically, when open all measurements are passed on to the estimator to let it find the position and converge. Later, when the position has stabilized, the filter closes down and only lets “good” measurements through. In theory this level of functionality should be be enough, after the estimator has converged it should never lose tracking as long as it is fed good data. The real world is more complex, and there is also a feature that can open the filter up again if it looks like the estimator is diverging.
The first test in the filter is to check that the TDoA value (the measurement) is smaller than the distance between the two anchors involved in the measurement. Remember that the measurements we get in a TDoA system is the difference in distance to two anchors, not the actual distance. A measurement that is larger than the distance between the anchors is not physically possible and we can be sure that the measurement is bad and it is discarded.
The second stage is to examine the error, where the error is defined as the difference between the measured TDoA value and the TDoA value at our estimated position.
float error = measurement - predicted;
This error does not really tell us how far away from the estimated position the measurement is, but it turns out to be good enough. The error is compared to an accepted distance, and is considered good if it is smaller than the accepted distance.
sampleIsGood = (fabsf(error) < acceptedDistance);
The area between the blue and orange lines represents the positions where the error is smaller than some fixed value.
The rest of the code is related to opening and closing the filter. This mechanism is based on an integrator where the time since the last received measurement is added when the error is smaller than a certain level (integratorTriggerDistance), and remove if larger. If the value of the integrator is large, the filter closes, and if it is smaller than a threshold it opens up. This mechanism implements a hysteresis that is independent on the received packet rate.
The acceptedDistance and integratorTriggerDistance are based on the standard deviation of the measurement that is used by the kalman estimator. The idea is that they are based on the noise level of the measurements.
Feedback
The filter has been tested in our flight lab and on data recorded during The Big Loco Test Show. The real world is complex though and it is hard for us to predict the behavior in situations we have note seen. Please let us know if you run into any problems!
The new outlier filter was pushed after the 2023.02 release and is currently only available on the master branch in github (by default). You have to compile from source if you want to try it out. If no alarming problems surface, it will be the the default filter in the next release.
This week’s guest blogpost is from Florian Goralsky from Bok o Bok about their dance piece with multiple Crazyflies. Enjoy!
Flying bodies across the fields is a contemporary dance piece for four performers and a swarm of drones, exploring the phenomenon of the disappearance of bees and the use of pollinating drones to compensate for this loss. The piece attempts to answer this crucial question in a poetical way: can the machine create life and save us from ecological disaster?
We’re super excited to talk about a performance that we’ve been working on for the past two years in collaboration with Bitcraze. It premiered at the Environmental Forum, Centre Pompidou Paris, in 2021, and we’ve had the opportunity to showcase it at different venues since then. We are happy to share our thoughts about it!
Choreographic research
Beyond symbolizing current attempts to use drones to pollinate fields, the presence of the Crazyflie drones, supports the back and forth between nature and technology. We integrate a swarm, performing complex choreographies, which refer to the functioning of a beehive, including the famous “bee dance”, discovered by Karl von Frisch, which is used to transmit information on the food sources. Far from having a spectacular performance as its only goal, the synchronization of autonomous drones highlights bio-inspired computer techniques, focused on collective intelligence.
Making a dance performance with drones needs a high accuracy and adaptability, both before and during the show. Usually, we only have a few hours, sometimes even a few minutes, to setup the system according to the space. We quickly realized we needed pre-recorded choreographies, and hybrid choreographies where the pilot could have a few degrees of freedom on pre-defined behaviors.
GUI Editor + Python Server
Taking this into account, we developed a web GUI editor, that is able to send choreographies created with any device to a Websocket Python server. The system supports any absolute positioning system (We use the Lighthouse), and then converts all the setpoints and actions to the Crazyflie API HighLevelCommander class. This system allows us to create, update, and test complex choreographies in a few minutes on various devices.
Preview position of six drones at a certain time.Early support of the CompressedTrajectories format, with Cubic bezier curves.
What is next?
We are looking forward to developing more dancers-drones interactions in the future. It will imply, in addition to the Lighthouse system, other sensors, in order to open up new possibilities: realtime path-finding, obstacle avoidance even during a recorded choreography (to allow improvisation), etc.
We’re happy to announce that the 2023.02 release is available for download!
The main new features of this release are:
Out of tree controllers
We have made it easier to add a new controller to the firmware in the Crazyflie. Controllers can now be added in an app, the same way as an estimator can be added. The main advantage is that all the code is contained in the app which makes it easy to upgrade the underlying firmware when new releases are available. You can read about how to use this feature in the firmware repository documentation.
Support to configure ESCs with BLHeli Configurator
On brushless Crazyflies, ESCs can now be configured using the BLHeli Configurator. See PR #1170
A UKF (Unscented Kalman Filter) state estimator has been added
An Unscented Kalman Filter (UKF) estimator has been added based by Klaus Kefferpütz from the paper ‘Error-State Unscented Kalman-Filter for UAV Indoor Navigation‘. The estimator is still slightly experimental and does not yet support all positioning methods (see this issue). Because of this, it is not available by default, but you can try it by enabling it using kbuild! You can read about the UKF estimator in the repository documentation.
Platform filter in client flash dialog
A filter has been added to the bootloader dialog in the client to make it easier to find the correct release. Releases are now filtered based on platform to avoid the clutter of mixing releases for cf2, tag, bolt and flapper.
Stability and bug fixes
We have fixed several bugs in the firmware and client software that, but you can check the release notes for each of these for further details.
We have created a simple deprecation policy to clarify future changes of the APIs. The short version is that we from now on will mention deprecated functionality in release notes and that the deprecated functionality will remain in the code base for 6 months before it is removed. Please see the development overview for more information.
A common task with the Crazyflie is to add a new controller or estimator. As we get some questions on how to do this, we will outline the process in this post. We will show how to add a custom controller and estimator that runs in the Crazyflie, built as an out-of-tree build.
This post assumes some basic knowledge about the Crazyflie firmware, the C programming language, how to build the firmware and flash it to a Crazyflie. If you need some more information on these topics, please see the “Getting started with development” tutorial. For an overview of how estimators and controllers are used by the stabilizer module, please see the firmware documentation.
Overview
The Crazyflie firmware is designed to make it easy to add custom controllers and estimators, a plugin system keeps the code clean and well separated. We will look at the details later, but the basic principle is to first write your new controller or estimator and then register it in the firmware. When the code has been compiled and flashed to the Crazyflie, the new module is activated by setting a parameter from the client or a python script.
We will implement the example as an app, which is a great way to make sure you can upgrade the underlying firmware without messing up your code. An app is a piece of code that exists somewhere in you file system outside of the main firmware source code. This setup minimizes the dependencies and the main firmware source tree can be upgraded without affecting your app (in most cases). That means there is no need for merges or complex management of source trees.
Registration of modules
Let’s first look at how controllers and estimators are registered and called in the plugin framework. We will use the controllers to show how it works, but the estimators are implemented in a similar way and it should be easy to understand how it works.
Note that there has been some updates of the Crazyflie firmware source code lately and any reference to the source code will be to the latest version (as of today).
The starting point of the controller implementation can be found in the src/modules/src/controller.c file, here we can find an array called controllerFunctions that holds a list of all the controllers in the system.
We can see that there is currently four controllers in the list: the PID controller, the Mellinger controller, the INDI controller and finally the Brescianini controller. There is also an “empty” controller at the top that is not important in this context and we will simply ignore it. At the bottom we find the out-of-tree controller, we will discuss this later.
Each controller must implement three functions: an initialization function, a test function and a controller function that performs the actual controller work. Signatures for the three functions are defined in controller.c. The functions are added to the list as function pointers that can be called by the stabilizer when needed.
There is a parameter, stabilizer.controller, in the stabilizer that tells the system which controller to use in the stabilizer loop. This parameter simply contains the index in the controllerFunctions list that will be used. For example, the default value 1 will make the stabilizer loop call the controllerPid function every iteration. If the value of the stabilizer.controller parameter is changed, the initialization function for the new controller will be called and subsequent calls from the stabilizer loop will be done to the new controller function.
We will not go into details of how to implement the actual controller here, but the existing controllers can be used as examples.
Suppose you want to add a new controller. It would be possible to add a new file in the Crazyflie firmware with your new controller implementation, add the function pointers to the list in controller.c and that would work just fine. The problem with such implementation would be that it is hard to maintain, your new files would be mixed with the files in the main firmware file tree, and even worse, you would have to modify the controller.c file to add your controller. The next time there is a new awesome feature in the firmware source code and you want to upgrade to the latest version, you will run into problems as you have to handle the files you modified!
A better solution is to use an app instead as apps are built out-of-tree, that is not in the main source tree. This removes the problem of merging changes in the main source files, all you have to do is to pull in the new file tree and recompile.
But how to register your new controller in the controller list? This is what the last line in the list of controllers is for
If CONFIG_CONTROLLER_OOT is defined we add a controller with the three functions controllerOutOfTreeInit, controllerOutOfTreeTest and controllerOutOfTree. All you have to do in your app is to define CONFIG_CONTROLLER_OOT and make sure the functions in your controller are named like above. That’s it!
Example implementation
Now we will create a new app and add a new controller, step by step. We assume that you have a newly cloned firmware repository in your filesystem to work on.
We will show the linux flavor of commands, but it should be easy to convert to other platforms.
Create a new app
The easiest way is to start from an existing app to get started, let’s use the hello world app. Copy the app and move into the new directory
cp -r examples/app_hello_world examples/my_controller
cd examples/my_controller/
Let’s rename hello_world.c
mv src/hello_world.c src/my_controller.c
We have to tell kbuild that we renamed the file. Open src/Kbuild in your favorite editor and update it to
obj-y += my_controller.o
Now let’s fix the basics in my_controller.c, open it in your editor and change according to the comments bellow:
#include <string.h>#include <stdint.h>#include <stdbool.h>#include "app.h"#include "FreeRTOS.h"#include "task.h"// Edit the debug name to get nice debug prints#define DEBUG_MODULE "MYCONTROLLER"#include "debug.h"// We still need an appMain() function, but we will not really use it. Just let it quietly sleep.
void appMain() {
DEBUG_PRINT("Waiting for activation ...\n");
while(1) {
vTaskDelay(M2T(2000));
// Remove the DEBUG_PRINT.// DEBUG_PRINT("Hello World!\n");
}
}Code language:PHP(php)
Now, lets add our new controller. We will not add a real implementation here as it would be a bit too large for this post, instead we will just call into the PID controller to make sure the Crazyflie still can fly. Add this code after appMain() in my_controller.c.
// The new controller goes here --------------------------------------------// Move the includes to the the top of the file if you want to#include "controller.h"// Call the PID controller in this example to make it possible to fly. When you implement you own controller, there is// no need to include the pid controller.#include "controller_pid.h"
void controllerOutOfTreeInit() {
// Initialize your controller data here...// Call the PID controller instead in this example to make it possible to fly
controllerPidInit();
}
bool controllerOutOfTreeTest() {
// Always return truereturntrue;
}
void controllerOutOfTree(control_t *control, const setpoint_t *setpoint, const sensorData_t *sensors, const state_t *state, const uint32_t tick) {
// Implement your controller here...// Call the PID controller instead in this example to make it possible to fly
controllerPid(control, setpoint, sensors, state, tick);
}
Code language:PHP(php)
Finally we need to tell the firmware that we have implemented the out-of-tree controller and that it should be added to the list. We do this by adding CONFIG_CONTROLLER_OOT to the app-config file. When you are done it should look like this:
Start your Crazyflie and the python client. Connect the client to the Crazyflie and open the console log tab. Make sure you are running your app by looking for the line:
MYCONTROLLER: Waiting for activation ...Code language:HTTP(http)
Now let’s activate our new controller! Open the parameter tab, find the stabilizer group and the controller parameter. Set it to 5 and check the console log that the out-of-tree controller was activated:
CONTROLLER: Using OutOfTree (5) controllerCode language:HTTP(http)
That’s it! Your new controller is activated and the Crazyflie is ready to fly.
Note: In the client, the comment for the stabilizer.controller parameter will not contain the out-of-tree controller, and it will look like only values 0-4 are valid even though 5 also works.
Conclusions
In this post we have shown how to add a new controller to the Crazyflie firmware. The process for adding an estimator is very similar, and hopefully it should be easy to understand how to do it based on the example above.
As you can see, very little code (apart from the actual controller/estimator) is required to add your own controller or estimator, and we hope that it will enable you to put your energy into the actual control problem, rather than the nitty gritty details of the code.
