H-ModQuad: Modularity for Versatility

2022-07-25 | David | Leave a comment

This week we have a guest blog post from Jiawei Xu and David Saldaña from the Swarmslab at Lehigh University. Enjoy!

Limits of flying vehicles

Advancements in technology have made quadrotor drones more accessible and easy to integrate into a wide variety of applications. Compared to traditional fixed-wing aircraft, quadrotors are more flexible to design and more suitable for motioning, such as statically hovering. Some examples of quadrotor applications include photographers using mounting cameras to take bird’s eye view images, and delivery companies using them to deliver packages. However, while being more versatile than other aerial platforms, quadrotors are still limited in their capability due to many factors.

First, quadrotors are limited by their lift capacity, i.e., strength. For example, a Crazyflie 2.1 is able to fly and carry a light payload such as an AI deck, but it is unable to carry a GoPro camera. A lifter quadrotor that is equipped with more powerful components can transport heavier payload but also consumes more energy and requires additional free space to operate. The difference in the strength of individual quadrotors creates a dilemma in choosing which drone components are better suited for a task.

Second, a traditional quadrotor’s motion in translation is coupled with its roll and pitch. Let’s take a closer look at Crazyflie 2.1, which utilizes a traditional quadrotor design. Its four motors are oriented in the same direction – along the positive z-axis of the drone frame, which makes it impossible to move horizontally without tilting. While such control policies that convert the desired motion direction into tilting angles are well studied, proven to work, and implemented on a variety of platforms [1][2], if, for instance, we want to stack a glass filled with milk on top of a quadrotor and send it from the kitchen to the bedroom, we should still expect milk stains on the floor. This lack of independent control for rotation and translation is another primary reason why multi-rotor drones lack versatility.

*Fig 1. A crazyflie has four propellers generating thrust forces in parallel. Credit to:* *https://robots.ros.org/crazyflie/*

Improving strength

These versatility problems are caused by the hardware of a multi-rotor drone designed specifically to deal with a certain set of tasks. If we push the boundary of these preset tasks, the requirements on the strength and controllability of the multi-rotor drone will eventually be impossible to satisfy. However, there is one inspiration we take from nature to improve the versatility in the strength of multi-rotor drones – modularity! Ants are weak individual insects that are not versatile enough to deal with complex tasks. However, when a group of ants needs to cross natural boundaries, they will swarm together to build capable structures like bridges and boats. In our previous work, ModQuad [3], we created modules that can fly by themselves and lift light payloads. As more ModQuad modules assemble together into larger structures, they can provide an increasing amount of lift force. The system shows that we can combine weak modules with improving the versatility of the structure’s carrying weight. To carry a small payload like a pin-hole camera, a single module is able to accomplish the task. If we want to lift a heavier object, we only need to assemble multiple modules together up to the required lift.

Improving controllability

On a traditional quadrotor, each propeller is oriented vertically. This means the device is unable to generate force in the horizontal direction. By attaching modules side by side in a ModQuad structure, we are aligning more rotors in parallel, which still does not contribute to the horizontal force the structure can generate. That is how we came up with the idea of H-ModQuad — we would like to have a versatile multi-rotor drone that is able to move in an arbitrary direction at an arbitrary attitude. By tilting the rotors of quadrotor modules and docking different types of modules together, we obtain a structure whose rotors are not pointing in the same direction, some of which are able to generate a force along the horizontal direction.

H-ModQuad Design

H-ModQuad has two major characteristics: modularity and heterogeneity, which can be indicated by the “Mod” and “H-” in the name. Modularity means that the vehicle (we call a structure) is composed of multiple smaller modules which are able to fly by themselves. Heterogeneity means that we can have modules of different types in a structure.

As mentioned before, insects like ants utilize modularity to enhance the group’s versatility. Aside from a large number of individuals in a swarm that can adapt to the different scales of the task requirement, the individuals in a colony specializing in different tasks are of different types, such as the queen, the female workers, and the males. The differentiation of the types in a hive helps the group adapt to tasks of different physical properties. We take this inspiration to develop two types of modules.

In our related papers [4][5], we introduced two types of modules which are R-modules and T-modules.

*Fig 2. Major components of an H-ModQuad “T-module” we are using in our project. We use Bitcraze Crazyflie Bolt as the central control board.*

An example T-module is shown in the figure above. As shown in the image, the rotors in a T-module are tilted around its arm connected with the central board. Each pair of diagonal rotors are tilted in the opposite direction, and each pair of adjacent rotors are either tilting in the same direction or in the opposite direction. We arrange the tilting of the rotors so that all the propellers generate the same thrust force, making the structure torque-balanced. The advantage of the T-module is that it allows the generation of more torque around the vertical axis. One single module can also generate forces in all horizontal directions.

An R-module has all its propellers oriented in the same direction that is not on the z-axis of the module. In this configuration, when assembling multiple modules together, rotors from different modules will point in different directions in the overall structure. The picture below shows a fully-actuated structure composed of R-modules. The advantage of R-modules is that the rotor thrusts inside a module are all in the same direction, which is more efficient when hovering.

*Structure 1: Composed of four types of R-modules.*

Depending on what types of modules we choose and how we arrange those modules, the assembled structure can obtain different actuation capabilities. Structure 1 is composed of four R-modules, which is able to translate in horizontal directions efficiently without tilting. The picture in the intro shows a structure composed of four T-modules of two types. It can hover while maintaining a tilting angle of up to 40 degrees.

Control and implementation

We implemented our new geometric controller for H-ModQuad structures based on Crazyflie Firmware on Crazyflie Bolt control boards. Specifically, aside from tuning the PID parameters, we have to change the power_distribution.c and controller_mellinger.c so that the code conforms to the structure model. In addition, we create a new module that embeds the desired states along predefined trajectories in the firmware. When we send a timestamp to a selected trajectory, the module retrieves and then sends the full desired state to the Mellinger Controller to process. All modifications we make on the firmware so that the drone works the way we want can be found at our github repository. We also recommend using the modified crazyflie_ros to establish communication between the base station and the drone.

Videos

Challenges and Conclusion

Different from the original Crazyflie 2.x, Bolt allows the usage of brushless motors, which are much more powerful. We had to design a frame using carbon fiber rods and 3-D printed connecting parts so that the chassis is sturdy enough to hold the control board, the ESC, and the motors. It takes some time to find the sweet spot of the combination of the motor model, propeller size, batteries, and so on. Communicating with four modules at the same time is also causing some problems for us. The now-archived ROS library, crazyflie_ros, sometimes loses random packages when working with multiple Crazyflie drones, leading to the stuttering behavior of the structure in flight. That is one of the reasons why we decided to migrate our code base to the new Crazyswarm library instead. The success of our design, implementation, and experiments with the H-ModQuads is proof of work that we are indeed able to use modularity to improve the versatility of multi-rotor flying vehicles. For the next step, we are planning to integrate tool modules into the H-ModQuads to show how we can further increase the versatility of the drones such that they can deal with real-world tasks.

Reference

[1] D. Mellinger and V. Kumar, “Minimum snap trajectory generation and control for quadrotors,” in 2011 IEEE International Conference on Robotics and Automation, 2011, pp. 2520–2525.

[2] T. Lee, M. Leok, and N. H. McClamroch, “Geometric tracking control of a quadrotor uav on se(3),” in 49th IEEE Conference on Decision and Control (CDC), 2010, pp. 5420–5425.

[3] D. Saldaña, B. Gabrich, G. Li, M. Yim and V. Kumar, “ModQuad: The Flying Modular Structure that Self-Assembles in Midair,” 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 691-698, doi: 10.1109/ICRA.2018.8461014.

[4] J. Xu, D. S. D’Antonio, and D. Saldaña, “Modular multi-rotors: From quadrotors to fully-actuated aerial vehicles,” arXiv preprint arXiv:2202.00788, 2022.

[5] J. Xu, D. S. D’Antonio and D. Saldaña, “H-ModQuad: Modular Multi-Rotors with 4, 5, and 6 Controllable DOF,” 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 190-196, doi: 10.1109/ICRA48506.2021.9561016.

Flapper Drones – bioinspired robots powered by Crazyflie Bolt

2022-07-11 | Matěj Karásek | Leave a comment

As the Crazyflie ecosystem expands, more and more novel aerial (but also ground or hybrid) robots are being built with one of the Crazyflie controllers onboard. For recent examples, you can check e.g. the recent blogpost about ICRA 2022.

In this post, I will introduce yet another Crazyflie-Bolt-powered aerial robot, the Flapper Nimble+ from our company Flapper Drones, which unlike other flying robots doesn’t have any propellers but uses flapping wings instead.

The best aerial robot design is…

Small drones, or micro air vehicles, have seen a lot of progress and new developments in the last 20 years. The most widespread design nowadays is a quadcopter, such as the Crazyflie 2.1. But is a quadcopter the ultimate (micro) drone solution? At Flapper Drones, we believe nature might provide even better designs… For some applications at least! 😊

Flying like a bird…

Flapper Drones is a spinoff of the MAVLab of the Delft University of Technology. At the MAVLab, we have been researching bio-inspired flight as part of the DelFly project since 2005. From the beginning, the goal has been to develop a lightweight, mission capable micro air vehicle, the design of which would draw inspiration from nature. Over the years, many such MAV concepts have been designed, built and tested, including the DelFly Micro, the world’s smallest camera-equipped MAV, or the DelFly Explorer, the first autonomous flapping-wing MAV equipped with a stereovision system. All these designs were propelled by a pair of flapping wings, while being controlled (and passively stabilized) by a tail such as birds or men-made airplanes.

… or an insect

The latest design, the DelFly Nimble is insect-inspired instead. What does that mean? The Nimble has no tail, which would provide the damping needed for stable flight. Instead, it is stabilized actively, by adjustments of the motion of its flapping wings. This is what all flying insects and also hummingbirds do. Flies, for example, sense their body motions with their halteres, drum-stick like biological gyroscopes, and adapt their wing motion accordingly to stay balanced…. or to be agile, when someone is trying to swat them!

And while the Nimble was originally built just to demonstrate that an insect-inspired flying robot can be built, eventually we could also use it to learn more about the flight of insects:

Flapper Drones – how do they work?

The Flapper Nimble+ is the commercial (and enlarged) version of the DelFly Nimble, developed and produced by Flapper Drones. To our knowledge, it is the first, and so far the only hover-capable tailless flapping-wing drone available!

The thrust keeping the Nimble+ airborne is created by its four flapping wings, which flap back and forth horizontally, about 10 to 12 times per second.

The wing actuation mechanism allows to adjust the flapping frequency of the left and right wing pairs independently, which enables control of the roll rotation. Pitch rotation is controlled by adjusting the mean wing position within the stroke plane, which shifts the mean thrust force forward or backward with respect to the center of mass, and also introduces a stabilizing dihedral angle in forward flight. Finally, yawing motion is achieved by tilting the wing roots of the left and right wing pair asymmetrically:

Advantages of flapping wings

The use of flapping-wing drones such as the Flapper Nimble+ brings several advantages. Next to their attractive biological appearance, the soft flapping wings produce less intrusive, low frequency sound and are safer, compared to propellers. As the wings move back and forth, minor mid-air collisions are not a problem. The wings bounce off objects leaving no damage, and the drone keeps flying as this only represents a minor disturbance:

The aerial drag characteristic is also different and helps with precise indoor flight. As soon as zero attitude is commanded, the Nimble+ goes into halt in a matter of several wingbeats, making it an ideal choice for novice drone pilots as well as in constrained or cluttered indoor spaces. Finally, the flapping wings can provide additional lift force as they also glide in forward flight. This can improve the power efficiency by over 20%, compared to hovering.

Otherwise, Flapper Drones can be operated as any other drone, with vertical take offs and landings, quick maneuvers and flight in any direction:

Crazyflie Bolt & compatibility

The Flapper Nimble+ is powered by the Crazyflie Bolt 1.1, where the Bolt’s BMI088 IMU and STM32F4 MCU are suitable substitutes to the halteres and brains of the real fly. We made this choice, because this enables compatibility with most of the Crazyflie ecosystem, but also, because we felt the only way a Crazyflie would do justice to its name is if it had flapping wings😊

Currently, the Nimble+ uses a fork of the Crazyflie firmware, which is of course open source. Moreover, with the recently introduced platform functionality, we will be able to include the Flapper platform into the official crazyflie firmware very soon (expected still in July 2022). This means that the Flapper remains compatible with the official Python libraries, the PC client or the smartphone app. But also third-party projects like the Crazyswarm or the Skybrush should only require minor adjustments, if any, to operate a swarm of Flappers. Thus, for the existing Crazyflie users, switching from a Crazyflie to the Flapper should be a breeze!

The Flapper Nimble+ is hardware compatible with most of the Crazyflie expansion decks. While software support remains experimental (the Flapper Nimble+ is not a native Crazyflie product, after all), many of the decks work out of the box and others might need just minor firmware modifications. Would you like to fly the Nimble+ autonomously? Add an LPS or Lighthouse deck and you’re good to go!

For more details regarding deck compatibility, you can check this overview.

Applications

While the Nimble+ was originally designed for drone shows and similar entertainment applications, the open-source firmware and expansion decks enabling autonomous flight make it ideal also as for academic research and, in general, as a development platform. Are you researching swarming, and would you like to make your swarm even more bio-inspired? Are you developing new sensors, or new controllers (possibly even bioinspired), which you would like to test on a new type of flying platform? Are you interested in the aerodynamics of flapping wings, or the flight dynamics of insect-like flight? Or are you just curious and would you like to learn more about bioinspired flight? In all these cases, a Flapper might be what you are looking for!

The 114-g and 49-cm wide Flapper Nimble+ has been designed as a modular system where any part can easily be replaced. Flapper Drones provides all the spares, which are available upon request. If you are interested in using the Nimble+ for entertainment, rather than research, you can modify the appearance by creating your own body shells, which can also be illuminated by RGB Leds (a suitable interface and power supply is already integrated). Or even by altering the design of the wings. Finally, you can easily extend the Flapper with your own sensors, or other devices. Would you like to add a tail? A gripper? A perching device? This is all possible, as long as these additions fit within the payload limit of about 25 grams.

Available soon in the webstore!

Did you get (bio)inspired, and would you like to try an insect-like flying robot yourself? Then we have some good news! The Flapper Nimble+ will soon be available for sale in an exclusive partnership with Bitcraze and their webstore. Checkout the product description and leave your email address behind, such that you get a notification when the Flappers are in stock and ready to ship. The first batch of 10 units is expected to be available at the end of summer, so do not wait too long 😉

Want to learn more?

To learn more about Flapper Drones, you can check our website, or watch the talk I gave at the last miniBAM:

Digital Futures Drone Arena – a one-of-a-kind hackathon in Stockholm in June

2022-05-23 | Airi Lampinen | Leave a comment

This week, we welcome Airi Lampinen from Stockholm University, to talk about the Crazyflie competition she’s organizing in Stockholm.

Welcome to our one-of-a-kind hackathon with Bitcraze’s Crazyflie in Stockholm, Sweden, on June 15-17, 2022! If you are curious about how technology and humans may play together, enthusiastic about the Crazyflie, or eager to learn how to use the Crazyflie, this event is for you.

What, where, when? The Inaugural Challenge at the Digital Futures Drone Arena takes place on June 15-17, 2022 at KTH’s Reactor Hall – a dismantled nuclear reactor hall – which – especially if you haven’t been to this cool space before – makes attending the event worthwhile in its own right. In 2016, the reactor hall was used to film the music video for Alan Walker’s song Faded (Restrung).

Who can join? Anyone irrespective of age, profession and past experience with drones is welcome to participate. We welcome up to 10 teams of 2-4 people. We provide all the necessary drone hardware to the participants. We use the Crazyflie 2.1 and the Lighthouse positioning system. All that a team needs to bring along is a computer. Registration is open, with a final deadline on June 5 – we encourage those interested to sign up as soon as possible to secure their spot!

Program & prizes? On the first day of the hackathon, we will run short tutorials for those with no or little previous drone experience. The teams will then have access to the Reactor Hall to work on the challenge and conduct trial runs with their drone – we offer long hours but each team is free to choose how much they want to work. (The goal here is to have a good time!) The competition itself takes place on the third and final day. We’ve got exciting prizes for the most successful teams!

Read more about the challenge, the prizes, and how to sign up on our website: http://www.dronearena.info/

The event is organized as a part of the Digital Futures demonstrator project Digital Futures Drone Arena led by Luca Mottola from RISE and Airi Lampinen from Stockholm University.

Bitcraze Announcements

We have also some Bitcraze news to share with you:

Last wednesday, we had our very first mini BAM, and it led to 2 hours of interesting talks and exciting discissions ! If you’ve missed it, you can find the recordings in your Youtube Channel: here for Flapper Drones’ presentation, and here for Collmot‘s talk. We plan on having at least one another mini BAM before the end of the year, so stay tuned if you’re interested in those events.

Finally, as I talked about in this blogpost, we are looking for a new team mate to add to the Bitcraze crew. You’re interested? Check out our jobs page if you want to learn more !

Introduction to Robotics at Princeton

2022-01-24 | Anirudha Majumdar | 1 Comment

This week we have a guest blog post from Anirudha Majumdar from Mechanical & Aerospace Engineering, Princeton University. Enjoy !

Overview

The course “Introduction to Robotics” at Princeton (MAE 345 / ECE 345 / COS 346) introduces students to fundamental theoretical and algorithmic principles in robotics. We use the Crazyflies to bring the excitement of drones to students in their very first introductory course in robotics. The course is primarily aimed at undergraduate students in their third and fourth years, and also offers a graduate-level track. In Fall 2021, approximately seventy undergraduate students and ten graduate students enrolled in the course from a variety of different majors: Mechanical and Aerospace Engineering, Computer Science, Electrical and Computer Engineering, Mathematics, Operations Research and Financial Engineering, Civil and Environmental Engineering, and Architecture.

The course covers the following technical topics:

Feedback Control;
Motion Planning;
State estimation, localization, and mapping;
Computer vision and learning;
Broader topics: Robotics and the law, ethics, and economics.

One of the primary aims of the course is to allow students to obtain hands-on experience implementing various algorithms on a hardware platform. For this, we use the Crazyflie 2.1 quadrotors with a camera attachment (Fig. 1). The Crazyflie is an ideal hardware platform for our course due to its light weight, reliability, safety, and open-source code. Throughout the semester, students work in teams to implement different algorithms from the course:

(i) linear quadratic regulator (LQR) for stabilizing the drone

(ii) rapidly-exploring randomized trees (RRTs) for motion planning through a forest of PVC pipe obstacles

(iii) the Lucas-Kanade optical flow algorithm

(iv) object-following with neural networks.

In the final project, students implement algorithms for autonomous vision-based navigation through novel (i.e., a priori unknown) obstacle environments (Fig. 2). Below, we describe the modified Crazyflies used for the course, along with the hands-on projects in more detail.

Open-source course materials and website

We have made the course materials (lecture notes, slides, and assignments) freely available through our course website. The assignments include theory questions, programming assignments in Python, and hardware implementation with the Crazyflie. All programming and hardware assignments are freely-available through GitHub. We hope that the course materials will be useful to instructors at other institutions in getting the next generation of engineers excited about robotics!

*Fig. 1. Crazyflie with a camera attachment.*

Final project for course: vision-based navigation and target-seeking.

Hardware components and lab spaces

The feedback control and motion planning hardware assignments can be completed with the following parts list:

For the vision-based assignments, we attach cameras to the drones, which transmit images in real-time to a receiver unit plugged into to a laptop. This requires the following additional parts:

We set up three netted arenas for students to complete hardware assignments. Each arena is approximately 12 ft x 8 ft in size; two of the arenas are shown in Fig. 3.

*Fig. 3. Two netted flying arenas used for the course.*

Crazyflie assignments

Lab 1: Linear Quadratic Regulator (LQR)

In the first hardware lab, students implement and tune a Linear Quadratic Regulator (LQR) controller to make the drone hover. This builds on the theoretical foundations of feedback control covered in the first module of the course. Implementing LQR control on a physical system allows students to get a sense for the “reality gap”, i.e., assumptions made by the theory that are not perfectly satisfied in real life (e.g., linear dynamics, perfect state estimation, and perfect knowledge of the dynamics). This assignment makes use of a modified version of the Crazyflie firmware that allows us to upload different controller gains (obtained using LQR) and run these on the drones. Code for this assignment is available here.

Lab 2: Rapidly-Exploring Randomized Trees (RRTs)

In the second lab, students implement the RRT algorithm for motion planning. Each netted arena is set up with PVC pipe obstacles (similar to Fig. 2). Students measure the obstacle locations and radii and use the RRT algorithm to find a collision-free path from a starting location to a goal location. The Crazyflie then uses its waypoint tracking capabilities to execute the motion plan. Code for this assignment is available here.

Lab 3: Lucas-Kanade optical flow

In the computer vision module of the course, we first introduce students to “classical” vision approaches (before discussing modern deep learning techniques). One of the primary algorithms we discuss in this module is the Lucas-Kanade algorithm for computing optical flow (which has many applications such as computing time-to-collision). Students record a video using the drone’s on-board camera attachment and process this offline to compute optical flow. Code for this assignment is available here.

Lab 4: Object tracking

In the computer vision module of the course, we introduce students to deep learning-based approaches to vision. As a demonstration of the power of deep learning, we use neural networks to perform real-time object detection. Specifically, students use neural networks trained on the Coco dataset to detect a person (or an object such as a cup) using the drone’s camera, and use the resulting bounding box to command the drone in a way that makes it follow the person or object. The neural networks can be run on standard laptop computers at >= 30Hz, allowing the drone to follow the person or object in real-time. Code for this assignment is available here.

Final project: Vision-based obstacle avoidance and target-seeking

In the final project for the course, students implement vision-based navigation and target-seeking using the Crazyflie drones. Teams must program the drones to autonomously navigate through a forest of PVC pipes, whose locations are not known beforehand. Students utilize different computer vision algorithms (e.g., edge detection, contour finding, etc.) to detect obstacles and program strategies to avoid these obstacles. The goal is to navigate to the other side of the flying arena and land in front of a target object (in the form of a book). Again, teams use different approaches (e.g., neural networks) to find the book and then land in front of it. Further details and starter code for the project can be found here. See above for a video.

Acknowledgements

The following teaching staff have contributed greatly to the development of the materials for this course: Vincent Pacelli, Julienne LaChance, Jon Prevost, Alec Farid, Meghan Booker, Lena Rosendahl, David Snyder, Allen Ren, and Eric Lepowsky.

A Modified EKF for Constant Altitude Flights

2022-01-10 | Dominik | Leave a comment

Today we will have a guest blogpost by Dominik Natter, working in the Robotics & Control group at SINTEF in Trondheim, Norway. Enjoy!!

In this blogpost we will teach you how to fly the Crazyflie beyond edges without crashing, using only on-board sensors. Come join in!

flying over edges — Safe flights across edges are achievable!

Introduction

UAVs have seen tremendous progress in the last decades and have since moved from research labs to various real-world environments. Small UAVs (so-called micro air vehicles, MAVs) like the Crazyflie open up even more possibilities. For example, their size allows them to traverse narrow passages or fly in cluttered environments (as recently showcased in this blog post). However, in order to achieve these complex tasks the community must further improve the cognitive ability of these MAVs in order to avoid crashes.

One task on this list and today’s topic is the possibility to fly at constant altitude irrespective of the terrain. This feature has been discussed in the community already two years ago. To understand the problem, let’s look at the currently implemented solution: With the Flow deck mounted the Crazyflie uses a 1D lidar sensor to estimate its vertical position. This vertical position (more or less) equals the current sensor reading. On flat floors this solution works very well. However, if the Crazyflie shall traverse through a narrow window or fly above irregular terrain its altitude will change based on the sensor readings. This can lead to unstable flights, as in the following video, or even crashes!

You might wonder: why not use any of the other great tools from the Bitcraze universe? Indeed, the Lighthouse positioning system and the Loco positioning system work well for absolute positioning (as we have seen earlier, e.g., in this blog post). However, the required setups are often not available in difficult environments. Alternatively, the barometer could be used to achieve a solution based solely on on-board sensors. In fact, Bitcraze has proposed an altitude hold functionality a few years ago. This is a cool feature, but its positioning accuracy of “roughly ±15cm” is not fully satisfying. Finally, relying on the on-board IMU alone will inevitably lead to drifting over time.

Thus, we propose a solution based on the Flow deck and the Multiranger deck. This approach, only based on on-board sensors, allows to fly at constant altitude with obstacles above, below, or even both above and below the Crazyflie. Kristoffer Skare developed this solution when he worked with us as an intern in 2021.

Technical Description

As a first step, the upward-facing lidar of the Multiranger deck is incorporated in the same way the downward-facing lidar of the Flow deck is used in the firmware. This additional measurement can then be used in the extended Kalman filter (EKF) to improve the state estimation. Currently, the EKF estimates and outputs 1 value for the altitude. For our purpose two more states are added to the EKF: one state is defined as the height of the object under the Crazyflie compared to the height where the altitude state is defined as 0. Similarly, the other state is defined as the height of the object above the Crazyflie compared to the same reference height. The Crazyflie keeps therefore track of the environment in order to keep its own altitude constant. To achieve this, an edge detection was implemented: The errors between the predicted and measured distance are tracked in both the upward or downward range measurement. If either of these errors is too large the algorithm assumes that the floor or roof has changed (while the original EKF would think the drone’s position has changed, triggering a change in thrust). Thus, the corresponding state gets updated. For more details on the technical implementation and the code itself, check out our pull request.

Results

To analyze our approach we have used a Qualisys motion capture system. We have conducted many different tests: flying over different obstacles, flying at different velocities, flying at different altitudes, or even flying under different lighting conditions. Exemplarily, in this post we will have a look at a baseline example, a good estimate, and a bad estimate. In each picture you can see the altitude (in meters) over time (in seconds) for different flight speeds (in centimeters per second). You will see three lines: The motion capture ground truth (blue), the altitude estimated by our code (orange), and the new state keeping track of the floor height (green). For each plot, the Crazyflie takes off, flies in positive x direction, and lands.

In the baseline experiment, it flies over a flat floor. Clearly, the altitude estimates follow the ground truth values well, and the floor is correctly estimated to be flat.

In the next example, we have added a box with an approximate height of 0.225 m and made the Crazyflie fly over it. Despite the obstacle the altitude estimates follow the ground truth values well. Note how the floor estimates indicates the shape of the box.

experiment with box — Experiment flying over a box

Because the algorithm is based on an edge detection, we had a hunch that smoothly changing obstacles will pose a problem. Indeed, the estimates can be messy as we see in the next example. Here, the Crazyflie flies over an orthogonal triangle, with the short leg at 0.23 m pointing upwards and the long leg with 0.65 m pointing in flight direction (thus forming a slope). For different flight speeds different the estimates turn out quite differently.

experiment with slope — Experiment flying over a slope

If you don’t like looking at plots, check out this video with some cool shots instead!

Conclusion

To summarize, we propose a solution for constant altitude flight with Crazyflies, using the Flow deck and the Multiranger deck. We have tested it successfully under various circumstances. Still, we see some potential for improvement, e.g. when dealing with slopes. In addition, the current implementation is quite a change to the original EKF, which poses a problem for integration.

Thus, a way forward can be an out-of-tree build to ease the use of the solution for the community. At SINTEF we certainly plan to deploy this code in all of our tests in 2022, which will hopefully allow us to gather more experience and thus find further ways to improve or tune the system.

We want to emphasize that this is not a perfect solution. That means a) you should use it with care and b) you are very much welcomed to contribute. E.g. feel free to chime in in the pull request, test the code in your environments, propose improvements, or implement an out-of-tree build! :) Maybe you can even come up with an alternative approach for constant altitude flights?

If you want to check out more of our work, visit our website. Also, keep reading this amazing blog from Bitcraze as we try to be back some day (if Bitcraze wants us hehe)!

Real-time control of a swarm of Crazyflies in artificial forests

2021-11-08 | Enrica | 4 Comments

This week we have a guest blog post from Enrica Soria from the Laboratory of Intelligent Systems Faculty of Ecole Polytechnique Fédérale de Lausanne (EPFL) . Enjoy!

From Star Wars to Black Mirror, sci-fi movies predict a future where thousands of drones will fill our sky. Curving sharply around trees or soaring over buildings, they fly just like a flock of starlings. To turn this vision into a reality, real drone swarms need to increase their autonomy and operate in a decentralized fashion. In a decentralized swarm, each robot makes its own decision based only on local information. Decentralization not only allows the swarm to be more robust to the failure of single individuals, but also removes the dependency from a single computing unit, thus making the swarm more scalable in terms of size.

We at LIS (EPFL) have shown that predictive controllers can improve the safety of aerial swarms by predicting and optimizing the agents’ future behavior in an iterative process. However, the centralized nature of this method allowed us to only control five drones and prevented us from scaling up to a large number of drones. For this reason, we have worked on a novel decentralized and scalable swarm controller that allows the safe and cohesive flight of aerial swarms in cluttered environments. In our latest article, published in IEEE Robotics and Automation Letters (RA-L), we describe how we designed the controller, show its scalability in size, and demonstrate its robustness to noise. We studied the swarms’ performance and compared how it changes in two different environments: a forest and funnel-like environment.

The Crazyflie 2.1 was the perfect platform for our experiments. They are lightweight, modular, and tough. This quadcopter can survive big hits when things don’t go as planned… and, if you work on swarms, things can go wrong!

*The fleet of Crazyflies equipped with a single marker.*

With our algorithm, sixteen robots were able to fly through an artificial forest that we set up in our indoor motion capture arena. In our previous work, we installed four markers on each quadcopter and used the rigid body tracking from Motive (the Optitrack software). The large volume of our experimental room required the usage of big markers for long-distance detection, which added considerable weight to the drone. Hence, in our new work, we use a single marker per drone. Tracking is supported by the ‘crazyswarm’ package and communication with the entire swarm only requires two radio links. However, despite our model being decentralized, in our implementation robots relay the information to an external brain, which does the computations for them. In the future, all the necessary code will be embedded onboard, removing the dependency on external infrastructure.

*Our predictive swarm of Crazyflies flying among obstacles in our indoor experimental room.*

Video about the article

This work is a step forward towards the fully autonomous deployment of drone swarms in our cities. By enabling safe navigation in cluttered environments, drone fleets will be able to integrate with conventional air traffic, search for missing people, inspect dangerous areas, transport injured people to hospitals quicker, and deliver important packages right to our doors.

For further details, check out our article here!

Sniffy Bug: A Fully Autonomous Swarm of Gas-Seeking Nano Quadcopters in Cluttered Environments

2021-07-14 | Bart Duisterhof | 1 Comment

This week we have a guest blog post from Bart Duisterhof and Prof. Guido de Croon from the MAVlab, Faculty of Aerospace Engineering from the Delft University of Technology. Enjoy!

Tiny drones are ideal candidates for fully autonomous jobs that are too dangerous or time-consuming for humans. A commonly shared dream would be to have swarms of such drones help in search-and-rescue scenarios, for instance to localize gas leaks without endangering human lives. Drones like the CrazyFlie are ideal for such tasks, since they are small enough to navigate in narrow spaces, safe, agile, and very inexpensive. However, their small footprint also makes the design of an autonomous swarm extremely challenging, both from a software and hardware perspective.

From a software perspective, it is really challenging to come up with an algorithm capable of autonomous and collaborative navigation within such tight resource constraints. State-of-the-art solutions like SLAM require too much memory and processing power. A promising line of work is to use bug algorithms [1], which can be implemented as computationally efficient finite state machines (FSMs), and can navigate around obstacles without requiring a map.

A downside of using FSMs is that the resulting behavior can be very sensitive to their hyperparameters, and therefore may not generalize outside of the tested environments. This is especially true for the problem of gas source localization (GSL), as wind conditions and obstacle configurations drastically change the problem. In this blog post, we show how we tackled the complex problem of swarm GSL in cluttered environments by using a simple bug algorithm with evolved parameters, and then tested it onboard a fully autonomous swarm of CrazyFlies. We will focus on the problems that were encountered along the way, and the design choices we made as a result. At the end of this post, we will also add a short discussion about the future of nano drones.

Why gas source localization?

Overall we are interested in finding novel ways to enable autonomy on constrained devices, like CrazyFlies. Two years ago, we showed that a swarm of CrazyFlie drones was able to explore unknown, cluttered environments and come back to the base station. Since then, we have been working on an even more complex task: using such a swarm for Gas Source Localization (GSL).

There has been a lot of research focussing on autonomous GSL in robotics, since it is an important but very hard problem. The difficulty of the task comes from the complexity of how odor can spread in an environment. In an empty room without wind, a gas will slowly diffuse from the source. This can allow a robot to find it by moving up gradient, just like small bacteria like E. Coli do. However, if the environment becomes larger with many obstacles and walls, and wind comes into play, the spreading of gas is much less regular. Large parts of the environment may have no gas or wind at all, while at the same time there may be pockets of gas away from the source. Moreover, chemical sensors for robots are much less capable than the smelling organs of animals. Available chemical sensors for robots are typically less sensitive, noisier, and much slower.

Due to these difficulties, most work in the GSL field has focused on a single robot that has to find a gas source in environments that are relatively small and without obstacles. Relatively recently, there have been studies in which groups of robots solve this task in a collaborative fashion, for example with Particle Swarm Optimization (PSO). This allows robots to find the source and escape local maxima when present. Until now this concept has been shown in simulation [2] and on large outdoor drones equipped with LiDAR and GPS [3], but never before on tiny drones in complex, GPS-denied, indoor environments.

Required Infrastructure

In our project, we introduce a new bug algorithm, Sniffy Bug, which uses PSO for gas source localization. In order to tune the FSM of Sniffy Bug, we used an artificial evolution. For time reasons, evolution typically takes place in simulation. However, early in the project, we realized that this would be a challenge, as no end-to-end gas modeling pipeline existed yet. It is important to have an easy-to-use pipeline that does not require any aerodynamics domain knowledge, such that as many researchers as possible can generate environments to test their algorithms. It would also make it easier to compare contributions and to better understand in which conditions certain algorithms work or don’t work. The GADEN ROS package [4] is a great open-source tool for modeling gas distribution when you have an environment and flow field, but for our objective, we needed a fully automated tool that could generate a great variety of random environments on-demand with just a few parameters. Below is an overview of our simulation pipeline: AutoGDM.

AutoGDM, a fully automated gas dispersion modeling (GDM) simulation pipeline.

First, we use a procedural environment generator proposed in [5] to generate random walls and obstacles inside of the environment. An important next step is to generate a 3D flowfield by means of computational fluid dynamics (CFD). A hard requirement for us was that AutoGDM needed to be free to use, so we chose to use the open-source CFD tool OpenFOAM. It’s used for cutting-edge aerodynamics research, and also the tool suggested by the authors of GADEN. Usually, using OpenFOAM isn’t trivial, as a large number of parameters need to be selected that require field expertise, resulting in a complicated process. Next, we integrate GADEN into our pipeline, to go from environment definition (CAD files) and a flow field to a gas concentration field. Other parts that needed to be automated were the random selection of boundary conditions, which has a large impact on the actual flow field, and source placement, which has an equally large impact on the concentration field.

After we built this pipeline, we started looking for a robot simulator to couple it to. Since we weren’t planning on using a camera, our main requirement was for the simulator to be efficient (preferably in 2D) so that evolutions would take relatively little time. We decided to use Swarmulator [6], a lightweight C++ robot simulator designed for swarming and we plugged in our gas data.

Algorithm Design

Roughly speaking, we considered two categories of algorithms for controlling the drones: 1) a neural network, and 2) an FSM that included PSO, with evolved parameters. Since we used a tiny neural network for light seeking with a CrazyFlie in our previous work, we first evolved neural networks in simulation. One of the first experiments is shown below.

A single agent in simulation seeking a light source using a tiny neural network.

While it worked pretty well in simple environments with few obstacles, it seemed challenging to make this work in real life with complex obstacles and multiple agents that need to collaborate. Given the time constraints of the project, we have opted for evolving the FSM. This also facilitated crossing the reality gap, as the simulated evolution could build on basic behaviors that we developed and validated on the real platform, including obstacle avoidance with four tiny laser rangers, while communicating with and avoiding other drones. An additional advantage of PSO with respect to the reality gap is that it only needs gas concentration and no gradient of the gas concentration or wind direction (which many algorithms in literature use). On a real robot at this scale, estimating the gas concentration gradient or the direction of a light breeze is hard if not impossible.

Hardware

Our CrazyFlie needs to be able to avoid obstacles, execute velocity commands, sense gas, and estimate the other agent’s position in its own frame. For navigation, we added the flow deck and laser rangers, whereas for gas sensing we used a TGS8100 gas sensor that was used on a CrazyFlie before in previous work [7]. The sensor is lightweight and inexpensive, but accurately estimating gas concentrations can be difficult because of its size. It tends to drift and needs time to recover after a spike in concentration is observed. Another thing we noticed is that it is possible to break them, a crash can definitely destroy the sensor.

To estimate the relative position between agents, we use a Decawave Ultra-Wideband (UWB) module and communicate states, as proposed in [8]. We also use the UWB module to communicate gas information between agents and collaboratively seek the source. The complete configuration is visible below.

A 37.5 g nano quadcopter, capable of fully autonomous waypoint tracking, obstacle avoidance, relative localization, communication and gas sensing.

Evaluation in Simulation

After we optimized the parameters of our model using Swarmulator and AutoGDM, and of course trying many different versions of our algorithm, we ended up with the final Sniffy Bug algorithm. Below is a video that shows evolved Sniffy Bug evaluated in six different environments. The red dots are an agent’s personal target waypoint, whereas the yellow dot is the best-known position for the swarm.

Sniffy Bug evaluated in Swarmulator environments.

Simulation showed that Sniffy Bug is effective at locating the gas source in randomly generated environments. The drones successfully collaborate by means of PSO.

Real Flight Testing

After observing Sniffy Bug in simulation we were optimistic, but unsure about performance in real life. First, inspired by previous works, we disperse alcohol through the air by placing liquid alcohol into a can which is then dispersed using a computer fan.

Dispersion of liquid alcohol in flight tests.

We test Sniffy Bug in our flight arena of size 10 x 10 meters with large obstacles that are shaped like walls and orange poles. The image below shows four flight tests of Sniffy Bug in cluttered environments, flying fully autonomously, i.e., without the help from any external infrastructure.

Time-lapse images of real-world experiments in our flight arena. Sniffy was evaluated on four distinct environments, 10 x 10 meters in size, seeking a real isopropyl alcohol source. The trajectories of the nano quadcopters are clearly visible due to their blue lights.

In the total of 24 runs we executed, we compared Sniffy Bug with manually selected and evolved parameters. The figure below shows that the evolved parameters are more efficient in locating the source as compared to the manual parameters.

Maximum recorded gas reading by the swarm, for each time step for each run.

This does not only show that our system can successfully locate a gas source in challenging environments, but it also demonstrates the usefulness of the simulation pipeline. The parameters that were learned in simulation yield a high-performance model, validating the environment generation, randomization, and gas modeling parts of our pipeline.

Conclusion and Discussion

With this work, we believe we have made an important step towards swarms of gas-seeking drones. The proposed solution is shown to work in real flight tests with obstacles, and without any external systems to help in localization or communication. We believe this methodology can be extended to larger environments or even to 3 dimensions, since PSO is a robust, multi-dimensional heuristic search method. Moreover, we hope that AutoGDM will help the community to better compare gas seeking algorithms, and to more easily learn parameters or models in simulation, and deploy them in the real world.

To improve Sniffy Bug’s performance, adding more laser rangers will definitely help. When working with only four laser rangers you realize how little information it actually provides. If one of the rangers senses a low value it is unclear if a slim pole or a massive wall is detected, adding inefficiency to the algorithm. Adding more laser rangers or using other sensor modalities like vision will help to avoid also more complex obstacles than walls and poles in a reliable manner.

Another interesting discussion can be held on the hardware required for real deployment. When working with 40 grams of maximum take-off weight, the sensors and actuators that can be selected are limited. For example, the low-power and lightweight flow deck works great but fails in low-light scenarios or with smoke. Future work exploring novel sensors for highly constrained nano robots could really help increase the Technological Readiness Level (TRL) of these systems.

Finally, this has been a really fun project to work on for us and we can’t wait to hear your thoughts on Sniffy Bug!

References

[1] K. N. McGuire, C. De Wagter, K. Tuyls, H. J. Kappen, and G. C. H. E.de Croon, “Minimal navigation solution for a swarm of tiny flying robotsto explore an unknown environment,”Science Robotics, vol. 4, no. 35,2019.

[2] W. Jatmiko, K. Sekiyama and T. Fukuda, “A pso-based mobile robot for odor source localization in dynamic advection-diffusion with obstacles environment: theory, simulation and measurement,” in IEEE Computational Intelligence Magazine, vol. 2, no. 2, pp. 37-51, May 2007, doi: 10.1109/MCI.2007.353419.

[3] Steiner, JA, Bourne, JR, He, X, Cropek, DM, & Leang, KK. “Chemical-Source Localization Using a Swarm of Decentralized Unmanned Aerial Vehicles for Urban/Suburban Environments.” Proceedings of the ASME 2019 Dynamic Systems and Control Conference. Volume 3, Park City, Utah, USA. October 8–11, 2019. V003T21A006. ASME. https://doi.org/10.1115/DSCC2019-9099

[4] . Monroy, V. Hernandez-Bennetts, H. Fan, A. Lilienthal, andJ. Gonzalez-Jimenez, “Gaden: A 3d gas dispersion simulator for mobilerobot olfaction in realistic environments,”MDPI Sensors, vol. 17, no.7: 1479, pp. 1–16, 2017.

[5] K. McGuire, G. de Croon, and K. Tuyls, “A comparative study of bug algorithms for robot navigation,”Robotics and Autonomous Systems, vol.121, p. 103261, 2019.

[6] https://github.com/coppolam/swarmulator

[7] J. Burgues, V. Hern ́andez, A. J. Lilienthal, and S. Marco, “Smellingnano aerial vehicle for gas source localization and mapping,”Sensors(Switzerland), vol. 19, no. 3, 2019.[8] S. Li, M. Coppola, C. D. Wagter, and G. C. H. E. de Croon, “An autonomous swarm of micro flying robots with range-based relative localization,” Arxiv, 2020.

[8] S. Li, M. Coppola, C. D. Wagter, and G. C. H. E. de Croon, “An autonomous swarm of micro flying robots with range-based relative localization,” Arxiv, 2020.

Links

ArXiv: https://arxiv.org/abs/2107.05490

Code: https://github.com/tudelft/sniffy-bug

Video:

Please reach out if you have any questions or ideas, you can reach us at: b.p.duisterhof@gmail.com or g.c.h.e.decroon@tudelft.nl

Ultra-Wideband Swarm Ranging

2021-06-21 | Feng Shan | Leave a comment

This week we have a guest blog post from Dr Feng Shan at School of Computer Science and Engineering
Southeast University, China. Enjoy!

It is possible to utilize tens and thousands of Crazyflies to form a swarm to complete complicated cooperative tasks, such as searching and mapping. These Crazyflies are in short distance to each other and may move dynamically, so we study the dynamic and dense swarms. The ultra-wideband (UWB) technology is proposed to serve as the fundamental technique for both networking and localization, because UWB is so time sensitive that an accurate distance can be calculated using the transmission and receive timestamps of data packets. We have therefore designed a UWB Swarm Ranging Protocol with key features: simple yet efficient, adaptive and robust, scalable and supportive. It is implemented on Crazyflie 2.1 with onboard UWB wireless transceiver chips DW1000.

Fig.1. Nine Crazyflies are in a compact space ranging the distance with each other.

The Basic Idea

The basic idea of the swarm ranging protocol was inspired by Double Sided-Two Way Ranging (DS-TWR), as shown below.

Fig.2. The exsiting Double Sided-Two Way Ranging (DS-TWR) protocol.

There are four types of message in DS-TWR, i.e., poll, response, final and report message, exchanging between the two sides, A and B. We define their transmission and receive timestamps are T_p, R_p, T_r, R_r, T_f, and R_f, respectively. We define the reply and round time duration for the two sides as follows.

Let t_p be the time of flight (ToF), namely radio signal propagation time. ToF can be calculated as Eq. (2).

Then, the distance can be estimated by the ToF.

In our proposed Swarm Ranging Protocol, instead of four types of messages, we use only one type of message, which we call the ranging message.

Fig.3. The basic idea of the proposed Swarm Ranging Protocol.

Three sides A, B and C take turns to transmit six messages, namely A₁, B₁, C₁, A₂, B₂, and C₂. Each message can be received by the other two sides because of the broadcast nature of wireless communication. Then every message generates three timestamps, i.e., one transmission and two receive timestamps, as shown in Fig.3(a). We can see that each pair has two rounds of message exchange as shown in Fig.3(b). Hence, there are sufficient timestamps to calculate the ToF for each pair, that means all three pairs can be ranged with each side transmitting only two messages. This observation inspires us to design our ranging protocol.

Protocol Design

The formal definition of the i-th ranging message that broadcasted by Crazyflie X is as follows.

X_i is the message identification, e.g., sender and sequence number; T_{x_i-1} is the transmission timestamp of X_i-1, i.e., the last sent message; RxM is the set of receive timestamps and their message identification, e.g., RxM = {(A₂, R_A₂), (B₂, R_B₂)}; v is the velocity of X when it generates message X_i.

As mentioned above, six timestamps (T_p, R_p, T_r, R_r, T_f, R_f,) are needed to calculate the ToF. Therefore, for each neighbor, an additional data structure is designed to store these timestamps which we named it the ranging table, as shown in Fig.4. Each device maintains one ranging table for each known neighbor to store the timestamps required for ranging.

Fig.4. The *ranging table*, one for each neighbor.

Let’s focus on a simple scenario where there are a number of Crazyflies, A, B, C, etc, in a short distance. Each one of them transmit a message that can be heard by all others, and they broadcast ranging messages at the same pace. As a result, between any two consecutive message transmission, a Crazyflie can hear messages from all others. The message exchange between A and Y is as follows.

Fig.5. Message exchange between A and Y.

The following steps show how the ranging messages are generated and the ranging tables are updated to correctly compute the distance between A and Y.

Fig.6. How the ranging message and ranging table works to compute distance.

The message exchange between A and Y could be also A and B, A and C, etc, because they are equal, that’s means A could perform the ranging process above with all of it’s neighbors at the same time.

To handle dense and dynamic swarm, we improved the data structure of ranging table.

Fig.7. The improved ranging table for dense and dynamic swarm.

There are three new notations P, t_n, t_s, denoting the newest ranging period, the next (expected) delivery time and the expiration time, respectively.

For any Crazyflie, we allow it to have different ranging period for different neighbors, instead of setting a constant period for all neighbors. So, not all neighbors’ timestamps are required to be carried in every ranging message, e.g., the receive timestamp to a far apart and motionless neighbor is required less often. t_n is used to measure the priority of neighbors. Also, when a neighbor is not heard for a certain duration, we set it as expired and will remove its ranging table.

If you are interested in our protocol, you can find much more details in our paper, that has just been published on IEEE International Conference on Computer Communications (INFOCOM) 2021. Please refer the links at the bottom of this article for our paper.

Implementation

We have implemented our swarm ranging protocol for Crazyflie and it is now open-source. Note that we have also implemented the Optimized Link State Routing (OLSR) protocol, and the ranging messages are one of the OLSR messages type. So the “Timestamp Message” in the source file is the ranging message introduced in this article.

The procedure that handles the ranging messages is triggered by the hardware interruption of DW1000. During such procedure, timestamps in ranging tables are updated accordingly. Once a neighbor’s ranging table is complete, the distance is calculated and then the ranging table is rearranged.

All our codes are stored in the folder crazyflie-firmware/src/deck/drivers/src/swarming.

The following figure is a ranging performance comparison between our ranging protocol and token-ring based TWR protocol. It’s clear that our protocol handles the large number of drones smoothly.

We also conduct a collision avoidance experiment to test the real time ranging accuracy. In this experiment, 8 Crazyflie drones hover at the height 70cm in a compact area less than 3m by 3m. While a ninth Crazyflie drone is manually controlled to fly into this area. Thanks to the swarm ranging protocol, a drone detects the coming drone by ranging distance, and lower its height to avoid collision once the distance is small than a threshold, 30cm.

Build & Run

Clone our repository

git clone --recursive https://github.com/SEU-NetSI/crazyflie-firmware.git

Go to the swarming folder.

cd crazyflie-firmware/src/deck/drivers/src/swarming

Then build the firmware.

make clean
make

Flash the cf2.bin.

cfloader flash path/to/cf2.bin stm32-fw

Open the client, connect to one of the drones and add log variables. (We use radio channel as the address of the drone) Our swarm ranging protocol allows the drones to ranging with multiple targets at the same time. The following shows that our swarm ranging protocol works very efficiently.

Summary

We designed a ranging protocol specially for dense and dynamic swarms. Only a single type of message is used in our protocol which is broadcasted periodically. Timestamps are carried by this message so that the distance can be calculated. Also, we implemented our proposed ranging protocol on Crazyflie drones. Experiment shows that our protocol works very efficiently.

Robust Kalman Filter for Ultra-wideband Localization

2021-06-07 | Wenda Zhao | 6 Comments

This week we have a guest blog post from Wenda Zhao, Ph.D. candidate at the Dynamic System Lab (with Prof. Angela Schoellig), University of Toronto Institute for Aerospace Studies (UTIAS). Enjoy!

Accurate indoor localization is a crucial enabling capability for indoor robotics. Small and computationally-constrained indoor mobile robots have led researchers to pursue localization methods leveraging low-power and lightweight sensors. Ultra-wideband (UWB) technology, in particular, has been shown to provide sub-meter accurate, high-frequency, obstacle-penetrating ranging measurements that are robust to radio-frequency interference, using tiny integrated circuits. UWB chips have already been included in the latest generations of smartphones (iPhone 12, Samsung Galaxy S21, etc.) with the expectation that they will support faster data transfer and accurate indoor positioning, even in cluttered environments.

A Crazyflie with an IMU and UWB tag flies through a cardboard tunnel. A vision-based motion capture system would not be able to achieve this due to the occlusion.

In our lab, we show that a Crazyflie nano-quadcopter can stably fly through a cardboard tunnel with only an IMU and UWB tag, from Bitcraze’s Loco Positioning System (LPS), for state estimation. However, it is challenging to achieve a reliable localization performance as we show above. Many factors can reduce the accuracy and reliability of UWB localization, for either two-way ranging (TWR) or time-difference-of-arrival (TDOA) measurements. Non-line-of-sight (NLOS) and multi-path radio propagation can lead to erroneous, spurious measurements (so-called outliers). Even line-of-sight (LOS) UWB measurements exhibit error patterns (i.e., bias), which are typically caused by the UWB antenna’s radiation characteristics. In our recent work, we present an M-estimation-based robust Kalman filter to reduce the influence of outliers and achieve robust UWB localization. We contributed an implementation of the robust Kalman filter for both TWR and TDOA (PR #707 and #745) to Bitcraze’s crazyflie-firmware open-source project.

Methodology

The conventional Kalman filter, a primary sensor fusion mechanism, is sensitive to measurement outliers due to its minimum mean-square-error (MMSE) criterion. To achieve robust estimation, it is critical to properly handle measurement outliers. We implement a robust M-estimation method to address this problem. Instead of using a least-squares, maximum-likelihood cost function, we use a robust cost function to downweigh the influence of outlier measurements [1]. Compared to Random Sample Consensus (RANSAC) approaches, our method can handle sparse UWB measurements, which are often a challenge for RANSAC.

From the Bayesian maximum-a-posteriori perspective, the Kalman filter state estimation framework can be derived by solving the following minimization problem:

Therein, x_kand y_k are the system state and measurements at timestep k. P_k and R_k denote the prior covariance and measurement covariance, respectively. The prior and posteriori estimates are denoted as x_k check and x_k hat and the measurement function without noise is indicated as g(x_k,0). Through Cholesky factorization of P_k and R_k, the original optimization problem is equivalent to

where e_x,k,i and e_y,k,jare the elements of e_x,k and e_y,k. To reduce the influence of outliers, we incorporate a robust cost function into the Kalman filter framework as follows:

where rho() could be any robust function (G-M, SC-DCS, Huber, Cauchy, etc.[2]).

By introducing a weight function for the process and measurement uncertainties—with e as input—we can translate the optimization problem into an Iteratively Reweighted Least Squares (IRLS) problem. Then, the optimal posteriori estimate can be computed through iteratively solving the least-squares problem using the robust weights computed from the previous solution. In our implementation, we use the G-M robust cost function and the maximum iteration is set to be two for computational reasons. For further details about the robust Kalman filter, readers are referred to our ICRA/RA-L paper and the onboard firmware (mm_tdoa_robust.c and mm_distance_robust.c).

Performance

We demonstrate the effectiveness of the robust Kalman filter on-board a Crazyflie 2.1. The Crazyflie is equipped with an IMU and an LPS UWB tag (in TDOA2 mode). With the conventional onboard extended Kalman filter, the drone is affected by measurement outliers and jumps around significantly while trying to hover. In contrast, with the robust Kalman filter, the drone shows a more reliable localization performance.

The robust Kalman filter implementations for UWB TWR and TDOA localization have been included in the crazyflie-firmware master branch as of March 2021 (2021.03 release). This functionality can be turned on by setting a parameter (robustTwr or robustTdoa) in estimator_kalman.c. We encourage LPS users to check out this new functionality.

As we mentioned above, off-the-shelf, low-cost UWB modules also exhibit distinctive and reproducible bias patterns. In our recent work, we devised experiments using the LPS UWB modules and showed that the systematic biases have a strong relationship with the pose of the tag and the anchors as they result from the UWB radio doughnut-shaped antenna pattern. A pre-trained neural network is used to approximate the systematic biases. By combining bias compensation with the robust Kalman filter, we obtain a lightweight, learning-enhanced localization framework that achieves accurate and reliable UWB indoor positioning. We show that our approach runs in real-time and in closed-loop on-board a Crazyflie nano-quadcopter yielding enhanced localization performance for autonomous trajectory tracking. The dataset for the systematic biases in UWB TDOA measurements is available on our Open-source Code & Dataset webpage. We are also currently working on a more comprehensive dataset with IMU, UWB, and optical flow measurements and again based on the Crazyflie platform. So stay tuned!

Reference

[1] L. Chang, K. Li, and B. Hu, “Huber’s M-estimation-based process uncertainty robust filter for integrated INS/GPS,” IEEE Sensors Journal, 2015, vol. 15, no. 6, pp. 3367–3374.

[2] K. MacTavish and T. D. Barfoot, “At all costs: A comparison of robust cost functions for camera correspondence outliers,” in IEEE Conference on Computer and Robot Vision (CRV). 2015, pp. 62–69.

Links

The authors are with the Dynamic Systems Lab, Institute for Aerospace Studies, University of Toronto, Canada, and affiliated with the Vector Institute for Artificial intelligence in Toronto.

Feel free to contact us if you have any questions or suggestions: wenda.zhao@robotics.utias.utoronto.ca.

Please cite this as:

<code>@ARTICLE{Zhao2021Learningbased,
author={W. {Zhao} and J. {Panerati} and A. P. {Schoellig}},
title={Learning-based Bias Correction for Time Difference of Arrival Ultra-wideband Localization of Resource-constrained Mobile Robots},
journal={IEEE Robotics and Automation Letters},
volume={6},
number={2},
pages={3639-3646},
year={2021},
publisher={IEEE}
doi={10.1109/LRA.2021.3064199}}
</code>

Hardware emulation thesis

2021-02-15 | Max | Leave a comment

Hi!

Since the middle of January, Bitcraze has had two additional guests: us! We, Josefine & Max, are currently doing our master thesis at Bitcraze during our final semester at LTH.

Unfortunately, the pandemic means remote working which could have caused some difficulties with hardware and equipment accessibility. Fortunately, for us, our goal with the thesis is to emulate the Crazyflie 2.1 hardware in the open source software Renode. We can therefore do the work at home.

Since this is the first time either of us have tried to emulate hardware it is exciting to see if it will even be possible, especially as we do not know what limitations Renode might have. Thus far, four weeks in, there have been some hiccups and crashes but also progress and success. One example was when we got the USART up and running and it became possible to start printing debug messages and another was when the LED lights were connected and it was possible to see when they were turned on and off.

Example of output from partially implemented hardware.

If all goes well, the emulated hardware could be used as a part of the CI pipeline to automatically hunt for bugs. Academically, it would be used to further study testing methods of control firmware.

Getting a glimpse into the workings at Bitcraze and the lovely people working here has been most interesting and we are looking forward to the time ahead. Until next time, hopefully with a working emulation, Josefine & Max.