An abridged version of this research brief was published on TechCrunch, the article can be found here.

If you would prefer to listen to this brief and see its accompanying slide deck, please watch the video below:

The problems with sensor calibration today

Sensors are critical components of the modern vehicle. They are the eyes of a car, enabling everything from existing ADAS (Advanced Driver-Assistance Systems) features such as automated breaking and lane keeping to potential removal of the driver altogether. When vehicles become highly automated, the accuracy of these sensors becomes increasingly important. The consequences of these “eyes” not pointing in the right direction or not seeing clearly could be catastrophic; your car could needlessly break in the middle of the highway or suddenly swerve into another lane. Sufficiently high and safe accuracy is essential, and calibration is one of the main factors ensuring that a vehicle’s sensors are operating at the highest fidelity.

Sensors find themselves out of calibration due to daily normal use, changes in operating conditions such as temperature or standard vibrations, or something more severe like accidents or vehicle part replacements. Unfortunately, very little emphasis has been placed on addressing the issue. This comes as no surprise; the automotive product cycle is incredibly long, and automated vehicles simply haven’t been tested long enough yet to thoroughly expose this issue.

The current most prominent solution, especially for suites of sensors, is to pick functionally or economically inferior sensor configurations and/or to simply hope that the sensors never become miscalibrated from their initial factory settings in the first place. Yet while this is obviously unsafe, there exist no common metrics to measure what it means for a sensor to be miscalibrated and no common standards that companies can hold their sensor calibrations up against. Every player in this space has their own unique sensor suites and an accompanying set of unique calibration practices, further complicating the matter.

More pertinent in the short term is that current aftermarket, maintenance, and return-to-service options are woefully underprepared to address the issue. This is clearly apparent when considering the calibration of ADAS systems already in the market today. A typical ADAS calibration procedure can last anywhere from 15 minutes to 2 hours, and requires expensive equipment (i.e. service information, calibration scanning and alignment tools, large and clear paved areas, and wheel alignment racks). The vehicle itself also needs to be prepared to meticulous standards; the fuel tank must be full, the tires must be properly inflated, the vehicle must be perfectly flat on a balanced floor, etc. Most garages and mechanics are under-equipped and insufficiently trained to conduct what is an incredibly tedious, manual, and technically complex procedure. This ultimately causes improper calibration that endangers the vehicle’s passengers and those around them.

Given the emphasis placed on safety when considering the application of automation to vehicles, we at Trucks firmly believe that sensor calibration is an aspect of a vehicle’s autonomy stack that needs to be improved.

The different types of sensor calibration

Sensor calibration is generally broken down into three types.

1. ‍Intrinsic: refers to the internal parameters and configurations of a sensor.‍
2. Extrinsic: refers to the parameters relating multiple sensors together.
3. ‍Temporal: refers to the syncing of sensors that collect data at differing frequencies.

Most standard LiDARs and cameras can automatically perform intrinsic calibration of the noise caused by standard day-to-day operations (vibration, temperature changes, etc.), thus making intrinsic calibration somewhat of a solved problem. Similarly, temporal miscalibrations don’t cause too many complications yet, as simply syncing different sensor streams by matching the frames closest to each other has proven to be good enough for today’s operating frequencies. Extrinsic calibration, however, poses the most problems to fleets given the ever increasing reliance on a system of multiple sensors to overcome the shortcomings of individual sensors. For this reason, this brief focuses primarily on extrinsic calibration.

The status quo of sensor calibration

As per Jacob Lambert, when looking at sensor calibration for cameras, LiDARs, and other perception sensors across a broad spectrum beyond just automated vehicles, there are three classes of calibrations that have been attempted.

1. Target based: Perhaps most familiar, this method uses easily recognizable targets such as checkerboards. Sensors are able to track these targets across different frames and compute re-projection errors to compute relevant calibration parameters. A major pain point of this approach is that finding a suitable target that functions well across different modes of sensors is difficult. Target based extrinsic calibration also requires all sensors to detect the target at the same time, therefore limiting this approach to systems of sensors with overlapping fields of view. Ultimately, this approach is the most robust (since it is easy to debug) and has thus become the most widespread and standard form of calibration.
2. Ego-motion based: Given that 3D localization is largely a solved problem in robotics, one can use the localization of each sensor to compute their trajectories. The sensors’ computed trajectories, plus the assumption that they are rigidly mounted to the vehicle, allows one to compute the extrinsic calibration of the system. However, this method is severely limited not only by the quality of localization but also the fact that it requires significant movement along all of a sensor’s degrees-of-freedom to achieve proper localization to begin with. Due to these factors, ego-motion based calibration is impractical for automated vehicles and thus limited to drones and other smaller robots.
3. Appearance based: Appearance based methods rely on parameters of scenes that can be extracted from all sensor types, parameters such as intensity distribution. Once you are able to observe the intensity distribution or other appearance parameters of a scene across different sensors over multiple frames, you can compute the transform between the sensors. Appearance-based calibration doesn’t require any specific features in the environment, making it the ideal solution for any place, any time calibration. However, research thus far has been inconclusive, and results have been highly contingent on the sensor initialization and data used. While this is difficult to practically implement today, we believe this approach has the most potential to become the ideal solution for calibration, especially as the quality of LIDARs improves and their ability to capture intensity and other appearance parameters matches that of cameras.

Improving Sensor Calibration

Much of transportation research thus far has been a massively collaborative undertaking between universities and private players, so we expect to see many of these solutions in practice soon.

Innovations in Academia

The following three academic institutions that have spent considerable time tackling sensor calibration and each have a unique approach:

The STARS Lab at the University of Toronto

As per Professor Jonathan Kelly, head of the STARS Lab, calibration is a task too sensitive for a human to do reliably and correctly. For this reason, his lab proposes online self-calibration, where the vehicle is able to calibrate its sensors autonomously and in real time.

The enabling insight to their approach is to find sensors that provide ground truth positioning information. Likely candidates for ground truth data are IMU or GPS sensors, given that current iterations of these sensors are already extremely accurate and rarely need re-calibration. However, one can also use statistical training on the sensors’ noise, error, and failure models to gait their reliability and select one as a ground truth source.

Once the vehicle has a source of ground truth, it can move in specially designed maneuvers such as figure eights and detect common objects in the environment (i.e. stop signs, traffic lights, etc.) via its perception sensors. It can then overlay the transform of detections over the ground truth to compute calibration parameters.

STARS lab has received numerous enquiries from industry to develop a calibration as a service offering. Such an offering would be based on either a chip or in the cloud, depending on the intensity of computations required for the use-case at hand.

The Stanford Driving Team

Stanford’s automotive labs, including the Driving Team at SAIL, also propose on-line self-calibration as the best solution to this issue. In addition to comparing to ground truth data, their approach relies heavily on probabilistic error models of sensors.

Error models are extracted by pre-training them with examples of correct and incorrect calibration settings. These error models are then used to detect miscalibrated sensors. Once miscalibration is detected, the algorithm then uses those error models, overlayed on the observed data and ground truth data, to track the drift of calibration parameters as a vehicle performs specified maneuvers. These tracked drifts allow the vehicle to adjust its calibration accordingly.

One interesting note here is that Zoox CEO Jesse Levinson conducted his PhD at the Stanford Driving Team, and a significant component of his thesis focused on sensor calibration. Given Levinson’s background in this space, as well as the massive operational expenses calibration on poses to AV fleets, we suspect that Zoox is be a front-runner in the race to optimize sensor calibration for fleet operations.

The Takeda Lab at Nagoya University

The Takeda Lab has a quite different view on the matter. As per Lambert, any place, any time self-calibration is impractical in the near future and thus only results in lower quality calibration overall. Given the established robustness of target based calibration, the Takeda Lab believes that the solution should be an open source toolbox or measuring stick that can be used to compare the calibration of the different vehicles.

A target based calibration measuring stick would first need to visualize both the targets detected and the relevant reprojection errors between each frame. It should then contain a number of statistics on these errors, ranging from histograms and means to number of data points and the distance between them. Given that any target based calibration is capable of producing these results, having such results publicly available allows regulators and the transportation community as a whole to develop standards to hold all sensor calibrations against.

In addition, such a toolbox should also include calibration checkers on the vehicles to monitor the health of calibration parameters. These checkers would likely be simple detection processes that detect common objects such as stop signs or traffic lights.

Innovations in Industry

Industry players have very different approaches to improving sensor calibration based on where they lie in the autonomy supply chain. Many of these companies have largely maintained confidentiality regarding their calibration work, but the information below presents an insight into the overarching ideas on their minds.

OEMs

Some OEMs have filed patents on potential solutions to improve extrinsic sensor calibration, but the content of these patents suggest that not much emphasis has been placed on extrinsic calibration. Consider published patent application US20160161602A1 and issued patent US10197400B2. These either propose target based calibration that has already been well established in academia or suggest solutions and accessories that are simply infeasible at scale.

In our research and discussions with OEMs, we found a range of sensor calibration readiness. Some have more of a full lifecycle approach to vehicle development and were interested in sharing their innovations in calibration, while others did not have much plans for calibration outside of what their afterservice organization could handle. If automated vehicles become fleet-only for the near term, it is likely that OEMs become more involved in the calibration function for their customers.

AV Fleets

As mentioned before, every fleet somewhat reinvents the wheel when it comes to sensor calibration given how unique sensor suites are to each fleet. Sensor calibration is in some cases required almost daily, and the current standard is a tedious procedure that takes up to 60 minutes for an engineer or field technician to perform. Some fleets, such as May Mobility, see the value in a 3rd party calibration as a service provider given that calibration is ultimately a maintenance task that takes away valuable time from R&D. However, other fleets are more skeptical given the amount of customization that goes into fleets and their sensor suites today. We see that the opportunity for a 3rd party as such lies somewhere in the middle, and that it will only grow as sensor suites begin to consolidate and standardize to a common set of perception sensors.

Sensor Hardware Suppliers

Many sensor suppliers have developed their own in-house intrinsic calibration systems and methods for how they expose this process to their customers. AEye, for example, provides an SDK to its customers so that technicians can trigger the self-calibration process and view the diagnostic results (details of which have been omitted due to confidentiality). These results can then be used for either maintenance or integration into system-wide extrinsic calibration.

Given the difficulty of implementing accurate self-calibration, it is crucial for sensor suppliers to make conscious design decisions that mitigate factors that cause miscalibration. In the same way that current MEMS, IMU, or airbag sensors don’t need regular calibration or maintenance, OEMs ultimately want sensors that don’t require their vehicles to be sent to dealerships or shops at any regularity. One design decision that AEye has made in this vein is making solid state instead of rotating iDARs. Other such decisions could be making better housings for sensors that protect them from vibrations or temperature changes.

Software Suppliers

Calibration is not limited to just sensors, the AI perception algorithms built upon them need to be calibrated as well. Perception software is often directly trained on sensor data, making them highly dependent on the specific sensors and parameters that they were trained on. When any of these sensors are miscalibrated or replaced, the overlaying AI algorithms need to be re-calibrated and retrained as well. While this may be an acceptable task in R&D labs, it is simply not scalable in the real world to expect aftermarket maintenance to recalibrate and retrain AI algorithms.

UK Software Start-up LGN’s solution to this problem is to feed and train AI with latent space instead of the direct sensor feed. A sensor’s latent space is the abstraction of its input data derived using it’s intrinsic parameters. There are multiple benefits of using the latent space to train AI:

1. The latent space captures a level of redundancy within input data. This makes the AI more robust to most data perturbations caused by miscalibration, occlusion, etc. of sensors and allows you to normalize your AI and data across a more diverse array of sensors and operating conditions.
2. In a multi-sensor system, you can predict what the sensor input data should look like because you can accurately estimate what the latent space should look like.

The ability to predict what your sensors should be seeing is extremely powerful, allowing you to pinpoint anomalies or edge-cases within your data. Once you confirm that an anomaly was caused by miscalibration, you can decode the latent space that should have been observed to reverse calibrate the extrinsic parameters of your system as well as the intrinsic parameters of the faulty sensors. Ultimately, the latent space can act as the connective tissue built upon existing intrinsic calibration processes that links together intrinsic and extrinsic calibration parameters.

Aftermarket Solutions

Companies like Bosch are also leaning into the trend of standardizing target based calibration, developing products like the following to help reduce uncertainty and consistency issues when mechanics and technicians prepare vehicles for calibration.

Ultimately however, equipment as such is not only incredibly expensive, but also requires space and maintenance that most garages cannot afford. This is where companies like Bridgestone and other after-market product suppliers/retailers see an opportunity.

Consider first an analogy proposed by Darren Liccardo: maintenance in the aviation space. Maintenance shops for aviation technology are highly regulated with thorough records of all work done on the aircrafts. Furthermore, they are highly specialized; they are equipped with the latest technologies and those who work there are specially trained for the task at hand. As cars and other vehicles become more and more technically complex, we can no longer rely upon the under-regulated networks of garages and shops that exist today.

As mentioned by Reilly Brennan in his post on the future of auto maintenance, the advent of electrification and autonomy makes obsolete many traditional vehicle maintenance tasks and services currently offered by garages. Companies like Bridgestone that have massive amounts of real estate and capital available, as well as traditional vehicle maintenance centers, should see this as an opportunity to evolve with vehicles and fill this exact need by developing specialized shops for AV maintenance that are equipped with the right technology and properly trained technicians.

Insurance Providers

One of the potential vectors for calibration is that insurers see the value before regulators do. In such a circumstance, some policies could become dependent on frequent sensor calibration. For vehicles operating in a highly automated manner (even at lower levels of automation, such as Level 1 use of automated cruise control), there should be a level of service requirement for sensors to stay in an acceptable performance range. Calibration is the key to maintaining such quality and it is likely we will see insurers demand that sensors are calibrated frequently.

An analogy for this space is the legal tussle over a speeding ticket. When pressed in a court of law by a defendant, a police officer often has to prove his or her radar gun had been calibrated soon before the reading was taken. The same is likely to be asked of a vehicle by law enforcement officers and insurers.

Ensuring that sensors operate at the highest fidelity

As our vehicles become highly automated, sensor calibration is a critical element that needs to be improved and standardized. The rate at which ADAS and automation is hitting the market is much faster than the rate at which sensor calibration procedures are being improved, and this has the potential to cause massive problems in operations, after-market, and return-to-service scenarios. It is crucial that OEMs, suppliers, fleet operators, academia, and creative third parties come together to develop the technology and infrastructure to insure that the information our vehicles operate on are of the highest fidelity.

If you have any thoughts on sensor calibration, or are working on ways to improve it, let’s chat! Feel free to contact me at puneeth [at] trucks [dot] vc.

I’d like to thank the following people for taking the time to chat with me about their thoughts on sensor calibration:

• ‍Darren Liccardo — Catapult Ventures
• ‍Dr. Jonathan Kelly — University of Toronto
• ‍Dr. Steve Vozar — May Mobility
• ‍Dr. Jan Becker — Apex.AI
• ‍Jacob Lambert — Nagoya University
• ‍David Oroshnik and Pritesh Solanki — AEye
• ‍Daniel Warner and Dr. Vladimir Ceperic — LGN
• ‍Maria Dunn — Bridgestone

Additionally, I’d like to thank the authors of the following papers and articles for providing an immensely valuable perspective on this ma‍Automatic Online Calibration of Cameras and Lasers by Jesse Levinson and Sebastian TAutomatic Laser Calibration, Mapping, and Localization for Autonomous Vehicles by Jesse Levinson

• ‍ADAS Sensor Calibration Increases Repair Costs by the American Automobile Association