7 minute read
The LM GT3 model (#37425) improves phone detection and is the first openpilot DM model trained using labels generated by a locally hosted large Vision-Language Model (VLM).
In this model, the driver’s phone-usage ground truth is generated by asking a VLM a YES/NO question and using the output probability of the YES token. The language model is set to predict only one token. The previous approach relied on a custom classifier network trained on a set of hand-labeled driver images.
To evaluate this new pipeline, we hand-labeled a 1k validation set of driver images with drivers either using or not using phones, called the phone_using set. We also curated a smaller set named phone_mounted, consisting of 80 driver images with a phone mounted on the dash but not in use. The phone_mounted set represents difficult negative samples.
The VLM notably reduces False Positives and improves detection of active handheld phone use as True Positives, while keeping all mounted phones as True Negatives.
ROC curves for phone_using on the 1k validation set. phone_mounted not depicted since all are negative samples, whose false positives reduce to 0 when phone_using true positive rate is ~50%.
With the model deployed in the field, we were also able to see its positive effects on safe driving habits when using openpilot. We compared phone usage time, as detected by the improved model, across the same group of devices before and after this model change. We saw a significant drop when openpilot was enabled, and even observed a small drop when it was not enabled, suggesting that drivers did reduce their phone usage overall.
Phone usage behavior progression on the same sample group of 71 devices running master/nightly/staging: ‘before’ segments sampled from 60-day window before new DM model; ‘after’ segments from 20-day window after. All data re-evaluated offline using the same model.
The two road cameras use a different image processing path on the ISP than the driver camera, and have had better image quality due to some missing pieces on the driver camera processing side. We have now enabled equivalent tone mapping blocks (#37873), as well as moved from on-sensor binning to post-demosaic downscaling (#37876). This means the driver camera will get the same high level of dynamic range and pixel details as the road-facing cameras.
Driver camera pictures old vs new, day and night.
Alerts from looking around are reduced during active maneuvers (#37751). When steering through a turn, the head pose yaw tolerance is widened in the direction of the steering angle, so glancing where the car is going no longer triggers an alert.
Adaptive head pose thresholds during turns. This drops ~7% of all alerts. Angles shown are for illustrative purposes only, as these thresholds are also based on the scene and speed.
This PR also extends the standstill exemption for DM alerting, so that an audible alert no longer unnecessarily kicks in the moment the car starts moving.
The driverMonitoringState message has been restructured (#37799) to be both more readable and have a cleaner abstraction boundary. Downstream processes, such as selfdrived, can access top-level states and handle alerts without needing to know about DM policy details. Debugging with the new logs should also become easier.
In the comma four onroad UI, we added a color change to the DM icon to subtly indicate distraction (#37826) before any alert is triggered. It also serves as feedback when the driver gets an audible DM alert and tries to look back at the device.
DM icon turns orange as soon as alert countdown starts, and returns to green as attention is restored.
The DM icon is now calibrated to the road instead of showing raw pose from the model (#37149), so the green arc will be at the top of the icon when the driver is looking straight ahead.
Comparison of DM icon behavior when driver is looking straight ahead.
Direct sunlight on a parked comma four can heat the device above the fixed 75°C onroad threshold, temporarily blocking openpilot from going onroad even though the device would cool down once the car started moving.
This release raises the threshold to 85°C and updates the comma four onroad thermal bands (#37891). Fleet analysis projected that raising the threshold from 75°C to 85°C would reduce affected devices by ~90%.
Number of comma four devices thermally blocked over the last 30 days. Raising the threshold from 75°C to 85°C reduced affected devices from 125 to 11.
Thermal blocks were not isolated to a specific region, but occurred wherever solar load could push device temperature above the 75°C threshold. Incidents were concentrated in the US, consistent with fleet distribution, with additional clusters across Europe, the Middle East, and East Asia.
Worldwide distribution of comma four devices thermally blocked before the threshold change.
The figure below shows temperature traces from three comma four devices with high average offroad temperatures. The raised 85°C threshold allows these devices to go onroad during most peak-heating conditions.
Temperature traces from three comma four devices. Grey offroad points capture the offroad heating and cooling cycle, while green points show onroad driving. The old and new thresholds are shown as dashed lines.
The original 75°C threshold was set conservatively to keep the SoC well below thermal throttling limits. CPU or GPU throttling can cause openpilot processes to lag and eventually disengage, so the onroad thermal policy must keep the device below throttling limits.
Thermal throttling is managed by the Limits Management Hardware (LMH), which reduces frequency and voltage when a thermal trip point is reached. The CPU clusters, GPU, and RAM each have independent thermal zones and trip points. Since the RAM package is stacked directly on the SoC, its thermal limit can also trigger CPU throttling.
Qualcomm’s default trip points (95°C) are designed for passively cooled smartphones, while comma four has active cooling and a larger thermal mass. Because of this, the CPU, GPU, and RAM trip points were increased by 10°C to 105°C (agnos-kernel-sdm845#110).
The comma four was designed to operate like this, with a better heatsink design, a second heatsink on the bottom of the chip, and a very quiet fan. The software was still running a conservative thermal policy. Raising the trip points aligns the software with the hardware’s cooling performance.
To validate the new trip points, a comma four was heat-soaked to 84°C before going onroad while CPU, GPU, and RAM temperatures were logged. Under initial load, CPU and GPU temperatures briefly peaked at 96°C before stabilizing around 68°C once the fan ramped up. RAM stabilized near 60°C.
CPU, GPU, and RAM temperatures during ignition after heat-soaking the device at 84°C. The new 105°C throttling threshold is shown in red.
The new threshold was merged into master on April 24, 2026. Since then, no comma four running openpilot master has been thermally blocked from going onroad.
Daily count of comma four thermal blocks
The longitudinal maneuver report enabled fast iteration on acceleration and jerk tuning by comparing commanded vs. measured response during scripted maneuvers. This release adds the same concept for lateral control. (#37562)
On a straight, level road at constant speed, openpilot commands a set of lateral acceleration profiles and records the car’s response. Three maneuvers are run at 20 and 30 mph:
| Maneuver | Command |
|---|---|
| Step right | +0.5 m/s² for 1 s, then -0.5 m/s² for 1.5 s |
| Step left | -0.5 m/s² for 1 s, then +0.5 m/s² for 1.5 s |
| Sine 0.5 Hz | 1.0 m/s² amplitude, 2 s period |
Each maneuver repeats three times and only starts after the car has held speed for two seconds. Comparing commanded and measured lateral acceleration exposes rate limits, delays, and tuning issues.
Sine 0.5 Hz maneuver at 30 mph on a Tesla Model Y and Kia EV6. The Tesla tracks the command closely, while the Kia is steering-rate limited.
Lateral maneuver mode can be enabled in developer settings before going onroad. The maneuvers then execute automatically once conditions are met. The route rlogs need to be uploaded to connect, after which the HTML report can be generated with tools/lateral_maneuvers/generate_report.py route_id.
openpilot is built by comma together with the openpilot community. Get involved in the community to push the driving experience forward.
#driving-feedback on DiscordWant to build the next release with us? We’re hiring.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。