Love is NOT in the air for SMS safety scores as they may change to an Item Response Theory model. What does this mean?
In 2017, it was suggested that the FMCSA should radically improve the CSA system with the highly complex Item Response Theory (IRT) model.
However, according to FMCSA’s recent notice, they will not adopt this model but make many changes to the existing SMS system.
There is a lot to break down here, so let’s look at what was cut and what changes are coming.
Don’t forget you can comment in the next 90 days on the FMCSA website.
What was the IRT Model?
A Congress-mandated report by the National Academies of Science issued recommendations that the U.S. Department of Transportation overhaul its Compliance, Safety, Accountability (CSA) carrier rating system.
At the time, the report criticized that some BASICs lacked correlation with:
- Crash risk
- Data insufficiency
- Use of relative rankings
- Use of non-fault or non-preventable crashes
- State variations in inspections and violations
- Lack of consistency in violation coding
- Lack of transparency of the SMS algorithm, and
- Public availability of SMS rankings
The report also offered several recommendations that FMCSA should:
- Reconfigure the SMS statistical model (percentile ranking) with an “item response theory” (IRT) model that “more accurately targets at-risk carriers”
- Make the scoring system more transparent and easier for carriers to understand and replicate
- Depart from using relative metrics as the sole means for targeting carriers
- Better collaborate with state partners and other data providers to collect higher quality data, such as miles traveled and number of power units
[Related: Carriers Must Go Digital to Meet Future Off-Site Audit Expectations]
However, FMCSA found many issues in the testing and models of the IRT approach.
What issues did IRT have?
FMCSA’s IRT modeling work revealed many limitations and practical challenges with using an IRT model.
As a result, FMCSA has concluded that IRT modeling does not perform well for the Agency’s use in identifying motor carriers for safety interventions, and therefore, does not improve overall safety.
Here were the reasons why:
- IRT is heavily biased towards identifying smaller carriers that have few inspections with violations and limited on-road exposure to crash risk. When the safety event groups and data sufficiency standards used in SMS were applied to the IRT model, IRT produced similar results to SMS.
- IRT does not use Vehicle Miles Traveled (VMT) or Power Units (PUs) to adjust for differences in on-road exposure in the Unsafe Driving BASIC. As a result, IRT identified carriers with much lower crash rates in that BASIC compared to SMS.
- IRT modeling is not readily understandable by most stakeholders or the public. IRT’s inherent complexity makes it challenging for the industry and public to replicate and interpret results.
- A motor carrier cannot independently compute its IRT results. IRT results can be computed only for the entire carrier population. A carrier would not be able to identify how specific violations or areas of regulatory noncompliance impacted its prioritization status or how it could improve its status.
- IRT’s runtime is incompatible with FMCSA’s operational needs. The IRT model takes four weeks to run as compared to two days for SMS. The long runtime would make it difficult to make even minor changes to the system.
[Related: Best DOT Audit Guide in the Industry]
What changes will the FMCSA make?
FMCSA proposes the following combined improvements to SMS:
- Reorganized and updated safety categories (BASICs), including new segmentation
- Consolidated and group violations, including adding 14 not in the SMS
- Simplified violation severity weights
- Proportionate percentiles instead of safety event groups
- Improved Intervention Thresholds
- Greater focus on recent violations, and
- Updated Utilization Factor
Let’s break these down.
Reorganizing Safety Categories (BASICs)
Your CSA scores are based on multiple factors called Behavioral Analysis and Safety Improvement Categories or “BASIC” categories. Roadside inspection violations, as well as investigation results, fall under 1 of 7 categories, including:
- Unsafe driving – moving and parking violations, such as speeding, improper lane changes, no seatbelt, cell phone/handheld device use, improper parking, etc.
- Crash indicator – DOT reportable crashes (injury, towaway or fatality)
- Hours of Service (HOS) compliance – falsifying your record-of-duty status, inadequate paperwork for ELD, driving, on-duty and rest break violations
- Vehicle maintenance – mechanical issues and not making required repairs
- Controlled substance/alcohol – driving under the influence
- Hazardous materials compliance – unsafe or incorrect handling and/or documentation of hazardous materials, including improper or inadequate placards
- Driver fitness – Unfit to drive due to physical health or lack of training (sickness, no medical card, driving a vehicle you are not qualified to drive (i.e.- tanker with no ‘N’ endorsement, etc.)
The proposed changes would make the new safety categories:
- Unsafe Driving
- Crash Indicator
- Hours of Service (HOS) Compliance
- Vehicle Maintenance
- Vehicle Maintenance: Driver Observed
- HM Compliance, and
- Driver Fitness
Controlled Substances/Alcohol has the fewest violations of any BASIC, and those violations are also cited relatively infrequently.
The EFA demonstrated that controlled substances and alcohol violations were strongly associated with the Unsafe Driving BASIC and supported removing the Controlled Substances/Alcohol category as a standalone BASIC.
The new Unsafe Driving safety category now includes the drug and alcohol violations that were previously captured in the Controlled Substances/Alcohol BASIC.
In addition, violations for operating while under an OOS Order belong in the new Unsafe Driving safety category.
Currently, SMS places these types of violations across multiple BASICs based on the underlying OOS violation. Moving and consolidating these violations to the new Unsafe Driving safety category would allow motor carriers and enforcement officials to more effectively identify and correct driver-based safety problems related to disregarding OOS Orders.
Vehicle Maintenance is the largest BASIC in terms of both the number of violation identifiers included in the BASIC and the number of violations cited during inspections.
Breaking this category into two separate categories would provide greater specificity to help carriers improve and enforcement officials to conduct targeted investigations.
Consolidated Violations and Severity Weights
FMCSA’s analysis during IRT modeling confirmed that similar violation provisions could be consolidated to mitigate differences that result from inspectors citing different violation codes. Grouping similar violations together would also allow motor carriers and enforcement officials to identify and address specific safety issues more easily. The following table shows a summary of the consolidated violations by safety category.
SMS assigns each violation a specific severity weight that is intended to correlate with the crash risk associated with that violation.
FMCSA proposes to simplify violation severity weights by assigning each consolidated violation group a weight of either one or two.
- OOS violations and violations in the Unsafe Driving safety category that are disqualifying offenses under 49 CFR 383.51 would be assigned a weight of two and all other violations would be assigned a weight of one.
- If an OOS violation is combined with a non-OOS violation in the consolidated violation grouping, the consolidated group would be assigned the higher weight of two.
FMCSA’s evaluation found that simplifying the severity weights identifies carriers with higher crash rates. This change would maintain the safety focus on those violations severe enough to result in an OOS Order while removing the subjectivity and complications of distinguishing each violation by severity on a scale of 1 through 10.
Proportionate Percentiles
FMCSA places motor carriers into safety event groups in SMS based on their number of inspections and crashes. For example, carriers in the HOS Compliance BASIC with 3 to 10 driver inspections are compared to each other, while carriers with 11 to 20 driver inspections are compared to each other, and so forth.
When a carrier moves into a different safety event group, they sometimes experience large percentile jumps based solely on a no-violation inspection in this new safety group.
FMCSA proposes to smooth out the groups with median benchmarks to prevent these sudden jumps in percentiles.
Intervention Thresholds
The FMCSA sends warning letters when safety performance data indicates they are not complying with safety regulations.
The BASIC “intervention thresholds” were established because they are strong indicators of future crash risk and exceeding them can put you on FMCSA’s priority lists which can later result in fines and violations.
- Unsafe Driving, Crash Indicator, and HOS Compliance BASICs have the strongest correlation to crash risk. Therefore, those BASICs have lower Intervention Thresholds than the other BASICs, at 65 percent for property carriers, 60 percent for HM carriers, and 50 percent for passenger carriers.
- The Intervention Thresholds for the Vehicle Maintenance, Controlled Substances/Alcohol, and Driver Fitness BASICs currently are set at 80 percent for property carriers, 75 percent for HM carriers, and 65 percent for passenger carriers.
- The HM Compliance Intervention Thresholds are set at 80 percent for all carriers.
FMCSA’s found that the Driver Fitness and HM Compliance safety categories have the lowest correlation to crash risk and believes raising the Intervention Thresholds in those safety categories would allow the Agency to focus on populations with a greater safety risk.
DOT Audits are Inevitable
While most of these changes to the SMS system sounds fair, it will shake up and impact carriers that were previously close to an investigation threshold.
This may mean you will be targeted more heavily than before.
Are you prepared for a DOT Audit?
There is no question that you will go through a DOT Audit, the question is will you pass a DOT Audit. Our DOT Compliance team is available to help with any type of audit, including New Entrant Audits, DOT Compliance Reviews, Focused Reviews, IFTA Audits or IRP Audits.
Want to be proactive in your safety management before the DOT makes these changes?
Our complete DOT Compliance Programs promotes proactive safety and will complement or become your current safety department, without the cost of employing the many staff members it takes to run an effective safety program.