FMCSA | SambaSafety
The State of the CSA Score: IRT and Facing the Future – Fully Informed

The State of the CSA Score: IRT and Facing the Future – Fully Informed

In the dozen years that I have been involved in the Trucking Industry, I have met many amazing people and seen advances in safety technology, data, and analytics that are mind-boggling.  I have met hundreds of business owners and executives who operate their businesses with a vision of literally delivering America, but a mission built on Safety.  If I have heard it once, I’ve heard it 100 times – competition stops at safety’s door.

In 2007, I founded Vigillo and now proudly serve as an executive with SambaSafety. While SambaSafety is a supplier to the industry and exists as a for-profit business, we live and breathe the mission of safety.

Change has been constant in my industry tenure and perhaps none as momentous as the pending changes to the foundation of the CSA scoring methodology – used by industry and the FMCSA to gauge a carrier’s commitment to safety.

The specific topic is the work SambaSafety, and previously Vigillo, has done to work through the transition from CSA to the new IRT Model as defined by the National Academies of Sciences, Engineering and Medicine (NAS) in their report of June 2017:

https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/mission/policy/82226/nas-report-final-june-2017.pdf

Change as big as the shift from CSA Classic to an IRT model has led to a robust debate among industry participants. Below are excerpts from some concerns we’ve heard and our response.

 

Question: While it is possible to guess what the new IRT model will look like, how can any early representation produce results that will align with the final model?

We didn’t guess at the design of the model. Just the opposite. NAS provided a 130-page roadmap that describes in detail the IRT model.  With counsel from IRT experts, we followed that roadmap to produce our preview.

Moreover, we have a strong track record helping the industry to anticipate and interpret the CSA scoring methodology. Vigillo released the industry’s first CSA management platform in 2008, two full years before the official launch by FMCSA.

We have helped support changes to the CSA program, culminating in the FAST ACT and the IRT recommendation. In its early years, I provided data to illustrate CSA deficiencies to Congress as CSA criticism grew after its launch. I collaborated with the American Trucking Associations and others in the development of the language that was inserted into the FAST Act which ultimately launched CSA Reform.  At the request of the ATA, I gave the industry comments at the first meeting of the NAS and accompanied the ATA Chairman in the delivery of those comments. Over the next 18 months, I attended every public meeting held by NAS. Before the draft report was published publicly, I was one of the primary reviewers. Ultimately, I led the efforts at SambaSafety to retain the services of IRT experts including one of the NAS Panelists.

Back in 2008, Vigillo gave thousands of motor carriers a preview of what CSA was, how it worked, and how it would impact their safety programs. Advance knowledge of these kinds of significant changes at FMCSA is essential. That’s why we’re doing it again with IRT.

 

Question: Why worry about IRT now when it’s not near implementation?

The NAS report provided for a two-year timeline following the initial 18-month study.  That would call for implementation in June of 2019. FMCSA has stated publicly that its target is September 2019, not significantly behind.

Our best information says that we’ll see careful rollout and solicitation of industry feedback over the following six months.  Full implementation is expected in Q2 2020. The reforms are moving along pretty much as outlined in the FAST Act.

IRT produces very different results for many carriers.  IRT uses a methodology that is easy to interpret but nearly impossible for a non-statistician to calculate. Many of our customers want a preview of their IRT score to be included in the discussion, right from the start. It’s imperative for them to know how their safety culture will be assessed and it allows them to provide informed feedback to the FMCSA during the solicitation period.

At SambaSafety, we live the mission of safety, and we innovate. We invest in R&D and are continually building new products and services to serve that mission.  We think our job is to get there first. We have a proven track record of doing that time and time again. Our customers benefit from our ability to anticipate change, understand formerly hidden risks, participate in the discussion of the new model and prepare for a two-year look back.

It’s a choice to be proactive and get involved early to anticipate significant industry changes like IRT.  Hundreds of carriers are making that choice.  We think that’s a good thing for safety.

 

Question: Does IRT really improve upon legacy CSA?

It does, no question about it.

Let’s look at three of the defects of CSA as it exists today:

  1.  Safety Event Groups – Spiking Scores
  2. Disparate State Enforcement – Geography sways Scores
  3. A relatively small number of scored carriers – Less than 20% of carriers have sufficient data for any CSA Score

First, Safety Event Groups, as constructed today, group carriers in an attempt to compare similar carriers.  It has never worked well and has some problematic side effects.  Five of the seven Groups are based on inspection count.  A carrier can move from one group to another by virtue of just one inspection, even a clean inspection, and scores spike dramatically as a carrier moves up the ladder in Safety Event Groups. Related to this, a Carrier’s CSA Measure produces very different CSA Percentile Scores depending on what group they are in.  As a result, Carrier A with a BASIC Score of say 50% is not comparable to Carrier B with 50% in a different Group. 

IRT does away with Safety Event Groups in favor of a Risk Exposure Index.  IRT utilizes this Exposure Index which is created by blending Power Unit Count, Driver Count, VMT and Inspection Count to normalize carriers for comparison purposes. No groups to leap between and all carriers scores are comparable to each other.

Second, Law Enforcement Disparity has always plagued CSA.  CMV Enforcement on the New Jersey Turnpike is a very different animal than enforcement at the Texas border.  Our country is vast, diverse, and immensely challenging from an enforcement perspective. Enforcement knows their unique challenges and focuses on the specifics that they believe enhances safety in their own back yards. CSA does not formally recognize these challenges and punishes carriers who operate in targeted enforcement zones vs. carriers who do business elsewhere.

IRT is not about the frequency of violations.  IRT looks at patterns of violations; it’s not just counting them.  As a result, the wild swings we see in CSA Scores due to disparate enforcement are largely smoothed out. 

Third, CSA today cannot score a carrier with insufficient inspection data.  As a result, only about 100,000 carriers receive any CSA Score at all. IRT has a lower threshold for what constitutes sufficient data for it to evaluate and provides scores for almost 200,000 carriers.

There are other improvements that IRT brings in terms of a more scientific foundation, ability to adapt to changing patterns in the data and it is set up well for additional data to be added in the future.  We can hardly imagine the magnitude of varying types and amounts of data that trucking operations will generate in the future. IRT is uniquely capable of incorporating new types of data to make continuous improvements to our understanding of what constitutes an excellent Safety Culture in the Trucking Industry.

 

Steve Bryan

 

Happy Anniversary FMCSA – Meet The Parents

Happy February everybody, and Happy Anniversary!

Its been exactly 6 months (August 1 – Feb 1) since FMCSA launched their 24 month Crash Preventability Demonstration Program to see what impact it would have if FMCSA reviewed DOT Record-able crashes and made Preventable/Non-Preventable determinations for purposes of CSA crash scores. So happy 6-month (26 week) anniversary!

According to Match.com findings a 6-month relationship is important because this is when three key relationship milestones take place:

  • Revealing of one’s imperfections (24 weeks)
  • First argument (23 weeks)
  • Parental introductions take place (24 weeks)

So I ran some numbers FMCSA, its time to meet Momma.

Imperfection #1 – Permitted time period out of sync with CSA Scores

Lets start at the highest level.  Motor Carrier (CSMS) CSA scoring is based on the most immediate past 2 years of data including crashes.  In the two years prior to January 1, 2018, FMCSA recorded 307,732 crashes.  (MCMIS as of December snapshot).

All Crashes – Two Years

But FMCSA only allows crashes after June 1, 2017 to be submitted for reviews under its new program that started August 1, 2017.  So less than 25% of CSA crashes can be submitted at this time, it will only catch up with the 2 year CSA window, as the Demonstration Program comes to a close at the end of 24 months.

Crashes Qualifying for Review – 76,122 (24.7% of all Crashes)

Crash Locations – States (Oh look, Texas wins again!)

So as we sit here at the 6-month milestone, we can only judge the preventability project on the 1,149 submitted crashes out of 76,122 crashes that could possibly be submitted under the limitations put in place for this program, or 24.7% of all possible crashes. I never understood the reasoning behind limiting crash submissions to those since June, when FMCSA uses crashes back two years for scoring purposes. Seems fair to align those dates, but hey, we were early in the relationship and we didn’t want to have that first argument.

Imperfection #2 –  Transparency could be better

On August 1, as soon as the Demonstration Program launched, Vigillo started capturing all crashes submitted through DataQ’s.  FMCSA publishes the outcomes of crash reviews publicly, but cycles them through every 30 days, removing the previous month from public view.  There is no running total, so we created one.  We set up a daily capture and can now report that 1,149 crashes have been submitted for review by 588 different motor carriers since August 1.  That’s just 1.5% of the possible 76,122 that have been recorded in that same time period. This is starting to look an awful lot like the DataQ’s system we already dated and broke up with.  Lets get some reporting in there team, this is a demonstration program, lets see the results without having to do backflips. Motor Carriers, lets get busy, submit those crashes.

Imperfection #3 – Only certain predetermined categories can be submitted

FMCSA predetermined that only crashes in the following categories can be submitted for review:

  • When the commercial motor vehicle (CMV) was struck by a motorist driving under the influence (or related offense)
  • When the CMV was struck by a motorist driving the wrong direction
  • When the CMV was struck in the rear
  • When the CMV was struck while it was legally stopped or parked, including when the vehicle is unattended
  • When the CMV struck an individual committing, or attempting to commit, suicide by stepping or driving in front of the CMV
  • When the CMV sustained disabling damage after striking an animal in the roadway
  • When the crash was the result of an infrastructure failure, or falling trees, rocks, or debris
  • When the CMV was struck by cargo or equipment from another vehicle

And this is the breakdown now 6 months into the Program:

Actual Submissions by Crash category – 1,149 submitted

Actual Review Outcomes by Crash category – 596 Non-Preventable (52%)

Our first argument – Many clearly Non-Preventable Crashes are not being reviewed.

So to recap, Crashes going back two years should be permitted for submission, we need better and more transparent reporting from FMCSA, and with the huge investment in technology, including camera’s, there should not be predetermined categories of acceptable crashes.  It is impossible for anyone outside FMCSA to know how many crashes are being submitted and rejected out of hand because they are not in one of the pre-determined categories, and even less knowable, how many carriers don’t even try.

The link below is from a motor carrier who will remain anonymous, and it shows a crash that is clearly not preventable by the driver or motor carrier.  But it is a side collision, and FMCSA has predetermined that no side collision can be reviewed.  It is a Demonstration Program, lets demonstrate what is possible today with technology and video.  Until you do, I’m not sure how I feel about continuing our relationship, or having you meet my parents.

 

Steve

 

UPDATE: https://www.overdriveonline.com/fmcsa-clarifies-how-to-submit-crash-preventability-review-requests-for-pilot-program/?utm_source=daily_utm_medium=email_utm_content=02-07-2018_utm_campaign=Overdrive_ust_id=b5c8550d89cbb74bab233e6e8dc92bec

FMCSA UPDATE: WEDNESDAY 12/21/2016

Hi everyone,

We have not updated you about the FMCSA upgrades in a  little while but we’ve been back to Washington DC and met with the FMCSA since last time and have some news:

The FMCSA upgrade is still not complete but thankfully the FMCSA are now posting news about the upgrade status, you can see those below. At Vigillo what we are seeing is that some DOT’s still cannot log in to FMCSA Portal. The FMCSA are very aware of this and we contact them multiple times per day and have escalated this to the very highest levels.

SMS 

The SMS BASIC Percentile scores for this month were calculated with inspection, violation and crash data only up through November 16th, not up to the SMS Snapshot date of November 25th. The data for the latter half of November (Nov 17th – Nov 25th) will be included in next month’s SMS update. This may cause you some anomalies in your BASIC Percentile scores this month as only half the usual amount of data is being used. Still, the CSA Methodology is a relative scoring system so much of this should mostly balance out but we do see some significant Percentile changes for the month.

We are seeing that some States are still not reporting inspection, violation or crash data, or are not reporting it completely. We don’t yet know if this data will be backdated at a later time but we will keep you updated on this.

Here is what the FMCSA are reporting:

  • Display of the Safety Measurement System (SMS) Behavior Analysis and Safety Improvement Categories (BASICs) is currently disabled within the Portal. To view your SMS data, log into the CSA SMS Website.
  • Not all State inspections/crashes are uploaded at this time. FMCSA is working/troubleshooting with some individual States to complete the inspection/crash upload.
  • Individual Carrier Safety Profile generation is not working for some carriers. FMCSA is working to restore this functionality to all carriers

QC Mobile API

The QC Mobile API used to programmatically retrieve L&I information is back online although it does have some intermittent problems and was down for much of last weekend. It is working correctly at this moment. The FMCSA are working on this.

The FMCSA have reported to us that they expect to have all problems relating to the Compass Portal upgrade addressed by the time of the next SMS monthly update which is on the week of January 9th 2017.

We will keep you updated as we find out more and Happy Holidays from Vigillo.