CSA | SambaSafety
The State of the CSA Score: IRT and Facing the Future – Fully Informed

The State of the CSA Score: IRT and Facing the Future – Fully Informed

In the dozen years that I have been involved in the Trucking Industry, I have met many amazing people and seen advances in safety technology, data, and analytics that are mind-boggling.  I have met hundreds of business owners and executives who operate their businesses with a vision of literally delivering America, but a mission built on Safety.  If I have heard it once, I’ve heard it 100 times – competition stops at safety’s door.

In 2007, I founded Vigillo and now proudly serve as an executive with SambaSafety. While SambaSafety is a supplier to the industry and exists as a for-profit business, we live and breathe the mission of safety.

Change has been constant in my industry tenure and perhaps none as momentous as the pending changes to the foundation of the CSA scoring methodology – used by industry and the FMCSA to gauge a carrier’s commitment to safety.

The specific topic is the work SambaSafety, and previously Vigillo, has done to work through the transition from CSA to the new IRT Model as defined by the National Academies of Sciences, Engineering and Medicine (NAS) in their report of June 2017:

https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/mission/policy/82226/nas-report-final-june-2017.pdf

Change as big as the shift from CSA Classic to an IRT model has led to a robust debate among industry participants. Below are excerpts from some concerns we’ve heard and our response.

 

Question: While it is possible to guess what the new IRT model will look like, how can any early representation produce results that will align with the final model?

We didn’t guess at the design of the model. Just the opposite. NAS provided a 130-page roadmap that describes in detail the IRT model.  With counsel from IRT experts, we followed that roadmap to produce our preview.

Moreover, we have a strong track record helping the industry to anticipate and interpret the CSA scoring methodology. Vigillo released the industry’s first CSA management platform in 2008, two full years before the official launch by FMCSA.

We have helped support changes to the CSA program, culminating in the FAST ACT and the IRT recommendation. In its early years, I provided data to illustrate CSA deficiencies to Congress as CSA criticism grew after its launch. I collaborated with the American Trucking Associations and others in the development of the language that was inserted into the FAST Act which ultimately launched CSA Reform.  At the request of the ATA, I gave the industry comments at the first meeting of the NAS and accompanied the ATA Chairman in the delivery of those comments. Over the next 18 months, I attended every public meeting held by NAS. Before the draft report was published publicly, I was one of the primary reviewers. Ultimately, I led the efforts at SambaSafety to retain the services of IRT experts including one of the NAS Panelists.

Back in 2008, Vigillo gave thousands of motor carriers a preview of what CSA was, how it worked, and how it would impact their safety programs. Advance knowledge of these kinds of significant changes at FMCSA is essential. That’s why we’re doing it again with IRT.

 

Question: Why worry about IRT now when it’s not near implementation?

The NAS report provided for a two-year timeline following the initial 18-month study.  That would call for implementation in June of 2019. FMCSA has stated publicly that its target is September 2019, not significantly behind.

Our best information says that we’ll see careful rollout and solicitation of industry feedback over the following six months.  Full implementation is expected in Q2 2020. The reforms are moving along pretty much as outlined in the FAST Act.

IRT produces very different results for many carriers.  IRT uses a methodology that is easy to interpret but nearly impossible for a non-statistician to calculate. Many of our customers want a preview of their IRT score to be included in the discussion, right from the start. It’s imperative for them to know how their safety culture will be assessed and it allows them to provide informed feedback to the FMCSA during the solicitation period.

At SambaSafety, we live the mission of safety, and we innovate. We invest in R&D and are continually building new products and services to serve that mission.  We think our job is to get there first. We have a proven track record of doing that time and time again. Our customers benefit from our ability to anticipate change, understand formerly hidden risks, participate in the discussion of the new model and prepare for a two-year look back.

It’s a choice to be proactive and get involved early to anticipate significant industry changes like IRT.  Hundreds of carriers are making that choice.  We think that’s a good thing for safety.

 

Question: Does IRT really improve upon legacy CSA?

It does, no question about it.

Let’s look at three of the defects of CSA as it exists today:

  1.  Safety Event Groups – Spiking Scores
  2. Disparate State Enforcement – Geography sways Scores
  3. A relatively small number of scored carriers – Less than 20% of carriers have sufficient data for any CSA Score

First, Safety Event Groups, as constructed today, group carriers in an attempt to compare similar carriers.  It has never worked well and has some problematic side effects.  Five of the seven Groups are based on inspection count.  A carrier can move from one group to another by virtue of just one inspection, even a clean inspection, and scores spike dramatically as a carrier moves up the ladder in Safety Event Groups. Related to this, a Carrier’s CSA Measure produces very different CSA Percentile Scores depending on what group they are in.  As a result, Carrier A with a BASIC Score of say 50% is not comparable to Carrier B with 50% in a different Group. 

IRT does away with Safety Event Groups in favor of a Risk Exposure Index.  IRT utilizes this Exposure Index which is created by blending Power Unit Count, Driver Count, VMT and Inspection Count to normalize carriers for comparison purposes. No groups to leap between and all carriers scores are comparable to each other.

Second, Law Enforcement Disparity has always plagued CSA.  CMV Enforcement on the New Jersey Turnpike is a very different animal than enforcement at the Texas border.  Our country is vast, diverse, and immensely challenging from an enforcement perspective. Enforcement knows their unique challenges and focuses on the specifics that they believe enhances safety in their own back yards. CSA does not formally recognize these challenges and punishes carriers who operate in targeted enforcement zones vs. carriers who do business elsewhere.

IRT is not about the frequency of violations.  IRT looks at patterns of violations; it’s not just counting them.  As a result, the wild swings we see in CSA Scores due to disparate enforcement are largely smoothed out. 

Third, CSA today cannot score a carrier with insufficient inspection data.  As a result, only about 100,000 carriers receive any CSA Score at all. IRT has a lower threshold for what constitutes sufficient data for it to evaluate and provides scores for almost 200,000 carriers.

There are other improvements that IRT brings in terms of a more scientific foundation, ability to adapt to changing patterns in the data and it is set up well for additional data to be added in the future.  We can hardly imagine the magnitude of varying types and amounts of data that trucking operations will generate in the future. IRT is uniquely capable of incorporating new types of data to make continuous improvements to our understanding of what constitutes an excellent Safety Culture in the Trucking Industry.

 

Steve Bryan

 

2019 Truckload Carriers Association Annual Conference

2019 Truckload Carriers Association Annual Conference

SambaSafety is proud to be here in force as one of your Allied partners.  We’re exhibiting, presenting, and sponsoring at what is one of our favorite events of the year.

Almost everything you know about CSA is changing, and Sambasafety is at the forefront of helping carriers understand those dramatic changes and the new methodology called Item Response Theory (IRT).  IRT produces a new generation of CSA Scores including a new single Safety Culture Score. For TCA Attendees only, stop by and visit us at booth 428, drop off your business card, and we’ll follow up after the show and give you a free, sneak peek at your new scores.  Just write “Sneak Peek” on your business card and drop it with us at the booth.

We love Vegas, but don’t gamble on being caught off guard when the new scores release later this year.  Let us show you how it all works and how the new IRT/CSA will evaluate your safety culture.

The New CSA/IRT Scorecard

The New CSA/IRT Scorecard

The Next Generation of CSA

Introducing the new CSA/IRT Scorecard

The new CSA will measure a motor carrier’s

Safety Culture with a single score

CSA is undergoing major changes and almost everything we know about CSA—severity weights, time weights, BASIC measures and Safety Event Groups—will no longer factor into the new scoring methodology under the FMCSA’s planned 2019 release.

Our new IRT/CSA Scorecard provides you access to your new CSA Safety Culture Score—an entire year ahead of the FMCSA planned release date.

Establish Your Safety Score NOW!

Thank you! The information has been submitted successfully.

Be the first to know your CSA Safety Culture Score

The two-year look back is already in effect. We are offering you visibility into your new CSA Safety Culture Score to help you get ahead of how it measures your company’s safety culture—and show you what you can do now to prepare.

CSA Safety Culture Score

The new CSA FAST Act Score Model utilizes Item Response Theory (IRT) methodology and almost completely changes the building blocks of the current CSA scoring model.

It more accurately measures a company’s Safety Culture as your new score is based on inspections, violations and violation groups, and also factors in relevant variables such as drivers, inspections, VMTs, number of power units and more.

BASIC Comparison Scores

Get a side-by-side view of your current  CSA scores next to your new CSA/IRT  scores and start to understand the new  CSA scoring model.

Violation Groups

See your activity by violation group. The new  methodology identifies the same possible 945  CSA violations and assigns them to one of 66  violation groups. These groups are assigned  across the BASICs.

Industry Benchmark

How do you rank in the Exposure Risk Index?  See how you stack up against like motor  carriers in the industry and where you rank  amongst your peers for safety culture.

Violation Group Detail

Get the big picture with violation count, two  year timeline, severity indicator, and violations  falling off. See where you need to focus to  maintain or improve your CSA Safety Culture Score.

Violation Detail

Drill down to the specific list of individual  violations that make up your overall violation  count in any given group.

Driver Detail

The driver detail page will show you violation, date and location by driver.

Protect Your Business Now!

Click Here

FMCSA submits CSA overhaul plan to Congress

FMCSA submits CSA overhaul plan to Congress

Here we go!!!

As reported this morning by our friends at CCJ: FMCSA submits CSA overhaul plan to Congress…read more.

Almost three years since the FAST Act demanded CSA Reforms, looks like FMCSA is rolling it out.

Major changes pending to CSA methodology; Vigillo, a SambaSafety Company has been working in parallel to ensure we are ready this year to help customers interpret and act on the new scores and move Beyond Compliance with the right combination of data and Wisdom Deployed.

Are you ready for this?

Existing Methodology Example:

 

New methodology Example:

 

Motor Carrier Survey – for MC&E

Hello everyone, Our biggest annual event is upon us.  The American Trucking Assn’s Management Conference and Exhibition (MCE) starts tomorrow in Orlando, FL.  On Sunday morning, I will be participating in a panel discussion of the changes to CSA that have been suggested by the National Academies in the CSA Reform study they completed in June.

One of the recommendations made is to implement a new statistical model into CSA called Item Response Theory (IRT).  In my session Sunday morning at MC&E at 9:45 AM in room W305AB (shameless plug, please attend if you’re here), I will be using the results of a silly surey to explain fundamentally how IRT works.  I could use your help, I need some data.  Please click the survey link below and answer the 10 survey questions.  Make up a nickname, its totally anonymous.  I’ll use the aggregate responses, along with some specific right/wrong responses and show how IRT analyzes things like survey questions and map that to how IRT might do a better job of analyzing vioilation data in CSA.

www.pollev.com/vigillo

Appreciate your help, see you Sunday morning in Orlando.

 

Steve

CSA Reform Status – Vigillo Comments

Truck-Driving-with-graphs3

PDF Version Here

 IMPROVING MOTOR CARRIER SAFETY MEASUREMENT

Panel on the Review of the Compliance, Safety, and Accountability (CSA) Program of the Federal Motor Carrier Safety Administration

Initial Comments by Steve Bryan, President & Founder of Vigillo, A SambaSafety Company

June 28, 2017

Yesterday, June 27, 2017, 18 months after the FAST Act removed CSA Scores from public view, the National Academies of Science, Engineering and Medicine (NAS) published their preliminary findings outlining their recommendations to reform the Compliance, Safety and Accountability program we all know and love as CSA.

Their 132 page report starts with a nod to the FMCSA for the work they have done on CSA and for the funding for this study (amount undisclosed).  NAS starts by thanking Joseph DeLorenzo, director, FMCSA Office of Enforcement and Compliance, for his presentations, which were not just useful, but “enormously useful”, way to go Joe!

That was followed by acknowledgements to the other twenty-five presenters the first of which was yours truly (it was alphabetical). We are all thanked but our contributions are not characterized on a usefulness scale.  Item Response Theory (IRT) applied to the relative informational value of each of us would have been nice, but hey, more on that later.

NAS categorizes their report into the following four sections:

  1. Assessment of the Current Safety Measurement System
  2. A More Natural Statistical Model
  3. Data Improvements
  4. Transparency, Reproducibility, and Public Disclosure of Safety Rankings

In one paragraph, I’ll summarize. My take on this entire 132 page report is stated as follows.

“Good job FMCSA on what you have done with CSA so far, we (NAS) give you a solid 5 out of 10 for your efforts so far.  But there needs to be a sound statistical model applied to replace the ad hoc nature of how CSA is formulated today. More, and more reliable data needs to be gathered, and the data utilized needs to be more transparent and software developed to make such data more useful by all stakeholders.  Only then should CSA Scores be released to the public.”

(more…)