Putting tech to the test: How do we make sure new solutions work?

However, that evaluation process itself is tricky, not least as with so many providers out there, it’s hard to parse through the noise. Solution vendors come armed with lofty claims about the potential of their technology and promise the earth, but in practice the results may not pan out – wasting time, resources and energy. 

How, then, can insurers ensure they are evaluating solutions quickly, and optimally? At Tractable, we’ve successfully moved over 30 of the world’s top insurers through from initial testing phases to becoming full-fledged customers, using our AI on real claims and helping real policyholders recover faster when they’ve had an accident.

As a result, we’ve seen a successful trial or two – as well as ones that haven’t gone to plan. Here’s our advice on how to get the most out of a test, with a couple of examples of how we’ve seen this play out in practice. 

Put the solution under pressure: test ‘as live’

At Tractable, we’re confident that our AI solution is the market leader, with the aim that it’s as accurate as a human assessor, and used successfully by leading companies in the US and elsewhere, including Tokio Marine, Ageas, MS&AD, and many more. We process claims worth over $2bn a year, accelerating claims from, in some instances, taking weeks into minutes – being processed on a single phone call…

But then, we would say that, wouldn’t we? 

And that’s the problem – it’s all too easy for us, and other companies that offer tech solutions, to make claims about the efficacy of what they offer. And it’s not easy to tell the truth from what might be well-written hyperbole. 

That’s why testing is crucial – but this step too can fail if a test isn’t precise enough. For example, there have been times when an insurer has asked Tractable to show its results against other providers that claim to have a working AI – but said that every party could return the results in 48 hours, or even longer.

That timeframe isn’t useful, as it doesn’t put any pressure on the AI. You need to create a situation where the vendor has to rely on its technology to do the work, not provide a get-out clause whereby it can manually check the computer’s math and make changes accordingly. 

So our suggestion: propose a live test. At Tractable, we want realistic – and even difficult – testing conditions that weed out solutions that can’t help, and highlight those that can. Only that way can you really be sure that a technology can do what its vendors claim it does – while also ensuring that the vendor itself is a good potential fit. 

Let’s take a look at how this might work in practice.

Ensure the solution can handle scale

One of the challenges for an insurer is knowing whether a solution can perform as well as the provider claims outside of a controlled environment. 

It’s relatively easy for a vendor to ‘prove’ that a technology performs at a promised standard if you’re measuring it on a few claims that have been carefully pre-selected to include no surprises. However, applying that solution at scale, in an environment where every variable (from weather, to vehicle make and model, to accident type, to image quality) is a potential unknown, is another question entirely. 

Introducing more realistic testing conditions means the buyer can have more confidence in what they are buying, even before a purchase is made. For example, earlier in 2021 one of the world’s largest insurers asked us to carry out a live test of our AI solution to see how well it would solve their problems. The test itself was challenging – it required us to process 1,000 claims in just a few hours, and to analyze the results in a workshop with senior leadership. 

That’s a big ask: normally, an insurer invites you to process a much smaller number of claims – say 100 or so – but this company really wanted to put our technology through its paces. Their leaders made it clear that they wanted to test the AI’s ability to work at speed, at a scale that made potential human intervention impossible – ensuring they established as accurate an impression as possible of its capacity and any limitations.  

We wanted that too – but there was a catch. Normally, you only upload so many claims for a company that’s actually a customer – so you’ve already built all the integrations you need for that to happen. But this was a test, which meant none of that infrastructure was in place yet. So, we had to construct, from scratch, an automation system that would take data from the images and ingest it into our assessment system – allowing us to run the ‘real life’ test for the client as per their specifications, and enabling us to show them the results in the desired time frame. 

By doing so, we were able to prove to the client that we could meet their expectations on accuracy and timing (as the results from the AI exceeded their targets). However, we were also able to show that we can work with a very important customer in a way that proves we will be a reliable and supportive partner to them, which is key when partnering with one of the world’s largest and most credible insurers. They need to be sure they can trust both your technology, and you.