KEY CHALLENGES
A provider of sanctions screening engines was interested in assessing how effective their screening engine was at identifying sanction list names and how the various system configuration setting options might affect screening effectiveness.
In short, the screening provider wanted to improve the performance of their screening engine and approached SQA for assistance.
SQA PROPOSED SOLUTION
SQA Consulting regularly tests sanctions screening systems for large and small Financial Institutions worldwide, providing assurance on the effectiveness of the screening system.
This requirement was similar to our usual screening assurance work in that we could use our in-house Screening Assessment Centre (SAC) software to generate a test file to assess the effectiveness and efficiency of the provider’s screening engine. The difference here is that the provider wished to assess screening effectiveness using a range of different configuration settings.
In this case, the provider ended up iterating the configuration settings 12 times. The SQA test file was run through the system each time and we returned the analysis of the results within a couple of hours for each test run.
SOLUTION BENEFITS
SQA’s rapid turn-around of test results at short notice allowed the screening provider to progress testing at their own pace to their own test schedule. Not only was the comprehensive nature of the SQA test file of significant benefit in achieving the required testing, but the speed at which multiple test runs could be completed was highly beneficial in achieving the required testing most efficiently in a short time frame.
On completion of testing, SQA presented a quadrant diagram, as shown below, plotting the screening engine’s screening effectiveness vs screening efficiency for each of the configuration settings. This allowed the screening provider to assess how their screening engine’s performance varied using different combinations of configuration settings.
The blue dots represent comparison benchmark results for similar screening tests conducted at various other Financial Institutions. The red dots are the test results for the provider’s different test executions for different system configuration settings.
This graph allows the screening engine’s effectiveness and efficiency to be compared against the benchmark comparisons, and also for the different configuration settings, helping to determine the optimum configuration setting to achieve the desired balance of screening effectiveness and efficiency.
If you are the developer or a user of a sanctions screening system and would be interested in undertaking similar testing to fine-tune and optimise the performance of your screening system, then please contact us and we will be happy to assist.
