This article aims to provide a high-level view of the various considerations when setting thresholds and developing TM rules. Identifying new rules for development is not covered in this article.
Tuning Phases
When the rule is being designed, the initial analysis will be required to determine if it will work or not. This will involve ensuring the relevant data is available, and that the concept is feasible. For example, if the rule aims to compare actual turnover against expected turnover, the initial analysis may need to check the following:
Detailed analysis will then be required to set up the new rule, prove that it does work correctly and that it will add value and to determine the levels to set the thresholds at. This will involve an iterative process with sample alerts for review by the investigator and a feedback loop with the analyst.
Once the rule is in production on your TM system, periodic review is necessary to determine that the rule is still of benefit and that the thresholds are still valid.
Tools for Tuning
Test Environment -Approaches will vary slightly depending on the TM system in place and the set up of the test environment. Some approaches may involve the analysis to be completed outside of the TM system. For example, directly within the Datawarehouse environment, or others will have development directly on a test environment within the TM system. It is good practice to have the test approach fully documented, including roles and responsibilities, details of test environments and access control to the data and the environments, etc.
Data – Accessibility of data and the use of live data are both key factors in rule tuning and development. Having the right data available in a timely manner on the TM system is critical to enable accurate rules to be developed. Often it is not until the data is under the microscope as part of rule development that the real challenges are noticed. Customer segmentation can often be one where challenges arise particularly for business customers. The supermarket category is all well and good until you see the giant store in the same category as the small corner shop.
The use of live data as opposed to test data will be critical for proving that the rule is beneficial and that it will help identify suspicious activities. Data security and access controls should be appropriately managed. Using recent data that has already been processed by the TM system is also beneficial to indicate if the new rule is identifying new risks that have not yet been flagged by the other rules.
Resources – Two heads are better than one in this case – combining the skills of a data analyst with the experience of an AML investigator will prove invaluable. The data analyst will be able to crunch the numbers, identify trends and confirm that the rule does what it is supposed to do, whereas the AML investigator will be able to review the sample alerts and confirm if they are identifying suspicious activity – confirming that the rule will or will not add value. It is an iterative approach between the analyst and investigator where tweaks to the rule, both data inputs, and thresholds, will be made and the alerts reviewed again by the investigator until a decision point is reached as to whether the rule will add value or it should be cancelled.
The relevant senior manager or working group should be included as necessary for sign off on the rules and thresholds. Having a functioning governance structure in place for TM is essential. There is an expectation that decisions regarding introducing new rules, changing existing rules and setting of thresholds are fully documented and go through a robust approval process.
Threshold Setting and Tuning
Determining an appropriate threshold and subsequent threshold reviews can be a source for much debate and consideration.
Analysis has indicated that there is typically a high volume of false positives generated by rules-based TM systems (90-95% false positives). This is a staggeringly high statistic meaning that AML investigators are devoting significant time and effort closing alerts that do not present a real risk. This stresses the importance of having the right rules running with the right thresholds.
The phrase don’t use a sledgehammer to crack a nut is very relevant here – deploying a couple of rules with a broad range of thresholds with a view to catching as much as possible is not an effective approach. This will result in high volumes of false positives and valuable resources being wasted reviewing low risk alerts resulting in investigator fatigue.
Applying a strategic approach to determining the rules and thresholds is a must. Consideration of the actual risk the rule is looking to monitor and aligning this to the overall TM strategy and applying a proportionate response will be beneficial.
There will always be outliers, and it won’t be possible to have a rule to catch all of these. Simple approaches with the rules and data will go a long way to improve performance. For example, as opposed to hard coded values in the thresholds, using multiples of values can both broaden the reach of a rule and make it more targeted. For instance, it may be more standard behaviour for your business customer to lodge cash.
Instead of using a defined cash value for which you want an alert generated i.e. show me those business customers who lodged €75k, instead look for those customers which for example lodge 50% more cash than they did last month as this may be potentially more suspicious and require investigation. Seasonal variations can obviously impact but rules can be further developed to try and account for this. Customer segmentation can also alleviate some of the unnecessary alerting but, often the level of segmentation is lacking.
While it is not considered good practice to let resourcing levels drive thresholds, the impact on resources of introducing a new rule at a certain threshold must be considered. For example, the addition of a new rule which will increase workload by 50% overnight may mean all alerts are now at risk of being processed late. Introducing the rule on a phased basis or targeted at specific customer segments could be a far more effective means of addressing the specific risk. It should be noted that regulators have called out the need for funding to be available to maintain the TM programme, as such, delays in adding to your suite of rules in production on the TM system due to resourcing issues is unlikely to be accepted as a valid argument, should the case arise.
The Alert to SAR/STR ratio (some call it the alert to case ratio or the conversion rate) is of significant value in determining how effective thresholds are for each rule – it can easily indicate the false positive rate for the rule by simply looking at how many of the alerts resulted in the generation of a SAR/STR. This is where the input from the AML investigator is invaluable in helping determine the appropriate threshold to set. Considerations such as do the alerts results in a SAR being made, would the SAR have been reported anyway from a different alert, or manual report by branch staff are all relevant.
For new rules in development, the Alert to SAR ratio can be calculated based on the work of the AML investigator when testing the outputs – the test alerts need to be processed as if they are real alerts, with the AML investigator determining if they do present a real suspicion or not. Again, this stresses the need for access to live data when in the development and testing stages.
For Rules which are already in production on the TM system, the Alert to SAR ratio is a good indicator of the value of the rule on a consistent basis. If it changes dramatically it can be a good indicator that something has changed, potentially with the availability of data. It is a valuable indicator to include in the suite of MI for the TM programme.
Looking at the inverse of the Alert to SAR ratio – the volume of alerts not converted to alerts is often referred to as the False Positive rate – effectively the same data but a different way of looking at it.
Agreeing your internal expectations for the Alert to SAR rate is an important exercise. This should be considered and documented as part of the steering groups responsibilities. Depending on the risk being monitored by a rule a lower Alert to SAR ratio may be deemed appropriate.
Sometimes the starting thresholds for the analysis may simply be an arbitrary number that the analysts will use to get a benchmark to start the initial conservations with the AML investigators.
Above and Below the Line testing is a powerful method for refining a threshold. It is particularly helpful when the rule is well understood, and the overall data set and approach is agreed. It involves looking at the impact the thresholds have on identification of a real suspicion – i.e. converting the Alert to a SARs/STRs. When the threshold is increased/decreased by a certain percentage, by how much will this change the volumes of SARs created? The table below provides an illustration of how effective it can be for refining the threshold.
If the thresholds are decreased, more alerts are generated.
If the thresholds are increased, less alerts are generated.
Rule 1
Current Threshold
Decrease Threshold
10%
Scenario A
Decrease Threshold
20%
Scenario B
Increase Threshold
10%
Scenario C
Increase Threshold
20%
Scenario D
Volume of Alerts
100
110
140
90
80
Change in Alert Volume
N/A
+10
+40
-10
-20
SAR Volume
10
10
11
10
7
Change in SAR Volume
N/A
No change
+1
No change
-3
Conclusion
N/A
10% increase in workload with no impact on volume of SARs
40% increase in workload for gain of 1 additional SAR
10% reduction in workload with no impact on volume of SARs
20% reduction in workload but a 30% decrease in volumes of SARs
Scenario C would present the best outcome in most cases. The threshold has been increased by 10%, this resulted in a 10% reduction in the volume of alerts generated with no impact on the number of SARs, i.e. the same number of SARs were generated, yet less alerts needed to be reviewed.
The argument could be made for Scenario B due to the additional SAR being made but this gain has been offset by a significant 40% increase in the workload. If the risk being monitored was considered a business-critical risk, scenario B may then be considered the better option.
This type of analysis in conjunction with good MI will enhance the decision-making process around rule thresholds on an ongoing basis.
In conclusion, it is all well and good having a fantastic TM system in place, but it will only ever be as good as the rules you build. Having the right team and infrastructure in place to develop and analyse the rules will enable a consistent reliable approach ensuring you get value from your TM system investment on a continuous basis.
SQA Consulting can support the development and ongoing assurance reviews of your TM programme. Please contact us to find out more.
