Transactional fraud is no longer fought by reviewing static rules in a spreadsheet. A European payment gateway processes millions of operations a day and has milliseconds to decide whether a transaction is legitimate, suspicious or clearly fraudulent. Within that window, machine learning models evaluate hundreds of variables, compare current behaviour against the account holder's history and return a risk score. This article explains how those systems work, which anomaly detection and malware detection architectures are used, and which regulatory framework — the GDPR, PSD2 and the EU Artificial Intelligence Act — shapes their deployment.
Anomaly detection: the statistical basis of fraud
At its core, fraud detection is an anomaly detection problem over highly imbalanced data: fraud usually accounts for between 0.1% and 2% of transactions. That rarity shapes the entire architecture of the model.
The techniques fall into three families. The first is supervised learning, which trains classifiers — gradient boosting with XGBoost or LightGBM, regularised logistic regression or neural networks — on historical labels of confirmed fraud. It works very well with known patterns, but is blind to new schemes. The second family is unsupervised learning, which needs no labels: the Isolation Forest algorithm isolates rare observations by splitting the feature space at random, while autoencoders learn to reconstruct normal transactions and flag as anomalous any operation whose reconstruction error exceeds a threshold. The third, semi-supervised learning, combines both: the model is trained on data that is mostly legitimate, and deviations are detected.
A mature system does not pick a single family. A typical financial institution chains an Isolation Forest for fast screening, a supervised model to score risk and an auditable rules engine for the borderline cases that demand a regulatory explanation.
Feature engineering and real-time learning
The performance of a fraud detector depends less on the algorithm than on the variables it receives. The most predictive ones are temporal aggregations: the number of transactions by the same cardholder in the last hour, the geographical distance between two consecutive purchases, the deviation of the amount from the average of the last 90 days, or how often a new device is used. Computing these variables in real time requires a streaming architecture with a feature store that serves precomputed values at latencies below 50 milliseconds.
There are three common mistakes at this stage. First, data leakage: including as a variable information that is only known after the fraud has been labelled, which artificially inflates accuracy in testing. Second, failing to respect temporal order during validation, evaluating on transactions that predate the training set. Third, ignoring concept drift: fraudsters adapt their tactics, so a frozen model loses effectiveness within weeks and needs periodic retraining with monitoring of the input distribution.
Malware detection with machine learning
Artificial intelligence applied to malware operates on two planes. Static analysis examines the binary without executing it: it extracts file headers, byte sequences (n-grams), the entropy of the sections — packed or encrypted code has high entropy — and calls to system API functions. Dynamic analysis runs the sample in an isolated environment (a sandbox) and records its behaviour: network connections, registry writes, processes it attempts to inject into. From those traces, classifiers distinguish benign from malicious behaviour.
The current frontier is the adversarial attack: small modifications to the binary that do not alter its functionality but evade the classifier. Defending against it requires adversarial training, diverse model ensembles and, above all, not relying on a single detection layer. The European cybersecurity body ENISA stresses this principle of defence in depth.
Automated incident response and orchestration
Detection is only half the job. SOAR platforms (security orchestration, automation and response) take the model's output and execute actions: blocking a card, requiring step-up authentication, isolating a machine from the network or opening a ticket in the security operations centre. The key is calibrating the threshold. A threshold that is too low generates false positives that block legitimate customers and trigger calls to support; one that is too high lets fraud through. The relevant metric is not overall accuracy — useless with imbalanced classes — but precision and recall on the fraud class, summarised by the precision-recall curve and the area beneath it.
A design detail that sets mature systems apart is feedback. Every case reviewed by an analyst — confirmed as fraud or dismissed as legitimate — returns to the training set as a new label. That closed learning loop lets the model adapt to emerging tactics without waiting for a large-scale retraining, but it requires governing label quality: an analyst who flags by inertia degrades the model as much as one who reviews with judgement improves it. That is why it is worth periodically auditing inter-analyst agreement and measuring the average resolution time as an indicator of process health.
Relationship graphs: detecting organised fraud rings
The costliest fraud is rarely the work of an isolated actor. Organised fraud rings share devices, addresses, mule accounts or behavioural patterns that an individual transactional model does not see, because it analyses each operation separately. Graph analysis changes the unit of observation: it models customers, devices, accounts and merchants as nodes, and the transactions or shared attributes as edges. Over that graph, community detection algorithms reveal suspicious clusters — dozens of accounts sharing a single device or channelling funds toward a common point — that would be invisible transaction by transaction. Graph neural networks take the idea further by propagating risk signals between neighbouring nodes, so that proximity to a known fraud raises the suspicion on connected nodes.
| Approach | Needs labels | Detects new fraud | Explainability | Maintenance cost |
|---|---|---|---|---|
| Static rules | No | No | High | Low, but rigid |
| Supervised (XGBoost) | Yes | Limited | Medium (with SHAP) | Medium |
| Unsupervised (Isolation Forest) | No | Yes | Low | Medium |
| Autoencoder | No | Yes | Low | High |
Regulatory framework: GDPR, PSD2 and the AI Act
Any anti-fraud system processes personal data and makes decisions that affect people, so the General Data Protection Regulation requires a legal basis, data minimisation and transparency. Article 22 of the GDPR limits fully automated decisions with significant legal effects: blocking a payment is one of them, so there must be reviewable human intervention and the data subject's right to obtain an explanation. The PSD2 Directive imposes strong customer authentication and explicitly recognises transactional risk analysis as a way to exempt that second factor when the risk is low, which turns model quality into a lever for user experience.
Since 2024, the EU Artificial Intelligence Act has added risk management, data governance and activity logging obligations to high-risk AI systems, a category into which certain credit scoring uses may fall. The Spanish Data Protection Agency publishes specific guidance on how to reconcile profiling with the rights of data subjects.
Steps to deploy an AI anti-fraud system
A realistic deployment follows an ordered sequence. It starts with a data inventory and a data protection impact assessment. Next comes building the feature store and a baseline of auditable rules to serve as a reference. On that base, the model is trained while respecting temporal order in validation, the thresholds are calibrated with real business data, and it is deployed in shadow mode — scoring without acting — to measure the impact before enabling blocks. Finally, drift is monitored and the traceability of each decision is documented for audit.
Frequently asked questions
How much fraud data is needed to train a reliable model?
There is no fixed number, but with highly imbalanced classes the positive cases matter more than the total volume. With a few thousand confirmed frauds and resampling or class-weighting techniques you can obtain useful models; below that, it is best to rely on unsupervised methods.
Does AI replace anti-fraud analysts?
No. The model prioritises and filters so that analysts spend their time on the ambiguous, high-value cases. Human intervention is also a legal requirement for decisions with significant effect under Article 22 of the GDPR.
What is the difference between a false positive and a false negative here?
A false positive blocks a legitimate customer and generates friction and support cost; a false negative lets real fraud through and causes a direct loss. The balance between the two is set by the threshold, depending on the relative cost the business assigns to each error.
Is it mandatory to explain why a transaction was blocked?
When the decision is automated and has significant effects, the data subject has the right to information about the logic applied. That is why it is worth using interpretability techniques such as SHAP and keeping an auditable rules engine alongside the model.
Conclusion
Automated fraud detection is not about replacing rules with an opaque model, but about combining several layers — unsupervised screening, supervised scoring and auditable rules — over variables computed in real time, and accepting that the adversary evolves and forces retraining. The real competitive differentiator lies in calibrating the threshold well to minimise friction for the legitimate customer without opening the door to the fraudster, and in doing so within the framework imposed by the GDPR, PSD2 and the AI Act. At Summum we support financial institutions and merchants in that design: from the data inventory to the shadow-mode deployment, always with the traceability that regulatory audit requires.