Detects fraudulent phone calls as they unfold. Live audio is transcribed in overlapping chunks by OpenAI Whisper, and each running transcript is scored by a fine-tuned multilingual BERT classifier that outputs a fraud probability and a risk level.
Scam calls work because the victim only realizes what is happening after the money is gone. A detector that scores a call while it is still in progress can warn the person at the moment it matters, rather than after the fact. This project builds that pipeline end to end: speech in, risk score out.
- Audio chunking — the incoming call is sliced into 6-second windows with 0.5s overlap, so a sentence split across a boundary is not lost.
- Transcription — OpenAI Whisper (
small) transcribes each chunk. Thesmallcheckpoint was chosen as the speed/accuracy tradeoff that keeps the loop real-time. - Running transcript — chunks are accumulated; the classifier reads the most recent ~1200 characters, so the score reflects where the call has gotten to, not just the last few words.
- Classification — a fine-tuned
bert-base-multilingual-casedoutputsP(fraud). - Risk banding —
< 0.35→ LOW,< 0.80→ MEDIUM, else HIGH.
A Gradio interface replays an audio file through the pipeline to simulate a live call, logging the risk score as it evolves chunk by chunk.
quanshiyun/anti_fraud_dataset — Chinese-language call transcripts labeled fraud / legitimate.
- 24,246 examples, near-perfectly balanced (12,132 legitimate / 12,114 fraud)
- Split 80/10/10 (stratified, seed 42): 19,396 train / 2,425 val / 2,425 test
Evaluated on the held-out test set (n = 2,425).
| Model | ROC-AUC | Accuracy | Precision (fraud) | Recall (fraud) | F1 (fraud) |
|---|---|---|---|---|---|
| TF-IDF + Logistic Regression (baseline) | 0.919 | 0.844 | 0.785 | 0.946 | 0.858 |
| mBERT (fine-tuned) | 0.978 | 0.918 | 0.894 | 0.947 | 0.920 |
Confusion matrix, mBERT:
| Predicted legit | Predicted fraud | |
|---|---|---|
| Actual legit | 1078 | 136 |
| Actual fraud | 64 | 1147 |
The two models catch fraud at nearly identical rates (recall 0.946 vs 0.947). The gain from mBERT is almost entirely in precision: false positives drop from 314 to 136, a 57% reduction. That distinction matters more than accuracy here — a detector that cries wolf on one call in four is one users will learn to ignore, and an alert only helps if people still trust it by the tenth call.
Both models are tuned toward recall over precision, which is the right asymmetry for this task: a missed scam costs the victim money, while a false alarm costs a moment of attention.
git clone https://github.com/drori12/real-time-fraud-call-detection.git
cd real-time-fraud-call-detection
pip install -r requirements.txtThen open the notebook, or run it directly in Colab via the badge above (a GPU runtime is recommended for fine-tuning).
Whisper runs locally — no OpenAI API key is required.
- Chinese only. The classifier was fine-tuned on Chinese transcripts and Whisper is invoked with
language="zh". Applying it to another language requires re-training on labeled data in that language. - Transcription quality is the ceiling. Whisper errors propagate directly into the classifier; noisy lines, heavy accents, and cross-talk degrade detection.
- Latency is untested under real load. The pipeline is evaluated by replaying audio files, not on live streaming audio.
- Novel scam scripts. The model learns patterns present in the training corpus; a scam using unfamiliar framing may score low.
- The dataset is balanced; real call traffic is not. Fraud is rare in the wild, so precision on a realistic base rate would be substantially lower than the table above suggests.
├── Chinese_fraud_detection.ipynb # full pipeline: data → baseline → mBERT → Whisper → Gradio
├── requirements.txt
└── README.md
MIT