Flagship · powered n=100 / 200 · measured 2026-08-30 · Apple M3 Pro / MPS
TinyLlama + Alpaca leaked membership. It did not spit the data back.
memaudit plants canaries in a fine-tune and reports membership leakage and regurgitation. This page is the sellable run: TinyLlama-1.1B-Chat, real Alpaca, LoRA r=8, one epoch, 100 inserted canaries, 200 held-out controls, budget 0.874%, model-scored high_ppl canaries.
TinyLlama-1.1B-Chat · Alpaca · LoRA r=8 · 1 epoch · 100 canaries / 200 controls · budget 0.874% · model-scored high_ppl · 20,000 host rows
The stack every fine-tuner already recognizes.
TinyLlama-1.1B-Chat-v1.0, tatsu-lab/alpaca official
text column (20,000 shuffled rows), Hugging Face
Trainer + LoRA r=8 on
q_proj,k_proj,v_proj,o_proj, one epoch, lr 2e-4.
README API: generate_canaries +
inject + callback.
n=100,
n_controls=200,
include_prob=1.0 so 100 canaries actually land,
requested family high_ppl,
actual generator model_scored_high_ppl
(TinyLlama rejection sampling into the PPL band; 300/300 in band),
reps {1, 4, 16}.
Scoring ref="auto" /
disable_adapter().
| Knob | Value |
|---|---|
| Budget | 0.874% of tokens (22,228 canary / 2,521,431 host) — ≤1% cap, host grown from 5k to 20k |
| Hardware | Apple M3 Pro, 18 GB unified, MPS · Python 3.12.11 · torch 2.7.1 · transformers 4.56.2 |
| Wall-clock | load 3 s · data 304 s · canary gen 1,921 s · train 10,685 s · audit 829 s · wall 3 h 35 min (22:19–01:55 IST) |
| Method | base_calibrated_min_k_plus_plus · reference.mode=disable_adapter |
| Report | schema 1.2.0 · tool 0.1.0 · examples/alpaca-powered-report.json |
How to read this, then the numbers.
Membership asks whether an attacker with logprobs can tell a canary was trained on. Regurgitation asks whether the model emits the secret from a prefix. They routinely disagree. A one-epoch LoRA (r=8) of TinyLlama-1.1B-Chat on 20,000 real Stanford Alpaca rows, with an honest 0.874% canary budget and model-scored high-perplexity canaries, did leak membership at the pre-declared operating point: 10 of 100 inserted canaries were detectable at 1% FPR (TPR 0.100, 95% CI [0.049, 0.176]). Ranking agreed — AUC 0.837. The same threshold decomposes as 1x 0/34, 4x 1/33, 16x 9/33: the pooled 10% is substantially a duplication/exposure stress signal, not a 10% detection probability for a single-exposure record. The same run was 0/100 under this prefix/decoding/exact-match protocol (not a claim of no extraction risk).
Limitations: 1.1B, one epoch, 20,000 rows, max_length 256,
laptop MPS. Thresholds are calibrated on this run's held-out controls. This is evidence of resistance
(or not) to the attacks actually run. It is not a GDPR / AI Act / CNIL determination.
pip install "memaudit[peft,trl]" python examples/alpaca_case_study.py
Prior powered run, labeled appendix only.
Measured 2026-08-28 on the same machine. Requested high_ppl
without passing model= at generation; with no model and no corpus,
generation used the uniform_vocab fallback.
TPR@1%FPR 0.180 (18/100), CI [0.110, 0.269], AUC 0.776, tier curve
1x 0/34 · 4x 2/33 · 16x 16/33, budget 0.907%, regurgitation 0/100.
Report: examples/alpaca-powered-report-v0.1-uniformvocab.json.
The n=12 run is a first look, not the headline.
Same stack, 5,000 Alpaca rows, README default inclusion coin 0.5 → 12 of 32 candidates inserted,
120 held-out. TPR@1%FPR 0.500 (6/12), CI [0.211, 0.789], AUC 0.880, regurgitation 0/12.
The interval is too wide to sell. Report:
examples/alpaca-case-study-report.json.
Already-measured distilgpt2 n=100/200: TPR 0.000 [0.000, 0.036], risky-config AUC 0.848, regurgitation 0/100. Those CIs are sellable and stay on the landing-page table.