SYNTHESIS NOTE
Psychology, Society, and Alignment Language, Text, and Discourse

How much worse is misuse risk from open foundation models?

Can we measure whether open foundation models actually increase misuse risk beyond what bad actors could already accomplish with existing technology? Current research hasn't adequately answered this question across cyber, biotech, and information warfare domains.

Synthesis note · 2026-06-03 · sourced from Alignment

The open-vs-closed release debate is heated and under-evidenced. This position paper clarifies it by defining open foundation models (broadly available weights — Llama 2, Stable Diffusion XL) via five distinctive properties (greater customizability, deeper inspectability, poor monitoring, etc.) that drive both their benefits (innovation, competition, distributed decision-making power, transparency) and risks. Its analytical contribution is a marginal-risk framework: assess misuse not in absolute terms but relative to pre-existing technology (search engines, prior models). Applying it across vectors (cyberattacks, bioweapons, disinformation), it finds current research insufficient to characterize the marginal risk — and shows that past disagreements stem from focusing on different parts of the framework under different assumptions.

The keeper is the marginal reframing: the policy question is not "could an open model help a bad actor?" but "how much does it help beyond what they could already do?" — and on that question the evidence is mostly missing, which is itself the finding.

This is a discourse/governance anchor for the vault. It complements the empirical risk register of Where do frontier AI models actually pose the greatest risk today? — both insist on measured marginal risk over speculation — and informs the open-weights side of the alignment-and-society conversation.

Inquiring lines that use this note as a source 3

This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.

Related concepts in this collection 2

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
12 direct connections · 93 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

open foundation models need a marginal-risk framework because current evidence cannot characterize their misuse risk relative to existing technology