What are the big ai labs getting wrong about biosafety

Basically all of the big frontier AI labs are semi-intentionally not testing for certain biosafety capabilities because they know the models fail at the tasks, and that makes everything a little more dangerous.

What are the big ai labs doing?

The big AI labs are making a big deal about model biosafety, and running really quite extreme safeguards on their models. If you are a microbiologist and you wish to use claude to organize your email, the autorouter will fall back from Fable to Opus. (or it used to)

This might lead you to believe that there are well-understood risks, since Anthropic is leaving money on the table about it. Not so.

These safeguards have been implemented on the strengths of things like good performance on the Virology Capabilities Test (multiple choice very difficult virology exam) and a decrease in the number of 'biodefense expert'-judged "critical failures" on a text plan to create a bioweapon created over multiple days. 2024 wet lab uplift trials(an uplift trial is where you give someone a tool and measure how much better they are at the task with it) ('the task' for these trials are like 'doing a bacterial transformation' 'IPTG induction' etc) produced no uplift. People outside frontier labs are running their own tests, but we'll get to those.

LLMs perform well on other tasks. AI can generate bacteriophage genomes and a small percentage are actually virulent! Models can design viral capsid proteins and are reasonably capable of evading DNA synthesis company safeguards. These are all impressive capabilities.

The AI is very good at homework, but has not been shown to be dangerous in the lab.

Why are they doing that?

This approach is very understandable! Homework is easy to grade. Actually attempting to make things in the lab is expensive.

Furthermore, there's something you might have missed: basically all of these safeguards are focused around preventing 1) human 2) viral 3) pathogens. Someone designs and then synthesizes a viral pathogen 'from scratch' and then releases it. It's called the Virology capabilities test!

Why is that bad, then, smart guy

This narrow view of what 'biorisk' is means two things: 1) Nothing even resembling the actual task will ever be attempted (good) 2) whether or not the model is actually capable of the bad thing is still almost completely unknown. Any unaligned model can generate a plausible-looking flu genome that might be very dangerous. Actually evaluating whether it is is almost impossible! (well, there's one way. but that would be insane.)

This focus on models of biorisk that are inherently untestable has meant that there is now significant uncertainty about the real capabilities of these models on other tasks. Part of that is probably because running the wet lab uplift tests might show that novice uplift is nothing special, and AI companies are not in the habit of releasing reassuring biorisk results.

A brief aside about what other people are doing

SecureBio and other groups in the biosafety risk space are running their own trials. The pattern from above(good at homework, bad at novice uplift) holds. 2025 wet lab uplift trials produced negative uplift(statistically insignificant). LANL ran a pilot study that produced positive(if insignificant) uplift. There are quite a few experiments showing quite good performance in 'optimize this cloning protocol' or similar tasks in the reverse-centaur mode (model proposes what is to be done, skilled humans do it). These are mostly interesting since these tests are not standardized (there are only a handful) and rarely conducted pre-release. Are models even getting better at novice uplift? Hard to say.

What we should actually be afraid of instead of Scary Virus

If we expand our imagination beyond 'someone downloads sars 3' there are a huge number of bad things that might be possible with todays' models.

Claudemeal but real (sourdough starter that produces GHB). Pickles that produce dioxins. Orange trees that produce botulinum toxin in the peels. Dose a city rat with a gene drive so that they're all fluorescent in a few years, or sterile, or secrete something interesting in their saliva. Agriterror that involves releasing pollen to cross-contaminate the genetics of seed corn. Engineering pesticide resistant ragweed. Even staying within traditional bioweapons for a moment there are very few antifungals. There are really quite a few possibilities if we look beyond (virus AND infects-humans AND deadly).

Now I think that it's quite unlikely LLMs will change the way biotech is done as aggressively as they have changed how software is done. It is now sort of feasible to write your own operating system, or decompile an entire AAA video game, or a hundred other things that would not have been practical to do as one person five years ago. What biological things are possible now or will be possible soon?

This is where we circle back to the limited biosafety tests done before model release. Frankly if an LLM can order dna that codes for certain viral capsids from dna synthesis companies, who cares? If how_to_acquire_bioweapon.txt is judged to be free of 'critical failures' by a biodefense expert, that's actually still quite different from it actually working. Biodefense experts are not red-teaming with live bioweapons to find out how difficult it is. (I hope not, anyway.) Models that fail the tests(i.e. have the capabilities) might be safe to release, and models that pass the tests might still be dangerous in other ways.

so what should be done then

Repeated actual wet lab uplift tests with a standardized task, done well before model release. Don't ask them to end the world, ask them to make a gram of insulin. These tests would be cheap(ish) to run. Not as cheap as capsid folding experiments, but pretty cheap, and pretty safe. And if random undergraduates can reliably make a gram of anything they want, well. That tells you something concrete.

The status quo is that wet lab uplift trials are sporadic, not organized, conducted after models are released, and rarely repeat the same task with different models. As long as the uplift observed continues to be very small, this is fine. One day it may suddenly stop being fine, and we won't know until the model is already released.

so what do you want out of it

I think the status quo for biosafety is too brittle. The tests being done on models pre-release are poor measures of what we are afraid of, and the risks are high. Only frontier labs can do these tests safely. If the models are released to be tested by other groups we will all find out about their new capabilities together, from the newspapers.

We are sleepwalking toward a pretty significant biorisk. Not an existential risk, but significant ones. Personally I estimate that we have quite a bit of runway, but we really should get in the habit of checking where we are going before we get there. I want to live in the good future, and that means more realistic tests of lower-stakes bad outcomes that can be compared between models, even (especially) if the wet lab uplift tests consistently show that the models are not actually that powerful.

New models cost several billion dollars to train. I don't think it's unreasonable to round up 24 college freshmen for a few days once or twice a year. You probably wouldn't even have to pay them.

P.S.

People are designing and synthesizing new pharmaceuticals in their garages and testing them on rats. Is this guy deluded about how successful he's being? Why is he succeeding where uplift trials have failed? Interesting questions.