Wet Oaf

know what you like and what you thrive on

2026-09-23T00:00:00-07:00

Denouement

The other day I was preparing for a windsurfing test, one I wasn't totally confident I would pass. A couple days before I thought to myself "why not take it easy for a day, and instead of practicing at the very limit of my abilities why not remind myself why I like this in the first place?"

it was a disaster

I went up to a larger size board hoping for a relaxed and easy time. Instead, every fall felt personal and every success was mundane and unrewarding. Of course I can tack on a 190 liter board. I can tack on a 160! (most of the time). By making things easier(physically) I made them harder for myself(emotionally).

I think basically I have set up most of the hobbies that I find really rewarding as like, not quite type 2 fun but perhaps type 1.8 fun. Windsurfing is mostly uphauling over and over in swells you can't quite balance on and a single moment of transcendent skipping over water or actually gybing without falling perhaps every hour.

I'm not sure what point I'm even trying to make here. "If you ain't falling you ain't learning" and the real joy of learning something new is in learning, perhaps? If you like type 2 fun don't try and have type 1 fun it won't work? If you're frustrated with something, have you tried making it harder?

moral

Perhaps in five years I'll have psychically digested this into something worth saying. Enjoy this wisdom cud. Or don't, I'm not your mom.

PS

I passed easily, I was worried for nothing.

Anthropic September Dual-Use

2026-09-11T00:00:00-07:00

well i guess i gotta

We're talking about the September 2026 anthropic report on dual-use biology stuff today.

tl;dr it's not that much stuff, considering

Case 1: Chungunyika gain-of-function

Probably the most lurid, this is explicit gain-of-function chungyunika virus research conducted at a military lab in a (reading between the lines) sanctioned country. People do do this kind of thing "normally", but it is both difficult and dangerous and of all the cases this is the most easily blocked, and in fact it was.

Fun hit: the infrastructure to route biosafety queries claude wouldn't answer to other more-permissive models was itself developed with claude. Really the big ai cloud providers have an absolutely unprecedented level of visibility into what people at companies are doing. (echoes of the reveal that the houthis in saudi arabia were trying to develop guided missile technology using Claude. the prototype crashed, or at least the account asked 'why did our missile crash' a couple days later)

Case 2: Bird flu gain-of-function

This is also relatively normal-ish bird flu biology research that was completely delegated from opus to haiku. I guess there's exactly one guy out there who is using Haiku to write their entire study design, grant, data analysis, and write up the article at the end? A task for which it was not especially good at (by design i suppose, that's what the safeguards are supposed to do). This is a live research program involving several prompts over multiple days.

Case 3: orthopoxvirus gene deletion

This one is kind of interesting. Running a viral gene knockout study to test which genes affect host immune evasion is exactly what you would do to hunt for new antiviral targets if you were trying to come up with new drugs to treat orthopoxviruses(monkeypox, smallpox, etc). This is someone who cleverly anonymized their access and then one-shot a grant proposal and never came back. I wonder if it was ever submitted, never mind performed.

Fun hit: anthropic says this is "equally useful to someone seeking to attenuate a virus and to someone seeking to preserve, enhance, or transfer that function in others". a) I don't agree b) neither does the biosafety classfier, which simply wrote the whole proposal in Opus.

Case 4+5: Toxins, Proteins, etc

Case 4 looks like a totally standard bioinformatics research project to me. Protein toxins are peptides after all, and they're a huge and new drug class for a reason. It doesn't look like anything was actually synthesized and this research looks completely legitimate to me (and indeed it is, or was, state-sponsored health research.)

Case 5 is someone who was using Claude just to write up quarterly progress reports without mentioning the exact research they were working on, which at the very least shows awareness that every prompt you send up there is not private. This is in almost no sense biology-related as a task.

reading between the lines a bit

All of these biorisk cases were banned for restricted-region evasion, and the only really scary one (case study 1) immediately re-established access and increased the sophistication of their workarounds. An uncharitable take might be that the biorisk safeguards were entirely worthless this year, the only questionable research being performed this year was being done by a sophsticated enough actor to simply bypass the safeguards(at least in part). Given the five case studies we have here I think they were probably chosen specifically because they involve ToS violations, specifically to avoid "anthropic detects sketchy biorisk prompting, does not ban user" as a takeaway. Much of the biorisk safeguard activations are false positives, after all, including at least four of these. Maybe all five depending on whose military bases are hosting pathogen research.

So one lesson might be that people are not trying to do sketchy biorisk things at all. In august Anthropic did significantly relax their biosafety safeguards in August of this year, so perhaps they agree. Perhaps all the sketchy biosafety things are happening in supported customer countries and we are simply not hearing about them (unlikely).

P.S

I find the new york times coverage of this irresponsibly breathless, which is unsurprising.

We Should Be Doing More Dumb Tests

2026-09-08T00:00:00-07:00

What are the big ai labs getting wrong about biosafety

Basically all of the big frontier AI labs are semi-intentionally not testing for certain biosafety capabilities because they know the models fail at the tasks, and that makes everything a little more dangerous.

What are the big ai labs doing?

The big AI labs are making a big deal about model biosafety, and running really quite extreme safeguards on their models. If you are a microbiologist and you wish to use claude to organize your email, the autorouter will fall back from Fable to Opus. (or it used to)

This might lead you to believe that there are well-understood risks, since Anthropic is leaving money on the table about it. Not so.

These safeguards have been implemented on the strengths of things like good performance on the Virology Capabilities Test (multiple choice very difficult virology exam) and a decrease in the number of 'biodefense expert'-judged "critical failures" on a text plan to create a bioweapon created over multiple days. 2024 wet lab uplift trials(an uplift trial is where you give someone a tool and measure how much better they are at the task with it) ('the task' for these trials are like 'doing a bacterial transformation' 'IPTG induction' etc) produced no uplift. People outside frontier labs are running their own tests, but we'll get to those.

LLMs perform well on other tasks. AI can generate bacteriophage genomes and a small percentage are actually virulent! Models can design viral capsid proteins and are reasonably capable of evading DNA synthesis company safeguards. These are all impressive capabilities.

The AI is very good at homework, but has not been shown to be dangerous in the lab.

Why are they doing that?

This approach is very understandable! Homework is easy to grade. Actually attempting to make things in the lab is expensive.

Furthermore, there's something you might have missed: basically all of these safeguards are focused around preventing 1) human 2) viral 3) pathogens. Someone designs and then synthesizes a viral pathogen 'from scratch' and then releases it. It's called the Virology capabilities test!

Why is that bad, then, smart guy

This narrow view of what 'biorisk' is means two things: 1) Nothing even resembling the actual task will ever be attempted (good) 2) whether or not the model is actually capable of the bad thing is still almost completely unknown. Any unaligned model can generate a plausible-looking flu genome that might be very dangerous. Actually evaluating whether it is is almost impossible! (well, there's one way. but that would be insane.)

This focus on models of biorisk that are inherently untestable has meant that there is now significant uncertainty about the real capabilities of these models on other tasks. Part of that is probably because running the wet lab uplift tests might show that novice uplift is nothing special, and AI companies are not in the habit of releasing reassuring biorisk results.

A brief aside about what other people are doing

SecureBio and other groups in the biosafety risk space are running their own trials. The pattern from above(good at homework, bad at novice uplift) holds. 2025 wet lab uplift trials produced negative uplift(statistically insignificant). LANL ran a pilot study that produced positive(if insignificant) uplift. There are quite a few experiments showing quite good performance in 'optimize this cloning protocol' or similar tasks in the reverse-centaur mode (model proposes what is to be done, skilled humans do it). These are mostly interesting since these tests are not standardized (there are only a handful) and rarely conducted pre-release. Are models even getting better at novice uplift? Hard to say.

What we should actually be afraid of instead of Scary Virus

If we expand our imagination beyond 'someone downloads sars 3' there are a huge number of bad things that might be possible with todays' models.

Claudemeal but real (sourdough starter that produces GHB). Pickles that produce dioxins. Orange trees that produce botulinum toxin in the peels. Dose a city rat with a gene drive so that they're all fluorescent in a few years, or sterile, or secrete something interesting in their saliva. Agriterror that involves releasing pollen to cross-contaminate the genetics of seed corn. Engineering pesticide resistant ragweed. Even staying within traditional bioweapons for a moment there are very few antifungals. There are really quite a few possibilities if we look beyond (virus AND infects-humans AND deadly).

Now I think that it's quite unlikely LLMs will change the way biotech is done as aggressively as they have changed how software is done. It is now sort of feasible to write your own operating system, or decompile an entire AAA video game, or a hundred other things that would not have been practical to do as one person five years ago. What biological things are possible now or will be possible soon?

This is where we circle back to the limited biosafety tests done before model release. Frankly if an LLM can order dna that codes for certain viral capsids from dna synthesis companies, who cares? If how_to_acquire_bioweapon.txt is judged to be free of 'critical failures' by a biodefense expert, that's actually still quite different from it actually working. Biodefense experts are not red-teaming with live bioweapons to find out how difficult it is. (I hope not, anyway.) Models that fail the tests(i.e. have the capabilities) might be safe to release, and models that pass the tests might still be dangerous in other ways.

so what should be done then

Repeated actual wet lab uplift tests with a standardized task, done well before model release. Don't ask them to end the world, ask them to make a gram of insulin. These tests would be cheap(ish) to run. Not as cheap as capsid folding experiments, but pretty cheap, and pretty safe. And if random undergraduates can reliably make a gram of anything they want, well. That tells you something concrete.

The status quo is that wet lab uplift trials are sporadic, not organized, conducted after models are released, and rarely repeat the same task with different models. As long as the uplift observed continues to be very small, this is fine. One day it may suddenly stop being fine, and we won't know until the model is already released.

so what do you want out of it

I think the status quo for biosafety is too brittle. The tests being done on models pre-release are poor measures of what we are afraid of, and the risks are high. Only frontier labs can do these tests safely. If the models are released to be tested by other groups we will all find out about their new capabilities together, from the newspapers.

We are sleepwalking toward a pretty significant biorisk. Not an existential risk, but significant ones. Personally I estimate that we have quite a bit of runway, but we really should get in the habit of checking where we are going before we get there. I want to live in the good future, and that means more realistic tests of lower-stakes bad outcomes that can be compared between models, even (especially) if the wet lab uplift tests consistently show that the models are not actually that powerful.

New models cost several billion dollars to train. I don't think it's unreasonable to round up 24 college freshmen for a few days once or twice a year. You probably wouldn't even have to pay them.

P.S.

People are designing and synthesizing new pharmaceuticals in their garages and testing them on rats. Is this guy deluded about how successful he's being? Why is he succeeding where uplift trials have failed? Interesting questions.