recipe for ruin

Nielsen's notes on ASI xrisk introduced the thought experiment: If you ask an all-knowing oracle a question like

"Can you give me a simple, easy-to-follow recipe, where with $10,000, a month's work, and without arousing any security concerns, a bright teenager can build a device that will destroy the entire world?"

or even

"Is there an arrangement of 1,000 protons, neutrons, and electrons that, if created, would destroy the world?"

what would it be able to come up with? This separates the question of the threats that we might in principle worry about, from the question of how intelligent an ASI might be and whether it would be able to find them in practice.

This is related to the question of whether the universe favors offense or defense, at a deep level. If sufficiently powerful offensive weapons exist at the tips of the tech tree --- or even just at points above us on the tech tree before the discovery of corresponding defenses --- then we should be extremely cautious about continuing to climb the tree.

Note that nuclear weapons are close to an answer to both these questions --- a simple arrangement of matter that can be enormously destructive and is hard to defend against. It so happens that they're not cheap to construct, so we've mostly prevented proliferation, but all it would take is a simple procedure for enriching uranium in your garage, and suitcase nukes become a huge problem.

Killer viruses are also a possible answer. It is not clear whether the landscape here favors offense or defense (are there viruses that can't be effectively vaccinated against? can we get off a vaccination campaign in time to matter?).