VaspHandler — DFT calculations across a lab of unreliable machines

VASP is the workhorse of solid-state physics — you give it a crystal and it gives you energies, band structures, forces. It is also slow, and the computers a materials lab actually owns are a dozen desktops of mixed age under different desks, switched off by whoever leaves last. VaspHandler is what I wrote so that this pile of hardware behaves like one machine.

Each computer runs a small daemon. Anyone in the lab submits a job from a command line or from a page on this site; the job is broken into leased tasks, the tasks go to whichever machines are awake, results are copied to more than one disk, and the whole state is mirrored to a website so you can see what is running without walking into the building. A calculation that is interrupted picks itself up from its own last written geometry rather than starting again. The lab is installed from a pendrive that carries its own Python, because none of the machines can be trusted to have one.

No election

The first several versions used Raft. Every machine was a council seat, and the council elected an admin to run the queue. This is the correct textbook answer, and it was wrong for this lab.

A five-seat council needs three machines awake. This lab is a dozen machines that people switch off on their way home. Losing the majority meant losing the lab until somebody drove in the next morning — and nobody had done anything wrong. It cost days, twice, before I accepted that the protocol was solving a problem the lab did not have and creating one it did.

So now each machine is given a rank at install time, and the admin is simply the reachable machine with the best rank. No vote, no quorum. One machine on its own runs the lab. What this gives up is written down instead of glossed over: during a network partition two machines can briefly both believe they are the admin. That is affordable only because of properties the system already had — tasks are leased and idempotent, results are content-addressed, an interrupted calculation resumes from its own files — so a split costs some duplicated work, while a lost quorum cost a fortnight. The docstring on the leadership module carries the full argument, including the conditions under which it would stop holding.

Removing the council also cut the test suite from a hundred seconds to thirty, which was a consequence rather than an optimisation: most of the old runtime was tests waiting for real elections.

Tested against a lab that does not exist

Alongside pytest there is labsim: a simulated lab of real daemons with a fake VASP and a power switch. It can switch machines off mid-task, partition them, bring them back — everything the real lab does by accident, on demand. Most of the important bugs were found there before they could be found in the building.

Not all of them. One release died at startup on every real machine because the first branch a lab machine takes — asking the network what version to run — was the one branch every test disabled. That taught me to run a static checker inside the suite as a test that fails, and to always start the installed daemon from the installed interpreter before calling a release built.

One file for the present tense

Six versions in a row shipped with a document claiming something that had stopped being true, and twice a false claim decided how somebody spent their day. The cause was arithmetic: the same fact lived in a README, a handover, four typeset documents and their copies on the stick. Eight places, updated by hand.

Present-tense facts now live in one file, and a test fails the suite when that file and the repository disagree. It is the single rule from this project I have carried into every project since.