Yesterday’s post was about axes. This one is about a small part of a molecule that refuses to stay put: the methyl group. It took two days, five calculations and rules written down in advance before we knew how to treat it. This post explains what a methyl group does, why our usual method struggles with it, and what we decided.
Many of the molecules in this project are flat rings of carbon atoms. Sometimes one hydrogen on the ring is replaced by a methyl group: one carbon atom carrying three hydrogen atoms, written CH₃. Seen from the ring, the methyl group looks like a small propeller with three blades.
That propeller can turn. As it turns, each hydrogen passes close to the atoms next to it on the ring, which pushes back a little. So the propeller has three comfortable positions per full turn, with a small hill between each pair of them. The height of that hill is called the barrier. A methyl group with a high barrier mostly rocks back and forth in one valley. A methyl group with almost no barrier spins nearly freely.
Our test molecule is 4-methylimidazole, a five-membered ring with two nitrogen atoms and one methyl group. Its barrier is low compared with a chemical bond, but high compared with the energy needed to set the propeller rocking. In the language of the field it is a hindered rotor: it rocks, and only occasionally hops over a hill.
To predict the positions of the bands in an infrared spectrum, we treat every vibration as a spring. Each normal mode is one pattern of motion in which all atoms move together. The first guess, the harmonic one, treats each spring as ideal. Real springs are not ideal: they stiffen or soften as they stretch, and they pull on each other. That is anharmonicity, and it shifts each band by a few to tens of wavenumbers (cm⁻¹, the unit of the spectrum).
Our method for these shifts is VPT2. It starts from the springs and adds small corrections. Its central assumption is that every atom makes small motions around one resting position. For a ring that holds well. For the propeller it does not: turning the methyl group is a large motion, around a circle, with three resting positions instead of one.
The propeller also touches other vibrations. The three hydrogens can fold together and apart like an umbrella; these are the CH₃ bends, two bands near 1508 and 1521 cm⁻¹, a wavelength of about 6.6 micrometres, inside the part of the spectrum we predict. When the propeller turns, those bends change a little. VPT2 sees this as a coupling between the turning and the bending, and it shifts the bends for it. The question of this post is whether that shift is real.
We compute some of VPT2’s numbers twice, by two independent routes. If both routes give the same number, it is probably right. If they disagree, something in the calculation is noisy. Two of the molecules in our study of these shifts, both with a methyl group, had a disagreement of 5.6 and 7.2 cm⁻¹, far above what we accept.
A disagreement like that has two possible causes. The first is numerical noise from the integration grid, the set of points on which the program evaluates its formulas. A finer grid removes that kind of noise. The second is that the motion is too large for the method. A finer grid does not help then.
On 3 November we repeated the calculation for 4-methylimidazole on a finer grid. We had written down beforehand how to tell the two causes apart: if the disagreement fell to half or less, it was grid noise; if three quarters or more remained, it was the large motion. For a similar case in October, a vinyl group on styrene, only 0.19 of the disagreement had remained: grid noise. For the methyl group 0.93 remained. The worst pairs were all the propeller with the methyl group’s own C–H bonds. The verdict: the propeller’s large motion, not the grid.
That left a practical question. A common remedy in the literature is to take the turning motion out of VPT2 altogether, so that it cannot shift anything else. Doing that moved the CH₃ bends by up to 5.9 cm⁻¹. That is about twice the accuracy we promise per family of bands (3 cm⁻¹). So the choice mattered, and we did not know which choice was right.
A literature search on 4 November gave a reason to doubt the remedy. Atomic motion can be described in two ways. One way uses straight-line displacements of every atom along x, y and z. The other uses bond lengths, bond angles and twist angles. A turning propeller is naturally an angle: in straight lines, its hydrogens would leave their circle. A 2022 study on acetic acid (Mendolicchio, Bloino and Barone) found that in angle-based coordinates the couplings of a methyl group become 10 to 100 times smaller, while the bands themselves move by only 0 to 4 cm⁻¹. That suggested the large couplings might be an artefact of straight lines that cancels in the full result.
We tested this with a second, independent program, Psience. We took the turning motion out of VPT2 in both kinds of coordinates. The bends moved by 6.5 cm⁻¹ in straight-line coordinates and by 7.5 cm⁻¹ in angle coordinates. Both descriptions said the same thing, so this test did not decide. (The same comparison also exposed an error in our own code, the subject of yesterday’s post.)
Next we measured the barrier itself. We turned the propeller in 13 steps through a third of a turn and let the rest of the molecule relax at each step. Our level of theory, B3LYP, puts the barrier at 228 cm⁻¹. A microwave measurement, which reads a molecule’s rotation very precisely, gives 317 cm⁻¹ (Gougoula and co-workers, 2019). So our method makes the hill 28 % too low. A real molecule is therefore more hindered than our model of it. The propeller’s own rocking frequency follows from the same scan: 91 cm⁻¹, far below the window of the spectrum we predict.
VPT2 is a shortcut: it replaces the exact problem by springs plus small corrections. For three motions only, the propeller and the two bends, we can avoid the shortcut and solve the problem directly. This is called a variational calculation.
It works like this. We place the atoms at many combinations of three settings: the propeller’s angle, and how far each of the two bends is folded. At every combination we compute the energy. Together those energies form a map of the landscape the atoms move in. Then we solve the quantum mechanics of the three motions on that map, without assuming small motions. To isolate the coupling, we solve it twice: once on the real map, and once on a map from which the link between turning and bending has been removed. The difference in the bend’s band is the true effect of the coupling. Then we compare it with what VPT2 says.
Before computing anything, we wrote down three rules:
The first map had 24 propeller angles, 15 degrees apart, and 11 settings for each bend: 2,904 energies. It was not converged. Halving the angles moved the answer by 0.97 cm⁻¹, more than three times the allowed 0.3. So there was no verdict.
We then doubled the angles to 48, 7.5 degrees apart. Every second angle of the new map was an angle of the old one, so we reused those 2,904 energies and computed 2,904 new ones, about three and a half hours on the laptop. This time the map was converged: halving the angles moved the answer by 0.03 cm⁻¹.
| Variational | VPT2, straight lines | VPT2, angles | |
|---|---|---|---|
| Bend near 1508 cm⁻¹ | +1.4 | +6.5 | +7.5 |
| Bend near 1521 cm⁻¹ | +0.1 | +3.3 | +4.4 |
The shift of each CH₃ bend caused by its coupling to the turning propeller, in cm⁻¹.
Rule 2 fails: 1.4 and 6.5 are far apart. Rule 3 holds: 1.4 is under 1.5, and both VPT2 values are above 3. So by the rule written in advance, the coupling VPT2 puts between turning and bending is not really there for this molecule.
We wrote three limits down with the verdict.
The project now has a rule for hindered methyl groups. The CH₃ bends get their anharmonic shift from VPT2 with the turning motion taken out. The propeller’s own rocking band comes from the scan, not from VPT2. The methyl group’s C–H stretches, near 3 micrometres, lie outside the window we predict and stay as they are.
The rule says how to compute a methyl band. It does not yet say how far such a band can be trusted. The certificate that comes with every predicted spectrum will keep marking methyl bands as outside the measured accuracy, until a test on more than one molecule, again registered in advance, gives them an error bar of their own. A nearly free methyl group, like the one on toluene, has an even lower hill and is not covered by the rule at all.
The code for these calculations was written and tested with an AI assistant, Claude, under my direction. The variational read is in commit 792304b2, and the rule in 9e0387ee.
]]>A molecule floating in space does not care which way it faces. Turn it a quarter turn, or upside down, and it is still the same molecule with the same energy and the same infrared spectrum. A computer program, however, has to write the positions of the atoms down as numbers along three axes, x, y and z, so it must choose which way those axes point. A good program gives the same answer whatever it chooses.
This week we found two places where that went wrong. The first was in Psi4, someone else’s program. The second was in our own code. Neither of them changed a result of this project. Both are worth telling, because they show how a small choice about axes can hide in a calculation for weeks.
Our cheaper calculations use density functional theory, DFT for short. DFT finds a molecule’s energy by adding up a formula over all of the space around the atoms. A computer cannot add up infinitely many points, so it picks a finite set of points around every atom and adds up the formula only there. That set of points is called the integration grid.
You can compare this with estimating the area of a pond. You lay a net of evenly spaced pins over the map and count how many pins fall in the water. The answer is close, but not exact, and it changes slightly if you shift or turn the net. A DFT grid has the same kind of small error, and that error also changes slightly when the grid turns against the atoms.
The makers of Psi4 wanted the energy to stay the same when a user turns a molecule. Their solution: before calculating, Psi4 lines its grid up with the molecule itself. If the molecule turns, the grid turns along with it, so the pins land on the same spots and the small error stays the same.
To do that, Psi4 needs a direction that belongs to the molecule. It uses what we can call the molecule’s charge axes. Imagine every atomic nucleus as a small weight, heavier the more charge it has. Every object made of weights has three natural axes, the way a book has a long side, a short side and a thickness. Psi4 points its grid along those three axes.
Benzene is a flat ring of six carbon atoms, a regular hexagon. Within its plane it is almost round. Its two in-plane axes are nearly equal: in our numbers 189.944 against 189.953, a difference of less than one part in ten thousand. For a book that would mean the long side and the short side are the same length. Which direction counts as “long” is then decided by a hair.
A molecule vibrates. In each normal mode, all atoms move together in a fixed pattern. When benzene moves a little along one of these patterns, the hair changes and the charge axes can swing round by several degrees, while the atoms themselves have hardly moved. For vibration number 5, a small push turns the axes by 6.6 degrees one way and 5.2 degrees the other way. The grid turns with them. For vibration number 10, the same push turns the axes by only 0.07 degrees.
A vibration behaves like a spring. A spectrum needs two things from that spring: how stiff it is, and how its stiffness changes when the atoms move further apart. The second part is called the fourth-power term. It is small, but it shifts the bands of the spectrum.
There are two ways to get these numbers. The direct way, analytic derivatives, lets the program calculate the stiffness from its formulas in one go. The indirect way, finite differences, pushes the atoms a little along the vibration, calculates the energy at each step, and fits a curve through those energies. The project mostly uses the direct way. On 28 October we used the indirect way on purpose, as a second route to check benzene’s numbers, once in PySCF and once in Psi4.
The indirect way is very sensitive to bumps in the curve. If the grid turns between two steps, each energy carries a slightly different grid error, and the curve gets small bumps that have nothing to do with the molecule. For vibration number 5 of benzene, Psi4’s curve gave a fourth-power term of 849.7, where PySCF, another program, gave 265.0 (both in wavenumbers, cm⁻¹, the unit of the spectrum).
On 28 October we had a suspicion, not a proof: the order matched. The vibrations whose axes turned most were the ones where Psi4 missed most, and vibration 10, whose axes hardly turned, was the one where Psi4 agreed with PySCF.
On 3 November we tested the suspicion inside one program, so that nothing else could differ. PySCF normally keeps its grid fixed in space. We made PySCF calculate the same 25 energies twice. The first time it used its normal fixed grid. The second time we first turned each geometry onto its own charge axes, the way Psi4 does. Turning a whole molecule does not change its true energy, so any difference between the two runs had to come from the grid.
Before running, we wrote down what would count as proof: vibrations 5 and 11 should change a lot, vibration 10 should hardly change, and vibration 5 should change more than 11, because its axes turn further. The results:
| Vibration | Axes turn by | Fourth-power term, fixed grid | Fourth-power term, grid on charge axes |
|---|---|---|---|
| 5 | 6.6° / 5.2° | −3.5 | 1,092.8 |
| 11 | 3.2° / 3.3° | 12.4 | 132.2 |
| 10 | 0.07° | 132.3 | 132.5 |
All three rules held. The same program, with the same method and the same grid, gives a very different curve once its grid turns with the charge axes. Psi4’s misses on benzene have the size and the direction you expect from this effect alone.
Psi4’s energies are not wrong in the usual sense. The turning trick was meant to remove a small error, and for most molecules it does. For almost round molecules such as benzene, the trick makes the error jump from one step to the next instead.
For our project this was already handled, for two reasons. First, the project’s first collection of molecules was calculated in Psi4 by finite differences, so those numbers could carry this effect. Since then most molecules that train our network get their stiffness from PySCF by the direct way, without steps. That is true for 750 of the 972 molecules in the training set. The last 222 are queued on a server. Second, where we do need steps, for the fourth-power terms, PySCF calculates the full stiffness directly at each pushed geometry, and PySCF keeps its grid fixed in space, so the grid does not turn between the steps. We have not reported this to Psi4 yet. Our evidence is indirect: Psi4 has no switch to turn the orientation off, so we could only copy its behaviour in another program.
A molecule does not only vibrate. It also tumbles through space. When atoms move while the molecule rotates, they feel a sideways push. This push is the same Coriolis effect that makes a ball rolled across a turning merry-go-round seem to curve. In a molecule, that push couples the vibrations to each other a little, and it shifts the bands by a small amount.
Our method for these shifts is called VPT2. It corrects the “ideal spring” picture of a vibration for the ways real bonds stiffen and soften. One of its terms is the Coriolis term. That term multiplies two ingredients for each of the three axes:
Both ingredients must refer to the same axis. Otherwise the term multiplies the coupling around one axis with the spin around another.
Our code calculated the couplings around the x, y and z axes of the file the molecule happened to be stored in. It took the rotational constants around the molecule’s own axes, sorted from the easiest spin to the hardest. These two sets of axes are the same only when the stored molecule already lies along its own axes, in exactly that order.
The code had been like this since 21 September. Our only test of the Coriolis term used data from Psi4, and Psi4 always turns a molecule onto its own axes before saving it. So in the one test we had, the molecule lay exactly right, and the mistake could not show. Psi4’s habit of turning molecules, the subject of Part 1, had hidden our own error.
On 4 November we ran our own calculations through a second, independent program, Psience (version 1.6.1), to check something else. For water, two of the three bands differed from Psience by 23.0 cm⁻¹. That is far too large to be rounding, and it led straight to the mixed-up axes.
The size of the Coriolis term depends on the rotational constant, and that constant depends on the molecule’s size. Water is small and light, and it spins easily, so the term is large. The molecules of this project are big rings of carbon, and they turn slowly, so the term is small. A wrong pairing of axes therefore mattered a lot for water, and very little for our molecules. For 4-methylimidazole (12 atoms) the bands moved by at most 0.04 cm⁻¹. For our molecules of 18 to 20 atoms they moved by at most 0.005 cm⁻¹. For benzene they did not move at all.
The repaired code calculates the couplings around the same axes, in the same order, as the rotational constants. A new test turns a molecule through space and checks that every band stays the same to one millionth of a wavenumber. That is exactly the property the old code lacked. Water now agrees with Psience to within 0.012 cm⁻¹ on all three bands.
Then we redid everything the term had touched: 72 stored calculations and 28 sets of band labels, followed by every reading we had made from them, once with the old inputs and once with the new ones. The largest change in a score was 0.04 cm⁻¹. Not one verdict changed.
One band label did move by 5.5 cm⁻¹. That band is a near-even mix of two vibrations, and which of the two gets the label can flip on a tiny change. Its underlying value did not move.
Both problems were about axes. A molecule’s real answer does not depend on which way it faces, so there is a simple check: turn the molecule, and see that nothing changes. Our new test does exactly that. The pyscf experiment of Part 1 is the same check, applied to another program’s grid.
The code in both parts was written and tested with an AI assistant, Claude, under my direction, like the rest of the project’s software. The fix is in commit 648a001e, the redo in 63ab66bf, and the grid experiment in c84d4ca8.
]]>The Spectrum Atlas has a new page: the pipeline show. It follows one molecule from its name to its predicted infrared spectrum, step by step, and ends with a certificate that says how far the prediction can be trusted. This post explains what you see on that page and why each step is there.
The page has three columns. On the left you pick a molecule, sorted by its number of carbon atoms. In the middle the pipeline is drawn as a metro line with six stations, and the stations light up as the molecule passes through them. On the right the spectrum appears, with the certificate under it.
Every number on the page is read from the files of a real run. The page itself contains no numbers that someone typed in. If a run had gone wrong, the page would show the wrong result, not a nicer one.
Molecule pages in the Atlas now link to the show. Where the Atlas lists a molecule that went through the pipeline, its page has a card “Through the pipeline” that opens the show at that molecule. Phenanthrene is one example.
1. Molecule. The input is a short line of text that describes the molecule, a SMILES string. The pipeline first checks whether the molecule is one it may answer for: neutral, with paired electrons, and made only of elements the network has seen in training. That check happens before any computation, so a molecule outside the network’s range costs nothing.
2. Start geometry. A free chemistry library, RDKit, turns the text into a first three-dimensional shape.
3. Optimisation. A quantum chemistry program, psi4, moves the atoms until the forces on them vanish. This gives the molecule’s resting shape, its equilibrium geometry, with the cheap method B3LYP.
4. Hessian. A second program, PySCF, computes how stiff the molecule is in every direction, the Hessian, and how much each vibration changes the molecule’s charge distribution. The stiffness decides where the bands of the spectrum lie. The charge change decides how strong each band is.
5. The network. Here the project’s own part comes in. The neural network predicts a correction to the cheap stiffness. Three copies of the network, trained from different random starts (seeds), each give their correction, and the pipeline adds their average to the B3LYP Hessian. Where the three copies disagree, the certificate shows the spread.
6. Spectrum and certificate. From the corrected stiffness the pipeline computes the positions of the bands, and from step 4 their heights. Then it writes the certificate: for every band, which family of vibrations it belongs to, how large the network’s error was on molecules it never saw, and whether that family is licensed.
The network does not yet correct towards the expensive reference this project is about, coupled cluster. For now it corrects B3LYP towards ωB97X, a more careful DFT method that we can afford to compute for hundreds of molecules. ωB97X plays the role of the better method, a proxy, so that we can test whether the whole machine works before we pay for the real reference.
That choice also gives every molecule in the show a known right answer, because the corpus holds an ωB97X calculation for each of them. So the certificate can put the network’s spectrum next to the right answer and give the difference.
The show holds 38 molecules. None of them was used to train the network. They fall into three groups.
Eight molecules belong to the first hold-out set, hold-out (a): the molecules we set aside before training to measure the error. They have 12 to 26 atoms. Their band positions are on average 1.64 wavenumbers from the ωB97X answer (the median over the eight), and 3.22 in the worst case.
Four belong to the second hold-out set, hold-out (b), with 23 to 28 atoms. Their median is 2.39 wavenumbers, their worst 2.73.
The other 26 are in the catalogue but were never part of training or of either hold-out set. Their median is 1.48 wavenumbers, their worst 4.60, for a naphthalene with two nitro groups.
To give a sense of scale: for phenanthrene, B3LYP alone is 23.5 wavenumbers from ωB97X on average. With the network’s correction the distance is 1.2.
Thirty-four of the 38 molecules come within 3 wavenumbers on average. That average includes the bands the certificate does not license. The licence works per family of vibrations, not per molecule. A family is licensed where the network’s measured error is at most 3 wavenumbers on both hold-out sets. Three families pass: the ring vibrations in the plane, the C–H stretches and the C–H bends out of the plane. The fourth family, “other”, holds the vibrations that fit none of the three, most of them slow motions of the whole skeleton at the low end of the spectrum. It misses the line with 3.62 and 3.89. So the certificate marks those bands “not licensed”, even where they happen to be right.
Running whole molecules through the chain found three faults that tests on single parts had missed.
The order of the atoms mattered. The same molecule can be written as several different SMILES strings, and each writing numbers the atoms differently. B3LYP does not care: its bands moved by less than a billionth of a wavenumber. The network did care. For phenanthrene another writing moved single corrected bands by up to 2.9 wavenumbers. Until we find the cause, the pipeline always starts from one fixed writing, the canonical SMILES, so the same molecule always gets the same answer.
Two of our test molecules had been seen. The certificate now also says whether the network trained on the molecule. That check found that two of the ten molecules in hold-out (a) were in the training set under another name. So part of hold-out (a)’s error had been measured on molecules the network knew. Those two are now out of the training for the next version, and the show uses only the eight clean ones. Hold-out (b) was clean: none of its 39 molecules had been seen.
A flat molecule sits on a seam. In a flat molecule many angles lie at exactly half a turn, where the number jumps from plus to minus. A tiny change in the shape can flip such an angle to the other side, and the network then sees a different input. We measured how much this costs: at most 0.12 wavenumbers on the average error of hold-out (a). So it is a reproducibility flaw, not a large error.
The 38 runs in the show did not compute steps 3 and 4 on the laptop. The corpus had already done exactly those calculations at the same settings, so the pipeline takes them from there and says so on the station. That is why a run in the show takes less than three seconds. When the corpus computed steps 3 and 4, they took about an hour and a half for a typical molecule (the median) and three hours for the longest. A new molecule pays that time. A full run of phenanthrene on 30 October, with every step computed, took 76 minutes.
The page and the pipeline were written and tested with an AI assistant, like the rest of the project’s code.
The scripts, the runs and the records behind every number are in the project’s repository.
]]>On 22 September this notebook found out that benzene’s carbon–hydrogen stretches had come out forty-five to sixty wavenumbers too low, and why (The hill was not where we stood). Our expensive calculations are made at the cheap method’s resting shape of the molecule, its equilibrium geometry, not at their own. A bond that the better method would make slightly shorter is measured while it is slightly too long, like a guitar string tuned at the wrong tension. That post also found a way to fix this without moving anything: a short formula, which we called the geometry term, predicts how far each vibration shifts when the molecule relaxes to the better method’s resting shape. On benzene it worked.
That was one molecule. A rule that holds on one molecule is a hint, not a rule. This post is about the test on eighteen more.
The geometry term multiplies two things. The first is how far each atom would move if the molecule relaxed to the better method’s shape. The second is how quickly each vibration’s stiffness changes when the atoms move: a third derivative of the energy, the same numbers the anharmonic step of this project computes anyway (see Half price). Both are cheap compared with a full new calculation at the better level.
So the question had two parts. Does the formula predict the shifts on molecules other than benzene? And does it still work if we only estimate where the atoms would go, with one step from the forces at the start, instead of actually relaxing the molecule? The second part matters for the product: one step costs one gradient, relaxing costs several.
We used the eighteen molecules whose third derivatives we already had: naphthalene, azulene, styrene, quinoxaline and fourteen others with 12 to 20 atoms. The cheap method is B3LYP, the method our whole corpus starts from. In the role of the better method we used ωB97X, a more careful DFT method. Coupled cluster on eighteen molecules would have cost far more than this test is worth, and ωB97X is the same stand-in the network’s earlier tests used.
For each molecule the laptop computed ωB97X’s vibrations twice: at B3LYP’s resting shape, and again after relaxing the molecule to ωB97X’s own resting shape. The difference is the measured shift of every vibration. The formula gives the predicted shift. The laptop needed just under a day for all eighteen.
Before the first molecule ran, I wrote down four lines (the registration), counting only vibrations that move by at least 1 wavenumber:
My prediction: all four hold.
Two molecules made trouble. Styrene and its cousin with an extra carbon–carbon triple bond both carry a small CH=CH₂ group on the ring. At ωB97X’s resting shape they lie flat, but for one motion, the group twisting about the bond to the ring, flat turns out not to be the bottom of a valley. It is the top of a very low hill: the molecule would rather twist a little. A motion that sits on top of a hill does not vibrate at all, and the calculation returns an imaginary number for its frequency (41i and 43i cm⁻¹ in chemists’ notation). There is no shift to measure for it.
So before reading any result, I added a rule to the registration (the amendment): for those two molecules, that one twisting motion is left out, and every other vibration of all eighteen molecules counts. As a check, the answer is also reported without the two molecules at all, so that the leaving-out cannot have decided anything. The alternative was to let the two molecules twist into their real, slightly bent resting shape. Then most of the change between the two shapes would be one large swing along the softest motion, which a first-order formula is not meant to handle, and we would be testing something else.
All four lines hold (the read). Over 758 vibrations, the typical shift (the root mean square) is 10.99 wavenumbers. After the correction, 2.41 is left: 22 % of the shift, where the line allowed 30 %. The one-step estimate leaves exactly the same 2.41. The correction helps in all eighteen molecules. Without styrene and its cousin the picture is the same: 11.16 before, 2.39 after.
| region of the spectrum | vibrations | typical shift | left after the correction |
|---|---|---|---|
| below 1000 cm⁻¹ | 332 | 8.40 | 2.19 |
| 1000–1900 cm⁻¹ | 289 | 13.99 | 3.10 |
| 1900–2700 cm⁻¹ | 6 | 21.94 | 1.12 |
| C–H stretches, above 2700 cm⁻¹ | 131 | 8.12 | 0.36 |
The 1900–2700 row holds only six vibrations, too few to judge by the third line’s own rule; they are shown for completeness. The carbon–hydrogen stretches, the motions that started this story on benzene, almost disappear: 8.12 wavenumbers of shift, 0.36 left.
I had predicted that the shift would be largest in the carbon–hydrogen stretches, because those bonds change length the most between the two methods. It is not. The biggest shifts are in the middle of the spectrum, 1000 to 1900 wavenumbers, where the ring vibrations live. The formula handles them too, just less completely (13.99 down to 3.10). The four lines held; the reason I gave for expecting it did not.
This is a test between two DFT methods, not against coupled cluster. It shows that the formula carries from one molecule to eighteen and that the cheap one-step estimate is good enough. It does not show how well it does on the real anchors; that comes when the anchors’ own gradients are used.
In five molecules, matching each vibration at one shape to the same vibration at the other shape was uncertain: two vibrations had partly mixed. The molecule where this was worst also has the largest left-over error, 4.69 wavenumbers. So part of what is left may be a bookkeeping limit of the test rather than a failure of the formula.
And the formula is not yet in the product. It goes into the design now.
A corrected vibration is still the two things the September post described: the stiffness measured at the cheap shape, plus the geometry term. What changed today is that the second part is no longer a benzene result. For eighteen molecules it removes about four fifths of the shift, and it does that with one gradient at the better level and third derivatives the anharmonic step computes anyway.
The full numbers, the scripts and the lines written before the runs are in the project’s repository.
]]>The network in this project learns in two steps. First it learns from a large set of cheap quantum-chemistry calculations (DFT) how the vibrations of a molecule shift between two cheap methods. Then it should learn the last step from a much better and much more expensive method, coupled cluster. A coupled-cluster calculation of one whole molecule’s vibrations is what we call an anchor: a teacher’s answer the network can check itself against. We have about ten. The question this post answers is how many we need, and what they would cost.
The honest way to find out would be to train on some anchors and test on others. With ten, that test rests on one or two molecules at a time, and the answer would mostly be luck. So we answered the question on a stand-in task where we can afford hundreds of teachers.
The stand-in works like this. On the DFT data, we bend the answers the way the real coupled-cluster answers differ from DFT on the anchors we already have. The clearest pattern there is in the out-of-plane bending of the C–H bonds. Compared with the DFT difference the network learned first, the coupled-cluster correction to these bands is close to zero on a single ring, about half as large and of the opposite sign on fused rings like naphthalene, and almost as large and of the opposite sign when the molecule has a five-membered ring. The bent answers carry that pattern. Then we give the trained network the bent answers of k molecules, let it adjust, and measure its error on 343 molecules it never saw. We did this for k = 100, 120, 150 and 170 with three random picks of molecules each, and for k = 200 with the whole pool of 200, each time with three versions of the network.
The target is the one the project has used throughout: an error of at most 3 cm⁻¹ per family of vibrations.
So 167 is the number for three of the four families. The fourth is the same family the network struggles with on DFT alone, and more teachers of this kind are not the fix for it.
We expected that choosing the teachers carefully would help: make sure every kind of C–H arrangement (one hydrogen alone on a ring edge, two side by side, three, four) appears often enough. It did not. A carefully covering selection did no better than random picks. What does matter is the mix of ring types, because that is what the correction follows. So the plan keeps the mix of the pool the experiment drew from: 89 fused, 67 single-ring and 44 five-ring molecules out of 200, which for 167 teachers means 74, 56 and 37.
One anchor costs rented server time, and the cost grows steeply with the size of the molecule. With the price rule we measured on the last wave of anchors:
Both numbers are honest, and the truth for the plan lies between them. The experiment measured how many teachers are needed when they look like its pool. Whether small teachers teach as well as a mixed set has not been measured, and the cheaper number assumes they do.
Not in either number: molecules with a positive charge (they need a different coupled-cluster route we have not built yet), a cheap extra step that corrects for the basis set, and the DFT calculations for the 128 chosen molecules that do not have them yet — those are cheap and run on the laptop.
How the missing anchors will be made is decided in November. This post gives that decision its two inputs: how many (167, for three of the four families) and what that costs (between about €730 and €3,000 of rented time).
The full numbers, the scripts and the lines written before the runs are in the project’s repository.
This post says we have “about ten” anchors and calls them “the ten we have”. The count is eleven, the same eleven as in Rehearsing the last step: benzene, naphthalene, pyridine, fluorobenzene, benzonitrile and anthracene since 6–7 October, and quinoxaline, quinoline, isoquinoline, azulene and acenaphthylene since 20 October. (The post of 20 October called quinoxaline “one of seven”: that was true that morning, before the other four came in the same day.) The argument does not change: with eleven, a test that trains on some anchors and checks on the rest still rests on one or two molecules at a time.
]]>Our network learns from examples: for each molecule in its training set, it sees the answer it should give and adjusts itself to come closer. If those answers are noisy, the network learns the noise along with the physics. This week we replaced the answers, and the network got better. The vibrations we hoped would improve did not.
The answers are Hessians: tables of how stiff a molecule is against every small push of its atoms. They hold everything the network needs to know about where the vibrations of a molecule sit. Until now we computed most of them by finite differences: push the atoms a tiny step, compute the forces, push back, compute again, and subtract. It works, but every subtraction of two nearly equal numbers leaves a little noise behind.
On 3 October we measured that noise per kind of vibration. It was small for the C–H stretches (0.2 wavenumbers), larger for the ring and skeletal motions (1.5), and largest for the C–H out-of-plane bends (3.3), which is exactly the size of the error we were aiming for. We wrote then that the honest next step was cleaner answers, computed analytically: from exact formulas for the derivatives instead of small steps.
A rented server spent the past weeks on that, and the training set now has analytic Hessians for 862 molecules. On Tuesday night the laptop trained the network again on them, with nothing else changed: same design, same recipe, same 750 training molecules, three trainings with different random starts. Four hours later we had version 1.3.
We keep a list of network versions, and only one of them is the one the rest of the project uses. A new version replaces it only if it is worse on none of five measures, on both groups of molecules the network never sees during training (hold-out sets). Ten of those molecules resemble the training set. The other 39 are built on ring frameworks it has never met.
Four of the measures are the typical frequency error for each kind of vibration. The fifth asks how well the network gets the way the vibrations of neighbouring rings mix: it is the share of that mixing error the network leaves over, so 0.17 means it removes 83 per cent.
To make the comparison fair, the old version was scored again against the same cleaner answers as the new one. Errors in wavenumbers, lower is better:
| C–H stretch | out-of-plane bend | ring, in plane | other | ring mixing | |
|---|---|---|---|---|---|
| familiar frameworks, old (v1.1) | 1.25 | 2.36 | 1.82 | 3.35 | 0.192 |
| familiar frameworks, new (v1.3) | 1.15 | 1.99 | 1.66 | 3.35 | 0.172 |
| unfamiliar frameworks, old (v1.1) | 2.55 | 3.33 | 2.35 | 4.66 | 0.289 |
| unfamiliar frameworks, new (v1.3) | 2.50 | 2.70 | 2.12 | 3.89 | 0.263 |
Worse on none, lower on nine of ten. Version 1.3 is now the network the project uses (the read). (There was a version 1.2 in between: a different input to the network, which was tested and rejected.)
Not every one of those nine improvements is solid. Three trainings with different random starts do not end in exactly the same place, and most of the differences in the table are within that spread. Three are clearly beyond it: the out-of-plane bends on the familiar frameworks, and the ring mixing on both groups. The rest point the same way, but on their own they would not convince us.
The out-of-plane bends improved most, which is what the noise measurement of 3 October predicted: their answers were the noisiest, so cleaning them helped most.
The column that did not move is “other”: the slow, floppy motions of the whole molecule, and the motions of side groups attached to the rings. On the familiar frameworks it stayed at 3.35 to two decimals. Before the training we wrote down what we would conclude: if the cleaner answers lowered that number by at least 0.3, the noise had been the problem; if not, it was not. It did not, so the noise was not the problem. On the unfamiliar frameworks the same column did fall, from 4.66 to 3.89, but that is within the spread of the three trainings.
That is a useful negative. On 3 October we had already guessed that this family’s error was a learning gap rather than a noise floor, and now we know it. Whatever holds the floppy motions at 3.35, it is in what the network learns or in what it is shown, not in the quality of the answers. They stay the open family on our list.
Cleaner answers made a better network, at the cost of a few weeks of rented computing and four hours on a laptop, and the improvement landed where the noise measurement said it would. The floppy motions are now the clearest remaining problem, and we know one thing that is not causing it.
The full numbers, the scripts and the lines written before the training are in the project’s repository.
]]>This project stands on software other people wrote and give away for free. Almost all of our quantum chemistry runs on PySCF, a Python library kept up by a group of researchers and volunteers. Next to it we use a handful of smaller packages: one for anharmonic corrections, one for geometry optimisation, one for a fast approximate method.
Software that many people use still has corners that nobody has walked into yet. Our calculations are large and unusual enough that we walked into several. Each time we had two choices: work around the problem in our own code, or fix it at the source so that the next person does not hit it. Where we could, we fixed it at the source. Each fix goes to the maintainers as a pull request: a proposed change, usually with a test that fails without the change and passes with it. The maintainers read it, ask questions, and decide whether to take it.
One thing should be said plainly. The code in these pull requests was written and tested by an AI coding assistant, Claude Code, which works on this project with me. I decide what we take on and hand in the pull requests; tracking down the cause, writing the change and proving it with tests is its work.
So far that has been nineteen pull requests to six projects. Eleven have been accepted, six are waiting for review (four of them submitted on 27 October), and two we withdrew because a better fix came along. Below is each of them, in a sentence or two.
PySCF is a free Python library for quantum chemistry, written and kept up by researchers and volunteers; almost every calculation in this project runs on it. Its documentation is at pyscf.org.
PySCF-forge holds new methods for PySCF until they are ready to move into PySCF itself.
PySCF properties computes how molecules and crystals respond to electric and magnetic fields, among them the brightness of infrared lines.
pyVPT2 computes the anharmonic corrections to vibrational frequencies by second-order perturbation theory, on top of the quantum-chemistry package Psi4 (documentation).
optking searches for the shape in which a molecule is most stable, a geometry optimisation. It started as part of Psi4 and is now a separate Python program.
dxtb, from Stefan Grimme’s group in Bonn, is a fast approximate quantum-chemistry method (extended tight binding) written in PyTorch, so that it can be combined with machine learning (documentation).
We submitted these four on the morning of 27 October, after reworking all four to follow the style rules that PySCF publishes for contributors. None of them has been reviewed yet, so they may still change or be turned down. The first two fix problems that can hit any molecule. The other two are about cations, molecules that have lost one electron. Many PAHs in space are cations, and the most accurate method we use, CCSD(T), ran out of memory on every cation beyond the smallest.
When you compute the vibrations of a molecule with a standard quantum-chemistry method, the frequencies come out a few per cent too high. The usual repair is simple: multiply every computed frequency by one number, a scale factor, slightly below one. The NASA Ames PAH database uses three such numbers, one for each region of the spectrum (0.956, 0.952 and 0.960). The largest infrared library of PAHs to date, 144,111 spectra published this year by Mai, Lu and Wang, uses exactly those three numbers.
This project tries to do without the scale factor. Instead of stretching the whole spectrum by a fixed amount, a neural network learns how each vibration of each molecule should move. That only makes sense if a scale factor actually leaves something behind. This week we measured how much, and where.
Before we looked at any number, we wrote down what we would compute and what would count as a pass. That note is in the repository with a date and a commit (the registration), so the results below could not be bent to fit afterwards.
Our network learns in two stages. In the first stage it practises on a stand-in: the difference between two cheaper methods (called B3LYP and ωB97X), which we can compute for hundreds of molecules. The real target, coupled cluster, is too expensive for that. I wrote about the two stages last week.
On the stand-in, the question is easy to make fair. We fitted the best scale factors we could to 1,073 molecules — more than the network ever sees in one training — and asked how far each vibration still ends up from where the second method puts it. Then we asked the same of the network, on molecules neither had seen: ten held back from training, and 39 built on ring frameworks the network had never met.
We split the vibrations into four kinds:
The typical error left over (a root-mean-square average over all vibrations of that kind), in wavenumbers, the unit spectroscopists use:
| C–H stretch | out-of-plane bend | ring, in plane | other | |
|---|---|---|---|---|
| best three-region scale factor | 2.0 | 9.2 | 15.5 | 8.6 |
| best scale factor per kind of vibration | 2.0 | 6.6 | 15.3 | 8.5 |
| our network | 1.3 | 2.4 | 1.8 | 3.4 |
On the 39 molecules with unfamiliar frameworks the network’s errors are larger (2.6, 3.8, 2.6 and 6.0), but it still removes between a third and more than four fifths (33 to 82 per cent) of what the scale factor leaves (the read).
The ring vibrations are the striking column. A scale factor leaves about 15 wavenumbers there, and even a separate factor for every kind of vibration does not help. The reason shows up when you look at how the ring vibrations actually move between the two methods: on average they go up by about 12 wavenumbers, but from one vibration to the next that varies by 16. A single number cannot follow that. The network can, because it looks at each vibration and its surroundings in the molecule.
For the C–H stretches it is the other way round. They all move by almost the same amount (about 44, give or take 1.5), so a scale factor does nearly as well as the network. That is no surprise: it is the case scale factors were made for.
The stand-in is a rehearsal. The real test is measured spectra, so we took the two molecules for which we have both a coupled-cluster calculation and good measurements of every vibration: benzene and naphthalene. Three things came out.
Our reference calculation is good where it should be. For benzene, spectroscopists have worked back from the measured spectrum to the frequencies the vibrations would have if the bonds were perfect springs. Against those, our coupled-cluster calculation is typically 5.5 wavenumbers off (root mean square) for the ring vibrations. The C–H stretches, however, come out 38 too low. The likely cause: we compute the coupled-cluster vibrations at the shape the cheaper method gives the molecule, and its C–H bonds are slightly longer than coupled cluster would make them: the forces in our own calculation pull every hydrogen about 0.004 ångström closer to its carbon (the check). A longer bond vibrates more slowly. We saw the same offset in an earlier check, so it looks systematic, and the final pipeline will have to correct for it.
Scaled spectra are decent, with offsets. Mai, Lu and Wang’s scaled spectra land typically 22 wavenumbers from the measured bands of benzene and 15 from those of naphthalene. For benzene the ring vibrations are on average about 25 too low: an offset of one kind of vibration, the kind a single number cannot remove.
Real bonds are not perfect springs. Our coupled-cluster frequencies are further from the measured bands than the scaled ones: the C–H stretches by about 90 wavenumbers. That is not an error in the calculation. Real bonds get softer the more they stretch (anharmonicity), so real vibrations sit lower than perfect-spring ones. A scale factor quietly folds this effect into its one number, together with everything else. We want to keep the pieces apart, so we add it as a separate step, which we are building alongside the network (the comparison).
On the practice task the network does clearly better than any scale factor, in every kind of vibration, and most of all for the ring vibrations, which carry much of the structure of a PAH spectrum. Against experiment the picture is more modest, and more useful: a coupled-cluster correction alone does not beat a scale factor; it needs its companion step for anharmonicity, and a fix for the C–H offset that comes from the molecule’s shape.
Two caveats. The first part is the stand-in, not experiment. And the second part rests on two molecules; it shows where the differences are, not how large they will be across all PAHs.
The full numbers, the scripts and the note written beforehand are in the project’s repository.
The second paragraph says the NASA Ames PAH database uses the scale factors 0.956, 0.952 and 0.960. That was stated too firmly. Those three numbers are the factors Mai, Lu and Wang give for the database, citing its 2018 paper: 0.960 above 2500 cm⁻¹, 0.952 between 1111 and 2500, and 0.956 below 1111. They are the numbers their library uses. Nothing in this post’s results depends on them: the practice-task table fits its own best factors, and the comparison with experiment uses their scaled spectra as they published them. The database file itself, as served for download, stores other factors: 0.9597, 0.9691 and 0.9794, as this notebook found when it read the file on 11 September (Who are we up against?). Why the published factors and the stored ones differ we have not settled.
]]>The project tries to predict the infrared spectra of large carbon molecules accurately without paying for the most expensive calculations. The plan has two parts. A cheap quantum-chemistry method gives every molecule’s vibrations, but with a systematic error. A neural network learns to correct that error.
The correction we really want points towards coupled cluster, the reference method. But one coupled-cluster calculation of the vibrations costs days for a small molecule, and it is impossible for the large ones we care about. So we cannot show the network many examples of the real correction. We do it in two stages:
This post is about stage 2, which did not work at first.
A neural network is a stack of layers. The first layers turn a molecule into a description the network can work with; the last few layers, which I call the head, turn that description into the correction. When a network adjusts to new examples, you choose which layers may change.
We let only the very last layer change, and we added a brake: a penalty that keeps that layer close to what it learned from the stand-in. The idea was caution. With only eleven anchors, a network that changes too much can learn the eleven by heart and get worse on everything else.
The registered test (written down before we looked) asked: do more anchors help? We looked at the out-of-plane C–H bending vibrations, where the hydrogen atoms move up and down out of the plane of the rings. These are the hardest for the network and important in the spectrum.
The answer was no. On the six original anchors the error went from 14.06 to 13.69 cm⁻¹ when five new anchors joined, which is within the noise (the read). The predicted brightness of the lines was also worse than not correcting at all, on all three anchors where we could check it.
“The idea does not work” was one possible reading. But there are other possible causes: too few anchors, wrong settings, the wrong layers allowed to change, the network’s design. Overnight I tested more than twenty of them, one at a time, on the data we had.
The test that decided it is an unusual one. Normally you test a network on molecules it has not seen, because the question is whether it can predict new ones. This time I did the opposite on purpose: I let the network adjust on all eleven anchors and then tested it on those same eleven.
That is like asking a student to answer the exact questions they just studied, with the answers in front of them. A student who still gets them wrong does not need more practice questions: something stops them from taking the answers in. The same holds for the network. If it cannot reproduce the anchors it was just shown, more anchors will not help; the way it is allowed to learn is too narrow.
That is what happened. With only the last layer allowed to change, the network stayed 7.76 cm⁻¹ off on the very anchors it had been shown. With the whole head allowed to change, it reproduced them to within 0.80 cm⁻¹. It could take the answers in; we had not let it. Tested properly again, on anchors it had not seen, the error halved: from 12.29 to 6.54 cm⁻¹, averaged over three repeats (the night’s report).
A second finding explains why the brake hurt. For the out-of-plane vibrations, the coupled-cluster correction does not point the same way as the stand-in correction; it partly points the opposite way. A brake that holds the network near the stand-in holds it near the wrong answer.
The next morning we ran the registered test again, this time with the whole head free and the predictions written down first. Both came out as predicted. The out-of-plane error now falls with every added anchor, from 7.24 to 6.25 cm⁻¹ (14 % lower; the read), and the line brightness beats “no correction” on all three anchors where we can check it.
That is encouraging, but it is not proof. I chose the whole-head approach by looking at these same eleven anchors. It is like adjusting a recipe by tasting one pot of soup and then judging the recipe by tasting the same pot again. To judge the recipe you need other pots.
We will not have more coupled-cluster anchors before next summer. So we rehearsed stage 2 in a setting where we do have hundreds of correct answers: the cheaper methods. We built two rehearsals.
For each k, from 5 to 200, four ways of adjusting competed: the last layer with the brake (the old way), the last layer without the brake, the whole head free, and a fresh network that learns from the k molecules alone. The fresh network tells us what the earlier practice is worth.
Before anything ran, we wrote down what we expected (the registration): in the first rehearsal the whole head should beat the old way; in the second probably not, because there the correction does not change, only the molecules.
Both expectations held (the read).
In the first rehearsal, the old way did not learn at all. With the brake on, the out-of-plane error stayed between 8 and 10 cm⁻¹ whether the network saw 5 molecules or 200: the brake kept it on the wrong correction. With the whole head free, the error fell steadily, from 6.6 cm⁻¹ with 5 molecules to 4.0 with 20 and 2.8 with 200, and the line brightness improved with it. One nuance: with only five molecules, freeing just the last layer (without the brake) did slightly better (5.9) than freeing the whole head. From twenty molecules on, the whole head led.
The earlier practice was worth something even though it had learned the wrong correction, and most of all when examples were few. With 20 molecules, the adjusted network (4.0) was already better than a fresh network that had seen 50 (4.8). With more examples the gap closed, and at 200 molecules the two were equal.
In the second rehearsal, the network was right before it adjusted at all: about 2.7 cm⁻¹ on a family it had never seen. Adjusting changed little. That was expected too: what it had learned carried over to the new molecules.
Our eleven anchors failed the first test because of how we let the network learn, not because there were too few of them or because the idea is wrong.
The rehearsal also gives a first answer to the question the next phase has to put a price on: how many anchors buy how much accuracy. With the whole head free, the out-of-plane error in the rehearsal was:
| molecules shown | 5 | 10 | 20 | 50 | 100 | 200 |
|---|---|---|---|---|---|---|
| error (cm⁻¹) | 6.6 | 5.1 | 4.0 | 3.8 | 3.2 | 2.8 |
Most of the gain comes from the first twenty or so: going from 5 to 20 molecules removes more error than going from 20 to 200. After that, each doubling buys a few tenths of a wavenumber. Our own target for this family is 3 cm⁻¹ or better; in the rehearsal that took between 100 and 200 molecules. For coupled cluster that would be far more anchors than we have, so for the next phase the curve says: a few dozen anchors buy most of the improvement, and the last step to 3 cm⁻¹ is the expensive part.
It is a rehearsal, so it has limits: the stand-in replaces coupled cluster, there is one way of making the correction different, and one held-out family. It is a guide for the real step, not a guarantee.
]]>Two things held this project back in the past fortnight. One was a measurement that seemed to say our method stops working when molecules get bigger. The other was money: the most accurate reference numbers we use, the anchors, needed a rented server that we had stopped paying for. This week both moved, and neither moved by luck: each was a test with its line written down before the result.
The worry about size. The method learns a correction: how a cheap calculation of a molecule’s vibrations should be adjusted to look like an expensive one. It learns from a corpus of about a thousand molecules, none with more than four fused rings and none larger than about thirty atoms. The molecules astronomers care about are bigger, from forty atoms to a hundred and more. So the question is whether the correction learned on small molecules still holds on large ones, where we cannot afford to teach it.
An earlier test, chain 39, said it got much worse: the error of a size-shifted test set was almost twice that of a matched control (a ratio of 1.96). But that test had a flaw we knew about: its small training molecules also had fewer rings, so a bigger molecule was always also a molecule with a new ring count. Size and ring count were tangled together.
Chain 41 untangles them. We took thirty molecules with five fused rings, 32 to 40 atoms: perylene, benzo[a]pyrene, benzo[e]pyrene, benzo[k]fluoranthene, picene and dibenz[a,h]anthracene, each with four relatives carrying small changes. One model was trained on the corpus as it stands, at most four rings, and tested on them. A second, the control, was also given the five-ring molecules of three of the six families, and tested on the other three. If the first model is much worse than the control, five rings must be in the training data. If they are about equal, the correction carries over.
The measure is the share of the ring coupling error that a model leaves behind, compared with making no correction at all: 0 would be perfect, 1 would be no help. Before any number, we wrote down two lines. A ratio of the two models of 1.3 or less: the correction carries. A ratio of 1.8 or more: steep, and the corpus would need every ring count we ever want to predict.
The result: 0.428 for the model that never saw five rings, 0.406 for the control. A ratio of 1.05 (the read). In frequencies, after the correction, the two models are off by 3.97 and 3.41 cm⁻¹ on these molecules, where making no correction at all is off by 22.63. And giving the control twice as many five-ring molecules barely changed it (0.413 with half, 0.406 with all), so this is not a case of the control simply having had too few examples.
So the steep penalty of chain 39 came mostly from the new ring count, not from size itself. For the ring couplings, the quantity this test was built to measure, size up to 40 atoms is not a wall.
Equally good is not yet good enough. Two models can agree and both be poor, so the ratio alone does not settle it. The absolute numbers settle part of it: without a single five-ring molecule in training, the model still removes about 82 % of the frequency error on these molecules, from 22.6 to 4.0 cm⁻¹. But our target is 3 cm⁻¹ or less in every family of vibrations, and on the five-ring molecules no family gets there, with or without five-ring examples (per family). The out-of-plane hydrogen bends and the catch-all family sit at 4 to 5 cm⁻¹, as they also do on some families inside the corpus. And one family tells a different story from the ratio: the carbon–hydrogen stretches are off by 3.2 cm⁻¹ without five-ring examples and by 1.2 to 1.4 with them. A handful of larger ring systems in the corpus clearly helps. These family numbers were promised beside the ratio before the test, but our read script left them out; we computed them only after the result was known.
The cheaper ruler. The second story is about the anchors: molecules for which we have the expensive kind of calculation, coupled cluster, the reference the network is checked and tuned against. Even that is done in two parts. The coupled-cluster part uses a small basis set, a coarse brush, because a fine brush would be far too costly. A cheaper method, MP2, then measures how much the answer changes from the coarse brush to the fine one, and that change is added on top.
That MP2 step was the expensive part in memory: 26 gigabytes per process, which only a rented server had. Five new anchors were finished in their coupled-cluster part and waiting for it, about 98 server-hours, at a time when we had decided to stop renting.
There is a well-known shortcut, density fitting, which builds the huge table of electron interactions from a much smaller one. The program we use for most of our work cannot do the derivatives we need with that shortcut; another program, psi4, can. The risk is that the shortcut changes the answer. So the test, again written down first: take two molecules whose MP2 step we already had the old way, pyridine and naphthalene, redo the step with the shortcut, and compare the vibrational frequencies that come out. Within 0.5 cm⁻¹ for every vibration: adopt it. Up to 2: note the difference and carry it as an error. Beyond: refuse it.
Before that, a sanity check on water: the shortcut’s derivatives agree with derivatives worked out step by step from its energies to 5 × 10⁻⁸ atomic units, against a line of 10⁻⁶.
The swap: pyridine’s frequencies moved by at most 0.42 cm⁻¹, naphthalene’s by at most 0.46. Both under the line of 0.5, so the shortcut is adopted (the read). I had predicted less than 0.3, and that was wrong: the line holds, with less room to spare than I expected.
What it buys: one step of the fine-brush calculation for naphthalene took 136 seconds on the laptop, against 1,996 seconds the old way on a large server. The largest memory use was 3.3 gigabytes. All five waiting anchors together come to about eight and a half hours on the laptop we already own.
The first one is in. Quinoxaline, a two-ring molecule with two nitrogen atoms, went through the whole chain this morning. Its two softest vibrations come out at 115 and 119 cm⁻¹ with the coarse brush, and at 167 and 179 with the fine-brush step added. The cheap calculation the network starts from has them at 177 and 185. The coarse brush alone would have taught the network to pull these two down by about 60 cm⁻¹; with the step, by about 10. Most of that big pull was the brush, not the molecule. It passed our promotion rule and is now one of seven anchors in use (the anchor). The other four follow today.
What this is not. Chain 41 compares two density functional methods, a cheap one and a more careful one, as a stand-in for the real target; it says the correction carries in size, not yet that the coupled-cluster correction does. It reaches 40 atoms; the 50- to 100-atom molecules are the next test, chain 42. Two of the thirty molecules left the test before any result was read, because the reference calculations for them showed an unstable vibration; they come back when better reference calculations exist. And the density-fitting shortcut was tested on two molecules and adopted with a margin smaller than predicted: if one of the new anchors behaves differently, the old way is still there, on a server, when the budget allows.
]]>