Spectrum Atlas

Methods

The pipeline starts from a cheap calculation of a molecule's infrared spectrum. A trained model then predicts how a far more accurate calculation (coupled cluster, CCSD(T)) would shift each vibration. That prediction is shown for a family of vibrations only after it has been checked against anchors: molecules for which the accurate calculation was actually done.

The model is still in training on a growing set of molecules, so no page shows a predicted correction yet. Six molecules have an anchor: benzene, naphthalene, anthracene, pyridine, fluorobenzene and benzonitrile. Their pages show the anchor's spectrum first, and benzene's is also checked against a measured laboratory spectrum (NIST). Three more anchors (quinoxaline, azulene, acenaphthylene) are computed or in progress for a registered test and are not shown until it is read. Every other molecule shows the cheap spectrum only. Everything below links to the pre-registration or the note behind it.

The pictures — the architecture sheets, read from the repository at build time

Rectangles are steps (operation + software), rounded shapes are data objects, dashed shapes are not built yet. (loads a diagram library, ≈ 600 kB)

Overview 00_overview.mmd

Overview of plan 05 (level 1): the kinds of process and the data objects that connect them. Target architecture; no decisions, no data. Rectangle = process (here: a whole sheet); rounded ends = data object; arrow = data flow. Dashed = not built yet. 7 October 2026: the candidate generator (module 06) feeds the corpus as a registered source; the Spectrum Atlas publishes the catalogue with the status and the provenance of every molecule; training exists (the carried ΔH model); test and licence, and the spectrum pipeline as one chain, do not yet.

flowchart LR
  linkStyle default stroke:#8a9bb0,stroke-width:2.2px
  classDef planned stroke-dasharray: 6 4,stroke:#b8860b,fill:#fff3c4,color:#111
  classDef ext fill:#cfd8e3,stroke:#3d5a80,color:#111
  classDef data fill:#e9ecef,stroke:#555,color:#111
  classDef proc fill:#f7f7fb,stroke:#333,stroke-width:1.5px,color:#111
  classDef research fill:#fdf2e3,stroke:#8a5a00,color:#111

  MOL(["Molecule: geometry, charge, multiplicity"]):::data
  COND(["Observation conditions: internal energy after UV absorption (emission) or temperature (absorption); resolution of the instrument"]):::data
  PUB(["PubChem fused aromatics, frozen set"]):::ext
  LABDB(["Laboratory spectra"]):::ext
  PAHDB(["Opponents: scaled DFT spectra (PAHdb and others) and other predictors"]):::ext
  GENF["Candidate generator (module 06)"]:::proc
  CANDS(["New fused ring systems with PubChem membership"]):::data
  LABF["Label factory (sheet 4)"]:::proc
  LABELS(["Label: ΔH blocks with error margin per family and geometry term per mode, sealed"]):::data
  CORPF["Corpus step (sheet 3)"]:::proc
  CORPUS(["Corpus records: two DFT Hessians, proxy correction, local-coordinate features and deck responses per molecule"]):::data
  ATLASF["Atlas export (website)"]:::proc
  ATLAS(["Spectrum Atlas: catalogue with status and provenance per molecule"]):::data
  TRAIN["Training (sheet 6)"]:::proc
  TRAINED(["Trained ΔH model: ensemble of members"]):::data
  TESTSET(["Test set"]):::data
  EVAL["Test and licence (sheet 7)"]:::planned
  LICCAL(["Licence table and calibration"]):::data
  PIPE["Spectrum pipeline (sheet 8)"]:::planned
  SPEC(["Spectrum: bands with position, intensity, profile and error margin; licence status per family"]):::data
  RES["Research process (sheets 1 and 2)"]:::research

  PUB --> GENF --> CANDS --> CORPF
  MOL --> LABF --> LABELS
  MOL --> CORPF --> CORPUS
  CORPUS --> ATLASF
  LABELS --> ATLASF --> ATLAS
  LABELS --> TRAIN
  CORPUS --> TRAIN
  TRAIN --> TRAINED
  TRAINED --> EVAL
  TRAIN --> TESTSET --> EVAL
  LABDB --> EVAL
  PAHDB --> EVAL
  EVAL --> LICCAL
  MOL --> PIPE
  LABELS --> PIPE
  COND --> PIPE
  TRAINED --> PIPE
  LICCAL --> PIPE
  PIPE --> SPEC
  RES -. decides on .-> LABF
  RES -. decides on .-> TRAIN
  RES -. decides on .-> EVAL
  RES -. decides on .-> PIPE

source

Data creation: the corpus 30_data_creation_corpus.mmd

Data creation — the corpus: two DFT Hessians per molecule and the proxy correction (level 3). Target architecture. Rectangle = process step (operation + software); rounded ends = data object (the thing). Every step yields one data object. Dashed = not built yet. 7 October 2026: the molecules come from a manifest with a source per row (parents by hand, enumerated children, Hessian-QM9, pool 3, the candidate generator of module 06 with its gate); what is computed next is chosen by the composition rule. The two Hessians are analytic (pyscf) — the finite-difference psi4 Hessians of the first layers stay in the record as the first route and are being replaced pool-wide. The record keeps the Cartesian Hessians, the modes, the local-coordinate features and the probe deck's pattern responses (the mode tokens of 6 September were dropped by decision 49). 9 October 2026, the cheap level's reach (measured): both DFT Hessians by finite differences (psi4, 6-31G*, 8 threads of a CPX62) take 4.0 h for perylene (32 atoms), 5.3 h for a 33-atom, 6.6 h for a 35-atom and 7.6 h for a 38-atom molecule of pool 3; the analytic route (pyscf, the labels server) takes 6,000–14,000 s per functional at 30 atoms. So the corpus can hold the large families themselves (≈ 40 atoms); chain 39 (9 Oct) showed that it has to — the network does not carry the correction across a large gap in size and ring count.

flowchart LR
  linkStyle default stroke:#8a9bb0,stroke-width:2.2px
  classDef planned stroke-dasharray: 6 4,stroke:#b8860b,fill:#fff3c4,color:#111
  classDef data fill:#e9ecef,stroke:#555,color:#111
  classDef ext fill:#cfd8e3,stroke:#3d5a80,color:#111

  SRCX(["Parent molecules, Hessian-QM9, pool 3"]):::ext
  PUB(["PubChem fused aromatics, frozen set"]):::ext
  ENUM["Enumeration of substituted children (RDKit)"]
  KIDS(["Enumerated children of the parent cores"]):::data
  GEN["Candidate generation and gate (module 06; PyTorch, RDKit)"]
  CANDS(["New fused ring systems with PubChem membership"]):::data
  MAN["Manifest assembly with a source per row (own software)"]
  MANI(["Manifest: molecule, layer, source, status"]):::data
  SEL["Selection by the composition rule (own software)"]
  MOL(["Molecule: SMILES or geometry, charge, multiplicity"]):::data
  OPT["Geometry optimisation at low level (psi4)"]
  GEO(["Optimised geometry"]):::data
  DFTL["Analytic Hessian at low level, B3LYP (pyscf)"]
  SK(["Hessian H0 and dipole derivatives"]):::data
  DFTH["Analytic Hessian at high level, ωB97X (pyscf)"]
  H1(["Hessian H1"]):::data
  MODE["Mode analysis (own software)"]
  MODES(["Normal modes: L, frequencies, families, symmetry blocks"]):::data
  PROXY["Proxy correction (own software)"]
  DHP(["Proxy correction: ΔH = H1 − H0 (Cartesian), with its projections on the modes and on the primitive-pair coordinates"]):::data
  LOCC["Local-coordinate features (own software, geomeTRIC)"]
  PF(["Primitive internal coordinates and their pairs — diagonal, atom-sharing, ring bond–bond — with pair features from H0"]):::data
  DECK["Deck responses (own software)"]
  RESP(["Pattern responses R = ½ aᵀ ΔH a of the probe deck: the measurement planner's records"]):::data
  REC["Record assembly (own software)"]
  CORP(["Corpus records: two DFT Hessians, proxy correction, local-coordinate features and deck responses per molecule"]):::data

  SRCX --> ENUM --> KIDS --> MAN
  PUB --> GEN --> CANDS --> MAN
  SRCX --> MAN --> MANI --> SEL --> MOL
  MOL --> OPT --> GEO
  GEO --> DFTL --> SK --> MODE --> MODES
  GEO --> DFTH --> H1
  SK --> PROXY
  H1 --> PROXY
  MODES --> PROXY --> DHP --> REC
  GEO --> LOCC
  SK --> LOCC --> PF --> REC
  DHP --> DECK
  MODES --> DECK --> RESP --> REC
  SK --> REC
  REC --> CORP

source

Data creation: the labels 40_data_creation_labels.mmd

Data creation — the label factory: how one label comes about (level 3). Target architecture: no decisions, no data. Rectangle = process step (operation + software); rounded ends = data object (the thing). Every step yields one data object. Dashed = not built yet. 7 October 2026: redrawn to the built route. A label is a canonical CCSD(T) Hessian by central differences of analytic gradients over the symmetry-unique displacements, raised to cc-pVTZ quality by the MP2 basis step (the composite level), checked by two routes per quantity and kept in the anchor registry (one carried version per molecule). The open-shell gradients are the planned branch for cations; the deck-transport route of September is replaced by the full symmetric deck. 26 October 2026: the local-correlation (LNO) route is gone (its out-of-plane offset does not shrink); the MP2 basis step is psi4's density-fitted MP2 gradient.

flowchart LR
  linkStyle default stroke:#8a9bb0,stroke-width:2.2px
  classDef planned stroke-dasharray: 6 4,stroke:#b8860b,fill:#fff3c4,color:#111
  classDef data fill:#e9ecef,stroke:#555,color:#111

  MOL(["Molecule: geometry, charge, multiplicity"]):::data
  SK(["Hessian H0 and dipole derivatives"]):::data
  SYM["Symmetry analysis (own software)"]
  DECK(["Deck: symmetry-unique Cartesian displacements"]):::data
  CCG["CCSD(T) gradients with relaxed density (pyscf, own (T) kernels in C)"]
  GRADS(["Gradients, energies and dipoles at every displacement"]):::data
  OPEN["UCCSD(T) gradients for open shell (pyscf, own unrestricted (T) kernels)"]:::planned
  OPENG(["Open-shell gradients, energies and dipoles at every displacement"]):::data
  FD["Central differences and symmetry reconstruction (own software)"]
  HCC(["CCSD(T)/cc-pVDZ Hessian and dipole derivatives"]):::data
  MP2["DF-MP2 gradients in two basis sets (psi4)"]
  MROWS(["MP2 Hessian rows, cc-pVDZ and cc-pVTZ"]):::data
  COMP["Composite assembly (own software)"]
  HCOMP(["Composite Hessian: CCSD(T)/cc-pVDZ plus the MP2 basis step"]):::data
  DIFF["Difference to the cheap level (own software)"]
  DH(["ΔH blocks per family: diagonal and couplings"]):::data
  CHECK["Consistency check (own software)"]
  NOISE(["Noise term per quantity: difference between two routes or between symmetry partners"]):::data
  BUDGET["Error budget (own software)"]
  MARG(["Error margin per family: noise and additivity error of the composite"]):::data
  CUB(["Cubic constants of the low level (from the VPT2 step of sheet 8)"]):::data
  GEO["Geometry term (own software)"]:::planned
  GTERM(["Geometry term per mode"]):::data
  SEAL["Sealing and registration (own software)"]
  LABEL(["Label: ΔH blocks with error margin per family and geometry term per mode, sealed"]):::data

  MOL --> SYM --> DECK
  DECK --> CCG --> GRADS --> FD
  DECK --> OPEN --> OPENG --> FD
  FD --> HCC --> COMP
  DECK --> MP2 --> MROWS --> COMP
  COMP --> HCOMP --> DIFF
  SK --> DIFF --> DH --> SEAL
  GRADS --> CHECK
  HCC --> CHECK
  MROWS --> CHECK
  CHECK --> NOISE --> BUDGET
  DH --> BUDGET --> MARG --> SEAL
  GRADS --> GEO
  CUB --> GEO --> GTERM --> SEAL
  SEAL --> LABEL

source

The correction model 50_deltaH_model_components.mmd

Components of the ΔH model (level 4): what happens inside the step "Forward pass (ΔH model, PyTorch)" of sheet 8 (the spectrum pipeline). Rectangle = process step (operation + software); rounded ends = data object (the thing). Every step yields one data object. Dashed = not built yet. 7 October 2026: redrawn to the carried design (the hybrid model, ΔH-network v1.1, chain 34). An equivariant body turns atoms, positions, charge state and the rotation invariants of H0 into per-atom features; a pair head predicts one force-constant correction per pair of primitive internal coordinates on a fixed pattern, from the features of the atoms involved, the pair features and the cheap force constants; a learned scale per pair class adds a share of the cheap force constant (SQM-like); ΔH = Bᵀ ΔF B is exactly symmetric and translation- and rotation-free. The tensor head (Cartesian 3×3 blocks) and the stand-alone pairwise head of rungs B and C remain in the code as measured variants, not in the carried design.

flowchart LR
  linkStyle default stroke:#8a9bb0,stroke-width:2.2px
  classDef planned stroke-dasharray: 6 4,stroke:#b8860b,fill:#fff3c4,color:#111
  classDef data fill:#e9ecef,stroke:#555,color:#111

  GEO(["Molecule: atomic numbers, coordinates of the cheap minimum, charge, multiplicity"]):::data
  SK(["Hessian H0 of the cheap level"]):::data
  PRIMC["Primitive internal coordinates and the pair pattern (geomeTRIC, own software)"]
  PRIM(["Primitive pairs of the pattern, Wilson B matrix, cheap force constants F0 per pair"]):::data
  FEAT["Pair features (own software)"]
  PF(["Pair features: pair classes, ring relations, projections of H0"]):::data
  EQB["Equivariant body: message passing over atoms with scalar and vector channels, charge-state embedding, fed by the rotation invariants of H0 (PyTorch)"]
  ATOMF(["Per-atom scalar and vector features"]):::data
  HEAD["Pair head: pooled atom features of the two primitives, pair features, F0 (PyTorch)"]
  DFR(["ΔF residual per primitive pair"]):::data
  SQM["Class scaling: ΔF = α per pair class × F0 + residual (PyTorch)"]
  DFL(["ΔF: force-constant correction per primitive pair"]):::data
  BACK["Back-transformation ΔH = Bᵀ ΔF B (own software)"]
  DHC(["ΔH in Cartesian coordinates of one member"]):::data
  PROJ["Projection onto the cheap modes: Lᵀ ΔH L (own software)"]
  KLOC(["ΔH blocks per family of one member, couplings included"]):::data
  ENSA["Ensemble averaging over the members (own software)"]:::planned
  OUT(["ΔH blocks per family, with ensemble uncertainty"]):::data

  GEO --> PRIMC --> PRIM
  PRIM --> FEAT
  SK --> FEAT --> PF --> HEAD
  GEO --> EQB
  SK --> EQB --> ATOMF --> HEAD
  PRIM --> HEAD --> DFR --> SQM
  PRIM --> SQM --> DFL --> BACK
  PRIM --> BACK --> DHC --> PROJ --> KLOC --> ENSA --> OUT

source

Test and licence 70_test_and_licence.mmd

Test and licence (level 3b): from the trained ΔH model and the test set come the licence table and the calibration that the spectrum pipeline uses alongside the trained model. The model's weights no longer change here. Rectangle = process step (operation + software); rounded ends = data object (the thing). Every step yields one data object. Dashed = not built yet. Built: the test per family on the fixed hold-outs (analytic labels where they exist), the promotion of a model against lines registered before its run, and the licence (decision 65: a family is licensed where the carried model's error is within 3 cm⁻¹ on both hold-outs; neutral molecules). Not yet: the calibration of the ensemble spread and the score against laboratory spectra and opponents.

flowchart LR
  linkStyle default stroke:#8a9bb0,stroke-width:2.2px
  classDef planned stroke-dasharray: 6 4,stroke:#b8860b,fill:#fff3c4,color:#111
  classDef data fill:#e9ecef,stroke:#555,color:#111
  classDef ext fill:#cfd8e3,stroke:#3d5a80,color:#111

  ENS(["Trained ΔH model: ensemble of members"]):::data
  TESTSET(["Test set"]):::data
  LABDB(["Laboratory spectra with the margin per reference column"]):::ext
  PAHDB(["Opponents: scaled DFT spectra (PAHdb and others) and other predictors"]):::ext
  TEST["Test on the hold-out molecules per family (own software)"]
  TESTR(["Test errors per family and charge state, alongside the opponents"]):::data
  PROMO["Promotion against the registered lines and the learning curve (own software)"]
  STAT(["Model status: carried, candidate or superseded"]):::data
  LIC["Licence determination: within 3 cm⁻¹ on both hold-outs (own software)"]
  LICT(["Licence table per family (neutral; charge states once cations are trained)"]):::data
  CAL["Uncertainty calibration (own software)"]:::planned
  CALR(["Calibration of the ensemble spread"]):::data

  ENS --> TEST
  TESTSET --> TEST
  LABDB --> TEST
  PAHDB --> TEST
  TEST --> TESTR
  TESTR --> PROMO --> STAT
  TESTR --> LIC --> LICT
  TESTR --> CAL --> CALR

source

The spectrum pipeline 80_spectrum_pipeline.mmd

Spectrum pipeline (level 3): a molecule in, its certified infrared spectrum out. As built: modules/08_industry_synthesis/m08/pipeline.py, pipeline 0.4 (versions in modules/PIPELINE.md). Rectangle = process step (operation + software); rounded ends = data object (the thing). Every step yields one data object. Dashed = not built yet. 30 October 2026: brought to the built pipeline (TASKS 153). The licence enters at the certificate, per family and band (decision 65), not as a filter on ΔH blocks: the whole ΔH is applied and every band says whether its family is licensed. A measured label (sheet 4) is not a ΔH source here, and observation conditions (temperature, emission) are later work (TASKS 11). Planned: reuse of a corpus Hessian (TASKS 150), the training-set check (TASKS 155), the geometry term and the anharmonic step (TASKS 53). 

flowchart LR
  linkStyle default stroke:#8a9bb0,stroke-width:2.2px
  classDef planned stroke-dasharray: 6 4,stroke:#b8860b,fill:#fff3c4,color:#111
  classDef data fill:#e9ecef,stroke:#555,color:#111

  MOL(["Molecule: SMILES, neutral and closed shell"]):::data
  ADM["Admission (own software, RDKit)"]
  ADMM(["Admitted molecule: canonical SMILES, start geometry, carried members"]):::data
  OPT["Geometry optimisation (DFT, psi4)"]
  GEO(["Optimised geometry"]):::data
  DFT["Hessian and dipole derivatives (DFT, pyscf, analytic)"]
  CORP(["Corpus records: Hessians and dipole derivatives of computed molecules"]):::data
  REUSE["Reuse in the canonical atom order (own software)"]:::planned
  SK(["Hessian H0 and dipole derivatives"]):::data
  MODE["Mode analysis (own software)"]
  MODES(["Normal modes: L, frequencies, families"]):::data
  LOCC["Local-coordinate features (own software, geomeTRIC)"]
  PF(["Primitive pairs with their features; atoms and coordinates"]):::data
  FWD["Forward pass (ΔH model: equivariant body and hybrid pair head, ΔH = Bᵀ ΔF B; PyTorch)"]
  DH(["ΔH of each member and their mean"]):::data
  APPLY["Assembly of H (own software)"]
  H(["Force constants H = H0 + ΔH"]):::data
  EIG["Diagonalisation (own software)"]
  POS(["Band positions with the members' range"]):::data
  INT["Intensity calculation, double harmonic (own software)"]
  INTS(["Band heights"]):::data
  VPT["VPT2 (pyVPT2 on pyscf Hessians)"]:::planned
  ANHC(["Anharmonic constants, with route difference per constant"]):::data
  GEOP["Geometry term (own software)"]:::planned
  POSG(["Band positions with geometry term"]):::data
  ANH["Anharmonic correction (own software)"]:::planned
  ANHS(["Anharmonic shifts per band, overtones and combination bands"]):::data
  SHAPE["Profile formation (own software)"]
  SPEC0(["Spectrum: bands with position, height and Lorentzian profile"]):::data
  TSET(["Training set of the carried members"]):::data
  TCHK["Training-set check (own software)"]:::planned
  TFLAG(["Trained on or unseen, with the ωB97X truth where the corpus holds it"]):::data
  LICT(["Licence table and measured errors per family (sheet 7)"]):::data
  LTOL(["Laboratory tolerance per spectral region"]):::data
  CERT["Certificate (own software)"]
  SPEC(["Certified spectrum: per band position, height, members' range, family, licence, measured error, laboratory tolerance"]):::data

  MOL --> ADM --> ADMM --> OPT --> GEO --> DFT --> SK
  ADMM --> REUSE
  CORP --> REUSE --> SK
  SK --> MODE --> MODES
  SK --> LOCC --> PF --> FWD --> DH --> APPLY --> H --> EIG --> POS --> SHAPE
  MODES --> FWD
  ADMM --> FWD
  SK --> APPLY
  H --> INT
  SK --> INT --> INTS --> SHAPE
  SK --> VPT --> ANHC
  POS --> GEOP --> POSG --> SHAPE
  ANHC --> GEOP
  ANHC --> ANH --> ANHS --> SHAPE
  MODES --> ANH
  SHAPE --> SPEC0 --> CERT --> SPEC
  ADMM --> TCHK
  TSET --> TCHK --> TFLAG --> CERT
  MODES --> CERT
  LICT --> CERT
  LTOL --> CERT

source

Six lessons

  1. Compute every number twice, by two different methods. One vibration of benzene in our training set was off by 133 cm⁻¹. At first it looked as if the model could not learn benzene. A second calculation by another method showed that the number itself was wrong: an artefact of the step-by-step (finite-difference) way it had been computed. Since then every computed quantity gets a second route. Quality policy.
  2. Code that runs and looks right can still be wrong. For six days our coupled-cluster calculations called a library function in a way that quietly used an ingredient of a simpler method. Nothing crashed, the numbers looked reasonable and our own checks passed, yet the frequencies were off by 2.5 to 7.7 cm⁻¹ on water. The library's own tests showed the correct call all along. Now, the first time we use a library function, we write down which of the library's own tests makes the same call. Incident note.
  3. A check needs its own check. The infrared intensities come from an equation that pyscf solves step by step. Sometimes the solver stopped before the answer was finished, without a warning. We now test every answer against the equation it should satisfy, recomputed 49 molecules, and added our evidence to the known pyscf issue. The second method we then used to confirm the result had a weakness of its own: on one molecule it needed a second electric-field strength before the two methods agreed. Blog post, pyscf issue #2702.
  4. A result on a few molecules is one point on a curve. On 30 September we concluded that more training data would not help, because the errors stayed flat as the data grew. They stayed flat because of a flaw in the training targets, which held every model at the same floor. Once that was fixed, more data clearly helped. A result on a small set now counts as one point on a learning curve, and we only stop when the curve has stopped falling. Ledger.
  5. The couplings can only be learned in local terms. When one bond in a molecule vibrates, its neighbours feel it. Written in the molecule's overall vibration patterns (its normal modes), these couplings could not be learned from any amount of data. Written as pairs of bonds and angles, the same data teach them. E7.
  6. The coupled-cluster correction is local too, one bond further. About 95 % of benzene's correction and 93 % of naphthalene's sits in pairs of bonds and angles at most two bonds apart. An earlier reading said 98 % for benzene; it came from the calculations of lesson 2 and was redone. E8, benzene, naphthalene.