Test_extended_meson_field: add view_open timers to measure MemoryManager H2D transfers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -31,53 +31,25 @@ Key configure options:
 | `--with-gmp=`, `--with-mpfr=`, `--with-fftw=`, `--with-lime=` | paths to libs |
 | `--enable-hdf5`, `--enable-mkl`, `--enable-lapack` | optional features |
 
-GPU builds additionally need `--enable-gen-simd-width=64` (sets 512-bit SIMD width for GPU warp/wavefront sizing) and `--enable-unified=no --enable-shm=nvlink` for multi-GPU runs.
-
-To speed up compilation, `--disable-fermion-reps --disable-gparity` skips instantiating G-parity and higher-representation fermion operators.
-
 Platform recipes from `README.md`:
 - **KNL**: `--enable-simd=KNL --enable-comms=mpi3-auto --enable-mkl`
 - **Skylake/Haswell**: `--enable-simd=AVX512` or `AVX2` + `--enable-comms=mpi3-auto`
 - **AMD EPYC**: `--enable-simd=AVX2 --enable-comms=mpi3`
 - **A64FX (Fugaku)**: `--enable-simd=A64FX --enable-comms=mpi3 --enable-shm=shmget` (see `SVE_README.txt`)
 
-Complete, working `configure` invocations for specific HPC systems (Frontier/ROCm, Perlmutter/CUDA, Summit, SDCC-A100, etc.) live in `systems/<platform>/config-command`. These are the canonical references for production builds.
-
 Required external libs: GMP, MPFR, OpenSSL, zlib.
 
-### Use `systems/` for real machines
-
-`systems/<machine>/` holds the known-good build for each production platform (`Frontier`, `Aurora`, `Perlmutter`, `Summit`, `Tursa`, `Lumi`, `Booster`, `Crusher`, `SDCC-*`, `mac-arm`, …). Each contains a `config-command` (the exact `../../configure` invocation) and a `sourceme.sh` (module loads and env). **Prefer copying/adapting these over hand-rolling configure flags** — they encode compiler workarounds, `LDFLAGS`, and shared-memory settings that are easy to get wrong. `systems/WorkArounds.txt` records known vendor bugs.
-
-Note the GPU builds use `--enable-simd=GPU --enable-gen-simd-width=64`, so `Nsimd` is *not* 1 on device (it is `64/sizeof(scalar)`).
-
-### Regenerating `Make.inc` — required after adding or deleting source files
-
-`Make.inc` files are generated, not tracked in git (`.gitignore`d). `scripts/filelist` walks `Grid/`, `tests/*`, `benchmarks/`, `examples/`, and `HMC/` and writes the file lists and per-test `bin_PROGRAMS` rules. Every new `.cc`/`.h` in `Grid/`, and every new `Test_*.cc` / `Benchmark_*.cc` / `Example_*.cc`, is invisible to the build until you run:
-
-```bash
-./scripts/filelist    # from the source root, then re-run configure/make
-```
-
-`bootstrap.sh` runs it for you on the first setup.
-
-## Running Tests and Benchmarks
+## Running Tests
 
 ```bash
 # From build directory
 make check                          # root-level tests (Test_simd, Test_cshift, etc.)
 make -C tests/<subdir> tests        # build tests in a subdirectory
-make tests                          # build all tests across all subdirectories
 ./tests/core/Test_simd              # run a single test binary directly
-mpirun -n 4 ./tests/core/Test_cshift --grid 16.16.16.16 --mpi 1.1.1.4
 ```
 
-`make check` is a thin smoke test — building a subdirectory with `make -C tests/<subdir> tests` and running the relevant binaries directly is the normal development loop. Test binaries take Grid's standard command-line arguments (`--grid`, `--mpi`, `--accelerator-threads`, `--threads`, `--debug-signals`, `--log`); see `Grid/util/Init.cc`.
-
 Test subdirectories and their focus: `core` (SIMD, stencil, comms), `solver` (CG, GMRES, eigensolvers), `hmc` (MD integrators), `forces` (fermion forces), `lanczos`, `IO`, `smearing`, `sp2n`, `debug`.
 
-Tests and benchmarks that need optional fermion representations are guarded by `disable_tests_without_instantiations.h` / `disable_benchmarks_without_instantiations.h`, so a `--disable-fermion-reps --disable-gparity` build silently compiles them to no-ops.
-
 ## Architecture
 
 ### Layer stack (bottom to top)
@@ -90,7 +62,7 @@ Tests and benchmarks that need optional fermion representations are guarded by `
 
 4. **Cartesian/comms layer** (`Grid/cartesian/`, `Grid/communicator/`) — `GridCartesian` holds the MPI topology and local/global geometry. `Grid/cshift/` implements nearest-neighbour halo exchange; `Grid/stencil/` is the optimised multi-hop stencil used by Dirac operators.
 
-5. **Algorithm layer** (`Grid/algorithms/`) — iterative solvers (CG, GMRES, BiCGSTAB, mixed-precision), eigensolvers (Lanczos, LAPACK), FFT, smearing, and multigrid.
+5. **Algorithm layer** (`Grid/algorithms/`) — iterative solvers (CG, GMRES, BiCGSTAB, mixed-precision), eigensolvers (Lanczos, LAPACK), FFT, smearing.
 
 6. **QCD layer** (`Grid/qcd/`) — gauge and fermion actions, HMC integrators, observables.
 
@@ -103,54 +75,9 @@ Tests and benchmarks that need optional fermion representations are guarded by `
 - `smearing/` — APE, Stout, HEX, gradient flow
 - `observables/` — Polyakov loop, plaquette, topological charge
 
-### GPU acceleration and the view/memory-manager discipline
-### Multigrid (`Grid/algorithms/multigrid/`)
-
-Aggregation-based algebraic multigrid for Wilson-type fermions. Key files: `CoarsenedMatrix.h` (coarse operator), `GeneralCoarsenedMatrix.h` and `GeneralCoarsenedMatrixMultiRHS.h` (general coarsening supporting multi-RHS solves), `Aggregates.h` (near-null vector construction), `Geometry.h` (coarse-grid geometry). `MultiGrid.h` is the top-level include.
-
 ### GPU acceleration
 
-GPU support is injected via macros in `Grid/threads/Accelerator.h` — `accelerator_for(i, n, nsimd, {...})`, `accelerator_forNB` (non-blocking, must be followed by `accelerator_barrier()`), `accelerator_for2dNB`, and `accelerator_inline`. On a CPU build these degrade to `thread_for` (OpenMP). Unified virtual memory is on by default (`--enable-unified=yes`); device-aware MPI (`--enable-accelerator-aware-mpi`) avoids device→host copies on transfers.
-
-Lattice data is **not** directly addressable inside a kernel. You must open a view with the correct access mode so `Grid/allocator/MemoryManager.h` can move/mark the data:
-
-```cpp
-autoView(out_v, out, AcceleratorWriteDiscard);   // RAII; closes at end of scope
-autoView(in_v,  in,  AcceleratorRead);
-accelerator_for(ss, grid->oSites(), Nsimd, {
-    coalescedWrite(out_v[ss], coalescedRead(in_v[ss]));
-});
-```
-
-Modes are `AcceleratorRead/Write/WriteDiscard` and `CpuRead/Write/WriteDiscard`. Getting the mode wrong (e.g. `AcceleratorRead` on a field you write) produces stale-data bugs that only appear on GPU builds. Inside kernels use `coalescedRead`/`coalescedWrite` rather than raw `operator[]` — they map the SIMD lane onto `threadIdx.x` so accesses stay coalesced.
-
-### Repo-local debugging skills (`skills/`)
-
-`skills/` contains hard-won, Grid-specific playbooks written as invocable skill files. Consult them before debugging in these areas rather than reasoning from first principles:
-
-| File | Covers |
-|---|---|
-| `gpu-memory-performance.md` | `acceleratorThreads()`, LambdaApply thread mapping, `coalescedRead` idiom, fused vs staged HBM access |
-| `gpu-runtime-correctness.md` | GPU runtime returning early from sync, silent wrong answers |
-| `communication-overlap.md` | 7-phase halo pipeline, per-packet events, host-staging vs GPU-direct RDMA |
-| `mpi-heterogeneous.md` | `MPI_Sendrecv` device-buffer aliasing, deterministic reductions |
-| `compiler-validation.md` | Isolating GPU compiler codegen bugs, minimal reproducers |
-| `correctness-verification.md` | Double-run fingerprinting, per-packet checksums, flight recorder |
-| `hang-diagnosis.md` | Diagnosing MPI/accelerator hangs |
-
-The key loop macros (defined in `Grid/threads/Accelerator.h`) are:
-- `accelerator_for(iter, num, nsimd, {...})` — maps to CUDA/HIP kernel or OpenMP loop; `nsimd` is the innermost SIMD lane count
-- `accelerator_forNB(...)` — non-blocking variant (no implicit barrier)
-- `accelerator_for2dNB(iter1, num1, iter2, num2, nsimd, {...})` — 2D kernel launch
-- `thread_for(iter, num, {...})` — CPU OpenMP loop (never dispatches to GPU)
-
-On CPU builds, `accelerator_for` aliases to `thread_for`.
-
-### Solver patterns
-
-`SchurRedBlack` (`Grid/algorithms/iterative/SchurRedBlack.h`) implements red-black (even/odd) preconditioning for fermion operators. Most production fermion solves use `SchurRedBlackDiagMooeeSolve` or similar wrappers that internally call a `ConjugateGradient` on the Schur complement.
-
-Mixed-precision solvers (`ConjugateGradientMixedPrec`, `BiCGSTABMixedPrec`) drive a double-precision outer loop with single-precision inner solves.
+GPU support is injected via macros (`accelerator_for`, `accelerator_for2dNB`). The `Grid/simd/` SIMD types map to scalar on GPU device code; host code paths remain vectorised. Unified virtual memory is on by default (`--enable-unified=yes`); device-aware MPI (`--enable-accelerator-aware-mpi`) avoids device→host copies on transfers.
 
 ### Memory and I/O
 
@@ -158,26 +85,14 @@ Mixed-precision solvers (`ConjugateGradientMixedPrec`, `BiCGSTABMixedPrec`) driv
 - `Grid/parallelIO/` — distributed parallel reader/writer for ILDG (via LIME), SciDAC, and native binary formats
 - `Grid/serialisation/` — text, binary, HDF5, XML/JSON serialisation of arbitrary Grid objects
 
-### Executables
+### HMC applications
 
-- `HMC/` — production HMC driver programmes (e.g. `Mobius2p1f.cc`, `DWF_plus_DSDR_nf2plus1_Shamir_Gparity.cc`)
-- `benchmarks/` — `Benchmark_dwf`, `Benchmark_ITT`, `Benchmark_comms`, `Benchmark_memory_bandwidth`, … used to qualify a new machine
-- `examples/` — small, readable programmes (`Example_plaquette.cc`, `Example_Mobius_spectrum.cc`) that are the best starting point for learning the API
-
-Each of these directories auto-builds every top-level `.cc` as its own binary via `scripts/filelist`.
-
-Every programme is wrapped in `Grid_init(&argc, &argv)` / `Grid_finalize()` (`Grid/util/Init.h`).
+`HMC/` contains production-ready HMC driver programmes (e.g. `Mobius2p1f.cc`, `DWF_plus_DSDR_nf2plus1_Shamir_Gparity.cc`). These are built separately from the library tests.
 
 ## Key Conventions
 
 - **C++17** is required throughout.
 - Template structure: most classes are templated on `<_FImpl>` (fermion impl) or `<Gimpl>` (gauge impl), which encode the representation and precision. Instantiation is controlled by `--enable-fermion-instantiations`.
-- **Tensor indices are positional, not labelled.** The `Grid/tensors/` arithmetic recurses structurally over the `iScalar`/`iVector`/`iMatrix` nest: each level defines only the {scalar,vector,matrix}² products at its own level, with element types resolved by automatic type deduction, so every colour/spin/lorentz combination composes from ~200 lines (versus the pre-C++11 QDP++/PETE approach of machine-generating every case). An index's meaning derives entirely from its nesting depth counted from the outside; `iScalar` is the identity/broadcast case at every level. Never insert or remove a nesting level casually — the multiplication tables contract by position.
-- **Multigrid coarsening deepens the tensor nest by one level.** A coarse site vector is `iVector<CComplex,nbasis>`, and `innerProduct` on it returns `iScalar<CComplex>` — one level deeper than the fine block scalar. So the block-inner-product scalar type gains one `iScalar` wrapper per MG level (fine: `vTComplex`; level 2: `iScalar<vTComplex>`; see `examples/Example_pvdagm_3level.cc`). When calling `blockInnerProduct`/`blockZAXPY`/`blockOrthogonalise` on coarse fields, the coarse scalar type must match `decltype(innerProduct(siteVector(),siteVector()))` exactly; a wrong depth fails to compile (no viable `operator=` deep in the instantiation chain) rather than mis-contracting.
 - The `RealD`/`RealF`/`ComplexD`/`ComplexF` typedefs are used everywhere; avoid raw `double`/`float`.
-- Use `GRID_ASSERT(cond)` (defined in `Grid/GridStd.h`), not bare `assert` — it prints a Grid-formatted message and aborts cleanly under MPI.
-- Logging is stream-based, not macro-based: `std::cout << GridLogMessage << ... << std::endl;`. Channels declared in `Grid/log/Log.h` include `GridLogError`, `GridLogWarning`, `GridLogDebug`, `GridLogPerformance`, `GridLogIterative`, `GridLogSolver`, `GridLogHMC`, `GridLogComms`, `GridLogMemory`, `GridLogDslash`, `GridLogIRL`, `GridLogMG`. A subset is switched on at runtime with e.g. `--log Error,Warning,Message,Performance,Iterative,Integrator,Debug,Colours` (names given without the `GridLog` prefix).
-- Performance-critical paths use `GRID_TRACE(name)` from `Grid/perfmon/Tracing.h` (compiled out unless `--enable-tracing` selects a backend) and the `GridStopWatch` timers in `Grid/perfmon/Timer.h`.
+- Logging uses `Grid_log`, `Grid_error` macros (from `Grid/log/`); performance-critical paths use the `GRID_TRACE` / timer macros from `Grid/perfmon/`.
 - Reductions across MPI ranks go through `GridBase::GlobalSum` / `GlobalMax`; never reduce with bare MPI calls inside library code.
-- Everything lives in `NAMESPACE_BEGIN(Grid)` / `NAMESPACE_END(Grid)` macros; follow the surrounding file rather than writing `namespace Grid { }`.
-- British spelling is used in identifiers and comments (`colour`, `neighbour`, `serialisation`).

Grid/algorithms/Algorithms.h

@@ -53,6 +53,7 @@ NAMESPACE_CHECK(approx);
 #include <Grid/algorithms/deflation/MultiRHSBlockCGLinalg.h>
 // Not really deflation, but useful
 #include <Grid/algorithms/blas/MomentumProject.h>
+#include <Grid/algorithms/blas/A2ASpatialSum.h>
 NAMESPACE_CHECK(deflation);
 #include <Grid/algorithms/iterative/ConjugateGradient.h>
 NAMESPACE_CHECK(ConjGrad);

Grid/algorithms/blas/A2ASpatialSum.h

@@ -0,0 +1,213 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: Grid/algorithms/blas/A2ASpatialSum.h
+
+    Copyright (C) 2025
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#pragma once
+
+NAMESPACE_BEGIN(Grid);
+
+/*
+  A2ASpatialSum
+
+  Replaces the scalar spatial accumulation loop in A2A extended meson field
+  contractions with a batched GEMM over local time slices, enabling GPU offload.
+
+  Given:
+    leftv[N_i][osite]    — conjugated left SpinColourVectors (SIMD-packed)
+    loopRight[N_j][osite]— type-contracted right SpinColourVectors (SIMD-packed)
+
+  Computes:
+    EMF[i,j,t] = sum_{x,s,c} leftv[i][x,t,s,c] * loopRight[j][x,t,s,c]
+
+  via batched GEMM over nt local time slices, then GlobalSumVector across MPI.
+
+  Memory layout (all C row-major):
+    W_buf  [nt][N_i][nxyz*Nsc]  — W[t][i][x*Nsc+sc]  = leftv[i] at (x,t)
+    LR_buf [nt][N_j][nxyz*Nsc]  — LR[t][j][x*Nsc+sc] = loopRight[j] at (x,t)
+    EMF_buf[nt][N_j][N_i]       — column-major result; EMF[i,j,t] = EMF_buf[t][j][i]
+
+  BLAS call (column-major, OP_T on A so A is read as W[i][k]):
+    C = A^T * B  where A=W[N_i×K C-row], B=LR[N_j×K C-row], C=[N_j×N_i C-row]
+    → C[i,j] = sum_k W[i][k] * LR[j][k] = EMF[i,j] ✓
+*/
+template<class vobj>
+class A2ASpatialSum
+{
+public:
+  typedef typename vobj::scalar_type   scalar;
+  typedef typename vobj::scalar_object sobj;
+
+  GridBase *grid;
+  int N_i, N_j;
+  int nt, nxyz, Nsc;
+
+  deviceVector<scalar>   W_buf;
+  deviceVector<scalar>   LR_buf;
+  deviceVector<scalar>   EMF_buf;
+  deviceVector<scalar *> W_ptrs;
+  deviceVector<scalar *> LR_ptrs;
+  deviceVector<scalar *> EMF_ptrs;
+
+  A2ASpatialSum() : grid(nullptr), N_i(0), N_j(0), nt(0), nxyz(0), Nsc(0) {}
+
+  void Allocate(int _N_i, int _N_j, GridBase *_grid)
+  {
+    grid = _grid;
+    N_i  = _N_i;
+    N_j  = _N_j;
+    Coordinate ldims = grid->LocalDimensions();
+    nt   = ldims[grid->Nd() - 1];
+    nxyz = grid->lSites() / nt;
+    Nsc  = sizeof(sobj) / sizeof(scalar);
+
+    W_buf.resize(nt * N_i * nxyz * Nsc);
+    LR_buf.resize(nt * N_j * nxyz * Nsc);
+    EMF_buf.resize(nt * N_j * N_i);
+
+    // Build persistent batch pointer arrays
+    W_ptrs.resize(nt);
+    LR_ptrs.resize(nt);
+    EMF_ptrs.resize(nt);
+    scalar *Wh   = &W_buf[0];
+    scalar *LRh  = &LR_buf[0];
+    scalar *EMFh = &EMF_buf[0];
+    int lN_i = N_i, lN_j = N_j, lnxyz = nxyz, lNsc = Nsc;
+    for (int t = 0; t < nt; t++) {
+      acceleratorPut(W_ptrs[t],   Wh   + t * lN_i * lnxyz * lNsc);
+      acceleratorPut(LR_ptrs[t],  LRh  + t * lN_j * lnxyz * lNsc);
+      acceleratorPut(EMF_ptrs[t], EMFh + t * lN_j * lN_i);
+    }
+  }
+
+  void PackLeft(const std::vector<Lattice<vobj>> &leftv)
+  {
+    GRID_ASSERT((int)leftv.size() == N_i);
+    PackVectors(leftv, &W_buf[0], N_i);
+  }
+
+  void PackRight(const std::vector<Lattice<vobj>> &loopRight)
+  {
+    GRID_ASSERT((int)loopRight.size() == N_j);
+    PackVectors(loopRight, &LR_buf[0], N_j);
+  }
+
+private:
+  // Pack vecs[N] lattice fields into buf[nt][N][nxyz*Nsc], extracting all SIMD lanes.
+  void PackVectors(const std::vector<Lattice<vobj>> &vecs, scalar *buf, int N)
+  {
+    int nd     = grid->_ndimension;
+    int osites = grid->oSites();
+    int Nsimd  = vobj::Nsimd();
+    int lN     = N;
+    int lNsc   = Nsc;
+    int lnxyz  = nxyz;
+    Coordinate rdimensions = grid->_rdimensions;
+    Coordinate ldims       = grid->LocalDimensions();
+    Coordinate simd        = grid->_simd_layout;
+
+    for (int n = 0; n < N; n++) {
+      autoView(src_v, vecs[n], AcceleratorRead);
+      accelerator_for(sf, osites, Nsimd, {
+#ifdef GRID_SIMT
+        {
+          int lane = acceleratorSIMTlane(Nsimd);
+#else
+          for (int lane = 0; lane < Nsimd; lane++) {
+#endif
+          Coordinate icoor(nd), ocoor(nd), lcoor(nd);
+          Lexicographic::CoorFromIndex(icoor, lane, simd);
+          Lexicographic::CoorFromIndex(ocoor, sf, rdimensions);
+          for (int d = 0; d < nd; d++)
+            lcoor[d] = rdimensions[d] * icoor[d] + ocoor[d];
+
+          int     l_t = lcoor[nd - 1];
+          Coordinate xyz_coor = lcoor;
+          xyz_coor[nd - 1] = 0;
+          int64_t l_xyz;
+          Lexicographic::IndexFromCoor(xyz_coor, l_xyz, ldims);
+
+          sobj    data   = extractLane(lane, src_v[sf]);
+          scalar *data_s = (scalar *)&data;
+
+          int64_t base = (int64_t)l_t * lN * lnxyz * lNsc
+                       + (int64_t)n   * lnxyz * lNsc
+                       + l_xyz * lNsc;
+          for (int sc = 0; sc < lNsc; sc++)
+            buf[base + sc] = data_s[sc];
+        }
+      });
+    }
+  }
+
+public:
+
+  // Batched GEMM + MPI reduction → result[nt_global][N_i][N_j]
+  //
+  // BLAS (column-major, OP_T on A):
+  //   C[N_j×N_i] = A^T[N_i×K] * B[N_j×K]    with K=nxyz*Nsc
+  //   reading A as C row-major [N_i][K] and B as C row-major [N_j][K]
+  //   → C[i,j] = sum_k W[i,k] * LR[j,k] = EMF[i,j] ✓
+  void Sum(Eigen::Tensor<ComplexD, 3> &result)
+  {
+    GridBLAS BLAS;
+
+    int K = nxyz * Nsc;
+    BLAS.gemmBatched(GridBLAS_OP_T, GridBLAS_OP_N,
+                     N_i, N_j, K,
+                     scalar(1.0),
+                     W_ptrs,
+                     LR_ptrs,
+                     scalar(0.0),
+                     EMF_ptrs);
+    BLAS.synchronise();
+
+    // Copy from device and distribute into global-t layout
+    int nt_global = result.dimension(0);
+    int nd        = grid->Nd();
+    int lt_start  = grid->LocalStarts()[nd - 1];
+
+    std::vector<scalar> host_emf(nt * N_j * N_i);
+    acceleratorCopyFromDevice(&EMF_buf[0], host_emf.data(),
+                              nt * N_j * N_i * sizeof(scalar));
+
+    // EMF_buf[t][j*N_i + i] = EMF[i,j] for local t
+    std::vector<scalar> global_emf(nt_global * N_i * N_j, scalar(0.0));
+    for (int lt = 0; lt < nt; lt++) {
+      int gt = lt + lt_start;
+      for (int i = 0; i < N_i; i++)
+      for (int j = 0; j < N_j; j++)
+        global_emf[gt * N_i * N_j + i * N_j + j] = host_emf[lt * N_j * N_i + j * N_i + i];
+    }
+    grid->GlobalSumVector(global_emf.data(), nt_global * N_i * N_j);
+
+    for (int gt = 0; gt < nt_global; gt++)
+    for (int i  = 0; i  < N_i;      i++)
+    for (int j  = 0; j  < N_j;      j++)
+      result(gt, i, j) = global_emf[gt * N_i * N_j + i * N_j + j];
+  }
+};
+
+NAMESPACE_END(Grid);

Grid/algorithms/blas/BatchedBlas.h

@@ -68,59 +68,6 @@ NAMESPACE_BEGIN(Grid);
 enum GridBLASOperation_t { GridBLAS_OP_N, GridBLAS_OP_T, GridBLAS_OP_C } ;
 enum GridBLASPrecision_t { GridBLAS_PRECISION_DEFAULT, GridBLAS_PRECISION_16F, GridBLAS_PRECISION_16BF, GridBLAS_PRECISION_TF32 };
 
-///////////////////////////////////////////////////////////////////////////
-// BLAS scalar constants: the put() wrapper OWNS the residency policy so
-// call sites just pass values (scalars-live-on-the-host rule).
-//
-// Policy per backend:
-//  - CUDA / HIP : handles are put in HOST pointer mode at Init (cuBLAS docs
-//    2.2.7: host mode is the documented default; 2.1.5: host-mode scalars
-//    are consumed AT CALL TIME, "can be freed just after the return of the
-//    call even though the kernel launch is asynchronous").  put() stores
-//    the value in persistent host memory and returns its address: ZERO
-//    host->device copies.
-//  - SYCL : the oneMKL group-API alpha/beta arrays are dereferenced
-//    USM-side (spec is silent for the group API; implementation observed
-//    to require USM-accessible storage -- host stack pointers fault).
-//    put() keeps a device-resident copy with VALUE CACHING: the copy is
-//    issued only when the value changes (accumulation pattern
-//    beta = (p==0 ? 0 : 1) costs two copies per Mult instead of npoint).
-//
-// Motivation (rocprof, Frontier, 2026-08-13): per-call alpha/beta device
-// staging generated ~92k tiny staged hipMemcpys in a 12s solve window
-// (~26% of host API time) at the latency-bound coarse level.
-// NB not thread safe -- matches the single-threaded host BLAS call
-// pattern of the per-call staging it replaces.
-///////////////////////////////////////////////////////////////////////////
-template<class T>
-class GridBLASDeviceConstant {
-#ifdef GRID_SYCL
-  deviceVector<T> dev;
-  T               host;
-  int             valid;
-public:
-  GridBLASDeviceConstant() : dev(1), valid(0) {};
-  T * put(T v) {
-    if ( (!valid) || (v != host) ) {
-      acceleratorCopyToDevice((void *)&v,(void *)&dev[0],sizeof(T));
-      host  = v;
-      valid = 1;
-    }
-    return &dev[0];
-  }
-#else
-  // CUDA / HIP in HOST pointer mode (and CPU/Eigen, where the pointer is
-  // unused): persistent host storage, no copies ever.
-  T host;
-public:
-  GridBLASDeviceConstant() {};
-  T * put(T v) {
-    host = v;
-    return &host;
-  }
-#endif
-};
-
 class GridBLAS {
 public:
 
@@ -134,31 +81,11 @@ public:
 #ifdef GRID_CUDA
       std::cout << "cublasCreate"<<std::endl;
       cublasCreate(&gridblasHandle);
-      // HOST pointer mode: scalars consumed at call time from host memory
-      // (cuBLAS docs 2.1.5/2.2.7) -- no device staging of alpha/beta.
-      // DEVICE mode would be a deliberate opt-in for device-produced
-      // scalars (e.g. a future graph-captured solver).
-      cublasSetPointerMode(gridblasHandle, CUBLAS_POINTER_MODE_HOST);
-      {
-        cublasPointerMode_t pm;
-        cublasGetPointerMode(gridblasHandle,&pm);
-        std::cout << "GridBLAS: cuBLAS pointer mode "
-                  << ((pm==CUBLAS_POINTER_MODE_DEVICE)?"DEVICE":"HOST") <<std::endl;
-      }
+      cublasSetPointerMode(gridblasHandle, CUBLAS_POINTER_MODE_DEVICE);
 #endif
 #ifdef GRID_HIP
       std::cout << "hipblasCreate"<<std::endl;
       hipblasCreate(&gridblasHandle);
-      // Explicit HOST mode: the hipBLAS default is UNDOCUMENTED in the
-      // headers (enum 0 == HOST by cuBLAS-mirroring convention only);
-      // set it and print it so every log carries the ground truth.
-      hipblasSetPointerMode(gridblasHandle, HIPBLAS_POINTER_MODE_HOST);
-      {
-        hipblasPointerMode_t pm;
-        hipblasGetPointerMode(gridblasHandle,&pm);
-        std::cout << "GridBLAS: hipBLAS pointer mode "
-                  << ((pm==HIPBLAS_POINTER_MODE_DEVICE)?"DEVICE":"HOST") <<std::endl;
-      }
 #endif
 #ifdef GRID_SYCL
       gridblasHandle = theGridAccelerator;
@@ -313,11 +240,11 @@ public:
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
     
-    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static GridBLASDeviceConstant<ComplexD> alpha_c;
-    static GridBLASDeviceConstant<ComplexD> beta_c;
-    ComplexD *alpha_p = alpha_c.put(alpha);
-    ComplexD *beta_p  = beta_c.put(beta);
+    static deviceVector<ComplexD> alpha_p(1);
+    static deviceVector<ComplexD> beta_p(1);
+    // can prestore the 1 and the zero on device
+    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(ComplexD));
+    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(ComplexD));
     RealD t0=usecond();
     //    std::cout << "ZgemmBatched mnk  "<<m<<","<<n<<","<<k<<" count "<<batchCount<<std::endl;
 #ifdef GRID_HIP
@@ -571,11 +498,11 @@ public:
       lda = k;
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
-    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static GridBLASDeviceConstant<ComplexF> alpha_c;
-    static GridBLASDeviceConstant<ComplexF> beta_c;
-    ComplexF *alpha_p = alpha_c.put(alpha);
-    ComplexF *beta_p  = beta_c.put(beta);
+    static deviceVector<ComplexF> alpha_p(1);
+    static deviceVector<ComplexF> beta_p(1);
+    // can prestore the 1 and the zero on device
+    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(ComplexF));
+    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(ComplexF));
     RealD t0=usecond();
 
     GRID_ASSERT(Bkn.size()==batchCount);
@@ -768,456 +695,6 @@ public:
      RealD bytes = 1.0*sizeof(ComplexF)*(m*k+k*n+m*n)*batchCount;
   }
   
-  ///////////////////////////////////////////////////////////////////////////////////
-  // Explicit-leading-dimension complex single GEMM.
-  //
-  // A,B,C may be SLICES of larger parent allocations: lda/ldb/ldc are the
-  // PARENT strides (>= the compact values the ten-argument overload derives).
-  // Motivating use: software split-K for tiny-output/huge-K dense multiplies
-  // (arXiv:2409.03904 fig 11; cf MultiRHSBlockCGLinalg) -- batch over K-chunks
-  // of a dense slab by pointer offset j*Kchunk with lda = the full K extent,
-  // then reduce the partial C's.  Backends pass lda straight through; only
-  // the compact overload invented them.
-  ///////////////////////////////////////////////////////////////////////////////////
-  void gemmBatched(GridBLASOperation_t OpA,
-		   GridBLASOperation_t OpB,
-		   int m,int n, int k,
-		   ComplexF alpha,
-		   deviceVector<ComplexF*> &Amk, int lda,
-		   deviceVector<ComplexF*> &Bkn, int ldb,
-		   ComplexF beta,
-		   deviceVector<ComplexF*> &Cmn, int ldc,
-		   GridBLASPrecision_t precision = GridBLAS_PRECISION_DEFAULT)
-  {
-    RealD t2=usecond();
-    int32_t batchCount = Amk.size();
-
-    GRID_ASSERT( lda >= ((OpA==GridBLAS_OP_N) ? m : k) );
-    GRID_ASSERT( ldb >= ((OpB==GridBLAS_OP_N) ? k : n) );
-    GRID_ASSERT( ldc >= m );
-
-    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static GridBLASDeviceConstant<ComplexF> alpha_c;
-    static GridBLASDeviceConstant<ComplexF> beta_c;
-    ComplexF *alpha_p = alpha_c.put(alpha);
-    ComplexF *beta_p  = beta_c.put(beta);
-    RealD t0=usecond();
-
-    GRID_ASSERT(Bkn.size()==batchCount);
-    GRID_ASSERT(Cmn.size()==batchCount);
-#ifdef GRID_HIP
-    GRID_ASSERT(precision == GridBLAS_PRECISION_DEFAULT);
-    hipblasOperation_t hOpA;
-    hipblasOperation_t hOpB;
-    if ( OpA == GridBLAS_OP_N ) hOpA = HIPBLAS_OP_N;
-    if ( OpA == GridBLAS_OP_T ) hOpA = HIPBLAS_OP_T;
-    if ( OpA == GridBLAS_OP_C ) hOpA = HIPBLAS_OP_C;
-    if ( OpB == GridBLAS_OP_N ) hOpB = HIPBLAS_OP_N;
-    if ( OpB == GridBLAS_OP_T ) hOpB = HIPBLAS_OP_T;
-    if ( OpB == GridBLAS_OP_C ) hOpB = HIPBLAS_OP_C;
-#if defined(HIP_VERSION_MAJOR) && (HIP_VERSION_MAJOR >=7)
-    auto err = hipblasCgemmBatched(gridblasHandle,
-				   hOpA,
-				   hOpB,
-				   m,n,k,
-				   (hipComplex *) &alpha_p[0],
-				   (hipComplex **)&Amk[0], lda,
-				   (hipComplex **)&Bkn[0], ldb,
-				   (hipComplex *) &beta_p[0],
-				   (hipComplex **)&Cmn[0], ldc,
-				   batchCount);
-#else
-    auto err = hipblasCgemmBatched(gridblasHandle,
-                                   hOpA,
-                                   hOpB,
-                                   m,n,k,
-                                   (hipblasComplex *) &alpha_p[0],
-                                   (hipblasComplex **)&Amk[0], lda,
-                                   (hipblasComplex **)&Bkn[0], ldb,
-                                   (hipblasComplex *) &beta_p[0],
-                                   (hipblasComplex **)&Cmn[0], ldc,
-                                   batchCount);
-#endif
-    GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
-#endif
-#ifdef GRID_CUDA
-    cublasOperation_t hOpA;
-    cublasOperation_t hOpB;
-    if ( OpA == GridBLAS_OP_N ) hOpA = CUBLAS_OP_N;
-    if ( OpA == GridBLAS_OP_T ) hOpA = CUBLAS_OP_T;
-    if ( OpA == GridBLAS_OP_C ) hOpA = CUBLAS_OP_C;
-    if ( OpB == GridBLAS_OP_N ) hOpB = CUBLAS_OP_N;
-    if ( OpB == GridBLAS_OP_T ) hOpB = CUBLAS_OP_T;
-    if ( OpB == GridBLAS_OP_C ) hOpB = CUBLAS_OP_C;
-    cublasStatus_t err;
-    if (precision == GridBLAS_PRECISION_DEFAULT) {
-      err = cublasCgemmBatched(gridblasHandle,
-			       hOpA,
-			       hOpB,
-			       m,n,k,
-			       (cuComplex *) &alpha_p[0],
-			       (cuComplex **)&Amk[0], lda,
-			       (cuComplex **)&Bkn[0], ldb,
-			       (cuComplex *) &beta_p[0],
-			       (cuComplex **)&Cmn[0], ldc,
-			       batchCount);
-    } else {
-      cublasComputeType_t compute_precision = toDataType(precision);
-      err = cublasGemmBatchedEx(gridblasHandle,
-				hOpA,
-				hOpB,
-				m,n,k,
-				(void *) &alpha_p[0],
-				(void **)&Amk[0], CUDA_C_32F, lda,
-				(void **)&Bkn[0], CUDA_C_32F, ldb,
-				(void *) &beta_p[0],
-				(void **)&Cmn[0], CUDA_C_32F, ldc,
-				batchCount, compute_precision, CUBLAS_GEMM_DEFAULT);
-    }
-    GRID_ASSERT(err==CUBLAS_STATUS_SUCCESS);
-#endif
-#ifdef GRID_SYCL
-    GRID_ASSERT(precision == GridBLAS_PRECISION_DEFAULT);
-    int64_t m64=m;
-    int64_t n64=n;
-    int64_t k64=k;
-    int64_t lda64=lda;
-    int64_t ldb64=ldb;
-    int64_t ldc64=ldc;
-    int64_t batchCount64=batchCount;
-
-    oneapi::mkl::transpose iOpA;
-    oneapi::mkl::transpose iOpB;
-
-    if ( OpA == GridBLAS_OP_N ) iOpA = oneapi::mkl::transpose::N;
-    if ( OpA == GridBLAS_OP_T ) iOpA = oneapi::mkl::transpose::T;
-    if ( OpA == GridBLAS_OP_C ) iOpA = oneapi::mkl::transpose::C;
-    if ( OpB == GridBLAS_OP_N ) iOpB = oneapi::mkl::transpose::N;
-    if ( OpB == GridBLAS_OP_T ) iOpB = oneapi::mkl::transpose::T;
-    if ( OpB == GridBLAS_OP_C ) iOpB = oneapi::mkl::transpose::C;
-
-    oneapi::mkl::blas::column_major::gemm_batch(*gridblasHandle,
-						&iOpA,
-						&iOpB,
-						&m64,&n64,&k64,
-						(ComplexF *) &alpha_p[0],
-						(const ComplexF **)&Amk[0], (const int64_t *)&lda64,
-						(const ComplexF **)&Bkn[0], (const int64_t *)&ldb64,
-						(ComplexF *) &beta_p[0],
-						(ComplexF **)&Cmn[0], (const int64_t *)&ldc64,
-						(int64_t)1,&batchCount64,std::vector<sycl::event>());
-    synchronise();
-#endif
-#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
-    GRID_ASSERT(precision == GridBLAS_PRECISION_DEFAULT);
-    // Reference implementation: Eigen with explicit outer stride
-    typedef Eigen::Map<Eigen::MatrixXcf,0,Eigen::OuterStride<> > eMat;
-    if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_N) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk * eBkn ;
-	else
-	  eCmn = alpha * eAmk * eBkn ;
-	});
-    } else if ( (OpA == GridBLAS_OP_C ) && (OpB == GridBLAS_OP_N) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk.adjoint() * eBkn ;
-	else
-	  eCmn = alpha * eAmk.adjoint() * eBkn ;
-	});
-    } else if ( (OpA == GridBLAS_OP_T ) && (OpB == GridBLAS_OP_N) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk.transpose() * eBkn ;
-	else
-	  eCmn = alpha * eAmk.transpose() * eBkn ;
-	});
-    } else if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_C) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk * eBkn.adjoint() ;
-	else
-	  eCmn = alpha * eAmk * eBkn.adjoint() ;
-	});
-    } else if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_T) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk * eBkn.transpose() ;
-	else
-	  eCmn = alpha * eAmk * eBkn.transpose() ;
-	});
-    } else if ( (OpA == GridBLAS_OP_C ) && (OpB == GridBLAS_OP_C) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk.adjoint() * eBkn.adjoint() ;
-	else
-	  eCmn = alpha * eAmk.adjoint() * eBkn.adjoint() ;
-	} );
-    } else if ( (OpA == GridBLAS_OP_T ) && (OpB == GridBLAS_OP_T) ) {
-      thread_for (p, batchCount, {
-	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-	if (std::abs(beta) != 0.0)
-	  eCmn = beta * eCmn + alpha * eAmk.transpose() * eBkn.transpose() ;
-	else
-	  eCmn = alpha * eAmk.transpose() * eBkn.transpose() ;
-	} );
-    } else {
-      assert(0);
-    }
-#endif
-    RealD t1=usecond();
-    RealD flops = 8.0*m*n*k*batchCount;
-    RealD bytes = 1.0*sizeof(ComplexF)*(m*k+k*n+m*n)*batchCount;
-  }
-
-  ///////////////////////////////////////////////////////////////////////////////////
-  // Explicit-leading-dimension complex double GEMM.  Mirror of the ComplexF
-  // overload above; motivating use is the fp64 distributed recursive Schur
-  // inversion (RecursiveSchurInverse), whose operands are column windows of
-  // larger row-slab allocations.
-  ///////////////////////////////////////////////////////////////////////////////////
-  void gemmBatched(GridBLASOperation_t OpA,
-                   GridBLASOperation_t OpB,
-                   int m,int n, int k,
-                   ComplexD alpha,
-                   deviceVector<ComplexD*> &Amk, int lda,
-                   deviceVector<ComplexD*> &Bkn, int ldb,
-                   ComplexD beta,
-                   deviceVector<ComplexD*> &Cmn, int ldc)
-  {
-    RealD t2=usecond();
-    int32_t batchCount = Amk.size();
-
-    GRID_ASSERT( lda >= ((OpA==GridBLAS_OP_N) ? m : k) );
-    GRID_ASSERT( ldb >= ((OpB==GridBLAS_OP_N) ? k : n) );
-    GRID_ASSERT( ldc >= m );
-
-    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static GridBLASDeviceConstant<ComplexD> alpha_c;
-    static GridBLASDeviceConstant<ComplexD> beta_c;
-    ComplexD *alpha_p = alpha_c.put(alpha);
-    ComplexD *beta_p  = beta_c.put(beta);
-    RealD t0=usecond();
-
-    GRID_ASSERT(Bkn.size()==batchCount);
-    GRID_ASSERT(Cmn.size()==batchCount);
-#ifdef GRID_HIP
-    hipblasOperation_t hOpA;
-    hipblasOperation_t hOpB;
-    if ( OpA == GridBLAS_OP_N ) hOpA = HIPBLAS_OP_N;
-    if ( OpA == GridBLAS_OP_T ) hOpA = HIPBLAS_OP_T;
-    if ( OpA == GridBLAS_OP_C ) hOpA = HIPBLAS_OP_C;
-    if ( OpB == GridBLAS_OP_N ) hOpB = HIPBLAS_OP_N;
-    if ( OpB == GridBLAS_OP_T ) hOpB = HIPBLAS_OP_T;
-    if ( OpB == GridBLAS_OP_C ) hOpB = HIPBLAS_OP_C;
-#if defined(HIP_VERSION_MAJOR) && (HIP_VERSION_MAJOR >=7)
-    auto err = hipblasZgemmBatched(gridblasHandle,
-                                   hOpA,
-                                   hOpB,
-                                   m,n,k,
-                                   (hipDoubleComplex *) &alpha_p[0],
-                                   (hipDoubleComplex **)&Amk[0], lda,
-                                   (hipDoubleComplex **)&Bkn[0], ldb,
-                                   (hipDoubleComplex *) &beta_p[0],
-                                   (hipDoubleComplex **)&Cmn[0], ldc,
-                                   batchCount);
-#else
-    auto err = hipblasZgemmBatched(gridblasHandle,
-                                   hOpA,
-                                   hOpB,
-                                   m,n,k,
-                                   (hipblasDoubleComplex *) &alpha_p[0],
-                                   (hipblasDoubleComplex **)&Amk[0], lda,
-                                   (hipblasDoubleComplex **)&Bkn[0], ldb,
-                                   (hipblasDoubleComplex *) &beta_p[0],
-                                   (hipblasDoubleComplex **)&Cmn[0], ldc,
-                                   batchCount);
-#endif
-    GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
-#endif
-#ifdef GRID_CUDA
-    cublasOperation_t hOpA;
-    cublasOperation_t hOpB;
-    if ( OpA == GridBLAS_OP_N ) hOpA = CUBLAS_OP_N;
-    if ( OpA == GridBLAS_OP_T ) hOpA = CUBLAS_OP_T;
-    if ( OpA == GridBLAS_OP_C ) hOpA = CUBLAS_OP_C;
-    if ( OpB == GridBLAS_OP_N ) hOpB = CUBLAS_OP_N;
-    if ( OpB == GridBLAS_OP_T ) hOpB = CUBLAS_OP_T;
-    if ( OpB == GridBLAS_OP_C ) hOpB = CUBLAS_OP_C;
-    auto err = cublasZgemmBatched(gridblasHandle,
-                                  hOpA,
-                                  hOpB,
-                                  m,n,k,
-                                  (cuDoubleComplex *) &alpha_p[0],
-                                  (cuDoubleComplex **)&Amk[0], lda,
-                                  (cuDoubleComplex **)&Bkn[0], ldb,
-                                  (cuDoubleComplex *) &beta_p[0],
-                                  (cuDoubleComplex **)&Cmn[0], ldc,
-                                  batchCount);
-    GRID_ASSERT(err==CUBLAS_STATUS_SUCCESS);
-#endif
-#ifdef GRID_SYCL
-    int64_t m64=m;
-    int64_t n64=n;
-    int64_t k64=k;
-    int64_t lda64=lda;
-    int64_t ldb64=ldb;
-    int64_t ldc64=ldc;
-    int64_t batchCount64=batchCount;
-
-    oneapi::mkl::transpose iOpA;
-    oneapi::mkl::transpose iOpB;
-
-    if ( OpA == GridBLAS_OP_N ) iOpA = oneapi::mkl::transpose::N;
-    if ( OpA == GridBLAS_OP_T ) iOpA = oneapi::mkl::transpose::T;
-    if ( OpA == GridBLAS_OP_C ) iOpA = oneapi::mkl::transpose::C;
-    if ( OpB == GridBLAS_OP_N ) iOpB = oneapi::mkl::transpose::N;
-    if ( OpB == GridBLAS_OP_T ) iOpB = oneapi::mkl::transpose::T;
-    if ( OpB == GridBLAS_OP_C ) iOpB = oneapi::mkl::transpose::C;
-
-    oneapi::mkl::blas::column_major::gemm_batch(*gridblasHandle,
-                                                &iOpA,
-                                                &iOpB,
-                                                &m64,&n64,&k64,
-                                                (ComplexD *) &alpha_p[0],
-                                                (const ComplexD **)&Amk[0], (const int64_t *)&lda64,
-                                                (const ComplexD **)&Bkn[0], (const int64_t *)&ldb64,
-                                                (ComplexD *) &beta_p[0],
-                                                (ComplexD **)&Cmn[0], (const int64_t *)&ldc64,
-                                                (int64_t)1,&batchCount64,std::vector<sycl::event>());
-    synchronise();
-#endif
-#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
-    // Reference implementation: Eigen with explicit outer stride
-    typedef Eigen::Map<Eigen::MatrixXcd,0,Eigen::OuterStride<> > eMat;
-    if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_N) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk * eBkn;
-        }
-        else
-        {
-          eCmn = alpha * eAmk * eBkn;
-        }
-      });
-    } else if ( (OpA == GridBLAS_OP_C ) && (OpB == GridBLAS_OP_N) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk.adjoint() * eBkn;
-        }
-        else
-        {
-          eCmn = alpha * eAmk.adjoint() * eBkn;
-        }
-      });
-    } else if ( (OpA == GridBLAS_OP_T ) && (OpB == GridBLAS_OP_N) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk.transpose() * eBkn;
-        }
-        else
-        {
-          eCmn = alpha * eAmk.transpose() * eBkn;
-        }
-      });
-    } else if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_C) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk * eBkn.adjoint();
-        }
-        else
-        {
-          eCmn = alpha * eAmk * eBkn.adjoint();
-        }
-      });
-    } else if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_T) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk * eBkn.transpose();
-        }
-        else
-        {
-          eCmn = alpha * eAmk * eBkn.transpose();
-        }
-      });
-    } else if ( (OpA == GridBLAS_OP_C ) && (OpB == GridBLAS_OP_C) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk.adjoint() * eBkn.adjoint();
-        }
-        else
-        {
-          eCmn = alpha * eAmk.adjoint() * eBkn.adjoint();
-        }
-      });
-    } else if ( (OpA == GridBLAS_OP_T ) && (OpB == GridBLAS_OP_T) ) {
-      thread_for (p, batchCount, {
-        eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
-        eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
-        eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
-        if (std::abs(beta) != 0.0)
-        {
-          eCmn = beta * eCmn + alpha * eAmk.transpose() * eBkn.transpose();
-        }
-        else
-        {
-          eCmn = alpha * eAmk.transpose() * eBkn.transpose();
-        }
-      });
-    } else {
-      assert(0);
-    }
-#endif
-    RealD t1=usecond();
-    RealD flops = 8.0*m*n*k*batchCount;
-    RealD bytes = 1.0*sizeof(ComplexD)*(m*k+k*n+m*n)*batchCount;
-  }
-
   ///////////////////////////////////////////////////////////////////////////
   // Single precision real GEMM
   ///////////////////////////////////////////////////////////////////////////
@@ -1244,11 +721,11 @@ public:
       lda = k;
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
-    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static GridBLASDeviceConstant<RealF> alpha_c;
-    static GridBLASDeviceConstant<RealF> beta_c;
-    RealF *alpha_p = alpha_c.put(alpha);
-    RealF *beta_p  = beta_c.put(beta);
+    static deviceVector<RealF> alpha_p(1);
+    static deviceVector<RealF> beta_p(1);
+    // can prestore the 1 and the zero on device
+    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(RealF));
+    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(RealF));
     RealD t0=usecond();
 
     GRID_ASSERT(Bkn.size()==batchCount);
@@ -1405,11 +882,11 @@ public:
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
     
-    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static GridBLASDeviceConstant<RealD> alpha_c;
-    static GridBLASDeviceConstant<RealD> beta_c;
-    RealD *alpha_p = alpha_c.put(alpha);
-    RealD *beta_p  = beta_c.put(beta);
+    static deviceVector<RealD> alpha_p(1);
+    static deviceVector<RealD> beta_p(1);
+    // can prestore the 1 and the zero on device
+    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(RealD));
+    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(RealD));
     RealD t0=usecond();
 
     GRID_ASSERT(Bkn.size()==batchCount);

Grid/algorithms/blas/BatchedInverse.h

@@ -1,282 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: BatchedInverse.h
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#pragma once
-
-#include <Grid/algorithms/blas/BatchedBlas.h>
-
-#ifdef GRID_HIP
-#include <rocsolver/rocsolver.h>
-#endif
-// GRID_CUDA: batched LU inversion lives in cuBLAS (getrfBatched/getriBatched);
-//            cublas_v2.h already included via BatchedBlas.h.
-// GRID_SYCL: oneapi/mkl.hpp already included via BatchedBlas.h (lapack::getrf/getri).
-
-NAMESPACE_BEGIN(Grid);
-
-///////////////////////////////////////////////////////////////////////////////
-// GridBLASInverse: cross-platform batched dense matrix inversion.
-//
-// HIGH LEVEL contract (deliberately NOT a getrf/getrs interface): invert a
-// batch of dense N x N matrices IN PLACE,
-//
-//     A[i] <- A[i]^{-1}       i = 0 .. batchCount-1
-//
-// Layout: column major, lda = N, contiguous per batch element; pointer list
-// exactly as GridBLAS::gemmBatched (deviceVector<T*> of device pointers).
-// Each backend chooses HOW:
-//   HIP  : rocSOLVER getrf_batched + getri_batched
-//   CUDA : cuBLAS getrfBatched + getriBatched (out-of-place getri; workspace
-//          hidden here, result copied back so the surface stays in-place)
-//   SYCL : oneMKL LAPACK getrf + getri per batch element (USM, in-order queue)
-//   CPU  : Eigen PartialPivLU (the correctness oracle for all of the above)
-//
-// The int32 vendor-batched entry points bound N < 2^31 (asserted); the huge
-// single-matrix ILP64 path (getrf_64 + blocked identity-getrs harvest, proven
-// in the dense coarse-coarse setup at N=69120) migrates here as a batch==1
-// large-N dispatch in a follow-up -- the recursive Schur leaves are the
-// batched consumers this surface is shaped for.
-//
-// NB GPU-backend call signatures are written to vendor documentation but the
-// air-gapped development loop compiles only the CPU/Eigen path; verify the
-// rocSOLVER/cuBLAS/oneMKL calls against headers on first device compile.
-// Semantics are locked by the CPU unit test (Test_batched_blas).
-///////////////////////////////////////////////////////////////////////////////
-class GridBLASInverse {
-public:
-
-#ifdef GRID_HIP
-  // rocSOLVER runs on a rocblas_handle (distinct type from hipblasHandle_t)
-  static rocblas_handle & Handle(void) {
-    static rocblas_handle h;
-    static int init = 0;
-    if ( !init ) {
-      auto st = rocblas_create_handle(&h);
-      GRID_ASSERT(st == rocblas_status_success);
-      init = 1;
-    }
-    return h;
-  }
-#endif
-#ifdef GRID_CUDA
-  // cuBLAS batched LU shares the GridBLAS handle
-  static cublasHandle_t & Handle(void) {
-    GridBLAS::Init();
-    return GridBLAS::gridblasHandle;
-  }
-#endif
-#ifdef GRID_SYCL
-  static sycl::queue * & Handle(void) {
-    GridBLAS::Init();
-    return GridBLAS::gridblasHandle;
-  }
-#endif
-
-  GridBLASInverse() {};
-  ~GridBLASInverse() {};
-
-  void inverseBatched(int64_t N, deviceVector<ComplexF*> &Amat)
-  {
-    int32_t batchCount = Amat.size();
-    GRID_ASSERT(batchCount > 0);
-
-#ifdef GRID_HIP
-    GRID_ASSERT( N < 2147483647L );
-    rocblas_int n   = (rocblas_int)N;
-    rocblas_int lda = (rocblas_int)N;
-
-    deviceVector<rocblas_int> ipiv((uint64_t)batchCount*N);
-    deviceVector<rocblas_int> info(batchCount);
-
-    auto st1 = rocsolver_cgetrf_batched(Handle(), n, n,
-					(rocblas_float_complex *const *)&Amat[0], lda,
-					&ipiv[0], (rocblas_stride)N,
-					&info[0], batchCount);
-    GRID_ASSERT(st1 == rocblas_status_success);
-    auto st2 = rocsolver_cgetri_batched(Handle(), n,
-					(rocblas_float_complex *const *)&Amat[0], lda,
-					&ipiv[0], (rocblas_stride)N,
-					&info[0], batchCount);
-    GRID_ASSERT(st2 == rocblas_status_success);
-    accelerator_barrier();
-    std::vector<rocblas_int> info_h(batchCount);
-    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(rocblas_int));
-    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0); // singular pivot => abort loudly
-#endif
-#ifdef GRID_CUDA
-    GRID_ASSERT( N < 2147483647L );
-    int n = (int)N;
-
-    deviceVector<int> ipiv((uint64_t)batchCount*N);
-    deviceVector<int> info(batchCount);
-
-    auto st1 = cublasCgetrfBatched(Handle(), n,
-				   (cuComplex **)&Amat[0], n,
-				   &ipiv[0], &info[0], batchCount);
-    GRID_ASSERT(st1 == CUBLAS_STATUS_SUCCESS);
-
-    // getri is OUT of place: hidden workspace keeps the surface in-place
-    deviceVector<ComplexF>  work((uint64_t)batchCount*N*N);
-    deviceVector<ComplexF*> Cptr(batchCount);
-    std::vector<ComplexF*>  Cptr_h(batchCount);
-    std::vector<ComplexF*>  Aptr_h(batchCount);
-    for(int i=0;i<batchCount;i++) Cptr_h[i] = &work[(uint64_t)i*N*N];
-    acceleratorCopyToDevice(&Cptr_h[0],&Cptr[0],batchCount*sizeof(ComplexF*));
-    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexF*));
-
-    auto st2 = cublasCgetriBatched(Handle(), n,
-				   (const cuComplex *const *)&Amat[0], n,
-				   &ipiv[0],
-				   (cuComplex **)&Cptr[0], n,
-				   &info[0], batchCount);
-    GRID_ASSERT(st2 == CUBLAS_STATUS_SUCCESS);
-    accelerator_barrier();
-    std::vector<int> info_h(batchCount);
-    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(int));
-    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0);
-    for(int i=0;i<batchCount;i++)
-      acceleratorCopyDeviceToDevice(Cptr_h[i],Aptr_h[i],(uint64_t)N*N*sizeof(ComplexF));
-#endif
-#ifdef GRID_SYCL
-    // Per-element oneMKL LAPACK on the in-order queue; group API optimisation later.
-    sycl::queue *q = Handle();
-    std::vector<ComplexF*> Aptr_h(batchCount);
-    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexF*));
-
-    int64_t lwf = oneapi::mkl::lapack::getrf_scratchpad_size<std::complex<float> >(*q,N,N,N);
-    int64_t lwi = oneapi::mkl::lapack::getri_scratchpad_size<std::complex<float> >(*q,N,N);
-    deviceVector<ComplexF> scratchf(lwf);
-    deviceVector<ComplexF> scratchi(lwi);
-    deviceVector<int64_t>  ipiv(N);
-    for(int i=0;i<batchCount;i++){
-      oneapi::mkl::lapack::getrf(*q,N,N,(std::complex<float>*)Aptr_h[i],N,&ipiv[0],
-				 (std::complex<float>*)&scratchf[0],lwf);
-      oneapi::mkl::lapack::getri(*q,N,  (std::complex<float>*)Aptr_h[i],N,&ipiv[0],
-				 (std::complex<float>*)&scratchi[0],lwi);
-    }
-    q->wait();
-#endif
-#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
-    // Reference implementation; the oracle the unit test locks semantics with.
-    thread_for (p, batchCount, {
-      Eigen::Map<Eigen::MatrixXcf> eA(Amat[p],N,N);
-      Eigen::PartialPivLU<Eigen::MatrixXcf> lu(eA);
-      eA = lu.inverse();
-      });
-#endif
-  }
-
-  void inverseBatched(int64_t N, deviceVector<ComplexD*> &Amat)
-  {
-    int32_t batchCount = Amat.size();
-    GRID_ASSERT(batchCount > 0);
-
-#ifdef GRID_HIP
-    GRID_ASSERT( N < 2147483647L );
-    rocblas_int n   = (rocblas_int)N;
-    rocblas_int lda = (rocblas_int)N;
-
-    deviceVector<rocblas_int> ipiv((uint64_t)batchCount*N);
-    deviceVector<rocblas_int> info(batchCount);
-
-    auto st1 = rocsolver_zgetrf_batched(Handle(), n, n,
-					(rocblas_double_complex *const *)&Amat[0], lda,
-					&ipiv[0], (rocblas_stride)N,
-					&info[0], batchCount);
-    GRID_ASSERT(st1 == rocblas_status_success);
-    auto st2 = rocsolver_zgetri_batched(Handle(), n,
-					(rocblas_double_complex *const *)&Amat[0], lda,
-					&ipiv[0], (rocblas_stride)N,
-					&info[0], batchCount);
-    GRID_ASSERT(st2 == rocblas_status_success);
-    accelerator_barrier();
-    std::vector<rocblas_int> info_h(batchCount);
-    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(rocblas_int));
-    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0);
-#endif
-#ifdef GRID_CUDA
-    GRID_ASSERT( N < 2147483647L );
-    int n = (int)N;
-
-    deviceVector<int> ipiv((uint64_t)batchCount*N);
-    deviceVector<int> info(batchCount);
-
-    auto st1 = cublasZgetrfBatched(Handle(), n,
-				   (cuDoubleComplex **)&Amat[0], n,
-				   &ipiv[0], &info[0], batchCount);
-    GRID_ASSERT(st1 == CUBLAS_STATUS_SUCCESS);
-
-    deviceVector<ComplexD>  work((uint64_t)batchCount*N*N);
-    deviceVector<ComplexD*> Cptr(batchCount);
-    std::vector<ComplexD*>  Cptr_h(batchCount);
-    std::vector<ComplexD*>  Aptr_h(batchCount);
-    for(int i=0;i<batchCount;i++) Cptr_h[i] = &work[(uint64_t)i*N*N];
-    acceleratorCopyToDevice(&Cptr_h[0],&Cptr[0],batchCount*sizeof(ComplexD*));
-    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexD*));
-
-    auto st2 = cublasZgetriBatched(Handle(), n,
-				   (const cuDoubleComplex *const *)&Amat[0], n,
-				   &ipiv[0],
-				   (cuDoubleComplex **)&Cptr[0], n,
-				   &info[0], batchCount);
-    GRID_ASSERT(st2 == CUBLAS_STATUS_SUCCESS);
-    accelerator_barrier();
-    std::vector<int> info_h(batchCount);
-    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(int));
-    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0);
-    for(int i=0;i<batchCount;i++)
-      acceleratorCopyDeviceToDevice(Cptr_h[i],Aptr_h[i],(uint64_t)N*N*sizeof(ComplexD));
-#endif
-#ifdef GRID_SYCL
-    sycl::queue *q = Handle();
-    std::vector<ComplexD*> Aptr_h(batchCount);
-    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexD*));
-
-    int64_t lwf = oneapi::mkl::lapack::getrf_scratchpad_size<std::complex<double> >(*q,N,N,N);
-    int64_t lwi = oneapi::mkl::lapack::getri_scratchpad_size<std::complex<double> >(*q,N,N);
-    deviceVector<ComplexD> scratchf(lwf);
-    deviceVector<ComplexD> scratchi(lwi);
-    deviceVector<int64_t>  ipiv(N);
-    for(int i=0;i<batchCount;i++){
-      oneapi::mkl::lapack::getrf(*q,N,N,(std::complex<double>*)Aptr_h[i],N,&ipiv[0],
-				 (std::complex<double>*)&scratchf[0],lwf);
-      oneapi::mkl::lapack::getri(*q,N,  (std::complex<double>*)Aptr_h[i],N,&ipiv[0],
-				 (std::complex<double>*)&scratchi[0],lwi);
-    }
-    q->wait();
-#endif
-#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
-    thread_for (p, batchCount, {
-      Eigen::Map<Eigen::MatrixXcd> eA(Amat[p],N,N);
-      Eigen::PartialPivLU<Eigen::MatrixXcd> lu(eA);
-      eA = lu.inverse();
-      });
-#endif
-  }
-};
-
-NAMESPACE_END(Grid);

Grid/algorithms/iterative/AdefGeneric.h

@@ -53,22 +53,7 @@ class TwoLevelCG : public LinearFunction<Field>
   // Fine operator, Smoother, CoarseSolver
   LinearOperatorBase<Field>   &_FineLinop;
   LinearFunction<Field>   &_Smoother;
-
-  GridStopWatch ProjectTimer;
-  GridStopWatch PromoteTimer;
-  GridStopWatch CoarseTimer;
-  GridStopWatch SmoothTimer;
-  GridStopWatch MatrixTimer;
-  GridStopWatch M3Timer;
-  GridStopWatch LinalgTimer;
-
-  int64_t M3Calls;
-  int64_t SmoothCalls;
-  int64_t MatrixCalls;
-  int64_t ProjectCalls;
-  int64_t CoarseCalls;
-  int64_t PromoteCalls;
-
+  
   // more most opertor functions
   TwoLevelCG(RealD tol,
 	     Integer maxit,
@@ -118,20 +103,12 @@ class TwoLevelCG : public LinearFunction<Field>
     RealD tn;
     
     GridStopWatch HDCGTimer;
-    ProjectTimer.Reset();
-    PromoteTimer.Reset();
-    CoarseTimer.Reset();
-    SmoothTimer.Reset();
-    MatrixTimer.Reset();
-    M3Timer.Reset();
-    LinalgTimer.Reset();
-    M3Calls = SmoothCalls = MatrixCalls = ProjectCalls = CoarseCalls = PromoteCalls = 0;
     HDCGTimer.Start();
     //////////////////////////
     // x0 = Vstart -- possibly modify guess
     //////////////////////////
     Vstart(x,src);
-
+    
     // r0 = b -A x0
     _FineLinop.HermOp(x,mmp[0]);
     axpy (r, -1.0,mmp[0], src);    // Recomputes r=src-Ax0
@@ -168,40 +145,33 @@ class TwoLevelCG : public LinearFunction<Field>
       int peri_kp = (k+1) % mmax;
 
       rtz=rtzp;
-      M3Timer.Start();
       d= PcgM3(p[peri_k],mmp[peri_k]);
-      M3Timer.Stop();
-      M3Calls++;
       a = rtz/d;
-
+    
       // Memorise this
       pAp[peri_k] = d;
-
-      LinalgTimer.Start();
+      
       axpy(x,a,p[peri_k],x);
       RealD rn = axpy_norm(r,-a,mmp[peri_k],r);
-      LinalgTimer.Stop();
 
       // Compute z = M x
       PcgM1(r,z);
-
+      
       {
 	RealD n1,n2;
 	n1=norm2(r);
 	n2=norm2(z);
 	std::cout << GridLogMessage<<"HDCG::fPcg iteration "<<k<<" : vector r,z "<<n1<<" "<<n2<<"\n";
       }
-      LinalgTimer.Start();
       rtzp =real(innerProduct(r,z));
-      LinalgTimer.Stop();
       std::cout << GridLogMessage<<"HDCG::fPcg iteration "<<k<<" : inner rtzp "<<rtzp<<"\n";
 
       //    PcgM2(z,p[0]);
       PcgM2(z,mu); // ADEF-2 this is identity. Axpy possible to eliminate
-
+      
       p[peri_kp]=mu;
 
-      // Standard search direction  p -> z + b p
+      // Standard search direction  p -> z + b p    
       b = (rtzp)/rtz;
       
       int northog;
@@ -232,25 +202,8 @@ class TwoLevelCG : public LinearFunction<Field>
       if ( rn <= rsq ) { 
 
 	HDCGTimer.Stop();
-	std::cout<<GridLogMessage<<"HDCG: fPcg converged in "<<k<<" iterations and "<<HDCGTimer.Elapsed()<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg breakdown"<<std::endl;
-	auto mspc = [](GridStopWatch &sw, int64_t n) -> double {
-	  return (n > 0) ? sw.useconds() * 1e-3 / n : 0.0;
-	};
-	std::cout<<GridLogMessage<<"HDCG: fPcg M3 (fine MVM)  "<<M3Timer.Elapsed()
-		 <<"  "<<M3Calls<<" calls  "<<mspc(M3Timer,M3Calls)<<" ms/call"<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg linalg         "<<LinalgTimer.Elapsed()<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg smoother       "<<SmoothTimer.Elapsed()
-		 <<"  "<<SmoothCalls<<" calls  "<<mspc(SmoothTimer,SmoothCalls)<<" ms/call"<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg matrix (in M1) "<<MatrixTimer.Elapsed()
-		 <<"  "<<MatrixCalls<<" calls  "<<mspc(MatrixTimer,MatrixCalls)<<" ms/call"<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg project        "<<ProjectTimer.Elapsed()
-		 <<"  "<<ProjectCalls<<" calls  "<<mspc(ProjectTimer,ProjectCalls)<<" ms/call"<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg coarse         "<<CoarseTimer.Elapsed()
-		 <<"  "<<CoarseCalls<<" calls  "<<mspc(CoarseTimer,CoarseCalls)<<" ms/call"<<std::endl;
-	std::cout<<GridLogMessage<<"HDCG: fPcg promote        "<<PromoteTimer.Elapsed()
-		 <<"  "<<PromoteCalls<<" calls  "<<mspc(PromoteTimer,PromoteCalls)<<" ms/call"<<std::endl;
-
+	std::cout<<GridLogMessage<<"HDCG: fPcg converged in "<<k<<" iterations and "<<HDCGTimer.Elapsed()<<std::endl;;
+	
 	_FineLinop.HermOp(x,mmp[0]);			  
 	axpy(tmp,-1.0,src,mmp[0]);
 	
@@ -522,29 +475,35 @@ class TwoLevelADEF2 : public TwoLevelCG<Field>
     CoarseField PleftProj(this->coarsegrid);
     CoarseField PleftMss_proj(this->coarsegrid);
 
-    this->SmoothTimer.Start();
+    GridStopWatch SmootherTimer;
+    GridStopWatch MatrixTimer;
+    SmootherTimer.Start();
     this->_Smoother(in,Min);
-    this->SmoothTimer.Stop();
-    this->SmoothCalls++;
+    SmootherTimer.Stop();
 
-    this->MatrixTimer.Start();
+    MatrixTimer.Start();
     this->_FineLinop.HermOp(Min,out);
-    this->MatrixTimer.Stop();
-    this->MatrixCalls++;
+    MatrixTimer.Stop();
     axpy(tmp,-1.0,out,in);          // tmp  = in - A Min
 
-    this->ProjectTimer.Start();
-    this->_Aggregates.ProjectToSubspace(PleftProj,tmp);
-    this->ProjectTimer.Stop();
-    this->ProjectCalls++;
-    this->CoarseTimer.Start();
+    GridStopWatch ProjTimer;
+    GridStopWatch CoarseTimer;
+    GridStopWatch PromTimer;
+    ProjTimer.Start();
+    this->_Aggregates.ProjectToSubspace(PleftProj,tmp);     
+    ProjTimer.Stop();
+    CoarseTimer.Start();
     this->_CoarseSolver(PleftProj,PleftMss_proj); // Ass^{-1} [in - A Min]_s
-    this->CoarseTimer.Stop();
-    this->CoarseCalls++;
-    this->PromoteTimer.Start();
-    this->_Aggregates.PromoteFromSubspace(PleftMss_proj,tmp);// tmp = Q[in - A Min]
-    this->PromoteTimer.Stop();
-    this->PromoteCalls++;
+    CoarseTimer.Stop();
+    PromTimer.Start();
+    this->_Aggregates.PromoteFromSubspace(PleftMss_proj,tmp);// tmp = Q[in - A Min]  
+    PromTimer.Stop();
+    std::cout << GridLogPerformance << "PcgM1 breakdown "<<std::endl;
+    std::cout << GridLogPerformance << "\tSmoother   " << SmootherTimer.Elapsed() <<std::endl;
+    std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
+    std::cout << GridLogPerformance << "\tProj       " << ProjTimer.Elapsed() <<std::endl;
+    std::cout << GridLogPerformance << "\tCoarse     " << CoarseTimer.Elapsed() <<std::endl;
+    std::cout << GridLogPerformance << "\tProm       " << PromTimer.Elapsed() <<std::endl;
 
     axpy(out,1.0,Min,tmp); // Min+tmp
   }

Grid/algorithms/iterative/ConjugateGradient.h

@@ -212,17 +212,13 @@ public:
 		  << "\tTarget " << Tolerance << std::endl;
 
 	//	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tConstruct  " << ConstructTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tNorm       " << NormTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tAssign     " << AssignTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tSolver     " << SolverTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "Solver breakdown "<<std::endl;
-	std::cout << GridLogMessage << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tSolver Elapsed    " << SolverTimer.Elapsed() <<std::endl;
+        std::cout << GridLogPerformance << "Time breakdown "<<std::endl;
+	std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
+	std::cout << GridLogPerformance << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
+	std::cout << GridLogPerformance << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
+	std::cout << GridLogPerformance << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
+	std::cout << GridLogPerformance << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
 
 	std::cout << GridLogDebug << "\tMobius flop rate " << DwfFlops/ usecs<< " Gflops " <<std::endl;
 

Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h

@@ -236,5 +236,4 @@ public:
   }
 };
 NAMESPACE_END(Grid);
-#undef GCRLogLevel
 #endif

Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h

@@ -38,14 +38,13 @@ Author: Peter Boyle <paboyle@ph.ed.ac.uk>
 ///////////////////////////////////////////////////////////////////////////////////////////////////////
 NAMESPACE_BEGIN(Grid);
 
-#define GCRLogLevel std::cout << GridLogMessage <<std::string(level,'\t')<< name<<" "
+#define GCRLogLevel std::cout << GridLogMessage <<std::string(level,'\t')<< " Level "<<level<<" " 
 
 template<class Field>
 class PrecGeneralisedConjugateResidualNonHermitian : public LinearFunction<Field> {
 public:                                                
   using LinearFunction<Field>::operator();
   RealD   Tolerance;
-  RealD   SSQ;
   Integer MaxIterations;
   int verbose;
   int mmax;
@@ -55,18 +54,11 @@ public:
   GridStopWatch PrecTimer;
   GridStopWatch MatTimer;
   GridStopWatch LinalgTimer;
-  std::string name;
-  int ZeroGuess  = 0;  // caller contract: guess is always zero => first-cycle r0 = src, skip the apply
-  int FirstCycle = 0;
 
   LinearFunction<Field>     &Preconditioner;
   LinearOperatorBase<Field> &Linop;
 
-  void Name(std::string _name) { name = _name; };
-
-  void Level(int n) { Name("Level " + std::to_string(n)); level = n; }
-
-  void SetZeroGuess(int z) { ZeroGuess = z; };
+  void Level(int lv) { level=lv; };
 
   PrecGeneralisedConjugateResidualNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,LinearFunction<Field> &Prec,int _mmax,int _nstep) : 
     Tolerance(tol), 
@@ -75,8 +67,8 @@ public:
     Preconditioner(Prec),
     mmax(_mmax),
     nstep(_nstep)
-  {
-    Level(1);
+  { 
+    level=1;
     verbose=1;
   };
 
@@ -85,7 +77,6 @@ public:
     //    psi=Zero();
     RealD cp, ssq,rsq;
     ssq=norm2(src);
-    SSQ=ssq;
     rsq=Tolerance*Tolerance*ssq;
       
     Field r(src.Grid());
@@ -98,12 +89,11 @@ public:
     SolverTimer.Start();
 
     steps=0;
-    FirstCycle=1;
     for(int k=0;k<MaxIterations;k++){
 
       cp=GCRnStep(src,psi,rsq);
 
-      GCRLogLevel <<"PGCR("<<mmax<<","<<nstep<<") "<< steps <<" steps cp = "<<sqrt(cp/ssq)<<" target "<<sqrt(rsq/ssq) <<std::endl;
+      GCRLogLevel <<"PGCR("<<mmax<<","<<nstep<<") "<< steps <<" steps cp = "<<cp<<" target "<<rsq <<std::endl;
 
       if(cp<rsq) {
 
@@ -152,25 +142,21 @@ public:
     GCRLogLevel<< "PGCR nStep("<<nstep<<")"<<std::endl;
 
     //////////////////////////////////
-    // r0 = src - A x0.  ZeroGuess: on the first cycle x0==0 by caller
-    // contract (enforced here), so r0 = src exactly; skip the apply.
-    // Restart cycles (psi!=0) always do the full computation.
+    // initial guess x0 is taken as nonzero.
+    // r0=src-A x0 = src
     //////////////////////////////////
-    if (ZeroGuess && FirstCycle) {
-      psi = Zero();
-      LinalgTimer.Start();
-      r = src;
-      LinalgTimer.Stop();
-    } else {
-      MatTimer.Start();
-      Linop.Op(psi,Az);
-      MatTimer.Stop();
-      LinalgTimer.Start();
-      r=src-Az;
-      LinalgTimer.Stop();
-    }
-    FirstCycle=0;
+    MatTimer.Start();
+    Linop.Op(psi,Az);
+    //    zAz = innerProduct(Az,psi);
+    zAAz= norm2(Az);
+    MatTimer.Stop();
+    
 
+    LinalgTimer.Start();
+    r=src-Az;
+    LinalgTimer.Stop();
+    GCRLogLevel<< "PGCR true residual r = src - A psi   "<<norm2(r) <<std::endl;
+    
     /////////////////////
     // p = Prec(r)
     /////////////////////
@@ -195,7 +181,6 @@ public:
     
     cp =norm2(r);
     LinalgTimer.Stop();
-    GCRLogLevel<< "PGCR true residual "<< sqrt(cp/SSQ)     <<std::endl;
 
     for(int k=0;k<nstep;k++){
 
@@ -214,12 +199,13 @@ public:
       cp = axpy_norm(r,-a,q[peri_k],r);
       LinalgTimer.Stop();
 
-      GCRLogLevel<< "PGCR step["<<steps<<"]  resid " << sqrt(cp/SSQ)<<std::endl;
+      GCRLogLevel<< "PGCR step["<<steps<<"]  resid " << cp << " target " <<rsq<<std::endl; 
 
       if((k==nstep-1)||(cp<rsq)){
 	return cp;
       }
 
+
       PrecTimer.Start();
       Preconditioner(r,z);// solve Az = r
       PrecTimer.Stop();
@@ -253,6 +239,4 @@ public:
   }
 };
 NAMESPACE_END(Grid);
-
-#undef GCRLogLevel
 #endif

Grid/algorithms/multigrid/Aggregates.h

@@ -66,21 +66,7 @@ public:
   {
   };
   
-  void GlobalOrthonormalise(void)
-  {
-    // Normalise all vectors
-    for(int i=0;i<nbasis; i++){
-      RealD scale = std::pow(norm2(subspace[i]),-0.5); 
-      subspace[i] = subspace[i]*scale;
-    }
-    for(int i=0;i<nbasis; i++){
-      for(int j=0;j<i; j++){
-	basisOrthogonalize(subspace,subspace[i],j);
-      }
-      RealD scale = std::pow(norm2(subspace[i]),-0.5); 
-      subspace[i] = subspace[i]*scale;
-    }
-  }  
+  
   void Orthogonalise(void){
     CoarseScalar InnerProd(CoarseGrid); 
     //    std::cout << GridLogMessage <<" Block Gramm-Schmidt pass 1"<<std::endl;
@@ -124,16 +110,14 @@ public:
       
       hermop.Op(noise,Mn); std::cout<<GridLogMessage << "noise   ["<<b<<"] <n|MdagM|n> "<<norm2(Mn)<<std::endl;
 
-      for(int i=0;i<2;i++){
+      for(int i=0;i<4;i++){
 
 	CG(hermop,noise,subspace[b]);
 
 	noise = subspace[b];
 	scale = std::pow(norm2(noise),-0.5); 
 	noise=noise*scale;
-	
-	hermop.Op(noise,Mn); std::cout<<GridLogMessage << "intermediate["<<i<<"] <i|MdagM|i> "<<norm2(Mn)<<std::endl;
-	
+
       }
 
       hermop.Op(noise,Mn); std::cout<<GridLogMessage << "filtered["<<b<<"] <f|MdagM|f> "<<norm2(Mn)<<std::endl;
@@ -150,8 +134,7 @@ public:
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,10,DiracOp,simple_fine,30,30);
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,10,DiracOp,simple_fine,12,12);
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,30,DiracOp,simple_fine,12,12);
-    //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.0005,30,DiracOp,simple_fine,20,20);
-    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.0005,30,DiracOp,simple_fine,10,10);
+    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,30,DiracOp,simple_fine,10,10);
     FineField noise(FineGrid);
     FineField src(FineGrid);
     FineField guess(FineGrid);
@@ -184,8 +167,6 @@ public:
 	noise = subspace[b];
 	scale = std::pow(norm2(noise),-0.5); 
 	noise=noise*scale;
-	
-	DiracOp.Op(noise,Mn); std::cout<<GridLogMessage << "intermediate["<<i<<"] <f|Op|f> "<<innerProduct(noise,Mn)<<" <f|OpDagOp|f>"<<norm2(Mn)<<std::endl;
 
       }
 
@@ -193,7 +174,6 @@ public:
       subspace[b]   = noise;
 
     }
-    GlobalOrthonormalise();
   }
 
   ////////////////////////////////////////////////////////////////////////////////////////////////

Grid/algorithms/multigrid/GeneralCoarsenedMatrix.h

@@ -31,7 +31,6 @@ Author: Peter Boyle <pboyle@bnl.gov>
 
 #include <Grid/lattice/PaddedCell.h>
 #include <Grid/stencil/GeneralLocalStencil.h>
-#include <Grid/algorithms/deflation/MultiRHSBlockProject.h>
 
 NAMESPACE_BEGIN(Grid);
 
@@ -67,10 +66,6 @@ public:
   std::vector<CoarseMatrix> _Adag;
   std::vector<CoarseVector> MultTemporaries;
 
-  int64_t MultCalls;
-  double  MultFlopsAccum;
-  double  MultUsecAccum;
-
   ///////////////////////
   // Interface
   ///////////////////////
@@ -109,20 +104,19 @@ public:
   }
   */
   
-  GeneralCoarsenedMatrix(NonLocalStencilGeometry &_geom,GridBase *FineGrid, GridCartesian * CoarseGrid,int _herm=1)
+  GeneralCoarsenedMatrix(NonLocalStencilGeometry &_geom,GridBase *FineGrid, GridCartesian * CoarseGrid)
     : geom(_geom),
       _FineGrid(FineGrid),
       _CoarseGrid(CoarseGrid),
-      hermitian(_herm),
+      hermitian(1),
       Cell(_geom.Depth(),_CoarseGrid),
-      Stencil(Cell.grids.back(),geom.shifts),
-      MultCalls(0), MultFlopsAccum(0.0), MultUsecAccum(0.0)
+      Stencil(Cell.grids.back(),geom.shifts)
   {
     {
       int npoint = _geom.npoint;
     }
     _A.resize(geom.npoint,CoarseGrid);
-    if ( !hermitian ) _Adag.resize(geom.npoint,CoarseGrid);
+    //    _Adag.resize(geom.npoint,CoarseGrid);
   }
   void M (const CoarseVector &in, CoarseVector &out)
   {
@@ -130,10 +124,10 @@ public:
   }
   void Mdag (const CoarseVector &in, CoarseVector &out)
   {
-    if(hermitian)
-      Mult(_A,in,out);
-    else
-      Mult(_Adag,in,out);
+    GRID_ASSERT(hermitian);
+    Mult(_A,in,out);
+    //    if ( hermitian ) M(in,out);
+    //    else Mult(_Adag,in,out);
   }
   void Mult (std::vector<CoarseMatrix> &A,const CoarseVector &in, CoarseVector &out)
   {
@@ -233,28 +227,29 @@ public:
     text+=usecond();
     ttot+=usecond();
     
-    MultCalls++;
-    MultFlopsAccum += flops;
-    MultUsecAccum  += ttot;
-    std::cout << GridLogPerformance
-	      << "Coarse Mult call " << MultCalls
-	      << "  tot " << ttot << " us"
-	      << "  kernel " << tmult << " us"
-	      << "  kernel " << flops/tmult*1e-3 << " GFlop/s"
-	      << "  overall " << MultFlopsAccum/MultUsecAccum*1e-3 << " GFlop/s (cumul)"
-	      << "  bw " << bytes/tmult*1e-3 << " GB/s"
-	      << std::endl;
+    std::cout << GridLogPerformance<<"Coarse 1rhs Mult Aviews "<<tviews<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Mult exch "<<texch<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Mult mult "<<tmult<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<" of which mult2  "<<tmult2<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Mult ext  "<<text<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Mult temps "<<ttemps<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Mult copy  "<<tcopy<<" us"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Mult tot  "<<ttot<<" us"<<std::endl;
+    //    std::cout << GridLogPerformance<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Kernel flops "<< flops<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Kernel flop/s "<< flops/tmult<<" mflop/s"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse Kernel bytes/s "<< bytes/tmult<<" MB/s"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse overall flops/s "<< flops/ttot<<" mflop/s"<<std::endl;
+    std::cout << GridLogPerformance<<"Coarse total bytes   "<< bytes/1e6<<" MB"<<std::endl;
 
   };
   
   void PopulateAdag(void)
   {
-#if 0
-    // Serial global peek/poke reference implementation
     for(int64_t bidx=0;bidx<CoarseGrid()->gSites() ;bidx++){
       Coordinate bcoor;
       CoarseGrid()->GlobalIndexToGlobalCoor(bidx,bcoor);
-
+      
       for(int p=0;p<geom.npoint;p++){
 	Coordinate scoor = bcoor;
 	for(int mu=0;mu<bcoor.size();mu++){
@@ -267,36 +262,6 @@ public:
 	pokeSite(adj(link),_Adag[pp],bcoor);
       }
     }
-#else
-    // Parallel: _Adag[pp](x) = adj( _A[p](x + s_pp) ),  pp = Reverse(p), s_pp = -s_p.
-    // The neighbour fetch reuses the same padded-cell + stencil machinery as Mult,
-    // reading one matrix element per coalesced access so no whole site matrix
-    // (230KB at nbasis=60) ever lands on a GPU thread stack (HIP limit 128KB).
-    // Halo sites compute garbage neighbours; Cell.Extract discards them.
-    // Must run on the unpadded _A, i.e. before ExchangeCoarseLinks.
-    const int Nsimd = CComplex::Nsimd();
-    for(int p=0;p<geom.npoint;p++){
-      int pp = geom.Reverse(p);
-      CoarseMatrix Apad = Cell.ExchangePeriodic(_A[p]);
-      CoarseMatrix Dpad(Apad.Grid());
-      int64_t osites = Apad.Grid()->oSites();
-      {
-	autoView( Apad_v   , Apad, AcceleratorRead);
-	autoView( Dpad_v   , Dpad, AcceleratorWriteDiscard);
-	autoView( Stencil_v, Stencil, AcceleratorRead);
-	accelerator_for(sj, osites*nbasis, Nsimd, {
-	    int32_t ss = sj/nbasis;
-	    int32_t j  = sj%nbasis;
-	    auto SE = Stencil_v.GetEntry(pp,ss);
-	    for(int i=0;i<nbasis;i++){
-	      auto z = coalescedReadGeneralPermute(Apad_v[SE->_offset](i,j),SE->_permute,Nd);
-	      coalescedWrite(Dpad_v[ss](j,i),conjugate(z));
-	    }
-	});
-      }
-      _Adag[pp] = Cell.Extract(Dpad);
-    }
-#endif
   }
   /////////////////////////////////////////////////////////////
   // 
@@ -452,19 +417,10 @@ public:
 	int osites=CoarseGrid()->oSites();
 	autoView( A_v  , _A[k], AcceleratorWrite);
 	autoView( FT_v  , FT[k], AcceleratorRead);
-	accelerator_for(sss, osites, nbasis, {
-#ifdef GRID_SIMT
-	    int j = acceleratorSIMTlane(nbasis);
-	    A_v[sss](i,j) = FT_v[sss](j);
-#else
-	    // CPU build: acceleratorSIMTlane()==0 -- an un-looped SIMT tensor
-	    // index writes ONLY j=0 and silently drops the other nbasis-1
-	    // columns (caught by Test_schur_dense_coarse import certificate,
-	    // 2026-08-14).  Loop explicitly.
+	accelerator_for(sss, osites, 1, {
 	    for(int j=0;j<nbasis;j++){
 	      A_v[sss](i,j) = FT_v[sss](j);
 	    }
-#endif
         });
       }
       tinv+=usecond();
@@ -472,8 +428,8 @@ public:
 
     // Only needed if nonhermitian
     if ( ! hermitian ) {
-      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
-      PopulateAdag();
+      //      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
+      //      PopulateAdag();
     }
 
     // Need to write something to populate Adag from A
@@ -561,9 +517,13 @@ public:
     // Now compute the matrix elements of linop between the orthonormal
     // set of vectors.
     ///////////////////////////////////////////////////////////////////////
+    FineField phaV(grid); // Phased block basis vector
+    FineField MphaV(grid);// Matrix applied
     std::vector<FineComplexField> phaF(npoint,grid);
     std::vector<CoarseComplexField> pha(npoint,CoarseGrid());
-
+    
+    CoarseVector coarseInner(CoarseGrid());
+    
     typedef typename CComplex::scalar_type SComplex;
     FineComplexField one(grid); one=SComplex(1.0);
     FineComplexField zz(grid); zz = Zero();
@@ -582,52 +542,37 @@ public:
       pha[p]  =exp(pha[p]*ci);
 
       blockZAXPY(phaF[p],pha[p],one,zz);
-
+      
     }
     tphase+=usecond();
-
-    // Import basis into BLAS layout once; blockProject then reads it once per
-    // basis vector rather than once per (i,p) as in scalar blockProject.
-    // Process all npoint in a single batch.
-    MultiRHSBlockProject<FineField> Projector;
-    Projector.Allocate(nbasis, grid, CoarseGrid());
-    Projector.ImportBasis(U.subspace);
-
-    std::vector<FineField>    phaV_batch(npoint, grid);
-    std::vector<FineField>    MphaV_batch(npoint, grid);
-    std::vector<CoarseVector> proj_batch(npoint, CoarseGrid());
-    std::vector<CoarseVector> ComputeProj(npoint, CoarseGrid());
-    std::vector<CoarseVector>          FT(npoint, CoarseGrid());
-
-    // Pre-allocate BLAS_F and BLAS_C to avoid repeated hipMalloc/hipFree of
-    // ~5.6 GB per blockProject call, which hangs on ROCm for large allocations.
-    Projector.BLAS_F.resize(Projector.fine_vol * Projector.words * npoint);
-    Projector.BLAS_C.resize(Projector.coarse_vol * nbasis * npoint);
-
+    
+    std::vector<CoarseVector> ComputeProj(npoint,CoarseGrid());
+    std::vector<CoarseVector>          FT(npoint,CoarseGrid());
     for(int i=0;i<nbasis;i++){// Loop over basis vectors
-      accelerator_barrier(); // ensure prior iteration's async writes are retired
       std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<"/"<<nbasis<< std::endl;
+      for(int p=0;p<npoint;p++){ // Loop over momenta in npoint
+	tphaseBZ-=usecond();
+	phaV = phaF[p]*V.subspace[i];
+	tphaseBZ+=usecond();
 
-      tphaseBZ-=usecond();
-      for(int p=0;p<npoint;p++)
-	phaV_batch[p] = phaF[p] * V.subspace[i];
-      tphaseBZ+=usecond();
-      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" phaseBZ done"<< std::endl;
+	/////////////////////////////////////////////////////////////////////
+	// Multiple phased subspace vector by matrix and project to subspace
+	// Remove local bulk phase to leave relative phases
+	/////////////////////////////////////////////////////////////////////
+	tmat-=usecond();
+	linop.Op(phaV,MphaV);
+	tmat+=usecond();
+	//	std::cout << i << " " <<p << " MphaV "<<norm2(MphaV)<<" "<<norm2(phaV)<<std::endl;
 
-      tmat-=usecond();
-      for(int p=0;p<npoint;p++)
-	linop.Op(phaV_batch[p], MphaV_batch[p]);
-      tmat+=usecond();
-      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" mat done"<< std::endl;
+	tproj-=usecond();
+	blockProject(coarseInner,MphaV,U.subspace);
+	coarseInner = conjugate(pha[p]) * coarseInner;
 
-      // One batched GEMM reads BLAS_V once for all npoint vectors.
-      tproj-=usecond();
-      Projector.blockProject(MphaV_batch, proj_batch);
-      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" blockProject done"<< std::endl;
-      for(int p=0;p<npoint;p++)
-	ComputeProj[p] = conjugate(pha[p]) * proj_batch[p];
-      tproj+=usecond();
-      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" proj done"<< std::endl;
+	ComputeProj[p] = coarseInner;
+	tproj+=usecond();
+	//	std::cout << i << " " <<p << " ComputeProj "<<norm2(ComputeProj[p])<<std::endl;
+
+      }
 
       tinv-=usecond();
       for(int k=0;k<npoint;k++){
@@ -635,23 +580,14 @@ public:
 	for(int l=0;l<npoint;l++){
 	  FT[k]= FT[k]+ invMkl(l,k)*ComputeProj[l];
 	}
-
+      
 	int osites=CoarseGrid()->oSites();
 	autoView( A_v  , _A[k], AcceleratorWrite);
 	autoView( FT_v  , FT[k], AcceleratorRead);
-	accelerator_for(sss, osites, nbasis, {
-#ifdef GRID_SIMT
-	    int j = acceleratorSIMTlane(nbasis);
-	    A_v[sss](i,j) = FT_v[sss](j);
-#else
-	    // CPU build: acceleratorSIMTlane()==0 -- an un-looped SIMT tensor
-	    // index writes ONLY j=0 and silently drops the other nbasis-1
-	    // columns (caught by Test_schur_dense_coarse import certificate,
-	    // 2026-08-14).  Loop explicitly.
+	accelerator_for(sss, osites, 1, {
 	    for(int j=0;j<nbasis;j++){
 	      A_v[sss](i,j) = FT_v[sss](j);
 	    }
-#endif
         });
       }
       tinv+=usecond();
@@ -659,13 +595,13 @@ public:
 
     // Only needed if nonhermitian
     if ( ! hermitian ) {
-      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
-      PopulateAdag();
+      //      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
+      //      PopulateAdag();
     }
 
-    //    for(int p=0;p<geom.npoint;p++){
-    //      std::cout << " _A["<<p<<"] "<<norm2(_A[p])<<std::endl;
-    //    }
+    for(int p=0;p<geom.npoint;p++){
+      std::cout << " _A["<<p<<"] "<<norm2(_A[p])<<std::endl;
+    }
 
     // Need to write something to populate Adag from A
     ExchangeCoarseLinks();
@@ -680,7 +616,7 @@ public:
   void ExchangeCoarseLinks(void){
     for(int p=0;p<geom.npoint;p++){
       _A[p] = Cell.ExchangePeriodic(_A[p]);
-      if ( !hermitian ) _Adag[p]= Cell.ExchangePeriodic(_Adag[p]);
+      //      _Adag[p]= Cell.ExchangePeriodic(_Adag[p]);
     }
   }
   virtual  void Mdiag    (const Field &in, Field &out){ GRID_ASSERT(0);};

Grid/algorithms/multigrid/RecursiveSchurInverse.h

@@ -1,647 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: RecursiveSchurInverse.h
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#pragma once
-
-#include <Grid/algorithms/blas/BatchedBlas.h>
-#include <Grid/algorithms/blas/BatchedInverse.h>
-
-NAMESPACE_BEGIN(Grid);
-
-///////////////////////////////////////////////////////////////////////////////
-// RecursiveSchurInverse: distributed dense inversion by recursive Schur
-// complement over a binary rank-range tree.
-//
-// CONTRACT: the caller presents an N x N matrix in RANK-MAJOR row ordering,
-// distributed by rows -- rank r owns global rows [rowStart[r], rowStart[r+1])
-// -- and receives its rows of the INVERSE in the same layout.  This class
-// knows nothing of lattices or coarse operators; it consumes a GridBase for
-// world collectives, GridBLAS for GEMMs and GridBLASInverse for the leaf
-// inversions (all of which have Eigen reference backends, so the whole
-// algorithm unit-tests on a CPU-only laptop build under mpirun).
-//
-// PRECISION (decision 2026-08-14, superseding the fp32-merge design): the
-// ENTIRE inversion runs in fp64 (ComplexD).  The apply-side fp32 gain is
-// taken where it matters -- inside the iterative process -- by rounding the
-// finished inverse ONCE when the caller stores it in the fp32 apply slab.
-// Consequences: merge-growth error accumulates in eps64 and the terminal
-// rounding gives representation-only ~eps32 accuracy independent of growth;
-// the Newton-Schulz refinement and the fp32 escalation ladder are DELETED
-// (resurrectable from git history if a future scale forces reduced-precision
-// merges).  Setup cost: ~2x panel-gather bytes and ~2x transient memory,
-// once per setup; fp64 GEMM runs at fp32 rate on CDNA2/PVC.
-//
-// EXECUTION MODEL: SPMD full-tree walk.  Every rank executes the identical
-// recursion call sequence; participation in DATA is ownership-gated, and
-// every collective is a world-communicator zero-fill GlobalSumVector.  No
-// sub-communicators exist, so no deadlock surface exists.
-//
-// STORAGE CONVENTION (pinned by unit test T1b, Test_schur_inverse.cc):
-// BlockRows is COLUMN-MAJOR with ld = rows, matching the BLAS world:
-// element (i,j) lives at data[ i + j*ld ]; a column window [col0, col0+w)
-// is the contiguous slice starting at data[ col0*ld ].
-///////////////////////////////////////////////////////////////////////////////
-
-///////////////////////////////////////////////////////////////////////////////
-// My rows of a distributed dense matrix: rows x cols, column major, ld = rows.
-///////////////////////////////////////////////////////////////////////////////
-class BlockRows
-{
-public:
-  deviceVector<ComplexD> data;
-  int64_t                rows;
-  int64_t                cols;
-  int64_t                ld;
-
-  BlockRows()
-  {
-    rows = 0;
-    cols = 0;
-    ld   = 0;
-  }
-  void Resize(int64_t r, int64_t c)
-  {
-    rows = r;
-    cols = c;
-    ld   = r;
-    data.resize((uint64_t)r*c);
-  }
-  ComplexD *ColumnWindow(int64_t col0)
-  {
-    GRID_ASSERT( col0 >= 0 );
-    GRID_ASSERT( col0 <= cols );
-    return &data[(uint64_t)col0*ld];
-  }
-};
-
-class RecursiveSchurInverse
-{
-public:
-  GridBase             *grid;       // world collectives only
-  int64_t               N;          // global matrix dimension
-  int                   P;          // ranks
-  int                   me;         // this rank
-  std::vector<int64_t>  rowStart;   // P+1 entries: rank-major row ownership
-  int64_t               myRow0;
-  int64_t               myNrows;
-  int64_t               panelBytes; // gather panel budget (DENSE_PANEL_BYTES)
-
-  GridBLAS              BLAS;
-  GridBLASInverse       INV;
-
-  // Growth telemetry (diagnostic, not load-bearing at fp64): one entry per
-  // merge node, walk order
-  std::vector<double>   telNormB;      // ||B||_F = ||A11inv A12||_F
-  std::vector<double>   telSratio;     // ||S||_F / ||A22||_F
-  double                telLeafMaxInv; // max |(leaf inverse)_ij| over leaves
-
-  // Phase timers/counters (this rank), accumulated across the whole walk:
-  // where does the setup wall actually go?  Reported by ReportTelemetry.
-  double                tMemset;       // device panel zero-fill
-  double                tDeposit;      // owner rows -> panel (device kernel)
-  double                tAllreduce;    // GlobalSumVector on device panels
-  double                tGemm;         // strided gemm + synchronise
-  double                tLeaf;         // leaf inversions
-  double                tARmin;        // fastest single panel collective
-  double                tARmax;        // slowest single panel collective
-  uint64_t              bytesAllreduce;
-  uint64_t              nAllreduce;    // panel collectives
-  uint64_t              nGatherGemm;   // GatherGemm calls
-
-  // PERSISTENT DEVICE panel, grow-only across the whole walk.
-  // Persistent: fresh per-call allocations defeat the MPI registration
-  // cache (measured 13% at N=138k).  DEVICE-RESIDENT: the collective runs
-  // on the device pointer via GPU-aware MPI, and panel assembly (zero-fill
-  // + owner deposit) is device-side -- the entire host round-trip (stage
-  // D2H, 92s memset, deposit, H2D at N=138k) is deleted.  Measured
-  // motivation (MPI_benchmark/gather_mpi, Frontier, 16-128 ranks):
-  // host-buffer allreduce 0.8-1.3 GB/s vs device-buffer 5.9-6.8 GB/s;
-  // device allgatherv/ring 20-24 GB/s is the planned stage-2 pattern.
-  // DEVICE BUILDS REQUIRE GPU-aware MPI (MPICH_GPU_SUPPORT_ENABLED=1,
-  // standard in production); CPU builds are unaffected (device == host).
-  deviceVector<ComplexD> dPanelBuf;
-
-  ///////////////////////////////////////////////////////////////////////////
-  // Ownership-table validation: a proper partition of [0,N).
-  // Static and communicator-free so synthetic tables unit-test directly.
-  ///////////////////////////////////////////////////////////////////////////
-  static void CheckRowStart(const std::vector<int64_t> &table, int64_t N)
-  {
-    int P = (int)table.size() - 1;
-    GRID_ASSERT( P >= 1 );
-    GRID_ASSERT( table[0] == 0 );
-    GRID_ASSERT( table[P] == N );
-    for(int r=0; r<P; r++)
-    {
-      GRID_ASSERT( table[r+1] >= table[r] );  // zero-row ranks permitted
-    }
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // Build the ownership table from each rank's local row count: zero-fill
-  // allgather (the standing comms idiom) then prefix sum.  Every rank
-  // returns the identical table.
-  ///////////////////////////////////////////////////////////////////////////
-  static std::vector<int64_t> MakeRowStart(GridBase *g, int64_t myNrows)
-  {
-    int P  = g->ProcessorCount();
-    int me = g->ThisRank();
-
-    std::vector<uint64_t> counts(P, 0);
-    counts[me] = (uint64_t)myNrows;
-    g->GlobalSumVector(&counts[0], P);
-
-    std::vector<int64_t> table(P+1);
-    table[0] = 0;
-    for(int r=0; r<P; r++)
-    {
-      table[r+1] = table[r] + (int64_t)counts[r];
-    }
-    CheckRowStart(table, table[P]);
-    return table;
-  }
-
-  RecursiveSchurInverse(GridBase *g,
-                        int64_t N_,
-                        std::vector<int64_t> &rowStart_,
-                        int64_t panelBytes_)
-  {
-    grid       = g;
-    N          = N_;
-    P          = g->ProcessorCount();
-    me         = g->ThisRank();
-    rowStart   = rowStart_;
-    panelBytes = panelBytes_;
-
-    GRID_ASSERT( (int)rowStart.size() == P+1 );
-    CheckRowStart(rowStart, N);
-
-    myRow0  = rowStart[me];
-    myNrows = rowStart[me+1] - rowStart[me];
-
-    telLeafMaxInv = 0.0;
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // THE communication primitive (plan 3.4 / 4B.3).
-  //
-  //   C(:, colC : colC+widthB) <-   beta  * C(:, colC : colC+widthB)
-  //                               + alpha * A(:, colA : colA+widthA) * Bsub
-  //
-  // Bsub is the widthA x widthB sub-block of a row-distributed operand
-  // owned by ranks [rB0, rB1): owner r contributes its rows of
-  // B(:, colB : colB+widthB) at sub-block row offset
-  // rowStart[r] - rowStart[rB0].  The sub-block is gathered in panelBytes
-  // row-chunks, DEVICE-NATIVE: device zero-fill + device deposit kernel +
-  // world GlobalSumVector on the device pointer (GPU-aware MPI on device
-  // builds; see the dPanelBuf comment for the measured motivation).
-  //
-  // SPMD rules: EVERY rank calls (the collectives are world-wide);
-  // non-owners of B add zeros; ranks with A.rows == 0 skip all local
-  // compute but still make every collective call.  Column offsets are
-  // LOCAL buffer offsets -- non-participants pass 0.
-  ///////////////////////////////////////////////////////////////////////////
-  void GatherGemm(ComplexD alpha,
-                  BlockRows &A, int64_t colA, int64_t widthA,
-                  int rB0, int rB1,
-                  BlockRows &B, int64_t colB, int64_t widthB,
-                  ComplexD beta,
-                  BlockRows &C, int64_t colC)
-  {
-    GRID_ASSERT( rB0 >= 0 );
-    GRID_ASSERT( rB1 >  rB0 );
-    GRID_ASSERT( rB1 <= P );
-
-    int64_t k = rowStart[rB1] - rowStart[rB0];
-    int64_t m = A.rows;
-    int64_t n = widthB;
-    GRID_ASSERT( widthA == k );
-    GRID_ASSERT( n >= 1 );
-
-    int owner = ( me >= rB0 ) && ( me < rB1 ) && ( B.rows > 0 );
-    int64_t myOff = 0;
-    if ( owner )
-    {
-      myOff = rowStart[me] - rowStart[rB0];
-    }
-
-    if ( m > 0 )
-    {
-      GRID_ASSERT( colA + widthA <= A.cols );
-      GRID_ASSERT( colC + widthB <= C.cols );
-      GRID_ASSERT( C.rows == m );
-    }
-
-    nGatherGemm++;
-    GRID_TRACE("GatherGemm");
-
-    if ( owner )
-    {
-      GRID_ASSERT( colB + widthB <= B.cols );
-    }
-
-    int64_t kc = panelBytes / ( (int64_t)sizeof(ComplexD) * n );
-    if ( kc < 1 ) kc = 1;
-    if ( kc > k ) kc = k;
-    GRID_ASSERT( kc*n < 2147483647L );   // GlobalSumVector count is int
-
-    deviceVector<ComplexD> &dPanel = dPanelBuf;
-    if ( dPanel.size() < (uint64_t)kc*n ) dPanel.resize((uint64_t)kc*n);
-    deviceVector<ComplexD*> ap(1);
-    deviceVector<ComplexD*> bp(1);
-    deviceVector<ComplexD*> cp(1);
-    std::vector<ComplexD*>  ptr(1);
-
-    for(int64_t k0=0; k0<k; k0+=kc)
-    {
-      int64_t kchunk = std::min(kc, k-k0);
-
-      tMemset -= usecond();
-      acceleratorMemSet(&dPanel[0], 0, (uint64_t)kchunk*n*sizeof(ComplexD));
-      tMemset += usecond();
-      if ( owner )
-      {
-        int64_t i0 = std::max(k0,        myOff);
-        int64_t i1 = std::min(k0+kchunk, myOff+B.rows);
-        if ( i1 > i0 )
-        {
-          // Device-side deposit straight from the B window: strided
-          // block copy as a flat kernel (len rows of n columns).
-          int64_t   len   = i1-i0;
-          int64_t   brows = B.rows;
-          int64_t   dof   = i0-k0;
-          int64_t   sof   = i0-myOff;
-          ComplexD *src   = B.ColumnWindow(colB);
-          ComplexD *dst   = &dPanel[0];
-          tDeposit -= usecond();
-          accelerator_for(idx, (uint64_t)(len*n), 1, {
-            int64_t j = idx / len;
-            int64_t i = idx % len;
-            dst[(uint64_t)(dof + i + j*kchunk)] = src[(uint64_t)(sof + i + j*brows)];
-          });
-          tDeposit += usecond();
-        }
-      }
-      double tar = -usecond();
-      grid->GlobalSumVector(&dPanel[0], (int)(kchunk*n));
-      tar += usecond();
-      tAllreduce += tar;
-      tARmin = std::min(tARmin, tar);
-      tARmax = std::max(tARmax, tar);
-      bytesAllreduce += (uint64_t)kchunk*n*sizeof(ComplexD);
-      nAllreduce++;
-
-      if ( m > 0 )
-      {
-        ComplexD beta_use = ( k0==0 ) ? beta : ComplexD(1.0,0.0);
-
-        ptr[0] = A.ColumnWindow(colA + k0);
-        acceleratorCopyToDevice(&ptr[0], &ap[0], sizeof(ComplexD*));
-        ptr[0] = &dPanel[0];
-        acceleratorCopyToDevice(&ptr[0], &bp[0], sizeof(ComplexD*));
-        ptr[0] = C.ColumnWindow(colC);
-        acceleratorCopyToDevice(&ptr[0], &cp[0], sizeof(ComplexD*));
-
-        tGemm -= usecond();
-        BLAS.gemmBatched(GridBLAS_OP_N, GridBLAS_OP_N,
-                         (int)m, (int)n, (int)kchunk,
-                         alpha,    ap, (int)A.ld,
-                                   bp, (int)kchunk,
-                         beta_use, cp, (int)C.ld);
-        BLAS.synchronise();
-        tGemm += usecond();
-      }
-    }
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // Local Frobenius norm-squared of a full-height column window.
-  // NO comms; callers GlobalSum the result.  Host staging, setup-scale.
-  ///////////////////////////////////////////////////////////////////////////
-  double FrobNorm2Local(BlockRows &X, int64_t col0, int64_t w)
-  {
-    if ( X.rows == 0 ) return 0.0;
-    GRID_ASSERT( col0 + w <= X.cols );
-    uint64_t len = (uint64_t)X.rows*w;
-    std::vector<ComplexD> h(len);
-    acceleratorCopyFromDevice(X.ColumnWindow(col0), &h[0], len*sizeof(ComplexD));
-    // Member real()/imag(): portable across std::complex (CPU) and
-    // thrust::complex (HIP), where std::norm does not resolve.
-    double s = 0.0;
-    for(uint64_t i=0; i<len; i++)
-    {
-      double re = h[i].real();
-      double im = h[i].imag();
-      s += re*re + im*im;
-    }
-    return s;
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // dst(:, dcol0 : dcol0+w) = - src(:, 0:w).  Both operands have ld == rows
-  // so full-height windows are contiguous: flat elementwise device copy.
-  ///////////////////////////////////////////////////////////////////////////
-  void NegateCopy(BlockRows &src, BlockRows &dst, int64_t dcol0, int64_t w)
-  {
-    GRID_ASSERT( src.rows == dst.rows );
-    GRID_ASSERT( w <= src.cols );
-    GRID_ASSERT( dcol0 + w <= dst.cols );
-    if ( src.rows == 0 ) return;
-    uint64_t len = (uint64_t)src.rows*w;
-    ComplexD *s = &src.data[0];
-    ComplexD *d = dst.ColumnWindow(dcol0);
-    accelerator_for(i, len, 1, {
-      d[i] = -s[i];
-    });
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // Leaf inversion.  Purely LOCAL -- the calling rank owns the whole
-  // width x width leaf (width == my row count); no collectives, so the
-  // SPMD walk stays uniform with other ranks doing nothing.  The window
-  // is contiguous (ld == rows == width): invert IN PLACE via
-  // GridBLASInverse.  Everything is already fp64; no promote/demote.
-  ///////////////////////////////////////////////////////////////////////////
-  void LeafInvert(int64_t col0, int64_t width, BlockRows &Arows)
-  {
-    GRID_TRACE("SchurLeaf");
-    GRID_ASSERT( width == Arows.rows );
-    GRID_ASSERT( col0 + width <= Arows.cols );
-    int64_t  w   = width;
-    uint64_t len = (uint64_t)w*w;
-    tLeaf -= usecond();
-
-    deviceVector<ComplexD*> bp(1);
-    std::vector<ComplexD*>  ptr(1);
-    ptr[0] = Arows.ColumnWindow(col0);
-    acceleratorCopyToDevice(&ptr[0], &bp[0], sizeof(ComplexD*));
-    INV.inverseBatched(w, bp);
-
-    // Telemetry: max |element| of the leaf inverse
-    {
-      std::vector<ComplexD> h(len);
-      acceleratorCopyFromDevice(Arows.ColumnWindow(col0), &h[0], len*sizeof(ComplexD));
-      double mx = 0.0;
-      for(uint64_t i=0; i<len; i++)
-      {
-        double re = h[i].real();
-        double im = h[i].imag();
-        mx = std::max(mx, re*re + im*im);
-      }
-      telLeafMaxInv = std::max(telLeafMaxInv, std::sqrt(mx));
-    }
-    tLeaf += usecond();
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // The recursion (plan 3.5 / 4B.3).  Inverts the diagonal block of the
-  // rank-major matrix spanned by ranks [r0, r1), living in every member
-  // rank's column window [col0, col0+width) -- IN PLACE.
-  //
-  // SPMD: every rank calls with IDENTICAL (r0, r1, width) and its own
-  // local (col0, Arows); ranks outside [r0, r1) participate in the
-  // collectives only (dummy operands, zero contributions).  The collective
-  // sequence -- 5 GatherGemm calls + 3 scalar GlobalSums per merge node --
-  // is identical on every rank by construction.
-  //
-  //   I = [r0, mid)   J = [mid, r1)   widths WI, WJ
-  //   1. recurse I:      A11 -> A11inv
-  //   2. B = A11inv.A12                     (I rows)
-  //   3. C = A21.A11inv                     (J rows)
-  //   4. S = A22 - A21.B    in place        (J rows)   [alpha=-1, beta=1]
-  //   5. recurse J:      S -> Sinv
-  //   6. T = Sinv.C                         (J rows)
-  //   7. U = B.Sinv                         (I rows)
-  //   8. X11 = A11inv + U.C  in place       (I rows)   [beta=1]
-  //   9. X12 = -U, X21 = -T  local negates; X22 = Sinv already in place
-  ///////////////////////////////////////////////////////////////////////////
-  void SchurNode(int r0, int r1, int64_t col0, int64_t width, BlockRows &Arows)
-  {
-    int span = r1 - r0;
-    GRID_ASSERT( span >= 1 );
-    GRID_ASSERT( width == rowStart[r1] - rowStart[r0] );
-
-    if ( span == 1 )
-    {
-      if ( ( me == r0 ) && ( myNrows > 0 ) )
-      {
-        LeafInvert(col0, width, Arows);
-      }
-      return;
-    }
-
-    int     mid = ( r0 + r1 ) / 2;
-    int64_t WI  = rowStart[mid] - rowStart[r0];
-    int64_t WJ  = rowStart[r1]  - rowStart[mid];
-
-    // Zero-width child ranges (all ranks of a half owning no rows) are a
-    // KNOWN LIMITATION: fail loudly rather than divide mysteriously.
-    GRID_ASSERT( WI > 0 );
-    GRID_ASSERT( WJ > 0 );
-
-    int inI = ( me >= r0  ) && ( me < mid );
-    int inJ = ( me >= mid ) && ( me < r1  );
-
-    ComplexD one  ( 1.0,0.0);
-    ComplexD mone (-1.0,0.0);
-    ComplexD zero ( 0.0,0.0);
-
-    BlockRows dummy;
-
-    // 1. A11 -> A11inv
-    SchurNode(r0, mid, col0, WI, Arows);
-
-    // 2. B = A11inv . A12   (I rows; gather A12 from I owners)
-    BlockRows Bbuf;
-    if ( inI ) Bbuf.Resize(myNrows, WJ);
-    {
-      BlockRows &Aop = inI ? Arows : dummy;
-      BlockRows &Cop = inI ? Bbuf  : dummy;
-      int64_t    cA  = inI ? col0  : 0;
-      GatherGemm(one, Aop, cA, WI,
-                 r0, mid,
-                 Arows, col0+WI, WJ,
-                 zero, Cop, 0);
-    }
-    double nB = FrobNorm2Local(Bbuf, 0, inI ? WJ : 0);
-    grid->GlobalSumVector(&nB, 1);
-    telNormB.push_back(std::sqrt(nB));
-
-    // 3. C = A21 . A11inv   (J rows; gather A11inv from I owners)
-    BlockRows Cbuf;
-    if ( inJ ) Cbuf.Resize(myNrows, WI);
-    {
-      BlockRows &Aop = inJ ? Arows : dummy;
-      BlockRows &Cop = inJ ? Cbuf  : dummy;
-      int64_t    cA  = inJ ? col0  : 0;
-      GatherGemm(one, Aop, cA, WI,
-                 r0, mid,
-                 Arows, col0, WI,
-                 zero, Cop, 0);
-    }
-
-    // 4. S = A22 - A21 . B  in place on my A22 window (J rows)
-    double nA22 = FrobNorm2Local( inJ ? Arows : dummy, inJ ? col0+WI : 0, inJ ? WJ : 0 );
-    grid->GlobalSumVector(&nA22, 1);
-    {
-      BlockRows &Aop = inJ ? Arows   : dummy;
-      BlockRows &Cop = inJ ? Arows   : dummy;
-      int64_t    cA  = inJ ? col0    : 0;
-      int64_t    cC  = inJ ? col0+WI : 0;
-      GatherGemm(mone, Aop, cA, WI,
-                 r0, mid,
-                 Bbuf, 0, WJ,
-                 one, Cop, cC);
-    }
-    double nS = FrobNorm2Local( inJ ? Arows : dummy, inJ ? col0+WI : 0, inJ ? WJ : 0 );
-    grid->GlobalSumVector(&nS, 1);
-    telSratio.push_back( std::sqrt(nS) / ( std::sqrt(nA22) + 1.0e-300 ) );
-
-    // 5. S -> Sinv
-    SchurNode(mid, r1, col0+WI, WJ, Arows);
-
-    // 6. T = Sinv . C       (J rows; gather C from J owners)
-    BlockRows Tbuf;
-    if ( inJ ) Tbuf.Resize(myNrows, WI);
-    {
-      BlockRows &Aop = inJ ? Arows   : dummy;
-      BlockRows &Cop = inJ ? Tbuf    : dummy;
-      int64_t    cA  = inJ ? col0+WI : 0;
-      GatherGemm(one, Aop, cA, WJ,
-                 mid, r1,
-                 Cbuf, 0, WI,
-                 zero, Cop, 0);
-    }
-
-    // 7. U = B . Sinv       (I rows; gather Sinv from J owners)
-    BlockRows Ubuf;
-    if ( inI ) Ubuf.Resize(myNrows, WJ);
-    {
-      BlockRows &Aop = inI ? Bbuf : dummy;
-      BlockRows &Cop = inI ? Ubuf : dummy;
-      GatherGemm(one, Aop, 0, WJ,
-                 mid, r1,
-                 Arows, col0+WI, WJ,
-                 zero, Cop, 0);
-    }
-
-    // 8. X11 = A11inv + U . C  in place (I rows; gather C from J owners)
-    {
-      BlockRows &Aop = inI ? Ubuf  : dummy;
-      BlockRows &Cop = inI ? Arows : dummy;
-      int64_t    cC  = inI ? col0  : 0;
-      GatherGemm(one, Aop, 0, WJ,
-                 mid, r1,
-                 Cbuf, 0, WI,
-                 one, Cop, cC);
-    }
-
-    // 9. Off-diagonal signs, local
-    if ( inI ) NegateCopy(Ubuf, Arows, col0+WI, WJ);
-    if ( inJ ) NegateCopy(Tbuf, Arows, col0,    WI);
-  }
-
-  ///////////////////////////////////////////////////////////////////////////
-  // PUBLIC ENTRY.  Arows: my rows of the rank-major N x N matrix (fp64).
-  // On exit Arows holds my rows of the inverse, still fp64; the caller
-  // owns the single terminal rounding into its fp32 apply storage.
-  ///////////////////////////////////////////////////////////////////////////
-  void Invert(BlockRows &Arows)
-  {
-    GRID_ASSERT( Arows.rows == myNrows );
-    GRID_ASSERT( Arows.cols == N );
-
-    telNormB.resize(0);
-    telSratio.resize(0);
-    telLeafMaxInv = 0.0;
-
-    tMemset        = 0.0;
-    tDeposit       = 0.0;
-    tAllreduce     = 0.0;
-    tGemm          = 0.0;
-    tLeaf          = 0.0;
-    tARmin         = 1.0e30;
-    tARmax         = 0.0;
-    bytesAllreduce = 0;
-    nAllreduce     = 0;
-    nGatherGemm    = 0;
-
-    SchurNode(0, P, 0, N, Arows);
-
-    RealD mx = telLeafMaxInv;
-    grid->GlobalMax(mx);
-    telLeafMaxInv = mx;
-  }
-
-  // All telemetry values are globally reduced or boss-local; safe to
-  // stream on every rank (Grid quiesces stdout to the boss unless
-  // --debug-stdout).  NOTE: tAllreduce INCLUDES wait/imbalance -- a rank
-  // arriving early books its wait here; the min/max spread across ranks
-  // separates true wire time (~min) from skew (max-min).
-  void ReportTelemetry(void)
-  {
-    for(uint64_t i=0; i<telNormB.size(); i++)
-    {
-      std::cout << GridLogPerformance
-                << "SchurNode " << i
-                << " ||B||_F "       << telNormB[i]
-                << " ||S||/||A22|| " << telSratio[i]
-                << std::endl;
-    }
-    std::cout << GridLogPerformance
-              << "Schur leaves max|Ainv| " << telLeafMaxInv
-              << std::endl;
-
-    RealD armax =  tAllreduce;
-    RealD armin = -tAllreduce;
-    grid->GlobalMax(armax);
-    grid->GlobalMax(armin);
-    armin = -armin;
-
-    std::cout << GridLogMessage << "Schur phases (boss rank, seconds):"
-              << "  memset "    << tMemset/1.0e6
-              << "  deposit "   << tDeposit/1.0e6
-              << "  allreduce " << tAllreduce/1.0e6
-              << "  gemm "      << tGemm/1.0e6
-              << "  leaf "      << tLeaf/1.0e6
-              << std::endl;
-    std::cout << GridLogMessage << "Schur comms:"
-              << "  GatherGemm calls " << nGatherGemm
-              << "  panel allreduces " << nAllreduce
-              << "  allreduce GB "     << bytesAllreduce/1024./1024./1024.
-              << "  allreduce s min/max over ranks " << armin/1.0e6
-              << " / " << armax/1.0e6
-              << std::endl;
-    std::cout << GridLogMessage << "Schur comms per-call (boss):"
-              << "  min " << (nAllreduce ? tARmin/1.0e3 : 0.0) << " ms"
-              << "  avg " << (nAllreduce ? tAllreduce/nAllreduce/1.0e3 : 0.0) << " ms"
-              << "  max " << tARmax/1.0e3 << " ms"
-              << std::endl;
-  }
-};
-
-NAMESPACE_END(Grid);

Grid/allocator/MemoryManager.cc

@@ -63,10 +63,12 @@ void MemoryManager::PrintBytes(void)
   std::cout << " MemoryManager : "<<(total_device>>20)<<" accelerator Mbytes "<<std::endl;
   std::cout << " MemoryManager : "<<(total_host>>20)  <<" cpu         Mbytes "<<std::endl;
   uint64_t cacheBytes;
-  cacheBytes = HostCacheBytes();
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" cpu alloc cache Mbytes "<<std::endl;
-  cacheBytes = DeviceCacheBytes();
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" acc alloc cache Mbytes "<<std::endl;
+  cacheBytes = CacheBytes[Cpu];
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" cpu cache Mbytes "<<std::endl;
+  cacheBytes = CacheBytes[Acc];
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" acc cache Mbytes "<<std::endl;
+  cacheBytes = CacheBytes[Shared];
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" shared cache Mbytes "<<std::endl;
   
 #ifdef GRID_CUDA
   cuda_mem();

Grid/allocator/MemoryManager.h

@@ -113,7 +113,7 @@ private:
   static void *Insert(void *ptr,size_t bytes,AllocationCacheEntry *entries,int ncache,int &victim,uint64_t &cbytes) ;
   static void *Lookup(size_t bytes,AllocationCacheEntry *entries,int ncache,uint64_t &cbytes) ;
 
-public:  
+ public:
   static void PrintBytes(void);
   static void Audit(std::string s);
   static void Init(void);
@@ -215,7 +215,6 @@ public:
   static void NotifyDeletion(void * CpuPtr);
   static void Print(void);
   static void PrintAll(void);
-  static void EvictAll(void);
   static void PrintState( void* CpuPtr);
   static int   isOpen   (void* CpuPtr);
   static void  ViewClose(void* CpuPtr,ViewMode mode);

Grid/allocator/MemoryManagerCache.cc

@@ -79,25 +79,6 @@ void MemoryManager::EntryErase(uint64_t CpuPtr)
   auto AccCache = EntryLookup(CpuPtr);
   AccViewTable.erase(CpuPtr);
 }
-/////////////////////////////////////////////////////////////////////////////////
-// LRU membership invariant:
-//
-//    LRU_valid == 1   <=>   AccPtr != NULL  &&  accLock == 0  &&  cpuLock == 0
-//
-// i.e. the LRU queue contains exactly the device-resident, completely unlocked
-// entries -- the evictable set.  Membership is maintained EAGERLY at the lock
-// 0<->1 edges, O(1) via the stored LRU_entry iterator:
-//
-//    AcceleratorViewOpen   lock 0->1 : LRUremove   (gated on LRU_valid)
-//    AcceleratorViewClose  accLock->0: LRUinsert   (AccPtr necessarily exists)
-//    CpuViewOpen           lock 0->1 : LRUremove   (gated on LRU_valid)
-//    CpuViewClose          cpuLock->0: LRUinsert   (iff AccPtr exists)
-//    Evict/AccDiscard                : LRUremove   (frees the device copy)
-//
-// Consequences: victims taken from LRU.back() are evictable by construction;
-// Evict() on a locked entry is an invariant violation (asserted), and the
-// eviction loops (EvictVictims/EvictAll) cannot spin.
-/////////////////////////////////////////////////////////////////////////////////
 void  MemoryManager::LRUinsert(AcceleratorViewEntry &AccCache)
 {
   GRID_ASSERT(AccCache.LRU_valid==0);
@@ -149,21 +130,21 @@ void MemoryManager::Evict(AcceleratorViewEntry &AccCache)
 {
   ///////////////////////////////////////////////////////////////////////////
   // Make CPU consistent, remove from Accelerator, remove from LRU, LEAVE CPU only entry
-  // Cannot be locked. If allocated must be in LRU pool.
+  // Cannot be acclocked. If allocated must be in LRU pool.
   //
-  // (Historical: a Nov 2022 incident (two CpuPtrs; eviction called from
-  //  CpuViewOpen -- since excised) could present a cpuLocked entry here, and
-  //  silent-return guards were added.  The LRU membership invariant (see
-  //  LRUinsert) now excludes ALL locked entries from the queue eagerly at the
-  //  lock edges, so a locked victim is an invariant violation: asserted.)
+  // Nov 2022... Felix issue: Allocating two CpuPtrs, can have an entry in LRU-q with CPUlock.
+  //                          and require to evict the AccPtr copy. Eviction was a mistake in CpuViewOpen
+  //                          but there is a weakness where CpuLock entries are attempted for erase
+  //                          Take these OUT LRU queue when CPU locked?
+  //                          Cannot take out the table as cpuLock data is important.
   ///////////////////////////////////////////////////////////////////////////
   GRID_ASSERT(AccCache.state!=Empty);
   
   mprintf("MemoryManager: Evict CpuPtr %lx AccPtr %lx cpuLock %ld accLock %ld",
 	  (uint64_t)AccCache.CpuPtr,(uint64_t)AccCache.AccPtr,
 	  (uint64_t)AccCache.cpuLock,(uint64_t)AccCache.accLock); 
-  GRID_ASSERT(AccCache.accLock==0);
-  GRID_ASSERT(AccCache.cpuLock==0);
+  if (AccCache.accLock!=0) return;
+  if (AccCache.cpuLock!=0) return;
   if(AccCache.state==AccDirty) {
     Flush(AccCache);
   }
@@ -269,19 +250,6 @@ void  MemoryManager::EvictVictims(uint64_t bytes)
     }
   }
 }
-void  MemoryManager::EvictAll(void)
-{
-  while(LRU.size()>0){
-    if ( DeviceLRUBytes > 0){
-      uint64_t victim = LRU.back(); // From the LRU
-      auto AccCacheIterator = EntryLookup(victim);
-      auto & AccCache = AccCacheIterator->second;
-      Evict(AccCache);
-    } else {
-      return;
-    }
-  }
-}
 uint64_t MemoryManager::AcceleratorViewOpen(uint64_t CpuPtr,size_t bytes,ViewMode mode,ViewAdvise hint)
 {
   ////////////////////////////////////////////////////////////////////////////
@@ -415,13 +383,6 @@ void MemoryManager::CpuViewClose(uint64_t CpuPtr)
   GRID_ASSERT(AccCache.accLock==0);
 
   AccCache.cpuLock--;
-  // Return to LRU queue when fully unlocked -- mirrors AcceleratorViewClose.
-  // Asymmetry vs the Acc side: a device copy need not exist for a host view;
-  // only device-resident entries belong in the (evictable) LRU queue.
-  if( (AccCache.cpuLock==0) && (AccCache.AccPtr!=(uint64_t)NULL) ) {
-    dprintf("CpuViewClose %lx cpuLock decremented to zero, move to LRU queue",(uint64_t)CpuPtr);
-    LRUinsert(AccCache);
-  }
 }
 /*
  *  Action  State   StateNext         Flush    Clone
@@ -488,14 +449,6 @@ uint64_t MemoryManager::CpuViewOpen(uint64_t CpuPtr,size_t bytes,ViewMode mode,V
     GRID_ASSERT(0); // should be unreachable
   }
 
-  GRID_ASSERT(AccCache.cpuLock>0);
-  // If view is opened on host must remove from LRU -- mirrors AcceleratorViewOpen.
-  // LRU_valid==1 here implies this is the 0->1 lock edge of a device-resident entry.
-  if(AccCache.LRU_valid==1){
-    dprintf("CpuViewOpen: entry removed from LRU ");
-    LRUremove(AccCache);
-  }
-
   AccCache.transient= transient? EvictNext : 0;
 
   return AccCache.CpuPtr;

Grid/communicator/Communicator_base.h

@@ -238,16 +238,6 @@ public:
   }
   void AllToAll(int dim  ,void *in,void *out,uint64_t words,uint64_t bytes);
   void AllToAll(void  *in,void *out,uint64_t words         ,uint64_t bytes);
-
-  ////////////////////////////////////////////////////////////////////////////
-  // Variable count all to all. Counts and displacements are in units of
-  // "bytes" sized words and are indexed by rank within this communicator.
-  // For exchanges that are a permutation but do not divide evenly between
-  // ranks; AllToAll above is the uniform count special case.
-  ////////////////////////////////////////////////////////////////////////////
-  void AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
-                 void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
-                 uint64_t bytes);
   
   template<class obj> void Broadcast(int root,obj &data)
   {

Grid/communicator/Communicator_mpi3.cc

@@ -945,25 +945,5 @@ void CartesianCommunicator::AllToAll(void  *in,void *out,uint64_t words,uint64_t
   MPI_Alltoall(in,iwords,object,out,iwords,object,communicator);
   MPI_Type_free(&object);
 }
-void CartesianCommunicator::AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
-                                      void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
-                                      uint64_t bytes)
-{
-  FlightRecorder::StepLog("AllToAllV");
-  GRID_ASSERT(sendcounts.size()==(size_t)_Nprocessors);
-  GRID_ASSERT(senddispls.size()==(size_t)_Nprocessors);
-  GRID_ASSERT(recvcounts.size()==(size_t)_Nprocessors);
-  GRID_ASSERT(recvdispls.size()==(size_t)_Nprocessors);
-  // MPI counts are "int"; the caller sizes the word to keep them in range
-  int ibytes = bytes;
-  GRID_ASSERT(bytes == (uint64_t)ibytes);
-  MPI_Datatype object;
-  MPI_Type_contiguous(ibytes,MPI_BYTE,&object);
-  MPI_Type_commit(&object);
-  int ierr = MPI_Alltoallv(in ,(int *)&sendcounts[0],(int *)&senddispls[0],object,
-                           out,(int *)&recvcounts[0],(int *)&recvdispls[0],object,communicator);
-  GRID_ASSERT(ierr==0);
-  MPI_Type_free(&object);
-}
 
 NAMESPACE_END(Grid);

Grid/communicator/Communicator_none.cc

@@ -113,17 +113,6 @@ void CartesianCommunicator::AllToAll(void  *in,void *out,uint64_t words,uint64_t
 {
   bcopy(in,out,bytes*words);
 }
-void CartesianCommunicator::AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
-                                      void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
-                                      uint64_t bytes)
-{
-  // Single rank: the exchange degenerates to a copy of our own segment
-  GRID_ASSERT(sendcounts.size()==1);
-  GRID_ASSERT(recvcounts.size()==1);
-  GRID_ASSERT(sendcounts[0]==recvcounts[0]);
-  bcopy((char *)in +(uint64_t)senddispls[0]*bytes,
-        (char *)out+(uint64_t)recvdispls[0]*bytes,bytes*(uint64_t)sendcounts[0]);
-}
 
 int  CartesianCommunicator::RankWorld(void){return 0;}
 void CartesianCommunicator::Barrier(void){}

Grid/lattice/Lattice_base.h

@@ -289,7 +289,7 @@ public:
   ///////////////////////////////////////////
   // move constructor
   ///////////////////////////////////////////
-  Lattice(Lattice && r) noexcept { 
+  Lattice(Lattice && r){ 
     this->_grid = r.Grid();
     this->_odata      = r._odata;
     this->_odata_size = r._odata_size;
@@ -330,7 +330,7 @@ public:
   ///////////////////////////////////////////
   // Move assignment possible if same type
   ///////////////////////////////////////////
-  inline Lattice<vobj> & operator = (Lattice<vobj> && r) noexcept {
+  inline Lattice<vobj> & operator = (Lattice<vobj> && r){
 
     resize(0); // deletes if appropriate
     this->_grid       = r.Grid();

Grid/lattice/Lattice_reduction_gpu.h

@@ -438,11 +438,5 @@ inline typename vobj::scalar_object sum_gpu_large(const vobj *lat, Integer osite
   result = sumD_gpu_large(lat,osites);
   return result;
 }
-template<class Word> Word checksum_gpu(Word *vec,uint64_t L)
-{
-  Word w;
-  bzero(&w,sizeof(w));
-  return w;
-}
 
 NAMESPACE_END(Grid);

Grid/lattice/Lattice_trace.h

@@ -124,68 +124,6 @@ Lattice<iScalar<iScalar<iMatrix<vComplexD, N> > > > Inverse(const Lattice<iScala
   return ret;
 }
 
-template<int N>
-Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > > , N> > Inverse(const Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > >, N> > &Umu)
-{
-  GridBase *grid=Umu.Grid();
-  auto lvol = grid->lSites();
-  Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > >, N > > ret(grid);
-  
-  autoView(Umu_v,Umu,CpuRead);
-  autoView(ret_v,ret,CpuWrite);
-  thread_for(site,lvol,{
-    Eigen::MatrixXcd EigenU = Eigen::MatrixXcd::Zero(N,N);
-    Coordinate lcoor;
-    grid->LocalIndexToLocalCoor(site, lcoor);
-    iMatrix<iScalar<iScalar<iScalar<ComplexD> > >, N > Us;
-    iMatrix<iScalar<iScalar<iScalar<ComplexD> > >, N > Ui;
-    peekLocalSite(Us, Umu_v, lcoor);
-    for(int i=0;i<N;i++){
-      for(int j=0;j<N;j++){
-	EigenU(i,j) = Us(i,j)()()();
-      }}
-    Eigen::MatrixXcd EigenUinv = EigenU.inverse();
-    for(int i=0;i<N;i++){
-      for(int j=0;j<N;j++){
-	Ui(i,j)()()() = EigenUinv(i,j);
-      }}
-    pokeLocalSite(Ui,ret_v,lcoor);
-  });
-  return ret;
-}
-
-template<int N>
-Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > > , N> > Inverse(const Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > >, N> > &Umu)
-{
-  GridBase *grid=Umu.Grid();
-  auto lvol = grid->lSites();
-  Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > >, N > > ret(grid);
-  
-  autoView(Umu_v,Umu,CpuRead);
-  autoView(ret_v,ret,CpuWrite);
-  thread_for(site,lvol,{
-    Eigen::MatrixXcd EigenU = Eigen::MatrixXcd::Zero(N,N);
-    Coordinate lcoor;
-    grid->LocalIndexToLocalCoor(site, lcoor);
-    iMatrix<iScalar<iScalar<iScalar<iScalar<ComplexD> > > >, N > Us;
-    iMatrix<iScalar<iScalar<iScalar<iScalar<ComplexD> > > >, N > Ui;
-    peekLocalSite(Us, Umu_v, lcoor);
-    for(int i=0;i<N;i++){
-      for(int j=0;j<N;j++){
-	EigenU(i,j) = Us(i,j)()()()();
-      }}
-    Eigen::MatrixXcd EigenUinv = EigenU.inverse();
-    for(int i=0;i<N;i++){
-      for(int j=0;j<N;j++){
-	Ui(i,j)()()()() = EigenUinv(i,j);
-      }}
-    pokeLocalSite(Ui,ret_v,lcoor);
-  });
-  return ret;
-}
-
-
-
 
 NAMESPACE_END(Grid);
 #endif

Grid/parallelIO/BinaryIO.cc

@@ -2,7 +2,3 @@
 
 int                    Grid::BinaryIO::latticeWriteMaxRetry = -1;
 Grid::BinaryIO::IoPerf Grid::BinaryIO::lastPerf;
-
-// Target size of a single contiguous file extent under BINARYIO_AGGREGATE.
-// 4MB is around the knee for Lustre; exposed so it can be swept at runtime.
-uint64_t               Grid::BinaryIO::aggregateTargetBytes = 4*1024*1024;

Grid/parallelIO/BinaryIO.h

@@ -39,7 +39,6 @@
 #endif
 
 #include <arpa/inet.h>
-#include <sys/stat.h>
 #include <algorithm>
 
 NAMESPACE_BEGIN(Grid);
@@ -88,7 +87,6 @@ class BinaryIO {
 
   static IoPerf lastPerf;
   static int latticeWriteMaxRetry;
-  static uint64_t aggregateTargetBytes;
 
   /////////////////////////////////////////////////////////////////////////////
   // more byte manipulation helpers
@@ -255,392 +253,12 @@ class BinaryIO {
   // Read or Write distributed lexico array of ANY object to a specific location in file 
   //////////////////////////////////////////////////////////////////////////////////////
 
-  static const int BINARYIO_AGGREGATE     = 0x20;
   static const int BINARYIO_MASTER_APPEND = 0x10;
   static const int BINARYIO_UNORDERED     = 0x08;
   static const int BINARYIO_LEXICOGRAPHIC = 0x04;
   static const int BINARYIO_READ          = 0x02;
   static const int BINARYIO_WRITE         = 0x01;
 
-#ifdef USE_MPI_IO
-  /////////////////////////////////////////////////////////////////////////////
-  // Aggregation: self controlled transposition onto an I/O friendly layout.
-  //
-  // Under BINARYIO_LEXICOGRAPHIC the subarray file view handed to MPI-IO has
-  // contiguous runs of only lLattice[0]*sizeof(fobj) bytes -- a few KB for
-  // typical local volumes. Rather than rely on collective buffering to repair
-  // that, redistribute the payload ourselves so every rank owns a contiguous
-  // range of the global lexicographic site ordering, then issue large plain
-  // contiguous writes.
-  //
-  // "Un-splitting" the nunsplit fastest dimensions means the row of ranks
-  // sharing the remaining process coordinates collectively owns whole global
-  // hyperplanes. All data movement is then confined to that row communicator.
-  // Every rank still owns exactly lSites() sites afterwards, so the exchange is
-  // a pure permutation and needs no divisibility condition on the process grid.
-  /////////////////////////////////////////////////////////////////////////////
-  struct AggregationPlan {
-    int      nunsplit{0};  // number of fastest dimensions un-split
-    int      rowsize{0};   // ranks in the aggregation (row) communicator
-    int      rowrank{0};   // our logical (lexicographic) index within the row
-    uint64_t lsites{0};    // sites per rank -- invariant under the permutation
-    uint64_t chunk{0};     // sites in one globally contiguous run owned by the row
-    std::unique_ptr<CartesianCommunicator> rowcomm;
-    // counts and displacements are indexed by rank within rowcomm
-    std::vector<int>      sendcounts, senddispls, recvcounts, recvdispls;
-    std::vector<uint64_t> scatter;     // recv slot -> slot in the aggregated buffer
-    std::vector<uint64_t> extentGsite; // global lex site index of extent start
-    std::vector<uint64_t> extentLocal; // offset of extent within aggregated buffer
-    std::vector<uint64_t> extentSites; // sites in this extent
-  };
-
-  static inline void BuildAggregationPlan(GridBase *grid,uint64_t fobjSize,AggregationPlan &p)
-  {
-    int ndim           = grid->Dimensions();
-    Coordinate psizes  = grid->ProcessorGrid();
-    Coordinate pcoor   = grid->ThisProcessorCoor();
-    Coordinate gLattice= grid->GlobalDimensions();
-    Coordinate lLattice= grid->LocalDimensions();
-    Coordinate lstart  = grid->LocalStarts();
-
-    uint64_t lsites = grid->lSites();
-    p.lsites = lsites;
-
-    //////////////////////////////////////////////////////////////////////////
-    // Un-splitting dims 0..k-1 gives the row a contiguous run of
-    //     chunk(k) = prod_{d<k} gLattice[d] * lLattice[k]
-    // sites, and each rank writes extents of min(chunk,lsites). Take the
-    // smallest k that reaches the target so we disturb as few dimensions --
-    // and move as little data -- as possible.
-    //////////////////////////////////////////////////////////////////////////
-    int k = ndim-1;
-    for(int trial=1; trial<ndim; trial++){
-      uint64_t chunk = lLattice[trial];
-      for(int d=0; d<trial; d++) chunk *= gLattice[d];
-      if ( std::min(chunk,lsites)*fobjSize >= aggregateTargetBytes ) { k = trial; break; }
-    }
-    p.nunsplit = k;
-
-    //////////////////////////////////////////////////////////////////////////
-    // The box the row collectively owns, expressed in global coordinates.
-    // Restricting the global lexicographic order to this box preserves the
-    // ordering, so the row index below is monotone in the global index.
-    //////////////////////////////////////////////////////////////////////////
-    Coordinate B(ndim), S(ndim);
-    for(int d=0; d<ndim; d++){
-      if ( d<k ) { B[d] = gLattice[d]; S[d] = 0;         }
-      else       { B[d] = lLattice[d]; S[d] = lstart[d]; }
-    }
-
-    uint64_t chunk = lLattice[k];
-    for(int d=0; d<k; d++) chunk *= gLattice[d];
-    p.chunk = chunk;
-
-    //////////////////////////////////////////////////////////////////////////
-    // Row communicator: the ranks sharing the process coordinates of the slow
-    // (still split) dimensions. This is the sub-division the Cartesian
-    // communicator already performs for AllToAll(dim,...), widened from one
-    // dimension to the k fastest.
-    //////////////////////////////////////////////////////////////////////////
-    Coordinate row(ndim,1);
-    for(int d=0; d<k; d++) row[d] = psizes[d];
-    int srank;
-    p.rowcomm.reset(new CartesianCommunicator(row,*grid,srank));
-    p.rowsize = p.rowcomm->ProcessorCount();
-
-    //////////////////////////////////////////////////////////////////////////
-    // Our logical index in the row is the forward lexicographic index of the
-    // un-split process coordinates, so that increasing logical index means
-    // increasing global lexicographic position in the file. The communicator
-    // numbers its own ranks by the reversed (MPI) convention, so build the map
-    // between the two rather than assuming either.
-    //////////////////////////////////////////////////////////////////////////
-    int64_t logical=0, lstride=1;
-    for(int d=0; d<k; d++){ logical += pcoor[d]*lstride; lstride *= psizes[d]; }
-    GRID_ASSERT(lstride == (int64_t)p.rowsize);
-    p.rowrank = (int)logical;
-
-    std::vector<uint64_t> commOf(p.rowsize,0);
-    commOf[p.rowrank] = (uint64_t)p.rowcomm->ThisRank();
-    p.rowcomm->GlobalSumVector(&commOf[0],p.rowsize);
-
-    uint64_t mystart = (uint64_t)p.rowrank * lsites;
-    uint64_t myend   = mystart + lsites;
-
-    Coordinate lcoor(ndim), bcoor(ndim), gcoor(ndim);
-
-    //////////////////////////////////////////////////////////////////////////
-    // Send side. Walking our local sites in local lexicographic order walks
-    // the row index monotonically, so the send buffer is iodata untouched and
-    // we need only the per destination counts.
-    //////////////////////////////////////////////////////////////////////////
-    std::vector<int> sendLogical(p.rowsize,0);
-    for(uint64_t L=0; L<lsites; L++){
-      Lexicographic::CoorFromIndex(lcoor,L,lLattice);
-      for(int d=0; d<ndim; d++) bcoor[d] = (d<k) ? (lstart[d]+lcoor[d]) : lcoor[d];
-      int64_t ri; Lexicographic::IndexFromCoor(bcoor,ri,B);
-      sendLogical[ ri/(int64_t)lsites ]++;
-    }
-    p.sendcounts.assign(p.rowsize,0);
-    p.senddispls.assign(p.rowsize,0);
-    { int64_t disp=0;
-      for(int d=0; d<p.rowsize; d++){          // send buffer is in logical order
-        int c = (int)commOf[d];
-        p.sendcounts[c] = sendLogical[d];
-        p.senddispls[c] = (int)disp;
-        disp += sendLogical[d];
-      }
-    }
-
-    //////////////////////////////////////////////////////////////////////////
-    // Receive side. For each slot of our aggregated range work out which rank
-    // of the row owns it. Within one source the slots arrive in increasing row
-    // index order, which is the order the source sends them in.
-    //////////////////////////////////////////////////////////////////////////
-    std::vector<int> recvLogical(p.rowsize,0), recvDisplLogical(p.rowsize,0);
-    std::vector<int> source(lsites);
-    for(uint64_t pos=0; pos<lsites; pos++){
-      Lexicographic::CoorFromIndex(bcoor,(int64_t)(mystart+pos),B);
-      int64_t j=0, jstride=1;
-      for(int d=0; d<k; d++){ j += (bcoor[d]/lLattice[d])*jstride; jstride *= psizes[d]; }
-      source[pos] = (int)j;
-      recvLogical[j]++;
-    }
-    p.recvcounts.assign(p.rowsize,0);
-    p.recvdispls.assign(p.rowsize,0);
-    { int64_t disp=0;
-      for(int s=0; s<p.rowsize; s++){         // recv buffer is in logical order
-        int c = (int)commOf[s];
-        recvDisplLogical[s] = (int)disp;
-        p.recvcounts[c] = recvLogical[s];
-        p.recvdispls[c] = (int)disp;
-        disp += recvLogical[s];
-      }
-    }
-    p.scatter.resize(lsites);
-    {
-      std::vector<int> fill(p.rowsize,0);
-      for(uint64_t pos=0; pos<lsites; pos++){
-        int j = source[pos];
-        p.scatter[ recvDisplLogical[j] + fill[j]++ ] = pos;
-      }
-    }
-
-    //////////////////////////////////////////////////////////////////////////
-    // The two sides are derived independently; make them check each other.
-    //////////////////////////////////////////////////////////////////////////
-    {
-      std::vector<uint64_t> sendc(p.rowsize),recvc(p.rowsize);
-      for(int c=0;c<p.rowsize;c++) sendc[c]=(uint64_t)p.sendcounts[c];
-      p.rowcomm->AllToAll(&sendc[0],&recvc[0],1,sizeof(uint64_t));
-      for(int c=0;c<p.rowsize;c++) GRID_ASSERT((int)recvc[c]==p.recvcounts[c]);
-    }
-
-    //////////////////////////////////////////////////////////////////////////
-    // Decompose our range into globally contiguous file extents.
-    //////////////////////////////////////////////////////////////////////////
-    for(uint64_t c = mystart/chunk; c <= (myend-1)/chunk; c++){
-      uint64_t lo = std::max(mystart, c*chunk);
-      uint64_t hi = std::min(myend, (c+1)*chunk);
-      Lexicographic::CoorFromIndex(bcoor,(int64_t)(c*chunk),B);
-      for(int d=0;d<ndim;d++) gcoor[d] = (d<k) ? bcoor[d] : bcoor[d]+S[d];
-      int64_t gbase; Lexicographic::IndexFromCoor(gcoor,gbase,gLattice);
-      p.extentGsite.push_back( (uint64_t)gbase + (lo - c*chunk) );
-      p.extentLocal.push_back( lo - mystart );
-      p.extentSites.push_back( hi - lo );
-    }
-  }
-
-  static inline void ReportAggregationPlan(GridBase *grid,const AggregationPlan &p,uint64_t fobjSize,const char *what)
-  {
-    if ( !grid->IsBoss() ) return;
-    std::cout << GridLogMessage << "IOobject: aggregate " << what
-              << " un-splitting " << p.nunsplit << " fastest dimensions, row of "
-              << p.rowsize << " ranks" << std::endl;
-    std::cout << GridLogMessage << "IOobject: aggregate " << p.extentSites.size()
-              << " extent(s)/rank, first " << p.extentSites[0]*fobjSize/1024./1024. << " MB"
-              << " (target " << aggregateTargetBytes/1024./1024. << " MB)" << std::endl;
-    std::cout << GridLogMessage << "IOobject: aggregate buffer overhead "
-              << p.lsites*fobjSize/1024./1024. << " MB/rank" << std::endl;
-  }
-
-  ////////////////////////////////////////////////////////////////////////////
-  // Stage timings.  The interesting quantity is the slowest rank, since every
-  // stage is followed sooner or later by a synchronisation, so reduce with
-  // GlobalMax rather than reporting whatever the boss happened to see.
-  ////////////////////////////////////////////////////////////////////////////
-  static inline void ReportStages(GridBase *grid,const char *what,
-                                  const std::vector<const char *> &names,
-                                  std::vector<RealD> &useconds)
-  {
-    GRID_ASSERT(names.size()==useconds.size());
-    for(uint64_t i=0;i<useconds.size();i++) grid->GlobalMax(useconds[i]);
-    if ( grid->IsBoss() ) {
-      std::cout << GridLogMessage << "IOobject: aggregate " << what << " stages (max over ranks, s):";
-      for(uint64_t i=0;i<names.size();i++)
-        std::cout << " " << names[i] << " " << useconds[i]/1.0e6;
-      std::cout << std::endl;
-    }
-  }
-
-  template<class fobj>
-  static inline void AggregateExchange(GridBase *grid,AggregationPlan &p,std::vector<fobj> &iodata,
-                                       std::vector<fobj> &aggregated,int forward)
-  {
-    uint64_t lsites = p.lsites;
-    GridStopWatch talloc,tperm,tcomm;
-
-    talloc.Start();
-    std::vector<fobj> tmp(lsites);
-    talloc.Stop();
-
-    if ( forward ) { // iodata (local order) -> aggregated (lexicographic order)
-      tcomm.Start();
-      p.rowcomm->AllToAllV(&iodata[0],p.sendcounts,p.senddispls,
-                           &tmp[0],   p.recvcounts,p.recvdispls,sizeof(fobj));
-      tcomm.Stop();
-      tperm.Start();
-      thread_for(s,lsites,{ aggregated[p.scatter[s]] = tmp[s]; });
-      tperm.Stop();
-    } else {         // aggregated -> iodata, the exact mirror
-      tperm.Start();
-      thread_for(s,lsites,{ tmp[s] = aggregated[p.scatter[s]]; });
-      tperm.Stop();
-      tcomm.Start();
-      p.rowcomm->AllToAllV(&tmp[0],   p.recvcounts,p.recvdispls,
-                           &iodata[0],p.sendcounts,p.senddispls,sizeof(fobj));
-      tcomm.Stop();
-    }
-
-    std::vector<RealD> us = { (RealD)talloc.useconds(), (RealD)tperm.useconds(), (RealD)tcomm.useconds() };
-    ReportStages(grid,forward?"exchange (write)":"exchange (read)",
-                 {"alloc","permute","alltoallv"},us);
-  }
-
-  template<class fobj>
-  static inline void AggregateWrite(GridBase *grid,AggregationPlan &p,std::vector<fobj> &aggregated,
-                                    std::string file,uint64_t offset)
-  {
-    //////////////////////////////////////////////////////////////////////////
-    // All ranks write concurrently into a shared file, so the file must exist
-    // before any of them open it for update, but it does NOT have to be the
-    // right length first: the extents tile the record exactly, so writing them
-    // extends a short file to precisely offset+payload.
-    //
-    // Records are created in sequence, so this payload ends the file: the
-    // length must end up precisely offset+payload. Anything beyond is left
-    // over from whatever the file previously held and must not survive -- a
-    // shorter new record written over a longer old one would otherwise leave
-    // a trailing fragment of the previous contents masquerading as data.
-    // That is the only case needing a truncate, so stat first and truncate
-    // afterwards only when the size actually came out wrong. Measured on
-    // Frontier, an unconditional truncate up front cost 0.22 to 5.4 s per
-    // record -- 15 to 25% of a 19 GB write and 100% of a small one -- while
-    // create, open and close together cost a few milliseconds. It is per
-    // record, so multi record files do not amortise it away.
-    //
-    // ::truncate is used because the C++ standard library cannot express this.
-    // std::filebuf has no length operation at all; ios::trunc only truncates to
-    // zero; seeking past the end and writing a byte can grow a file but never
-    // shrink one; and there is no portable way to recover a descriptor from a
-    // stream in order to call ftruncate. C++17 does finally offer
-    // std::filesystem::resize_file, but that would be Grid's first <filesystem>
-    // dependency and needs -lstdc++fs on the older toolchains still in use.
-    //////////////////////////////////////////////////////////////////////////
-    GridStopWatch tcreate,ttrunc,tbar,topen,twrite,tclose,tskew;
-    uint64_t need = offset + (uint64_t)grid->_gsites*sizeof(fobj);
-
-    tcreate.Start();
-    if ( grid->IsBoss() ) {
-      // opening for update needs the file to exist; create one only if not
-      std::fstream probe(file,std::ios::binary|std::ios::out|std::ios::in);
-      if ( !probe.is_open() ) {
-        std::ofstream create(file,std::ios::binary|std::ios::out);
-        create.close();
-      }
-    }
-    tcreate.Stop();
-
-    tbar.Start();
-    grid->Barrier();
-    tbar.Stop();
-
-    std::ofstream fout;
-    fout.exceptions( std::fstream::failbit | std::fstream::badbit );
-    try {
-      topen.Start();
-      fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
-      topen.Stop();
-      twrite.Start();
-      for(uint64_t e=0;e<p.extentSites.size();e++){
-        fout.seekp(offset + p.extentGsite[e]*sizeof(fobj));
-        fout.write((char *)&aggregated[p.extentLocal[e]],p.extentSites[e]*sizeof(fobj));
-      }
-      twrite.Stop();
-      tclose.Start();
-      fout.close();   // flushes the stream buffer; does not force writeback
-      tclose.Stop();
-    } catch (const std::fstream::failure& exc) {
-      std::cout << GridLogError << "Error in aggregate write to " << file << std::endl;
-      std::cout << GridLogError << "Exception description: " << exc.what() << std::endl;
-      GridAbort();
-    }
-
-    ////////////////////////////////////////////////////////////////////////
-    // Timed apart from the truncate that follows it.  seek+write above is the
-    // slowest rank; this barrier is what the fastest rank then waits, so the
-    // pair separates the write cost from the spread across ranks. Folding it
-    // into the truncate makes a millisecond stat look like a second.
-    ////////////////////////////////////////////////////////////////////////
-    tskew.Start();
-    grid->Barrier();                 // every extent must be on its way first
-    tskew.Stop();
-
-    ttrunc.Start();
-    if ( grid->IsBoss() ) {
-      struct stat sb;
-      int ierr = ::stat(file.c_str(),&sb);
-      GRID_ASSERT(ierr==0);
-      if ( (uint64_t)sb.st_size != need ) {   // only when a longer record preceded us
-        ierr = ::truncate(file.c_str(),(off_t)need);
-        GRID_ASSERT(ierr==0);
-      }
-    }
-    grid->Barrier();
-    ttrunc.Stop();
-
-    std::vector<RealD> us = { (RealD)tcreate.useconds(), (RealD)tbar.useconds(),
-                              (RealD)topen.useconds(),   (RealD)twrite.useconds(),
-                              (RealD)tclose.useconds(),  (RealD)tskew.useconds(),
-                              (RealD)ttrunc.useconds() };
-    ReportStages(grid,"write",{"create","barrier","open","seek+write","close","skew","stat+truncate"},us);
-  }
-
-  template<class fobj>
-  static inline void AggregateRead(GridBase *grid,AggregationPlan &p,std::vector<fobj> &aggregated,
-                                   std::string file,uint64_t offset)
-  {
-    GridStopWatch topen,tread,tclose;
-    std::ifstream fin;
-    topen.Start();
-    fin.open(file,std::ios::binary|std::ios::in);
-    topen.Stop();
-    tread.Start();
-    for(uint64_t e=0;e<p.extentSites.size();e++){
-      fin.seekg(offset + p.extentGsite[e]*sizeof(fobj));
-      fin.read((char *)&aggregated[p.extentLocal[e]],p.extentSites[e]*sizeof(fobj));
-      GRID_ASSERT(fin.fail()==0);
-    }
-    tread.Stop();
-    tclose.Start();
-    fin.close();
-    tclose.Stop();
-
-    std::vector<RealD> us = { (RealD)topen.useconds(), (RealD)tread.useconds(), (RealD)tclose.useconds() };
-    ReportStages(grid,"read",{"open","seek+read","close"},us);
-  }
-#endif
-
   template<class word,class fobj>
   static inline void IOobject(word w,
 			      GridBase *grid,
@@ -684,18 +302,6 @@ class BinaryIO {
       lStart[d] = 0;
     }
 
-    //////////////////////////////////////////////////////////////////////////////
-    // Aggregate the lexicographic layout onto contiguous per rank extents
-    // ourselves rather than leaving it to MPI-IO collective buffering
-    //////////////////////////////////////////////////////////////////////////////
-    int aggregate = (control & BINARYIO_AGGREGATE)
-                 && (control & BINARYIO_LEXICOGRAPHIC)
-                 && !(control & BINARYIO_MASTER_APPEND)
-                 && (nrank > 1);
-#ifndef USE_MPI_IO
-    GRID_ASSERT(aggregate==0); // BINARYIO_AGGREGATE requires MPI
-#endif
-
 #ifdef USE_MPI_IO
     std::vector<int> distribs(ndim,MPI_DISTRIBUTE_BLOCK);
     std::vector<int> dargs   (ndim,MPI_DISTRIBUTE_DFLT_DARG);
@@ -723,8 +329,6 @@ class BinaryIO {
     ierr = MPI_Type_contiguous(numword,mpiword,&mpiObject);    GRID_ASSERT(ierr==0);
     ierr = MPI_Type_commit(&mpiObject);
 
-    // The subarray view is what aggregation exists to avoid; do not build it
-    if ( !aggregate ) {
     //////////////////////////////////////////////////////////////////////////////
     // File global array data type
     //////////////////////////////////////////////////////////////////////////////
@@ -736,7 +340,6 @@ class BinaryIO {
     //////////////////////////////////////////////////////////////////////////////
     ierr=MPI_Type_create_subarray(ndim,&lLattice[0],&lLattice[0],&lStart[0],MPI_ORDER_FORTRAN, mpiObject,&localArray);    GRID_ASSERT(ierr==0);
     ierr=MPI_Type_commit(&localArray);    GRID_ASSERT(ierr==0);
-    }
 #endif
 
     //////////////////////////////////////////////////////////////////////////////
@@ -755,19 +358,7 @@ class BinaryIO {
 
       timer.Start();
 
-      if ( aggregate ) {
-#ifdef USE_MPI_IO
-        std::cout<< GridLogMessage<<"IOobject: aggregate read I/O "<< file<< std::endl;
-        AggregationPlan plan;
-        BuildAggregationPlan(grid,sizeof(fobj),plan);
-        ReportAggregationPlan(grid,plan,sizeof(fobj),"read");
-        std::vector<fobj> aggregated(lsites);
-        AggregateRead(grid,plan,aggregated,file,offset);
-        AggregateExchange(grid,plan,iodata,aggregated,0);
-#else
-        GRID_ASSERT(0);
-#endif
-      } else if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
+      if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
 #ifdef USE_MPI_IO
 	std::cout<< GridLogMessage<<"IOobject: MPI read I/O "<< file<< std::endl;
 	ierr=MPI_File_open(grid->communicator,(char *) file.c_str(), MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);    GRID_ASSERT(ierr==0);
@@ -825,25 +416,7 @@ class BinaryIO {
       grid->Barrier();
 
       timer.Start();
-      if ( aggregate ) {
-#ifdef USE_MPI_IO
-        std::cout << GridLogMessage <<"IOobject: aggregate write I/O " << file << std::endl;
-        AggregationPlan plan;
-        BuildAggregationPlan(grid,sizeof(fobj),plan);
-        ReportAggregationPlan(grid,plan,sizeof(fobj),"write");
-        std::vector<fobj> aggregated(lsites);
-        AggregateExchange(grid,plan,iodata,aggregated,1);
-        AggregateWrite(grid,plan,aggregated,file,offset);
-        ////////////////////////////////////////////////////////////////////////
-        // Not every rank ends at the end of the payload, so the position can
-        // not be recovered from a file handle. Callers (Lime record chaining)
-        // rely on this being the first byte past the record.
-        ////////////////////////////////////////////////////////////////////////
-        offset = offset + (uint64_t)grid->_gsites*sizeof(fobj);
-#else
-        GRID_ASSERT(0);
-#endif
-      } else if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
+      if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
 #ifdef USE_MPI_IO
         std::cout << GridLogMessage <<"IOobject: MPI write I/O " << file << std::endl;
         ierr = MPI_File_open(grid->communicator, (char *)file.c_str(), MPI_MODE_RDWR | MPI_MODE_CREATE, MPI_INFO_NULL, &fh);
@@ -888,26 +461,12 @@ class BinaryIO {
         
 	std::ofstream fout; 
 	fout.exceptions ( std::fstream::failbit | std::fstream::badbit );
-
-        ////////////////////////////////////////////////////////////////////
-        // Grid's model is that the boss rank performs the metadata
-        // operations and every other rank only seeks and writes into a file
-        // that already exists. Opening with ios::out on all ranks broke that:
-        // it is O_TRUNC, so a rank opening late truncated the file back to
-        // zero after an earlier rank had written its segment, leaving a hole
-        // in its place. The barriers around this block are outside it and do
-        // not order the opens against the writes. Let the boss create and
-        // empty the file, then everyone opens for update only. Same resulting
-        // length, one metadata operation instead of one per rank, no race.
-        ////////////////////////////////////////////////////////////////////
-        if ( !offset && grid->IsBoss() ) { // offset zero: this record starts the file
-          std::ofstream create(file,std::ios::binary|std::ios::out);
-          create.close();
-        }
-        grid->Barrier();
-
 	try {
-	  fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
+	  if (offset) { // Must already exist and contain data
+	    fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
+	  } else {     // Allow create
+	    fout.open(file,std::ios::binary|std::ios::out);
+	  }
 	} catch (const std::fstream::failure& exc) {
 	  std::cout << GridLogError << "Error in opening the file " << file << " for output" <<std::endl;
 	  std::cout << GridLogError << "Exception description: " << exc.what() << std::endl;
@@ -918,7 +477,7 @@ class BinaryIO {
 	  exit(1);
 #endif
 	}
-
+	
 	if ( control & BINARYIO_MASTER_APPEND )  {
 	  try {
 	    fout.seekp(0,fout.end);
@@ -989,7 +548,7 @@ class BinaryIO {
 				       uint32_t &nersc_csum,
 				       uint32_t &scidac_csuma,
 				       uint32_t &scidac_csumb,
-				       int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE
+				       int control=BINARYIO_LEXICOGRAPHIC
 				       )
   {
     typedef typename vobj::scalar_object sobj;
@@ -1028,7 +587,7 @@ class BinaryIO {
 					  uint32_t &nersc_csum,
 					  uint32_t &scidac_csuma,
 					  uint32_t &scidac_csumb,
-					  int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
+					  int control=BINARYIO_LEXICOGRAPHIC)
   {
     typedef typename vobj::scalar_object sobj;
     typedef typename vobj::Realified::scalar_type word;    word w=0;
@@ -1115,7 +674,7 @@ class BinaryIO {
     std::cout << GridLogMessage << "RNG read I/O on file " << file << std::endl;
 
     std::vector<RNGstate> iodata(lsites);
-    IOobject(w,grid,iodata,file,offset,format,BINARYIO_READ|BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE,
+    IOobject(w,grid,iodata,file,offset,format,BINARYIO_READ|BINARYIO_LEXICOGRAPHIC,
 	     nersc_csum,scidac_csuma,scidac_csumb);
 
     timer.Start();
@@ -1194,7 +753,7 @@ class BinaryIO {
     });
     timer.Stop();
 
-    IOobject(w,grid,iodata,file,offset,format,BINARYIO_WRITE|BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE,
+    IOobject(w,grid,iodata,file,offset,format,BINARYIO_WRITE|BINARYIO_LEXICOGRAPHIC,
 	     nersc_csum,scidac_csuma,scidac_csumb);
     iodata.resize(1);
     {

Grid/parallelIO/IldgIO.h

@@ -212,7 +212,7 @@ class GridLimeReader : public BinaryIO {
   // Read a generic lattice field and verify checksum
   ////////////////////////////////////////////
   template<class vobj>
-  void readLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
+  void readLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC)
   {
     typedef typename vobj::scalar_object sobj;
     scidacChecksum scidacChecksum_;
@@ -414,7 +414,7 @@ class GridLimeWriter : public BinaryIO
   // in communicator used by the field.Grid()
   ////////////////////////////////////////////////////
   template<class vobj>
-  void writeLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
+  void writeLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC)
   {
     ////////////////////////////////////////////////////////////////////
     // NB: FILE and iostream are jointly writing disjoint sequences in the
@@ -519,7 +519,7 @@ class ScidacWriter : public GridLimeWriter {
   template <class vobj, class userRecord>
   void writeScidacFieldRecord(Lattice<vobj> &field,userRecord _userRecord,
                               const unsigned int recordScientificPrec = 0,
-			      int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
+			      int control=BINARYIO_LEXICOGRAPHIC)
   {
     GridBase * grid = field.Grid();
 
@@ -561,7 +561,7 @@ class ScidacReader : public GridLimeReader {
   ////////////////////////////////////////////////
   template <class vobj, class userRecord>
   void readScidacFieldRecord(Lattice<vobj> &field,userRecord &_userRecord,
-			     int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE) 
+			     int control=BINARYIO_LEXICOGRAPHIC) 
   {
     typedef typename vobj::scalar_object sobj;
     GridBase * grid = field.Grid();

Grid/perfmon/PerfCount.h

@@ -51,8 +51,8 @@ Author: paboyle <paboyle@ph.ed.ac.uk>
 #endif
 #ifdef __x86_64__
 #ifdef GRID_CUDA
-accelerator_inline uint64_t __rdtsc(void) {  return 0; }
-accelerator_inline uint64_t __rdpmc(int ) {  return 0; }
+//accelerator_inline uint64_t __rdtsc(void) {  return 0; }
+//accelerator_inline uint64_t __rdpmc(int ) {  return 0; }
 #else
 #include <x86intrin.h>
 #endif
@@ -93,7 +93,8 @@ inline uint64_t cyclecount(void){
 }
 #elif defined __x86_64__
 inline uint64_t cyclecount(void){ 
-  return (uint64_t)0;
+  uint64_t ret = __rdtsc();
+  return (uint64_t)ret;
 }
 #else
 

Grid/qcd/action/pseudofermion/TwoFlavourBosonPseudoFermion.h

@@ -1,133 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./lib/qcd/action/pseudofermion/TwoFlavourBosonPseudoFermion.h
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#pragma once
-
-NAMESPACE_BEGIN(Grid);
-
-///////////////////////////////////////////////////////////////////////////////
-// Two flavour BOSON (wrong-sign) pseudofermion for any FermionOperator B:
-//
-//    S2 = chi^dag Bdag B chi = |B chi|^2
-//
-//    integral  ==>  det( Bdag B )^-1 = |det B|^-2
-//
-// A compensator monomial: supplies an INVERSE determinant with NO solve in
-// the force or the action -- both are matrix multiplies.  The only solve is
-// the heatbath  chi = B^-1 eta,  once per trajectory (for B = the
-// Pauli-Villars operator this is a mass-one solve, trivially cheap).
-//
-// Primary use: two instances with B = PV cancel the |det PV|^2 excess of
-// TwoFlavourPVdagMPseudoFermionAction down to the DWF quotient
-// |det M|^2/|det PV|^2 (two unsquared instances rather than one squared
-// kernel: first powers of PV in the force, milder).  Being generic in B it
-// also serves Hasenbusch-chain compensation at intermediate masses, or any
-// future inverse-det bookkeeping.  (Sibling of the domain-decomposed boson
-// in DomainDecomposedBoundaryTwoFlavourBosonPseudoFermion.h, without the
-// boundary machinery.)
-//
-// Heatbath exact by construction: S2 after refresh = |B B^-1 eta|^2 = |eta|^2.
-///////////////////////////////////////////////////////////////////////////////
-template<class Impl>
-class TwoFlavourBosonPseudoFermionAction : public Action<typename Impl::GaugeField> {
-public:
-  INHERIT_IMPL_TYPES(Impl);
-
-private:
-  FermionOperator<Impl> & BOp;  // the operator whose |det|^-2 is supplied
-
-  LinearFunction<FermionField> &HeatbathSolver; // b -> B^-1 b (heatbath only)
-
-  FermionField Chi; // the pseudo fermion field for this trajectory
-
-public:
-  TwoFlavourBosonPseudoFermionAction(FermionOperator<Impl>  &_BOp,
-				     LinearFunction<FermionField> & HS
-				     ) : BOp(_BOp),
-					 HeatbathSolver(HS),
-					 Chi(_BOp.FermionGrid())
-  {};
-
-  virtual std::string action_name(){return "TwoFlavourBosonPseudoFermionAction";}
-
-  virtual std::string LogParameters(){
-    std::stringstream sstream;
-    sstream << GridLogMessage << "["<<action_name()<<"] has no parameters" << std::endl;
-    return sstream.str();
-  }
-
-  virtual void refresh(const GaugeField &U, GridSerialRNG &sRNG, GridParallelRNG& pRNG) {
-    // P(chi) = e^{- chi^dag BdagB chi} ; chi = B^-1 eta ; P(eta) = e^{-eta^dag eta}
-    // e^{-x^2/2 sig^2} => sig^2 = 0.5 ; eta enters with width 1/sqrt(2).
-    RealD scale = std::sqrt(0.5);
-    FermionField eta(BOp.FermionGrid());
-    gaussian(pRNG,eta);
-    eta = eta * scale;
-    refresh(U,eta);
-  }
-
-  // Deterministic-noise variant (test hook):
-  // after this,  S(U) == norm2(eta)  exactly (to solver tolerance).
-  void refresh(const GaugeField &U, const FermionField &eta) {
-    BOp.ImportGauge(U);
-    Chi = Zero();
-    HeatbathSolver(eta,Chi);       // Chi = B^-1 eta : the ONLY solve
-    std::cout << GridLogMessage << action_name() << " refresh |Chi|^2 = "<< norm2(Chi)<<std::endl;
-  }
-
-  //////////////////////////////////////////////////////
-  // S2 = |B chi|^2  -- matrix multiply only
-  //////////////////////////////////////////////////////
-  virtual RealD S(const GaugeField &U) {
-    BOp.ImportGauge(U);
-
-    FermionField w(BOp.FermionGrid());
-    BOp.M(Chi,w);                  // w = B chi
-    RealD action = norm2(w);
-    return action;
-  }
-
-  //////////////////////////////////////////////////////
-  // dS2 = chi^dag dBdag w + w^dag dB chi ,   w = B chi
-  // NO solves.
-  //////////////////////////////////////////////////////
-  virtual void deriv(const GaugeField &U,GaugeField & dSdU) {
-    BOp.ImportGauge(U);
-
-    FermionField w(BOp.FermionGrid());
-    GaugeField force(BOp.GaugeGrid());
-
-    BOp.M(Chi,w);                  // w = B chi
-
-    BOp.MDeriv(force, Chi, w, DaggerYes); dSdU =      force;
-    BOp.MDeriv(force, w, Chi, DaggerNo ); dSdU = dSdU+force;
-
-    dSdU *= -1.0;  // Grid action sign convention (cf TwoFlavourRatio.h)
-  };
-};
-
-NAMESPACE_END(Grid);

Grid/qcd/action/pseudofermion/TwoFlavourRatio4DPseudoFermion.h

@@ -1,264 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./lib/qcd/action/pseudofermion/TwoFlavourRatio4DPseudoFermion.h
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#pragma once
-
-NAMESPACE_BEGIN(Grid);
-
-///////////////////////////////////////////////////////////////////////////////
-// Two flavour ratio with FOUR dimensional pseudofermion, UNpreconditioned
-// (full grid) solves.
-//
-// Companion to TwoFlavourRatioEO4DPseudoFermion.h but with the solver
-// plumbing exposed as LinearFunction<FermionField> objects that already
-// know their operator -- the natural interface for the non-Hermitian
-// multigrid GCR stack (PVdagM), which solves M and Mdag DIRECTLY rather
-// than through SchurRedBlack normal equations.
-//
-// Why: with 5D pseudofermions the squared-operator formulation hands
-// normal-equation solvers (MdagM)^-1 phi AND Mdag^-1 phi from ONE Krylov
-// space; a direct solver must solve twice, halving its per-solve gain.
-// The 4D pseudofermion action needs one M^-1 and one M^-dag solve per
-// force evaluation FOR BOTH solver families, so the direct-solver gain
-// carries through undiluted.  In addition phi4 is Ls-agnostic, so the
-// force can be evaluated with a reduced-Ls operator pair while the
-// accept/reject uses full Ls (inexact force, exact action).
-//
-// Solver slots (all full-grid 5D LinearFunctions, solution overwritten,
-// zero guess imposed internally):
-//   DerivMinvSolver     : x = M^-1 b        (DenOp)
-//   DerivMdagInvSolver  : x = M^-dag b      (DenOp).  For G5R5-hermitian
-//                         actions this may be implemented by the caller as
-//                         G5R5 . DerivMinvSolver . G5R5 -- no adjoint
-//                         multigrid needed.
-//   ActionMinvSolver    : x = M^-1 b        (DenOp, accept/reject tolerance)
-//   HeatbathVinvSolver  : x = V^-1 b        (NumOp)
-//
-// 4D <-> 5D wall maps: the action is S = | P (M^-1 V) Pdag phi4 |^2 where
-// (P,Pdag) MUST be a mutually adjoint pair for S and deriv to be
-// consistent.  Two candidate conventions, selected by solution_walls:
-//   true  : P = P_- psi(0) + P_+ psi(Ls-1)   (solution walls, matches
-//           ExportPhysicalFermionSolution) and Pdag its literal adjoint.
-//   false : P = P_+ psi(0) + P_- psi(Ls-1)   (source walls, Pdag matches
-//           ImportUnphysicalFermion).
-// The heatbath is exact iff [P M^-1 V Pdag][P V^-1 M Pdag] = 1 (the 4D
-// effective-operator composition identity); which convention satisfies it
-// is settled numerically by the refresh test  S == 0.5*|eta4|^2  exactly.
-///////////////////////////////////////////////////////////////////////////////
-template<class Impl>
-class TwoFlavourRatio4DPseudoFermionAction : public Action<typename Impl::GaugeField> {
-public:
-  INHERIT_IMPL_TYPES(Impl);
-
-private:
-  typedef FermionOperator<Impl> FermOp;
-  FermionOperator<Impl> & NumOp;// the basic operator (V)
-  FermionOperator<Impl> & DenOp;// the basic operator (M)
-
-  LinearFunction<FermionField> &DerivMinvSolver;
-  LinearFunction<FermionField> &DerivMdagInvSolver;
-  LinearFunction<FermionField> &ActionMinvSolver;
-  LinearFunction<FermionField> &HeatbathVinvSolver;
-
-  FermionField phi4; // the pseudo fermion field for this trajectory
-
-  int solution_walls; // wall convention for the (P,Pdag) pair; see header
-
-  ////////////////////////////////////////////////////////////////////
-  // The mutually adjoint 4D <-> 5D pair.
-  //   Wall4D    : q4  = P psi5      (extract)
-  //   Wall4DAdj : psi5 = Pdag q4    (insert; literal adjoint of Wall4D)
-  ////////////////////////////////////////////////////////////////////
-  void Wall4D(const FermionField &psi5, FermionField &q4)
-  {
-    int Ls = NumOp.FermionGrid()->_fdimensions[0];
-    FermionField tmp(NumOp.FermionGrid());
-    if ( solution_walls ) {
-      // q4 = P_- psi(0) + P_+ psi(Ls-1)
-      axpby_ssp_pminus(tmp, 0., psi5, 1., psi5, 0, 0);
-      axpby_ssp_pplus (tmp, 1., tmp , 1., psi5, 0, Ls-1);
-    } else {
-      // q4 = P_+ psi(0) + P_- psi(Ls-1)
-      axpby_ssp_pplus (tmp, 0., psi5, 1., psi5, 0, 0);
-      axpby_ssp_pminus(tmp, 1., tmp , 1., psi5, 0, Ls-1);
-    }
-    ExtractSlice(q4, tmp, 0, 0);
-  }
-  void Wall4DAdj(const FermionField &q4, FermionField &psi5)
-  {
-    int Ls = NumOp.FermionGrid()->_fdimensions[0];
-    FermionField tmp(NumOp.FermionGrid());
-    tmp = Zero();
-    InsertSlice(q4, tmp, 0   , 0);
-    InsertSlice(q4, tmp, Ls-1, 0);
-    if ( solution_walls ) {
-      // psi(0) = P_- q4 ; psi(Ls-1) = P_+ q4
-      axpby_ssp_pminus(tmp, 0., tmp, 1., tmp, 0   , 0);
-      axpby_ssp_pplus (tmp, 0., tmp, 1., tmp, Ls-1, Ls-1);
-    } else {
-      // psi(0) = P_+ q4 ; psi(Ls-1) = P_- q4
-      axpby_ssp_pplus (tmp, 0., tmp, 1., tmp, 0   , 0);
-      axpby_ssp_pminus(tmp, 0., tmp, 1., tmp, Ls-1, Ls-1);
-    }
-    psi5 = tmp;
-  }
-
-public:
-  TwoFlavourRatio4DPseudoFermionAction(FermionOperator<Impl>  &_NumOp,
-				       FermionOperator<Impl>  &_DenOp,
-				       LinearFunction<FermionField> & DMS,
-				       LinearFunction<FermionField> & DMDS,
-				       LinearFunction<FermionField> & AMS,
-				       LinearFunction<FermionField> & HVS,
-				       int _solution_walls = 1
-				       ) : NumOp(_NumOp),
-					   DenOp(_DenOp),
-					   DerivMinvSolver(DMS),
-					   DerivMdagInvSolver(DMDS),
-					   ActionMinvSolver(AMS),
-					   HeatbathVinvSolver(HVS),
-					   phi4(_NumOp.GaugeGrid()),
-					   solution_walls(_solution_walls)
-  {};
-
-  virtual std::string action_name(){return "TwoFlavourRatio4DPseudoFermionAction";}
-
-  virtual std::string LogParameters(){
-    std::stringstream sstream;
-    sstream << GridLogMessage << "["<<action_name()<<"] solution_walls " << solution_walls << std::endl;
-    return sstream.str();
-  }
-
-  virtual void refresh(const GaugeField &U, GridSerialRNG &sRNG, GridParallelRNG& pRNG) {
-
-    // P(phi4) = e^{- phi4^dag Beff^dag Beff phi4}   ;  Beff = P M^-1 V Pdag
-    //
-    // NumOp == V
-    // DenOp == M
-    //
-    // Take phi4 = P V^-1 M Pdag eta4   ( = Beff^-1 eta4 by the composition
-    // identity; verified numerically by S == 0.5 |eta4|^2 after refresh )
-    //
-    // P(eta) = e^{- eta^dag eta} ;  e^{-x^2/2 sig^2} => sig^2 = 0.5
-    // so eta enters with width 1/sqrt(2).
-    //
-    RealD scale = std::sqrt(0.5);
-
-    FermionField eta4(NumOp.GaugeGrid());
-    FermionField eta5(NumOp.FermionGrid());
-    FermionField tmp (NumOp.FermionGrid());
-    FermionField phi5(NumOp.FermionGrid());
-
-    gaussian(pRNG,eta4);
-
-    NumOp.ImportGauge(U);
-    DenOp.ImportGauge(U);
-
-    Wall4DAdj(eta4,eta5);            // eta5 = Pdag eta4
-    DenOp.M(eta5,tmp);               // tmp  = M eta5
-    phi5 = Zero();
-    HeatbathVinvSolver(tmp,phi5);    // phi5 = V^-1 M eta5
-    Wall4D(phi5,phi4);               // phi4 = P phi5
-    phi4 = phi4*scale;
-
-    std::cout << GridLogMessage << "4d pf (non-EO) refresh "<< norm2(phi4)<<"\n";
-  };
-
-  //////////////////////////////////////////////////////
-  // S = phi4^dag (Pdag^dag V^dag M^-dag P^dag) (P M^-1 V Pdag) phi4
-  //   = | P M^-1 V Pdag phi4 |^2
-  //////////////////////////////////////////////////////
-  virtual RealD S(const GaugeField &U) {
-
-    NumOp.ImportGauge(U);
-    DenOp.ImportGauge(U);
-
-    FermionField Y4  (NumOp.GaugeGrid());
-    FermionField phi5(NumOp.FermionGrid());
-    FermionField X   (NumOp.FermionGrid());
-    FermionField Y   (NumOp.FermionGrid());
-
-    Wall4DAdj(phi4,phi5);            // phi5 = Pdag phi4
-    NumOp.M(phi5,X);                 // X    = V phi5
-    Y = Zero();
-    ActionMinvSolver(X,Y);           // Y    = M^-1 V phi5
-    Wall4D(Y,Y4);                    // Y4   = P Y
-
-    RealD action = norm2(Y4);
-
-    return action;
-  };
-
-  //////////////////////////////////////////////////////
-  // dS/du = 2 Re [ (M^-dag Pdag w4)^dag dV Pdag phi4 ]
-  //       - 2 Re [ (M^-dag Pdag w4)^dag dM (M^-1 V Pdag phi4) ]
-  // with w4 = P M^-1 V Pdag phi4.
-  // Two first-power solves: one M^-1, one M^-dag.
-  //////////////////////////////////////////////////////
-  virtual void deriv(const GaugeField &U,GaugeField & dSdU) {
-
-    NumOp.ImportGauge(U);
-    DenOp.ImportGauge(U);
-
-    FermionField  phi5           (NumOp.FermionGrid());
-    FermionField  Vphi           (NumOp.FermionGrid());
-    FermionField  MinvVphi       (NumOp.FermionGrid());
-    FermionField  w4             (NumOp.GaugeGrid());
-    FermionField  Y              (NumOp.FermionGrid());
-    FermionField  MdagInvPdagW   (NumOp.FermionGrid());
-
-    GaugeField   force(NumOp.GaugeGrid());
-
-    Wall4DAdj(phi4,phi5);              // phi5     = Pdag phi4
-    NumOp.M(phi5,Vphi);                // Vphi     = V phi5
-    MinvVphi = Zero();
-    DerivMinvSolver(Vphi,MinvVphi);    // MinvVphi = M^-1 V phi5
-    std::cout << GridLogMessage << "4d pf (non-EO) deriv solve "<< norm2(MinvVphi)<<"\n";
-
-    // Project onto the physical 4D subspace and back: Y = Pdag P MinvVphi.
-    // Pdag here MUST be the literal adjoint of the P used in S, else the
-    // force is inconsistent with the action.
-    Wall4D(MinvVphi,w4);               // w4 = P MinvVphi
-    Wall4DAdj(w4,Y);                   // Y  = Pdag w4
-
-    MdagInvPdagW = Zero();
-    DerivMdagInvSolver(Y,MdagInvPdagW);   // = M^-dag Pdag w4  (adjoint solve)
-    std::cout << GridLogMessage << "4d pf (non-EO) deriv solve dag "<< norm2(MdagInvPdagW)<<"\n";
-
-    // phi^dag (Pdag' Vdag Mdag^-1 P') (dV) Pdag phi   + h.c.
-    NumOp.MDeriv(force, MdagInvPdagW, phi5, DaggerNo );  dSdU=force;
-    NumOp.MDeriv(force, phi5, MdagInvPdagW, DaggerYes);  dSdU=dSdU+force;
-
-    // - phi^dag ( ... Mdag^-1 ) dM ( M^-1 V ... ) phi  + h.c.
-    DenOp.MDeriv(force, MdagInvPdagW, MinvVphi, DaggerNo );  dSdU=dSdU-force;
-    DenOp.MDeriv(force, MinvVphi, MdagInvPdagW, DaggerYes);  dSdU=dSdU-force;
-
-    dSdU *= -1.0;
-  };
-};
-
-NAMESPACE_END(Grid);

Grid/qcd/action/pseudofermion/TwoFlavourRatioLeftPrec.h

@@ -1,206 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./lib/qcd/action/pseudofermion/TwoFlavourRatioLeftPrec.h
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#pragma once
-
-NAMESPACE_BEGIN(Grid);
-
-///////////////////////////////////////////////////////////////////////////////
-// Two flavour ratio with LEFT-PRECONDITIONED solves.
-//
-// Same action content as TwoFlavourRatio.h:
-//
-//    S = phi^dag V (Mdag M)^-1 Vdag phi     ==>   det[ Mdag M / Vdag V ]
-//
-// (V = NumOp the heavier / Pauli-Villars operator, M = DenOp the lighter),
-// but organised around the composite
-//
-//    F = Vdag M
-//
-// which is the 2-hop-coarsenable operator the non-Hermitian multigrid
-// serves.  Solving M X = b as F X = Vdag b is LEFT PRECONDITIONING by
-// Vdag; the determinant/action layer is the standard quotient, and all
-// novelty is confined to the solver contract.
-//
-// TwoFlavourRatio.h is tied to a normal-equations solver: one (MdagM)^-1
-// solve, then Y = M X gives Mdag^-1 Vdag phi almost free.  The left-
-// preconditioned idiom is DIFFERENT: the chain
-//
-//    b = Vdag phi
-//    z : Fdag z = b          (adjoint  F solve)
-//    Y = V z                 (= Mdag^-1 Vdag phi -- harvested from solve 1)
-//    s = Vdag Y              (= Vdag V z)
-//    X : F X = s             (forward F solve;  X = (MdagM)^-1 Vdag phi)
-//
-// yields Y BEFORE X (so S(U) needs only the adjoint solve), with Y's
-// accuracy independent of the second solve.  Force terms are then the
-// standard four MDeriv insertions of TwoFlavourRatio.
-//
-// Solver slots are LinearFunctions with the F-SOLVE contract (solution
-// overwritten, zero guess imposed internally):
-//    ForwardSolver(b,x)  :  F x    = b
-//    AdjointSolver(b,z)  :  Fdag z = b
-// implemented in production by the multigrid-GCR stack (forward cycle and
-// adjoint cycle); in tests by CG on the composite normal equations.
-// HeatbathSolver(b,x) : x = (Vdag V)^-1 b -- heavy operator, plain CG.
-//
-// Heatbath is exact by operator algebra:  phi = V (VdagV)^-1 Mdag eta
-// ==>  S = | Mdag^-1 Vdag phi |^2 = |eta|^2  (to solver tolerance); the
-// deterministic refresh(U,eta) hook below is the test point.
-//
-// Hasenbusch: nothing requires V to have mass one; any (heavier,lighter)
-// pair works, F(V,M) = Vdag M coarsenable by the same machinery, rungs'
-// solves are F-family (mrhs-batchable, mass-shared coarse space).
-///////////////////////////////////////////////////////////////////////////////
-template<class Impl>
-class TwoFlavourRatioLeftPrecPseudoFermionAction : public Action<typename Impl::GaugeField> {
-public:
-  INHERIT_IMPL_TYPES(Impl);
-
-private:
-  FermionOperator<Impl> & NumOp;// V
-  FermionOperator<Impl> & DenOp;// M
-
-  LinearFunction<FermionField> &DerivForwardSolver;  // F x = b, MD tolerance
-  LinearFunction<FermionField> &DerivAdjointSolver;  // Fdag z = b, MD tolerance
-  LinearFunction<FermionField> &ActionAdjointSolver; // Fdag z = b, accept/reject tolerance
-  LinearFunction<FermionField> &HeatbathSolver;      // (VdagV)^-1 b, heavy op
-
-  FermionField Phi; // the pseudo fermion field for this trajectory
-
-public:
-  TwoFlavourRatioLeftPrecPseudoFermionAction(FermionOperator<Impl>  &_NumOp,
-					     FermionOperator<Impl>  &_DenOp,
-					     LinearFunction<FermionField> & DFS,
-					     LinearFunction<FermionField> & DAS,
-					     LinearFunction<FermionField> & AAS,
-					     LinearFunction<FermionField> & HS
-					     ) : NumOp(_NumOp),
-						 DenOp(_DenOp),
-						 DerivForwardSolver(DFS),
-						 DerivAdjointSolver(DAS),
-						 ActionAdjointSolver(AAS),
-						 HeatbathSolver(HS),
-						 Phi(_NumOp.FermionGrid())
-  {};
-
-  virtual std::string action_name(){return "TwoFlavourRatioLeftPrecPseudoFermionAction";}
-
-  virtual std::string LogParameters(){
-    std::stringstream sstream;
-    sstream << GridLogMessage << "["<<action_name()<<"] has no parameters" << std::endl;
-    return sstream.str();
-  }
-
-  virtual void refresh(const GaugeField &U, GridSerialRNG &sRNG, GridParallelRNG& pRNG) {
-    // P(phi) = e^{- phi^dag V (MdagM)^-1 Vdag phi} ; phi = Vdag^-1 Mdag eta
-    // e^{-x^2/2 sig^2} => sig^2 = 0.5 ; eta enters with width 1/sqrt(2).
-    RealD scale = std::sqrt(0.5);
-    FermionField eta(NumOp.FermionGrid());
-    gaussian(pRNG,eta);
-    eta = eta * scale;
-    refresh(U,eta);
-  }
-
-  // Deterministic-noise variant (test hook):
-  // after this,  S(U) == norm2(eta)  exactly (to solver tolerance).
-  void refresh(const GaugeField &U, const FermionField &eta) {
-    NumOp.ImportGauge(U);
-    DenOp.ImportGauge(U);
-
-    FermionField tmp(NumOp.FermionGrid());
-    FermionField w  (NumOp.FermionGrid());
-
-    DenOp.Mdag(eta,tmp);           // tmp = Mdag eta
-    w = Zero();
-    HeatbathSolver(tmp,w);         // w   = (VdagV)^-1 Mdag eta
-    NumOp.M(w,Phi);                // Phi = V (VdagV)^-1 Mdag eta = Vdag^-1 Mdag eta
-    std::cout << GridLogMessage << action_name() << " refresh |Phi|^2 = "<< norm2(Phi)<<std::endl;
-  }
-
-  //////////////////////////////////////////////////////
-  // S = phi^dag V (MdagM)^-1 Vdag phi = | Mdag^-1 Vdag phi |^2
-  // ONE adjoint F solve:  Y = V Fdag^-1 Vdag phi = Mdag^-1 Vdag phi
-  //////////////////////////////////////////////////////
-  virtual RealD S(const GaugeField &U) {
-    NumOp.ImportGauge(U);
-    DenOp.ImportGauge(U);
-
-    FermionField b(NumOp.FermionGrid());
-    FermionField z(NumOp.FermionGrid());
-    FermionField Y(NumOp.FermionGrid());
-
-    NumOp.Mdag(Phi,b);             // b = Vdag phi
-    z = Zero();
-    ActionAdjointSolver(b,z);      // Fdag z = b
-    NumOp.M(z,Y);                  // Y = V z = Mdag^-1 Vdag phi
-
-    RealD action = norm2(Y);
-    return action;
-  }
-
-  //////////////////////////////////////////////////////
-  // dS/du = phi^dag dV (MdagM)^-1 Vdag phi
-  //       - phi^dag V (MdagM)^-1 [ Mdag dM + dMdag M ] (MdagM)^-1 Vdag phi
-  //       + phi^dag V (MdagM)^-1 dVdag phi
-  // Identical force insertions to TwoFlavourRatio.h; X and Y from the
-  // left-preconditioned chain (Y harvested from the adjoint solve).
-  //////////////////////////////////////////////////////
-  virtual void deriv(const GaugeField &U,GaugeField & dSdU) {
-    NumOp.ImportGauge(U);
-    DenOp.ImportGauge(U);
-
-    FermionField b(NumOp.FermionGrid());
-    FermionField z(NumOp.FermionGrid());
-    FermionField Y(NumOp.FermionGrid());
-    FermionField s(NumOp.FermionGrid());
-    FermionField X(NumOp.FermionGrid());
-
-    GaugeField force(NumOp.GaugeGrid());
-
-    NumOp.Mdag(Phi,b);             // b = Vdag phi
-    z = Zero();
-    DerivAdjointSolver(b,z);       // Fdag z = b
-    NumOp.M(z,Y);                  // Y = V z = Mdag^-1 Vdag phi   (solve-1 harvest)
-    NumOp.Mdag(Y,s);               // s = Vdag V z
-    X = Zero();
-    DerivForwardSolver(s,X);       // F X = s  ==>  X = (MdagM)^-1 Vdag phi
-
-    // phi^dag V (MdagM)^-1 dVdag phi
-    NumOp.MDeriv(force , X, Phi, DaggerYes);  dSdU  =      force;
-    // phi^dag dV (MdagM)^-1 Vdag phi
-    NumOp.MDeriv(force , Phi, X, DaggerNo );  dSdU  = dSdU+force;
-    //  - phi^dag V (MdagM)^-1 Mdag dM (MdagM)^-1 Vdag phi
-    //  - phi^dag V (MdagM)^-1 dMdag M (MdagM)^-1 Vdag phi
-    DenOp.MDeriv(force, Y, X, DaggerNo );     dSdU  = dSdU-force;
-    DenOp.MDeriv(force, X, Y, DaggerYes);     dSdU  = dSdU-force;
-
-    dSdU *= -1.0;
-  };
-};
-
-NAMESPACE_END(Grid);

Grid/simd/Simd.h

@@ -115,10 +115,10 @@ accelerator_inline ComplexF adj(const ComplexF& r ){ return(conjugate(r)); }
 
 #if defined(GRID_CUDA) || defined(GRID_HIP)
 //Provide for convenience
-inline std::complex<double> conjugate(const std::complex<double>& r){ return(conj(r)); }
-inline std::complex<float>  conjugate(const std::complex<float>& r) { return(conj(r)); }
-inline std::complex<double> adj(const std::complex<double>& r)      { return(conj(r)); }
-inline std::complex<float>  adj(const std::complex<float>& r)       { return(conj(r)); }
+accelerator_inline std::complex<double> conjugate(const std::complex<double>& r){ return(conj(r)); }
+accelerator_inline std::complex<float>  conjugate(const std::complex<float>& r) { return(conj(r)); }
+accelerator_inline std::complex<double> adj(const std::complex<double>& r)      { return(conj(r)); }
+accelerator_inline std::complex<float>  adj(const std::complex<float>& r)       { return(conj(r)); }
 #endif
 
 accelerator_inline RealF real(const RealF  & r){ return r; }

Grid/tensors/Tensor_inner.h

@@ -32,6 +32,33 @@ Author: Christoph Lehner <christoph@lhnr.de>
 
 NAMESPACE_BEGIN(Grid);
 
+//////////////////////////////////////
+// innerProductD scalar overloads must be visible before norm2 is defined
+//////////////////////////////////////
+accelerator_inline ComplexD innerProductD(const ComplexF &l,const ComplexF &r){  return innerProduct(l,r); }
+accelerator_inline ComplexD innerProductD(const ComplexD &l,const ComplexD &r){  return innerProduct(l,r); }
+accelerator_inline RealD    innerProductD(const RealD    &l,const RealD    &r){  return innerProduct(l,r); }
+accelerator_inline RealD    innerProductD(const RealF    &l,const RealF    &r){  return innerProduct(l,r); }
+
+accelerator_inline vComplexD innerProductD(const vComplexD &l,const vComplexD &r){  return innerProduct(l,r); }
+accelerator_inline vRealD    innerProductD(const vRealD    &l,const vRealD    &r){  return innerProduct(l,r); }
+accelerator_inline vComplexD innerProductD(const vComplexF &l,const vComplexF &r)
+{
+  vComplexD la,lb;
+  vComplexD ra,rb;
+  Optimization::PrecisionChange::StoD(l.v,la.v,lb.v);
+  Optimization::PrecisionChange::StoD(r.v,ra.v,rb.v);
+  return innerProduct(la,ra) + innerProduct(lb,rb);
+}
+accelerator_inline vRealD innerProductD(const vRealF &l,const vRealF &r)
+{
+  vRealD la,lb;
+  vRealD ra,rb;
+  Optimization::PrecisionChange::StoD(l.v,la.v,lb.v);
+  Optimization::PrecisionChange::StoD(r.v,ra.v,rb.v);
+  return innerProduct(la,ra) + innerProduct(lb,rb);
+}
+
 ///////////////////////////////////////////////////////////////////////////////////////
 // innerProduct Scalar x Scalar -> Scalar
 // innerProduct Vector x Vector -> Scalar
@@ -138,30 +165,8 @@ auto Reduce (const iScalar<l>& lhs) -> iScalar<decltype(Reduce(lhs._internal))>
 
 //////////////////////////////////////
 // innerProductD : if single promote to double and evaluate with sum 2x
+// (scalar/vector overloads are declared above norm2 for ADL visibility)
 //////////////////////////////////////
-accelerator_inline ComplexD innerProductD(const ComplexF &l,const ComplexF &r){  return innerProduct(l,r); }
-accelerator_inline ComplexD innerProductD(const ComplexD &l,const ComplexD &r){  return innerProduct(l,r); }
-accelerator_inline RealD    innerProductD(const RealD    &l,const RealD    &r){  return innerProduct(l,r); }
-accelerator_inline RealD    innerProductD(const RealF    &l,const RealF    &r){  return innerProduct(l,r); }
-
-accelerator_inline vComplexD innerProductD(const vComplexD &l,const vComplexD &r){  return innerProduct(l,r); }
-accelerator_inline vRealD    innerProductD(const vRealD    &l,const vRealD    &r){  return innerProduct(l,r); }
-accelerator_inline vComplexD innerProductD(const vComplexF &l,const vComplexF &r)
-{  
-  vComplexD la,lb;
-  vComplexD ra,rb;
-  Optimization::PrecisionChange::StoD(l.v,la.v,lb.v);
-  Optimization::PrecisionChange::StoD(r.v,ra.v,rb.v);
-  return innerProduct(la,ra) + innerProduct(lb,rb); 
-}
-accelerator_inline vRealD innerProductD(const vRealF &l,const vRealF &r)
-{  
-  vRealD la,lb;
-  vRealD ra,rb;
-  Optimization::PrecisionChange::StoD(l.v,la.v,lb.v);
-  Optimization::PrecisionChange::StoD(r.v,ra.v,rb.v);
-  return innerProduct(la,ra) + innerProduct(lb,rb); 
-}
 
 // Now do it for vector, matrix, scalar
 template<class l,class r,int N> accelerator_inline

MPI_benchmark/compile-command

@@ -1,4 +1 @@
-mpicxx  -fsycl halo_mpi.cc -o halo_mpi
-mpicxx -O2 -std=c++11 io_mpi.cc -o io_mpi
-# Frontier: (hipcc via mpicxx wrapper; ACC_HIP is the default in-file)
-mpicxx -O2 -x hip gather_mpi.cc -o gather_mpi -L${ROCM_PATH}/lib -lamdhip64
+mpicxx  -fsycl halo_mpi.cc -o halo_mpi

MPI_benchmark/gather_mpi.cc

@@ -1,373 +0,0 @@
-#include <cassert>
-#include <complex>
-#include <memory>
-#include <vector>
-#include <algorithm>
-#include <array>
-#include <string>
-#include <sstream>
-#include <cstring>
-#include <stdio.h>
-#include <stdlib.h>
-#include <strings.h>
-#include <ctime>
-#include <sys/time.h>
-
-#include <mpi.h>
-
-/**************************************************************
- * Distributed dense-panel gather benchmark / reproducer.
- *
- * Pattern under test: P ranks each OWN a contiguous block of rows
- * of a (krows x ncols) fp64-complex panel; every rank must end
- * holding the WHOLE panel.  This is the setup-time collective of a
- * distributed recursive Schur inversion (row-distributed dense
- * matrices, rank-major ordering => owned rows contiguous).
- *
- * Three implementations of identical semantics:
- *   A  zero-fill + MPI_Allreduce(SUM)   (simple; non-owners send zeros)
- *   B  MPI_Allgatherv                   (owners-only send)
- *   C  ring allgather via MPI_Sendrecv  (halo-exchange primitive)
- *
- * Measured motivation (Frontier, 288 ranks, ~340MB panels): pattern A
- * on host buffers delivered 0.48 GB/s effective payload -- ~8% of its
- * own per-node wire floor, ~13x below the efficiency the Sendrecv
- * halo exchange achieves on the same NICs (185 GB/s bidirectional,
- * see halo_mpi.cc).
- *
- * Config: what is the target
- **************************************************************
- */
-#undef  ACC_CUDA
-#define ACC_HIP
-#undef  ACC_SYCL
-#undef  ACC_NONE
-
-/**************************************************************
- * Some MPI globals
- **************************************************************
- */
-MPI_Comm WorldComm;
-MPI_Comm WorldShmComm;
-
-int WorldSize;
-int WorldRank;
-
-int WorldShmSize;
-int WorldShmRank;
-
-/**************************************************************
- * Allocate buffers on the GPU, SYCL needs an init call and context
- **************************************************************
- */
-#ifdef ACC_CUDA
-#include <cuda.h>
-void acceleratorInit(void){}
-void *acceleratorAllocDevice(size_t bytes)
-{
-  void *ptr=NULL;
-  auto err = cudaMalloc((void **)&ptr,bytes);
-  assert(err==cudaSuccess);
-  return ptr;
-}
-void acceleratorFreeDevice(void *ptr){  cudaFree(ptr);}
-void acceleratorMemSet(void *ptr,int val,size_t bytes){ cudaMemset(ptr,val,bytes);}
-void acceleratorCopyToDevice(const void *from,void *to,size_t bytes){ cudaMemcpy(to,from,bytes,cudaMemcpyHostToDevice);}
-#endif
-#ifdef ACC_HIP
-#include <hip/hip_runtime.h>
-void acceleratorInit(void){}
-inline void *acceleratorAllocDevice(size_t bytes)
-{
-  void *ptr=NULL;
-  auto err = hipMalloc((void **)&ptr,bytes);
-  if( err != hipSuccess ) {
-    ptr = (void *) NULL;
-    printf(" hipMalloc failed for %ld %s \n",bytes,hipGetErrorString(err));
-  }
-  return ptr;
-};
-inline void acceleratorFreeDevice(void *ptr){ auto r=hipFree(ptr);};
-inline void acceleratorMemSet(void *ptr,int val,size_t bytes){ auto r=hipMemset(ptr,val,bytes);};
-inline void acceleratorCopyToDevice(const void *from,void *to,size_t bytes){ auto r=hipMemcpy(to,from,bytes,hipMemcpyHostToDevice);};
-#endif
-#ifdef ACC_SYCL
-#include <sycl/CL/sycl.hpp>
-#include <sycl/usm.hpp>
-cl::sycl::queue *theAccelerator;
-void acceleratorInit(void)
-{
-  cl::sycl::gpu_selector selector;
-  cl::sycl::device selectedDevice { selector };
-  theAccelerator = new sycl::queue (selectedDevice);
-  auto name = theAccelerator->get_device().get_info<sycl::info::device::name>();
-  printf("AcceleratorSyclInit: Selected device is %s\n",name.c_str()); fflush(stdout);
-}
-inline void *acceleratorAllocDevice(size_t bytes){ return malloc_device(bytes,*theAccelerator);};
-inline void acceleratorFreeDevice(void *ptr){free(ptr,*theAccelerator);};
-inline void acceleratorMemSet(void *ptr,int val,size_t bytes){ theAccelerator->memset(ptr,val,bytes).wait();};
-inline void acceleratorCopyToDevice(const void *from,void *to,size_t bytes){ theAccelerator->memcpy(to,from,bytes).wait();};
-#endif
-#ifdef ACC_NONE
-void acceleratorInit(void){}
-inline void *acceleratorAllocDevice(size_t bytes){ return malloc(bytes);};
-inline void acceleratorFreeDevice(void *ptr){free(ptr);};
-inline void acceleratorMemSet(void *ptr,int val,size_t bytes){ memset(ptr,val,bytes);};
-inline void acceleratorCopyToDevice(const void *from,void *to,size_t bytes){ memcpy(to,from,bytes);};
-#endif
-
-/**************************************************************
- * Microsecond timer
- **************************************************************
- */
-inline double usecond(void) {
-  struct timeval tv;
-  gettimeofday(&tv,NULL);
-  return 1.0e6*tv.tv_sec + 1.0*tv.tv_usec;
-}
-
-/**************************************************************
- * Main benchmark routine.
- *
- * panel = krows x ncols complex<double>, column major, krows = P*myrows.
- * Rank r owns rows [r*myrows, (r+1)*myrows): with column-major layout
- * the owned data is strided; for B and C the owners' contribution is
- * packed contiguously (rank-major block layout), which is exactly how
- * the production code would call it.  Effective payload = full panel
- * bytes; every rank must hold it all at the end.
- **************************************************************
- */
-void Benchmark(size_t panel_bytes,bool use_device,int ncall)
-{
-  size_t words = panel_bytes/sizeof(double); // treat as flat doubles
-  size_t mywords = words/WorldSize;
-  words = mywords*WorldSize;                  // exact division
-  size_t bytes  = words*sizeof(double);
-  size_t mybytes= mywords*sizeof(double);
-
-  double *panel;
-  double *contrib;
-  if ( use_device ) {
-    panel   = (double *)acceleratorAllocDevice(bytes);
-    contrib = (double *)acceleratorAllocDevice(mybytes);
-    if ( panel==NULL || contrib==NULL ) { printf("alloc failed\n"); return; }
-  } else {
-    panel   = (double *)malloc(bytes);
-    contrib = (double *)malloc(mybytes);
-  }
-  // Owners deposit non-trivial data once (content is irrelevant to timing)
-  std::vector<double> init(mywords,1.0*WorldRank);
-  acceleratorCopyToDevice(&init[0],contrib,mybytes);
-  if ( !use_device ) memcpy(contrib,&init[0],mybytes);
-
-  double tA,tB,tC;
-
-  /*********************************************************
-   * A: zero-fill + Allreduce(SUM) -- the simple idiom
-   *********************************************************/
-  {
-    MPI_Barrier(WorldComm);
-    double t0=usecond();
-    for(int n=0;n<ncall;n++){
-      if ( use_device ) acceleratorMemSet(panel,0,bytes);
-      else              memset(panel,0,bytes);
-      // owner deposits its block at its rank-major offset
-      if ( use_device ) {
-        // device-to-device deposit stands in for the real kernel
-#ifdef ACC_HIP
-        auto r=hipMemcpy(panel+WorldRank*mywords,contrib,mybytes,hipMemcpyDeviceToDevice);
-#else
-        acceleratorCopyToDevice(contrib,panel+WorldRank*mywords,mybytes);
-#endif
-      } else {
-        memcpy(panel+WorldRank*mywords,contrib,mybytes);
-      }
-      int ierr=MPI_Allreduce(MPI_IN_PLACE,panel,words,MPI_DOUBLE,MPI_SUM,WorldComm);
-      assert(ierr==0);
-    }
-    MPI_Barrier(WorldComm);
-    tA=(usecond()-t0)/ncall;
-  }
-
-  /*********************************************************
-   * A2: as A but with a FRESH buffer every call -- the
-   * registration-cache killer (unpinned pages each call).
-   * Host-memory case only; isolates the production bug.
-   *********************************************************/
-  double tA2 = 0.0;
-  if ( !use_device ) {
-    MPI_Barrier(WorldComm);
-    double t0=usecond();
-    for(int n=0;n<ncall;n++){
-      double *fresh = (double *)malloc(bytes);
-      memset(fresh,0,bytes);
-      memcpy(fresh+WorldRank*mywords,contrib,mybytes);
-      int ierr=MPI_Allreduce(MPI_IN_PLACE,fresh,words,MPI_DOUBLE,MPI_SUM,WorldComm);
-      assert(ierr==0);
-      free(fresh);
-    }
-    MPI_Barrier(WorldComm);
-    tA2=(usecond()-t0)/ncall;
-  }
-
-  /*********************************************************
-   * D: chunked Allreduce on the persistent buffer -- hand
-   * decomposition of the primitive (the split-K analogue):
-   * does MPI fail to pipeline internally?
-   *********************************************************/
-  double tD;
-  {
-    size_t chunkwords = (32ULL<<20)/sizeof(double);   // 32MB chunks
-    MPI_Barrier(WorldComm);
-    double t0=usecond();
-    for(int n=0;n<ncall;n++){
-      if ( use_device ) acceleratorMemSet(panel,0,bytes);
-      else              memset(panel,0,bytes);
-      if ( use_device ) {
-#ifdef ACC_HIP
-        auto r=hipMemcpy(panel+WorldRank*mywords,contrib,mybytes,hipMemcpyDeviceToDevice);
-#else
-        acceleratorCopyToDevice(contrib,panel+WorldRank*mywords,mybytes);
-#endif
-      } else {
-        memcpy(panel+WorldRank*mywords,contrib,mybytes);
-      }
-      for(size_t off=0; off<words; off+=chunkwords){
-        size_t cw = std::min(chunkwords, words-off);
-        int ierr=MPI_Allreduce(MPI_IN_PLACE,panel+off,(int)cw,MPI_DOUBLE,MPI_SUM,WorldComm);
-        assert(ierr==0);
-      }
-    }
-    MPI_Barrier(WorldComm);
-    tD=(usecond()-t0)/ncall;
-  }
-
-  /*********************************************************
-   * B: Allgatherv -- owners-only send, identical result
-   *********************************************************/
-  {
-    std::vector<int> counts(WorldSize,(int)mywords);
-    std::vector<int> displs(WorldSize);
-    for(int r=0;r<WorldSize;r++) displs[r]=r*(int)mywords;
-
-    MPI_Barrier(WorldComm);
-    double t0=usecond();
-    for(int n=0;n<ncall;n++){
-      int ierr=MPI_Allgatherv(contrib,(int)mywords,MPI_DOUBLE,
-                              panel,&counts[0],&displs[0],MPI_DOUBLE,WorldComm);
-      assert(ierr==0);
-    }
-    MPI_Barrier(WorldComm);
-    tB=(usecond()-t0)/ncall;
-  }
-
-  /*********************************************************
-   * C: ring allgather from Sendrecv -- the halo primitive.
-   * P-1 steps; step s passes the block received at step s-1 to
-   * the right neighbour.  Per-step message = panel/P.
-   *********************************************************/
-  {
-    int right=(WorldRank+1)%WorldSize;
-    int left =(WorldRank-1+WorldSize)%WorldSize;
-
-    MPI_Barrier(WorldComm);
-    double t0=usecond();
-    for(int n=0;n<ncall;n++){
-      // start with my own block resident at my slot
-      if ( use_device ) {
-#ifdef ACC_HIP
-        auto r=hipMemcpy(panel+WorldRank*mywords,contrib,mybytes,hipMemcpyDeviceToDevice);
-#else
-        acceleratorCopyToDevice(contrib,panel+WorldRank*mywords,mybytes);
-#endif
-      } else {
-        memcpy(panel+WorldRank*mywords,contrib,mybytes);
-      }
-      for(int s=0;s<WorldSize-1;s++){
-        int sendblock=(WorldRank-s+WorldSize)%WorldSize;
-        int recvblock=(WorldRank-s-1+WorldSize)%WorldSize;
-        int ierr=MPI_Sendrecv(panel+sendblock*mywords,(int)mywords,MPI_DOUBLE,right,s,
-                              panel+recvblock*mywords,(int)mywords,MPI_DOUBLE,left, s,
-                              WorldComm,MPI_STATUS_IGNORE);
-        assert(ierr==0);
-      }
-    }
-    MPI_Barrier(WorldComm);
-    tC=(usecond()-t0)/ncall;
-  }
-
-  if ( !WorldRank ) {
-    double GB=bytes/1024./1024./1024.;
-    printf("\t%10.3f GB\t A allreduce %8.2f\t A2 fresh-buf %8.2f\t D chunked %8.2f\t B allgatherv %8.2f\t C ring %8.2f  GB/s\n",
-           GB,
-           GB/(tA/1.0e6),
-           (tA2>0.0) ? GB/(tA2/1.0e6) : 0.0,
-           GB/(tD/1.0e6),
-           GB/(tB/1.0e6),
-           GB/(tC/1.0e6));
-    fflush(stdout);
-  }
-
-  if ( use_device ) {
-    acceleratorFreeDevice(panel);
-    acceleratorFreeDevice(contrib);
-  } else {
-    free(panel);
-    free(contrib);
-  }
-}
-
-/**************************************
- * Command line junk
- **************************************/
-int main(int argc, char **argv)
-{
-  acceleratorInit();
-
-  MPI_Init(&argc,&argv);
-
-  WorldComm = MPI_COMM_WORLD;
-
-  MPI_Comm_split_type(WorldComm, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL,&WorldShmComm);
-
-  MPI_Comm_rank(WorldComm     ,&WorldRank);
-  MPI_Comm_size(WorldComm     ,&WorldSize);
-
-  MPI_Comm_rank(WorldShmComm     ,&WorldShmRank);
-  MPI_Comm_size(WorldShmComm     ,&WorldShmSize);
-
-  if( !WorldRank ) {
-    printf("***********************************\n");
-    printf("%d ranks\n",WorldSize);
-    printf("%d ranks-per-node\n",WorldShmSize);
-    printf("%d nodes\n",WorldSize/WorldShmSize);fflush(stdout);
-    printf("***********************************\n");
-    printf("Panel gather: every rank owns 1/%d of the rows;\n",WorldSize);
-    printf("every rank must finish holding the whole panel.\n");
-    printf("Effective payload rate = panel bytes / wall.  Same semantics x3:\n");
-    printf("  A zero-fill+Allreduce   B Allgatherv   C ring Sendrecv\n");
-  }
-
-  std::vector<size_t> sizes({ (size_t)8<<20, (size_t)64<<20, (size_t)256<<20, (size_t)1<<30 });
-
-  if( !WorldRank ) {
-    printf("=========================================================\n");
-    printf("= HOST memory                                            \n");
-    printf("=========================================================\n");fflush(stdout);
-  }
-  for( auto sz : sizes ) Benchmark(sz,false,5);
-
-  if( !WorldRank ) {
-    printf("=========================================================\n");
-    printf("= DEVICE memory                                          \n");
-    printf("=========================================================\n");fflush(stdout);
-  }
-  for( auto sz : sizes ) Benchmark(sz,true,5);
-
-  if( !WorldRank ) {
-    printf("=========================================================\n");
-    printf("= DONE   \n");
-    printf("=========================================================\n");
-  }
-  MPI_Finalize();
-}

MPI_benchmark/io_aurora.pbs

@@ -1,137 +0,0 @@
-#!/bin/bash
-
-# Standalone MPI-only I/O reproducer on Aurora.  Host only -- no SYCL, no
-# gpu_tile_compact.sh -- so unlike halo_mpi this needs nothing but MPI:
-#
-#   mpicxx -O2 -std=c++11 io_mpi.cc -o io_mpi
-#
-# 12 ranks per node, one per tile, which is what the machine is.  That is a
-# deliberate difference from io_frontier.slurm (8 per node, one per GCD), so
-# read the comparison carefully -- see WHAT IS AND IS NOT COMPARABLE below.
-
-# Run io_aurora_debug.pbs first.  If the cross validation fails there, this
-# scan is 2 hours of 128 nodes producing numbers for a broken file.
-
-#PBS -q prod
-#PBS -l filesystems=flare
-#PBS -l filesystems=home
-#PBS -l select=128
-#PBS -l walltime=02:00:00
-#PBS -A 15479
-
-cd $PBS_O_WORKDIR
-cp $PBS_NODEFILE nodefile
-
-# Only if mpiexec is not already in the environment.  io_mpi is host only
-# and needs no part of the Grid build environment.
-#source ../../sourceme.sh
-
-##########################################################################
-# Environment.  io_mpi has no OpenMP and never touches a GPU, so one thread
-# per rank and a NUMA NIC policy rather than a GPU one.
-#
-# The MPICH_DBG_* variables are deliberately absent: at 1536 ranks they
-# produce gigabytes of log and perturb the timings they would explain.
-# MPICH_MPIIO_STATS/TIMERS are also off here -- they are per collective and
-# 1536 ranks x 6 rungs x 3 reps is unreadable.  Get them from the debug run.
-##########################################################################
-export OMP_NUM_THREADS=1
-export MPICH_CH4_SHM=XPMEM
-export MPICH_OFI_NIC_POLICY=NUMA
-
-##########################################################################
-# WHAT IS AND IS NOT COMPARABLE WITH THE FRONTIER SCAN
-#
-# Held identical at every rung of both scans:
-#   local volume  8.8.32.128 = 262144 sites x 576 B = 151 MB/rank
-#   file view     32768 contiguous runs of 4608 B per rank
-#   aggregation   k=2, row of 16, 8 extents of 18 MB
-# (verified: 4.4.3.1 at 48 ranks and 4.4.3.2 at 96 ranks give exactly the
-# same plan as Frontier's 4.4.2.1 at 32 ranks.)
-#
-# NOT identical, because 12 ranks/node is 1.5x the clients per node:
-#   record size at a given NODE count is 1.5x Frontier's
-#   client count at a given NODE count is 1.5x Frontier's
-#
-# So compare the two machines at equal RANK count (Aurora 4 nodes vs
-# Frontier 6, and so on) if what you want is equal client count and equal
-# record size; compare at equal NODE count if what you want is each machine
-# used as it is meant to be used.  Both are legitimate, they answer
-# different questions, and a table that does not say which one it is
-# reporting is worthless.  The quantity that carries the MPI-IO pathology --
-# per rank local volume and the resulting file view -- is invariant either
-# way, which is the point.
-##########################################################################
-
-##########################################################################
-# WHICH FILESYSTEM.  Point this at Lustre for the like-for-like comparison
-# with Frontier's Orion.  DAOS is a different architecture -- its numbers
-# are interesting but they are NOT a reproduction of the Frontier result,
-# and mixing them into one table would misrepresent both.  Label every set
-# of numbers with the filesystem it came from.
-##########################################################################
-# PROJECT is the flare project DIRECTORY name, which is not the -A account
-# number.  Set it once; the mkdir below fails loudly rather than writing
-# somewhere unintended.
-PROJECT=LatticeQCD_aesp_CNDA
-WORK=/lus/flare/projects/$PROJECT/$USER/iompi.$PBS_JOBID
-mkdir -p $WORK || { echo "cannot create $WORK -- set PROJECT correctly"; exit 1; }
-cd       $WORK
-
-# Match Frontier's default: no explicit striping.  Record what was inherited.
-lfs getstripe -d $WORK 2>/dev/null || echo "(no lfs getstripe -- not Lustre?)"
-
-# The largest rung writes three files of 232 GB, so budget ~700 GB and check
-# the quota before submitting.  Each run unlinks the three files first, so
-# that is peak usage, not cumulative.
-
-NRANKS=12          # one per tile
-
-BIN=$PBS_O_WORKDIR/io_mpi
-ARGS="--target 4194304 --reps 3"
-
-run () {   # run <nodes> <grid> <mpi> <comment> [extra args...]
-  local nodes=$1 gr=$2 mp=$3 note=$4
-  local ntot=$(( nodes * NRANKS ))
-  shift 4
-  echo
-  echo "==================================================================="
-  echo "=== nodes=$nodes ranks=$ntot grid=$gr mpi=$mp   $note"
-  echo "=== extra: $@"
-  echo "==================================================================="
-  mpiexec -np $ntot -ppn $NRANKS -envall $BIN --grid $gr --mpi $mp $ARGS "$@"
-  echo "=== exit $?"
-}
-
-#####################################################################
-# Phase 0.  Correctness.  Both count branches of MPI_Alltoallv are
-# covered; the labels below were checked, not assumed.  The whole-file
-# crc32 is serial, so keep these small.
-#####################################################################
-run 1 16.16.16.24  2.2.1.3  "correctness, UNIFORM counts, row of 4"      --reps 0 --serial-crc
-run 2 24.12.8.8    3.2.2.2  "correctness, NON-UNIFORM counts, row of 12" --reps 0 --serial-crc
-run 4 16.16.32.24  2.2.4.3  "correctness, NON-UNIFORM, non-zero offset"  --reps 0 --serial-crc --offset 1024
-
-#####################################################################
-# Phase 1.  Weak scan at 151 MB/rank.  Identical plan at every rung:
-# k=2, row of 16, 8 extents of 18 MB, 32768 runs of 4608 B in the view.
-#####################################################################
-#          nodes  global lattice    mpi        record
-run    4     32.32.96.128    4.4.3.1   "7.2 GB"    --no-validate
-run    8     32.32.96.256    4.4.3.2   "14.5 GB"   --no-validate
-run   16     32.32.96.512    4.4.3.4   "29.0 GB"   --no-validate
-run   32     32.32.192.512   4.4.6.4   "58.0 GB"   --no-validate
-run   64     32.32.192.1024  4.4.6.8   "116.0 GB"  --no-validate
-run  128     32.32.384.1024  4.4.12.8  "231.9 GB"  --no-validate
-
-#####################################################################
-# Phase 2.  The three questions a reviewer asks immediately.
-#####################################################################
-run  128     32.32.384.1024  4.4.12.8  "231.9 GB, durable"  --no-validate --fsync --drop-cache
-run  128     32.32.384.1024  4.4.12.8  "231.9 GB, cb hints" --no-validate \
-     --hints romio_cb_write=enable,romio_cb_read=enable,cb_nodes=128,cb_buffer_size=16777216
-run  128     32.32.384.1024  4.4.12.8  "231.9 GB, mem subarray" --no-validate --mem-subarray
-
-echo
-echo "=== done.  Files left in $WORK"
-ls -l $WORK

MPI_benchmark/io_frontier.slurm

@@ -1,109 +0,0 @@
-#!/bin/bash -l
-
-# Standalone MPI-only I/O reproducer on Frontier.  No Grid, no accelerator,
-# so no GCD/NUMA wrapper is needed -- the point of the exercise is that this
-# depends on nothing but an MPI installation and a filesystem.
-#
-#   mpicxx -O2 -std=c++11 io_mpi.cc -o io_mpi
-#
-# Weak scan: the local volume, and therefore the file view structure, is held
-# identical at every rung and only the number of Lustre clients changes:
-#
-#   local volume  8.8.32.128 = 262144 sites x 576 B = 151 MB/rank
-#   file view     32768 contiguous runs of 4608 B per rank, at every rung
-#   aggregate     k=2, row of 16, 8 extents of 18 MB, at every rung
-#
-# so any change in the relative bandwidth of the two lexicographic paths is a
-# property of the client count alone.
-#
-# The PERF lines are MiB/s (bytes/1024/1024/s), which is what BinaryIO.h
-# computes for lastPerf.mbytesPerSecond and prints as "MB/s", so the two
-# tools can be compared directly.  Grid's timed region is used here too:
-# barrier, start, [plan build + exchange + I/O], barrier, stop, quoting the
-# boss rank's stopwatch.  --reuse-plan hoists the plan build out, which is
-# how to show it is not where the time goes; do not use it when comparing
-# against Grid's own numbers.
-#
-# io_aurora.pbs runs 12 ranks per node, one per tile, because that is what
-# that machine is.  The per rank local volume and the file view are the same
-# there as here, but the record size and client count at a given NODE count
-# are 1.5x.  See the header of that script before tabulating the two
-# together.
-
-#SBATCH --job-name=ioMPI
-#SBATCH --nodes=128
-#SBATCH --ntasks-per-node=8
-#SBATCH --cpus-per-task=7
-#SBATCH --time=02:00:00
-#SBATCH --account=phy157_dwf
-#SBATCH --exclusive
-#SBATCH --mem=0
-
-module load cce/21.0.0
-module load cpe/26.03
-
-WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/iompi.$SLURM_JOB_ID
-mkdir -p $WORK
-cd       $WORK
-
-# Do NOT stripe by default.  Wide striping is what rescues the collective at
-# scale and costs every other path a factor of 1.2-2.2; the default layout is
-# what a user gets without knowing to ask.  Uncomment to reproduce that
-# interaction, and record which one you ran.
-#lfs setstripe -c -1 -S 8M $WORK
-lfs getstripe -d $WORK
-
-BIN=$SLURM_SUBMIT_DIR/io_mpi
-ARGS="--target 4194304 --reps 3"
-
-# ROMIO's own view of what it did.  Verbose, but the first thing anyone
-# reading the report will ask for.
-# export MPICH_MPIIO_STATS=1
-# export MPICH_MPIIO_TIMERS=1
-
-run () {   # run <nodes> <grid> <mpi> <comment> [extra args...]
-  local nodes=$1 gr=$2 mp=$3 note=$4
-  local nranks=$(( nodes * 8 ))
-  shift 4
-  echo
-  echo "==================================================================="
-  echo "=== nodes=$nodes ranks=$nranks grid=$gr mpi=$mp   $note"
-  echo "=== extra: $@"
-  echo "==================================================================="
-  srun -N$nodes -n$nranks --ntasks-per-node=8 $BIN --grid $gr --mpi $mp $ARGS "$@"
-  echo "=== exit $?"
-}
-
-#####################################################################
-# Phase 0.  Correctness, including the non-uniform Alltoallv branch
-# (odd process factor in an un-split dimension).  Small, and the
-# whole-file crc32 is serial, so keep the volume down here.
-#####################################################################
-run 1 12.12.8.8   2.2.2.1  "correctness, uniform counts"     --reps 0 --serial-crc
-run 3 24.12.8.8   3.2.2.2  "correctness, NON-uniform counts" --reps 0 --serial-crc
-run 4 16.16.16.32 2.2.2.4  "correctness, non-zero offset"    --reps 0 --serial-crc --offset 1024
-
-#####################################################################
-# Phase 1.  Weak scan, 151 MB/rank.  Timing only.
-#####################################################################
-run    4     32.32.64.128    4.4.2.1   "4.8 GB"    --no-validate
-run    8     32.32.64.256    4.4.2.2   "9.7 GB"    --no-validate
-run   16     32.32.64.512    4.4.2.4   "19.3 GB"   --no-validate
-run   32     32.32.128.512   4.4.4.4   "38.6 GB"   --no-validate
-run   64     32.32.128.1024  4.4.4.8   "77.3 GB"   --no-validate
-run  128     32.32.256.1024  4.4.8.8   "154.6 GB"  --no-validate
-
-#####################################################################
-# Phase 2.  Answer the two questions a reviewer will ask immediately.
-#####################################################################
-# Is the gap an artefact of measuring cache rather than the filesystem?
-run  128     32.32.256.1024  4.4.8.8   "154.6 GB, durable"  --no-validate --fsync --drop-cache
-# Does the collective recover if it is given the hints it wants?
-run  128     32.32.256.1024  4.4.8.8   "154.6 GB, cb hints" --no-validate \
-     --hints romio_cb_write=enable,romio_cb_read=enable,cb_nodes=128,cb_buffer_size=16777216
-# Does the degenerate memory subarray matter?
-run  128     32.32.256.1024  4.4.8.8   "154.6 GB, mem subarray" --no-validate --mem-subarray
-
-echo
-echo "=== done.  Files left in $WORK"
-ls -l $WORK

examples/Example_pvdagm.cc

@@ -1,803 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./tests/Test_padded_cell.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-RealD FineSmootherShift = 0.1;
-int   FineSmootherOrder = 8;
-int   FineSmootherTol   = 0;
-//RealD CoarseSmootherShift = 0.1;
-//int   CoarseSmootherOrder = 8;
-//int   CoarseSmootherTol   = 0;
-RealD CoarseSolverShift   = 0.002;
-RealD CoarseSolverTol     = 0.03;
-int   CoarseSolverOrder   = 200;
-int   CoarseMmax          = 20;   // coarse GCR restart length (was hardcoded 20)
-RealD mass=0.00078;
-void ParseEnvironment(void)
-{
-    
-  if(getenv("MASS") ) mass = atof(getenv("MASS"));
-  if(getenv("FineSmootherShift"))   FineSmootherShift   = atof(getenv("FineSmootherShift"));
-  if(getenv("FineSmootherOrder"))   FineSmootherOrder   = atoi(getenv("FineSmootherOrder"));
-  if(getenv("CoarseSolverShift"))   CoarseSolverShift   = atof(getenv("CoarseSolverShift"));
-  if(getenv("CoarseSolverTol"))     CoarseSolverTol     = atof(getenv("CoarseSolverTol"));
-  if(getenv("CoarseSolverOrder"))   CoarseSolverOrder   = atoi(getenv("CoarseSolverOrder"));
-  if(getenv("CoarseMmax"))          CoarseMmax          = atoi(getenv("CoarseMmax"));
-  if(getenv("DiagInvPrec"))
-  {
-    std::cout << GridLogMessage << "WARNING: DiagInvPrec option REMOVED (diagonal-inverse preconditioning wrecks fine->coarse null-vector inheritance); IGNORED" << std::endl;
-  }
-
-  //  if(getenv("CoarseSmootherShift")) CoarseSmootherShift = atof(getenv("CoarseSmootherShift"));
-  //  if(getenv("CoarseSmootherOrder")) CoarseSmootherOrder = atoi(getenv("CoarseSmootherOrder"));
-  
-  std::cout << GridLogMessage << "PARAM: FineSmootherShift   "<<FineSmootherShift<<std::endl;
-  std::cout << GridLogMessage << "PARAM: FineSmootherOrder   "<<FineSmootherOrder<<std::endl;
-  //  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift "<<CoarseSmootherShift<<std::endl;
-  //  std::cout << GridLogMessage << "PARAM: CoarseSmootherOrder "<<CoarseSmootherOrder<<std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverShift   "<<CoarseSolverShift<<std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverTol     "<<CoarseSolverTol<<std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder   "<<CoarseSolverOrder<<std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseMmax          "<<CoarseMmax<<std::endl;
-
-  std::cout << GridLogMessage << "PARAM: MASS "<<mass<<std::endl;
-}
-
-template <class T> void readFile(T& out, std::string const fname){  
-  #ifdef HAVE_LIME
-    // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-    std::cout << Grid::GridLogMessage << "Reads at: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    // Grid::ScidacReader SR(out.Grid()->IsBoss());
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    SR.readScidacFieldRecord(out, record);
-    SR.close();
-  #endif
-}
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-    SW.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SW.writeScidacFieldRecord(subspace[k], record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SR.readScidacFieldRecord(subspace[k], record);
-    SR.close();
-  #endif
-}
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  int nApp;
-  int nAppDag;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    //    std::cout << GridLogMessage<< "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    nApp++;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    //    std::cout << GridLogMessage<<"AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-    nAppDag++;
-  }
-
-  void clear() {
-    nApp = 0;
-    nAppDag = 0;
-  }
-
-  void getApplications() {
-    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
-    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
-    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
-  }
-
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    //    std::cout <<GridLogMessage<< "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
-  }
-};
-template<class Matrix,class Field>
-class MdagPVLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    //    std::cout <<GridLogMessage<< "Op: PVdag M "<<std::endl;
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    //    std::cout <<GridLogMessage<< "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    //    std::cout << GridLogMessage<<"HermOp: PVdag M Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
-  }
-};
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  int nApp;
-  int nAppDag;
-public:
-  RealD shift;
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV) , nApp(0), nAppDag(0){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    //    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    nApp++;
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    //    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    nAppDag++;
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    //    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-  void clear() {
-    nApp = 0;
-    nAppDag = 0;
-  }
-  void getApplications() {
-    std::cout << GridLogMessage << "# applications of ShiftedPVdagM: " << nApp << std::endl;
-    std::cout << GridLogMessage << "# applications of ShiftedPVdagM^dag: " << nAppDag << std::endl;
-    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
-  }
-};
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-  std::string name;
-
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_,
-		   std::string _name = std::string("unnamed"))
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      name(_name),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out) 
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    //    auto CoarseGrid = _CoarseOperator.Grid();
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    std::cout<<GridLogMessage << "Calling PreSmoother " <<std::endl;
-
-    //    std::cout<<GridLogMessage << "Calling PreSmoother input residual "<<norm2(in) <<std::endl;
-    double t;
-    // Fine Smoother
-    //    out = in;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-
-    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Update the residual
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);   
-    //    std::cout<<GridLogMessage <<"Residual-1 now " <<norm2(vec1)<<std::endl;
-
-    // Fine to Coarse 
-    t=-usecond();
-    _Aggregates.ProjectToSubspace  (Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Coarse correction
-    t=-usecond();
-    Csol = Zero();
-    _CoarseSolve(Csrc,Csol);
-    //Csol=Zero();
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Coarse to Fine
-    t=-usecond();  
-    //    _CoarseOperator.PromoteFromSubspace(_Aggregates,Csol,vec1);
-    _Aggregates.PromoteFromSubspace(Csol,vec1); 
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Residual
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);  
-    //    std::cout<<GridLogMessage <<"Residual-2 now " <<norm2(vec1)<<std::endl;
-
-    // Fine Smoother
-    t=-usecond();
-    //    vec2=vec1;
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add( out,out,vec2);
-    std::cout<<GridLogMessage << "Done " <<std::endl;
-  }
-};
-
-template<class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
-void runMG(
-  GridCartesian *FGrid,
-  GridCartesian *Coarse5d, 
-  NextToNearestStencilGeometry5D geom,
-  PVdagM_t PVdagM,
-  ShiftedPVdagM_t ShiftedPVdagM,
-  // std::vector<LatticeFermion> subspace
-  Subspace AggregatesPD
-) {
-
-  // typedef Aggregation<vSpinColourVector,vTComplex,nbasis> Subspace;
-  // typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
-  // typedef LittleDiracOperator::CoarseVector CoarseVector;
-  ParseEnvironment();
-
-  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
-  int nbasis = subspace.size();
-  const int cb = 0 ;
-
-  LatticeFermion err(FGrid);
-  LatticeFermion prom(FGrid);
-  LatticeFermion tmp(FGrid);
-
-  CoarseVector c_src (Coarse5d);
-  CoarseVector c_res (Coarse5d);
-  CoarseVector c_proj(Coarse5d);
-  Complex one(1.0);
-
-  LatticeFermionD f_src(FGrid);
-  LatticeFermionD f_res(FGrid);
-  // typedef MGPreconditioner<vSpinColourVector,  vTComplex,nbasis> TwoLevelMG;
-  TrivialPrecon<CoarseVector> simple;
-  TrivialPrecon<LatticeFermionD> simple_fine;
-
-  // Subspace AggregatesPD(Coarse5d,FGrid,cb);
-
-  // Orthonormalize subspace and compute nulliness
-
-  ShiftedPVdagM.shift = CoarseSolverShift;
-  int nonherm = 0;
-  LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d,nonherm);
-  LittleDiracOpPV.CoarsenOperator(ShiftedPVdagM, AggregatesPD);
-  ShiftedPVdagM.shift = FineSmootherShift;
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"Testing coarsened operator "<<std::endl;
-
-  c_src = one;  // 1 in every element for vector 1.
-  blockPromote(c_src,err,subspace);
-
-  prom=Zero();
-  for(int b=0;b<nbasis;b++){
-    prom=prom+subspace[b];
-  }
-  err=err-prom; 
-  std::cout<<GridLogMessage<<"Promoted back from subspace: err "<<norm2(err)<<std::endl;
-  std::cout<<GridLogMessage<<"c_src "<<norm2(c_src)<<std::endl;
-  std::cout<<GridLogMessage<<"prom  "<<norm2(prom)<<std::endl;
-
-  // PVdagM.Op(prom,tmp);
-  // blockProject(c_proj,tmp,subspace);
-  // std::cout<<GridLogMessage<<" Called Big Dirac Op "<<norm2(tmp)<<std::endl;
-
-  // LittleDiracOpPV.M(c_src,c_res);
-  // std::cout<<GridLogMessage<<" Called Little Dirac Op c_src "<< norm2(c_src) << "  c_res "<< norm2(c_res) <<std::endl;
-
-  // std::cout<<GridLogMessage<<"Little dop : "<<norm2(c_res)<<std::endl;
-  // //  std::cout<<GridLogMessage<<" Little "<< c_res<<std::endl;
-  // std::cout<<GridLogMessage<<"Big dop in subspace : "<<norm2(c_proj)<<std::endl;
-  // //  std::cout<<GridLogMessage<<" Big "<< c_proj<<std::endl;
-  // c_proj = c_proj - c_res;
-  // std::cout<<GridLogMessage<<" ldop error: "<<norm2(c_proj)<<std::endl;
-  // //  std::cout<<GridLogMessage<<" error "<< c_proj<<std::endl;
-
-  ///////////////////////////////////////
-  // Coarse grid solver test
-  ///////////////////////////////////////
-
-  std::cout<<GridLogMessage<<"******************* "<<std::endl;
-  std::cout<<GridLogMessage<<" Coarse Grid Solve -- Level 2 "<<std::endl;
-  std::cout<<GridLogMessage<<"******************* "<<std::endl;
-  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-  // DiagonalInverse preconditioning REMOVED (library support withdrawn: it
-  // wrecks the collinearity that makes fine->coarse null-vector inheritance
-  // free).  TrivialPrecon reproduces the former DiagInvPrec=0 path exactly.
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>  L2PGCR(CoarseSolverTol, (CoarseSolverOrder+CoarseMmax-1)/CoarseMmax, LinOpCoarse,simple,CoarseMmax,CoarseMmax);
-  L2PGCR.SetZeroGuess(1);              // callers zero Csol / c_res
-  L2PGCR.Level(2);
-  L2PGCR.Name("Couter");
-  c_res=Zero();
-  L2PGCR(c_src,c_res);
-
-
-  ////////////////////////////////////////
-  // Fine grid smoother
-  ////////////////////////////////////////
-  //  NonHermitianLinearOperator<PVdagM_t,LatticeFermionD> LinOpSmooth(PVdagM);
-  
-  //  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.05,1,ShiftedPVdagM,simple_fine,8,8);
-  // Force 10 iters exactly, no early termination
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(FineSmootherTol,1,
-									    ShiftedPVdagM,simple_fine,
-									    FineSmootherOrder,FineSmootherOrder);
-  SmootherGCR.Level(1);
-  SmootherGCR.Name("Fsmoother");
-  SmootherGCR.SetZeroGuess(1);         // pre/post slots + direct call all zero their guess
-
-  f_src = one;  // 1 in every element for vector 1.
-  f_res=Zero();
-  SmootherGCR(f_src,f_res);
-
-  TwoLevelMG TwoLevelPrecon(AggregatesPD,
-			    PVdagM,
-			    simple_fine,
-			    SmootherGCR,
-			    LinOpCoarse,
-			    L2PGCR,
-			    "PVdagM");
-  
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,TwoLevelPrecon,32,32);
-  L1PGCR.SetZeroGuess(1);              // f_res=Zero() before the solve
-  L1PGCR.Level(1);
-  L1PGCR.Name("Fouter");
-
-  std::cout<<GridLogMessage<<"******************* "<<std::endl;
-  std::cout<<GridLogMessage<<" Running Multi Grid Solver "<<std::endl;
-  std::cout<<GridLogMessage<<"******************* "<<std::endl;
-  f_res=Zero();
-  L1PGCR(f_src,f_res);
-
-  std::cout << GridLogMessage << "Fine Grid Smoother -- Level 2 operator uses: " << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-  ShiftedPVdagM.getApplications();
-  ShiftedPVdagM.clear();
-
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  // TODO read in more parameters: nbasis, GCR iters, smoother order, m
-  // Might be impossible because nbasis needs to be a constant to be a template parameter
-  // Usage : $ ./Example_pvdagm <nbasis> <smooth> <outerIters> <m>
-  // std::string nbasisStr  = argv[1];
-  // std::string smoothStr  = argv[2];
-  // std::string outerStr   = argv[3];
-  // std::string mStr       = argv[4];
-  // int nbasis = std::stoi(nbasisStr);
-  // int smooth = std::stoi(smoothStr);
-
-  const int Ls=24;
-  RealD M5=1.8;
-
-  
-  //  const int nbasis = 40;
-  const int nbasis = 60;
-  
-  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", running Mobius kernel with b=1.5, c=0.5" << std::endl;
-  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
-
-  std::vector<int> lat_size {48, 48, 48, 96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Construct a coarsened grid
-  // Coordinate clatt = GridDefaultLatt();
-  Coordinate clatt = lat_size;
-  Coordinate Block({4,4,4,4});
-  std::cout << GridLogMessage << "Lattice size: " << lat_size << std::endl;
-  for(int d=0;d<clatt.size();d++){
-    clatt[d] = lat_size[d]/Block[d];
-  }
-
-  std::cout << GridLogMessage << "constructing coarse grid" << std::endl;
-  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  std::vector<int> cseeds({5,6,7,8});
-  GridParallelRNG          RNG5(FGrid);   RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG          RNG4(UGrid);   RNG4.SeedFixedIntegers(seeds4);
-  GridParallelRNG          CRNG(Coarse5d);CRNG.SeedFixedIntegers(cseeds);
-
-  LatticeFermion    src(FGrid); random(RNG5,src);
-  LatticeFermion result(FGrid); result=Zero();
-  LatticeFermion    ref(FGrid); ref=Zero();
-  LatticeFermion    tmp(FGrid);
-  LatticeFermion    err(FGrid);
-  LatticeGaugeField Umu(UGrid);
-
-
-  std::cout << GridLogMessage << "Reading in gauge field" << std::endl;
-  FieldMetaData header;
-  // std::string file("/sdcc/u/poare/PETSc-Grid/ckpoint_lat.4000");
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  /*
-
-  // DWF, m=0.01
-  // std::string eigenPath = "/hpcgpfs01/work/lqcd/staging/RBC/ckpoint_lat.4000/ks_evecs/PVdagM_Nm80_Nk40_Niter5000_337342/";
-
-  // DWF, m=0.001
-  // std::string eigenPath = "/hpcgpfs01/work/lqcd/staging/RBC/ckpoint_lat.4000/ks_evecs/PVdagM_Nm80_Nk40_Niter5000_m0p001_339143/";
-
-  // Mobius, m=0.001
-  // std::string eigenPath = "/hpcgpfs01/work/lqcd/staging/RBC/ckpoint_lat.4000/ks_evecs/PVdagM_Nm80_Nk40_Niter5000_346851/";
-
-  // Frontier path
-  std::string eigenPath = "/ccs/home/poare/lqcd/multigrid/spectra/ckpoint_lat.1000/...";
-  
-
-  std::cout << GridLogMessage << "Loading eigenvalues" << std::endl;
-  std::ifstream evalFile(eigenPath + "evals.txt");
-  std::string str;
-  std::vector<ComplexD> evals;
-  while (std::getline(evalFile, str)) {
-      std::cout << GridLogMessage << "Reading line: " << str << std::endl;
-      int i1 = str.find("(") + 1;
-      int i2 = str.find(",") + 1;
-      int i3 = str.find(")");
-      std::cout << "i1,i2,i3 = " << i1 << "," << i2 << "," << i3 << std::endl;
-      std::string reStr = str.substr(i1, i2 - i1);
-      std::string imStr = str.substr(i2, i3 - i2);
-      std::cout << GridLogMessage << "Parsed re = " << reStr << " and im = " << imStr << std::endl;
-      // ComplexD z (std::stof(reStr), std::stof(imStr));
-      ComplexD z (std::stod(reStr), std::stod(imStr));
-      evals.push_back(z);
-  }
-  std::cout << GridLogMessage << "Eigenvalues: " << evals << std::endl;
-
-  int Nevecs = 20;
-  std::vector<LatticeFermion> evecs;
-  LatticeFermion evec (FGrid);
-  for (int i = 0; i < Nevecs; i++) {
-    std::string evecPath = eigenPath + "evec" + std::to_string(i);
-    readFile(evec, evecPath);
-    evecs.push_back(evec);
-  }
-  std::cout << GridLogMessage << "Evecs loaded" << std::endl;
-
-  */
-  // TODO uncomment when evecs are computed!
-
-  // DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
-  // DomainWallFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5);
-
-  // Mobius
-  RealD b=1.5;// Scale factor b+c=2, b-c=1
-  RealD c=0.5;
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
-  MobiusFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5,b,c);
-
-  const int cb = 0 ;
-  LatticeFermion prom(FGrid);
-
-  // assert(nbasis <= Nevecs);     // need to have enough evecs
-
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector CoarseVector;
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-
-  // typedef PVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> PVdagM_t;
-  // typedef MdagPVLinearOperator<DomainWallFermionD,LatticeFermionD> MdagPV_t;
-  // typedef ShiftedPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD> PVdagM_t;
-  typedef MdagPVLinearOperator<MobiusFermionD,LatticeFermionD> MdagPV_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-
-  PVdagM_t PVdagM(Ddwf,Dpv);
-  MdagPV_t MdagPV(Ddwf,Dpv);
-  //  ShiftedPVdagM_t ShiftedPVdagM(2.0,Ddwf,Dpv); // 355
-  //  ShiftedPVdagM_t ShiftedPVdagM(1.0,Ddwf,Dpv); // 246
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.5,Ddwf,Dpv); // 183
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // 145
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 134
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 127 -- NULL space via inverse iteration
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 57 -- NULL space via inverse iteration; 3 iterations
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // 57 , tighter inversion
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // nbasis 20 -- 49 iters
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // nbasis 20 -- 70 iters; asymmetric 
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // 58; Loosen coarse, tighten fine
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 56 ... 
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 51 ...  with 24 vecs
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 31 ...  with 24 vecs and 2^4 blocking
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 43 ...  with 16 vecs and 2^4 blocking, sloppier
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 35  ...  with 20 vecs and 2^4 blocking
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 35  ...  with 20 vecs and 2^4 blocking, looser coarse
-  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 64  ...  with 20 vecs, Christoph setup, and 2^4 blocking, looser coarse
-
-  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv); // 
-
-  // Run power method on HOA??
-  PowerMethod<LatticeFermion>       PM;
-
-  CoarseVector c_src (Coarse5d);
-  CoarseVector c_res (Coarse5d);
-  CoarseVector c_proj(Coarse5d);
-  Complex one(1.0);
-
-  std::vector<LatticeFermion> subspace(nbasis,FGrid);
-
-  LatticeFermionD f_src(FGrid);
-  LatticeFermionD f_res(FGrid);
-  typedef MGPreconditioner<vSpinColourVector,  vTComplex,nbasis> TwoLevelMG;
-  TrivialPrecon<CoarseVector> simple;
-  TrivialPrecon<LatticeFermionD> simple_fine;
-
-  // Warning: This routine calls PVdagM.Op, not PVdagM.HermOp
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis> Subspace;
-
-  // Breeds right singular vectors with call to HermOp (V)
-  // int chebyOrd = 500;
-  // V.CreateSubspaceChebyshev(RNG5,PVdagM,
-	// 		    nbasis,
-	// 		    4000.0,0.003,
-	// 		    chebyOrd);
-  // AggregatesPD.CreateSubspaceChebyshev(RNG5,
-	// 			       PVdagM,
-	// 			       nbasis,
-	// 			       4000.0,
-	// 			       0.003,
-	// 			       chebyOrd);
-  
-  // Subspace testing (uncomment blocks when needed)
-
-  // - nbasis = 20, m=0.01, 35 outer iterations
-  // - nbasis = 40, m=0.01, 23 outer iterations
-  std::cout << GridLogMessage << "*** GCR setup ***" << std::endl;
-
-  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
-  // Set SUBSPACE_FILE to override the default path.
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                              + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-
-  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
-  uint64_t file_exists = 0;
-  if ( UGrid->IsBoss() ) {
-    std::ifstream f(subspace_file);
-    file_exists = f.good() ? 1 : 0;
-  }
-  UGrid->GlobalSum(file_exists);
-
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-
-  if ( file_exists ) {
-    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-    // Insurance: GLOBAL (whole-lattice) orthonormalise, matching what
-    // CreateSubspaceGCR applies to generated subspaces (Aggregates.h:196), so a
-    // reloaded file ends in the same state.  This replaces the block
-    // Orthogonalise() previously called here -- that is redundant (CoarsenOperator
-    // block-GS's the subspace internally) and would leave a loaded file block-
-    // orthonormal while a generated one is globally orthonormal.  Global GS is
-    // span-preserving, so the coarse operator is unchanged.
-    AggregatesGCR.GlobalOrthonormalise();
-    std::cout << GridLogMessage << "Subspace loaded and globally orthonormalised." << std::endl;
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
-    PVdagM.getApplications();
-    PVdagM.clear();
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
-  }
-  
-  std::cout << GridLogMessage << "Basis construction operator uses: " << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-
-  std::cout << GridLogMessage << "Calling runMG " << std::endl;
-  runMG<PVdagM_t, ShiftedPVdagM_t, Subspace, LittleDiracOperator, CoarseVector, TwoLevelMG>(
-    FGrid, 
-    Coarse5d, 
-    geom, 
-    PVdagM, 
-    ShiftedPVdagM, 
-    AggregatesGCR
-  );
-
-  //////////////////////////////////
-  // Standard CG
-  //////////////////////////////////
-#if 0
-  {
-  std::cout << "**************************************"<<std::endl;
-  std::cout << "Calling red black CG"<<std::endl;
-  std::cout << "**************************************"<<std::endl;
-    ConjugateGradient<LatticeFermionD>  CGfine(1.0e-8,30000,false);
-    SchurDiagMooeeOperator<MobiusFermionD, LatticeFermion> HermOpEO(Ddwf);
-      
-    LatticeFermion result(FrbGrid); result=Zero();
-    LatticeFermion    src(FrbGrid); random(RNG5,src);
-    result=Zero();
-
-    CGfine(HermOpEO, src, result);
-  }
-  {
-    std::cout << "**************************************"<<std::endl;
-    std::cout << "Calling MdagM CG"<<std::endl;
-    std::cout << "**************************************"<<std::endl;
-      
-    LatticeFermion result(FGrid); result=Zero();
-    LatticeFermion    src(FGrid); random(RNG5,src);
-    result=Zero();
-
-    MdagMLinearOperator<MobiusFermionD, LatticeFermionD> HermOp(Ddwf);
-    ConjugateGradient<LatticeFermionD>  CGfine(1.0e-8,100000,false);
-    CGfine(HermOp, src, result);
-  }
-  {
-    std::cout << "**************************************"<<std::endl;
-    std::cout << "Calling PVdagM GCR"<<std::endl;
-    std::cout << "**************************************"<<std::endl;
-
-    LatticeFermion result(FGrid); result=Zero();
-    LatticeFermion    src(FGrid); random(RNG5,src);
-    result=Zero();
-    PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> GCR(1.0e-8,3000,PVdagM,simple_fine,50,50);
-    GCR.Name("Fbaseline");
-    GCR.SetZeroGuess(1);               // result=Zero() above
-    GCR(src,result);
-  }
-#endif  
-
-  
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_3level.cc

@@ -1,699 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_3level.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-
-template <class T> void readFile(T& out, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    SR.readScidacFieldRecord(out, record);
-    SR.close();
-  #endif
-}
-
-template <class T> void writeFile(T& in, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(in.Grid()->IsBoss());
-    SW.open(fname);
-    SW.writeScidacFieldRecord(in, record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-    SW.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SW.writeScidacFieldRecord(subspace[k], record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SR.readScidacFieldRecord(subspace[k], record);
-    SR.close();
-  #endif
-}
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  int nApp;
-  int nAppDag;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    nApp++;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-    nAppDag++;
-  }
-  void clear() { nApp = 0; nAppDag = 0; }
-  void getApplications() {
-    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
-    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
-    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class MdagPVLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
-// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
-template<class Field>
-class LuscherGuesser : public LinearFunction<Field> {
-  const std::vector<Field> &psi;
-  Eigen::MatrixXcd          C_inv;
-public:
-  using LinearFunction<Field>::operator();
-  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
-    : psi(psi_), C_inv(Cinv_) {}
-  virtual void operator()(const Field &src, Field &guess) {
-    int N = psi.size();
-    Eigen::VectorXcd b(N);
-    for (int t = 0; t < N; t++)
-      b(t) = TensorRemove(innerProduct(psi[t], src));
-    Eigen::VectorXcd c = C_inv * b;
-    guess = Zero();
-    for (int s = 0; s < N; s++)
-      guess += ComplexD(c(s)) * psi[s];
-  }
-};
-
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-  CoarseSolver   & _CoarseGuesser;
-
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_,
-		   CoarseSolver &CoarseGuesser_)
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      _CoarseGuesser(CoarseGuesser_),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out)
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    double t;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
-
-    t=-usecond();
-    _Aggregates.ProjectToSubspace(Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _CoarseGuesser(Csrc,Csol);
-    _CoarseSolve(Csrc,Csol);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _Aggregates.PromoteFromSubspace(Csol,vec1);
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
-
-    t=-usecond();
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add(out,out,vec2);
-  }
-};
-
-// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
-// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
-// at the fine level.
-template<class Field>
-class ShiftedLinearOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op;
-  RealD shift;
-public:
-  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
-  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
-  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
-  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
-  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
-  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
-  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
-  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
-void runMG(
-  GridCartesian *FGrid,
-  GridCartesian *Coarse5d,
-  GridCartesian *CoarseCoarse5d,
-  NextToNearestStencilGeometry5D geom,
-  PVdagM_t &PVdagM,
-  ShiftedPVdagM_t &ShiftedPVdagM,
-  Subspace &AggregatesPD
-) {
-  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
-  assert((int)subspace.size() == NB);
-  const int nbasis = NB;
-  const int cb = 0;
-
-  CoarseVector c_src(Coarse5d);
-  CoarseVector c_res(Coarse5d);
-  Complex one(1.0);
-
-  LatticeFermionD f_src(FGrid);
-  LatticeFermionD f_res(FGrid);
-
-  TrivialPrecon<CoarseVector>      simpleC;
-  TrivialPrecon<LatticeFermionD>   simple_fine;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 0→1: coarsen PVdagM, build LinOpCoarse
-  //////////////////////////////////////////////////////////////////////
-  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
-  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
-
-  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-
-  //////////////////////////////////////////////////////////////////////
-  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
-  L2PGCR_baseline.Level(2);
-  L2PGCR_baseline.Name("Cbaseline");
-  c_src = one;
-  c_res = Zero();
-  L2PGCR_baseline(c_src,c_res);
-
-  //////////////////////////////////////////////////////////////////////
-  // psi_coarse: coarse projections of pre-GS fine null vectors.
-  // These are the Level 1 near-null vectors, promoted from Level 0.
-  // Used as the aggregation basis for Level 1→2 coarsening.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
-
-  //////////////////////////////////////////////////////////////////////
-  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
-  //////////////////////////////////////////////////////////////////////
-  {
-    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    LatticeFermion ftmp(FGrid);
-    for (int j = 0; j < nbasis; j++) {
-      PVdagM.Op(subspace[j], ftmp);
-      for (int i = 0; i < nbasis; i++)
-        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
-    }
-    RealD normW = W.norm();
-    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
-
-    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    CoarseVector Ac(Coarse5d);
-    for (int l = 0; l < nbasis; l++) {
-      LinOpCoarse.Op(psi_coarse[l], Ac);
-      for (int k = 0; k < nbasis; k++)
-        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
-    }
-    RealD normC      = C.norm();
-    RealD normCmCdag = (C - C.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
-    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
-    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2: set up aggregation using psi_coarse as subspace.
-  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
-  // geometry 3×6×4×4 where z-local at Level 1 is 6).
-  // psi_coarse are assigned directly; CoarsenOperator performs
-  // block-GS orthogonalisation before building LinOpCoarseCoarse.
-  //////////////////////////////////////////////////////////////////////
-  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
-  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
-  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
-  typedef iScalar<vTComplex>                                              vTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
-  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
-  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
-  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
-
-  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.subspace[k] = psi_coarse[k];
-
-  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
-  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
-  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
-
-  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
-
-  TrivialPrecon<CoarseCoarseVector> simpleCC;
-
-  //////////////////////////////////////////////////////////////////////
-  // Lüscher deflation guesser for L3PGCR.
-  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
-  //         CoarseCoarseVector space — these cover the zero-momentum
-  //         component of the near-null space of LinOpCC.
-  // Step 2: breed Nextra additional null vectors directly on LinOpCC
-  //         using GCR with random sources — these pick up near-null
-  //         modes at all spatial frequencies not spanned by step 1.
-  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
-  //         full augmented basis and invert directly via Eigen LU.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
-
-  {
-    int Nextra = nbasis;  // breed as many extra as we have promoted ones
-    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
-    GridParallelRNG RNG_CC(CoarseCoarse5d);
-    RNG_CC.SeedFixedIntegers({11,13,17,19});
-    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
-    CoarseCoarseVector tmp(CoarseCoarse5d);
-    for (int k = 0; k < Nextra; k++) {
-      CoarseCoarseVector src(CoarseCoarse5d);
-      gaussian(RNG_CC, src);
-      tmp = Zero();
-      nullGCR(src, tmp);
-      psi_cc.push_back(tmp);
-    }
-    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
-              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
-  }
-
-  const int Naug = psi_cc.size();
-  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
-  {
-    CoarseCoarseVector Acc(CoarseCoarse5d);
-    for (int l = 0; l < Naug; l++) {
-      LinOpCC.Op(psi_cc[l], Acc);
-      for (int k = 0; k < Naug; k++)
-        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
-    }
-  }
-  {
-    RealD normCcc      = Ccc.norm();
-    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
-    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
-  }
-  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
-  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 2 solver: plain GCR, no further coarsening
-  //////////////////////////////////////////////////////////////////////
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3PGCR(1.0e-1,200,LinOpCC,simpleCC,16,16);
-  L3PGCR.Level(3);
-  L3PGCR.Name("CCouter");
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarse-level GCR smoother for Level 1→2 V-cycle.
-  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
-  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
-  //////////////////////////////////////////////////////////////////////
-  RealD coarse_smoother_shift = 0.01;
-  int   coarse_smoother_nstep = 12;
-  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
-  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
-     
-  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
-  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
-  CoarseSmootherGCR.Level(2);
-  CoarseSmootherGCR.Name("Csmoother");
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2 V-cycle preconditioner.
-  //////////////////////////////////////////////////////////////////////
-  L1to2MG L1to2Precon(AggregatesL2,
-                       LinOpCoarse,
-                       simpleC,            // no pre-smoother (matches fine-grid setup)
-                       CoarseSmootherGCR,  // post-smoother: 12 GCR steps
-                       LinOpCC,
-                       L3PGCR,
-                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
-
-  //////////////////////////////////////////////////////////////////////
-  // Standalone Level 1 two-level solve test.
-  // Compare against plain PGCR baseline above.
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
-  L2MGsolver.Level(2);
-  L2MGsolver.Name("Couter");
-  c_res = Zero();
-  L2MGsolver(c_src,c_res);
-
-  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-
-  //////////////////////////////////////////////////////////////////////
-  // Full three-level outer solve
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
-  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
-  SmootherGCR.Level(1);
-  SmootherGCR.Name("Fsmoother");
-
-  f_src = one;
-
-  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
-  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
-  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
-                               PVdagM,
-                               simple_fine,
-                               SmootherGCR,
-                               LinOpCoarse,
-                               L2MGsolver,
-                               simpleC);
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
-  L1PGCR.Level(1);
-  L1PGCR.Name("Fouter");
-
-  f_res = Zero();
-  L1PGCR(f_src,f_res);
-
-  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls   = 24;
-  RealD     M5   = 1.8;
-  RealD     b    = 1.5;
-  RealD     c    = 0.5;
-  RealD     mass = 0.00078;
-  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
-
-  const int nbasis = 60;
-
-  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
-  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
-
-  std::vector<int> lat_size {48, 48, 48, 96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
-  Coordinate clatt = lat_size;
-  for (int d = 0; d < 4; d++) clatt[d] /= 2;
-  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
-
-  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
-  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
-  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
-  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
-  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
-  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
-  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
-  Coordinate clatt2 = clatt;
-  clatt2[0] /= 2;
-  clatt2[1] /= 2;
-  clatt2[2] /= 3;
-  clatt2[3] /= 3;
-  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
-
-  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  RealD b_ = 1.5;
-  RealD c_ = 0.5;
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
-  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
-  // Set SUBSPACE_FILE to override the default path.
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                              + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-
-  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
-  uint64_t file_exists = 0;
-  if ( UGrid->IsBoss() ) {
-    std::ifstream f(subspace_file);
-    file_exists = f.good() ? 1 : 0;
-  }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb = 0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-
-  if ( file_exists ) {
-    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-    // Insurance: GLOBAL (whole-lattice) orthonormalise, matching CreateSubspaceGCR
-    // (Aggregates.h:196), in case the cached file predates it.  Span-preserving
-    // and globally orthonormal -- NOT the block Orthogonalise() below, which would
-    // defeat the raw-null discipline (runMG promotes the RAW subspace to build L2;
-    // block-GS here -> psi_coarse = e_k).  The raw copy in runMG happens AFTER this.
-    AggregatesGCR.GlobalOrthonormalise();
-    //    AggregatesGCR.Orthogonalise();
-    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
-    PVdagM.getApplications();
-    PVdagM.clear();
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
-  }
-
-  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
-    FGrid,
-    Coarse5d,
-    CoarseCoarse5d,
-    geom,
-    PVdagM,
-    ShiftedPVdagM,
-    AggregatesGCR
-  );
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_3level_madj.cc

@@ -1,625 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_3level.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-
-template <class T> void readFile(T& out, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    SR.readScidacFieldRecord(out, record);
-    SR.close();
-  #endif
-}
-
-template <class T> void writeFile(T& in, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(in.Grid()->IsBoss());
-    SW.open(fname);
-    SW.writeScidacFieldRecord(in, record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-    SW.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SW.writeScidacFieldRecord(subspace[k], record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SR.readScidacFieldRecord(subspace[k], record);
-    SR.close();
-  #endif
-}
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  int nApp;
-  int nAppDag;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    nApp++;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-    nAppDag++;
-  }
-  void clear() { nApp = 0; nAppDag = 0; }
-  void getApplications() {
-    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
-    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
-    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class MdagPVLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-  CoarseSolver   & _CoarseGuesser;
-  std::string      _name;
-  
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_,
-		   CoarseSolver &CoarseGuesser_,
-		   std::string name = std::string("unnamed"))
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      _CoarseGuesser(CoarseGuesser_),
-      _name(name),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out)
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    double t;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-    std::cout<<GridLogMessage << _name << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
-
-    t=-usecond();
-    _Aggregates.ProjectToSubspace(Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _CoarseGuesser(Csrc,Csol);
-    _CoarseSolve(Csrc,Csol);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _Aggregates.PromoteFromSubspace(Csol,vec1);
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << _name << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
-
-    t=-usecond();
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << _name <<"PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add(out,out,vec2);
-  }
-};
-
-// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
-// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
-// at the fine level.
-template<class Field>
-class ShiftedLinearOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op;
-  RealD shift;
-public:
-  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
-  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
-  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
-  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
-  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
-  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
-  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
-  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
-void runMG(
-  GridCartesian *FGrid,
-  GridCartesian *Coarse5d,
-  GridCartesian *CoarseCoarse5d,
-  NextToNearestStencilGeometry5D geom,
-  PVdagM_t &PVdagM,
-  ShiftedPVdagM_t &ShiftedPVdagM,
-  Subspace &AggregatesPD
-) {
-  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
-  assert((int)subspace.size() == NB);
-  const int nbasis = NB;
-  const int cb = 0;
-
-  CoarseVector c_src(Coarse5d);
-  CoarseVector c_res(Coarse5d);
-  Complex one(1.0);
-
-  LatticeFermionD f_src(FGrid);
-  LatticeFermionD f_res(FGrid);
-
-  TrivialPrecon<CoarseVector>      simpleC;
-  TrivialPrecon<LatticeFermionD>   simple_fine;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 0→1: coarsen PVdagM, build LinOpCoarse
-  //////////////////////////////////////////////////////////////////////
-  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
-  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
-
-  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-
-  //////////////////////////////////////////////////////////////////////
-  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
-  L2PGCR_baseline.Level(2);
-  c_src = one;
-  c_res = Zero();
-  L2PGCR_baseline(c_src,c_res);
-  PVdagM.getApplications();
-  PVdagM.clear();
-
-  //////////////////////////////////////////////////////////////////////
-  // psi_coarse: coarse projections of pre-GS fine null vectors.
-  // These are the Level 1 near-null vectors, promoted from Level 0.
-  // Used as the aggregation basis for Level 1→2 coarsening.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
-
-  //////////////////////////////////////////////////////////////////////
-  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
-  //////////////////////////////////////////////////////////////////////
-  {
-    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    LatticeFermion ftmp(FGrid);
-    for (int j = 0; j < nbasis; j++) {
-      PVdagM.Op(subspace[j], ftmp);
-      for (int i = 0; i < nbasis; i++)
-        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
-    }
-    RealD normW = W.norm();
-    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
-
-    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    CoarseVector Ac(Coarse5d);
-    for (int l = 0; l < nbasis; l++) {
-      LinOpCoarse.Op(psi_coarse[l], Ac);
-      for (int k = 0; k < nbasis; k++)
-        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
-    }
-    RealD normC      = C.norm();
-    RealD normCmCdag = (C - C.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
-    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
-    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2: set up aggregation using psi_coarse as subspace.
-  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
-  // geometry 3×6×4×4 where z-local at Level 1 is 6).
-  // psi_coarse are assigned directly; CoarsenOperator performs
-  // block-GS orthogonalisation before building LinOpCoarseCoarse.
-  //////////////////////////////////////////////////////////////////////
-  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
-  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
-  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
-  typedef iScalar<vTComplex>                                              vTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
-  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
-  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
-  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
-
-  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.subspace[k] = psi_coarse[k];
-
-  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
-  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
-  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
-
-  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 2 solver: plain GCR, no further coarsening
-  //////////////////////////////////////////////////////////////////////
-  TrivialPrecon<CoarseCoarseVector> simpleCC;
-  // L3PGCR is an inner solver inside the L1→2 V-cycle; does not need to converge
-  // to fine-grid precision. Loose tolerance (3e-2) and large restart (64) to allow
-  // the Krylov space to span enough of the near-null spectrum of LinOpCC per cycle.
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3PGCR(1.0e-4,5,LinOpCC,simpleCC,64,64);
-  L3PGCR.Level(3);
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarse-level GCR smoother for Level 1→2 V-cycle.
-  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
-  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
-  //////////////////////////////////////////////////////////////////////
-  RealD coarse_smoother_shift = 0.0;
-  int   coarse_smoother_nstep = 8;
-  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
-  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
-     
-  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.0,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
-  CoarseSmootherGCR.Level(2);
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2 V-cycle preconditioner.
-  //////////////////////////////////////////////////////////////////////
-  L1to2MG L1to2Precon(AggregatesL2,
-		      LinOpCoarse,
-		      simpleC,            // no pre-smoother (matches fine-grid setup)
-		      CoarseSmootherGCR,  // post-smoother: 12 GCR steps
-		      LinOpCC,
-		      L3PGCR,
-		      simpleCC,
-		      std::string("LinOpC"));
-
-  //////////////////////////////////////////////////////////////////////
-  // Standalone Level 1 two-level solve test.
-  // Compare against plain PGCR baseline above.
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
-  L2MGsolver.Level(2);
-  c_res = Zero();
-  L2MGsolver(c_src,c_res);
-
-  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-
-  //////////////////////////////////////////////////////////////////////
-  // Full three-level outer solve
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.00,1,ShiftedPVdagM,simple_fine,16,16);
-  SmootherGCR.Level(1);
-
-  f_src = one;
-
-  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
-  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
-  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
-			      PVdagM,
-			      simple_fine,
-			      SmootherGCR,
-			      LinOpCoarse,
-			      L2MGsolver,
-			      simpleC,
-			      std::string("PVdagM"));
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
-  L1PGCR.Level(1);
-
-  f_res = Zero();
-  L1PGCR(f_src,f_res);
-
-  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls   = 24;
-  RealD     M5   = 1.8;
-  RealD     b    = 1.5;
-  RealD     c    = 0.5;
-  RealD     mass = 0.00078;
-  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
-
-  const int nbasis = 60;
-
-  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
-  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
-
-  std::vector<int> lat_size {48, 48, 48, 96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
-  Coordinate clatt = lat_size;
-  for (int d = 0; d < 4; d++) clatt[d] /= 2;
-  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
-
-  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
-  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
-  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
-  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
-  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
-  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
-  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
-  Coordinate clatt2 = clatt;
-  clatt2[0] /= 2;
-  clatt2[1] /= 2;
-  clatt2[2] /= 3;
-  clatt2[3] /= 3;
-  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
-
-  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  RealD b_ = 1.5;
-  RealD c_ = 0.5;
-
-  RealD madj = 1.0;
-  if ( getenv("MADJ") ) madj=atof(getenv("MADJ"));
-  std::cout << "PV mass set to "<<madj<<std::endl;
-
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,madj, M5,b_,c_);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
-  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(0.00,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
-  // Set SUBSPACE_FILE to override the default path.
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                              + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-
-  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
-  uint64_t file_exists = 0;
-  if ( UGrid->IsBoss() ) {
-    std::ifstream f(subspace_file);
-    file_exists = f.good() ? 1 : 0;
-  }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb = 0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-
-  if ( file_exists ) {
-    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-    // Re-orthogonalise after loading to ensure block-GS condition holds.
-    //    AggregatesGCR.Orthogonalise();
-    std::cout << GridLogMessage << "Subspace loaded and re-orthogonalised." << std::endl;
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
-    PVdagM.getApplications();
-    PVdagM.clear();
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
-  }
-
-  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
-    FGrid,
-    Coarse5d,
-    CoarseCoarse5d,
-    geom,
-    PVdagM,
-    ShiftedPVdagM,
-    AggregatesGCR
-  );
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_4level.cc

@@ -1,798 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_3level.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-
-template <class T> void readFile(T& out, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    SR.readScidacFieldRecord(out, record);
-    SR.close();
-  #endif
-}
-
-template <class T> void writeFile(T& in, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(in.Grid()->IsBoss());
-    SW.open(fname);
-    SW.writeScidacFieldRecord(in, record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-    SW.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SW.writeScidacFieldRecord(subspace[k], record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SR.readScidacFieldRecord(subspace[k], record);
-    SR.close();
-  #endif
-}
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  int nApp;
-  int nAppDag;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    nApp++;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-    nAppDag++;
-  }
-  void clear() { nApp = 0; nAppDag = 0; }
-  void getApplications() {
-    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
-    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
-    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class MdagPVLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
-// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
-template<class Field>
-class LuscherGuesser : public LinearFunction<Field> {
-  const std::vector<Field> &psi;
-  Eigen::MatrixXcd          C_inv;
-public:
-  using LinearFunction<Field>::operator();
-  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
-    : psi(psi_), C_inv(Cinv_) {}
-  virtual void operator()(const Field &src, Field &guess) {
-    int N = psi.size();
-    Eigen::VectorXcd b(N);
-    for (int t = 0; t < N; t++)
-      b(t) = TensorRemove(innerProduct(psi[t], src));
-    Eigen::VectorXcd c = C_inv * b;
-    guess = Zero();
-    for (int s = 0; s < N; s++)
-      guess += ComplexD(c(s)) * psi[s];
-  }
-};
-
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-  CoarseSolver   & _CoarseGuesser;
-
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_,
-		   CoarseSolver &CoarseGuesser_)
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      _CoarseGuesser(CoarseGuesser_),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out)
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    double t;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
-
-    t=-usecond();
-    _Aggregates.ProjectToSubspace(Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _CoarseGuesser(Csrc,Csol);
-    _CoarseSolve(Csrc,Csol);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _Aggregates.PromoteFromSubspace(Csol,vec1);
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
-
-    t=-usecond();
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add(out,out,vec2);
-  }
-};
-
-// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
-// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
-// at the fine level.
-template<class Field>
-class ShiftedLinearOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op;
-  RealD shift;
-public:
-  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
-  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
-  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
-  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
-  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
-  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
-  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
-  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
-void runMG(
-  GridCartesian *FGrid,
-  GridCartesian *Coarse5d,
-  GridCartesian *CoarseCoarse5d,
-  GridCartesian *CoarseCoarseCoarse5d,
-  NextToNearestStencilGeometry5D geom,
-  PVdagM_t &PVdagM,
-  ShiftedPVdagM_t &ShiftedPVdagM,
-  Subspace &AggregatesPD
-) {
-  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
-  assert((int)subspace.size() == NB);
-  const int nbasis = NB;
-  const int cb = 0;
-
-  CoarseVector c_src(Coarse5d);
-  CoarseVector c_res(Coarse5d);
-  Complex one(1.0);
-
-  LatticeFermionD f_src(FGrid);
-  LatticeFermionD f_res(FGrid);
-
-  TrivialPrecon<CoarseVector>      simpleC;
-  TrivialPrecon<LatticeFermionD>   simple_fine;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 0→1: coarsen PVdagM, build LinOpCoarse
-  //////////////////////////////////////////////////////////////////////
-  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
-  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
-
-  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-
-  //////////////////////////////////////////////////////////////////////
-  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
-  L2PGCR_baseline.Level(2);
-  L2PGCR_baseline.Name("Cbaseline");
-  c_src = one;
-  c_res = Zero();
-  L2PGCR_baseline(c_src,c_res);
-
-  //////////////////////////////////////////////////////////////////////
-  // psi_coarse: coarse projections of pre-GS fine null vectors.
-  // These are the Level 1 near-null vectors, promoted from Level 0.
-  // Used as the aggregation basis for Level 1→2 coarsening.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
-
-  //////////////////////////////////////////////////////////////////////
-  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
-  //////////////////////////////////////////////////////////////////////
-  {
-    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    LatticeFermion ftmp(FGrid);
-    for (int j = 0; j < nbasis; j++) {
-      PVdagM.Op(subspace[j], ftmp);
-      for (int i = 0; i < nbasis; i++)
-        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
-    }
-    RealD normW = W.norm();
-    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
-
-    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    CoarseVector Ac(Coarse5d);
-    for (int l = 0; l < nbasis; l++) {
-      LinOpCoarse.Op(psi_coarse[l], Ac);
-      for (int k = 0; k < nbasis; k++)
-        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
-    }
-    RealD normC      = C.norm();
-    RealD normCmCdag = (C - C.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
-    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
-    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2: set up aggregation using psi_coarse as subspace.
-  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
-  // geometry 3×6×4×4 where z-local at Level 1 is 6).
-  // psi_coarse are assigned directly; CoarsenOperator performs
-  // block-GS orthogonalisation before building LinOpCoarseCoarse.
-  //////////////////////////////////////////////////////////////////////
-  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
-  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
-  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
-  typedef iScalar<vTComplex>                                              vTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
-  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
-  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
-  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
-
-  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.subspace[k] = psi_coarse[k];
-
-  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
-  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
-  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
-
-  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
-
-  TrivialPrecon<CoarseCoarseVector> simpleCC;
-
-  //////////////////////////////////////////////////////////////////////
-  // Lüscher deflation guesser for L3PGCR.
-  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
-  //         CoarseCoarseVector space — these cover the zero-momentum
-  //         component of the near-null space of LinOpCC.
-  // Step 2: breed Nextra additional null vectors directly on LinOpCC
-  //         using GCR with random sources — these pick up near-null
-  //         modes at all spatial frequencies not spanned by step 1.
-  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
-  //         full augmented basis and invert directly via Eigen LU.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
-
-  {
-    int Nextra = nbasis;  // breed as many extra as we have promoted ones
-    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
-    GridParallelRNG RNG_CC(CoarseCoarse5d);
-    RNG_CC.SeedFixedIntegers({11,13,17,19});
-    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
-    CoarseCoarseVector tmp(CoarseCoarse5d);
-    for (int k = 0; k < Nextra; k++) {
-      CoarseCoarseVector src(CoarseCoarse5d);
-      gaussian(RNG_CC, src);
-      tmp = Zero();
-      nullGCR(src, tmp);
-      psi_cc.push_back(tmp);
-    }
-    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
-              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
-  }
-
-  const int Naug = psi_cc.size();
-  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
-  {
-    CoarseCoarseVector Acc(CoarseCoarse5d);
-    for (int l = 0; l < Naug; l++) {
-      LinOpCC.Op(psi_cc[l], Acc);
-      for (int k = 0; k < Naug; k++)
-        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
-    }
-  }
-  {
-    RealD normCcc      = Ccc.norm();
-    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
-    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
-  }
-  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
-  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 2→3: coarsen LinOpCC using the RAW promoted psi_cc as aggregation
-  // to build the Level 4 (coarse-coarse-coarse) operator.
-  //   psi_cc[0..nbasis-1] are the coarse-coarse near-null vectors, projected
-  //   from the RAW psi_coarse (themselves projected from the RAW fine null
-  //   vectors) -- the pre-block-GS chain the whole construction depends on.
-  //   CoarsenOperator block-GS orthogonalises AggregatesL3.subspace IN PLACE,
-  //   so assign COPIES of psi_cc and keep psi_cc itself raw.
-  //
-  // Tensor depth deepens once more: innerProduct(CoarseCoarseSiteObj,...) returns
-  // iScalar<vTTComplex>, so CComplex for the L2→L3 level is iScalar<iScalar<vTComplex>>.
-  //////////////////////////////////////////////////////////////////////
-  typedef typename CoarseCoarseVector::vector_object                     CoarseCoarseSiteObj;
-  typedef iScalar<vTTComplex>                                            vTTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseCoarseSiteObj,vTTTComplex,NB>     LittleDiracOperatorL3;
-  typedef typename LittleDiracOperatorL3::CoarseVector                   CoarseCoarseCoarseVector;
-  typedef Aggregation<CoarseCoarseSiteObj,vTTTComplex,NB>               SubspaceL3;
-  typedef MGPreconditioner<CoarseCoarseSiteObj,vTTTComplex,NB>          L2to3MG;
-
-  SubspaceL3 AggregatesL3(CoarseCoarseCoarse5d, CoarseCoarse5d, cb);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL3.subspace[k] = psi_cc[k];          // raw promoted; COPY, keeps psi_cc raw
-
-  NextToNearestStencilGeometry5D geom3(CoarseCoarseCoarse5d);
-  LittleDiracOperatorL3 LittleDiracOpL3(geom3, CoarseCoarse5d, CoarseCoarseCoarse5d);
-  LittleDiracOpL3.CoarsenOperator(LinOpCC, AggregatesL3);   // block-GS's AggregatesL3.subspace in place
-
-  NonHermitianLinearOperator<LittleDiracOperatorL3,CoarseCoarseCoarseVector> LinOpCCC(LittleDiracOpL3);
-  TrivialPrecon<CoarseCoarseCoarseVector> simpleCCC;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 4 bottom solve: GCR on a SHIFTED LinOpCCC.  This is the one level
-  // with no IRS shift, and it is the most non-normal (coarsest) operator, so
-  // the bare bottom GCR wanders in a field of values that wraps the origin and
-  // its iteration count blows out (observed 5..54 iters to hit 0.2).  Solving
-  // (A_ccc + l4_shift) instead slides the FoV off the origin; the correction is
-  // only ever a loose 0.2 approximation anyway, so the detuning is free.
-  // l4_shift defaults to 0.0 => bare LinOpCCC, baseline unchanged until opted in.
-  //////////////////////////////////////////////////////////////////////
-  RealD l4_shift = 0.0;
-  if(getenv("l4_shift")) l4_shift = atof(getenv("l4_shift"));
-  std::cout << GridLogMessage << "PARAM l4_shift = " << l4_shift << std::endl;
-
-  ShiftedLinearOperator<CoarseCoarseCoarseVector> ShiftedLinOpCCC(l4_shift, LinOpCCC);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector> L4PGCR(1.0e-1,200,ShiftedLinOpCCC,simpleCCC,16,16);
-  L4PGCR.Level(4);
-  L4PGCR.Name("CCCouter");
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 2→3 V-cycle: depth-2 SHIFTED smoother on LinOpCC + L4 bottom.
-  // The shift slides the coarse-coarse field of values off the origin so a
-  // 2-step smoother has something to bite on a non-normal operator (IRS idea).
-  //////////////////////////////////////////////////////////////////////
-  RealD cc_smoother_shift = 0.01;
-  int   cc_smoother_nstep = 2;
-  if(getenv("cc_smoother_shift")) cc_smoother_shift = atof(getenv("cc_smoother_shift"));
-  if(getenv("cc_smoother_nstep")) cc_smoother_nstep = atoi(getenv("cc_smoother_nstep"));
-
-  ShiftedLinearOperator<CoarseCoarseVector> ShiftedLinOpCC(cc_smoother_shift, LinOpCC);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-    CoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCC,simpleCC,cc_smoother_nstep,cc_smoother_nstep);
-  CoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
-  CoarseCoarseSmootherGCR.Level(3);
-  CoarseCoarseSmootherGCR.Name("CCsmoother");
-
-  L2to3MG L2to3Precon(AggregatesL3,
-                      LinOpCC,
-                      simpleCC,                  // no pre-smoother
-                      CoarseCoarseSmootherGCR,   // post-smoother: depth-2 shifted GCR
-                      LinOpCCC,
-                      L4PGCR,
-                      simpleCCC);                // trivial guesser at the bottom
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 3 (coarse-coarse) solve: GCR preconditioned by the L2→L3 V-cycle.
-  // Replaces the plain L3PGCR of the 3-level build -- the coarse-coarse level
-  // is now smoothed shallowly and recursed rather than solved deeply.
-  //////////////////////////////////////////////////////////////////////
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3MGsolver(1.0e-1,200,LinOpCC,L2to3Precon,16,16);
-  L3MGsolver.Level(3);
-  L3MGsolver.Name("CCouter");
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarse-level GCR smoother for Level 1→2 V-cycle.
-  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
-  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
-  //////////////////////////////////////////////////////////////////////
-  RealD coarse_smoother_shift = 0.01;
-  int   coarse_smoother_nstep = 2;   // depth-2 smoother on the coarse level
-  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
-  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
-     
-  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
-  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
-  CoarseSmootherGCR.Level(2);
-  CoarseSmootherGCR.Name("Csmoother");
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2 V-cycle preconditioner.
-  //////////////////////////////////////////////////////////////////////
-  L1to2MG L1to2Precon(AggregatesL2,
-                       LinOpCoarse,
-                       simpleC,            // no pre-smoother (matches fine-grid setup)
-                       CoarseSmootherGCR,  // post-smoother: depth-2 shifted GCR
-                       LinOpCC,
-                       L3MGsolver,         // coarse-coarse solve is now the L2→L3 V-cycle
-                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
-
-  //////////////////////////////////////////////////////////////////////
-  // Standalone Level 1 two-level solve test.
-  // Compare against plain PGCR baseline above.
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
-  L2MGsolver.Level(2);
-  L2MGsolver.Name("Couter");
-  c_res = Zero();
-  L2MGsolver(c_src,c_res);
-
-  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-
-  //////////////////////////////////////////////////////////////////////
-  // Full three-level outer solve
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
-  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
-  SmootherGCR.Level(1);
-  SmootherGCR.Name("Fsmoother");
-
-  f_src = one;
-
-  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
-  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
-  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
-                               PVdagM,
-                               simple_fine,
-                               SmootherGCR,
-                               LinOpCoarse,
-                               L2MGsolver,
-                               simpleC);
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
-  L1PGCR.Level(1);
-  L1PGCR.Name("Fouter");
-
-  f_res = Zero();
-  L1PGCR(f_src,f_res);
-
-  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls   = 24;
-  RealD     M5   = 1.8;
-  RealD     b    = 1.5;
-  RealD     c    = 0.5;
-  RealD     mass = 0.00078;
-  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
-
-  const int nbasis = 60;
-
-  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
-  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
-
-  std::vector<int> lat_size {48, 48, 48, 96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
-  Coordinate clatt = lat_size;
-  for (int d = 0; d < 4; d++) clatt[d] /= 2;
-  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
-
-  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
-  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
-  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
-  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
-  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
-  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
-  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
-  Coordinate clatt2 = clatt;
-  clatt2[0] /= 2;
-  clatt2[1] /= 2;
-  clatt2[2] /= 3;
-  clatt2[3] /= 3;
-  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
-
-  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
-
-  // Level 3 coarse-coarse-coarse grid: block clatt2 = {12,12,8,16} -> {6,12,8,8}.
-  // GEOMETRY (mpi 3.6.4.4, Nsimd=4 => SIMD layout {1,1,2,2}, factor 2 on z and t):
-  //   every grid needs z-local and t-local EVEN.  clatt2-local is {4,2,2,4}, so
-  //   z-local=2 is already at its minimum even value and CANNOT be blocked (2->1
-  //   is odd and trips the SIMD assertion); y-local=2 would go to 1 (degenerate).
-  //   Only x and t have room, so block {2,1,1,2}: clatt3 {6,12,8,8}, L4-local
-  //   {2,2,2,2} -- all dims even and >=2.  z stays unblocked by construction.
-  Coordinate clatt3 = clatt2;
-  clatt3[0] /= 2;   // x: 12 -> 6   (x-local 4 -> 2)
-  // clatt3[1] (y) unblocked: y-local 2 -> blocking gives 1 (degenerate)
-  // clatt3[2] (z) unblocked: z-local 2 is SIMD-pinned even, cannot halve
-  clatt3[3] /= 2;   // t: 16 -> 8   (t-local 4 -> 2)
-  std::cout << GridLogMessage << "Level 3 coarse-coarse-coarse lattice: " << clatt3 << std::endl;
-
-  GridCartesian *CoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt3, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  RealD b_ = 1.5;
-  RealD c_ = 0.5;
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
-  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
-  // Set SUBSPACE_FILE to override the default path.
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                              + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-
-  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
-  uint64_t file_exists = 0;
-  if ( UGrid->IsBoss() ) {
-    std::ifstream f(subspace_file);
-    file_exists = f.good() ? 1 : 0;
-  }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb = 0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-
-  if ( file_exists ) {
-    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-    // Insurance: GLOBAL (whole-lattice) orthonormalise, in case the cached file
-    // predates the GlobalOrthonormalise() that CreateSubspaceGCR now applies
-    // (Aggregates.h:196).  It is span-preserving and makes the vectors globally
-    // orthonormal -- it is NOT the block Orthogonalise() below, so it does NOT
-    // cause the psi_coarse->e_k trap.  The RAW subspace copy in runMG happens
-    // AFTER this call, so the raw-null (pre-block-GS) discipline is preserved.
-    AggregatesGCR.GlobalOrthonormalise();
-    // DO NOT block-orthogonalise here: runMG copies subspace[] as the RAW
-    // (pre-block-GS) basis and CoarsenOperator block-GS's it in place later.
-    // Orthogonalising now defeats the raw-null discipline (psi_coarse -> e_k)
-    // and poisons L2/L3/L4.  See project_block_orthogonalise_leak.
-    // AggregatesGCR.Orthogonalise();
-    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
-    PVdagM.getApplications();
-    PVdagM.clear();
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
-  }
-
-  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
-    FGrid,
-    Coarse5d,
-    CoarseCoarse5d,
-    CoarseCoarseCoarse5d,
-    geom,
-    PVdagM,
-    ShiftedPVdagM,
-    AggregatesGCR
-  );
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_5level.cc

@@ -1,954 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_5level.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-
-template <class T> void readFile(T& out, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    SR.readScidacFieldRecord(out, record);
-    SR.close();
-  #endif
-}
-
-template <class T> void writeFile(T& in, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(in.Grid()->IsBoss());
-    SW.open(fname);
-    SW.writeScidacFieldRecord(in, record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-    SW.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SW.writeScidacFieldRecord(subspace[k], record);
-    SW.close();
-  #endif
-}
-
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-  #ifdef HAVE_LIME
-    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    for (int k = 0; k < (int)subspace.size(); k++)
-      SR.readScidacFieldRecord(subspace[k], record);
-    SR.close();
-  #endif
-}
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  int nApp;
-  int nAppDag;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    nApp++;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-    nAppDag++;
-  }
-  void clear() { nApp = 0; nAppDag = 0; }
-  void getApplications() {
-    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
-    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
-    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class MdagPVLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
-// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
-template<class Field>
-class LuscherGuesser : public LinearFunction<Field> {
-  const std::vector<Field> &psi;
-  Eigen::MatrixXcd          C_inv;
-public:
-  using LinearFunction<Field>::operator();
-  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
-    : psi(psi_), C_inv(Cinv_) {}
-  virtual void operator()(const Field &src, Field &guess) {
-    int N = psi.size();
-    Eigen::VectorXcd b(N);
-    for (int t = 0; t < N; t++)
-      b(t) = TensorRemove(innerProduct(psi[t], src));
-    Eigen::VectorXcd c = C_inv * b;
-    guess = Zero();
-    for (int s = 0; s < N; s++)
-      guess += ComplexD(c(s)) * psi[s];
-  }
-};
-
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-  CoarseSolver   & _CoarseGuesser;
-
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_,
-		   CoarseSolver &CoarseGuesser_)
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      _CoarseGuesser(CoarseGuesser_),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out)
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    double t;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
-
-    t=-usecond();
-    _Aggregates.ProjectToSubspace(Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _CoarseGuesser(Csrc,Csol);
-    _CoarseSolve(Csrc,Csol);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    t=-usecond();
-    _Aggregates.PromoteFromSubspace(Csol,vec1);
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
-
-    t=-usecond();
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add(out,out,vec2);
-  }
-};
-
-// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
-// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
-// at the fine level.
-template<class Field>
-class ShiftedLinearOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op;
-  RealD shift;
-public:
-  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
-  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
-  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
-  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
-  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
-  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
-  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
-  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
-void runMG(
-  GridCartesian *FGrid,
-  GridCartesian *Coarse5d,
-  GridCartesian *CoarseCoarse5d,
-  GridCartesian *CoarseCoarseCoarse5d,
-  GridCartesian *CoarseCoarseCoarseCoarse5d,
-  NextToNearestStencilGeometry5D geom,
-  PVdagM_t &PVdagM,
-  ShiftedPVdagM_t &ShiftedPVdagM,
-  Subspace &AggregatesPD
-) {
-  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
-  assert((int)subspace.size() == NB);
-  const int nbasis = NB;
-  const int cb = 0;
-
-  CoarseVector c_src(Coarse5d);
-  CoarseVector c_res(Coarse5d);
-  Complex one(1.0);
-
-  LatticeFermionD f_src(FGrid);
-  LatticeFermionD f_res(FGrid);
-
-  TrivialPrecon<CoarseVector>      simpleC;
-  TrivialPrecon<LatticeFermionD>   simple_fine;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 0→1: coarsen PVdagM, build LinOpCoarse
-  //////////////////////////////////////////////////////////////////////
-  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
-  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
-
-  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-
-  //////////////////////////////////////////////////////////////////////
-  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
-  L2PGCR_baseline.Level(2);
-  L2PGCR_baseline.Name("Cbaseline");
-  c_src = one;
-  c_res = Zero();
-  L2PGCR_baseline(c_src,c_res);
-
-  //////////////////////////////////////////////////////////////////////
-  // psi_coarse: coarse projections of pre-GS fine null vectors.
-  // These are the Level 1 near-null vectors, promoted from Level 0.
-  // Used as the aggregation basis for Level 1→2 coarsening.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
-
-  //////////////////////////////////////////////////////////////////////
-  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
-  //////////////////////////////////////////////////////////////////////
-  {
-    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    LatticeFermion ftmp(FGrid);
-    for (int j = 0; j < nbasis; j++) {
-      PVdagM.Op(subspace[j], ftmp);
-      for (int i = 0; i < nbasis; i++)
-        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
-    }
-    RealD normW = W.norm();
-    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
-
-    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
-    CoarseVector Ac(Coarse5d);
-    for (int l = 0; l < nbasis; l++) {
-      LinOpCoarse.Op(psi_coarse[l], Ac);
-      for (int k = 0; k < nbasis; k++)
-        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
-    }
-    RealD normC      = C.norm();
-    RealD normCmCdag = (C - C.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
-    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
-    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2: set up aggregation using psi_coarse as subspace.
-  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
-  // geometry 3×6×4×4 where z-local at Level 1 is 6).
-  // psi_coarse are assigned directly; CoarsenOperator performs
-  // block-GS orthogonalisation before building LinOpCoarseCoarse.
-  //////////////////////////////////////////////////////////////////////
-  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
-  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
-  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
-  typedef iScalar<vTComplex>                                              vTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
-  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
-  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
-  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
-
-  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.subspace[k] = psi_coarse[k];
-
-  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
-  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
-  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
-
-  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
-
-  TrivialPrecon<CoarseCoarseVector> simpleCC;
-
-  //////////////////////////////////////////////////////////////////////
-  // Lüscher deflation guesser for L3PGCR.
-  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
-  //         CoarseCoarseVector space — these cover the zero-momentum
-  //         component of the near-null space of LinOpCC.
-  // Step 2: breed Nextra additional null vectors directly on LinOpCC
-  //         using GCR with random sources — these pick up near-null
-  //         modes at all spatial frequencies not spanned by step 1.
-  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
-  //         full augmented basis and invert directly via Eigen LU.
-  //////////////////////////////////////////////////////////////////////
-  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
-
-  {
-    int Nextra = nbasis;  // breed as many extra as we have promoted ones
-    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
-    GridParallelRNG RNG_CC(CoarseCoarse5d);
-    RNG_CC.SeedFixedIntegers({11,13,17,19});
-    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
-    CoarseCoarseVector tmp(CoarseCoarse5d);
-    for (int k = 0; k < Nextra; k++) {
-      CoarseCoarseVector src(CoarseCoarse5d);
-      gaussian(RNG_CC, src);
-      tmp = Zero();
-      nullGCR(src, tmp);
-      psi_cc.push_back(tmp);
-    }
-    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
-              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
-  }
-
-  const int Naug = psi_cc.size();
-  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
-  {
-    CoarseCoarseVector Acc(CoarseCoarse5d);
-    for (int l = 0; l < Naug; l++) {
-      LinOpCC.Op(psi_cc[l], Acc);
-      for (int k = 0; k < Naug; k++)
-        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
-    }
-  }
-  {
-    RealD normCcc      = Ccc.norm();
-    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
-    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
-    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
-  }
-  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
-  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 2→3: coarsen LinOpCC using the RAW promoted psi_cc as aggregation
-  // to build the Level 4 (coarse-coarse-coarse) operator.
-  //   psi_cc[0..nbasis-1] are the coarse-coarse near-null vectors, projected
-  //   from the RAW psi_coarse (themselves projected from the RAW fine null
-  //   vectors) -- the pre-block-GS chain the whole construction depends on.
-  //   CoarsenOperator block-GS orthogonalises AggregatesL3.subspace IN PLACE,
-  //   so assign COPIES of psi_cc and keep psi_cc itself raw.
-  //
-  // Tensor depth deepens once more: innerProduct(CoarseCoarseSiteObj,...) returns
-  // iScalar<vTTComplex>, so CComplex for the L2→L3 level is iScalar<iScalar<vTComplex>>.
-  //////////////////////////////////////////////////////////////////////
-  typedef typename CoarseCoarseVector::vector_object                     CoarseCoarseSiteObj;
-  typedef iScalar<vTTComplex>                                            vTTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseCoarseSiteObj,vTTTComplex,NB>     LittleDiracOperatorL3;
-  typedef typename LittleDiracOperatorL3::CoarseVector                   CoarseCoarseCoarseVector;
-  typedef Aggregation<CoarseCoarseSiteObj,vTTTComplex,NB>               SubspaceL3;
-  typedef MGPreconditioner<CoarseCoarseSiteObj,vTTTComplex,NB>          L2to3MG;
-
-  SubspaceL3 AggregatesL3(CoarseCoarseCoarse5d, CoarseCoarse5d, cb);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL3.subspace[k] = psi_cc[k];          // raw promoted; COPY, keeps psi_cc raw
-
-  NextToNearestStencilGeometry5D geom3(CoarseCoarseCoarse5d);
-  LittleDiracOperatorL3 LittleDiracOpL3(geom3, CoarseCoarse5d, CoarseCoarseCoarse5d);
-  LittleDiracOpL3.CoarsenOperator(LinOpCC, AggregatesL3);   // block-GS's AggregatesL3.subspace in place
-
-  NonHermitianLinearOperator<LittleDiracOperatorL3,CoarseCoarseCoarseVector> LinOpCCC(LittleDiracOpL3);
-  TrivialPrecon<CoarseCoarseCoarseVector> simpleCCC;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 3→4: coarsen LinOpCCC to build the Level 5 operator, using a
-  // TRUNCATED basis of only the first NB5 (< nbasis) raw promoted null vectors.
-  //   psi_ccc[k] = raw psi_cc projected through the (block-GS'd) L3 aggregation
-  //   -- the pre-block-GS chain continued one level deeper.  We keep only the
-  //   leading NB5: after the global orthogonalisation of the original fine null
-  //   vectors the early indices retain the most-null content (shared low-mode
-  //   components are peeled in first), so the leading NB5 are the crudely-most-
-  //   null slice.  This is the cheap "first 30" truncation test; a principled
-  //   sigma-ordered rotation of psi_ccc would replace the slice, not the idea.
-  //   NB: a positive result is conservative (sigma-ordering can only help); a
-  //   negative one is inconclusive until the sigma-ordered NB5 is tried.
-  //
-  // Tensor depth deepens once more: CComplex for the L3→L4 level is
-  // iScalar<vTTTComplex>.  NB5 (the coarse dimension) is independent of the
-  // depth -- it just makes the coarsest site vector NB5-dimensional.
-  //////////////////////////////////////////////////////////////////////
-  const int NB5 = 30;   // compile-time: changing it re-instantiates the L4/L5 tensors
-  std::cout << GridLogMessage << "PARAM NB5 (truncated coarsest basis) = " << NB5 << std::endl;
-  assert(NB5 <= nbasis);
-
-  std::vector<CoarseCoarseCoarseVector> psi_ccc(nbasis, CoarseCoarseCoarse5d);
-  for (int k = 0; k < nbasis; k++)
-    AggregatesL3.ProjectToSubspace(psi_ccc[k], psi_cc[k]);   // raw psi_cc -> L4 null vectors
-
-  //////////////////////////////////////////////////////////////////////
-  // Optional sigma-ordering of psi_ccc (SVD_REORDER set): replace the crude
-  // first-NB5 slice with the NB5 genuinely-most-null directions of span(psi_ccc)
-  // under LinOpCCC.  For a NON-NORMAL operator the nullness measure is the
-  // singular value of A restricted to the span -- eig of Q†A†AQ -- NOT the
-  // numerical range Q†AQ (which non-normality contaminates).  Robust route:
-  // whiten by the Gram (drop near-dependent directions), Hermitian-eig the
-  // whitened A†A, rotate.  The printed singular spectrum IS the SVD study: where
-  // it falls off tells you the natural NB5, and the same numbers illuminate why
-  // the earlier singular-subspace deflation re-entered.  Safe here because we
-  // ORDER vectors that then feed a Galerkin projection, not REMOVE a subspace.
-  // Default (unset) leaves psi_ccc in raw order == the "first 30" test.
-  //////////////////////////////////////////////////////////////////////
-  if ( getenv("SVD_REORDER") ) {
-    std::cout << GridLogMessage << "SVD_REORDER: sigma-ordering psi_ccc under LinOpCCC" << std::endl;
-
-    Eigen::MatrixXcd G(nbasis,nbasis);                    // Gram  = Psi^dag Psi
-    for (int i=0;i<nbasis;i++)
-    for (int j=0;j<nbasis;j++)
-      G(i,j) = TensorRemove(innerProduct(psi_ccc[i],psi_ccc[j]));
-
-    std::vector<CoarseCoarseCoarseVector> Apsi(nbasis, CoarseCoarseCoarse5d);
-    for (int j=0;j<nbasis;j++) LinOpCCC.Op(psi_ccc[j], Apsi[j]);
-
-    Eigen::MatrixXcd M(nbasis,nbasis);                    // A^dagA = Psi^dag A^dag A Psi
-    for (int i=0;i<nbasis;i++)
-    for (int j=0;j<nbasis;j++)
-      M(i,j) = TensorRemove(innerProduct(Apsi[i],Apsi[j]));
-
-    // Whiten by the Gram: G = Ug diag(g) Ug^dag; keep g > tol*max; T = Ug diag(1/sqrt g).
-    // Q = Psi T is then orthonormal (Q^dag Q = T^dag G T = I).
-    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> esG(G);
-    Eigen::VectorXd g = esG.eigenvalues();               // ascending, real
-    RealD gmax = g(nbasis-1);
-    RealD gtol = 1.0e-9 * gmax;
-    int keep = 0; for (int i=0;i<nbasis;i++) if (g(i) > gtol) keep++;
-    std::cout << GridLogMessage << "  Gram spectrum: min=" << g(0) << " max=" << gmax
-              << " cond=" << gmax/std::max(g(0),1.0e-300) << " keep=" << keep << "/" << nbasis << std::endl;
-    assert(keep >= NB5);
-
-    Eigen::MatrixXcd T(nbasis, keep);                    // whitening (largest-g first)
-    { int c=0;
-      for (int i=nbasis-1;i>=0;i--) if (g(i) > gtol) { T.col(c) = esG.eigenvectors().col(i)/std::sqrt(g(i)); c++; }
-    }
-
-    Eigen::MatrixXcd Mw = T.adjoint() * M * T;           // whitened A^dagA (keep x keep, Hermitian)
-    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> esM(Mw);
-    Eigen::VectorXd s2 = esM.eigenvalues();              // ascending sigma^2 (most-null first)
-    std::cout << GridLogMessage << "  Singular spectrum sigma_k (most-null first):" << std::endl;
-    for (int k=0;k<keep;k++)
-      std::cout << GridLogMessage << "    sigma[" << k << "] = " << std::sqrt(std::max(s2(k),0.0)) << std::endl;
-
-    Eigen::MatrixXcd R = T * esM.eigenvectors();         // coeffs over Psi, sigma-ordered orthonormal dirs
-    std::vector<CoarseCoarseCoarseVector> phi(keep, CoarseCoarseCoarse5d);
-    for (int k=0;k<keep;k++) {
-      phi[k] = Zero();
-      for (int j=0;j<nbasis;j++)
-        phi[k] = phi[k] + ComplexD(R(j,k)) * psi_ccc[j];
-    }
-    for (int k=0;k<keep;k++) psi_ccc[k] = phi[k];        // psi_ccc[0..NB5-1] now = most-null dirs
-    std::cout << GridLogMessage << "SVD_REORDER: psi_ccc replaced by sigma-ordered directions" << std::endl;
-  }
-
-  typedef typename CoarseCoarseCoarseVector::vector_object              CoarseCoarseCoarseSiteObj;
-  typedef iScalar<vTTTComplex>                                          vTTTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5> LittleDiracOperatorL4;
-  typedef typename LittleDiracOperatorL4::CoarseVector                  CoarseCoarseCoarseCoarseVector;
-  typedef Aggregation<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5>       SubspaceL4;
-  typedef MGPreconditioner<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5>  L3to4MG;
-
-  SubspaceL4 AggregatesL4(CoarseCoarseCoarseCoarse5d, CoarseCoarseCoarse5d, cb);
-  for (int k = 0; k < NB5; k++)
-    AggregatesL4.subspace[k] = psi_ccc[k];         // FIRST NB5 raw promoted vectors (truncation)
-
-  NextToNearestStencilGeometry5D geom4(CoarseCoarseCoarseCoarse5d);
-  LittleDiracOperatorL4 LittleDiracOpL4(geom4, CoarseCoarseCoarse5d, CoarseCoarseCoarseCoarse5d);
-  LittleDiracOpL4.CoarsenOperator(LinOpCCC, AggregatesL4);   // block-GS's AggregatesL4.subspace in place
-
-  NonHermitianLinearOperator<LittleDiracOperatorL4,CoarseCoarseCoarseCoarseVector> LinOpCCCC(LittleDiracOpL4);
-  TrivialPrecon<CoarseCoarseCoarseCoarseVector> simpleCCCC;
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 5 bottom solve: GCR on a SHIFTED LinOpCCCC (the coarsest, most
-  // non-normal operator).  l5_shift slides its field of values off the origin;
-  // defaults to 0.0 (bare LinOpCCCC) until opted in.  This is the level a dense
-  // direct inverse would eventually replace: rank = NB5 * sites(clatt4).
-  //////////////////////////////////////////////////////////////////////
-  RealD l5_shift = 0.0;
-  if(getenv("l5_shift")) l5_shift = atof(getenv("l5_shift"));
-  std::cout << GridLogMessage << "PARAM l5_shift = " << l5_shift << std::endl;
-
-  ShiftedLinearOperator<CoarseCoarseCoarseCoarseVector> ShiftedLinOpCCCC(l5_shift, LinOpCCCC);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseCoarseVector> L5PGCR(1.0e-1,200,ShiftedLinOpCCCC,simpleCCCC,16,16);
-  L5PGCR.Level(5);
-  L5PGCR.Name("CCCCouter");
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 3→4 V-cycle: depth-2 SHIFTED smoother on LinOpCCC + Level 5 bottom.
-  // Level 4 is no longer the bottom -- it is smoothed shallowly and recursed to
-  // Level 5, mirroring how Level 3 recurses to Level 4.
-  //////////////////////////////////////////////////////////////////////
-  RealD ccc_smoother_shift = 0.05;
-  int   ccc_smoother_nstep = 2;
-  if(getenv("ccc_smoother_shift")) ccc_smoother_shift = atof(getenv("ccc_smoother_shift"));
-  if(getenv("ccc_smoother_nstep")) ccc_smoother_nstep = atoi(getenv("ccc_smoother_nstep"));
-
-  ShiftedLinearOperator<CoarseCoarseCoarseVector> ShiftedLinOpCCC(ccc_smoother_shift, LinOpCCC);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector>
-    CoarseCoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCCC,simpleCCC,ccc_smoother_nstep,ccc_smoother_nstep);
-  CoarseCoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
-  CoarseCoarseCoarseSmootherGCR.Level(4);
-  CoarseCoarseCoarseSmootherGCR.Name("CCCsmoother");
-
-  L3to4MG L3to4Precon(AggregatesL4,
-                      LinOpCCC,
-                      simpleCCC,                     // no pre-smoother
-                      CoarseCoarseCoarseSmootherGCR, // post-smoother: depth-2 shifted GCR
-                      LinOpCCCC,
-                      L5PGCR,
-                      simpleCCCC);                   // trivial guesser at the bottom
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 4 (coarse-coarse-coarse) solve: GCR preconditioned by the L3→L4 V-cycle.
-  //////////////////////////////////////////////////////////////////////
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector> L4MGsolver(1.0e-1,200,LinOpCCC,L3to4Precon,16,16);
-  L4MGsolver.Level(4);
-  L4MGsolver.Name("CCCouter");
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 2→3 V-cycle: depth-2 SHIFTED smoother on LinOpCC + Level 4 solve.
-  // The shift slides the coarse-coarse field of values off the origin so a
-  // 2-step smoother has something to bite on a non-normal operator (IRS idea).
-  //////////////////////////////////////////////////////////////////////
-  RealD cc_smoother_shift = 0.01;
-  int   cc_smoother_nstep = 2;
-  if(getenv("cc_smoother_shift")) cc_smoother_shift = atof(getenv("cc_smoother_shift"));
-  if(getenv("cc_smoother_nstep")) cc_smoother_nstep = atoi(getenv("cc_smoother_nstep"));
-
-  ShiftedLinearOperator<CoarseCoarseVector> ShiftedLinOpCC(cc_smoother_shift, LinOpCC);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-    CoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCC,simpleCC,cc_smoother_nstep,cc_smoother_nstep);
-  CoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
-  CoarseCoarseSmootherGCR.Level(3);
-  CoarseCoarseSmootherGCR.Name("CCsmoother");
-
-  L2to3MG L2to3Precon(AggregatesL3,
-                      LinOpCC,
-                      simpleCC,                  // no pre-smoother
-                      CoarseCoarseSmootherGCR,   // post-smoother: depth-2 shifted GCR
-                      LinOpCCC,
-                      L4MGsolver,                // coarse solve is now the L3→L4 V-cycle
-                      simpleCCC);                // trivial guesser
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 3 (coarse-coarse) solve: GCR preconditioned by the L2→L3 V-cycle.
-  // Replaces the plain L3PGCR of the 3-level build -- the coarse-coarse level
-  // is now smoothed shallowly and recursed rather than solved deeply.
-  //////////////////////////////////////////////////////////////////////
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3MGsolver(1.0e-1,200,LinOpCC,L2to3Precon,16,16);
-  L3MGsolver.Level(3);
-  L3MGsolver.Name("CCouter");
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarse-level GCR smoother for Level 1→2 V-cycle.
-  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
-  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
-  //////////////////////////////////////////////////////////////////////
-  RealD coarse_smoother_shift = 0.01;
-  int   coarse_smoother_nstep = 2;   // depth-2 smoother on the coarse level
-  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
-  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
-     
-  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
-  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
-  CoarseSmootherGCR.Level(2);
-  CoarseSmootherGCR.Name("Csmoother");
-
-  //////////////////////////////////////////////////////////////////////
-  // Level 1→2 V-cycle preconditioner.
-  //////////////////////////////////////////////////////////////////////
-  L1to2MG L1to2Precon(AggregatesL2,
-                       LinOpCoarse,
-                       simpleC,            // no pre-smoother (matches fine-grid setup)
-                       CoarseSmootherGCR,  // post-smoother: depth-2 shifted GCR
-                       LinOpCC,
-                       L3MGsolver,         // coarse-coarse solve is now the L2→L3 V-cycle
-                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
-
-  //////////////////////////////////////////////////////////////////////
-  // Standalone Level 1 two-level solve test.
-  // Compare against plain PGCR baseline above.
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
-  L2MGsolver.Level(2);
-  L2MGsolver.Name("Couter");
-  c_res = Zero();
-  L2MGsolver(c_src,c_res);
-
-  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-
-  //////////////////////////////////////////////////////////////////////
-  // Full five-level outer solve
-  //////////////////////////////////////////////////////////////////////
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<" Five-level outer solve"<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
-  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
-  SmootherGCR.Level(1);
-  SmootherGCR.Name("Fsmoother");
-
-  f_src = one;
-
-  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
-  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
-  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
-                               PVdagM,
-                               simple_fine,
-                               SmootherGCR,
-                               LinOpCoarse,
-                               L2MGsolver,
-                               simpleC);
-
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
-  L1PGCR.Level(1);
-  L1PGCR.Name("Fouter");
-
-  f_res = Zero();
-  L1PGCR(f_src,f_res);
-
-  std::cout << GridLogMessage << "Five-level outer solve: PVdagM operator uses:" << std::endl;
-  PVdagM.getApplications();
-  PVdagM.clear();
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls   = 24;
-  RealD     M5   = 1.8;
-  RealD     b    = 1.5;
-  RealD     c    = 0.5;
-  RealD     mass = 0.00078;
-  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
-
-  const int nbasis = 60;
-
-  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
-  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
-
-  std::vector<int> lat_size {48, 48, 48, 96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
-  Coordinate clatt = lat_size;
-  for (int d = 0; d < 4; d++) clatt[d] /= 2;
-  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
-
-  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
-  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
-  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
-  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
-  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
-  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
-  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
-  Coordinate clatt2 = clatt;
-  clatt2[0] /= 2;
-  clatt2[1] /= 2;
-  clatt2[2] /= 3;
-  clatt2[3] /= 3;
-  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
-
-  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
-
-  // Level 3 coarse-coarse-coarse grid: block clatt2 = {12,12,8,16} -> {6,12,8,8}.
-  // GEOMETRY (mpi 3.6.4.4, Nsimd=4 => SIMD layout {1,1,2,2}, factor 2 on z and t):
-  //   every grid needs z-local and t-local EVEN.  clatt2-local is {4,2,2,4}, so
-  //   z-local=2 is already at its minimum even value and CANNOT be blocked (2->1
-  //   is odd and trips the SIMD assertion); y-local=2 would go to 1 (degenerate).
-  //   Only x and t have room, so block {2,1,1,2}: clatt3 {6,12,8,8}, L4-local
-  //   {2,2,2,2} -- all dims even and >=2.  z stays unblocked by construction.
-  Coordinate clatt3 = clatt2;
-  clatt3[0] /= 2;   // x: 12 -> 6   (x-local 4 -> 2)
-  // clatt3[1] (y) unblocked: y-local 2 -> blocking gives 1 (degenerate)
-  // clatt3[2] (z) unblocked: z-local 2 is SIMD-pinned even, cannot halve
-  clatt3[3] /= 2;   // t: 16 -> 8   (t-local 4 -> 2)
-  std::cout << GridLogMessage << "Level 3 coarse-coarse-coarse lattice: " << clatt3 << std::endl;
-
-  GridCartesian *CoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt3, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarse4d);
-
-  // Level 4 coarse^4 grid: block clatt3 = {6,12,8,8} -> {3,6,8,8} via {2,2,1,1}.
-  //   mpi 3.6.4.4 => clatt4-local {1,1,2,2}: z-local=2, t-local=2 stay EVEN (SIMD
-  //   factor 2 pins them), so z,t are unblocked; x,y (SIMD factor 1) halve to
-  //   local 1 -- fully distributed but legal for the halo-depth-1 NextToNearest
-  //   stencil.  1152 sites; with NB5=30 that is the 34,560-rank coarsest operator
-  //   a dense direct inverse would target.
-  Coordinate clatt4 = clatt3;
-  clatt4[0] /= 2;   // x: 6  -> 3   (x-local 2 -> 1)
-  clatt4[1] /= 2;   // y: 12 -> 6   (y-local 2 -> 1)
-  // clatt4[2] (z) unblocked: z-local 2 is SIMD-pinned even
-  // clatt4[3] (t) unblocked: t-local 2 is SIMD-pinned even
-  std::cout << GridLogMessage << "Level 4 coarse^4 lattice: " << clatt4 << std::endl;
-
-  GridCartesian *CoarseCoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt4, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarseCoarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  RealD b_ = 1.5;
-  RealD c_ = 0.5;
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
-  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
-  // Set SUBSPACE_FILE to override the default path.
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                              + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-
-  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
-  uint64_t file_exists = 0;
-  if ( UGrid->IsBoss() ) {
-    std::ifstream f(subspace_file);
-    file_exists = f.good() ? 1 : 0;
-  }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb = 0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-
-  if ( file_exists ) {
-    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-    // Insurance: GLOBAL (whole-lattice) orthonormalise, in case the cached file
-    // predates the GlobalOrthonormalise() that CreateSubspaceGCR now applies
-    // (Aggregates.h:196).  It is span-preserving and makes the vectors globally
-    // orthonormal -- it is NOT the block Orthogonalise() below, so it does NOT
-    // cause the psi_coarse->e_k trap.  It also (re)establishes the weak nullness
-    // gradient (shared most-null components peeled into the early indices) that
-    // the "first NB5" truncation relies on.  Idempotent if the file was already
-    // globally orthonormal.  The RAW subspace copy in runMG happens AFTER this
-    // call, so the raw-null (pre-block-GS) discipline is preserved.
-    AggregatesGCR.GlobalOrthonormalise();
-    // DO NOT block-orthogonalise here: runMG copies subspace[] as the RAW
-    // (pre-block-GS) basis and CoarsenOperator block-GS's it in place later.
-    // Orthogonalising now defeats the raw-null discipline (psi_coarse -> e_k)
-    // and poisons L2/L3/L4.  See project_block_orthogonalise_leak.
-    // AggregatesGCR.Orthogonalise();
-    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
-    PVdagM.getApplications();
-    PVdagM.clear();
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
-  }
-
-  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
-    FGrid,
-    Coarse5d,
-    CoarseCoarse5d,
-    CoarseCoarseCoarse5d,
-    CoarseCoarseCoarseCoarse5d,
-    geom,
-    PVdagM,
-    ShiftedPVdagM,
-    AggregatesGCR
-  );
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_census.cc

@@ -1,717 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_census.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// Spectral census of the coarsened PVdagM operator A_c.
-//
-// Measures the three sets that discriminate between the candidate explanations
-// for slow coarse-grid Krylov convergence:
-//
-//   0. Fine Ritz diagonal of RAW subspace vectors (pre-block-orthog).
-//      NB CoarsenOperator block-orthogonalises subspace[] IN PLACE; all
-//      nulliness/deflation bases must be built from a raw copy.
-//   1. Adjoint correctness check    <y|A x> == <A^dag y|x>      (fail-fast)
-//   2. Raw-vector coarse images vs A_c: RQ (must equal CENSUS 0 by Galerkin),
-//      ||A_c psi_c||/||psi_c||, and representability error
-//   3. sigma_max^2 = lambda_max(A_c^dag A_c)  via power method
-//   4. Low singular values          Chebyshev-filtered IRL on A_c^dag A_c
-//      -> sigma_min census = pseudospectrum of A_c evaluated at the origin
-//   5. Half-plane margin            lambda_min/max of H = (A_c + A_c^dag)/2
-//      -> min Re W(A_c);  positive-real check (Eisenstat-Elman-Schultz bound)
-//
-// Interpretation:
-//   sigma_min ~ min|lambda|, ~nbasis tiny then gap : effectively normal, bipartite
-//   sigma_min ~ min|lambda|, dense low tail        : normal but rank-starved
-//   sigma_min << min|lambda|                       : non-normal near origin
-//   lambda_min(H) < 0                              : half-plane condition violated
-//
-// Requires the dagger code path in GeneralCoarsenedMatrix:
-//   _Adag allocated, PopulateAdag active, _Adag exchanged, hermitian=0.
-//
-// Env vars:
-//   MASS               fermion mass                     (default 0.00078)
-//   SUBSPACE_FILE      subspace cache path
-//   CoarseSolverShift  shift baked into coarsening      (default 0.0: pure Galerkin)
-//   CENSUS_NSTOP       converged low modes wanted       (default 60)
-//   CENSUS_NK          Lanczos Nk                       (default 96)
-//   CENSUS_NM          Lanczos Nm                       (default 192)
-//   CENSUS_TOL         Lanczos residual                 (default 1e-5)
-//   CENSUS_MAXIT       Lanczos max restarts             (default 50)
-//   CHEBY_LO           filter low edge in sigma^2       (default 4.0)
-//   CHEBY_HI           filter high edge; 0 = auto from power method x1.1
-//   CHEBY_ORDER        filter order                     (default 401)
-//                      filter gain at 0 ~ cosh(order*2*sqrt(lo/hi)); with
-//                      hi~2200, lo=4, order=401 => gain ~ 1e14. lo=0.01 at
-//                      order 201 gives gain ~1.4 (stagnation).
-
-#include <Grid/Grid.h>
-#include <Grid/Grid_Eigen_Dense.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-using namespace std;
-using namespace Grid;
-
-RealD mass              = 0.00078;
-RealD CoarseSolverShift = 0.0;
-int   CensusNstop       = 60;
-int   CensusNk          = 96;
-int   CensusNm          = 192;
-RealD CensusTol         = 1.0e-5;
-int   CensusMaxIt       = 50;
-RealD ChebyLo           = 4.0;   // sigma^2 cutoff: amplifies sigma < 2. Filter gain ~ cosh(order*2*sqrt(lo/hi))
-RealD ChebyHi           = 0.0;   // 0 => auto: 1.1 * power-method sigma_max^2
-int   ChebyOrder        = 401;
-RealD CGdeflTol         = 1.0e-8; // CENSUS 6 deflated-CG tolerance
-int   CGdeflMaxIt       = 4000;   // CENSUS 6 deflated-CG max iterations
-int   DeflRank          = 0;      // CENSUS 6 deflation rank; 0 => all available per basis
-
-void ParseEnvironment(void)
-{
-  if(getenv("MASS"))              mass              = atof(getenv("MASS"));
-  if(getenv("CoarseSolverShift")) CoarseSolverShift = atof(getenv("CoarseSolverShift"));
-  if(getenv("CENSUS_NSTOP"))      CensusNstop       = atoi(getenv("CENSUS_NSTOP"));
-  if(getenv("CENSUS_NK"))         CensusNk          = atoi(getenv("CENSUS_NK"));
-  if(getenv("CENSUS_NM"))         CensusNm          = atoi(getenv("CENSUS_NM"));
-  if(getenv("CENSUS_TOL"))        CensusTol         = atof(getenv("CENSUS_TOL"));
-  if(getenv("CENSUS_MAXIT"))      CensusMaxIt       = atoi(getenv("CENSUS_MAXIT"));
-  if(getenv("CHEBY_LO"))          ChebyLo           = atof(getenv("CHEBY_LO"));
-  if(getenv("CHEBY_HI"))          ChebyHi           = atof(getenv("CHEBY_HI"));
-  if(getenv("CHEBY_ORDER"))       ChebyOrder        = atoi(getenv("CHEBY_ORDER"));
-  if(getenv("CGDEFL_TOL"))        CGdeflTol         = atof(getenv("CGDEFL_TOL"));
-  if(getenv("CGDEFL_MAXIT"))      CGdeflMaxIt       = atoi(getenv("CGDEFL_MAXIT"));
-  if(getenv("DEFL_RANK"))         DeflRank          = atoi(getenv("DEFL_RANK"));
-
-  std::cout << GridLogMessage << "PARAM: MASS              " << mass              << std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverShift " << CoarseSolverShift << std::endl;
-  std::cout << GridLogMessage << "PARAM: CENSUS_NSTOP      " << CensusNstop       << std::endl;
-  std::cout << GridLogMessage << "PARAM: CENSUS_NK         " << CensusNk          << std::endl;
-  std::cout << GridLogMessage << "PARAM: CENSUS_NM         " << CensusNm          << std::endl;
-  std::cout << GridLogMessage << "PARAM: CENSUS_TOL        " << CensusTol         << std::endl;
-  std::cout << GridLogMessage << "PARAM: CENSUS_MAXIT      " << CensusMaxIt       << std::endl;
-  std::cout << GridLogMessage << "PARAM: CHEBY_LO          " << ChebyLo           << std::endl;
-  std::cout << GridLogMessage << "PARAM: CHEBY_HI          " << ChebyHi           << std::endl;
-  std::cout << GridLogMessage << "PARAM: CHEBY_ORDER       " << ChebyOrder        << std::endl;
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-#ifdef HAVE_LIME
-  std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-  SW.open(fname);
-  for (int k = 0; k < (int)subspace.size(); k++)
-    SW.writeScidacFieldRecord(subspace[k], record);
-  SW.close();
-#endif
-}
-
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-#ifdef HAVE_LIME
-  std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
-  Grid::emptyUserRecord record;
-  Grid::ScidacReader SR;
-  SR.open(fname);
-  for (int k = 0; k < (int)subspace.size(); k++)
-    SR.readScidacFieldRecord(subspace[k], record);
-  SR.close();
-#endif
-}
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  RealD shift;
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-// H = (A + A^dag)/2 : Hermitian part of the coarse operator.
-// lambda_min(H) = min Re W(A) is the half-plane margin; the EES GCR
-// convergence theorem requires it positive.
-template<class Matrix,class Field>
-class HermitianPartOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-public:
-  HermitianPartOperator(Matrix &Mat): _Mat(Mat) {};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){ HermOp(in,out); }
-  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,out);
-    _Mat.Mdag(in,tmp);
-    out = 0.5*(out + tmp);
-  }
-};
-
-// s*I - Op : power method on this gives s - lambda_min(Op) for Hermitian Op.
-template<class Field>
-class ShiftedNegatedOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op;
-  RealD s;
-public:
-  ShiftedNegatedOperator(RealD _s, LinearOperatorBase<Field> &Op): _Op(Op), s(_s) {};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){ HermOp(in,out); }
-  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    _Op.HermOp(in,out);
-    out = s*in - out;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-  ParseEnvironment();
-
-  const int Ls=24;
-  RealD M5=1.8;
-  RealD b=1.5;
-  RealD c=0.5;
-  const int nbasis = 60;
-
-  std::cout << GridLogMessage << "Census of coarse PVdagM: mass=" << mass << " Ls=" << Ls << " nbasis=" << nbasis << std::endl;
-
-  std::vector<int> lat_size {48, 48, 48, 96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Blocking: default matches Example_pvdagm.cc; override with e.g. BLOCK=2.2.2.2
-  Coordinate clatt = lat_size;
-  Coordinate Block({4,4,6,4});
-  if ( getenv("BLOCK") ) {
-    GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block);
-    GRID_ASSERT(Block.size()==4);
-  }
-  for(int d=0;d<clatt.size();d++){
-    GRID_ASSERT(lat_size[d] % Block[d] == 0);
-    clatt[d] = lat_size[d]/Block[d];
-  }
-  std::cout << GridLogMessage << "Block " << Block << "  coarse lattice " << clatt << std::endl;
-
-  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  GridParallelRNG RNG5(FGrid);    RNG5.SeedFixedIntegers({5,6,7,8});
-  GridParallelRNG RNG4(UGrid);    RNG4.SeedFixedIntegers({1,2,3,4});
-  GridParallelRNG CRNG(Coarse5d); CRNG.SeedFixedIntegers({5,6,7,8});
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(CoarseSolverShift,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  //////////////////////////////////////////////////////////////////////
-  // Subspace: load from cache or generate
-  //////////////////////////////////////////////////////////////////////
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                              + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-
-  uint64_t file_exists = 0;
-  if ( UGrid->IsBoss() ) {
-    std::ifstream f(subspace_file);
-    file_exists = f.good() ? 1 : 0;
-  }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb = 0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-
-  if ( file_exists ) {
-    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Keep the RAW (pre-block-orthogonalisation) near-null vectors.
-  // CoarsenOperator block-orthogonalises subspace[] IN PLACE, after which
-  // subspace[k] is the orthonormal basis phi_k and Project(phi_k) = e_k,
-  // the block-constant unit vector -- NOT a near-null direction.
-  // All nulliness measurements and any deflation basis must use raw[].
-  //////////////////////////////////////////////////////////////////////
-  std::vector<LatticeFermionD> raw(nbasis,FGrid);
-  for(int k=0;k<nbasis;k++) raw[k] = AggregatesGCR.subspace[k];
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 0: fine-grid Ritz diagonal on the loaded/generated raw vectors.
-  // Expect Re <psi|A|psi>/<psi|psi> ~ the nulliness achieved at generation
-  // (~2e-3). O(0.1-10) values mean the cache holds orthogonalised vectors
-  // and must be regenerated.
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 0: fine Ritz diagonal of raw subspace vectors" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  {
-    LatticeFermionD Ap(FGrid);
-    for(int k=0;k<nbasis;k++){
-      PVdagM.Op(raw[k],Ap);
-      RealD    n2psi = norm2(raw[k]);
-      ComplexD rq    = innerProduct(raw[k],Ap)/n2psi;
-      std::cout << GridLogMessage << "CENSUS: raw[" << k << "]  <psi|A|psi>/<psi|psi> = " << rq
-                << "   ||A psi||/||psi|| = " << std::sqrt(norm2(Ap)/n2psi) << std::endl;
-    }
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarsen. hermitian=0 is REQUIRED: enables PopulateAdag so that
-  // Mdag applies A^dag rather than silently aliasing to A.
-  //////////////////////////////////////////////////////////////////////
-  LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d,0);
-  if ( CoarseSolverShift != 0.0 ) {
-    std::cout << GridLogMessage << "Coarsening SHIFTED operator, shift=" << CoarseSolverShift << std::endl;
-    LittleDiracOpPV.CoarsenOperator(ShiftedPVdagM, AggregatesGCR);
-  } else {
-    std::cout << GridLogMessage << "Coarsening pure Galerkin operator (no shift)" << std::endl;
-    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);
-  }
-
-  CoarseVector c_x(Coarse5d);
-  CoarseVector c_y(Coarse5d);
-  CoarseVector c_t1(Coarse5d);
-  CoarseVector c_t2(Coarse5d);
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 1: adjoint correctness (fail fast)
-  //   <y|A x> == <A^dag y|x>  for random x,y
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 1: adjoint correctness of dagger code path" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  random(CRNG,c_x);
-  random(CRNG,c_y);
-  LittleDiracOpPV.M   (c_x,c_t1);       // A x
-  LittleDiracOpPV.Mdag(c_y,c_t2);       // A^dag y
-  ComplexD ip1 = innerProduct(c_y,c_t1);   // <y|A x>
-  ComplexD ip2 = innerProduct(c_t2,c_x);   // <A^dag y|x>
-  RealD reldiff = abs(ip1-ip2)/abs(ip1);
-  std::cout << GridLogMessage << "CENSUS: <y|Ax>       = " << ip1 << std::endl;
-  std::cout << GridLogMessage << "CENSUS: <Adag y|x>   = " << ip2 << std::endl;
-  std::cout << GridLogMessage << "CENSUS: rel diff     = " << reldiff << "  (expect ~1e-14; FAIL if O(1))" << std::endl;
-  GRID_ASSERT(reldiff < 1.0e-8);
-
-  // Coarse near-null ("global") vectors psi_c[k] = P^dag raw[k], stored for the
-  // Ritz-matrix + deflation study in CENSUS 6 (filled in CENSUS 2's projection
-  // loop below, before raw[]/subspace[] are freed).
-  std::vector<CoarseVector> psi_c(nbasis,Coarse5d);
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 2: nulliness of the RAW vectors' coarse images against A_c.
-  //   psi_c[k] = P^dag raw[k].  Galerkin guarantees the Rayleigh quotient
-  //   equals CENSUS 0's fine value exactly (raw[k] is in span of its own
-  //   chopped pieces) -- agreement is a machine-precision validation of
-  //   the coarsening.  ||A_c psi_c||/||psi_c|| is the sigma-relevant norm.
-  //   The representability column ||raw - P psi_c||/||raw|| must be ~eps.
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 2: raw-vector coarse images against coarse operator" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  {
-    LatticeFermionD back(FGrid);
-    for(int k=0;k<nbasis;k++){
-      AggregatesGCR.ProjectToSubspace(c_x, raw[k]);
-      psi_c[k] = c_x;                       // store coarse near-null vector for CENSUS 6
-      AggregatesGCR.PromoteFromSubspace(c_x, back);
-      back = back - raw[k];
-      RealD represent = std::sqrt(norm2(back)/norm2(raw[k]));
-      LittleDiracOpPV.M(c_x, c_t1);
-      RealD    n2psi = norm2(c_x);
-      RealD    n2Apsi= norm2(c_t1);
-      ComplexD rq    = innerProduct(c_x,c_t1) / n2psi;
-      std::cout << GridLogMessage << "CENSUS: psi_c[" << k << "]  <psi|A|psi>/<psi|psi> = " << rq
-                << "   ||A psi||/||psi|| = " << std::sqrt(n2Apsi/n2psi)
-                << "   represent_err = " << represent << std::endl;
-    }
-  }
-
-  // Fine subspace + raw copy are needed only through CENSUS 2; CENSUS 3-5 are
-  // entirely coarse (LittleDiracOpPV only), and the CENSUS 4 evec save writes the
-  // coarse vectors directly.  Release the ~2*nbasis fine 5D fields (~14 GB/GCD at
-  // 2^4) HERE, before the order-ChebyOrder Lanczos whose padded coarse temporaries
-  // otherwise push host memory over the top on top of _A + _Adag (the AccCache
-  // CpuPtr!=NULL abort seen mid-iteration).
-  // Direct orthonormality check of the fine near-null vectors (GlobalOrthonormalise
-  // in CreateSubspaceGCR).  raw is freed just below, so this runs here, not CENSUS 6.
-  // If this is ~0 but the coarse Gram S (CENSUS 6) is not, the gap is representability,
-  // not orthonormality.
-  {
-    Eigen::MatrixXcd Gfine(nbasis,nbasis);
-    for(int i=0;i<nbasis;i++){
-      for(int j=i;j<nbasis;j++){
-        ComplexD g = innerProduct(raw[i],raw[j]);
-        Gfine(i,j) = std::complex<double>(g.real(),g.imag());
-        Gfine(j,i) = std::conj(Gfine(i,j));
-      }
-    }
-    double GmI = (Gfine - Eigen::MatrixXcd::Identity(nbasis,nbasis)).norm();
-    std::cout << GridLogMessage << "CENSUS 2b: fine Gram ||<raw_i|raw_j> - I||_F = " << GmI
-              << "   (expect ~0 if fine vectors orthonormal)" << std::endl;
-  }
-
-  raw.clear();                    raw.shrink_to_fit();
-  AggregatesGCR.subspace.clear(); AggregatesGCR.subspace.shrink_to_fit();
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 3: sigma_max^2 = lambda_max( A_c^dag A_c ) by power method
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 3: power method for sigma_max" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  MdagMLinearOperator<LittleDiracOperator,CoarseVector> HermOpAdagA(LittleDiracOpPV);
-  random(CRNG,c_x);
-  PowerMethod<CoarseVector> PM;
-  RealD sigmax2 = PM(HermOpAdagA,c_x);
-  std::cout << GridLogMessage << "CENSUS: lambda_max(AdagA) = " << sigmax2
-            << "   sigma_max = " << std::sqrt(sigmax2) << std::endl;
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 4: low singular values via Chebyshev-filtered IRL on A^dag A
-  //   The low end of sigma(A_c) is the pseudospectrum of A_c at z=0.
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 4: Chebyshev-filtered Lanczos, low sigma^2" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  RealD cheby_hi = (ChebyHi > 0.0) ? ChebyHi : 1.1*sigmax2;
-  std::cout << GridLogMessage << "Chebyshev filter [" << ChebyLo << "," << cheby_hi << "] order " << ChebyOrder << std::endl;
-
-  // eval/evec/Nconv hoisted out of the block so CENSUS 6 can deflate with them.
-  std::vector<RealD>        eval(CensusNm);
-  std::vector<CoarseVector> evec(CensusNm,Coarse5d);
-  int Nconv=0;
-  {
-    Chebyshev<CoarseVector>      Cheby(ChebyLo,cheby_hi,ChebyOrder);
-    FunctionHermOp<CoarseVector> OpCheby(Cheby,HermOpAdagA);
-    PlainHermOp<CoarseVector>    Op     (HermOpAdagA);
-
-    ImplicitlyRestartedLanczos<CoarseVector> IRL(OpCheby,Op,CensusNstop,CensusNk,CensusNm,CensusTol,CensusMaxIt);
-
-    random(CRNG,c_x);
-    IRL.calc(eval,evec,c_x,Nconv);
-
-    std::cout << GridLogMessage << "CENSUS: converged " << Nconv << " modes of AdagA" << std::endl;
-    for(int i=0;i<Nconv;i++){
-      std::cout << GridLogMessage << "CENSUS: sigma[" << i << "]^2 = " << eval[i]
-                << "   sigma = " << std::sqrt(std::max(eval[i],0.0)) << std::endl;
-    }
-
-    // Optionally persist the low right-singular-vector basis: this IS the
-    // deflation basis for the coarse solve (ADEF1 / MultiRHSDeflation).
-    // Set CENSUS_EVEC_FILE to enable.
-    if ( getenv("CENSUS_EVEC_FILE") && Nconv>0 ) {
-#ifdef HAVE_LIME
-      std::string evec_file(getenv("CENSUS_EVEC_FILE"));
-      std::string eval_file = evec_file + ".evals.xml";
-      std::cout << GridLogMessage << "CENSUS: saving " << Nconv << " singular vectors to " << evec_file << std::endl;
-      emptyUserRecord record;
-      ScidacWriter WR(evec[0].Grid()->IsBoss());
-      WR.open(evec_file);
-      for(int i=0;i<Nconv;i++) WR.writeScidacFieldRecord(evec[i],record);
-      WR.close();
-      XmlWriter WRx(eval_file);
-      std::vector<RealD> eval_out(eval.begin(),eval.begin()+Nconv); // don't shrink shared eval
-      write(WRx,"evals",eval_out);
-#endif
-    }
-  }
-
-  // NB: evec/eval stay sized CensusNm (Lattice has no default ctor, so
-  // std::vector<CoarseVector>::resize won't instantiate).  They match in size,
-  // which is all DeflatedGuesser asserts; CENSUS 6 only ever indexes [0,Nconv).
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 5: half-plane margin from the Hermitian part
-  //   lambda_min(H) = min Re W(A_c) > 0 <=> positive-real (EES applies)
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 5: Hermitian part H=(A+Adag)/2, half-plane margin" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  HermitianPartOperator<LittleDiracOperator,CoarseVector> HermPart(LittleDiracOpPV);
-
-  random(CRNG,c_x);
-  RealD lamHmax = PM(HermPart,c_x);
-  std::cout << GridLogMessage << "CENSUS: lambda_max(H) = " << lamHmax << std::endl;
-
-  // lambda_min(H): most-negative eigenvalue via Chebyshev-filtered IRL on H.
-  // A shifted power method cannot separate it from the dense low tail (which is
-  // why the earlier -0.006 is suspect); Cheby(lo, hi>=lambda_max) amplifies the
-  // most-negative mode hardest so IRL isolates the true bottom of the spectrum.
-  RealD hpLo    = getenv("HPLANE_CHEBY_LO")    ? atof(getenv("HPLANE_CHEBY_LO"))    : 0.1;
-  RealD hpHi    = getenv("HPLANE_CHEBY_HI")    ? atof(getenv("HPLANE_CHEBY_HI"))    : 1.1*lamHmax;
-  int   hpOrder = getenv("HPLANE_CHEBY_ORDER") ? atoi(getenv("HPLANE_CHEBY_ORDER")) : 61;
-  // Grid's Chebyshev filter MUST be odd order (positive for x < -1, where the low/
-  // negative modes map); an even order flips the sign there and the IRL blows up.
-  if(hpOrder%2==0){ hpOrder++;
-    std::cout<<GridLogMessage<<"HPLANE_CHEBY_ORDER forced odd -> "<<hpOrder<<std::endl; }
-  int   hpNstop = getenv("HPLANE_NSTOP")       ? atoi(getenv("HPLANE_NSTOP"))       : 8;
-  int   hpNk    = getenv("HPLANE_NK")          ? atoi(getenv("HPLANE_NK"))          : 24;
-  int   hpNm    = getenv("HPLANE_NM")          ? atoi(getenv("HPLANE_NM"))          : 48;
-  RealD hpTol   = getenv("HPLANE_TOL")         ? atof(getenv("HPLANE_TOL"))         : 1.0e-4;
-  int   hpMaxIt = getenv("HPLANE_MAXIT")       ? atoi(getenv("HPLANE_MAXIT"))       : 20;
-
-  Chebyshev<CoarseVector>      HCheby(hpLo,hpHi,hpOrder);
-  FunctionHermOp<CoarseVector> HOpCheby(HCheby,HermPart);
-  PlainHermOp<CoarseVector>    HOpPlain(HermPart);
-  ImplicitlyRestartedLanczos<CoarseVector> HIRL(HOpCheby,HOpPlain,hpNstop,hpNk,hpNm,hpTol,hpMaxIt);
-  std::vector<RealD>        heval(hpNm);
-  std::vector<CoarseVector> hevec(hpNm,Coarse5d);
-  int hNconv=0;
-  random(CRNG,c_x);
-  HIRL.calc(heval,hevec,c_x,hNconv);
-  RealD lamHmin = (hNconv>0) ? heval[0] : 9.99e99;
-  for(int kk=0;kk<hNconv;kk++) lamHmin = std::min(lamHmin, heval[kk]);
-  std::cout << GridLogMessage << "CENSUS: IRL H-bottom converged " << hNconv
-            << " eigenvalues; most-negative = " << lamHmin << std::endl;
-  std::cout << GridLogMessage << "CENSUS: lambda_min(H) = " << lamHmin
-            << "  (positive-real / half-plane margin; NEGATIVE => GCR unguaranteed)" << std::endl;
-
-  //////////////////////////////////////////////////////////////////////
-  // CENSUS 6: Ritz matrix of the coarse near-null basis + deflated-CG study
-  //
-  //   C_ij = <psi_c^i | A^dag A | psi_c^j>,   S_ij = <psi_c^i | psi_c^j>.
-  //   psi_c are NOT orthonormal (raw near-null projected to coarse), so the
-  //   Rayleigh-Ritz problem is the GENERALISED Hermitian one  C v = theta S v.
-  //   Its eigenpairs (theta_i, g_i = sum_j V(j,i) psi_c^j) are the best approximate
-  //   eigenpairs of A^dag A available from span{psi_c}; Eigen normalises so that
-  //   V^dag S V = I, hence <g_i|g_j> = delta_ij and the g_i are an orthonormal
-  //   DeflatedGuesser basis.  Compare theta_i to the Lanczos sigma_i^2, then run
-  //   three CG solves on A^dag A: [1] no deflation, [2] Lanczos-eigenvector
-  //   deflated guess, [3] Ritz global-vector deflated guess (g_i treated as pure
-  //   eigenvectors with eigenvalue theta_i).
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS 6: Ritz matrix C_ij = <psi_c^i|AdagA|psi_c^j> + deflated CG" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-
-  std::vector<CoarseVector> Apsi(nbasis,Coarse5d);
-  for(int j=0;j<nbasis;j++) HermOpAdagA.HermOp(psi_c[j],Apsi[j]);   // A^dag A psi_c^j
-
-  Eigen::MatrixXcd Cmat(nbasis,nbasis);
-  Eigen::MatrixXcd Smat(nbasis,nbasis);
-  for(int i=0;i<nbasis;i++){
-    for(int j=0;j<nbasis;j++){
-      ComplexD cij = innerProduct(psi_c[i],Apsi[j]);
-      ComplexD sij = innerProduct(psi_c[i],psi_c[j]);
-      Cmat(i,j) = std::complex<double>(cij.real(),cij.imag());
-      Smat(i,j) = std::complex<double>(sij.real(),sij.imag());
-    }
-  }
-
-  {
-    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> ses(Smat);
-    Eigen::MatrixXcd Id = Eigen::MatrixXcd::Identity(nbasis,nbasis);
-    double SmI = (Smat - Id).norm();   // ||S - I||_F : ~0 iff psi_c orthonormal
-    std::cout << GridLogMessage << "CENSUS 6: Gram S eig range [" << ses.eigenvalues()(0)
-              << ", " << ses.eigenvalues()(nbasis-1)
-              << "]   ||S - I||_F = " << SmI
-              << "   (expect ~0: fine vectors are GlobalOrthonormalise'd => psi_c orthonormal)" << std::endl;
-  }
-  Eigen::GeneralizedSelfAdjointEigenSolver<Eigen::MatrixXcd> ges(Cmat,Smat);
-  Eigen::VectorXd  theta = ges.eigenvalues();    // ascending, real
-  Eigen::MatrixXcd Vr    = ges.eigenvectors();   // columns; V^dag S V = I
-
-  int ncmp = std::min((int)nbasis,Nconv);
-  std::cout << GridLogMessage << "CENSUS 6: Ritz theta vs Lanczos sigma^2 (both ascending):" << std::endl;
-  for(int i=0;i<nbasis;i++){
-    if(i<ncmp)
-      std::cout << GridLogMessage << "CENSUS: theta["<<i<<"] = "<<theta(i)
-                <<"   sqrt = "<<std::sqrt(std::max(theta(i),0.0))
-                <<"   | sigma^2 = "<<eval[i]<<"   theta/sigma^2 = "<<theta(i)/eval[i] << std::endl;
-    else
-      std::cout << GridLogMessage << "CENSUS: theta["<<i<<"] = "<<theta(i)
-                <<"   sqrt = "<<std::sqrt(std::max(theta(i),0.0)) << std::endl;
-  }
-
-  // Ritz global vectors g_i = sum_j V(j,i) psi_c^j  (S-orthonormal), eigenvalue theta_i
-  std::vector<CoarseVector> gvec(nbasis,Coarse5d);
-  std::vector<RealD>        gval(nbasis);
-  for(int i=0;i<nbasis;i++){
-    gvec[i] = Zero();
-    for(int j=0;j<nbasis;j++){
-      ComplexD coeff(Vr(j,i).real(),Vr(j,i).imag());
-      axpy(gvec[i],coeff,psi_c[j],gvec[i]);
-    }
-    gval[i] = theta(i);
-  }
-
-  // How good are the diagonalised global vectors as actual eigenvectors of A^dag A?
-  {
-    CoarseVector Ag(Coarse5d), rr(Coarse5d);
-    int nchk = std::min((int)nbasis,16);
-    for(int i=0;i<nchk;i++){
-      HermOpAdagA.HermOp(gvec[i],Ag);
-      axpy(rr,-gval[i],gvec[i],Ag);          // rr = A^dag A g - theta g
-      RealD rn = std::sqrt(norm2(rr));
-      std::cout << GridLogMessage << "CENSUS 6: Ritz resid ["<<i<<"] ||AdagA g - theta g||/theta = "
-                << rn/std::max(gval[i],1.0e-30) << "   (theta="<<gval[i]<<")" << std::endl;
-    }
-  }
-
-  // --- Three CG solves on A^dag A, common random source ---
-  int rankLanc = (DeflRank>0) ? std::min(DeflRank,Nconv)       : Nconv;
-  int rankRitz = (DeflRank>0) ? std::min(DeflRank,(int)nbasis) : (int)nbasis;
-  std::cout << GridLogMessage << "CENSUS 6: CG tol "<<CGdeflTol<<" maxit "<<CGdeflMaxIt
-            << " ; deflation ranks -- Lanczos "<<rankLanc<<", Ritz "<<rankRitz << std::endl;
-
-  CoarseVector cg_src(Coarse5d);  random(CRNG,cg_src);
-  CoarseVector cg_x  (Coarse5d);
-  ConjugateGradient<CoarseVector> CGdefl(CGdeflTol,CGdeflMaxIt,false);
-
-  cg_x = Zero();
-  CGdefl(HermOpAdagA,cg_src,cg_x);
-  std::cout << GridLogMessage << "CENSUS 6: [1] no deflation          : iters = "
-            << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
-
-  if(rankLanc>0){
-    DeflatedGuesser<CoarseVector> guessL(evec,eval,rankLanc);
-    guessL(cg_src,cg_x);
-    CGdefl(HermOpAdagA,cg_src,cg_x);
-    std::cout << GridLogMessage << "CENSUS 6: [2] Lanczos-evec deflation : iters = "
-              << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
-  }
-
-  {
-    DeflatedGuesser<CoarseVector> guessR(gvec,gval,rankRitz);
-    guessR(cg_src,cg_x);
-    CGdefl(HermOpAdagA,cg_src,cg_x);
-    std::cout << GridLogMessage << "CENSUS 6: [3] Ritz-vector deflation  : iters = "
-              << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Summary
-  //////////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "CENSUS SUMMARY" << std::endl;
-  std::cout << GridLogMessage << "  sigma_max        = " << std::sqrt(sigmax2) << std::endl;
-  std::cout << GridLogMessage << "  lambda_max(H)    = " << lamHmax  << std::endl;
-  std::cout << GridLogMessage << "  lambda_min(H)    = " << lamHmin  << std::endl;
-  std::cout << GridLogMessage << "  low sigma census : see CENSUS 4 table above" << std::endl;
-  std::cout << GridLogMessage << "  Compare min sigma with |lambda| from Krylov-Schur (Patrick):" << std::endl;
-  std::cout << GridLogMessage << "    sigma_min ~ min|lambda| : effectively normal; deflation rank is the issue" << std::endl;
-  std::cout << GridLogMessage << "    sigma_min << min|lambda|: non-normal; need two-sided/singular-vector deflation" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_halfplane.cc

@@ -1,278 +0,0 @@
-/*
- * Example_pvdagm_halfplane.cc
- *
- * Standalone fine-operator diagnostic: the EES half-plane margin of the
- * (non-Hermitian) PV-preconditioned Mobius DWF operator
- *
- *     A(m_adj) = D_adj^dag D_light      (D_adj plays the Pauli-Villars role)
- *
- * as a function of the adjoint mass m_adj, dialled from the light quark mass
- * up to the Pauli-Villars mass (=1).  No coarse grid, no subspace, no Lanczos
- * -- pure power-method spectral tests on the fine grid.
- *
- * Purpose: A is the LEFT preconditioner for inverting the light operator.
- * To solve  D_light X = B  we iterate the preconditioned system
- *     (D_adj^dag D_light) X = D_adj^dag B ,
- * whose solution X is independent of m_adj -- only the conditioning and the
- * iterative convergence change.  m_adj = m_light is the usual CGNR (symmetric
- * normal equations); m_adj = 1 is the Pauli-Villars preconditioned system.
- * The sweep asks which m_adj keeps the preconditioned operator well-behaved
- * (positive-real / EES-guaranteed) while buying the wider spectral range.
- *
- * For the Hermitian part  H(A) = (A + A^dag)/2  we measure, per m_adj:
- *
- *   lambda_max(H)   -- power method on H
- *   lambda_min(H)   -- power method on (sI - H) => min Re W(A), the half-plane
- *                      margin.  EES (Eisenstat-Elman-Schultz 1983, Thm 3.3)
- *                      GUARANTEES GCR convergence with rate
- *                          [ 1 - lambda_min(H)^2 / sigma_max^2 ]^{1/2}
- *                      ONLY when lambda_min(H) > 0 (positive-real / A's field
- *                      of values in the open right half-plane).  A negative
- *                      value means the guarantee is lost (not that GCR
- *                      diverges); the magnitude is then the distance-to-
- *                      positive-realness, i.e. the shift/deflation needed to
- *                      recover it.
- *   sigma_max       -- power method on A^dag A (= A.HermOp)
- *
- * Endpoints:
- *   m_adj = m_light  =>  A = M^dag M, Hermitian PD, positive-real by
- *                        construction, lambda_min(H) = sigma_min^2 > 0 (the
- *                        squared / CGNR operator).
- *   m_adj = 1        =>  A = PV^dag M, the standard PVdagM operator.
- *
- * Env: MASS, M5, MOBIUS_B, MOBIUS_C, LS, CONFIG,
- *      MADJ_LIST (comma separated) OR MADJ_MIN / MADJ_MAX / MADJ_N (geometric).
- *
- * Caveat: lambda_min(H) via a shifted power method can be soft when it sits
- * near zero over a dense low spectrum.  The SIGN and the TREND across m_adj
- * are the robust signal; confirm an individual near-zero value with a proper
- * shifted Lanczos if it is load-bearing.
- */
-
-#include <Grid/Grid.h>
-
-using namespace std;
-using namespace Grid;
-
-//////////////////////////////////////////////////////////////////////
-// A = PV^dag M : Op = _PV.Mdag . _Mat.M ,  AdjOp = _Mat.Mdag . _PV.M
-//////////////////////////////////////////////////////////////////////
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){ // A^dag A
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// H = (A + A^dag)/2 for a general non-Hermitian LinearOperator A.
-//////////////////////////////////////////////////////////////////////
-template<class Field>
-class HermitianPartLinOp : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_A;
-public:
-  HermitianPartLinOp(LinearOperatorBase<Field> &A): _A(A) {};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){ HermOp(in,out); }
-  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _A.Op(in,out);      // A in
-    _A.AdjOp(in,tmp);   // A^dag in
-    out = 0.5*(out + tmp);
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// s*I - Op : power method on this gives s - lambda_min(Op) for Hermitian Op.
-//////////////////////////////////////////////////////////////////////
-template<class Field>
-class ShiftedNegatedOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op;
-  RealD s;
-public:
-  ShiftedNegatedOperator(RealD _s, LinearOperatorBase<Field> &Op): _Op(Op), s(_s) {};
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){ HermOp(in,out); }
-  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1=real(dot);
-    n2=norm2(out);
-  }
-  void HermOp(const Field &in, Field &out){
-    _Op.HermOp(in,out);
-    out = s*in - out;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  RealD mass = 0.00078;
-  RealD M5   = 1.8;
-  RealD b    = 1.5;
-  RealD c    = 0.5;
-  int   Ls   = 24;
-  std::string config("ckpoint_lat.1000");
-
-  if(getenv("MASS"))     mass = atof(getenv("MASS"));
-  if(getenv("M5"))       M5   = atof(getenv("M5"));
-  if(getenv("MOBIUS_B")) b    = atof(getenv("MOBIUS_B"));
-  if(getenv("MOBIUS_C")) c    = atof(getenv("MOBIUS_C"));
-  if(getenv("LS"))       Ls   = atoi(getenv("LS"));
-  if(getenv("CONFIG"))   config = std::string(getenv("CONFIG"));
-
-  // Adjoint-mass sweep: explicit list, or geometric MADJ_MIN..MADJ_MAX in MADJ_N steps.
-  std::vector<RealD> madj_list;
-  if(getenv("MADJ_LIST")){
-    std::stringstream ss(getenv("MADJ_LIST"));
-    std::string tok;
-    while(std::getline(ss,tok,',')) if(tok.size()) madj_list.push_back(std::stod(tok));
-  } else {
-    int   N  = getenv("MADJ_N")   ? atoi(getenv("MADJ_N"))   : 6;
-    RealD lo = getenv("MADJ_MIN") ? atof(getenv("MADJ_MIN")) : mass;
-    RealD hi = getenv("MADJ_MAX") ? atof(getenv("MADJ_MAX")) : 1.0;
-    GRID_ASSERT(N>=1);
-    for(int i=0;i<N;i++)
-      madj_list.push_back( (N==1) ? lo : lo*std::pow(hi/lo, double(i)/double(N-1)) );
-  }
-
-  // lambda_min(H) is the most-negative eigenvalue; resolved by Chebyshev-filtered
-  // Lanczos on H (a shifted power method cannot separate it from the dense low tail).
-  RealD HalfChebyLo    = getenv("HALF_CHEBY_LO")    ? atof(getenv("HALF_CHEBY_LO"))    : 0.1;
-  RealD HalfChebyHi    = getenv("HALF_CHEBY_HI")    ? atof(getenv("HALF_CHEBY_HI"))    : 0.0; // 0 => auto
-  int   HalfChebyOrder = getenv("HALF_CHEBY_ORDER") ? atoi(getenv("HALF_CHEBY_ORDER")) : 61;
-  // Grid's Chebyshev filter MUST be odd order: only then is the polynomial positive
-  // for x < -1, the region the low/negative modes map to.  An even order flips the
-  // sign there, the filtered operator explodes negative, and the IRL never converges.
-  if(HalfChebyOrder%2==0){ HalfChebyOrder++;
-    std::cout<<GridLogMessage<<"HALF_CHEBY_ORDER forced odd -> "<<HalfChebyOrder<<std::endl; }
-  int   HalfNstop      = getenv("HALF_NSTOP")       ? atoi(getenv("HALF_NSTOP"))       : 8;
-  int   HalfNk         = getenv("HALF_NK")          ? atoi(getenv("HALF_NK"))          : 24;
-  int   HalfNm         = getenv("HALF_NM")          ? atoi(getenv("HALF_NM"))          : 48;
-  RealD HalfTol        = getenv("HALF_TOL")         ? atof(getenv("HALF_TOL"))         : 1.0e-4;
-  int   HalfMaxIt      = getenv("HALF_MAXIT")       ? atoi(getenv("HALF_MAXIT"))       : 20;
-
-  std::vector<int> lat = {48,48,48,96};
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
-
-  std::cout << GridLogMessage << "PARAM: MASS(light) " << mass << "  M5 " << M5
-            << "  b " << b << "  c " << c << "  Ls " << Ls << std::endl;
-  std::cout << GridLogMessage << "PARAM: CONFIG " << config << std::endl;
-
-  LatticeGaugeField Umu(UGrid);
-  FieldMetaData header;
-  std::cout << GridLogMessage << "Reading gauge field " << config << std::endl;
-  NerscIO::readConfiguration(Umu,header,config);
-
-  // Fixed light operator (never changes across the sweep).
-  MobiusFermionD Dlight(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid, mass, M5, b, c);
-
-  LatticeFermionD x(FGrid);
-
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "FINE HALF-PLANE SWEEP  A(m_adj) = D_adj^dag D_light" << std::endl;
-  std::cout << GridLogMessage << "  m_adj = " << mass << " => M^dag M (positive-real); m_adj = 1 => PVdagM" << std::endl;
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-
-  for(auto madj : madj_list){
-
-    MobiusFermionD Dadj(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid, madj, M5, b, c);
-
-    PVdagMLinearOperator<MobiusFermionD,LatticeFermionD> A(Dlight,Dadj); // A = Dadj^dag Dlight
-    HermitianPartLinOp<LatticeFermionD>                  H(A);
-
-    PowerMethod<LatticeFermionD> PM;
-
-    random(RNG5,x); RealD lamHmax = PM(H,x);
-
-    // lambda_min(H): most-negative eigenvalue via Chebyshev-filtered IRL on H.
-    // Cheby(lo,hi) amplifies eigenvalues below lo; with hi>=lambda_max(H) the most
-    // negative mode is amplified hardest, so IRL isolates the true bottom of the
-    // (possibly indefinite) spectrum where the shifted power method could not.
-    RealD fhi = (HalfChebyHi>0.0)? HalfChebyHi : 1.1*lamHmax;
-    Chebyshev<LatticeFermionD>      Cheby(HalfChebyLo,fhi,HalfChebyOrder);
-    FunctionHermOp<LatticeFermionD> OpCheby(Cheby,H);
-    PlainHermOp<LatticeFermionD>    OpPlain(H);
-    ImplicitlyRestartedLanczos<LatticeFermionD> IRL(OpCheby,OpPlain,HalfNstop,HalfNk,HalfNm,HalfTol,HalfMaxIt);
-    std::vector<RealD>           heval(HalfNm);
-    std::vector<LatticeFermionD> hevec(HalfNm,FGrid);
-    int hNconv=0;
-    random(RNG5,x);
-    IRL.calc(heval,hevec,x,hNconv);
-    RealD lamHmin = (hNconv>0) ? heval[0] : 9.99e99;
-    for(int kk=0;kk<hNconv;kk++) lamHmin = std::min(lamHmin, heval[kk]);
-    std::cout << GridLogMessage << "  (IRL H-bottom: " << hNconv
-              << " converged, most-negative eval " << lamHmin << ")" << std::endl;
-
-    random(RNG5,x); RealD sigmax2 = PM(A,x);   // A.HermOp = A^dag A
-    RealD sigmax  = std::sqrt(sigmax2);
-
-    bool  posreal = (lamHmin > 0.0);
-    RealD ratefac = posreal ? std::sqrt(1.0 - lamHmin*lamHmin/sigmax2) : 0.0; // EES per-iter
-    RealD iters8  = (posreal && ratefac < 1.0) ? std::log(1.0e-8)/std::log(ratefac) : 0.0;
-
-    std::cout << GridLogMessage << "HALFPLANE: m_adj " << madj
-              << "  lambda_min(H) " << lamHmin
-              << "  lambda_max(H) " << lamHmax
-              << "  sigma_max " << sigmax
-              << "  positive_real " << (posreal ? "YES" : "NO ")
-              << (posreal
-                    ? ("  EES_rate " + std::to_string(ratefac) + "  EES_iters(1e-8) " + std::to_string(iters8))
-                    : ("  margin_below_zero " + std::to_string(-lamHmin) + " (EES guarantee lost)"))
-              << std::endl;
-  }
-
-  std::cout << GridLogMessage << "=================================================" << std::endl;
-  std::cout << GridLogMessage << "Reading: lambda_min(H) > 0 => EES guarantees GCR at the quoted rate." << std::endl;
-  std::cout << GridLogMessage << "         crossing to < 0 as m_adj -> 1 marks loss of positive-realness." << std::endl;
-  std::cout << GridLogMessage << "         (non-normality: eigenvalues may still be right-half-plane.)" << std::endl;
-  std::cout << GridLogMessage << "Done" << std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_mrhs_3level.cc

@@ -1,613 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_mrhs_3level.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// MultiRHS (valence) THREE-level multigrid for PVdagM.
-//
-// This is exactly the plain three-level algorithm of Example_pvdagm_3level_SVDdefl.cc
-// with L3_DEFL=0 (NO deflation), applied to the enlarged block-diagonal mRHS system:
-// the coarse and coarse-coarse levels run a SINGLE Krylov (one GCR polynomial, inner
-// products summed over rhs) on the packed 6D mrhs fields, so both coarse levels batch
-// through GEMM (MultiGeneralCoarsenedMatrix) -- the valence throughput win at BOTH levels.
-//
-// Level structure (each coarse level is a single-field PGCR on a packed 6D mrhs field):
-//   L1 (fine)         : std::vector<LatticeFermionD>, MrhsPGCRNonHermitian on PVdagM,
-//                       preconditioned by the L1->L2 mrhs V-cycle (MrhsTwoLevelMG).
-//   L2 (coarse)       : 6D mrhs coarse field, PGCR, preconditioned by the L2->L3 mrhs
-//                       V-cycle (MrhsCoarseThreeLevelPrec) -- coarse-coarse correction + coarse smoother.
-//   L3 (coarse-coarse): 6D mrhs coarse-coarse field, PGCR (the innermost solve).
-//
-// RAW-NULL DISCIPLINE (critical -- see project_block_orthogonalise_leak): the L2->L3
-// aggregation MUST be built from RAW fine near-null vectors (pre block-GS).  We take a
-// raw copy of the loaded subspace BEFORE the L1->L2 CoarsenOperator (which block-
-// orthonormalises in place) and project THAT.  Guards print ||<psi|psi> - I||: ~0.23 =
-// content preserved, ~N_coarse = the e_k leak is back.
-//
-// Env: MASS SUBSPACE_FILE NRHS
-//      BLOCK  (dotted, default 2.2.2.2)  BLOCK2 (dotted, default 2.2.3.3)
-//      FineSmootherShift FineSmootherOrder
-//      CoarseSmootherShift CoarseSmootherNstep
-//      CoarseSolverTol CoarseSolverOrder
-//      L3_TOL L3_MAXIT L3_NSTEP
-//      OuterMmax OuterNstep OuterTol
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-
-using namespace std;
-using namespace Grid;
-
-RealD FineSmootherShift    = 0.1;
-int   FineSmootherOrder    = 16;
-RealD CoarseSmootherShift  = 0.1;
-int   CoarseSmootherNstep  = 4;
-RealD CoarseSolverTol      = 0.03;
-int   CoarseSolverOrder    = 200;
-RealD L3Tol                = 2.5e-1;
-int   L3MaxIt              = 50;
-int   L3Nstep              = 50;
-RealD OuterTol             = 1.0e-8;
-int   OuterMmax            = 8;
-int   OuterNstep           = 8;
-int   Nrhs                 = 12;
-RealD mass                 = 0.00078;
-
-void ParseEnvironment(void)
-{
-  if(getenv("MASS"))               mass               = atof(getenv("MASS"));
-  if(getenv("FineSmootherShift"))  FineSmootherShift  = atof(getenv("FineSmootherShift"));
-  if(getenv("FineSmootherOrder"))  FineSmootherOrder  = atoi(getenv("FineSmootherOrder"));
-  if(getenv("CoarseSmootherShift"))CoarseSmootherShift= atof(getenv("CoarseSmootherShift"));
-  if(getenv("CoarseSmootherNstep"))CoarseSmootherNstep= atoi(getenv("CoarseSmootherNstep"));
-  if(getenv("CoarseSolverTol"))    CoarseSolverTol    = atof(getenv("CoarseSolverTol"));
-  if(getenv("CoarseSolverOrder"))  CoarseSolverOrder  = atoi(getenv("CoarseSolverOrder"));
-  if(getenv("L3_TOL"))             L3Tol              = atof(getenv("L3_TOL"));
-  if(getenv("L3_MAXIT"))           L3MaxIt            = atoi(getenv("L3_MAXIT"));
-  if(getenv("L3_NSTEP"))           L3Nstep            = atoi(getenv("L3_NSTEP"));
-  if(getenv("OuterTol"))           OuterTol           = atof(getenv("OuterTol"));
-  if(getenv("OuterMmax"))          OuterMmax          = atoi(getenv("OuterMmax"));
-  if(getenv("OuterNstep"))         OuterNstep         = atoi(getenv("OuterNstep"));
-  if(getenv("NRHS"))               Nrhs               = atoi(getenv("NRHS"));
-
-  std::cout << GridLogMessage << "PARAM: MASS               " << mass               << std::endl;
-  std::cout << GridLogMessage << "PARAM: NRHS               " << Nrhs               << std::endl;
-  std::cout << GridLogMessage << "PARAM: FineSmootherShift  " << FineSmootherShift  << std::endl;
-  std::cout << GridLogMessage << "PARAM: FineSmootherOrder  " << FineSmootherOrder  << std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift" << CoarseSmootherShift<< std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSmootherNstep" << CoarseSmootherNstep<< std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverTol    " << CoarseSolverTol    << std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder  " << CoarseSolverOrder  << std::endl;
-  std::cout << GridLogMessage << "PARAM: L3_TOL             " << L3Tol              << std::endl;
-  std::cout << GridLogMessage << "PARAM: OuterMmax          " << OuterMmax          << std::endl;
-  std::cout << GridLogMessage << "PARAM: OuterNstep         " << OuterNstep         << std::endl;
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-#ifdef HAVE_LIME
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-  SW.open(fname);
-  for (int k = 0; k < (int)subspace.size(); k++) SW.writeScidacFieldRecord(subspace[k], record);
-  SW.close();
-#endif
-}
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-#ifdef HAVE_LIME
-  Grid::emptyUserRecord record;
-  Grid::ScidacReader SR;
-  SR.open(fname);
-  for (int k = 0; k < (int)subspace.size(); k++) SR.readScidacFieldRecord(subspace[k], record);
-  SR.close();
-#endif
-}
-
-//////////////////////////////////////////////////////////////////////
-// A = PV^dag M (non-Hermitian), and shifted variant for smoothers.
-//////////////////////////////////////////////////////////////////////
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat; Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
-  void OpDiag (const Field &in, Field &out) { assert(0); }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
-  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); }
-  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(in,tmp); _Mat.Mdag(tmp,out); }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ HermOp(in,out); ComplexD d=innerProduct(in,out); n1=real(d); n2=norm2(out); }
-  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat; Matrix &_PV;
-public:
-  RealD shift;
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-  void OpDiag (const Field &in, Field &out) { assert(0); }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
-  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); out = out + shift*in; }
-  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(tmp,out); _Mat.Mdag(in,tmp); out = out + shift*in; }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-// Generic shift wrapper (for the coarse-level smoother on the 6D mrhs coarse operator).
-template<class Field>
-class ShiftedLinearOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op; RealD shift;
-public:
-  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : _Op(Op), shift(_shift) {}
-  void OpDiag  (const Field &in, Field &out) { assert(0); }
-  void OpDir   (const Field &in, Field &out,int dir,int disp) { assert(0); }
-  void OpDirAll (const Field &in, std::vector<Field> &out) { assert(0); }
-  void Op      (const Field &in, Field &out) { _Op.Op(in,out);    out = out + shift*in; }
-  void AdjOp   (const Field &in, Field &out) { _Op.AdjOp(in,out); out = out + shift*in; }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp  (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-//////////////////////////////////////////////////////////////////////
-// mrhs interfaces + single-polynomial mrhs PGCR (verbatim from Example_pvdagm_mrhs.cc):
-// reductions summed over rhs -> one alpha/beta per step for the enlarged system.
-//////////////////////////////////////////////////////////////////////
-template<class Field>
-class MrhsLinearFunction {
-public:
-  virtual void operator()(std::vector<Field> &in, std::vector<Field> &out) = 0;
-};
-
-template<class Field>
-class MrhsPGCRNonHermitian {
-public:
-  RealD Tolerance; Integer MaxIterations; int mmax,nstep,steps,level;
-  int ZeroGuess = 0; int FirstCycle = 0;   // caller contract: zero guess => first-cycle r0 = src
-  std::string name = "Level 1";
-  LinearOperatorBase<Field> &Linop;
-  MrhsLinearFunction<Field> &Preconditioner;
-  void Level(int lv){ name = "Level " + std::to_string(lv); level=lv; }
-  void Name(std::string n){ name = n; }
-  void SetZeroGuess(int z){ ZeroGuess=z; }
-  MrhsPGCRNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,MrhsLinearFunction<Field> &Prec,int _mmax,int _nstep)
-    : Tolerance(tol),MaxIterations(maxit),Linop(_Linop),Preconditioner(Prec),mmax(_mmax),nstep(_nstep){ level=1; }
-  static RealD vnorm2(std::vector<Field> &x){ RealD s=0; for(auto &f:x) s+=norm2(f); return s; }
-  static ComplexD vinnerProduct(std::vector<Field> &x,std::vector<Field> &y){ ComplexD s(0); for(int r=0;r<(int)x.size();r++) s+=innerProduct(x[r],y[r]); return s; }
-  static void vaxpy(std::vector<Field> &z,ComplexD a,std::vector<Field> &x,std::vector<Field> &y){ for(int r=0;r<(int)z.size();r++) axpy(z[r],a,x[r],y[r]); }
-  void vOp(std::vector<Field> &in,std::vector<Field> &out){ for(int r=0;r<(int)in.size();r++) Linop.Op(in[r],out[r]); }
-  void operator()(std::vector<Field> &src,std::vector<Field> &psi){
-    RealD cp,ssq,rsq; int nrhs=src.size(); GridBase *grid=src[0].Grid();
-    ssq=vnorm2(src); rsq=Tolerance*Tolerance*ssq;
-    std::vector<Field> r(nrhs,grid);
-    GridStopWatch T; T.Start(); steps=0; FirstCycle=1;
-    for(int k=0;k<MaxIterations;k++){
-      cp=GCRnStep(src,psi,rsq);
-      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR("<<mmax<<","<<nstep<<") "<<steps<<" steps cp = "<<cp<<" target "<<rsq<<std::endl;
-      if(cp<rsq){
-        T.Stop(); vOp(psi,r); for(int rr=0;rr<nrhs;rr++) axpy(r[rr],-1.0,src[rr],r[rr]);
-        RealD tr=vnorm2(r);
-        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR: Converged on iteration "<<steps
-                 <<" computed residual "<<std::sqrt(cp/ssq)<<" true residual "<<std::sqrt(tr/ssq)<<" target "<<Tolerance<<std::endl;
-        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR Time elapsed: Total "<<T.Elapsed()<<std::endl;
-        for(int rr=0;rr<nrhs;rr++){ RealD rn=std::sqrt(norm2(r[rr])/norm2(src[rr])); std::cout<<GridLogMessage<<"MrhsPGCR per-rhs true residual["<<rr<<"] = "<<rn<<std::endl; }
-        return;
-      }
-    }
-    std::cout<<GridLogMessage<<"MrhsPGCR: did not converge"<<std::endl;
-  }
-  RealD GCRnStep(std::vector<Field> &src,std::vector<Field> &psi,RealD rsq){
-    RealD cp; ComplexD a,b,rq; RealD zAAz; int nrhs=src.size(); GridBase *grid=src[0].Grid();
-    std::vector<Field> r(nrhs,grid),z(nrhs,grid),Az(nrhs,grid);
-    std::vector< std::vector<Field> > q(mmax,std::vector<Field>(nrhs,grid));
-    std::vector< std::vector<Field> > p(mmax,std::vector<Field>(nrhs,grid));
-    std::vector<RealD> qq(mmax);
-    std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR nStep("<<nstep<<")"<<std::endl;
-    if (ZeroGuess && FirstCycle) { for(int rr=0;rr<nrhs;rr++){ psi[rr]=Zero(); r[rr]=src[rr]; } }
-    else                         { vOp(psi,Az); for(int rr=0;rr<nrhs;rr++) r[rr]=src[rr]-Az[rr]; }
-    FirstCycle=0;
-    Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
-    p[0]=z; q[0]=Az; qq[0]=zAAz; cp=vnorm2(r);
-    for(int k=0;k<nstep;k++){
-      steps++; int kp=k+1, peri_k=k%mmax, peri_kp=kp%mmax;
-      rq=vinnerProduct(q[peri_k],r); a=rq/qq[peri_k];
-      vaxpy(psi,a,p[peri_k],psi); vaxpy(r,-a,q[peri_k],r); cp=vnorm2(r);
-      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR step["<<steps<<"]  resid "<<cp<<" target "<<rsq<<std::endl;
-      if((k==nstep-1)||(cp<rsq)) return cp;
-      Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
-      q[peri_kp]=Az; p[peri_kp]=z;
-      int northog=((kp)>(mmax-1))?(mmax-1):(kp);
-      for(int back=0;back<northog;back++){ int peri_back=(k-back)%mmax; GRID_ASSERT((k-back)>=0);
-        b=-real(vinnerProduct(q[peri_back],Az))/qq[peri_back];
-        vaxpy(p[peri_kp],b,p[peri_back],p[peri_kp]); vaxpy(q[peri_kp],b,q[peri_back],q[peri_kp]); }
-      qq[peri_kp]=vnorm2(q[peri_kp]);
-    }
-    GRID_ASSERT(0); return cp;
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// L2->L3 mrhs V-cycle: a LinearFunction on the 6D mrhs COARSE field.
-// Mirrors Example_pvdagm_mrhs.cc's MrhsTwoLevelMG one level down, and the
-// single-RHS MGPreconditioner of Example_pvdagm_3level_SVDdefl.cc:
-//   out = in (trivial pre)
-//   r = in - A_coarse out
-//   restrict (unpack 6D coarse -> blockProject -> pack 6D coarse-coarse)
-//   ONE coarse-coarse solve (L3, GEMM)
-//   prolong (unpack -> blockPromote -> pack); out += correction
-//   r = in - A_coarse out
-//   coarse smoother (shifted 6D coarse op); out += smooth(r)
-//////////////////////////////////////////////////////////////////////
-template<class CoarseField, class CoarseCoarseField>
-class MrhsCoarseThreeLevelPrec : public LinearFunction<CoarseField> {
-public:
-  LinearOperatorBase<CoarseField>          &_CoarseOp;          // mrhs coarse op (6D)
-  LinearFunction<CoarseField>              &_CoarseSmoother;    // shifted 6D coarse smoother
-  MultiRHSBlockProject<CoarseField>        &_Projector;         // L2->L3 (vector-based)
-  LinearFunction<CoarseCoarseField>        &_CoarseCoarseSolve; // L3 solve (6D cc)
-  GridBase *_Coarse5d, *_CoarseCoarse5d, *_CoarseCoarseMrhs;
-  int _nrhs;
-
-  MrhsCoarseThreeLevelPrec(LinearOperatorBase<CoarseField> &CoarseOp,
-                           LinearFunction<CoarseField> &CoarseSmoother,
-                           MultiRHSBlockProject<CoarseField> &Projector,
-                           LinearFunction<CoarseCoarseField> &CoarseCoarseSolve,
-                           GridBase *Coarse5d, GridBase *CoarseCoarse5d, GridBase *CoarseCoarseMrhs, int nrhs)
-    : _CoarseOp(CoarseOp), _CoarseSmoother(CoarseSmoother), _Projector(Projector),
-      _CoarseCoarseSolve(CoarseCoarseSolve),
-      _Coarse5d(Coarse5d), _CoarseCoarse5d(CoarseCoarse5d), _CoarseCoarseMrhs(CoarseCoarseMrhs), _nrhs(nrhs) {}
-
-  using LinearFunction<CoarseField>::operator();
-  virtual void operator()(const CoarseField &in, CoarseField &out) {
-    int nrhs=_nrhs; double t;
-    CoarseField vec1(in.Grid());
-    CoarseField vec2(in.Grid());
-
-    // trivial pre-smoother
-    out = in;
-
-    // residual (6D coarse)
-    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
-
-    // restrict: unpack 6D coarse -> vector<Coarse> -> blockProject -> vector<CoarseCoarse> -> pack 6D cc
-    std::vector<CoarseField>       csplit(nrhs,_Coarse5d);
-    std::vector<CoarseCoarseField> ccsplit(nrhs,_CoarseCoarse5d);
-    CoarseCoarseField CCsrc(_CoarseCoarseMrhs);
-    CoarseCoarseField CCsol(_CoarseCoarseMrhs);
-
-    t=-usecond();
-    for(int r=0;r<nrhs;r++) ExtractSliceFast(csplit[r],vec1,r,0);
-    _Projector.blockProject(csplit,ccsplit);
-    for(int r=0;r<nrhs;r++) InsertSliceFast(ccsplit[r],CCsrc,r,0);
-    t+=usecond();
-    std::cout<<GridLogMessage<<"L2->L3 restrict took "<<t/1000.0<<"ms"<<std::endl;
-
-    // L3 solve (6D coarse-coarse, GEMM)
-    t=-usecond();
-    CCsol=Zero();
-    _CoarseCoarseSolve(CCsrc,CCsol);
-    t+=usecond();
-    std::cout<<GridLogMessage<<"L3 coarse-coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
-
-    // prolong: unpack 6D cc -> blockPromote -> pack 6D coarse; add correction
-    t=-usecond();
-    for(int r=0;r<nrhs;r++) ExtractSliceFast(ccsplit[r],CCsol,r,0);
-    _Projector.blockPromote(csplit,ccsplit);
-    for(int r=0;r<nrhs;r++) InsertSliceFast(csplit[r],vec1,r,0);
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage<<"L2->L3 prolong took "<<t/1000.0<<"ms"<<std::endl;
-
-    // residual + coarse smoother (6D coarse)
-    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
-    vec2=Zero();
-    _CoarseSmoother(vec1,vec2);
-    add(out,out,vec2);
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// L1->L2 mrhs V-cycle (verbatim from Example_pvdagm_mrhs.cc):
-// per-rhs fine smoother + batched restriction + ONE coarse solve + batched prolong.
-// The coarse solve passed in is now itself three-level (preconditioned by L2->L3).
-//////////////////////////////////////////////////////////////////////
-template<class FineField, class MrhsCoarseVector, class FineSmoother>
-class MrhsTwoLevelMG : public MrhsLinearFunction<FineField> {
-public:
-  typedef MrhsCoarseVector CoarseVector;
-  LinearOperatorBase<FineField>   &_FineOperator;
-  FineSmoother                    &_PostSmoother;
-  MultiRHSBlockProject<FineField> &_Projector;
-  LinearFunction<CoarseVector>    &_CoarseSolve;
-  GridBase *_CoarseGrid, *_CoarseGridMrhs;
-  MrhsTwoLevelMG(LinearOperatorBase<FineField> &FineOp, FineSmoother &Post,
-                 MultiRHSBlockProject<FineField> &Projector, LinearFunction<CoarseVector> &CoarseSolve,
-                 GridBase *CoarseGrid, GridBase *CoarseGridMrhs)
-    : _FineOperator(FineOp),_PostSmoother(Post),_Projector(Projector),_CoarseSolve(CoarseSolve),
-      _CoarseGrid(CoarseGrid),_CoarseGridMrhs(CoarseGridMrhs){}
-  virtual void operator()(std::vector<FineField> &in, std::vector<FineField> &out){
-    int nrhs=in.size(); GridBase *fgrid=in[0].Grid(); double t;
-    std::vector<FineField> vec1(nrhs,fgrid),vec2(nrhs,fgrid);
-    for(int r=0;r<nrhs;r++) out[r]=in[r];
-    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
-    std::vector<CoarseVector> Csrc_split(nrhs,_CoarseGrid), Csol_split(nrhs,_CoarseGrid);
-    CoarseVector CsrcMrhs(_CoarseGridMrhs), CsolMrhs(_CoarseGridMrhs);
-    t=-usecond();
-    _Projector.blockProject(vec1,Csrc_split);
-    for(int r=0;r<nrhs;r++) InsertSliceFast(Csrc_split[r],CsrcMrhs,r,0);
-    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs project+pack took "<<t/1000.0<<"ms"<<std::endl;
-    t=-usecond(); CsolMrhs=Zero(); _CoarseSolve(CsrcMrhs,CsolMrhs); t+=usecond();
-    std::cout<<GridLogMessage<<"Mrhs coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
-    t=-usecond();
-    for(int r=0;r<nrhs;r++) ExtractSliceFast(Csol_split[r],CsolMrhs,r,0);
-    _Projector.blockPromote(vec1,Csol_split);
-    for(int r=0;r<nrhs;r++) add(out[r],out[r],vec1[r]);
-    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs unpack+promote took "<<t/1000.0<<"ms"<<std::endl;
-    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
-    t=-usecond();
-    for(int r=0;r<nrhs;r++){ vec2[r]=Zero(); _PostSmoother(vec1[r],vec2[r]); add(out[r],out[r],vec2[r]); }
-    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs post-smooth took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-  ParseEnvironment();
-
-  const int Ls=24; RealD M5=1.8, b=1.5, c=0.5;
-  const int nbasis=60; const int nrhs=Nrhs;
-  GRID_ASSERT(nrhs % vComplex::Nsimd() == 0);
-
-  std::vector<int> lat_size {48,48,48,96};
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Level 1 blocking (default 2^4)
-  Coordinate clatt = lat_size;
-  Coordinate Block({2,2,2,2});
-  if ( getenv("BLOCK") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block); GRID_ASSERT(Block.size()==4); }
-  for(int d=0;d<4;d++){ GRID_ASSERT(lat_size[d]%Block[d]==0); clatt[d]=lat_size[d]/Block[d]; }
-  std::cout << GridLogMessage << "Block  " << Block  << "  coarse lattice       " << clatt << std::endl;
-
-  // Level 2 blocking (default 2,2,3,3)  -- matches Example_pvdagm_3level_SVDdefl
-  Coordinate cclatt = clatt;
-  Coordinate Block2({2,2,3,3});
-  if ( getenv("BLOCK2") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK2")),Block2); GRID_ASSERT(Block2.size()==4); }
-  for(int d=0;d<4;d++){ GRID_ASSERT(clatt[d]%Block2[d]==0); cclatt[d]=clatt[d]/Block2[d]; }
-  std::cout << GridLogMessage << "Block2 " << Block2 << "  coarse-coarse lattice " << cclatt << std::endl;
-
-  GridCartesian *Coarse4d = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(cclatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
-
-  // 6D mrhs grids: rhs is dim 0, SIMD across rhs (pattern: Test_general_coarse_hdcg_phys48.cc)
-  Coordinate mpi=GridDefaultMpi();
-  Coordinate rhMpi ({1,1,mpi[0],mpi[1],mpi[2],mpi[3]});
-  Coordinate rhSimd({vComplex::Nsimd(),1,1,1,1,1});
-  Coordinate rhLatt ({nrhs,1,clatt[0], clatt[1], clatt[2], clatt[3]});
-  Coordinate rhLatt2({nrhs,1,cclatt[0],cclatt[1],cclatt[2],cclatt[3]});
-  GridCartesian *CoarseMrhs       = new GridCartesian(rhLatt, rhSimd,rhMpi);
-  GridCartesian *CoarseCoarseMrhs = new GridCartesian(rhLatt2,rhSimd,rhMpi);
-
-  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-
-  // Level 1 tensor types
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>       LittleDiracOperator;
-  typedef MultiGeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  MrhsLittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                                CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>                  Subspace;
-
-  // Level 2 tensor types (coarsening deepens the nest by one iScalar -- see CLAUDE.md)
-  typedef CoarseVector::vector_object                                      CoarseSiteObj;
-  typedef iScalar<vTComplex>                                              vTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>          LittleDiracOperatorL2;
-  typedef MultiGeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>     MrhsLittleDiracOperatorL2;
-  typedef LittleDiracOperatorL2::CoarseVector                              CoarseCoarseVector;
-  typedef Aggregation<CoarseSiteObj,vTTComplex,nbasis>                     SubspaceL2;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom (Coarse5d);
-  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);   // 33-point at L2->L3, matching SVDdefl
-
-  //////////////////////////////////////////////////////////////////////
-  // Subspace: load RAW (no Orthogonalise!), or generate.
-  //////////////////////////////////////////////////////////////////////
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                            + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-  uint64_t file_exists=0;
-  if ( UGrid->IsBoss() ){ std::ifstream f(subspace_file); file_exists=f.good()?1:0; }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb=0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-  if ( file_exists ){
-    std::cout << GridLogMessage << "*** Loading subspace from disk (kept RAW) ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-  }
-
-  // RAW copy of the fine null vectors BEFORE CoarsenOperator block-orthonormalises in place.
-  std::vector<LatticeFermionD> rawNull(nbasis,FGrid);
-  for(int k=0;k<nbasis;k++) rawNull[k]=AggregatesGCR.subspace[k];
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarsen L1->L2 and L2->L3 with SINGLE-RHS machinery, import into the mrhs
-  // operators via CopyMatrix.  The single-RHS L1->L2 coarse operator must stay
-  // alive to be the "fine" operator for the L2->L3 coarsening, so BOTH single-RHS
-  // ops (and their padded _A) live in one scope and free together.  [MEMORY: this
-  // is the setup peak -- L1->L2 padded _A (~large at 2^4) + L2->L3 padded _A.]
-  //////////////////////////////////////////////////////////////////////
-  MrhsLittleDiracOperator   mrhsLittleDiracOpPV(geom,  CoarseMrhs);
-  MrhsLittleDiracOperatorL2 mrhsLittleDiracOpL2(geom2, CoarseCoarseMrhs);
-  MultiRHSBlockProject<LatticeFermionD> MrhsProjector;
-  MultiRHSBlockProject<CoarseVector>    MrhsProjectorL2;
-  {
-    // --- L1->L2 single-RHS coarse operator (kept alive for the L2->L3 coarsening) ---
-    LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d);
-    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);   // orthonormalises AggregatesGCR.subspace in place
-    mrhsLittleDiracOpPV.CopyMatrix(LittleDiracOpPV);
-    MrhsProjector.Allocate(nbasis,FGrid,Coarse5d);
-    MrhsProjector.ImportBasis(AggregatesGCR.subspace);       // orthonormalised, matches the coarse op
-    NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-
-    // --- psi_coarse = P^dag (RAW fine null) -> Galerkin images, NOT e_k ---
-    std::vector<CoarseVector> psi_coarse(nbasis,Coarse5d);
-    for(int k=0;k<nbasis;k++) AggregatesGCR.ProjectToSubspace(psi_coarse[k], rawNull[k]);
-    rawNull.clear(); rawNull.shrink_to_fit();
-    {
-      RealD s2=0.0;
-      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
-        ComplexD sij=TensorRemove(innerProduct(psi_coarse[i],psi_coarse[j]));
-        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
-      }
-      std::cout<<GridLogMessage<<"GUARD: ||<psi_coarse|psi_coarse> - I||_F = "<<std::sqrt(s2)
-               <<"   (~0.23 good; ~sqrt(N_coarse)="<<std::sqrt((double)Coarse5d->gSites())<<" = e_k leak)"<<std::endl;
-    }
-
-    // --- L2->L3 single-RHS coarsening (coarsen the single-RHS LinOpCoarse) ---
-    SubspaceL2 AggregatesL2(CoarseCoarse5d,Coarse5d,cb);
-    for(int k=0;k<nbasis;k++) AggregatesL2.subspace[k]=psi_coarse[k];
-    LittleDiracOperatorL2 LittleDiracOpL2(geom2,Coarse5d,CoarseCoarse5d);
-    LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
-    mrhsLittleDiracOpL2.CopyMatrix(LittleDiracOpL2);
-    MrhsProjectorL2.Allocate(nbasis,Coarse5d,CoarseCoarse5d);
-    MrhsProjectorL2.ImportBasis(AggregatesL2.subspace);      // orthonormalised by CoarsenOperator
-
-    // --- guard psi_cc (RAW psi_coarse; AggregatesL2 holds a separate orthonormalised copy) ---
-    {
-      std::vector<CoarseCoarseVector> psi_cc(nbasis,CoarseCoarse5d);
-      for(int k=0;k<nbasis;k++) AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
-      RealD s2=0.0;
-      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
-        ComplexD sij=TensorRemove(innerProduct(psi_cc[i],psi_cc[j]));
-        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
-      }
-      std::cout<<GridLogMessage<<"GUARD: ||<psi_cc|psi_cc> - I||_F = "<<std::sqrt(s2)
-               <<"   (~0.23 good; ~sqrt(N_cc)="<<std::sqrt((double)CoarseCoarse5d->gSites())<<" = e_k leak)"<<std::endl;
-    }
-  } // both single-RHS ops + padded _A + AggregatesL2 + psi_coarse freed here
-
-  NonHermitianLinearOperator<MrhsLittleDiracOperator,CoarseVector>         mrhsLinOpCoarse(mrhsLittleDiracOpPV);
-  NonHermitianLinearOperator<MrhsLittleDiracOperatorL2,CoarseCoarseVector> mrhsLinOpCC(mrhsLittleDiracOpL2);
-
-  //////////////////////////////////////////////////////////////////////
-  // Solvers, innermost first.
-  //////////////////////////////////////////////////////////////////////
-  TrivialPrecon<CoarseVector>       simpleC;
-  TrivialPrecon<CoarseCoarseVector> simpleCC;
-  TrivialPrecon<LatticeFermionD>    simple_fine;
-
-  // L3 (coarse-coarse) solve: PGCR on the 6D cc operator
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-    L3PGCR(L3Tol,L3MaxIt,mrhsLinOpCC,simpleCC,L3Nstep,L3Nstep);
-  L3PGCR.Level(3);
-  L3PGCR.Name("CCouter");
-  L3PGCR.SetZeroGuess(1);              // caller zeroes CCsol
-
-  // L2 coarse smoother: shifted 6D coarse op, fixed nstep
-  ShiftedLinearOperator<CoarseVector> ShiftedMrhsCoarse(CoarseSmootherShift, mrhsLinOpCoarse);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
-    CoarseSmootherGCR(0.01,1,ShiftedMrhsCoarse,simpleC,CoarseSmootherNstep,CoarseSmootherNstep);
-  CoarseSmootherGCR.Level(2);
-  CoarseSmootherGCR.Name("Csmoother");
-  CoarseSmootherGCR.SetZeroGuess(1);   // caller zeroes vec2
-
-  // L2->L3 V-cycle preconditioner (operates on 6D coarse field)
-  MrhsCoarseThreeLevelPrec<CoarseVector,CoarseCoarseVector>
-    L2to3Precon(mrhsLinOpCoarse, CoarseSmootherGCR, MrhsProjectorL2, L3PGCR,
-                Coarse5d, CoarseCoarse5d, CoarseCoarseMrhs, nrhs);
-
-  // L2 coarse solve: PGCR on 6D coarse op, preconditioned by the L2->L3 V-cycle
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
-    L2PGCR(CoarseSolverTol, CoarseSolverOrder/16, mrhsLinOpCoarse, L2to3Precon, 16, 16);
-  L2PGCR.Level(2);
-  L2PGCR.Name("Couter");
-  L2PGCR.SetZeroGuess(1);              // caller zeroes CsolMrhs
-
-  // Fine smoother (per-rhs, looped in the L1->L2 V-cycle)
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD>
-    SmootherGCR(0.0,1,ShiftedPVdagM,simple_fine,FineSmootherOrder,FineSmootherOrder);
-  SmootherGCR.Level(1);
-  SmootherGCR.Name("Fsmoother");
-  SmootherGCR.SetZeroGuess(1);         // caller zeroes vec2[r]
-
-  // L1->L2 V-cycle (fine); its coarse solve is the three-level L2PGCR
-  typedef PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> FineSmoother_t;
-  MrhsTwoLevelMG<LatticeFermionD,CoarseVector,FineSmoother_t>
-    ThreeLevelPrecon(PVdagM, SmootherGCR, MrhsProjector, L2PGCR, Coarse5d, CoarseMrhs);
-
-  // Outer mrhs solve
-  MrhsPGCRNonHermitian<LatticeFermionD>
-    L1PGCR(OuterTol,1000,PVdagM,ThreeLevelPrecon,OuterMmax,OuterNstep);
-  L1PGCR.Level(1);
-  L1PGCR.Name("Fouter");
-  L1PGCR.SetZeroGuess(1);              // sol[r]=Zero() at source setup
-
-  //////////////////////////////////////////////////////////////////////
-  // Sources and solve
-  //////////////////////////////////////////////////////////////////////
-  std::vector<LatticeFermionD> src(nrhs,FGrid), sol(nrhs,FGrid);
-  for(int r=0;r<nrhs;r++){ gaussian(RNG5,src[r]); sol[r]=Zero(); }
-
-  std::cout << GridLogMessage << "**********************************************" << std::endl;
-  std::cout << GridLogMessage << " MultiRHS THREE-level solve: " << nrhs << " RHS " << std::endl;
-  std::cout << GridLogMessage << "**********************************************" << std::endl;
-
-  GridStopWatch w; w.Start();
-  L1PGCR(src,sol);
-  w.Stop();
-  std::cout << GridLogMessage << "MultiRHS 3-level solve total " << w.Elapsed()
-            << "  (per RHS: " << w.useconds()/1.0e6/nrhs << " s)" << std::endl;
-
-  { LatticeFermionD Ax(FGrid); RealD worst=0.0;
-    for(int r=0;r<nrhs;r++){ PVdagM.Op(sol[r],Ax); Ax=Ax-src[r];
-      RealD rn=std::sqrt(norm2(Ax)/norm2(src[r]));
-      std::cout << GridLogMessage << "FINAL: rhs["<<r<<"] true residual = " << rn << std::endl;
-      worst=std::max(worst,rn); }
-    std::cout << GridLogMessage << "FINAL: worst-case residual = " << worst << std::endl;
-  }
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_mrhs_3level_DenseCoarseMatrix.cc

@@ -1,663 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_pvdagm_mrhs_3level_DenseCoarseMatrix.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// MultiRHS (valence) THREE-level multigrid for PVdagM with a DENSE, EXACT,
-// non-iterative coarse-coarse bottom -- the LIBRARY-CLASS successor of
-// Example_pvdagm_mrhs_3level_dense.cc, which is FROZEN as the regression
-// baseline / champion-provenance artifact (21.7 s/RHS at BLOCK=2.2.3.3,
-// BLOCK2=8.4.2.4, nb60, CSO3/FSO6/CST0.04 on 36 Frontier nodes).
-//
-// The dense bottom is now Grid/algorithms/multigrid/DenseCoarseMatrix.h:
-//  - stencil -> dense DIRECT import (no probe assembly: rows are local data)
-//    + IMPORT CERTIFICATE  (DENSE_IMPORT_SIGN=-1 flips convention, no rebuild)
-//  - split-K apply via GridBLAS.gemmBatched with explicit leading dimensions
-//    (DENSE_SPLITK chunks, default 32) -- the fig-11 software split-K
-//  - deviceVector / GridBLAS throughout the apply: platform-agnostic
-//
-// INTERCHANGE: same SLAB_FILE per-rank format as the frozen example (stem MUST
-// encode cfg/mass/blocking/nbasis; the header guards only N/nrows/nbasis) and
-// the same env-var set, so existing sbatch scripts drive either binary.
-//
-// A/B acceptance (old binary = control):
-//   slab-cached  : outer counts match EXACTLY (identical apply data; split-K
-//                  changes only fp32 reduction order); wall delta = split-K gain.
-//   fresh setup  : outer equal-or-+-1 (import vs probe = rounding); VERIFY
-//                  ~7e-4 both; setup delta = import gain (~93 s probe retired).
-//
-// Level structure, solvers, and tuning knobs are UNCHANGED from the frozen
-// example.  Env: MASS SUBSPACE_FILE NRHS BLOCK BLOCK2 FineSmootherShift/Order
-// CoarseSmootherShift/Nstep CoarseSolverTol/Order DENSE_CC DENSE_CC_CHECK
-// DENSE_SPLITK DENSE_DEVICE_SUM DENSE_IMPORT_SIGN DENSE_APPLY_PROFILE
-// L3_TOL L3_MAXIT L3_NSTEP OuterMmax OuterNstep OuterTol
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/multigrid/DenseCoarseMatrix.h>
-
-#include <memory>
-
-using namespace std;
-using namespace Grid;
-
-RealD FineSmootherShift    = 0.1;
-int   FineSmootherOrder    = 16;
-RealD CoarseSmootherShift  = 0.1;
-int   CoarseSmootherNstep  = 4;
-RealD CoarseSolverTol      = 0.03;
-int   CoarseSolverOrder    = 200;
-RealD L3Tol                = 2.5e-1;
-int   L3MaxIt              = 50;
-int   L3Nstep              = 50;
-RealD OuterTol             = 1.0e-8;
-int   OuterMmax            = 8;
-int   OuterNstep           = 8;
-int   Nrhs                 = 12;
-int   UseDenseCC           = 1;
-RealD mass                 = 0.00078;
-
-void ParseEnvironment(void)
-{
-  if(getenv("MASS"))               mass               = atof(getenv("MASS"));
-  if(getenv("FineSmootherShift"))  FineSmootherShift  = atof(getenv("FineSmootherShift"));
-  if(getenv("FineSmootherOrder"))  FineSmootherOrder  = atoi(getenv("FineSmootherOrder"));
-  if(getenv("CoarseSmootherShift"))CoarseSmootherShift= atof(getenv("CoarseSmootherShift"));
-  if(getenv("CoarseSmootherNstep"))CoarseSmootherNstep= atoi(getenv("CoarseSmootherNstep"));
-  if(getenv("CoarseSolverTol"))    CoarseSolverTol    = atof(getenv("CoarseSolverTol"));
-  if(getenv("CoarseSolverOrder"))  CoarseSolverOrder  = atoi(getenv("CoarseSolverOrder"));
-  if(getenv("L3_TOL"))             L3Tol              = atof(getenv("L3_TOL"));
-  if(getenv("L3_MAXIT"))           L3MaxIt            = atoi(getenv("L3_MAXIT"));
-  if(getenv("L3_NSTEP"))           L3Nstep            = atoi(getenv("L3_NSTEP"));
-  if(getenv("OuterTol"))           OuterTol           = atof(getenv("OuterTol"));
-  if(getenv("OuterMmax"))          OuterMmax          = atoi(getenv("OuterMmax"));
-  if(getenv("OuterNstep"))         OuterNstep         = atoi(getenv("OuterNstep"));
-  if(getenv("NRHS"))               Nrhs               = atoi(getenv("NRHS"));
-  if(getenv("DENSE_CC"))           UseDenseCC         = atoi(getenv("DENSE_CC"));
-
-  std::cout << GridLogMessage << "PARAM: MASS               " << mass               << std::endl;
-  std::cout << GridLogMessage << "PARAM: NRHS               " << Nrhs               << std::endl;
-  std::cout << GridLogMessage << "PARAM: DENSE_CC           " << UseDenseCC         << std::endl;
-  std::cout << GridLogMessage << "PARAM: FineSmootherShift  " << FineSmootherShift  << std::endl;
-  std::cout << GridLogMessage << "PARAM: FineSmootherOrder  " << FineSmootherOrder  << std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift" << CoarseSmootherShift<< std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSmootherNstep" << CoarseSmootherNstep<< std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverTol    " << CoarseSolverTol    << std::endl;
-  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder  " << CoarseSolverOrder  << std::endl;
-  std::cout << GridLogMessage << "PARAM: L3_TOL             " << L3Tol              << std::endl;
-  std::cout << GridLogMessage << "PARAM: OuterMmax          " << OuterMmax          << std::endl;
-  std::cout << GridLogMessage << "PARAM: OuterNstep         " << OuterNstep         << std::endl;
-}
-
-template <class Field>
-void saveSubspace(std::vector<Field> &subspace, std::string const fname){
-#ifdef HAVE_LIME
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
-  SW.open(fname);
-  for (int k = 0; k < (int)subspace.size(); k++) SW.writeScidacFieldRecord(subspace[k], record);
-  SW.close();
-#endif
-}
-template <class Field>
-void loadSubspace(std::vector<Field> &subspace, std::string const fname){
-#ifdef HAVE_LIME
-  Grid::emptyUserRecord record;
-  Grid::ScidacReader SR;
-  SR.open(fname);
-  for (int k = 0; k < (int)subspace.size(); k++) SR.readScidacFieldRecord(subspace[k], record);
-  SR.close();
-#endif
-}
-
-//////////////////////////////////////////////////////////////////////
-// A = PV^dag M (non-Hermitian), and shifted variant for smoothers.
-//////////////////////////////////////////////////////////////////////
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat; Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
-  void OpDiag (const Field &in, Field &out) { assert(0); }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
-  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); }
-  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(in,tmp); _Mat.Mdag(tmp,out); }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ HermOp(in,out); ComplexD d=innerProduct(in,out); n1=real(d); n2=norm2(out); }
-  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat; Matrix &_PV;
-public:
-  RealD shift;
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-  void OpDiag (const Field &in, Field &out) { assert(0); }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
-  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); out = out + shift*in; }
-  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(tmp,out); _Mat.Mdag(in,tmp); out = out + shift*in; }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-// Generic shift wrapper (for the coarse-level smoother on the 6D mrhs coarse operator).
-template<class Field>
-class ShiftedLinearOperator : public LinearOperatorBase<Field> {
-  LinearOperatorBase<Field> &_Op; RealD shift;
-public:
-  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : _Op(Op), shift(_shift) {}
-  void OpDiag  (const Field &in, Field &out) { assert(0); }
-  void OpDir   (const Field &in, Field &out,int dir,int disp) { assert(0); }
-  void OpDirAll (const Field &in, std::vector<Field> &out) { assert(0); }
-  void Op      (const Field &in, Field &out) { _Op.Op(in,out);    out = out + shift*in; }
-  void AdjOp   (const Field &in, Field &out) { _Op.AdjOp(in,out); out = out + shift*in; }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
-  void HermOp  (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
-};
-
-//////////////////////////////////////////////////////////////////////
-// Dense CC solve on the PACKED 6D mrhs coarse-coarse field: drop-in
-// for the L3 PGCR, delegating to the library DenseCoarseMatrix.
-//////////////////////////////////////////////////////////////////////
-template<class DenseType, class CoarseCoarseField>
-class MrhsDenseCCSolve : public LinearFunction<CoarseCoarseField> {
-public:
-  DenseType &_Dense;
-  GridBase *_CoarseCoarse5d;
-  int _nrhs;
-  MrhsDenseCCSolve(DenseType &D, GridBase *cc5d, int nrhs)
-    : _Dense(D), _CoarseCoarse5d(cc5d), _nrhs(nrhs) {}
-  using LinearFunction<CoarseCoarseField>::operator();
-  virtual void operator()(const CoarseCoarseField &in, CoarseCoarseField &out){
-    if ( getenv("DENSE_CC_CHECK") ) {
-      // Audit path: per-rhs 5D unpack so ApplyBatch can run the _Op defect
-      // check per rhs.  ~50ms/call of slice/split overhead -- audit only.
-      CoarseCoarseField tmp(in.Grid());
-      tmp = in;
-      std::vector<CoarseCoarseField> split_in (_nrhs,_CoarseCoarse5d);
-      std::vector<CoarseCoarseField> split_out(_nrhs,_CoarseCoarse5d);
-      for(int r=0;r<_nrhs;r++) ExtractSliceFast(split_in[r], tmp, r, 0);
-      _Dense.ApplyBatch(split_in, split_out);
-      for(int r=0;r<_nrhs;r++) InsertSliceFast(split_out[r], out, r, 0);
-    } else {
-      _Dense.ApplyBatch6D(in, out, _nrhs);
-    }
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// mrhs interfaces + single-polynomial mrhs PGCR (verbatim from the
-// frozen Example_pvdagm_mrhs_3level_dense.cc)
-//////////////////////////////////////////////////////////////////////
-template<class Field>
-class MrhsLinearFunction {
-public:
-  virtual void operator()(std::vector<Field> &in, std::vector<Field> &out) = 0;
-};
-
-template<class Field>
-class MrhsPGCRNonHermitian {
-public:
-  RealD Tolerance; Integer MaxIterations; int mmax,nstep,steps,level;
-  int ZeroGuess = 0; int FirstCycle = 0;   // caller contract: zero guess => first-cycle r0 = src
-  std::string name = "Level 1";
-  LinearOperatorBase<Field> &Linop;
-  MrhsLinearFunction<Field> &Preconditioner;
-  void Level(int lv){ name = "Level " + std::to_string(lv); level=lv; }
-  void Name(std::string n){ name = n; }
-  void SetZeroGuess(int z){ ZeroGuess=z; }
-  MrhsPGCRNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,MrhsLinearFunction<Field> &Prec,int _mmax,int _nstep)
-    : Tolerance(tol),MaxIterations(maxit),Linop(_Linop),Preconditioner(Prec),mmax(_mmax),nstep(_nstep){ level=1; }
-  static RealD vnorm2(std::vector<Field> &x){ RealD s=0; for(auto &f:x) s+=norm2(f); return s; }
-  static ComplexD vinnerProduct(std::vector<Field> &x,std::vector<Field> &y){ ComplexD s(0); for(int r=0;r<(int)x.size();r++) s+=innerProduct(x[r],y[r]); return s; }
-  static void vaxpy(std::vector<Field> &z,ComplexD a,std::vector<Field> &x,std::vector<Field> &y){ for(int r=0;r<(int)z.size();r++) axpy(z[r],a,x[r],y[r]); }
-  void vOp(std::vector<Field> &in,std::vector<Field> &out){ for(int r=0;r<(int)in.size();r++) Linop.Op(in[r],out[r]); }
-  void operator()(std::vector<Field> &src,std::vector<Field> &psi){
-    RealD cp,ssq,rsq; int nrhs=src.size(); GridBase *grid=src[0].Grid();
-    ssq=vnorm2(src); rsq=Tolerance*Tolerance*ssq;
-    std::vector<Field> r(nrhs,grid);
-    GridStopWatch T; T.Start(); steps=0; FirstCycle=1;
-    for(int k=0;k<MaxIterations;k++){
-      cp=GCRnStep(src,psi,rsq);
-      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR("<<mmax<<","<<nstep<<") "<<steps<<" steps cp = "<<cp<<" target "<<rsq<<std::endl;
-      if(cp<rsq){
-        T.Stop(); vOp(psi,r); for(int rr=0;rr<nrhs;rr++) axpy(r[rr],-1.0,src[rr],r[rr]);
-        RealD tr=vnorm2(r);
-        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR: Converged on iteration "<<steps
-                 <<" computed residual "<<std::sqrt(cp/ssq)<<" true residual "<<std::sqrt(tr/ssq)<<" target "<<Tolerance<<std::endl;
-        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR Time elapsed: Total "<<T.Elapsed()<<std::endl;
-        for(int rr=0;rr<nrhs;rr++){ RealD rn=std::sqrt(norm2(r[rr])/norm2(src[rr])); std::cout<<GridLogMessage<<"MrhsPGCR per-rhs true residual["<<rr<<"] = "<<rn<<std::endl; }
-        return;
-      }
-    }
-    std::cout<<GridLogMessage<<"MrhsPGCR: did not converge"<<std::endl;
-  }
-  RealD GCRnStep(std::vector<Field> &src,std::vector<Field> &psi,RealD rsq){
-    RealD cp; ComplexD a,b,rq; RealD zAAz; int nrhs=src.size(); GridBase *grid=src[0].Grid();
-    std::vector<Field> r(nrhs,grid),z(nrhs,grid),Az(nrhs,grid);
-    std::vector< std::vector<Field> > q(mmax,std::vector<Field>(nrhs,grid));
-    std::vector< std::vector<Field> > p(mmax,std::vector<Field>(nrhs,grid));
-    std::vector<RealD> qq(mmax);
-    std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR nStep("<<nstep<<")"<<std::endl;
-    if (ZeroGuess && FirstCycle) { for(int rr=0;rr<nrhs;rr++){ psi[rr]=Zero(); r[rr]=src[rr]; } }
-    else                         { vOp(psi,Az); for(int rr=0;rr<nrhs;rr++) r[rr]=src[rr]-Az[rr]; }
-    FirstCycle=0;
-    Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
-    p[0]=z; q[0]=Az; qq[0]=zAAz; cp=vnorm2(r);
-    for(int k=0;k<nstep;k++){
-      steps++; int kp=k+1, peri_k=k%mmax, peri_kp=kp%mmax;
-      rq=vinnerProduct(q[peri_k],r); a=rq/qq[peri_k];
-      vaxpy(psi,a,p[peri_k],psi); vaxpy(r,-a,q[peri_k],r); cp=vnorm2(r);
-      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR step["<<steps<<"]  resid "<<cp<<" target "<<rsq<<std::endl;
-      if((k==nstep-1)||(cp<rsq)) return cp;
-      Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
-      q[peri_kp]=Az; p[peri_kp]=z;
-      int northog=((kp)>(mmax-1))?(mmax-1):(kp);
-      for(int back=0;back<northog;back++){ int peri_back=(k-back)%mmax; GRID_ASSERT((k-back)>=0);
-        b=-real(vinnerProduct(q[peri_back],Az))/qq[peri_back];
-        vaxpy(p[peri_kp],b,p[peri_back],p[peri_kp]); vaxpy(q[peri_kp],b,q[peri_back],q[peri_kp]); }
-      qq[peri_kp]=vnorm2(q[peri_kp]);
-    }
-    GRID_ASSERT(0); return cp;
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// L2->L3 mrhs V-cycle: LinearFunction on the 6D mrhs COARSE field.
-// The coarse-coarse solve slot takes EITHER the dense mrhs solve
-// (DENSE_CC=1) or the L3 PGCR (DENSE_CC=0).
-//////////////////////////////////////////////////////////////////////
-template<class CoarseField, class CoarseCoarseField>
-class MrhsCoarseThreeLevelPrec : public LinearFunction<CoarseField> {
-public:
-  LinearOperatorBase<CoarseField>          &_CoarseOp;          // mrhs coarse op (6D)
-  LinearFunction<CoarseField>              &_CoarseSmoother;    // shifted 6D coarse smoother
-  MultiRHSBlockProject<CoarseField>        &_Projector;         // L2->L3 (vector-based)
-  LinearFunction<CoarseCoarseField>        &_CoarseCoarseSolve; // L3 solve (6D cc)
-  GridBase *_Coarse5d, *_CoarseCoarse5d, *_CoarseCoarseMrhs;
-  int _nrhs;
-
-  MrhsCoarseThreeLevelPrec(LinearOperatorBase<CoarseField> &CoarseOp,
-                           LinearFunction<CoarseField> &CoarseSmoother,
-                           MultiRHSBlockProject<CoarseField> &Projector,
-                           LinearFunction<CoarseCoarseField> &CoarseCoarseSolve,
-                           GridBase *Coarse5d, GridBase *CoarseCoarse5d, GridBase *CoarseCoarseMrhs, int nrhs)
-    : _CoarseOp(CoarseOp), _CoarseSmoother(CoarseSmoother), _Projector(Projector),
-      _CoarseCoarseSolve(CoarseCoarseSolve),
-      _Coarse5d(Coarse5d), _CoarseCoarse5d(CoarseCoarse5d), _CoarseCoarseMrhs(CoarseCoarseMrhs), _nrhs(nrhs) {}
-
-  using LinearFunction<CoarseField>::operator();
-  virtual void operator()(const CoarseField &in, CoarseField &out) {
-    int nrhs=_nrhs; double t;
-    CoarseField vec1(in.Grid());
-    CoarseField vec2(in.Grid());
-
-    // trivial pre-smoother
-    out = in;
-
-    // residual (6D coarse)
-    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
-
-    // restrict: unpack 6D coarse -> vector<Coarse> -> blockProject -> vector<CoarseCoarse> -> pack 6D cc
-    std::vector<CoarseField>       csplit(nrhs,_Coarse5d);
-    std::vector<CoarseCoarseField> ccsplit(nrhs,_CoarseCoarse5d);
-    CoarseCoarseField CCsrc(_CoarseCoarseMrhs);
-    CoarseCoarseField CCsol(_CoarseCoarseMrhs);
-
-    t=-usecond();
-    for(int r=0;r<nrhs;r++) ExtractSliceFast(csplit[r],vec1,r,0);
-    _Projector.blockProject(csplit,ccsplit);
-    for(int r=0;r<nrhs;r++) InsertSliceFast(ccsplit[r],CCsrc,r,0);
-    t+=usecond();
-    std::cout<<GridLogMessage<<"L2->L3 restrict took "<<t/1000.0<<"ms"<<std::endl;
-
-    // L3 solve (dense mrhs GEMM, or PGCR)
-    t=-usecond();
-    CCsol=Zero();
-    _CoarseCoarseSolve(CCsrc,CCsol);
-    t+=usecond();
-    std::cout<<GridLogMessage<<"L3 coarse-coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
-
-    // prolong: unpack 6D cc -> blockPromote -> pack 6D coarse; add correction
-    t=-usecond();
-    for(int r=0;r<nrhs;r++) ExtractSliceFast(ccsplit[r],CCsol,r,0);
-    _Projector.blockPromote(csplit,ccsplit);
-    for(int r=0;r<nrhs;r++) InsertSliceFast(csplit[r],vec1,r,0);
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage<<"L2->L3 prolong took "<<t/1000.0<<"ms"<<std::endl;
-
-    // residual + coarse smoother (6D coarse)
-    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
-    vec2=Zero();
-    _CoarseSmoother(vec1,vec2);
-    add(out,out,vec2);
-  }
-};
-
-//////////////////////////////////////////////////////////////////////
-// L1->L2 mrhs V-cycle (verbatim from the frozen example)
-//////////////////////////////////////////////////////////////////////
-template<class FineField, class MrhsCoarseVector, class FineSmoother>
-class MrhsTwoLevelMG : public MrhsLinearFunction<FineField> {
-public:
-  typedef MrhsCoarseVector CoarseVector;
-  LinearOperatorBase<FineField>   &_FineOperator;
-  FineSmoother                    &_PostSmoother;
-  MultiRHSBlockProject<FineField> &_Projector;
-  LinearFunction<CoarseVector>    &_CoarseSolve;
-  GridBase *_CoarseGrid, *_CoarseGridMrhs;
-  MrhsTwoLevelMG(LinearOperatorBase<FineField> &FineOp, FineSmoother &Post,
-                 MultiRHSBlockProject<FineField> &Projector, LinearFunction<CoarseVector> &CoarseSolve,
-                 GridBase *CoarseGrid, GridBase *CoarseGridMrhs)
-    : _FineOperator(FineOp),_PostSmoother(Post),_Projector(Projector),_CoarseSolve(CoarseSolve),
-      _CoarseGrid(CoarseGrid),_CoarseGridMrhs(CoarseGridMrhs){}
-  virtual void operator()(std::vector<FineField> &in, std::vector<FineField> &out){
-    int nrhs=in.size(); GridBase *fgrid=in[0].Grid(); double t;
-    std::vector<FineField> vec1(nrhs,fgrid),vec2(nrhs,fgrid);
-    for(int r=0;r<nrhs;r++) out[r]=in[r];
-    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
-    std::vector<CoarseVector> Csrc_split(nrhs,_CoarseGrid), Csol_split(nrhs,_CoarseGrid);
-    CoarseVector CsrcMrhs(_CoarseGridMrhs), CsolMrhs(_CoarseGridMrhs);
-    t=-usecond();
-    _Projector.blockProject(vec1,Csrc_split);
-    for(int r=0;r<nrhs;r++) InsertSliceFast(Csrc_split[r],CsrcMrhs,r,0);
-    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs project+pack took "<<t/1000.0<<"ms"<<std::endl;
-    t=-usecond(); CsolMrhs=Zero(); _CoarseSolve(CsrcMrhs,CsolMrhs); t+=usecond();
-    std::cout<<GridLogMessage<<"Mrhs coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
-    t=-usecond();
-    for(int r=0;r<nrhs;r++) ExtractSliceFast(Csol_split[r],CsolMrhs,r,0);
-    _Projector.blockPromote(vec1,Csol_split);
-    for(int r=0;r<nrhs;r++) add(out[r],out[r],vec1[r]);
-    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs unpack+promote took "<<t/1000.0<<"ms"<<std::endl;
-    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
-    t=-usecond();
-    for(int r=0;r<nrhs;r++){ vec2[r]=Zero(); _PostSmoother(vec1[r],vec2[r]); add(out[r],out[r],vec2[r]); }
-    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs post-smooth took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-  ParseEnvironment();
-
-  const int Ls=24; RealD M5=1.8, b=1.5, c=0.5;
-  const int nbasis=60; const int nrhs=Nrhs;
-  GRID_ASSERT(nrhs % vComplex::Nsimd() == 0);
-
-  std::vector<int> lat_size {48,48,48,96};
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Level 1 blocking (default 2^4)
-  Coordinate clatt = lat_size;
-  Coordinate Block({2,2,2,2});
-  if ( getenv("BLOCK") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block); GRID_ASSERT(Block.size()==4); }
-  for(int d=0;d<4;d++){ GRID_ASSERT(lat_size[d]%Block[d]==0); clatt[d]=lat_size[d]/Block[d]; }
-  std::cout << GridLogMessage << "Block  " << Block  << "  coarse lattice       " << clatt << std::endl;
-
-  // Level 2 blocking: SUPERCOARSE default 8,4,3,6 -> CC [3,6,8,8], the dense floor.
-  Coordinate cclatt = clatt;
-  Coordinate Block2({8,4,3,6});
-  if ( getenv("BLOCK2") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK2")),Block2); GRID_ASSERT(Block2.size()==4); }
-  for(int d=0;d<4;d++){ GRID_ASSERT(clatt[d]%Block2[d]==0); cclatt[d]=clatt[d]/Block2[d]; }
-  std::cout << GridLogMessage << "Block2 " << Block2 << "  coarse-coarse lattice " << cclatt << std::endl;
-
-  GridCartesian *Coarse4d = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *Coarse5d = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(cclatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
-
-  // 6D mrhs grids: rhs is dim 0, SIMD across rhs
-  Coordinate mpi=GridDefaultMpi();
-  Coordinate rhMpi ({1,1,mpi[0],mpi[1],mpi[2],mpi[3]});
-  Coordinate rhSimd({vComplex::Nsimd(),1,1,1,1,1});
-  Coordinate rhLatt ({nrhs,1,clatt[0], clatt[1], clatt[2], clatt[3]});
-  Coordinate rhLatt2({nrhs,1,cclatt[0],cclatt[1],cclatt[2],cclatt[3]});
-  GridCartesian *CoarseMrhs       = new GridCartesian(rhLatt, rhSimd,rhMpi);
-  GridCartesian *CoarseCoarseMrhs = new GridCartesian(rhLatt2,rhSimd,rhMpi);
-
-  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
-
-  LatticeGaugeField Umu(UGrid);
-  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
-  FieldMetaData header;
-  std::string file("/ccs/home/poare/ckpoint_lat.1000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
-  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
-
-  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
-
-  // Level 1 tensor types
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>       LittleDiracOperator;
-  typedef MultiGeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  MrhsLittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector                                CoarseVector;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>                  Subspace;
-
-  // Level 2 tensor types (coarsening deepens the nest by one iScalar)
-  typedef CoarseVector::vector_object                                      CoarseSiteObj;
-  typedef iScalar<vTComplex>                                              vTTComplex;
-  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>          LittleDiracOperatorL2;
-  typedef MultiGeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>     MrhsLittleDiracOperatorL2;
-  typedef LittleDiracOperatorL2::CoarseVector                              CoarseCoarseVector;
-  typedef Aggregation<CoarseSiteObj,vTTComplex,nbasis>                     SubspaceL2;
-
-  // The library dense bottom over the L2 coarse operator
-  typedef DenseCoarseMatrix<CoarseSiteObj,vTTComplex,nbasis>               DenseCC_t;
-
-  PVdagM_t        PVdagM(Ddwf,Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv);
-
-  NextToNearestStencilGeometry5D geom (Coarse5d);
-  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
-
-  //////////////////////////////////////////////////////////////////////
-  // Subspace: load RAW (no Orthogonalise!), or generate.
-  //////////////////////////////////////////////////////////////////////
-  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
-                            + std::to_string(nbasis) + ".scidac";
-  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
-  uint64_t file_exists=0;
-  if ( UGrid->IsBoss() ){ std::ifstream f(subspace_file); file_exists=f.good()?1:0; }
-  UGrid->GlobalSum(file_exists);
-
-  const int cb=0;
-  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
-  if ( file_exists ){
-    std::cout << GridLogMessage << "*** Loading subspace from disk (kept RAW) ***" << std::endl;
-    loadSubspace(AggregatesGCR.subspace, subspace_file);
-  } else {
-    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
-    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
-    saveSubspace(AggregatesGCR.subspace, subspace_file);
-  }
-
-  // RAW copy of the fine null vectors BEFORE CoarsenOperator block-orthonormalises in place.
-  std::vector<LatticeFermionD> rawNull(nbasis,FGrid);
-  for(int k=0;k<nbasis;k++) rawNull[k]=AggregatesGCR.subspace[k];
-
-  //////////////////////////////////////////////////////////////////////
-  // Coarsen L1->L2 and L2->L3 with SINGLE-RHS machinery; import to mrhs via
-  // CopyMatrix.  The L2 (coarse-coarse) single-RHS operator is HOISTED to
-  // main scope: DenseCoarseMatrix imports its stencil and uses its M for
-  // certificates, so it must stay alive for the whole run.
-  //////////////////////////////////////////////////////////////////////
-  MrhsLittleDiracOperator   mrhsLittleDiracOpPV(geom,  CoarseMrhs);
-  MrhsLittleDiracOperatorL2 mrhsLittleDiracOpL2(geom2, CoarseCoarseMrhs);
-  MultiRHSBlockProject<LatticeFermionD> MrhsProjector;
-  MultiRHSBlockProject<CoarseVector>    MrhsProjectorL2;
-
-  LittleDiracOperatorL2 LittleDiracOpL2(geom2,Coarse5d,CoarseCoarse5d);
-  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC5d(LittleDiracOpL2);
-
-  {
-    // --- L1->L2 single-RHS coarse operator (scoped: its padded _A is the memory peak) ---
-    LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d);
-    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);   // orthonormalises AggregatesGCR.subspace in place
-    mrhsLittleDiracOpPV.CopyMatrix(LittleDiracOpPV);
-    MrhsProjector.Allocate(nbasis,FGrid,Coarse5d);
-    MrhsProjector.ImportBasis(AggregatesGCR.subspace);
-    NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
-
-    // --- psi_coarse = P^dag (RAW fine null) -> Galerkin images, NOT e_k ---
-    std::vector<CoarseVector> psi_coarse(nbasis,Coarse5d);
-    for(int k=0;k<nbasis;k++) AggregatesGCR.ProjectToSubspace(psi_coarse[k], rawNull[k]);
-    rawNull.clear(); rawNull.shrink_to_fit();
-    {
-      RealD s2=0.0;
-      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
-        ComplexD sij=TensorRemove(innerProduct(psi_coarse[i],psi_coarse[j]));
-        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
-      }
-      std::cout<<GridLogMessage<<"GUARD: ||<psi_coarse|psi_coarse> - I||_F = "<<std::sqrt(s2)
-               <<"   (~0.23 good; ~sqrt(N_coarse)="<<std::sqrt((double)Coarse5d->gSites())<<" = e_k leak)"<<std::endl;
-    }
-
-    // --- L2->L3 single-RHS coarsening ---
-    SubspaceL2 AggregatesL2(CoarseCoarse5d,Coarse5d,cb);
-    for(int k=0;k<nbasis;k++) AggregatesL2.subspace[k]=psi_coarse[k];
-    LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
-    mrhsLittleDiracOpL2.CopyMatrix(LittleDiracOpL2);
-    MrhsProjectorL2.Allocate(nbasis,Coarse5d,CoarseCoarse5d);
-    MrhsProjectorL2.ImportBasis(AggregatesL2.subspace);
-
-    // --- guard psi_cc ---
-    {
-      std::vector<CoarseCoarseVector> psi_cc(nbasis,CoarseCoarse5d);
-      for(int k=0;k<nbasis;k++) AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
-      RealD s2=0.0;
-      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
-        ComplexD sij=TensorRemove(innerProduct(psi_cc[i],psi_cc[j]));
-        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
-      }
-      std::cout<<GridLogMessage<<"GUARD: ||<psi_cc|psi_cc> - I||_F = "<<std::sqrt(s2)
-               <<"   (~0.23 good; ~sqrt(N_cc)="<<std::sqrt((double)CoarseCoarse5d->gSites())<<" = e_k leak)"<<std::endl;
-    }
-  } // single-RHS FINE op + padded _A + AggregatesL2 + psi_coarse freed here
-
-  NonHermitianLinearOperator<MrhsLittleDiracOperator,CoarseVector>         mrhsLinOpCoarse(mrhsLittleDiracOpPV);
-  NonHermitianLinearOperator<MrhsLittleDiracOperatorL2,CoarseCoarseVector> mrhsLinOpCC(mrhsLittleDiracOpL2);
-
-  //////////////////////////////////////////////////////////////////////
-  // DENSE coarse-coarse bottom: the LIBRARY class, constructed AFTER the
-  // fine coarsening frees its memory peak.  Imports the stencil of the
-  // hoisted single-RHS LittleDiracOpL2 directly (no probing).
-  //////////////////////////////////////////////////////////////////////
-  std::unique_ptr<DenseCC_t> DenseCC;
-  std::unique_ptr<MrhsDenseCCSolve<DenseCC_t,CoarseCoarseVector>> MrhsDenseCC;
-  if (UseDenseCC) {
-    std::cout << GridLogMessage << "**********************************************" << std::endl;
-    std::cout << GridLogMessage << " Dense CC inverse setup (library DenseCoarseMatrix)" << std::endl;
-    std::cout << GridLogMessage << "**********************************************" << std::endl;
-    DenseCC.reset(new DenseCC_t(LittleDiracOpL2, CoarseCoarse5d));
-    MrhsDenseCC.reset(new MrhsDenseCCSolve<DenseCC_t,CoarseCoarseVector>(*DenseCC, CoarseCoarse5d, nrhs));
-  }
-
-  //////////////////////////////////////////////////////////////////////
-  // Solvers, innermost first.
-  //////////////////////////////////////////////////////////////////////
-  TrivialPrecon<CoarseVector>       simpleC;
-  TrivialPrecon<CoarseCoarseVector> simpleCC;
-  TrivialPrecon<LatticeFermionD>    simple_fine;
-
-  // L3 (coarse-coarse) iterative solve: PGCR on the 6D cc operator (DENSE_CC=0 branch)
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
-    L3PGCR(L3Tol,L3MaxIt,mrhsLinOpCC,simpleCC,L3Nstep,L3Nstep);
-  L3PGCR.Level(3);
-  L3PGCR.Name("CCouter");
-
-  LinearFunction<CoarseCoarseVector> *ccSolve;
-  if (UseDenseCC) ccSolve = MrhsDenseCC.get();
-  else            ccSolve = &L3PGCR;
-
-  // L2 coarse smoother: shifted 6D coarse op, fixed nstep
-  ShiftedLinearOperator<CoarseVector> ShiftedMrhsCoarse(CoarseSmootherShift, mrhsLinOpCoarse);
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
-    CoarseSmootherGCR(0.01,1,ShiftedMrhsCoarse,simpleC,CoarseSmootherNstep,CoarseSmootherNstep);
-  CoarseSmootherGCR.Level(2);
-  CoarseSmootherGCR.Name("Csmoother");
-  CoarseSmootherGCR.SetZeroGuess(1);   // caller zeroes vec2: skip r0 apply every L2 iteration
-
-  // L2->L3 V-cycle preconditioner (operates on 6D coarse field)
-  MrhsCoarseThreeLevelPrec<CoarseVector,CoarseCoarseVector>
-    L2to3Precon(mrhsLinOpCoarse, CoarseSmootherGCR, MrhsProjectorL2, *ccSolve,
-                Coarse5d, CoarseCoarse5d, CoarseCoarseMrhs, nrhs);
-
-  // L2 coarse solve: PGCR on 6D coarse op, preconditioned by the L2->L3 V-cycle
-  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
-    L2PGCR(CoarseSolverTol, CoarseSolverOrder/16, mrhsLinOpCoarse, L2to3Precon, 16, 16);
-  L2PGCR.Level(2);
-  L2PGCR.Name("Couter");
-  L2PGCR.SetZeroGuess(1);              // caller zeroes CsolMrhs; restarts still recompute r
-
-  // Fine smoother (per-rhs, looped in the L1->L2 V-cycle)
-  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD>
-    SmootherGCR(0.0,1,ShiftedPVdagM,simple_fine,FineSmootherOrder,FineSmootherOrder);
-  SmootherGCR.Level(1);
-  SmootherGCR.Name("Fsmoother");
-  SmootherGCR.SetZeroGuess(1);         // caller zeroes vec2[r]: saves 12 fine mults/outer
-
-  // L1->L2 V-cycle (fine); its coarse solve is the three-level L2PGCR
-  typedef PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> FineSmoother_t;
-  MrhsTwoLevelMG<LatticeFermionD,CoarseVector,FineSmoother_t>
-    ThreeLevelPrecon(PVdagM, SmootherGCR, MrhsProjector, L2PGCR, Coarse5d, CoarseMrhs);
-
-  // Outer mrhs solve
-  MrhsPGCRNonHermitian<LatticeFermionD>
-    L1PGCR(OuterTol,1000,PVdagM,ThreeLevelPrecon,OuterMmax,OuterNstep);
-  L1PGCR.Level(1);
-  L1PGCR.Name("Fouter");
-  L1PGCR.SetZeroGuess(1);              // sol[r]=Zero() below; restarts recompute r as always
-
-  //////////////////////////////////////////////////////////////////////
-  // Sources and solve
-  //////////////////////////////////////////////////////////////////////
-  std::vector<LatticeFermionD> src(nrhs,FGrid), sol(nrhs,FGrid);
-  for(int r=0;r<nrhs;r++){ gaussian(RNG5,src[r]); sol[r]=Zero(); }
-
-  std::cout << GridLogMessage << "**********************************************" << std::endl;
-  std::cout << GridLogMessage << " MultiRHS THREE-level solve (DenseCoarseMatrix bottom): " << nrhs << " RHS " << std::endl;
-  std::cout << GridLogMessage << "**********************************************" << std::endl;
-
-  GridStopWatch w; w.Start();
-  L1PGCR(src,sol);
-  w.Stop();
-  std::cout << GridLogMessage << "MultiRHS 3-level dense solve total " << w.Elapsed()
-            << "  (per RHS: " << w.useconds()/1.0e6/nrhs << " s)" << std::endl;
-
-  { LatticeFermionD Ax(FGrid); RealD worst=0.0;
-    for(int r=0;r<nrhs;r++){ PVdagM.Op(sol[r],Ax); Ax=Ax-src[r];
-      RealD rn=std::sqrt(norm2(Ax)/norm2(src[r]));
-      std::cout << GridLogMessage << "FINAL: rhs["<<r<<"] true residual = " << rn << std::endl;
-      worst=std::max(worst,rn); }
-    std::cout << GridLogMessage << "FINAL: worst-case residual = " << worst << std::endl;
-  }
-
-  std::cout << GridLogMessage << "Done" << std::endl;
-  Grid_finalize();
-  return 0;
-}

skills/ref_a2a_emf_work.md

@@ -0,0 +1,61 @@
+---
+name: ref_a2a_emf_work
+description: "A2A Extended Meson Field GPU offload work — status, file locations, pending task"
+metadata: 
+  node_type: memory
+  type: project
+  originSessionId: 956e80aa-401d-481a-80bb-17f8abe1c131
+---
+
+## What was built
+
+`Grid/algorithms/blas/A2ASpatialSum.h` — batched GEMM spatial sum replacing scalar SIMD accumulation. Included via `Grid/algorithms/Algorithms.h`.
+
+`tests/Test_extended_meson_field.cc` — test with class `A2AExtendedMesonFieldRef` containing:
+- CPU reference path (`use_blas=false`)
+- BLAS path (`use_blas=true`) using `A2ASpatialSum`
+- Per-phase timing with `[ref type=N]` / `[blas type=N]` labels
+- 4 contraction types (0-3), all verified at machine precision (~4e-16 rel_err)
+
+## Pending task: GPU offload class
+
+**Goal**: Write `A2AExtendedMesonFieldGPU` in the same test file, replacing all `thread_for` loops with `accelerator_for`-based free function kernels.
+
+The `thread_for` blocks to replace all have the form:
+```cpp
+thread_for(r, rd, {
+  int so = r * grid->_ostride[orthogdim];
+  for (int n = 0; n < e1; n++)
+  for (int b = 0; b < e2; b++) {
+    int ss = so + n * stride + b;
+    // work
+  }
+});
+```
+Replace with `accelerator_for(ss, grid->oSites(), Nsimd, { ... })`.
+
+**Free functions to write** (each takes `Lattice<T>` args, opens views internally):
+- `A2ALoopPropagator` — outerProduct sum (loop build)
+- `A2APackLeftConjugated` — conjugate left fermion fields into `Lattice<SpinColourVector_v>`
+- `A2ALoopLeftContractionType0/1/2/3` — per-site loop × loop propagator → `tloop`
+- `A2ALoopRightContractionType0/1/2/3` — per-site tloop × right → `loopRight[j]`
+
+**Data structure changes required**:
+- `tloopv`: `std::vector<SpinColourMatrix_v>` → `Lattice<SpinColourMatrix_v>` (PropagatorField)
+- `leftv[i]`: `std::vector<SpinColourVector_v>` → `Lattice<SpinColourVector_v>`
+- `loopRight[j]`: `std::vector<SpinColourVector_v>` → `Lattice<SpinColourVector_v>`
+
+**Why**: `std::vector<vobj>` is host memory, not GPU accessible. See [[ref_lattice_vs_vector]].
+
+**`A2ASpatialSum` impact**: `PackLeft`/`PackRight` currently take `std::vector<std::vector<vobj>>`. Once leftv/loopRight become `std::vector<Lattice<vobj>>`, those signatures must change to match.
+
+## Timing on 8.8.8.16 (N_i=N_j=8, Nloop=4, 1 MPI rank)
+
+Dominant costs:
+- `loop_build`: 4-6 ms (outerProduct over 4 propagators)
+- `pack_loopright`: 0.9-2.2 ms (type-dependent)
+- `spatial_sum` (ref): ~1.5 ms
+- `A2ASpatialSum TOTAL`: 2.5-4.3 ms (PackLeft+PackRight dominate GEMM on small volume)
+
+## Related
+[[ref_accelerator_for]] [[ref_coalesced_views]] [[ref_lattice_vs_vector]] [[ref_grid_simt_pattern]]

skills/ref_accelerator_for.md

@@ -0,0 +1,43 @@
+---
+name: ref_accelerator_for
+description: Grid accelerator_for usage — converting block-strided thread_for to GPU-portable oSites loops
+metadata: 
+  node_type: memory
+  type: reference
+  originSessionId: 956e80aa-401d-481a-80bb-17f8abe1c131
+---
+
+## Pattern: block-strided thread_for → accelerator_for over oSites
+
+Old CPU-only pattern (block-strided over orthog dimension):
+```cpp
+thread_for(r, rd, {
+  int so = r * grid->_ostride[orthogdim];
+  for (int n = 0; n < e1; n++)
+  for (int b = 0; b < e2; b++) {
+    int ss = so + n * stride + b;
+    // work on site ss
+  }
+});
+```
+
+GPU-portable replacement:
+```cpp
+accelerator_for(ss, grid->oSites(), Nsimd, {
+  // work on site ss — one SIMT thread per (osite, lane) on GPU
+  // one thread per osite (lane loop implicit via GRID_SIMT) on CPU
+});
+```
+
+Key rules:
+- `accelerator_for(iter, count, Nsimd, body)` — Nsimd is `vobj::Nsimd()` or `grid->Nsimd()`
+- On CPU: expands to `thread_for` over count, `acceleratorSIMTlane` always returns 0 — must use `#ifdef GRID_SIMT` pattern if iterating lanes explicitly (see [[ref_grid_simt_pattern]])
+- On GPU: one SIMT thread per (iter × lane), `acceleratorSIMTlane(Nsimd)` returns actual lane
+- Loop body must capture only scalar/POD by value or via device-accessible pointers; no `std::vector` or host containers inside the body
+- `Coordinate` inside `accelerator_for` must be `AcceleratorVector<int, MaxDims>` (stack-allocated, device-safe) — Grid's `Coordinate` typedef already satisfies this
+
+## Where defined
+`Grid/threads/Accelerator.h` — CPU path ~line 607; GPU paths in conditional blocks above.
+
+## Model file
+`Grid/algorithms/blas/MomentumProject.h` — `ImportVector` is the canonical example of correct `accelerator_for` + SIMD lane extraction.

skills/ref_coalesced_views.md

@@ -0,0 +1,70 @@
+---
+name: ref_coalesced_views
+description: Grid coalescedRead/coalescedWrite and autoView — GPU-portable field access inside accelerator_for
+metadata: 
+  node_type: memory
+  type: reference
+  originSessionId: 956e80aa-401d-481a-80bb-17f8abe1c131
+---
+
+## View access modes
+
+```cpp
+autoView(v, field, AcceleratorRead);   // read-only, device-accessible
+autoView(v, field, AcceleratorWrite);  // write-only, device-accessible
+autoView(v, field, AcceleratorReadWrite); // read-write, device-accessible
+autoView(v, field, CpuRead);           // CPU only (avoids GPU migration)
+autoView(v, field, CpuWrite);          // CPU only
+```
+
+Views must be opened **before** `accelerator_for` and closed (go out of scope) **after**. Never open a view inside the accelerator_for body.
+
+## coalescedRead / coalescedWrite
+
+Inside `accelerator_for(ss, oSites, Nsimd, { ... })`:
+
+```cpp
+auto site = coalescedRead(v[ss]);          // reads SIMT lane; returns scalar_object on GPU, vobj on CPU
+coalescedWrite(v[ss], site);               // writes SIMT lane
+```
+
+- `coalescedRead(v[ss])` calls `v.operator()(ss)` which on GPU returns `extractLane(lane, v[ss])` — one lane per SIMT thread, contiguous across threads → coalesced
+- On CPU returns the full vobj (no lane extraction needed; handled transparently)
+- The returned type is `decltype(coalescedRead(v[ss]))` — use `auto` or match with scalar_object
+
+## Typical kernel pattern
+
+```cpp
+autoView(out_v, out, AcceleratorWrite);
+autoView(in_v,  in,  AcceleratorRead);
+accelerator_for(ss, grid->oSites(), vobj::Nsimd(), {
+  auto x = coalescedRead(in_v[ss]);
+  // modify x ...
+  coalescedWrite(out_v[ss], x);
+});
+```
+
+## Free function kernel signature
+
+```cpp
+template<class vobj>
+void MyKernel(Lattice<vobj> &out, const Lattice<vobj> &in)
+{
+  GridBase *grid = in.Grid();
+  autoView(out_v, out, AcceleratorWrite);
+  autoView(in_v,  in,  AcceleratorRead);
+  accelerator_for(ss, grid->oSites(), vobj::Nsimd(), {
+    auto x = coalescedRead(in_v[ss]);
+    coalescedWrite(out_v[ss], x);
+  });
+}
+```
+
+## What NOT to do
+- Do not access `std::vector` elements inside `accelerator_for` — not device-accessible
+- Do not use `CpuRead`/`CpuWrite` views inside `accelerator_for` — GPU will fault
+- Do not assign to `v[ss]` directly inside `accelerator_for` — use `coalescedWrite`
+- Do not open multiple write views on the same field simultaneously
+
+## Related
+[[ref_accelerator_for]] [[ref_lattice_vs_vector]]

skills/ref_grid_simt_pattern.md

@@ -0,0 +1,47 @@
+---
+name: ref_grid_simt_pattern
+description: Grid GRID_SIMT
+metadata: 
+  node_type: memory
+  type: reference
+  originSessionId: 956e80aa-401d-481a-80bb-17f8abe1c131
+---
+
+## The problem
+
+On CPU, `accelerator_for(sf, oSites, Nsimd, {...})` expands to `thread_for(sf, oSites, {...})` — one thread per osite. `acceleratorSIMTlane(Nsimd)` always returns **0** on CPU. If you need to iterate all Nsimd lanes (e.g. to extract SIMD-packed data), you must loop explicitly on CPU.
+
+On GPU, `accelerator_for` launches one SIMT thread per (osite × lane). `acceleratorSIMTlane(Nsimd)` returns the actual lane index [0, Nsimd).
+
+## Correct pattern (from MomentumProject::ImportVector)
+
+```cpp
+accelerator_for(sf, osites, Nsimd, {
+#ifdef GRID_SIMT
+  {
+    int lane = acceleratorSIMTlane(Nsimd);
+#else
+    for (int lane = 0; lane < Nsimd; lane++) {
+#endif
+      // body using lane
+    }
+  });
+```
+
+- On GPU: `GRID_SIMT` is defined → single-lane body, lane from hardware
+- On CPU: `GRID_SIMT` is not defined → explicit lane loop inside the osite thread
+
+## When is this needed?
+
+Only when you explicitly need the lane index, e.g.:
+- Extracting scalar data from SIMD-packed `vobj` via `extractLane(lane, src[sf])`
+- Computing full local coordinates from (osite, lane) → `Lexicographic::CoorFromIndex(icoor, lane, simd_layout)`
+
+When using `coalescedRead`/`coalescedWrite`, this pattern is **not needed** — those handle lane selection transparently.
+
+## Pitfall that caused a bug
+
+`A2ASpatialSum::PackVectors` originally used `accelerator_for` without the `#ifdef GRID_SIMT` lane loop. On CPU, only lane=0 was extracted, giving wrong norms (~8× too small for `GEN_SIMD_WIDTH=64`, `Nsimd=4`). Fix: add the `#ifdef GRID_SIMT` pattern. See [[ref_accelerator_for]].
+
+## Model file
+`Grid/algorithms/blas/MomentumProject.h`, function `ImportVector`, lines ~166-207.

skills/ref_lattice_vs_vector.md

@@ -0,0 +1,48 @@
+---
+name: ref_lattice_vs_vector
+description: When to use Lattice<T> vs std::vector<T> for GPU-portable field storage in Grid
+metadata: 
+  node_type: memory
+  type: reference
+  originSessionId: 956e80aa-401d-481a-80bb-17f8abe1c131
+---
+
+## Rule
+
+Use `Lattice<vobj>` (or `std::vector<Lattice<vobj>>`) for any field that will be read or written inside `accelerator_for`. `std::vector<vobj>` is host memory and is NOT device-accessible.
+
+## Before vs after GPU offload
+
+```cpp
+// CPU-only (host memory, not GPU accessible)
+std::vector<SpinColourVector_v> tloopv(oSites, Zero());
+// accessed directly: tloopv[ss]
+
+// GPU-portable
+Lattice<SpinColourVector_v> tloop(grid);
+// accessed via view: autoView(tloop_v, tloop, AcceleratorWrite);
+//                    coalescedWrite(tloop_v[ss], val);
+```
+
+## Corollary: function signatures
+
+CPU-only version:
+```cpp
+void PackLeft(const std::vector<std::vector<vobj>> &leftv);
+```
+
+GPU-portable version:
+```cpp
+void PackLeft(const std::vector<Lattice<vobj>> &leftv);
+```
+
+## deviceVector for raw device buffers
+
+`deviceVector<T>` (defined in Grid) is like `std::vector<T>` but in device-accessible memory. Use for raw scalar scratch/pack buffers (e.g. GEMM input/output staging). Not for structured lattice data.
+
+## Pointer arrays for batched BLAS
+
+`deviceVector<scalar *>` holds batch pointer arrays. Populate with `acceleratorPut(ptrs[t], base + offset)` — sets device-side pointer from host. See `A2ASpatialSum::Allocate`.
+
+## Related
+[[ref_coalesced_views]] [[ref_accelerator_for]]

systems/Aurora/aggregate_io.pbs

@@ -1,144 +0,0 @@
-#!/bin/bash
-
-# Grid's tests/IO/Test_aggregate_io on Aurora, 12 ranks per node (one per
-# tile).  This is the Grid-side counterpart of MPI_benchmark/io_aurora.pbs:
-# same three paths, same comparison, but exercising the real BinaryIO.h
-# code that production uses, with Grid's checksums and its munge/endian
-# conversion in the loop.  Run both -- if they disagree, the difference is
-# in Grid, not in MPI-IO.
-#
-# Submit from the BUILD directory: BIN below is $PBS_O_WORKDIR relative.
-
-#PBS -q debug-scaling
-#PBS -l filesystems=flare
-#PBS -l filesystems=home
-#PBS -l select=4
-#PBS -l walltime=01:00:00
-#PBS -A 15479
-##PBS -A LatticeQCD_aesp_CNDA
-
-cd $PBS_O_WORKDIR
-cp $PBS_NODEFILE nodefile
-
-##########################################################################
-# Grid source tree.  Unlike io_mpi this IS a Grid binary -- SYCL, c-lime,
-# the lot -- so sourceme.sh is required, and so is the tile wrapper.
-##########################################################################
-root=$HOME/Grid/systems/Aurora            # <-- adjust to your source tree
-source ${root}/sourceme.sh
-TILE=${root}/benchmarks/gpu_tile.sh
-
-BIN=$PBS_O_WORKDIR/tests/IO/Test_aggregate_io
-[ -x $BIN   ] || { echo "no $BIN -- submit from the build directory"; exit 1; }
-[ -x $TILE  ] || { echo "no $TILE -- check \$root"; exit 1; }
-
-export OMP_NUM_THREADS=4
-export MPICH_OFI_NIC_POLICY=GPU
-# The aggregate path's MPI_Alltoallv runs on HOST buffers, so NIC policy
-# NUMA is arguably the better match here even though GPU is the standard
-# for Grid on this machine.  Worth one comparison run; do not change it
-# silently, the rest of the Grid numbers on Aurora were taken with GPU.
-#export MPICH_OFI_NIC_POLICY=NUMA
-
-##########################################################################
-# Test_aggregate_io writes ref.bin / agg.bin / raw.bin / trunc.bin with
-# RELATIVE paths, so they land in the cwd.  Home is not the point of the
-# exercise.
-##########################################################################
-# PROJECT is the flare project DIRECTORY name, not the -A account number.
-PROJECT=LatticeQCD_aesp_CNDA
-WORK=/lus/flare/projects/$PROJECT/$USER/aggio.$PBS_JOBID
-mkdir -p $WORK || { echo "cannot create $WORK -- set PROJECT correctly"; exit 1; }
-cd       $WORK
-
-# Leave striping at the directory default, as the Frontier runs did.  At
-# 128 nodes there the aggregate path was 2.2x faster on the default layout
-# than on -c -1, while MPI-IO was 2.4x slower on it; the two paths want
-# opposite things.  Set it only to reproduce that interaction, and say so.
-#lfs setstripe -c -1 -S 8M $WORK
-lfs getstripe -d $WORK 2>/dev/null || echo "(no lfs getstripe -- not Lustre?)"
-
-# ROMIO's own account of what the reference path did.  Verbose; switch on
-# deliberately.
-#export MPICH_MPIIO_STATS=1
-#export MPICH_MPIIO_TIMERS=1
-
-AT=8        # accelerator threads; immaterial to an I/O test, kept canonical
-
-run () {   # run <nodes> <grid> <mpi> <comment> [extra args...]
-  local nodes=$1 gr=$2 mp=$3 note=$4
-  local ntot=$(( nodes * 12 ))
-  shift 4
-  echo
-  echo "==================================================================="
-  echo "=== nodes=$nodes ranks=$ntot grid=$gr mpi=$mp   $note"
-  echo "=== extra: $@"
-  echo "==================================================================="
-  # Deliberately NOT removing ref.bin/agg.bin/raw.bin here.  The test does
-  # it itself (Test_aggregate_io.cc:203 and :370) for exactly the stale-tail
-  # reason, and doing it in the script would destroy the files a later
-  # --io-read-only job needs.
-  mpiexec -np $ntot -ppn 12 -envall $TILE $BIN --mpi $mp --grid $gr \
-          --shm-mpi 0 --shm 2048 --device-mem 32000 --accelerator-threads $AT "$@"
-  echo "=== exit $?"
-}
-
-#####################################################################
-# Phase 0.  Logic corners, correctness only, no bandwidth claimed.
-#
-# These run the FULL default target sweep {1, 1024, 64K, 4M}, which walks
-# k from 1 to Nd-1 and so exercises every shape of aggregation plan on a
-# volume small enough that the serial byte-for-byte file comparison is
-# affordable.  1.3 MB/rank means nothing for bandwidth and is not meant to.
-#
-# Both branches of MPI_Alltoallv are covered; the labels were checked with
-# MPI_benchmark/io_mpi, not assumed.
-#####################################################################
-run 1 16.16.16.24 2.2.1.3 "UNIFORM counts, row of 4"      --io-reps 0
-run 1 12.12.24.8  2.2.3.1 "NON-UNIFORM counts, row of 12" --io-reps 0
-
-#####################################################################
-# Phase 1.  151 MB/rank, 3.6 GB record, 24 ranks.  Correctness AND
-# performance at the production 4 MB target only -- the byte-for-byte
-# comparison reads the whole file on one rank, so the four-target sweep
-# is not affordable here.
-#####################################################################
-run 2 16.16.96.256 2.2.3.2 "3.6 GB, row of 4, 32 extents, correctness + perf" \
-    --aggregate-target 4194304
-
-# Same volume per rank, but an odd process factor in an un-split dimension
-# puts the variable-count branch of Alltoallv where the bandwidth is real.
-run 2 24.16.64.256 3.2.2.2 "3.6 GB, NON-UNIFORM counts, row of 6, 22 extents" \
-    --aggregate-target 4194304
-
-#####################################################################
-# Phase 2.  604 MB/rank, 29.0 GB record, 48 ranks.  Correctness off.
-#
-# Per-rank volume is held at 16.16.32.128 in ALL FOUR, so the only thing
-# changing is how much work the aggregation has to do:
-#
-#   4.4.3.1   row 16    8 extents of 72 MB   <- old path: 9 KB runs
-#   4.2.3.2   row  8   16 extents of 36 MB
-#   2.2.12.1  row  4   32 extents of 18 MB
-#   1.1.3.16  row  1  128 extents of 4.5 MB  <- no exchange at all
-#
-# The last is the interesting control: with the fast dimensions unsplit
-# the aggregate path degenerates to plain large-extent POSIX and the
-# Alltoallv is trivial, so it separates "the transposition is cheap" from
-# "the writes are large".  Note the test's own header warns that 1.1.X.Y
-# makes the CORRECTNESS section vacuous -- rows of size one exchange
-# nothing.  That is fine here because correctness is off; it would not be
-# fine as a correctness run.
-#####################################################################
-run 4 64.64.96.128   4.4.3.1  "29.0 GB, row of 16"  --aggregate-target 4194304 --io-no-correctness
-run 4 64.32.96.256   4.2.3.2  "29.0 GB, row of 8"   --aggregate-target 4194304 --io-no-correctness
-run 4 32.32.384.128  2.2.12.1 "29.0 GB, row of 4"   --aggregate-target 4194304 --io-no-correctness
-run 4 16.16.96.2048  1.1.3.16 "29.0 GB, row of 1, no exchange" \
-    --aggregate-target 4194304 --io-no-correctness
-
-echo
-echo "=== done.  Output left in $WORK"
-echo "=== For COLD read numbers, submit a second job on a fresh allocation"
-echo "=== with WORK pointed at this directory and --io-read-only added."
-echo "=== Reading back what this job just wrote only measures page cache."
-ls -l $WORK

systems/Aurora/aggregate_io_scaling.pbs

@@ -1,117 +0,0 @@
-#!/bin/bash
-
-# Node count scan for Grid's tests/IO/Test_aggregate_io on Aurora, 12 ranks
-# per node.  The Aurora counterpart of systems/Frontier/aggregate_io_scaling.slurm.
-#
-# Everything that could confound the comparison is held fixed and only the
-# number of Lustre clients changes:
-#
-#   local volume   8.8.32.128        = 262144 sites x 576 B = 151 MB / rank
-#   aggregation    k=2, row of 16, 8 extents of 18 MB        (identical at
-#                                                             every node count)
-#   old path       contiguous run = lLattice[0]*576 = 4.6 KB (identical too)
-#
-# So the ratio of contiguous extent between the two layout preserving paths
-# is a constant 4000x across the whole scan, and any change in their relative
-# bandwidth is a property of the client count alone.
-#
-# The mpi decompositions differ from the Frontier scan because 12 ranks per
-# node is not 8, but they were chosen to give the SAME aggregation plan --
-# verified with MPI_benchmark/io_mpi, which uses the identical plan algorithm
-# and the same 576 B site object:
-#
-#   Frontier   4.4.2.1  at   32 ranks  ->  k=2, row 16, 8 extents
-#   Aurora     4.4.3.1  at   48 ranks  ->  k=2, row 16, 8 extents
-#   Aurora     4.4.12.8 at 1536 ranks  ->  k=2, row 16, 8 extents
-#
-# Consequence of 12 rather than 8: at a given NODE count the record and the
-# client count are 1.5x Frontier's.  Compare the machines at equal RANK count
-# for equal record size, or at equal NODE count for each machine used as it
-# is meant to be used.  Say which one any table is reporting.
-#
-# Trim the tail of the run list for a smaller allocation; the geometries are
-# independent.
-#
-# Submit from the BUILD directory.
-
-#PBS -q prod
-#PBS -l filesystems=flare
-#PBS -l filesystems=home
-#PBS -l select=128
-#PBS -l walltime=02:00:00
-#PBS -A 15479
-##PBS -A LatticeQCD_aesp_CNDA
-
-cd $PBS_O_WORKDIR
-cp $PBS_NODEFILE nodefile
-
-root=$HOME/Grid/systems/Aurora            # <-- adjust to your source tree
-source ${root}/sourceme.sh
-TILE=${root}/benchmarks/gpu_tile.sh
-
-BIN=$PBS_O_WORKDIR/tests/IO/Test_aggregate_io
-[ -x $BIN  ] || { echo "no $BIN -- submit from the build directory"; exit 1; }
-[ -x $TILE ] || { echo "no $TILE -- check \$root"; exit 1; }
-
-export OMP_NUM_THREADS=4
-export MPICH_OFI_NIC_POLICY=GPU
-
-# MPICH_DBG_* and MPICH_MPIIO_STATS are deliberately absent: at 1536 ranks
-# they produce gigabytes of log and perturb the timings they would explain.
-# Get those from systems/Aurora/aggregate_io.pbs on the debug queue.
-
-# PROJECT is the flare project DIRECTORY name, not the -A account number.
-PROJECT=LatticeQCD_aesp_CNDA
-WORK=/lus/flare/projects/$PROJECT/$USER/aggscale.$PBS_JOBID
-mkdir -p $WORK || { echo "cannot create $WORK -- set PROJECT correctly"; exit 1; }
-cd       $WORK
-
-# The largest rung writes three files of 232 GB, so budget ~700 GB for the
-# scan and check the quota before submitting.  The test unlinks its three
-# files at the start of each run, so that is peak, not cumulative.
-#
-# Do NOT stripe by default.  Measured on FRONTIER at 128 nodes / 1024 ranks,
-# 154.6 GB record, best of 3 (MB/s) -- these are Frontier numbers, quoted for
-# what to look for, not a prediction for Aurora:
-#
-#                     default layout    lfs setstripe -c -1 -S 8M
-#   raw                    97673                81728
-#   MPI-IO lexicographic   31675                77534
-#   aggregate             189535                86063
-#
-# The two layout preserving paths want opposite things.  Wide striping is
-# what rescues MPI-IO at scale, 32 -> 78 GB/s; it costs the aggregate path a
-# factor of 2.2 and costs raw as well.  With the default layout MPI-IO
-# plateaued at ~30 GB/s from 64 nodes upward while the aggregate path kept
-# scaling.  Uncomment only to reproduce that interaction, and record which.
-#lfs setstripe -c -1 -S 8M $WORK
-lfs getstripe -d $WORK 2>/dev/null || echo "(no lfs getstripe -- not Lustre?)"
-
-ARGS="--aggregate-target 4194304 --io-no-correctness --io-reps 3"
-AT=8
-
-run () {   # run <nodes> <grid> <mpi> <comment>
-  local nodes=$1 nranks=$(( $1 * 12 ))
-  echo
-  echo "==================================================================="
-  echo "=== SCAN nodes=$nodes ranks=$nranks  grid=$2  mpi=$3   $4"
-  echo "==================================================================="
-  mpiexec -np $nranks -ppn 12 -envall $TILE $BIN --grid $2 --mpi $3 \
-          --shm-mpi 0 --shm 2048 --device-mem 32000 --accelerator-threads $AT $ARGS
-  echo "=== exit $?"
-}
-
-#            nodes  global lattice      decomposition   record
-run    4     32.32.96.128    4.4.3.1   "7.2 GB"
-run    8     32.32.96.256    4.4.3.2   "14.5 GB"
-run   16     32.32.96.512    4.4.3.4   "29.0 GB"
-run   32     32.32.192.512   4.4.6.4   "58.0 GB"
-run   64     32.32.192.1024  4.4.6.8   "116.0 GB"
-run  128     32.32.384.1024  4.4.12.8  "231.9 GB"
-
-echo
-echo "=== done.  Files left in $WORK"
-echo "=== For cold read numbers, submit a SECOND job on a fresh allocation"
-echo "=== with WORK pointed at this directory and --io-read-only added to"
-echo "=== ARGS.  Reading back what this job wrote only measures page cache."
-ls -l $WORK

systems/Booster/config-command

@@ -8,6 +8,7 @@ LIME=/p/home/jusers/boyle2/juwels/gm2dwf/boyle/
     --disable-gparity \
     --disable-fermion-reps \
     --with-lime=$LIME \
+    --enable-accelerator-cshift \
     --disable-unified \
     CXX=nvcc \
     LDFLAGS="-cudart shared " \

systems/Frontier/aggregate_io.slurm

@@ -1,152 +0,0 @@
-#!/bin/bash -l
-
-#SBATCH --job-name=runIO
-#SBATCH --nodes=4
-#SBATCH --ntasks-per-node=8
-#SBATCH --cpus-per-task=7
-#SBATCH --gpus-per-node=8
-#SBATCH --time=02:00:00
-#SBATCH --account=phy157_dwf
-#SBATCH --gpu-bind=none
-#SBATCH --exclusive
-#SBATCH --mem=0
-#SBATCH -q debug
-
-cat << EOF > select_gpu
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export NUMA_MAP=(3 3 1 1 2 2 0 0)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-
-rank=\$SLURM_PROCID
-CMD="numactl -m \$NUMA -N \$NUMA  \$*"
-exec \$CMD
-EOF
-
-chmod +x ./select_gpu
-
-# GPU_MAP/NUMA_MAP above describe a full node.  At fewer than 8 ranks per node
-# Slurm hands each task cpus-per-task cores block-wise from core 0, so the NUMA
-# domain NUMA_MAP names is not in the task's allowed cpu mask and numactl fails
-# with EINVAL before Grid starts.  The GPU choice stays valid; only the memory
-# binding does not.  For the small correctness runs the binding is irrelevant,
-# so use a wrapper that picks the GCD and leaves placement to Slurm.
-cat << EOF > select_gcd
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-exec \$*
-EOF
-
-chmod +x ./select_gcd
-SELECT=$SLURM_SUBMIT_DIR/select_gpu
-SELECT_NONUMA=$SLURM_SUBMIT_DIR/select_gcd
-
-root=$HOME/ParallelIO/systems/Frontier/
-source ${root}/sourceme-rocm7.2.sh
-
-export OMP_NUM_THREADS=7
-export MPICH_GPU_SUPPORT_ENABLED=1
-
-# Test_aggregate_io writes ref.bin / agg.bin / trunc.bin with *relative* paths,
-# so they land in the cwd of each rank, which is the submission directory unless
-# we move.  Home is NFS; the whole point of the exercise is Lustre.
-WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/aggio.$SLURM_JOB_ID
-mkdir -p $WORK
-cd       $WORK
-
-# Striping has to be set on the directory before the files are created, and
-# all three paths inherit it, so the A/B comparison stays fair either way.
-# Left unset: at 128 nodes the aggregate path is 2.2x faster on the default
-# layout than on -c -1, while MPI-IO is 2.4x slower on it.  See the table in
-# aggregate_io_scaling.slurm.  Set it only to reproduce that interaction.
-#lfs setstripe -c 32 -S 8M $WORK
-lfs getstripe -d $WORK      # record the inherited layout; needed to read the numbers
-
-BIN=$SLURM_SUBMIT_DIR/tests/IO/Test_aggregate_io
-
-# ROMIO / Cray MPICH collective buffering report for the reference path.
-# Verbose; switch on when you want to see what the old path is doing.
-# export MPICH_MPIIO_STATS=1
-# export MPICH_MPIIO_TIMERS=1
-
-# Always state ntasks-per-node explicitly.  Letting srun derive it from -n
-# gives an uneven packing (3,2,2,2 for -n9 over 4 nodes) which no fixed
-# localid -> GCD map can describe.  Ranks per node must not exceed 8.
-run () {   # run <nodes> <ranks-per-node> <grid> <mpi> <comment> [extra args...]
-  local nodes=$1 rpn=$2 gr=$3 mp=$4 note=$5 wrap=$SELECT
-  local nranks=$(( $1 * $2 ))
-  shift 5
-  if [ $rpn -ne 8 ]; then wrap=$SELECT_NONUMA; fi   # NUMA_MAP only valid at 8
-  echo
-  echo "==================================================================="
-  echo "=== N=$nodes x $rpn = $nranks  grid=$gr  mpi=$mp   $note"
-  echo "=== extra: $@"
-  echo "==================================================================="
-  srun -N$nodes -n$nranks --ntasks-per-node=$rpn $wrap $BIN --grid $gr --mpi $mp \
-       --accelerator-threads 8 --shm 2048 --device-mem 32000 "$@"
-  echo "=== exit $?"
-}
-
-#####################################################################
-# Phase 0.  Non-uniform AllToAllV, correctness only.
-#
-# The variable-count branch of CartesianCommunicator::AllToAllV only runs
-# when the send counts differ between destinations, which needs an odd
-# process factor in an un-split (fast) dimension.  It does NOT need a toy
-# volume -- see phase 1b, which hits it at 226 MB/rank.  The one small run
-# kept here is the only way to reach the non-uniform case at k=1 with many
-# extents per rank; it is a logic corner, carries --io-reps 0, and reports
-# no bandwidth.  2.2.4.4 per rank is 9 KB and means nothing for I/O.
-#####################################################################
-run 3 3 6.6.4.4  3.3.1.1  "non-uniform at all four targets, k=1 and k=3" --io-reps 0
-
-#####################################################################
-# Phase 1.  Medium volume, 4.8 GB record, 151 MB/rank.
-# Correctness plus performance at the production 4MB target only; the
-# byte-for-byte comparison reads the whole file on one rank, so the
-# full four target sweep is not affordable above a few GB.
-#####################################################################
-run 4 8 32.32.64.128  4.4.2.1  "4.8 GB, correctness + perf" --aggregate-target 4194304
-
-#####################################################################
-# Phase 1b.  Non-uniform AllToAllV at a volume that means something.
-# 24 ranks = 3 nodes x 8, so the NUMA map stays valid.
-#   48.48.64.64 / 3.2.2.2  k=2 row=6  <=6 extents of 13.5 MB  NON-UNIFORM
-# 5.4 GB record, 226 MB/rank.  Correctness and performance together, so
-# the variable-count path is covered where the bandwidth is also real.
-#####################################################################
-run 3 8 48.48.64.64  3.2.2.2  "5.4 GB, NON-UNIFORM counts, correctness + perf" --aggregate-target 4194304
-
-#####################################################################
-# Phase 2.  Production volume, 19.3 GB record, 604 MB/rank.
-# Correctness off.  Modelled extents at the 4MB target (fobj = 576 B):
-#   1.1.4.8   k=2  row=1   16 extents of 36 MB   <- old path gives 36 KB runs
-#   4.4.2.1   k=2  row=16   8 extents of 72 MB   <- old path gives  9 KB runs
-#   2.2.2.4   k=2  row=4    8 extents
-#   8.4.1.1   k=2  row=32   4 extents
-#####################################################################
-run 4 8 64.64.64.128  1.1.4.8  "19.3 GB, baseline decomposition" --aggregate-target 4194304 --io-no-correctness
-run 4 8 64.64.64.128  4.4.2.1  "19.3 GB, row of 32"             --aggregate-target 4194304 --io-no-correctness
-run 4 8 64.64.64.128  2.2.2.4  "19.3 GB, row of 4"              --aggregate-target 4194304 --io-no-correctness
-run 4 8 64.64.64.128  8.4.1.1  "19.3 GB, row of 32, x heavy"    --aggregate-target 4194304 --io-no-correctness
-
-# 10.9 GB with non-uniform counts, 453 MB/rank, 27 MB extents, 24 ranks.
-run 3 8 48.96.64.64   3.4.2.1  "10.9 GB, NON-UNIFORM counts"    --aggregate-target 4194304 --io-no-correctness
-
-#####################################################################
-# Phase 3.  Large volume, 43.5 GB record, 1.36 GB/rank.
-# Host side buffering is 3 copies of the local slice, so ~4 GB/rank
-# and ~33 GB/node.  Two repetitions only.
-#####################################################################
-run 4 8 64.64.96.192  1.1.4.8  "43.5 GB, baseline"  --aggregate-target 4194304 --io-no-correctness --io-reps 2
-run 4 8 64.64.96.192  4.4.2.1  "43.5 GB, row of 32" --aggregate-target 4194304 --io-no-correctness --io-reps 2
-
-echo
-echo "=== done.  Output left in $WORK"
-ls -l $WORK

systems/Frontier/aggregate_io_scaling.slurm

@@ -1,104 +0,0 @@
-#!/bin/bash -l
-
-# Node count scan.  Everything that could confound the comparison is held
-# fixed and only the number of Lustre clients changes:
-#
-#   local volume   8.8.32.128        = 262144 sites x 576 B = 151 MB / rank
-#   aggregation    k=2, row of 16, 8 extents of 18 MB        (identical at
-#                                                             every node count)
-#   old path       contiguous run = lLattice[0]*576 = 4.6 KB (identical too)
-#
-# So the ratio of contiguous extent between the two layout preserving paths is
-# a constant 4000x across the whole scan, and any change in their relative
-# bandwidth is a property of the client count alone.  That is the variable the
-# 4 node runs could not move.
-#
-# Trim the tail of the run list if you want a smaller allocation; the
-# geometries are independent.
-
-#SBATCH --job-name=ioScale
-#SBATCH --nodes=128
-#SBATCH --ntasks-per-node=8
-#SBATCH --cpus-per-task=7
-#SBATCH --gpus-per-node=8
-#SBATCH --time=02:00:00
-#SBATCH --account=phy157_dwf
-#SBATCH --gpu-bind=none
-#SBATCH --exclusive
-#SBATCH --mem=0
-
-cat << EOF > select_gpu
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export NUMA_MAP=(3 3 1 1 2 2 0 0)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-
-rank=\$SLURM_PROCID
-CMD="numactl -m \$NUMA -N \$NUMA  \$*"
-exec \$CMD
-EOF
-
-chmod +x ./select_gpu
-SELECT=$SLURM_SUBMIT_DIR/select_gpu
-
-root=$HOME/ParallelIO/systems/Frontier/
-source ${root}/sourceme-rocm7.2.sh
-
-export OMP_NUM_THREADS=7
-export MPICH_GPU_SUPPORT_ENABLED=1
-
-WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/aggscale.$SLURM_JOB_ID
-mkdir -p $WORK
-cd       $WORK
-
-# The largest rung writes three files of 154 GB, so budget ~700 GB for the
-# whole scan and check the quota before submitting.
-#
-# Do NOT stripe by default.  Measured on Frontier at 128 nodes / 1024 ranks,
-# 154.6 GB record, best of 3 (MB/s):
-#
-#                     default layout    lfs setstripe -c -1 -S 8M
-#   raw                    97673                81728
-#   MPI-IO lexicographic   31675                77534
-#   aggregate             189535                86063
-#
-# The two layout preserving paths want opposite things.  Wide striping is what
-# rescues MPI-IO at scale, 32 -> 78 GB/s, which is presumably what careful
-# striping has been buying in production.  It costs the aggregate path a factor
-# of 2.2, and costs raw as well.  With the default layout MPI-IO plateaus at
-# ~30 GB/s from 64 nodes upward -- the 20-40 GB/s seen in production -- while
-# the aggregate path keeps scaling.  Uncomment only to reproduce that.
-#lfs setstripe -c -1 -S 8M $WORK
-lfs getstripe -d $WORK      # record the inherited layout; needed to read the numbers
-
-BIN=$SLURM_SUBMIT_DIR/tests/IO/Test_aggregate_io
-ARGS="--aggregate-target 4194304 --io-no-correctness --io-reps 3"
-
-run () {   # run <nodes> <grid> <mpi> <comment>
-  local nodes=$1 nranks=$(( $1 * 8 ))
-  echo
-  echo "==================================================================="
-  echo "=== SCAN nodes=$nodes ranks=$nranks  grid=$2  mpi=$3   $4"
-  echo "==================================================================="
-  srun -N$nodes -n$nranks --ntasks-per-node=8 $SELECT $BIN --grid $2 --mpi $3 \
-       --accelerator-threads 8 --shm 2048 --device-mem 32000 $ARGS
-  echo "=== exit $?"
-}
-
-#            nodes  global lattice      decomposition   record
-run    4     32.32.64.128    4.4.2.1   "4.8 GB"
-run    8     32.32.64.256    4.4.2.2   "9.7 GB"
-run   16     32.32.64.512    4.4.2.4   "19.3 GB"
-run   32     32.32.128.512   4.4.4.4   "38.6 GB"
-run   64     32.32.128.1024  4.4.4.8   "77.3 GB"
-run  128     32.32.256.1024  4.4.8.8   "154.6 GB"
-
-echo
-echo "=== done.  Files left in $WORK"
-echo "=== For cold read numbers, submit a SECOND job on a fresh allocation"
-echo "=== with WORK pointed at this directory and --io-read-only added to"
-echo "=== ARGS.  Reading back what this job wrote only measures page cache."
-ls -l $WORK

systems/Frontier/config-command

@@ -1,5 +1,3 @@
-CLIME=`spack find --paths c-lime@2-3-9 | grep c-lime| cut -c 15-`
-
 ../../configure --enable-comms=mpi-auto \
 --with-lime=$CLIME \
 --enable-unified=no \
@@ -10,13 +8,13 @@ CLIME=`spack find --paths c-lime@2-3-9 | grep c-lime| cut -c 15-`
 --disable-gparity \
 --disable-fermion-reps \
 --enable-simd=GPU \
---with-openssl=$OPENSSL \
 --with-gmp=$GMP \
 --with-mpfr=$MPFR \
+--with-openssl=$OPENSSL \
 --disable-fermion-reps \
 CXX=hipcc MPICXX=mpicxx \
 CXXFLAGS="-fPIC -I${ROCM_PATH}/include/ -I${MPICH_DIR}/include " \
- LDFLAGS="-L${ROCM_PATH}/lib -L${MPICH_DIR}/lib -lmpi -lmpi_gtl_hsa -lhipblas -lrocsolver -lrocblas -lhipfft -lamdhip64"
+ LDFLAGS="-L${ROCM_PATH}/lib -L${MPICH_DIR}/lib -lmpi -lmpi_gtl_hsa -lhipblas -lrocblas -lhipfft -lamdhip64"
 
 
 

systems/Frontier/schur_f1.slurm

@@ -1,82 +0,0 @@
-#!/bin/bash -l
-
-#SBATCH --job-name=schurF1
-#SBATCH --nodes=1
-#SBATCH --ntasks-per-node=8
-#SBATCH --cpus-per-task=7
-#SBATCH --gpus-per-node=8
-#SBATCH --time=00:15:00
-#SBATCH --account=phy157_dwf
-#SBATCH --gpu-bind=none
-#SBATCH --exclusive
-#SBATCH --mem=0
-#SBATCH -q debug
-
-#####################################################################
-# F1 (schur_recursive_inverse_plan.txt 4C): RecursiveSchurInverse on
-# GPU + GPU-aware comms -- no lattice physics, communicator only.
-# Laptop-proven semantics (Eigen oracle, ranks 1-4); this run
-# adjudicates ONLY the device paths: rocSOLVER zgetri leaves, ComplexD
-# strided gemmBatched merges, device panels, thrust-complex NegateCopy
-# kernel, multi-GCD GlobalSumVector panels.
-#
-# PRE-REGISTERED: every stage PASS; T4 certificates ~1e-14..1e-10
-# (eps64 growth model, as laptop); at P=1-equivalent leaves rocSOLVER
-# != Eigen so |X-Xref| = exact 0 is NOT expected, <=1e-12 is healthy.
-#####################################################################
-
-cat << EOF > select_gpu
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export NUMA_MAP=(3 3 1 1 2 2 0 0)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-CMD="numactl -m \$NUMA -N \$NUMA  \$*"
-exec \$CMD
-EOF
-chmod +x ./select_gpu
-
-# NUMA_MAP is only valid at 8 ranks/node (see aggregate_io.slurm);
-# below 8 use the GCD-only wrapper and leave placement to Slurm.
-cat << EOF > select_gcd
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-exec \$*
-EOF
-chmod +x ./select_gcd
-SELECT=$SLURM_SUBMIT_DIR/select_gpu
-SELECT_NONUMA=$SLURM_SUBMIT_DIR/select_gcd
-
-root=$HOME/ParallelIO/systems/Frontier/
-source ${root}/sourceme-rocm7.2.sh
-
-export OMP_NUM_THREADS=7
-export MPICH_GPU_SUPPORT_ENABLED=1
-
-BIN=$SLURM_SUBMIT_DIR/tests/debug/Test_schur_inverse
-
-run () {   # run <ranks-per-node> <grid> <mpi> <comment>
-  local rpn=$1 gr=$2 mp=$3 note=$4 wrap=$SELECT
-  if [ $rpn -ne 8 ]; then wrap=$SELECT_NONUMA; fi
-  echo
-  echo "==================================================================="
-  echo "=== F1  n=$rpn  grid=$gr  mpi=$mp   $note"
-  echo "==================================================================="
-  srun -N1 -n$rpn --ntasks-per-node=$rpn $wrap $BIN --grid $gr --mpi $mp \
-       --accelerator-threads 8 --shm 2048 --device-mem 32000
-  echo "=== exit $?"
-}
-
-# Rank-count sweep: even splits, the uneven n=3 case, and the full node.
-run 2 16.16.16.16 1.1.1.2 "even split, 2 GCD"
-run 3 16.16.16.24 1.1.1.3 "UNEVEN split, 3 GCD"
-run 4 16.16.16.16 1.1.1.4 "even split, 4 GCD"
-run 8 16.16.16.16 1.1.1.8 "full node, 8 GCD, depth-3 tree"
-
-echo
-echo "=== F1 done"

systems/Frontier/schur_f2.slurm

@@ -1,92 +0,0 @@
-#!/bin/bash -l
-
-#SBATCH --job-name=schurF2
-#SBATCH --nodes=1
-#SBATCH --ntasks-per-node=8
-#SBATCH --cpus-per-task=7
-#SBATCH --gpus-per-node=8
-#SBATCH --time=00:30:00
-#SBATCH --account=phy157_dwf
-#SBATCH --gpu-bind=none
-#SBATCH --exclusive
-#SBATCH --mem=0
-#SBATCH -q debug
-
-#####################################################################
-# F2 (schur_recursive_inverse_plan.txt 4C): the full DenseCoarseMatrix
-# glue on GPU with a REAL coarsening -- DWF MdagM + 0.5 shift, random
-# aggregation nbasis=4, per-dim blocks {4,4,3,3}, coarse 3.3.4.4/rank
-# (N = 576 * nranks).  The test binary FORCES DENSE_SCHUR=2 audit mode
-# and DENSE_PANEL_BYTES=65536 internally (setenv in main), so both
-# invert paths run and are compared on the same imported A; multi-chunk
-# panel gathers are exercised through the glue.
-#
-# First GPU execution in anger of: GRID_SIMT-guarded CoarsenOperator
-# population kernel; fresh stencil->dense import (fp32 + fp64) with the
-# transposed-contraction fix; fp64 rank-major import; distributed fp64
-# recursion; single terminal fp32 rounding; boss rocSOLVER single-GCD
-# path as the audit oracle.
-#
-# PRE-REGISTERED (laptop values, expect same magnitudes): operator
-# health herm-dev ~1e-15; import certificate ~1e-8; fp64 import
-# certificate ~5e-7 few-ulp; AUDIT rel ~3e-7 (pure fp32 rounding);
-# VERIFY ~1.6e-7; T6 ALL PASS.
-#####################################################################
-
-cat << EOF > select_gpu
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export NUMA_MAP=(3 3 1 1 2 2 0 0)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-CMD="numactl -m \$NUMA -N \$NUMA  \$*"
-exec \$CMD
-EOF
-chmod +x ./select_gpu
-
-# NUMA_MAP is only valid at 8 ranks/node (see aggregate_io.slurm);
-# below 8 use the GCD-only wrapper and leave placement to Slurm.
-cat << EOF > select_gcd
-#!/bin/bash
-export GPU_MAP=(0 1 2 3 7 6 5 4)
-export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
-unset ROCR_VISIBLE_DEVICES
-export HIP_VISIBLE_DEVICES=\$GPU
-exec \$*
-EOF
-chmod +x ./select_gcd
-SELECT=$SLURM_SUBMIT_DIR/select_gpu
-SELECT_NONUMA=$SLURM_SUBMIT_DIR/select_gcd
-
-root=$HOME/ParallelIO/systems/Frontier/
-source ${root}/sourceme-rocm7.2.sh
-
-export OMP_NUM_THREADS=7
-export MPICH_GPU_SUPPORT_ENABLED=1
-
-BIN=$SLURM_SUBMIT_DIR/tests/debug/Test_schur_dense_coarse
-
-run () {   # run <ranks-per-node> <grid> <mpi> <comment>
-  local rpn=$1 gr=$2 mp=$3 note=$4 wrap=$SELECT
-  if [ $rpn -ne 8 ]; then wrap=$SELECT_NONUMA; fi
-  echo
-  echo "==================================================================="
-  echo "=== F2  n=$rpn  grid=$gr  mpi=$mp   $note"
-  echo "==================================================================="
-  srun -N1 -n$rpn --ntasks-per-node=$rpn $wrap $BIN --grid $gr --mpi $mp \
-       --accelerator-threads 8 --shm 2048 --device-mem 32000
-  echo "=== exit $?"
-}
-
-# Uniform local fine volume 12.12.12.12 per rank (t = 12 * nranks).
-# n=2 covers the padded-Extract branch; n=8 the full node, N=4608.
-# The n=3 uneven case is laptop-proven; commented to keep the debug
-# slot short -- uncomment for the full sweep.
-run 2 12.12.12.24 1.1.1.2 "2 GCD, padded Extract on t, N=1152"
-#run 3 12.12.12.36 1.1.1.3 "3 GCD, UNEVEN, N=1728"
-run 8 12.12.12.96 1.1.1.8 "full node, 8 GCD, N=4608"
-
-echo
-echo "=== F2 done"

systems/Frontier/sourceme-rocm7.0.sh

@@ -1,13 +0,0 @@
-
-echo spack
-. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
-
-
-module load cce/21.0.0
-module load cpe/26.03
-module load rocm/7.0.2
-
-export LD_LIBRARY_PATH=/opt/rocm-7.0.2/lib/llvm/lib/:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CRAY_LD_LIBRARY_PATH
-
-module load emacs

systems/Frontier/sourceme-rocm7.2.sh

@@ -1,15 +0,0 @@
-
-echo spack
-. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
-
-export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
-export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
-export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
-export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
-
-module load cce/21.0.0
-module load cpe/26.03
-module load rocm/7.2.0
-export LD_LIBRARY_PATH=/opt/rocm-7.2.0/lib/llvm/lib/:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CRAY_LD_LIBRARY_PATH
-module load emacs

systems/Frontier/sourceme.sh

@@ -7,9 +7,8 @@ export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
 export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
 export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
 
-module load cce/20.0.0
-module load cpe/25.09
-module load rocm/6.4.0
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.0.2
 export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=/opt/rocm-6.4.0/lib/llvm/lib/:$LD_LIBRARY_PATH
-module load emacs
+export LD_LIBRARY_PATH=/opt/rocm-7.0.2/lib/llvm/lib/:$LD_LIBRARY_PATH

systems/mac-arm/config-command-mpi

@@ -3,15 +3,14 @@
 CXX=mpicxx ../../configure \
 	   --enable-simd=GEN \
 	   --enable-comms=mpi-auto \
-	   --enable-Sp=no \
+	   --enable-unified=yes \
 	   --disable-fermion-reps \
 	   --disable-gparity \
-	   --with-fftw=$FFTW \
-	   --enable-unified=yes \
-	   --prefix /Users/peterboyle/QCD/vtk/Grid/install \
+	   --prefix /Users/peterboyle/QCD/Grid-install \
 	   --with-lime=$CLIME \
 	   --with-openssl=$OPENSSL \
 	   --with-gmp=$GMP \
 	   --with-mpfr=$MPFR \
-	   --disable-debug 
+	   --with-fftw=$FFTW \
+	   --disable-debug
 

systems/mac-arm/sourceme.sh

@@ -1,11 +1,7 @@
 source /Users/peterboyle/QCD//Spack/spack//share/spack/setup-env.sh
-
 export FFTW=`spack find --paths fftw    | grep ^fftw   | awk '{print $2}' `
-#export HDF5=`spack find --paths hdf5+cxx   | grep ^hdf5   | awk '{print $2}' `
 export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
 export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
 export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
 export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
 
-export LD_LIBRARY_PATH=$MPFR/lib:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=$GMP/lib:$LD_LIBRARY_PATH

tests/IO/Test_aggregate_io.cc

@@ -1,458 +0,0 @@
-    /*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./tests/IO/Test_aggregate_io.cc
-
-    Copyright (C) 2015
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// Correctness and performance test for BINARYIO_AGGREGATE.
-//
-// Correctness, per aggregateTargetBytes:
-//   1. write via the MPI-IO lexicographic path      -> ref.bin
-//   2. write via the aggregate path                 -> agg.bin
-//   3. the two files must be byte identical         <- proves the layout matches
-//   4. read agg.bin back through the aggregate path <- proves the mirror inverts
-//   5. write and read the non-lexicographic path    -> raw.bin.  Its layout is
-//      different by construction (each rank owns one contiguous segment in
-//      rank order) so it cannot be compared byte for byte, but the NERSC and
-//      SciDAC checksums are computed from the global site index and are
-//      therefore layout independent: they must match the other two paths.
-//   6. a record written over a longer pre-existing file must leave the file at
-//      exactly offset+payload, with no trailing fragment of the old contents
-//
-// Performance: three paths, both directions, timed with the client page cache
-// dropped before every read so that a read back reports filesystem bandwidth
-// rather than memory bandwidth.  The non-lexicographic path is the zero
-// overhead reference: no transposition, no layout independence, one disjoint
-// contiguous segment per rank, which is the arrangement that reaches full
-// filesystem bandwidth on a leadership machine.  It is the upper bound the
-// other two are trying to approach.
-//
-// Options:
-//   --aggregate-target <bytes>   sweep this one target only (default: sweep
-//                                1, 1024, 64K, 4M)
-//   --io-reps <n>                repetitions in the performance section
-//                                (default 3; 0 disables it)
-//   --io-no-correctness          skip the correctness section, which reads the
-//                                whole file on one rank and is not affordable
-//                                at very large volume
-//   --io-read-only               time reads only, of files left in place by an
-//                                earlier job.  Reading back what this job just
-//                                wrote measures the client page cache; a fresh
-//                                allocation pointed at the same directory is
-//                                the only way to get a cold read without root.
-//
-// The exchange is only meaningfully exercised when the fast dimensions are
-// split across ranks; --mpi 1.1.X.Y leaves the rows of size one and the test
-// then passes vacuously.  Non-uniform AllToAllV counts additionally need an
-// odd process factor in a fast dimension and a small local volume.
-
-#include <Grid/Grid.h>
-#include <fcntl.h>
-
-using namespace Grid;
-
-/////////////////////////////////////////////////////////////////////////////
-// Compare in chunks.  Slurping both files into memory is fine for a few MB
-// and fatal for the multi-GB records this test is meant to reach.
-/////////////////////////////////////////////////////////////////////////////
-static bool FilesIdentical(std::string a,std::string b)
-{
-  std::ifstream fa(a,std::ios::binary), fb(b,std::ios::binary);
-  if ( !fa.good() || !fb.good() ) {
-    std::cout<<GridLogMessage<<"  could not open "<<a<<" and/or "<<b<<std::endl;
-    return false;
-  }
-  fa.seekg(0,std::ios::end); fb.seekg(0,std::ios::end);
-  uint64_t sa = (uint64_t)fa.tellg(), sb = (uint64_t)fb.tellg();
-  if ( sa != sb ) {
-    std::cout<<GridLogMessage<<"  size mismatch "<<sa<<" vs "<<sb<<std::endl;
-    return false;
-  }
-  fa.seekg(0,std::ios::beg); fb.seekg(0,std::ios::beg);
-
-  const uint64_t chunk = 8*1024*1024;
-  std::vector<char> va(chunk), vb(chunk);
-  uint64_t done=0;
-  while ( done < sa ) {
-    uint64_t n = std::min(chunk,sa-done);
-    fa.read(&va[0],n);
-    fb.read(&vb[0],n);
-    for(uint64_t i=0;i<n;i++){
-      if ( va[i]!=vb[i] ) {
-        std::cout<<GridLogMessage<<"  first differing byte at "<<done+i<<" of "<<sa<<std::endl;
-        return false;
-      }
-    }
-    done += n;
-  }
-  return true;
-}
-
-/////////////////////////////////////////////////////////////////////////////
-// Reading back a file we have just written measures the client page cache,
-// not the filesystem: the earlier runs of this test reported 8 GB/s on reads
-// and ~1 GB/s on writes for the same data.  POSIX_FADV_DONTNEED asks the
-// kernel to drop the cached pages for the file.  It is advisory and every
-// rank must do it, since each client caches independently, so treat this as
-// best effort rather than a guarantee of a cold read.
-/////////////////////////////////////////////////////////////////////////////
-static void DropCache(GridBase *grid,std::string file)
-{
-  grid->Barrier();
-  int fd = ::open(file.c_str(),O_RDONLY);
-  if ( fd >= 0 ) {
-#ifdef POSIX_FADV_DONTNEED
-    ::posix_fadvise(fd,0,0,POSIX_FADV_DONTNEED);
-#endif
-    ::close(fd);
-  }
-  grid->Barrier();
-}
-
-static uint64_t OptionU64(int argc,char **argv,const char *opt,uint64_t def)
-{
-  if ( GridCmdOptionExists(argv,argv+argc,opt) ) {
-    std::string arg = GridCmdOptionPayload(argv,argv+argc,opt);
-    return (uint64_t)std::stoull(arg);
-  }
-  return def;
-}
-
-int main(int argc,char **argv)
-{
-  Grid_init(&argc,&argv);
-
-  Coordinate latt = GridDefaultLatt();
-  Coordinate simd = GridDefaultSimd(Nd,vComplexD::Nsimd());
-  Coordinate mpi  = GridDefaultMpi();
-  GridCartesian grid(latt,simd,mpi);
-
-  typedef vLorentzColourMatrixD vobj;
-  typedef LorentzColourMatrixD  sobj;
-
-  GridParallelRNG pRNG(&grid);
-  pRNG.SeedFixedIntegers(std::vector<int>({1,2,3,4}));
-  LatticeGaugeFieldD Umu(&grid);
-  random(pRNG,Umu);
-
-  BinarySimpleMunger<sobj,sobj> munge;
-  const std::string format("IEEE64BIG");
-
-  const int lex = BinaryIO::BINARYIO_LEXICOGRAPHIC;
-  const int agg = BinaryIO::BINARYIO_LEXICOGRAPHIC|BinaryIO::BINARYIO_AGGREGATE;
-  const int raw = 0;   // no BINARYIO_LEXICOGRAPHIC: contiguous segment per rank
-
-  uint64_t payload = (uint64_t)grid._gsites*sizeof(sobj);
-
-  std::vector<uint64_t> targets = {1, 1024, 64*1024, 4*1024*1024};
-  if ( GridCmdOptionExists(argv,argv+argc,"--aggregate-target") ) {
-    targets.clear();
-    targets.push_back(OptionU64(argc,argv,"--aggregate-target",4*1024*1024));
-  }
-  uint64_t reps        = OptionU64(argc,argv,"--io-reps",3);
-  bool     correctness = !GridCmdOptionExists(argv,argv+argc,"--io-no-correctness");
-  // Read only: time reads of files left by an earlier job.  The only way to
-  // get a cold client cache without root is to read on an allocation that did
-  // not write the data, so run one job to write and a second, pointed at the
-  // same directory, with this flag.
-  bool     readonly    = GridCmdOptionExists(argv,argv+argc,"--io-read-only");
-  if ( readonly ) correctness = false;
-
-  std::cout<<GridLogMessage<<"Record payload "<<payload<<" bytes = "
-           <<payload/1024./1024.<<" MB, "
-           <<payload/(RealD)grid.ProcessorCount()/1024./1024.<<" MB/rank"<<std::endl;
-
-  int failures=0;
-
-  //////////////////////////////////////////////////////////////////////////
-  // Correctness
-  //////////////////////////////////////////////////////////////////////////
-  if ( correctness ) for(auto target : targets){
-
-    std::cout<<GridLogMessage<<"=== correctness, aggregateTargetBytes = "<<target<<" ==="<<std::endl;
-
-    uint32_t n1,a1,b1, n2,a2,b2, n3,a3,b3;
-    uint64_t off;
-
-    // Start from a clean slate.  The aggregate path sets the file length to
-    // exactly offset+payload; the MPI-IO path (MPI_MODE_CREATE) leaves any
-    // pre-existing tail in place.  Comparing stale files would therefore
-    // report a size mismatch that says nothing about the payload.
-    if ( grid.IsBoss() ) { ::unlink("ref.bin"); ::unlink("agg.bin"); ::unlink("raw.bin"); }
-    grid.Barrier();
-
-    off=0;
-    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"ref.bin",munge,off,format,n1,a1,b1,lex);
-
-    BinaryIO::aggregateTargetBytes = target;
-    off=0;
-    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"agg.bin",munge,off,format,n2,a2,b2,agg);
-
-    grid.Barrier();
-
-    if ( grid.IsBoss() ) {
-      if ( !FilesIdentical("ref.bin","agg.bin") ) {
-        std::cout<<GridLogError<<"  FAIL: aggregate file differs from lexicographic file"<<std::endl;
-        failures++;
-      } else {
-        std::cout<<GridLogMessage<<"  files byte identical"<<std::endl;
-      }
-    }
-    if ( (n1!=n2)||(a1!=a2)||(b1!=b2) ) {
-      std::cout<<GridLogError<<"  FAIL: checksum mismatch between paths"<<std::endl;
-      failures++;
-    }
-    // writeLatticeObject takes offset by value, so the out-parameter that
-    // IOobject sets never reaches us here and cannot be checked directly.
-    // The observable equivalent is the file length: both paths must leave the
-    // record ending at exactly offset+payload.
-    if ( grid.IsBoss() ) {
-      for(auto f : {std::string("ref.bin"),std::string("agg.bin")}){
-        std::ifstream fs(f,std::ios::binary|std::ios::ate);
-        uint64_t sz = (uint64_t)fs.tellg();
-        if ( sz != payload ) {
-          std::cout<<GridLogError<<"  FAIL: "<<f<<" is "<<sz<<" bytes, expected "<<payload<<std::endl;
-          failures++;
-        }
-      }
-    }
-
-    LatticeGaugeFieldD Uchk(&grid);
-    DropCache(&grid,"agg.bin");
-    off=0;
-    BinaryIO::readLatticeObject<vobj,sobj>(Uchk,"agg.bin",munge,off,format,n3,a3,b3,agg);
-    if ( (n3!=n1)||(a3!=a1)||(b3!=b1) ) {
-      std::cout<<GridLogError<<"  FAIL: read back checksum mismatch"<<std::endl;
-      failures++;
-    }
-    Uchk = Uchk - Umu;
-    RealD residual = norm2(Uchk);
-    std::cout<<GridLogMessage<<"  read back residual "<<residual<<std::endl;
-    if ( residual != 0.0 ) {
-      std::cout<<GridLogError<<"  FAIL: read back does not reproduce the field"<<std::endl;
-      failures++;
-    }
-
-    ////////////////////////////////////////////////////////////////////////
-    // Non-lexicographic.  Different file layout by construction, so compare
-    // by checksum and by round trip rather than by bytes.
-    ////////////////////////////////////////////////////////////////////////
-    uint32_t n4,a4,b4, n5,a5,b5;
-    off=0;
-    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"raw.bin",munge,off,format,n4,a4,b4,raw);
-    grid.Barrier();
-    if ( (n4!=n1)||(a4!=a1)||(b4!=b1) ) {
-      std::cout<<GridLogError<<"  FAIL: raw path checksum differs; the NERSC and"
-               <<" SciDAC checksums are layout independent and must agree"<<std::endl;
-      failures++;
-    }
-    DropCache(&grid,"raw.bin");
-    off=0;
-    BinaryIO::readLatticeObject<vobj,sobj>(Uchk,"raw.bin",munge,off,format,n5,a5,b5,raw);
-    if ( (n5!=n1)||(a5!=a1)||(b5!=b1) ) {
-      std::cout<<GridLogError<<"  FAIL: raw read back checksum mismatch"<<std::endl;
-      failures++;
-    }
-    Uchk = Uchk - Umu;
-    RealD rawresidual = norm2(Uchk);
-    std::cout<<GridLogMessage<<"  raw read back residual "<<rawresidual<<std::endl;
-    if ( rawresidual != 0.0 ) {
-      std::cout<<GridLogError<<"  FAIL: raw read back does not reproduce the field"<<std::endl;
-      failures++;
-    }
-  }
-
-  //////////////////////////////////////////////////////////////////////////
-  // Truncation.  offset!=0 is the case that matters: it is what ILDG and
-  // NERSC use, and it is the branch that used to only ever grow the file.
-  // The oversized starting file is made by extending a sparse one rather
-  // than writing padding from a single rank, which does not scale.
-  //////////////////////////////////////////////////////////////////////////
-  if ( correctness ) {
-    BinaryIO::aggregateTargetBytes = 4*1024*1024;
-    for(uint64_t testOffset : {(uint64_t)0, (uint64_t)1024}){
-
-      uint64_t expect = testOffset + payload;
-
-      if ( grid.IsBoss() ) {
-        { std::ofstream create("trunc.bin",std::ios::binary|std::ios::out); create.close(); }
-        int ierr = ::truncate("trunc.bin",(off_t)(expect+65536));
-        GRID_ASSERT(ierr==0);
-      }
-      grid.Barrier();
-
-      uint32_t n,a,b;
-      uint64_t off = testOffset;
-      BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"trunc.bin",munge,off,format,n,a,b,agg);
-      grid.Barrier();
-
-      if ( grid.IsBoss() ) {
-        std::ifstream f("trunc.bin",std::ios::binary|std::ios::ate);
-        uint64_t sz = (uint64_t)f.tellg();
-        f.close();
-        if ( sz != expect ) {
-          std::cout<<GridLogError<<"  FAIL: offset "<<testOffset<<" left file "<<sz
-                   <<" bytes, expected "<<expect<<std::endl;
-          failures++;
-        } else {
-          std::cout<<GridLogMessage<<"  truncation ok at offset "<<testOffset
-                   <<": file is exactly "<<sz<<" bytes"<<std::endl;
-        }
-      }
-    }
-  }
-
-  //////////////////////////////////////////////////////////////////////////
-  // Performance.  Four numbers per repetition: write and read, old path and
-  // new.  Reads are preceded by a cache drop; writes are not, so a write
-  // number is "time to hand the data to the client cache and close", the
-  // same convention for both paths.
-  //////////////////////////////////////////////////////////////////////////
-  if ( reps ) {
-
-    LatticeGaugeFieldD Uio(&grid);
-    uint32_t n,a,b;
-
-    // The NERSC and SciDAC checksums are computed from the global site index,
-    // so all three layouts must produce the same values.  This costs nothing
-    // and is the only correctness check available at a volume where the byte
-    // for byte comparison (single rank, whole file) is unaffordable.
-    uint32_t cn[6],ca[6],cb[6];
-    auto agreeing = [&](const char *what,int lo,int hi){
-      for(int i=lo+1;i<=hi;i++){
-        if ( (cn[i]!=cn[lo])||(ca[i]!=ca[lo])||(cb[i]!=cb[lo]) ) {
-          std::cout<<GridLogError<<"  FAIL: "<<what<<" checksums disagree between paths"<<std::endl;
-          return false;
-        }
-      }
-      return true;
-    };
-
-    for(auto target : targets){
-
-      BinaryIO::aggregateTargetBytes = target;
-      std::cout<<GridLogMessage<<"=== performance, aggregateTargetBytes = "<<target<<" ==="<<std::endl;
-
-      std::vector<RealD> wref,wagg,wraw,rref,ragg,rraw;
-
-      for(uint64_t rep=0;rep<reps;rep++){
-
-        uint64_t off;
-
-        // Unlink only before the first repetition.  Lustre metadata cost is
-        // per file, not per byte, so rep 0 reports "create the file and write
-        // it" and the later reps report the steady state of overwriting an
-        // existing file -- which is what a multi record file does for every
-        // record after the first, and what production actually looks like.
-        if ( (rep==0) && !readonly && grid.IsBoss() ) {
-          ::unlink("ref.bin"); ::unlink("agg.bin"); ::unlink("raw.bin");
-        }
-        grid.Barrier();
-
-        if ( !readonly ) {
-
-        off=0;
-        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"ref.bin",munge,off,format,n,a,b,lex);
-        wref.push_back(BinaryIO::lastPerf.mbytesPerSecond);
-        cn[0]=n; ca[0]=a; cb[0]=b;
-
-        off=0;
-        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"agg.bin",munge,off,format,n,a,b,agg);
-        wagg.push_back(BinaryIO::lastPerf.mbytesPerSecond);
-        cn[1]=n; ca[1]=a; cb[1]=b;
-
-        off=0;
-        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"raw.bin",munge,off,format,n,a,b,raw);
-        wraw.push_back(BinaryIO::lastPerf.mbytesPerSecond);
-        cn[2]=n; ca[2]=a; cb[2]=b;
-        if ( !agreeing("write",0,2) ) failures++;
-
-        }  // !readonly
-
-        DropCache(&grid,"ref.bin");
-        off=0;
-        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"ref.bin",munge,off,format,n,a,b,lex);
-        rref.push_back(BinaryIO::lastPerf.mbytesPerSecond);
-        cn[3]=n; ca[3]=a; cb[3]=b;
-
-        DropCache(&grid,"agg.bin");
-        off=0;
-        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"agg.bin",munge,off,format,n,a,b,agg);
-        ragg.push_back(BinaryIO::lastPerf.mbytesPerSecond);
-        cn[4]=n; ca[4]=a; cb[4]=b;
-
-        DropCache(&grid,"raw.bin");
-        off=0;
-        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"raw.bin",munge,off,format,n,a,b,raw);
-        rraw.push_back(BinaryIO::lastPerf.mbytesPerSecond);
-        cn[5]=n; ca[5]=a; cb[5]=b;
-        if ( !agreeing("read back",readonly?3:0,5) ) failures++;
-      }
-
-      if ( grid.IsBoss() ) {
-        auto report = [&](const char *name,std::vector<RealD> &v){
-          if ( v.empty() ) return;
-          RealD best=0, sum=0;
-          for(auto x : v){ if(x>best) best=x; sum+=x; }
-          // First sample includes file creation, later ones do not; quote both
-          // rather than a mean that mixes the two.
-          std::cout<<GridLogMessage<<"  PERF target="<<target<<" "<<name
-                   <<" best "<<best<<" MB/s, mean "<<sum/v.size()
-                   <<" MB/s, first(cold create) "<<v[0]<<" MB/s, samples";
-          for(auto x : v) std::cout<<" "<<x;
-          std::cout<<std::endl;
-        };
-        report("write raw       ",wraw);   // zero overhead reference
-        report("write MPI-IO    ",wref);
-        report("write aggregate ",wagg);
-        report("read  raw       ",rraw);
-        report("read  MPI-IO    ",rref);
-        report("read  aggregate ",ragg);
-
-        // Fraction of the zero overhead reference that each layout preserving
-        // path achieves.  This is the number the whole exercise is about.
-        auto best = [](std::vector<RealD> &v){ RealD m=0; for(auto x:v) if(x>m) m=x; return m; };
-        if ( !wraw.empty() && best(wraw) > 0 ) {
-          std::cout<<GridLogMessage<<"  PERF target="<<target
-                   <<" write fraction of raw: MPI-IO "<<best(wref)/best(wraw)
-                   <<"  aggregate "<<best(wagg)/best(wraw)<<std::endl;
-        }
-        if ( !rraw.empty() && best(rraw) > 0 ) {
-          std::cout<<GridLogMessage<<"  PERF target="<<target
-                   <<" read  fraction of raw: MPI-IO "<<best(rref)/best(rraw)
-                   <<"  aggregate "<<best(ragg)/best(rraw)<<std::endl;
-        }
-      }
-    }
-  }
-
-  if ( grid.IsBoss() ) {
-    if ( failures ) std::cout<<GridLogError  <<failures<<" FAILURE(S)"<<std::endl;
-    else            std::cout<<GridLogMessage<<"ALL AGGREGATE IO TESTS PASSED"<<std::endl;
-  }
-
-  Grid_finalize();
-  return failures!=0;
-}

tests/Test_extended_meson_field.cc

@@ -0,0 +1,841 @@
+/*************************************************************************************
+
+Grid physics library, www.github.com/paboyle/Grid
+
+Source file: tests/Test_extended_meson_field.cc
+
+Copyright (C) 2015-2025
+
+Author: Peter Boyle <pboyle@bnl.gov>
+Author: Masaaki Tomii <masaaki.tomii@uconn.edu> (original Hadrons kernels)
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along
+with this program; if not, write to the Free Software Foundation, Inc.,
+51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+#include "disable_tests_without_instantiations.h"
+#ifdef ENABLE_FERMION_INSTANTIATIONS
+
+#include <Grid/Grid.h>
+#include <Grid/qcd/utils/A2Autils.h>
+
+using namespace Grid;
+
+typedef WilsonImplD FImpl;
+typedef typename FImpl::FermionField FermionField;
+typedef typename FImpl::SiteSpinor   vobj;
+typedef typename vobj::scalar_type   scalar_type;
+typedef typename vobj::vector_type   vector_type;
+typedef iSpinColourMatrix<vector_type> SpinColourMatrix_v;
+typedef iSpinColourVector<vector_type> SpinColourVector_v;
+typedef iSpinMatrix<vector_type>       SpinMatrix_v;
+typedef iSinglet<vector_type>          Scalar_v;
+typedef iSinglet<scalar_type>          Scalar_s;
+typedef Lattice<SpinColourMatrix_v>    PropagatorField;
+
+// CPU reference + optionally batched GEMM spatial sum, ported from
+// Hadrons/Modules/MContraction/A2AExtendedMesonField.hpp
+// (M. Tomii, mtomii/Hadrons:local-2025-edits). Hadrons infrastructure removed.
+// thread_for / CpuRead / orthogdim=3 preserved.
+class A2AExtendedMesonFieldRef
+{
+public:
+  // result is indexed [nt][N_i][N_j].
+  // use_blas=true replaces the scalar spatial accumulation with A2ASpatialSum.
+  static void compute(
+      Eigen::Tensor<ComplexD, 3> &result,
+      const std::vector<FermionField> &left,
+      const std::vector<FermionField> &right,
+      const std::vector<FermionField> &loop1,
+      const std::vector<FermionField> &loop2,
+      const std::vector<Gamma::Algebra> &gamma1,
+      const std::vector<Gamma::Algebra> &gamma2,
+      int type,
+      bool use_blas = false)
+  {
+    GridBase *grid = left[0].Grid();
+
+    const int orthogdim = 3;
+    int rd     = grid->_rdimensions[orthogdim];
+    int ld     = grid->_ldimensions[orthogdim];
+    int Nd     = grid->_ndimension;
+    int Nsimd  = grid->Nsimd();
+
+    int nt  = result.dimension(0);
+    int N_i = (int)left.size();
+    int N_j = (int)right.size();
+
+    std::string tag = std::string(use_blas ? "[blas" : "[ref ") + " type=" + std::to_string(type) + "]";
+    auto Tms = [](double us) { return us * 1e-3; };
+    double t0;
+
+    // ------------------------------------------------------------------
+    // Loop propagator: sum_k outerProduct(loop1[k], loop2[k])
+    // ------------------------------------------------------------------
+    t0 = usecond();
+    PropagatorField loop(grid);
+    loop = Zero();
+    for (unsigned int k = 0; k < loop1.size(); ++k)
+      loop += outerProduct(loop1[k], loop2[k]);
+    std::cout << GridLogMessage << tag << " loop_build:      " << Tms(usecond()-t0) << " ms\n";
+
+    // ------------------------------------------------------------------
+    // Pack conjugated left vectors
+    // ------------------------------------------------------------------
+    t0 = usecond();
+    std::vector<FermionField> leftv(N_i, grid);
+    for (int i = 0; i < N_i; i++)
+      leftv[i] = conjugate(left[i]);
+    std::cout << GridLogMessage << tag << " pack_left:       " << Tms(usecond()-t0) << " ms\n";
+
+    // ------------------------------------------------------------------
+    // Per-site loop contraction into PropagatorField tloop (type-dependent)
+    // ------------------------------------------------------------------
+    t0 = usecond();
+    PropagatorField tloop(grid);
+    tloop = Zero();
+    {
+      autoView(tloopv, tloop, CpuWrite);
+      autoView(loopv,  loop,  CpuRead);
+      if (type == 0) {
+        thread_for(ss, grid->oSites(), {
+          for (int s1 = 0; s1 < Ns; ++s1)
+          for (int s2 = 0; s2 < Ns; ++s2)
+            tloopv[ss]()(s1,s2)(0,0) = loopv[ss]()(s1,s2)(0,0)
+                                     + loopv[ss]()(s1,s2)(1,1)
+                                     + loopv[ss]()(s1,s2)(2,2);
+        });
+      }
+      if (type == 1) {
+        thread_for(ss, grid->oSites(), {
+          tloopv[ss] = Zero();
+          for (int mu = 0; mu < (int)gamma1.size(); ++mu)
+            tloopv[ss] = tloopv[ss] + Gamma(gamma1[mu]) * loopv[ss] * Gamma(gamma2[mu]);
+        });
+      }
+      if (type == 2) {
+        thread_for(ss, grid->oSites(), {
+          tloopv[ss] = Zero();
+          for (int mu = 0; mu < (int)gamma2.size(); ++mu) {
+            SpinColourMatrix_v tmp = Gamma(gamma2[mu]) * loopv[ss];
+            int s1 = mu / Ns;
+            int s2 = mu % Ns;
+            for (int c1 = 0; c1 < Nc; ++c1)
+            for (int c2 = 0; c2 < Nc; ++c2)
+              tloopv[ss]()(s1,s2)(c1,c2) = tmp()(0,0)(c1,c2) + tmp()(1,1)(c1,c2)
+                                          + tmp()(2,2)(c1,c2) + tmp()(3,3)(c1,c2);
+          }
+        });
+      }
+      if (type == 3) {
+        thread_for(ss, grid->oSites(), {
+          SpinMatrix_v spinLoop = Zero();
+          for (int s1 = 0; s1 < Ns; ++s1)
+          for (int s2 = 0; s2 < Ns; ++s2)
+            spinLoop()(s1,s2)() = loopv[ss]()(s1,s2)(0,0)
+                                + loopv[ss]()(s1,s2)(1,1)
+                                + loopv[ss]()(s1,s2)(2,2);
+          tloopv[ss] = Zero();
+          for (int mu = 0; mu < (int)gamma1.size(); ++mu) {
+            SpinMatrix_v tmp2 = Gamma(gamma1[mu]) * spinLoop * Gamma(gamma2[mu]);
+            for (int s1 = 0; s1 < Ns; ++s1)
+            for (int s2 = 0; s2 < Ns; ++s2)
+              tloopv[ss]()(s1,s2)(0,0) = tloopv[ss]()(s1,s2)(0,0) + tmp2()(s1,s2)();
+          }
+        });
+      }
+    }
+    std::cout << GridLogMessage << tag << " tloop:           " << Tms(usecond()-t0) << " ms\n";
+
+    // Select addLoopRight kernel for this type
+    std::function<void(SpinColourVector_v &,
+                       const SpinColourMatrix_v &,
+                       const SpinColourVector_v &,
+                       const std::vector<Gamma::Algebra> &,
+                       const std::vector<Gamma::Algebra> &)> addLoopRight;
+
+    if (type == 0) {
+      addLoopRight = [](SpinColourVector_v &lR,
+                        const SpinColourMatrix_v &loopm,
+                        const SpinColourVector_v &rightv,
+                        const std::vector<Gamma::Algebra> &g1,
+                        const std::vector<Gamma::Algebra> &g2) {
+        SpinMatrix_v spinLoop = Zero();
+        for (int s1 = 0; s1 < Ns; ++s1)
+        for (int s2 = 0; s2 < Ns; ++s2)
+          spinLoop()(s1,s2)() = loopm()(s1,s2)(0,0);
+        for (int mu = 0; mu < (int)g1.size(); ++mu) {
+          SpinMatrix_v GLoop = Gamma(g2[mu]) * spinLoop;
+          auto trGLoop = GLoop()(0,0)() + GLoop()(1,1)() + GLoop()(2,2)() + GLoop()(3,3)();
+          SpinColourVector_v Grightv = Gamma(g1[mu]) * rightv;
+          for (int s = 0; s < Ns; ++s)
+          for (int c = 0; c < Nc; ++c)
+            lR()(s)(c) += Grightv()(s)(c) * trGLoop;
+        }
+      };
+    }
+    if (type == 1) {
+      addLoopRight = [](SpinColourVector_v &lR,
+                        const SpinColourMatrix_v &loopm,
+                        const SpinColourVector_v &rightv,
+                        const std::vector<Gamma::Algebra> &g1,
+                        const std::vector<Gamma::Algebra> &g2) {
+        for (int s = 0; s < Ns; ++s)
+        for (int c = 0; c < Nc; ++c) {
+          lR()(s)(c)
+            += loopm()(s,0)(c,0) * rightv()(0)(0)
+            +  loopm()(s,0)(c,1) * rightv()(0)(1)
+            +  loopm()(s,0)(c,2) * rightv()(0)(2)
+            +  loopm()(s,1)(c,0) * rightv()(1)(0)
+            +  loopm()(s,1)(c,1) * rightv()(1)(1)
+            +  loopm()(s,1)(c,2) * rightv()(1)(2)
+            +  loopm()(s,2)(c,0) * rightv()(2)(0)
+            +  loopm()(s,2)(c,1) * rightv()(2)(1)
+            +  loopm()(s,2)(c,2) * rightv()(2)(2)
+            +  loopm()(s,3)(c,0) * rightv()(3)(0)
+            +  loopm()(s,3)(c,1) * rightv()(3)(1)
+            +  loopm()(s,3)(c,2) * rightv()(3)(2);
+        }
+      };
+    }
+    if (type == 2) {
+      addLoopRight = [](SpinColourVector_v &lR,
+                        const SpinColourMatrix_v &loopm,
+                        const SpinColourVector_v &rightv,
+                        const std::vector<Gamma::Algebra> &g1,
+                        const std::vector<Gamma::Algebra> &g2) {
+        for (int mu = 0; mu < (int)g1.size(); ++mu) {
+          int s1 = mu / Ns;
+          int s2 = mu % Ns;
+          SpinColourVector_v Grightv = Gamma(g1[mu]) * rightv;
+          for (int s = 0; s < Ns; ++s)
+          for (int c = 0; c < Nc; ++c)
+            lR()(s)(c) += loopm()(s1,s2)(c,0) * Grightv()(s)(0)
+                       +  loopm()(s1,s2)(c,1) * Grightv()(s)(1)
+                       +  loopm()(s1,s2)(c,2) * Grightv()(s)(2);
+        }
+      };
+    }
+    if (type == 3) {
+      addLoopRight = [](SpinColourVector_v &lR,
+                        const SpinColourMatrix_v &loopm,
+                        const SpinColourVector_v &rightv,
+                        const std::vector<Gamma::Algebra> &g1,
+                        const std::vector<Gamma::Algebra> &g2) {
+        for (int s = 0; s < Ns; ++s)
+        for (int c = 0; c < Nc; ++c)
+          lR()(s)(c) += loopm()(s,0)(0,0) * rightv()(0)(c)
+                     +  loopm()(s,1)(0,0) * rightv()(1)(c)
+                     +  loopm()(s,2)(0,0) * rightv()(2)(c)
+                     +  loopm()(s,3)(0,0) * rightv()(3)(c);
+      };
+    }
+
+    // ------------------------------------------------------------------
+    // Pack loopRight[j] = type-kernel(tloop, right[j]) per site
+    // ------------------------------------------------------------------
+    t0 = usecond();
+    std::vector<FermionField> loopRight(N_j, grid);
+    {
+      autoView(tlv, tloop, CpuRead);
+      for (int j = 0; j < N_j; j++) {
+        loopRight[j] = Zero();
+        autoView(lRv, loopRight[j], CpuWrite);
+        autoView(rv,  right[j],     CpuRead);
+        thread_for(ss, grid->oSites(), {
+          addLoopRight(lRv[ss], tlv[ss], rv[ss], gamma1, gamma2);
+        });
+      }
+    }
+    std::cout << GridLogMessage << tag << " pack_loopright:  " << Tms(usecond()-t0) << " ms\n";
+
+    if (use_blas) {
+      // ------------------------------------------------------------------
+      // BLAS path: A2ASpatialSum (Allocate + PackLeft + PackRight + Sum)
+      // ------------------------------------------------------------------
+      A2ASpatialSum<SpinColourVector_v> spatial_sum;
+      double t_blas_start = usecond();
+
+      t0 = usecond();
+      spatial_sum.Allocate(N_i, N_j, grid);
+      std::cout << GridLogMessage << tag << " Allocate:        " << Tms(usecond()-t0) << " ms\n";
+
+      t0 = usecond();
+      spatial_sum.PackLeft(leftv);
+      std::cout << GridLogMessage << tag << " PackLeft:        " << Tms(usecond()-t0) << " ms\n";
+
+      t0 = usecond();
+      spatial_sum.PackRight(loopRight);
+      std::cout << GridLogMessage << tag << " PackRight:       " << Tms(usecond()-t0) << " ms\n";
+
+      t0 = usecond();
+      spatial_sum.Sum(result);
+      std::cout << GridLogMessage << tag << " Sum (GEMM+MPI):  " << Tms(usecond()-t0) << " ms\n";
+
+      std::cout << GridLogMessage << tag << " A2ASpatialSum:   " << Tms(usecond()-t_blas_start) << " ms  [TOTAL]\n";
+
+    } else {
+      // ------------------------------------------------------------------
+      // Reference path: SIMD spatial sum + scalar extraction
+      // ------------------------------------------------------------------
+      int MFrvol = rd * N_i * N_j;
+      int MFlvol = ld * N_i * N_j;
+
+      Vector<Scalar_v> lvSum(MFrvol);
+      thread_for(r, MFrvol, { lvSum[r] = Zero(); });
+
+      t0 = usecond();
+      {
+        int e1     = grid->_slice_nblock[orthogdim];
+        int e2     = grid->_slice_block [orthogdim];
+        int stride = grid->_slice_stride[orthogdim];
+        using LView = decltype(leftv[0].View(CpuRead));
+        using RView = decltype(loopRight[0].View(CpuRead));
+        std::vector<LView> lv_views;
+        std::vector<RView> lr_views;
+        lv_views.reserve(N_i);
+        lr_views.reserve(N_j);
+        for (int i = 0; i < N_i; i++) lv_views.push_back(leftv[i].View(CpuRead));
+        for (int j = 0; j < N_j; j++) lr_views.push_back(loopRight[j].View(CpuRead));
+        thread_for(r, rd, {
+          int so   = r * grid->_ostride[orthogdim];
+          int base = N_i * N_j * r;
+          for (int n = 0; n < e1; n++)
+          for (int b = 0; b < e2; b++) {
+            int ss = so + n * stride + b;
+            for (int ii = 0; ii < N_i; ii++)
+            for (int jj = 0; jj < N_j; jj++) {
+              int idx = jj + N_j * ii + base;
+              for (int s = 0; s < Ns; ++s)
+              for (int c = 0; c < Nc; ++c)
+                lvSum[idx]()()() += lv_views[ii][ss]()(s)(c) * lr_views[jj][ss]()(s)(c);
+            }
+          }
+        });
+        for (auto &v : lv_views) v.ViewClose();
+        for (auto &v : lr_views) v.ViewClose();
+      }
+      std::cout << GridLogMessage << tag << " spatial_sum:     " << Tms(usecond()-t0) << " ms\n";
+
+      Vector<Scalar_s> lsSum(MFlvol);
+      thread_for(r, MFlvol, { lsSum[r] = scalar_type(0.0); });
+
+      t0 = usecond();
+      thread_for(rt, rd, {
+        Coordinate icoor(Nd);
+        ExtractBuffer<Scalar_s> extracted(Nsimd);
+        for (int ii = 0; ii < N_i; ii++)
+        for (int jj = 0; jj < N_j; jj++) {
+          int ij_rdx = jj + N_j * (ii + N_i * rt);
+          extract(lvSum[ij_rdx], extracted);
+          for (int idx = 0; idx < Nsimd; idx++) {
+            grid->iCoorFromIindex(icoor, idx);
+            int ldx    = rt + icoor[orthogdim] * rd;
+            int ij_ldx = jj + N_j * (ii + N_i * ldx);
+            lsSum[ij_ldx] = lsSum[ij_ldx] + extracted[idx];
+          }
+        }
+      });
+      std::cout << GridLogMessage << tag << " simd_extract:    " << Tms(usecond()-t0) << " ms\n";
+
+      int pd = grid->_processors[orthogdim];
+      int pc = grid->_processor_coor[orthogdim];
+
+      t0 = usecond();
+      Vector<ComplexD> cache(nt * N_i * N_j, ComplexD(0.0));
+      for (int lt = 0; lt < ld; lt++)
+      for (int pt = 0; pt < pd; pt++) {
+        int t = lt + pt * ld;
+        for (int ii = 0; ii < N_i; ii++)
+        for (int jj = 0; jj < N_j; jj++) {
+          if (pt == pc) {
+            int ij_ldx = jj + N_j * (ii + N_i * lt);
+            cache[jj + N_j * (ii + N_i * t)] = lsSum[ij_ldx]()()();
+          }
+        }
+      }
+      grid->GlobalSumVector(cache.data(), nt * N_i * N_j);
+      std::cout << GridLogMessage << tag << " globalsum:       " << Tms(usecond()-t0) << " ms\n";
+
+      for (int t  = 0; t  < nt;  t++)
+      for (int ii = 0; ii < N_i; ii++)
+      for (int jj = 0; jj < N_j; jj++)
+        result(t, ii, jj) = cache[jj + N_j * (ii + N_i * t)];
+    }
+  }
+};
+
+// ================================================================
+// Free-function GPU kernels — accelerator_for, v(ss) reads,
+// coalescedWrite writes, vobj-level arithmetic throughout.
+// Gamma arrays passed as Vector<Gamma::Algebra> (UVM).
+// ================================================================
+
+void A2ALoopPropagator(PropagatorField &loop,
+                       const std::vector<FermionField> &loop1,
+                       const std::vector<FermionField> &loop2)
+{
+  int Nk      = (int)loop1.size();
+  uint64_t oSites = loop.Grid()->oSites();
+  int Nsimd   = SpinColourVector_v::Nsimd();
+
+  typedef decltype(loop1[0].View(AcceleratorRead)) View;
+  std::vector<View> v1, v2;
+  v1.reserve(Nk); v2.reserve(Nk);
+  for (int k = 0; k < Nk; k++) {
+    v1.push_back(loop1[k].View(AcceleratorRead));
+    v2.push_back(loop2[k].View(AcceleratorRead));
+  }
+
+  deviceVector<SpinColourVector_v *> l1p(Nk), l2p(Nk);
+  for (int k = 0; k < Nk; k++) {
+    acceleratorPut(l1p[k], &v1[k][0]);
+    acceleratorPut(l2p[k], &v2[k][0]);
+  }
+
+  autoView(loopv, loop, AcceleratorWrite);
+  SpinColourVector_v **l1 = &l1p[0];
+  SpinColourVector_v **l2 = &l2p[0];
+  int lNk = Nk;
+  accelerator_for(ss, oSites, Nsimd, {
+    auto result = outerProduct(coalescedRead(l1[0][ss]), coalescedRead(l2[0][ss]));
+    for (int k = 1; k < lNk; k++)
+      result = result + outerProduct(coalescedRead(l1[k][ss]), coalescedRead(l2[k][ss]));
+    coalescedWrite(loopv[ss], result);
+  });
+}
+
+void A2APackLeftConjugated(FermionField &out, const FermionField &in)
+{
+  autoView(outv, out, AcceleratorWrite);
+  autoView(inv,  in,  AcceleratorRead);
+  uint64_t Osites = in.Grid()->oSites();
+  int Nsimd = SpinColourVector_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    coalescedWrite(outv[ss], conjugate(inv(ss)));
+  });
+}
+
+// Type 0: colour-trace stored in (s1,s2)(0,0)
+void A2ALoopLeftContractionType0(PropagatorField &tloop, const PropagatorField &loop)
+{
+  autoView(tloopv, tloop, AcceleratorWrite);
+  autoView(loopv,  loop,  AcceleratorRead);
+  uint64_t Osites = loop.Grid()->oSites();
+  int Nsimd = SpinColourMatrix_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    auto l = loopv(ss);
+    auto tmp = l; tmp = Zero();
+    for (int s1 = 0; s1 < Ns; ++s1)
+    for (int s2 = 0; s2 < Ns; ++s2)
+      tmp()(s1,s2)(0,0) = l()(s1,s2)(0,0) + l()(s1,s2)(1,1) + l()(s1,s2)(2,2);
+    coalescedWrite(tloopv[ss], tmp);
+  });
+}
+
+// Type 1: tloop = sum_mu Gamma(g1[mu]) * loop * Gamma(g2[mu])
+void A2ALoopLeftContractionType1(PropagatorField &tloop, const PropagatorField &loop,
+                                  const Vector<Gamma::Algebra> &gamma1,
+                                  const Vector<Gamma::Algebra> &gamma2)
+{
+  int ng = (int)gamma1.size();
+  const Gamma::Algebra *g1 = gamma1.data();
+  const Gamma::Algebra *g2 = gamma2.data();
+  autoView(tloopv, tloop, AcceleratorWrite);
+  autoView(loopv,  loop,  AcceleratorRead);
+  uint64_t Osites = loop.Grid()->oSites();
+  int Nsimd = SpinColourMatrix_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    auto l = loopv(ss);
+    auto tmp = l; tmp = Zero();
+    for (int mu = 0; mu < ng; ++mu)
+      tmp = tmp + Gamma(g1[mu]) * l * Gamma(g2[mu]);
+    coalescedWrite(tloopv[ss], tmp);
+  });
+}
+
+// Type 2: for mu=[0..ng), s1=mu/Ns, s2=mu%Ns;
+//         tloop(s1,s2)(c1,c2) = Tr_spin( Gamma(g2[mu]) * loop )(c1,c2)
+void A2ALoopLeftContractionType2(PropagatorField &tloop, const PropagatorField &loop,
+                                  const Vector<Gamma::Algebra> &gamma2)
+{
+  int ng = (int)gamma2.size();
+  const Gamma::Algebra *g2 = gamma2.data();
+  autoView(tloopv, tloop, AcceleratorWrite);
+  autoView(loopv,  loop,  AcceleratorRead);
+  uint64_t Osites = loop.Grid()->oSites();
+  int Nsimd = SpinColourMatrix_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    auto l = loopv(ss);
+    auto tmp = l; tmp = Zero();
+    for (int mu = 0; mu < ng; ++mu) {
+      auto gtmp = Gamma(g2[mu]) * l;
+      int s1 = mu / Ns;
+      int s2 = mu % Ns;
+      for (int c1 = 0; c1 < Nc; ++c1)
+      for (int c2 = 0; c2 < Nc; ++c2)
+        tmp()(s1,s2)(c1,c2) = gtmp()(0,0)(c1,c2) + gtmp()(1,1)(c1,c2)
+                            + gtmp()(2,2)(c1,c2) + gtmp()(3,3)(c1,c2);
+    }
+    coalescedWrite(tloopv[ss], tmp);
+  });
+}
+
+// Type 3: colour-trace → spin matrix → sum_mu G1*spinLoop*G2 stored in (s1,s2)(0,0)
+void A2ALoopLeftContractionType3(PropagatorField &tloop, const PropagatorField &loop,
+                                  const Vector<Gamma::Algebra> &gamma1,
+                                  const Vector<Gamma::Algebra> &gamma2)
+{
+  int ng = (int)gamma1.size();
+  const Gamma::Algebra *g1 = gamma1.data();
+  const Gamma::Algebra *g2 = gamma2.data();
+  autoView(tloopv, tloop, AcceleratorWrite);
+  autoView(loopv,  loop,  AcceleratorRead);
+  uint64_t Osites = loop.Grid()->oSites();
+  int Nsimd = SpinColourMatrix_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    typedef decltype(coalescedRead(loopv[0])) calcSCMatrix;
+    typedef iSpinMatrix<typename calcSCMatrix::vector_type> calcSpinMatrix;
+    auto l = loopv(ss);
+    calcSpinMatrix spinLoop; spinLoop = Zero();
+    for (int s1 = 0; s1 < Ns; ++s1)
+    for (int s2 = 0; s2 < Ns; ++s2)
+      spinLoop()(s1,s2)() = l()(s1,s2)(0,0) + l()(s1,s2)(1,1) + l()(s1,s2)(2,2);
+    auto tmp = l; tmp = Zero();
+    for (int mu = 0; mu < ng; ++mu) {
+      calcSpinMatrix tmp2 = Gamma(g1[mu]) * spinLoop * Gamma(g2[mu]);
+      for (int s1 = 0; s1 < Ns; ++s1)
+      for (int s2 = 0; s2 < Ns; ++s2)
+        tmp()(s1,s2)(0,0) = tmp()(s1,s2)(0,0) + tmp2()(s1,s2)();
+    }
+    coalescedWrite(tloopv[ss], tmp);
+  });
+}
+
+// Type 0: loopRight = sum_mu Tr(G2*spinLoop) * G1*right
+//   where spinLoop(s1,s2) = tloop(s1,s2)(0,0)
+void A2ALoopRightContractionType0(FermionField &loopRight,
+                                   const PropagatorField &tloop,
+                                   const FermionField &right,
+                                   const Vector<Gamma::Algebra> &gamma1,
+                                   const Vector<Gamma::Algebra> &gamma2)
+{
+  int ng = (int)gamma1.size();
+  const Gamma::Algebra *g1 = gamma1.data();
+  const Gamma::Algebra *g2 = gamma2.data();
+  autoView(lRv, loopRight, AcceleratorWrite);
+  autoView(tlv, tloop,     AcceleratorRead);
+  autoView(rv,  right,     AcceleratorRead);
+  uint64_t Osites = right.Grid()->oSites();
+  int Nsimd = SpinColourVector_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    typedef decltype(coalescedRead(rv[0]))  calcSCVector;
+    typedef decltype(coalescedRead(tlv[0])) calcSCMatrix;
+    typedef iSpinMatrix<typename calcSCMatrix::vector_type> calcSpinMatrix;
+    auto loopm  = tlv(ss);
+    auto rightv = rv(ss);
+    calcSpinMatrix spinLoop; spinLoop = Zero();
+    for (int s1 = 0; s1 < Ns; ++s1)
+    for (int s2 = 0; s2 < Ns; ++s2)
+      spinLoop()(s1,s2)() = loopm()(s1,s2)(0,0);
+    calcSCVector lR; lR = Zero();
+    for (int mu = 0; mu < ng; ++mu) {
+      auto GLoop   = Gamma(g2[mu]) * spinLoop;
+      auto trGLoop = GLoop()(0,0)() + GLoop()(1,1)() + GLoop()(2,2)() + GLoop()(3,3)();
+      auto Grightv = Gamma(g1[mu]) * rightv;
+      for (int s = 0; s < Ns; ++s)
+      for (int c = 0; c < Nc; ++c)
+        lR()(s)(c) = lR()(s)(c) + Grightv()(s)(c) * trGLoop;
+    }
+    coalescedWrite(lRv[ss], lR);
+  });
+}
+
+// Type 1: loopRight = tloop * right  (SpinColourMatrix * SpinColourVector)
+void A2ALoopRightContractionType1(FermionField &loopRight,
+                                   const PropagatorField &tloop,
+                                   const FermionField &right)
+{
+  autoView(lRv, loopRight, AcceleratorWrite);
+  autoView(tlv, tloop,     AcceleratorRead);
+  autoView(rv,  right,     AcceleratorRead);
+  uint64_t Osites = right.Grid()->oSites();
+  int Nsimd = SpinColourVector_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    coalescedWrite(lRv[ss], tlv(ss) * rv(ss));
+  });
+}
+
+// Type 2: loopRight(s)(c) = sum_{mu,c'} tloop(s1,s2)(c,c') * (G(g1[mu])*right)(s)(c')
+void A2ALoopRightContractionType2(FermionField &loopRight,
+                                   const PropagatorField &tloop,
+                                   const FermionField &right,
+                                   const Vector<Gamma::Algebra> &gamma1)
+{
+  int ng = (int)gamma1.size();
+  const Gamma::Algebra *g1 = gamma1.data();
+  autoView(lRv, loopRight, AcceleratorWrite);
+  autoView(tlv, tloop,     AcceleratorRead);
+  autoView(rv,  right,     AcceleratorRead);
+  uint64_t Osites = right.Grid()->oSites();
+  int Nsimd = SpinColourVector_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    typedef decltype(coalescedRead(rv[0])) calcSCVector;
+    auto loopm  = tlv(ss);
+    auto rightv = rv(ss);
+    calcSCVector lR; lR = Zero();
+    for (int mu = 0; mu < ng; ++mu) {
+      int s1 = mu / Ns;
+      int s2 = mu % Ns;
+      auto Grightv = Gamma(g1[mu]) * rightv;
+      for (int s = 0; s < Ns; ++s)
+      for (int c = 0; c < Nc; ++c)
+        lR()(s)(c) = lR()(s)(c)
+                   + loopm()(s1,s2)(c,0) * Grightv()(s)(0)
+                   + loopm()(s1,s2)(c,1) * Grightv()(s)(1)
+                   + loopm()(s1,s2)(c,2) * Grightv()(s)(2);
+    }
+    coalescedWrite(lRv[ss], lR);
+  });
+}
+
+// Type 3: loopRight(s)(c) = sum_{s'} tloop(s,s')(0,0) * right(s')(c)
+void A2ALoopRightContractionType3(FermionField &loopRight,
+                                   const PropagatorField &tloop,
+                                   const FermionField &right)
+{
+  autoView(lRv, loopRight, AcceleratorWrite);
+  autoView(tlv, tloop,     AcceleratorRead);
+  autoView(rv,  right,     AcceleratorRead);
+  uint64_t Osites = right.Grid()->oSites();
+  int Nsimd = SpinColourVector_v::Nsimd();
+  accelerator_for(ss, Osites, Nsimd, {
+    typedef decltype(coalescedRead(rv[0])) calcSCVector;
+    auto loopm  = tlv(ss);
+    auto rightv = rv(ss);
+    calcSCVector lR; lR = Zero();
+    for (int s = 0; s < Ns; ++s)
+    for (int c = 0; c < Nc; ++c)
+      lR()(s)(c) = loopm()(s,0)(0,0) * rightv()(0)(c)
+                 + loopm()(s,1)(0,0) * rightv()(1)(c)
+                 + loopm()(s,2)(0,0) * rightv()(2)(c)
+                 + loopm()(s,3)(0,0) * rightv()(3)(c);
+    coalescedWrite(lRv[ss], lR);
+  });
+}
+
+// ================================================================
+// GPU-offloaded extended meson field: accelerator_for contractions
+// + A2ASpatialSum GEMM spatial reduction.
+// ================================================================
+class A2AExtendedMesonFieldGPU
+{
+public:
+  static void compute(
+      Eigen::Tensor<ComplexD, 3> &result,
+      const std::vector<FermionField> &left,
+      const std::vector<FermionField> &right,
+      const std::vector<FermionField> &loop1,
+      const std::vector<FermionField> &loop2,
+      const std::vector<Gamma::Algebra> &gamma1_in,
+      const std::vector<Gamma::Algebra> &gamma2_in,
+      int type)
+  {
+    GridBase *grid = left[0].Grid();
+    int N_i = (int)left.size();
+    int N_j = (int)right.size();
+
+    std::string tag = std::string("[gpu  type=") + std::to_string(type) + "]";
+    auto Tms = [](double us) { return us * 1e-3; };
+    double t0;
+
+    Vector<Gamma::Algebra> gamma1(gamma1_in.begin(), gamma1_in.end());
+    Vector<Gamma::Algebra> gamma2(gamma2_in.begin(), gamma2_in.end());
+
+    t0 = usecond();
+    for (auto &f : loop1) { autoView(v, f, AcceleratorRead); }
+    for (auto &f : loop2) { autoView(v, f, AcceleratorRead); }
+    std::cout << GridLogMessage << tag << " view_open_loop:  " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    PropagatorField loop(grid);
+    A2ALoopPropagator(loop, loop1, loop2);
+    std::cout << GridLogMessage << tag << " loop_build:      " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    for (int i = 0; i < N_i; i++) { autoView(v, left[i], AcceleratorRead); }
+    std::cout << GridLogMessage << tag << " view_open_left:  " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    std::vector<FermionField> leftv(N_i, grid);
+    for (int i = 0; i < N_i; i++)
+      A2APackLeftConjugated(leftv[i], left[i]);
+    std::cout << GridLogMessage << tag << " pack_left:       " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    PropagatorField tloop(grid);
+    tloop = Zero();
+    switch (type) {
+    case 0: A2ALoopLeftContractionType0(tloop, loop);                break;
+    case 1: A2ALoopLeftContractionType1(tloop, loop, gamma1, gamma2); break;
+    case 2: A2ALoopLeftContractionType2(tloop, loop, gamma2);         break;
+    case 3: A2ALoopLeftContractionType3(tloop, loop, gamma1, gamma2); break;
+    }
+    std::cout << GridLogMessage << tag << " tloop:           " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    { autoView(tlv, tloop, AcceleratorRead); }
+    for (int j = 0; j < N_j; j++) { autoView(rv, right[j], AcceleratorRead); }
+    std::cout << GridLogMessage << tag << " view_open_right: " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    std::vector<FermionField> loopRight(N_j, grid);
+    for (int j = 0; j < N_j; j++) {
+      switch (type) {
+      case 0: A2ALoopRightContractionType0(loopRight[j], tloop, right[j], gamma1, gamma2); break;
+      case 1: A2ALoopRightContractionType1(loopRight[j], tloop, right[j]);                 break;
+      case 2: A2ALoopRightContractionType2(loopRight[j], tloop, right[j], gamma1);         break;
+      case 3: A2ALoopRightContractionType3(loopRight[j], tloop, right[j]);                 break;
+      }
+    }
+    std::cout << GridLogMessage << tag << " pack_loopright:  " << Tms(usecond()-t0) << " ms\n";
+
+    A2ASpatialSum<SpinColourVector_v> spatial_sum;
+    double t_blas = usecond();
+
+    t0 = usecond();
+    spatial_sum.Allocate(N_i, N_j, grid);
+    std::cout << GridLogMessage << tag << " Allocate:        " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    spatial_sum.PackLeft(leftv);
+    std::cout << GridLogMessage << tag << " PackLeft:        " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    spatial_sum.PackRight(loopRight);
+    std::cout << GridLogMessage << tag << " PackRight:       " << Tms(usecond()-t0) << " ms\n";
+
+    t0 = usecond();
+    spatial_sum.Sum(result);
+    std::cout << GridLogMessage << tag << " Sum (GEMM+MPI):  " << Tms(usecond()-t0) << " ms\n";
+
+    std::cout << GridLogMessage << tag << " A2ASpatialSum:   " << Tms(usecond()-t_blas) << " ms  [TOTAL]\n";
+  }
+};
+
+int main(int argc, char *argv[])
+{
+  Grid_init(&argc, &argv);
+
+  Coordinate latt_size   = GridDefaultLatt();
+  Coordinate simd_layout = GridDefaultSimd(4, vComplexD::Nsimd());
+  Coordinate mpi_layout  = GridDefaultMpi();
+  GridCartesian grid(latt_size, simd_layout, mpi_layout);
+
+  int Nt    = latt_size[Tp];
+  int N_i   = 8;
+  int N_j   = 8;
+  int Nloop = 4;
+
+  if (GridCmdOptionExists(argv, argv+argc, "--Ni"))
+    N_i = std::stoi(GridCmdOptionPayload(argv, argv+argc, "--Ni"));
+  if (GridCmdOptionExists(argv, argv+argc, "--Nj"))
+    N_j = std::stoi(GridCmdOptionPayload(argv, argv+argc, "--Nj"));
+
+  GridParallelRNG pRNG(&grid);
+  pRNG.SeedFixedIntegers({1, 2, 3, 4});
+
+  std::vector<FermionField> left(N_i,   &grid);
+  std::vector<FermionField> right(N_j,  &grid);
+  std::vector<FermionField> loop1(Nloop, &grid);
+  std::vector<FermionField> loop2(Nloop, &grid);
+
+  for (auto &f : left)  random(pRNG, f);
+  for (auto &f : right) random(pRNG, f);
+  for (auto &f : loop1) random(pRNG, f);
+  for (auto &f : loop2) random(pRNG, f);
+
+  std::vector<Gamma::Algebra> GammaMU = {
+    Gamma::Algebra::GammaX,
+    Gamma::Algebra::GammaY,
+    Gamma::Algebra::GammaZ,
+    Gamma::Algebra::GammaT
+  };
+
+  Eigen::Tensor<ComplexD, 3> result_ref(Nt, N_i, N_j);
+  Eigen::Tensor<ComplexD, 3> result_blas(Nt, N_i, N_j);
+  Eigen::Tensor<ComplexD, 3> result_gpu(Nt, N_i, N_j);
+  double t_ref = 0, t_blas = 0, t_gpu = 0, start, stop;
+
+  for (int type = 0; type < 4; type++) {
+
+    result_ref.setZero();
+    start = usecond();
+    A2AExtendedMesonFieldRef::compute(result_ref,  left, right, loop1, loop2,
+                                      GammaMU, GammaMU, type, false);
+    stop = usecond(); t_ref = stop - start;
+
+    result_blas.setZero();
+    start = usecond();
+    A2AExtendedMesonFieldRef::compute(result_blas, left, right, loop1, loop2,
+                                      GammaMU, GammaMU, type, true);
+    stop = usecond(); t_blas = stop - start;
+
+    result_gpu.setZero();
+    start = usecond();
+    A2AExtendedMesonFieldGPU::compute(result_gpu, left, right, loop1, loop2,
+                                      GammaMU, GammaMU, type);
+    stop = usecond(); t_gpu = stop - start;
+
+    double norm2_ref = 0.0, norm2_blas = 0.0, norm2_gpu = 0.0;
+    double norm2_diff_blas = 0.0, norm2_diff_gpu = 0.0;
+    for (int t  = 0; t  < Nt;  t++)
+    for (int ii = 0; ii < N_i; ii++)
+    for (int jj = 0; jj < N_j; jj++) {
+      norm2_ref  += norm2(result_ref(t, ii, jj));
+      norm2_blas += norm2(result_blas(t, ii, jj));
+      norm2_gpu  += norm2(result_gpu(t, ii, jj));
+      ComplexD diff_blas = result_ref(t, ii, jj) - result_blas(t, ii, jj);
+      ComplexD diff_gpu  = result_ref(t, ii, jj) - result_gpu(t, ii, jj);
+      norm2_diff_blas += norm2(diff_blas);
+      norm2_diff_gpu  += norm2(diff_gpu);
+    }
+
+    double rel_blas = (norm2_ref > 0) ? std::sqrt(norm2_diff_blas / norm2_ref) : 0.0;
+    double rel_gpu  = (norm2_ref > 0) ? std::sqrt(norm2_diff_gpu  / norm2_ref) : 0.0;
+
+    std::cout << GridLogMessage
+              << "type=" << type
+              << "  norm2_ref="  << norm2_ref
+              << "  norm2_blas=" << norm2_blas
+              << "  norm2_gpu="  << norm2_gpu
+              << "  rel_blas="   << rel_blas
+              << "  rel_gpu="    << rel_gpu
+              << "  t_ref="  << t_ref  * 1e-6 << "s"
+              << "  t_blas=" << t_blas * 1e-6 << "s"
+              << "  t_gpu="  << t_gpu  * 1e-6 << "s"
+              << std::endl;
+
+    GRID_ASSERT(rel_blas < 1e-10);
+    GRID_ASSERT(rel_gpu  < 1e-10);
+  }
+
+  std::cout << GridLogMessage << "All types passed A2ASpatialSum and GPU regression." << std::endl;
+
+  Grid_finalize();
+  return EXIT_SUCCESS;
+}
+
+#endif

tests/Test_fft_memory.cc

@@ -1,261 +0,0 @@
-/*************************************************************************************
-
- Test_fft_memory.cc
-
- Memory growth test for PlannedFFT on a spin-colour matrix (propagator) field.
-
- The test creates a single PlannedFFT object (which allocates FFTW plans once),
- then repeatedly applies FFT_all_dim to the same propagator 400 times.
-
- If PlannedFFT is working correctly the RSS should remain flat after the first
- iteration — no new plans, no new deviceVector allocations beyond the per-call
- pencil buffer which is freed at the end of each FFT_dim_execute call.
-
- Build exactly like any other Grid test, e.g.:
-   make Test_fft_memory
- or compile manually:
-   $(CXX) $(CXXFLAGS) Test_fft_memory.cc -o Test_fft_memory $(LDFLAGS)
-
-*************************************************************************************/
-
-#include <Grid/Grid.h>
-using namespace Grid;
-
-// --------------------------------------------------------------------------
-// Helper: read RSS (resident set size) in kB from /proc/self/status.
-// Returns 0 on platforms where /proc is unavailable.
-// --------------------------------------------------------------------------
-static long getCPURSSKb()
-{
-  long rss = 0;
-  FILE *fp = fopen("/proc/self/status", "r");
-  if (!fp) return -1;
-  char line[256];
-  while (fgets(line, sizeof(line), fp)) {
-    if (strncmp(line, "VmRSS:", 6) == 0) {
-      sscanf(line + 6, "%ld", &rss);
-      break;
-    }
-  }
-  fclose(fp);
-  return rss;
-}
-
-static long getGPUUsedMb()
-{
-#if defined(GRID_CUDA)
-  size_t free_bytes  = 0;
-  size_t total_bytes = 0;
-  cudaError_t err = cudaMemGetInfo(&free_bytes, &total_bytes);
-  if (err != cudaSuccess) return -1;
-  return (long)((total_bytes - free_bytes) / (1024 * 1024));
- 
-#elif defined(GRID_HIP)
-  size_t free_bytes  = 0;
-  size_t total_bytes = 0;
-  hipError_t err = hipMemGetInfo(&free_bytes, &total_bytes);
-  if (err != hipSuccess) return -1;
-  return (long)((total_bytes - free_bytes) / (1024 * 1024));
- 
-#else
-  return -1;   // CPU-only build: no GPU to query
-#endif
-}
-
-// ============================================================
-//  Convenience struct — one snapshot of both sides
-// ============================================================
-struct MemSnapshot {
-  long cpu_rss_kb;   // host RSS in kB  (-1 if unavailable)
-  long gpu_used_mb;  // device used in MB (-1 if no GPU)
-};
- 
-static MemSnapshot takeSnapshot()
-{
-  MemSnapshot s;
-  s.cpu_rss_kb  = getCPURSSKb();
-  s.gpu_used_mb = getGPUUsedMb();
-  return s;
-}
- 
-// ============================================================
-//  Pretty-print one row of the monitoring table
-// ============================================================
-static void printRow(int iter,
-                     const MemSnapshot &now,
-                     const MemSnapshot &prev)
-{
-  long cpu_delta = (now.cpu_rss_kb  >= 0 && prev.cpu_rss_kb  >= 0)
-                 ? now.cpu_rss_kb  - prev.cpu_rss_kb  : 0;
-  long gpu_delta = (now.gpu_used_mb >= 0 && prev.gpu_used_mb >= 0)
-                 ? now.gpu_used_mb - prev.gpu_used_mb : 0;
- 
-  // Sign prefix so deltas are unambiguous
-  auto sign = [](long v) -> const char* { return v >= 0 ? "+" : ""; };
- 
-  std::cout << GridLogMessage
-            << std::setw(6)  << iter
-            << "  CPU: " << std::setw(10) << now.cpu_rss_kb  << " kB"
-            << "  (" << sign(cpu_delta) << std::setw(7) << cpu_delta << " kB)"
-            << "  GPU: " << std::setw(7)  << now.gpu_used_mb << " MB"
-            << "  (" << sign(gpu_delta) << std::setw(5) << gpu_delta << " MB)"
-            << "\n";
-}
-
-// ============================================================
-
-int main(int argc, char **argv)
-{
-  Grid_init(&argc, &argv);
-
-  int threads = GridThread::GetThreads();
-  std::cout << GridLogMessage
-            << "Grid is setup to use " << threads << " threads" << std::endl;
-
-  // ------------------------------------------------------------------
-  // Grid setup — use whatever lattice/mpi/simd was passed on the CLI,
-  // e.g.  --grid 8.8.8.8 --mpi 1.1.1.1
-  // ------------------------------------------------------------------
-  Coordinate latt_size   = GridDefaultLatt();
-  Coordinate simd_layout = GridDefaultSimd(Nd, vComplexD::Nsimd());
-  Coordinate mpi_layout  = GridDefaultMpi();
-
-  GridCartesian GRID(latt_size, simd_layout, mpi_layout);
-
-  int vol = 1;
-  for (int d = 0; d < (int)latt_size.size(); d++) vol *= latt_size[d];
-
-  std::cout << GridLogMessage << "Lattice : ";
-  for (int d = 0; d < Nd; d++) std::cout << latt_size[d] << " ";
-  std::cout << std::endl;
-
-  // ------------------------------------------------------------------
-  // Propagator field: SpinColourMatrix = 12x12 complex, i.e.
-  // LatticePropagatorD (= Lattice<iSpinColourMatrix<vComplexD>>).
-  // This is the standard QCD quark propagator type.
-  // ------------------------------------------------------------------
-  LatticePropagatorD prop(&GRID);
-  
-  // ------------------------------------------------------------------
-  // Fill the propagator with a momentum-space plane wave,
-  // following the pattern from Test_fft.cc.
-  // We set each spin-colour component (a,b) to exp(i * sum_mu p_mu x_mu)
-  // with a fixed momentum p = (1,2,1,2).
-  // ------------------------------------------------------------------
-  Coordinate pvec({1, 2, 1, 2});
-
-  LatticeComplexD phase(&GRID);
-  LatticeComplexD coor(&GRID);
-  ComplexD ci(0.0, 1.0);
-
-  phase = Zero();
-  for (int mu = 0; mu < Nd; mu++) {
-    RealD TwoPiL = M_PI * 2.0 / latt_size[mu];
-    LatticeCoordinate(coor, mu);
-    phase = phase + (TwoPiL * pvec[mu]) * coor;
-  }
-  phase = exp(phase * ci);   // e^{i p.x}
-
-  // Broadcast the phase into every spin-colour matrix entry
-  prop = Zero();
-  prop = prop + phase;
-
-  std::cout << GridLogMessage
-            << "Propagator norm2 = " << norm2(prop) << std::endl;
-
-  // ------------------------------------------------------------------
-  // Baseline snapshot BEFORE PlannedFFT construction
-  // ------------------------------------------------------------------
-  MemSnapshot snap_before_plan = takeSnapshot();
-  std::cout << GridLogMessage
-            << "[mem] Before PlannedFFT construction"
-            << "  CPU: " << snap_before_plan.cpu_rss_kb  << " kB"
-            << "  GPU: " << snap_before_plan.gpu_used_mb << " MB"
-            << std::endl;
- 
-  // ------------------------------------------------------------------
-  // Create the PlannedFFT — plans are allocated here ONCE for all
-  // dimensions and stored inside the object.
-  // ------------------------------------------------------------------
-  PlannedFFT<iSpinColourMatrix<vComplexD>> plannedFFT(&GRID);
-
-  // ------------------------------------------------------------------
-  // Snapshot AFTER plan construction — this is the true baseline
-  // for the loop, because cufftPlanMany itself grabs device memory.
-  // ------------------------------------------------------------------
-  MemSnapshot snap_after_plan = takeSnapshot();
-  std::cout << GridLogMessage
-            << "[mem] After  PlannedFFT construction"
-            << "  CPU: " << snap_after_plan.cpu_rss_kb  << " kB"
-            << "  GPU: " << snap_after_plan.gpu_used_mb << " MB"
-            << "  (plan overhead:"
-            << "  CPU +" << snap_after_plan.cpu_rss_kb - snap_before_plan.cpu_rss_kb << " kB"
-            << "  GPU +" << snap_after_plan.gpu_used_mb - snap_before_plan.gpu_used_mb << " MB)"
-            << std::endl;
-
-  MemoryManager::Print();
-  // ------------------------------------------------------------------
-  // 400-iteration loop.
-  // Each iteration computes the full 4d forward FFT of `prop`.
-  // We deliberately do NOT cache the result — we always start from
-  // the same `prop` so the FFT is recomputed identically each time.
-  // The point is to watch memory, not correctness.
-  // ------------------------------------------------------------------
-  const int Niter = 40;
-  const int Niter2 = 32;
-
-  // Print header for the memory table
-  std::cout << GridLogMessage
-              << "\n"
-              << std::setw(6)  << "iter"
-              << "  CPU: " << std::setw(10) << "RSS[kB]"
-              << "  (  delta  )"
-              << "  GPU: " << std::setw(7)  << "used[MB]"
-              << "  (delta)"
-              << "\n";
- 
-  MemSnapshot snap_prev = snap_after_plan;
-
-  for (int i = 0; i < Niter; i++) {
-    std::vector<LatticePropagatorD> G;
-
-    for (int j = 0; j < Niter2; j++) {
-      LatticePropagatorD prop_fft(&GRID);
-
-      // Full 4d forward FFT using the pre-built plans
-      plannedFFT.FFT_all_dim(prop_fft, prop, FFT::forward);
-
-      G.push_back(prop_fft);
-    }
- 
-    // cudaMemGetInfo reflects the state *after* any pooled frees have
-    // been committed, so this is accurate without an explicit sync —
-    // FFT_dim_execute already calls accelerator_barrier() internally.
-    MemSnapshot snap_now = takeSnapshot();
-    printRow(i, snap_now, snap_prev);
-    MemoryManager::Print();
-    snap_prev = snap_now;
-  }
-
-  // ------------------------------------------------------------------
-  // Summary
-  // ------------------------------------------------------------------
-  MemSnapshot snap_final = takeSnapshot();
- 
-  long cpu_growth = snap_final.cpu_rss_kb  - snap_after_plan.cpu_rss_kb;
-  long gpu_growth = snap_final.gpu_used_mb - snap_after_plan.gpu_used_mb;
- 
-  std::cout << GridLogMessage
-            << "\n==== Memory summary (baseline = after plan construction) ====\n"
-            << "  CPU RSS growth over " << Niter << " FFTs : "
-            << cpu_growth << " kB"
-            << (cpu_growth == 0 ? "  OK" : "  *** GROWING ***") << "\n"
-            << "  GPU used growth over " << Niter << " FFTs : "
-            << gpu_growth << " MB"
-            << (gpu_growth == 0 ? "  OK" : "  *** GROWING ***") << "\n"
-            << "  Note: first-call watermark from pool fill is expected and benign.\n"
-            << "        A leak shows as continuous growth beyond iter ~2-3.\n";
- 
-  Grid_finalize();
-  return 0;
-}

tests/debug/Test_batched_blas.cc

@@ -1,334 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./tests/debug/Test_batched_blas.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-/*  END LEGAL */
-
-//
-// Unit tests for the blas-layer dense primitives:
-//
-//  T1 : GridBLASInverse::inverseBatched (ComplexF and ComplexD) --
-//       || A A^{-1} - I ||_max over a random well-conditioned batch.
-//       On a CPU build this exercises the Eigen reference (the semantic
-//       oracle); the SAME binary exercises rocSOLVER/cuBLAS/oneMKL on a
-//       device build.
-//  T2 : explicit-leading-dimension gemmBatched == SOFTWARE SPLIT-K in
-//       miniature.  Y = op(A,T).X computed (a) in one compact batch=1 call
-//       and (b) as NK K-chunks by POINTER OFFSET with lda = full K, partials
-//       reduced on the host.  (a)==(b) validates the strided overload AND
-//       the production dense-slab apply pattern (arXiv:2409.03904 fig 11).
-//
-// Hard asserts throughout: regression gate for Grid/algorithms/blas.
-//
-#include <Grid/Grid.h>
-#include <Grid/algorithms/blas/BatchedInverse.h>
-
-#include <random>
-
-using namespace std;
-using namespace Grid;
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  GridBLAS        blas;
-  GridBLASInverse inverse;
-
-  std::mt19937 rng(12345);
-  std::uniform_real_distribution<double> dist(-1.0,1.0);
-
-  ////////////////////////////////////////////////////////////////
-  // T1a : batched inversion, ComplexF
-  ////////////////////////////////////////////////////////////////
-  {
-    const int64_t N = 64;
-    const int  batch = 4;
-    const uint64_t elems = (uint64_t)batch*N*N;
-
-    // Random diagonally-dominant batch: A = N*I + R, |R_ij| <= 1
-    std::vector<ComplexF> Ahost(elems);
-    for(uint64_t i=0;i<elems;i++) Ahost[i] = ComplexF(dist(rng),dist(rng));
-    for(int b=0;b<batch;b++)
-      for(int64_t d=0;d<N;d++)
-	Ahost[(uint64_t)b*N*N + d*N + d] += ComplexF((RealF)N,0.0);
-
-    deviceVector<ComplexF> Adev(elems);   // gets inverted in place
-    deviceVector<ComplexF> Aorig(elems);  // untouched copy for the residual
-    deviceVector<ComplexF> Cdev(elems);
-    acceleratorCopyToDevice(&Ahost[0],&Adev[0], elems*sizeof(ComplexF));
-    acceleratorCopyToDevice(&Ahost[0],&Aorig[0],elems*sizeof(ComplexF));
-
-    deviceVector<ComplexF*> Ap(batch);
-    deviceVector<ComplexF*> Op(batch);
-    deviceVector<ComplexF*> Cp(batch);
-    std::vector<ComplexF*> ptr_h(batch);
-    for(int b=0;b<batch;b++) ptr_h[b] = &Adev[(uint64_t)b*N*N];
-    acceleratorCopyToDevice(&ptr_h[0],&Ap[0],batch*sizeof(ComplexF*));
-    for(int b=0;b<batch;b++) ptr_h[b] = &Aorig[(uint64_t)b*N*N];
-    acceleratorCopyToDevice(&ptr_h[0],&Op[0],batch*sizeof(ComplexF*));
-    for(int b=0;b<batch;b++) ptr_h[b] = &Cdev[(uint64_t)b*N*N];
-    acceleratorCopyToDevice(&ptr_h[0],&Cp[0],batch*sizeof(ComplexF*));
-
-    inverse.inverseBatched(N,Ap);                      // A <- A^{-1}
-
-    ComplexF one (1.0,0.0);
-    ComplexF zero(0.0,0.0);
-    blas.gemmBatched(GridBLAS_OP_N,GridBLAS_OP_N,
-		     (int)N,(int)N,(int)N,
-		     one, Op, Ap, zero, Cp);           // C = A A^{-1}
-    blas.synchronise();
-
-    std::vector<ComplexF> Chost(elems);
-    acceleratorCopyFromDevice(&Cdev[0],&Chost[0],elems*sizeof(ComplexF));
-
-    double maxdev = 0.0;
-    for(int b=0;b<batch;b++)
-      for(int64_t j=0;j<N;j++)
-	for(int64_t i=0;i<N;i++){
-	  ComplexF expect = (i==j) ? ComplexF(1.0,0.0) : ComplexF(0.0,0.0);
-	  ComplexF got = Chost[(uint64_t)b*N*N + j*N + i];
-	  maxdev = std::max(maxdev,(double)abs(got-expect));
-	}
-    std::cout << GridLogMessage << "T1a inverseBatched ComplexF  ||A Ainv - I||_max = "
-	      << maxdev << ( maxdev < 1.0e-4 ? "   PASS" : "   FAIL" ) << std::endl;
-    GRID_ASSERT(maxdev < 1.0e-4);
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T1b : batched inversion, ComplexD
-  ////////////////////////////////////////////////////////////////
-  {
-    const int64_t N = 48;
-    const int  batch = 3;
-    const uint64_t elems = (uint64_t)batch*N*N;
-
-    std::vector<ComplexD> Ahost(elems);
-    for(uint64_t i=0;i<elems;i++) Ahost[i] = ComplexD(dist(rng),dist(rng));
-    for(int b=0;b<batch;b++)
-      for(int64_t d=0;d<N;d++)
-	Ahost[(uint64_t)b*N*N + d*N + d] += ComplexD((RealD)N,0.0);
-
-    deviceVector<ComplexD> Adev(elems);
-    deviceVector<ComplexD> Aorig(elems);
-    deviceVector<ComplexD> Cdev(elems);
-    acceleratorCopyToDevice(&Ahost[0],&Adev[0], elems*sizeof(ComplexD));
-    acceleratorCopyToDevice(&Ahost[0],&Aorig[0],elems*sizeof(ComplexD));
-
-    deviceVector<ComplexD*> Ap(batch);
-    deviceVector<ComplexD*> Op(batch);
-    deviceVector<ComplexD*> Cp(batch);
-    std::vector<ComplexD*> ptr_h(batch);
-    for(int b=0;b<batch;b++) ptr_h[b] = &Adev[(uint64_t)b*N*N];
-    acceleratorCopyToDevice(&ptr_h[0],&Ap[0],batch*sizeof(ComplexD*));
-    for(int b=0;b<batch;b++) ptr_h[b] = &Aorig[(uint64_t)b*N*N];
-    acceleratorCopyToDevice(&ptr_h[0],&Op[0],batch*sizeof(ComplexD*));
-    for(int b=0;b<batch;b++) ptr_h[b] = &Cdev[(uint64_t)b*N*N];
-    acceleratorCopyToDevice(&ptr_h[0],&Cp[0],batch*sizeof(ComplexD*));
-
-    inverse.inverseBatched(N,Ap);
-
-    ComplexD one (1.0,0.0);
-    ComplexD zero(0.0,0.0);
-    blas.gemmBatched(GridBLAS_OP_N,GridBLAS_OP_N,
-		     (int)N,(int)N,(int)N,
-		     one, Op, Ap, zero, Cp);
-    blas.synchronise();
-
-    std::vector<ComplexD> Chost(elems);
-    acceleratorCopyFromDevice(&Cdev[0],&Chost[0],elems*sizeof(ComplexD));
-
-    double maxdev = 0.0;
-    for(int b=0;b<batch;b++)
-      for(int64_t j=0;j<N;j++)
-	for(int64_t i=0;i<N;i++){
-	  ComplexD expect = (i==j) ? ComplexD(1.0,0.0) : ComplexD(0.0,0.0);
-	  ComplexD got = Chost[(uint64_t)b*N*N + j*N + i];
-	  maxdev = std::max(maxdev,(double)abs(got-expect));
-	}
-    std::cout << GridLogMessage << "T1b inverseBatched ComplexD  ||A Ainv - I||_max = "
-	      << maxdev << ( maxdev < 1.0e-10 ? "   PASS" : "   FAIL" ) << std::endl;
-    GRID_ASSERT(maxdev < 1.0e-10);
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T2 : explicit-ld gemmBatched == software split-K miniature.
-  // Slab layout as in the dense coarse-coarse apply: A is K x nrows
-  // column major (lda=K); Y = op(A,T).X with X K x nrhs (ldb=K).
-  ////////////////////////////////////////////////////////////////
-  {
-    const int nrows = 8;
-    const int nrhs  = 4;
-    const int K     = 256;
-    const int NK    = 8;        // split-K chunks
-    const int Kc    = K/NK;
-
-    std::vector<ComplexF> Ahost((uint64_t)K*nrows);
-    std::vector<ComplexF> Xhost((uint64_t)K*nrhs);
-    for(auto &z : Ahost) z = ComplexF(dist(rng),dist(rng));
-    for(auto &z : Xhost) z = ComplexF(dist(rng),dist(rng));
-
-    deviceVector<ComplexF> Adev(Ahost.size());
-    deviceVector<ComplexF> Xdev(Xhost.size());
-    deviceVector<ComplexF> Yref((uint64_t)nrows*nrhs);
-    deviceVector<ComplexF> Ypart((uint64_t)NK*nrows*nrhs);
-    acceleratorCopyToDevice(&Ahost[0],&Adev[0],Ahost.size()*sizeof(ComplexF));
-    acceleratorCopyToDevice(&Xhost[0],&Xdev[0],Xhost.size()*sizeof(ComplexF));
-
-    ComplexF one (1.0,0.0);
-    ComplexF zero(0.0,0.0);
-
-    // (a) reference: one compact batch=1 call (compact lda == K for OP_T)
-    {
-      deviceVector<ComplexF*> Ap(1), Xp(1), Yp(1);
-      std::vector<ComplexF*> h(1);
-      h[0]=&Adev[0]; acceleratorCopyToDevice(&h[0],&Ap[0],sizeof(ComplexF*));
-      h[0]=&Xdev[0]; acceleratorCopyToDevice(&h[0],&Xp[0],sizeof(ComplexF*));
-      h[0]=&Yref[0]; acceleratorCopyToDevice(&h[0],&Yp[0],sizeof(ComplexF*));
-      blas.gemmBatched(GridBLAS_OP_T,GridBLAS_OP_N,
-		       nrows,nrhs,K,
-		       one, Ap, Xp, zero, Yp);
-      blas.synchronise();
-    }
-
-    // (b) split-K: NK chunk-pointers into the SAME allocations, lda/ldb = K
-    {
-      deviceVector<ComplexF*> Ap(NK), Xp(NK), Yp(NK);
-      std::vector<ComplexF*> h(NK);
-      for(int j=0;j<NK;j++) h[j] = &Adev[(uint64_t)j*Kc];        // K-offset slice
-      acceleratorCopyToDevice(&h[0],&Ap[0],NK*sizeof(ComplexF*));
-      for(int j=0;j<NK;j++) h[j] = &Xdev[(uint64_t)j*Kc];
-      acceleratorCopyToDevice(&h[0],&Xp[0],NK*sizeof(ComplexF*));
-      for(int j=0;j<NK;j++) h[j] = &Ypart[(uint64_t)j*nrows*nrhs];
-      acceleratorCopyToDevice(&h[0],&Yp[0],NK*sizeof(ComplexF*));
-      blas.gemmBatched(GridBLAS_OP_T,GridBLAS_OP_N,
-		       nrows,nrhs,Kc,
-		       one, Ap, /*lda*/ K,
-		            Xp, /*ldb*/ K,
-		       zero,Yp, /*ldc*/ nrows);
-      blas.synchronise();
-    }
-
-    std::vector<ComplexF> Yref_h((uint64_t)nrows*nrhs);
-    std::vector<ComplexF> Ypart_h((uint64_t)NK*nrows*nrhs);
-    acceleratorCopyFromDevice(&Yref[0], &Yref_h[0], Yref_h.size()*sizeof(ComplexF));
-    acceleratorCopyFromDevice(&Ypart[0],&Ypart_h[0],Ypart_h.size()*sizeof(ComplexF));
-
-    double maxdev = 0.0;
-    double maxval = 0.0;
-    for(int i=0;i<nrows*nrhs;i++){
-      ComplexF sum(0.0,0.0);
-      for(int j=0;j<NK;j++) sum = sum + Ypart_h[(uint64_t)j*nrows*nrhs + i];
-      maxdev = std::max(maxdev,(double)abs(sum-Yref_h[i]));
-      maxval = std::max(maxval,(double)abs(Yref_h[i]));
-    }
-    double rel = maxdev/maxval;
-    std::cout << GridLogMessage << "T2  split-K strided gemmBatched  max rel dev vs compact = "
-	      << rel << ( rel < 1.0e-4 ? "   PASS" : "   FAIL" ) << std::endl;
-    GRID_ASSERT(rel < 1.0e-4);
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T3 : ComplexD explicit-ld gemmBatched (the RecursiveSchurInverse
-  // merge primitive): same split-K miniature as T2, double precision.
-  // On device builds this is the FIRST exercise of hipblasZ/cublasZ
-  // gemmBatched through the strided overload.
-  ////////////////////////////////////////////////////////////////
-  {
-    const int nrows = 8;
-    const int nrhs  = 4;
-    const int K     = 256;
-    const int NK    = 8;        // split-K chunks
-    const int Kc    = K/NK;
-
-    std::vector<ComplexD> Ahost((uint64_t)K*nrows);
-    std::vector<ComplexD> Xhost((uint64_t)K*nrhs);
-    for(auto &z : Ahost) z = ComplexD(dist(rng),dist(rng));
-    for(auto &z : Xhost) z = ComplexD(dist(rng),dist(rng));
-
-    deviceVector<ComplexD> Adev(Ahost.size());
-    deviceVector<ComplexD> Xdev(Xhost.size());
-    deviceVector<ComplexD> Yref((uint64_t)nrows*nrhs);
-    deviceVector<ComplexD> Ypart((uint64_t)NK*nrows*nrhs);
-    acceleratorCopyToDevice(&Ahost[0],&Adev[0],Ahost.size()*sizeof(ComplexD));
-    acceleratorCopyToDevice(&Xhost[0],&Xdev[0],Xhost.size()*sizeof(ComplexD));
-
-    ComplexD one (1.0,0.0);
-    ComplexD zero(0.0,0.0);
-
-    // (a) reference: one compact batch=1 call (compact lda == K for OP_T)
-    {
-      deviceVector<ComplexD*> Ap(1), Xp(1), Yp(1);
-      std::vector<ComplexD*> h(1);
-      h[0]=&Adev[0]; acceleratorCopyToDevice(&h[0],&Ap[0],sizeof(ComplexD*));
-      h[0]=&Xdev[0]; acceleratorCopyToDevice(&h[0],&Xp[0],sizeof(ComplexD*));
-      h[0]=&Yref[0]; acceleratorCopyToDevice(&h[0],&Yp[0],sizeof(ComplexD*));
-      blas.gemmBatched(GridBLAS_OP_T,GridBLAS_OP_N,
-                       nrows,nrhs,K,
-                       one, Ap, Xp, zero, Yp);
-      blas.synchronise();
-    }
-
-    // (b) split-K: NK chunk-pointers into the SAME allocations, lda/ldb = K
-    {
-      deviceVector<ComplexD*> Ap(NK), Xp(NK), Yp(NK);
-      std::vector<ComplexD*> h(NK);
-      for(int j=0;j<NK;j++) h[j] = &Adev[(uint64_t)j*Kc];        // K-offset slice
-      acceleratorCopyToDevice(&h[0],&Ap[0],NK*sizeof(ComplexD*));
-      for(int j=0;j<NK;j++) h[j] = &Xdev[(uint64_t)j*Kc];
-      acceleratorCopyToDevice(&h[0],&Xp[0],NK*sizeof(ComplexD*));
-      for(int j=0;j<NK;j++) h[j] = &Ypart[(uint64_t)j*nrows*nrhs];
-      acceleratorCopyToDevice(&h[0],&Yp[0],NK*sizeof(ComplexD*));
-      blas.gemmBatched(GridBLAS_OP_T,GridBLAS_OP_N,
-                       nrows,nrhs,Kc,
-                       one, Ap, /*lda*/ K,
-                            Xp, /*ldb*/ K,
-                       zero,Yp, /*ldc*/ nrows);
-      blas.synchronise();
-    }
-
-    std::vector<ComplexD> Yref_h((uint64_t)nrows*nrhs);
-    std::vector<ComplexD> Ypart_h((uint64_t)NK*nrows*nrhs);
-    acceleratorCopyFromDevice(&Yref[0], &Yref_h[0], Yref_h.size()*sizeof(ComplexD));
-    acceleratorCopyFromDevice(&Ypart[0],&Ypart_h[0],Ypart_h.size()*sizeof(ComplexD));
-
-    double maxdev = 0.0;
-    double maxval = 0.0;
-    for(int i=0;i<nrows*nrhs;i++){
-      ComplexD sum(0.0,0.0);
-      for(int j=0;j<NK;j++) sum = sum + Ypart_h[(uint64_t)j*nrows*nrhs + i];
-      maxdev = std::max(maxdev,(double)abs(sum-Yref_h[i]));
-      maxval = std::max(maxval,(double)abs(Yref_h[i]));
-    }
-    double rel = maxdev/maxval;
-    std::cout << GridLogMessage << "T3  split-K strided gemmBatched ComplexD  max rel dev vs compact = "
-              << rel << ( rel < 1.0e-13 ? "   PASS" : "   FAIL" ) << std::endl;
-    GRID_ASSERT(rel < 1.0e-13);
-  }
-
-  std::cout << GridLogMessage << "All batched-blas tests PASSED" << std::endl;
-
-  Grid_finalize();
-}

tests/debug/Test_hipfft_minimal.cc

@@ -47,49 +47,68 @@ static void tryPlanAndExec(int G, long howmany) {
   printf("--- G=%-4d  howmany=%-10ld  total_elems=%-12ld ---\n",
          G, howmany, nelems);
 
-  // --- A: create plan first, allocate buffer afterwards ---
+  // Allocate device buffer (hipfftDoubleComplex = 16 bytes each)
+  hipfftDoubleComplex *dbuf = nullptr;
+  hipError_t herr = hipMalloc(&dbuf, nelems * sizeof(hipfftDoubleComplex));
+  if (herr != hipSuccess) {
+    printf("  hipMalloc failed (%d) for %ld elems — skipping\n\n", (int)herr, nelems);
+    return;
+  }
+  hipMemset(dbuf, 0, nelems * sizeof(hipfftDoubleComplex));
+
+  // 1. hipfftPlanMany (one-step, nullptr embed) — current Grid path
+  {
+    hipfftHandle p;
+    hipfftResult rv = hipfftPlanMany(&p, 1, n,
+                                     nullptr, 1, G,
+                                     nullptr, 1, G,
+                                     HIPFFT_Z2Z, (int)howmany);
+    printf("  hipfftPlanMany   create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  hipfftPlanMany   execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipfftDestroy(p);
+    }
+  }
+
+  // 2. hipfftCreate + hipfftMakePlanMany (two-step) — also current Grid path
   {
     hipfftHandle p;
     size_t workSize = 0;
-    hipfftCreate(&p);
-    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
-                                         nullptr, 1, G, nullptr, 1, G,
-                                         HIPFFT_Z2Z, (int)howmany, &workSize);
-    printf("  plan-first  create : %d (%s)\n", (int)rv, hipfftResultString(rv));
-    if (rv == HIPFFT_SUCCESS) {
-      hipfftDoubleComplex *buf = nullptr;
-      hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
-      hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
-      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
-      hipDeviceSynchronize();
-      printf("  plan-first  execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
-      hipFree(buf);
+    hipfftResult rc = hipfftCreate(&p);
+    if (rc == HIPFFT_SUCCESS) {
+      hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                           nullptr, 1, G,
+                                           nullptr, 1, G,
+                                           HIPFFT_Z2Z, (int)howmany, &workSize);
+      printf("  hipfftMakePlanMany      : %d (%s)  workSize=%zu\n",
+             (int)rv, hipfftResultString(rv), workSize);
+      if (rv == HIPFFT_SUCCESS) {
+        rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+        hipDeviceSynchronize();
+        printf("  hipfftMakePlanMany exec : %d (%s)\n", (int)rv, hipfftResultString(rv));
+      }
+      hipfftDestroy(p);
+    } else {
+      printf("  hipfftCreate            : %d (%s)\n", (int)rc, hipfftResultString(rc));
     }
-    hipfftDestroy(p);
   }
 
-  // --- B: hipMalloc first, create plan afterwards ---
+  // 3. hipfftPlan1d (simplest API, batch = howmany)
   {
-    hipfftDoubleComplex *buf = nullptr;
-    hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
-    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
-
     hipfftHandle p;
-    size_t workSize = 0;
-    hipfftCreate(&p);
-    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
-                                         nullptr, 1, G, nullptr, 1, G,
-                                         HIPFFT_Z2Z, (int)howmany, &workSize);
-    printf("  malloc-first create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    hipfftResult rv = hipfftPlan1d(&p, G, HIPFFT_Z2Z, (int)howmany);
+    printf("  hipfftPlan1d     create : %d (%s)\n", (int)rv, hipfftResultString(rv));
     if (rv == HIPFFT_SUCCESS) {
-      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
       hipDeviceSynchronize();
-      printf("  malloc-first execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      printf("  hipfftPlan1d     execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipfftDestroy(p);
     }
-    hipfftDestroy(p);
-    hipFree(buf);
   }
 
+  hipFree(dbuf);
   printf("\n");
 }
 

tests/debug/Test_schur_dense_coarse.cc

@@ -1,291 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: Test_schur_dense_coarse.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-
-//
-// T6 of the RecursiveSchurInverse regression chain
-// (schur_recursive_inverse_plan.txt 4B.5): the DenseCoarseMatrix GLUE,
-// on a real (tiny) lattice coarse operator, CPU laptop build.
-//
-// Builds a genuine GeneralCoarsenedMatrix (DWF MdagM + 0.5 shift for a
-// guaranteed-invertible Galerkin coarse op, random aggregation basis,
-// nbasis=8, 4^4 x Ls/1 blocking) and constructs DenseCoarseMatrix in
-// DENSE_SCHUR=2 AUDIT mode with small DENSE_PANEL_BYTES (multi-panel
-// gathers exercised through the glue).  The constructor then runs, in
-// order, all the certificates this stage exists to check:
-//   - fresh ImportDense (no SLAB_FILE) + IMPORT CERTIFICATE vs Op.M
-//   - InvertDenseSingle (the oracle)
-//   - InvertDenseSchur: self-certifying rank-major map, fp64 diagonal
-//     import certificate vs the fp32 slab, distributed recursion,
-//     growth telemetry
-//   - AUDIT: max|Ainv_schur - Ainv_single| over the full slab
-//   - VERIFY ||A Ainv x - x||/||x|| through the SCHUR result
-// This program adds asserts on the audit number and a random-vector
-// round trip.
-//
-// Uniform local volume 12.12.12.12 (fine), per-dim blocks {4,4,3,3},
-// coarse 3.3.4.4/rank, nbasis 4 (N = 576n):
-//   mpirun -n 1 ./Test_schur_dense_coarse --grid 12.12.12.12 --mpi 1.1.1.1
-//   mpirun -n 2 ./Test_schur_dense_coarse --grid 12.12.12.24 --mpi 1.1.1.2
-//   mpirun -n 3 ./Test_schur_dense_coarse --grid 12.12.12.36 --mpi 1.1.1.3
-//   mpirun -n 4 ./Test_schur_dense_coarse --grid 12.12.12.48 --mpi 1.1.1.4
-//
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-#include <Grid/algorithms/multigrid/DenseCoarseMatrix.h>
-
-using namespace std;
-using namespace Grid;
-
-///////////////////////////////////////////////////////////////////////
-// MdagM + shift: Galerkin projection of a PD operator plus sigma I is
-// safely invertible whatever the (random) subspace quality.
-///////////////////////////////////////////////////////////////////////
-template<class Field>
-class ShiftedHermOpAdaptor : public LinearOperatorBase<Field>
-{
-  LinearOperatorBase<Field> &wrapped;
-  RealD                      shift;
-public:
-  ShiftedHermOpAdaptor(LinearOperatorBase<Field> &wrapme, RealD s)
-    : wrapped(wrapme), shift(s) {};
-
-  void Op(const Field &in, Field &out)
-  {
-    wrapped.HermOp(in, out);
-    out = out + shift*in;
-  }
-  void AdjOp(const Field &in, Field &out)
-  {
-    Op(in, out);
-  }
-  void HermOp(const Field &in, Field &out)
-  {
-    Op(in, out);
-  }
-  void OpDiag(const Field &in, Field &out)                        { GRID_ASSERT(0); }
-  void OpDir (const Field &in, Field &out, int dir, int disp)     { GRID_ASSERT(0); }
-  void OpDirAll(const Field &in, std::vector<Field> &out)         { GRID_ASSERT(0); }
-  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { GRID_ASSERT(0); }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls     = 4;
-  const int nbasis = 4;
-
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(),
-                                                                   GridDefaultSimd(Nd,vComplex::Nsimd()),
-                                                                   GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Per-dimension blocking {4,4,3,3}: fine 12.12.12.12 -> coarse
-  // 3.3.4.4.  Two constraints meet here (both MEASURED today):
-  //  - coarse dims of 2 hit the probing pathology (health probe below)
-  //  - GEN-simd lanes {1,1,2,2} must land on even coarse dims, so the
-  //    odd production-like 3s go on the lane-free x,y axes (exactly the
-  //    production [3,6,8,8] trick).
-  Coordinate blocks({4,4,3,3});
-  Coordinate clatt = GridDefaultLatt();
-  for(int d=0; d<clatt.size(); d++)
-  {
-    GRID_ASSERT( (clatt[d] % blocks[d]) == 0 );
-    clatt[d] = clatt[d]/blocks[d];
-  }
-  GridCartesian *Coarse4d = SpaceTimeGrid::makeFourDimGrid(clatt,
-                                                           GridDefaultSimd(Nd,vComplex::Nsimd()),
-                                                           GridDefaultMpi());
-  GridCartesian *Coarse5d = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  std::vector<int> cseeds({9,10,11,12});
-  GridParallelRNG RNG4(UGrid);    RNG4.SeedFixedIntegers(seeds4);
-  GridParallelRNG RNG5(FGrid);    RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG CRNG(Coarse5d); CRNG.SeedFixedIntegers(cseeds);
-
-  LatticeGaugeField Umu(UGrid);
-  SU<Nc>::HotConfiguration(RNG4,Umu);
-
-  RealD mass = 0.1;
-  RealD M5   = 1.8;
-  DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
-
-  MdagMLinearOperator<DomainWallFermionD,LatticeFermion> HermDefOp(Ddwf);
-  ShiftedHermOpAdaptor<LatticeFermionD> HOA(HermDefOp, 0.5);
-
-  std::cout << GridLogMessage << "Building random aggregation space, nbasis " << nbasis << std::endl;
-  typedef Aggregation<vSpinColourVector,vTComplex,nbasis> Subspace;
-  Subspace Aggregates(Coarse5d,FGrid,0);
-  Aggregates.CreateSubspaceRandom(RNG5);
-
-  std::cout << GridLogMessage << "Coarsening shifted MdagM" << std::endl;
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector CoarseVector;
-  NextToNextToNextToNearestStencilGeometry5D geom(Coarse5d);
-  LittleDiracOperator LittleDiracOp(geom,FGrid,Coarse5d);
-  LittleDiracOp.CoarsenOperator(HOA,Aggregates);
-
-  ///////////////////////////////////////////////////////////////////////
-  // Operator health probes (independent of DenseCoarseMatrix).
-  //
-  // MEASURED PATHOLOGY, banked 2026-08-14: on coarse dims of 2 (fine
-  // 8.8.8.8, block 4 -> coarse 2.2.2.2) the coarsened operator is
-  // rank 16/128 with 112 zero ROWS (output support = 2 of 16 sites)
-  // and Hermiticity violation 0.17 -- the probing construction breaks
-  // on the size-2 torus.  The import certificate cannot see this
-  // (dense and M share _A).  Out of scope here; coarse dims >= 3.
-  //
-  // Cheap any-size probes: output support + Hermiticity via inner
-  // products on random vectors.
-  ///////////////////////////////////////////////////////////////////////
-  {
-    CoarseVector px(Coarse5d);
-    CoarseVector py(Coarse5d);
-    CoarseVector Mx(Coarse5d);
-    CoarseVector My(Coarse5d);
-    random(CRNG, px);
-    random(CRNG, py);
-    LittleDiracOp.M(px, Mx);
-    LittleDiracOp.M(py, My);
-    ComplexD ip1 = innerProduct(py, Mx);        // <y, M x>
-    ComplexD ip3 = innerProduct(px, My);        // <x, M y>
-    RealD hermdev = abs(ip1 - conj(ip3)) / std::sqrt(norm2(Mx)*norm2(py));
-    RealD support = norm2(Mx) / norm2(px);
-    std::cout << GridLogMessage << "Operator health: ||Mx||^2/||x||^2 = " << support
-              << "  herm-dev " << hermdev << std::endl;
-    // Hermitian fine op => exactly Hermitian Galerkin coarse op.
-    // (A measured herm-dev of 1.3e-4 here was the CPU SIMT-lane
-    // CoarsenOperator bug -- fixed 2026-08-14, now 4e-15.  A loud
-    // failure here means _A population is broken again.)
-    GRID_ASSERT( support > 1.0e-3 );
-    GRID_ASSERT( hermdev < 1.0e-10 );
-  }
-
-  ///////////////////////////////////////////////////////////////////////
-  // Full-matrix conditioning probe at small N: dense columns by
-  // applying M to unit vectors, fp64 Eigen SVD.
-  ///////////////////////////////////////////////////////////////////////
-  {
-    int64_t Nprobe = Coarse5d->gSites() * nbasis;
-    if ( Nprobe <= 700 )
-    {
-      Eigen::MatrixXcd eA(Nprobe, Nprobe);
-      CoarseVector e(Coarse5d);
-      CoarseVector Me(Coarse5d);
-      for(int64_t j=0; j<Nprobe; j++)
-      {
-        int64_t gsite = j / nbasis;
-        int     b     = j % nbasis;
-        e = Zero();
-        Coordinate gcoor(Coarse5d->_ndimension);
-        Lexicographic::CoorFromIndex(gcoor, gsite, Coarse5d->GlobalDimensions());
-        typedef typename CoarseVector::vector_object::scalar_object csobj;
-        csobj s;
-        s = Zero();
-        ((ComplexD *)&s)[b] = ComplexD(1.0,0.0);
-        pokeSite(s, e, gcoor);
-        LittleDiracOp.M(e, Me);
-        for(int64_t i=0; i<Nprobe; i++)
-        {
-          int64_t gsi = i / nbasis;
-          int     bi  = i % nbasis;
-          Coordinate gci(Coarse5d->_ndimension);
-          Lexicographic::CoorFromIndex(gci, gsi, Coarse5d->GlobalDimensions());
-          csobj si;
-          peekSite(si, Me, gci);
-          // Explicit re/im at the thrust/std boundary (HIP builds)
-          ComplexD zz = ((ComplexD *)&si)[bi];
-          eA(i,j) = std::complex<double>(zz.real(), zz.imag());
-        }
-      }
-      Eigen::JacobiSVD<Eigen::MatrixXcd> svd(eA);
-      double smax = svd.singularValues()(0);
-      double smin = svd.singularValues()(Nprobe-1);
-      int64_t rank = 0;
-      for(int64_t i=0; i<Nprobe; i++)
-      {
-        if ( svd.singularValues()(i) > 1.0e-10*smax ) rank++;
-      }
-      double herm = (eA - eA.adjoint()).cwiseAbs().maxCoeff();
-      int64_t zrows = 0;
-      int64_t zcols = 0;
-      for(int64_t i=0; i<Nprobe; i++)
-      {
-        if ( eA.row(i).cwiseAbs().maxCoeff() < 1.0e-12 ) zrows++;
-        if ( eA.col(i).cwiseAbs().maxCoeff() < 1.0e-12 ) zcols++;
-      }
-      std::cout << GridLogMessage << "Operator probe: N=" << Nprobe
-                << "  sigma_max " << smax
-                << "  sigma_min " << smin
-                << "  rank " << rank << "/" << Nprobe
-                << "  herm-dev " << herm
-                << "  zero rows/cols " << zrows << "/" << zcols
-                << std::endl;
-    }
-  }
-
-  ///////////////////////////////////////////////////////////////////////
-  // T6: AUDIT mode, fresh import, multi-panel gathers.  The constructor
-  // runs every certificate in the chain (see banner).
-  ///////////////////////////////////////////////////////////////////////
-  setenv("DENSE_SCHUR","2",1);
-  setenv("DENSE_PANEL_BYTES","65536",1);
-  unsetenv("SLAB_FILE");
-
-  typedef DenseCoarseMatrix<vSpinColourVector,vTComplex,nbasis> DenseCC;
-  DenseCC dcm(LittleDiracOp, Coarse5d);
-
-  std::cout << GridLogMessage << "T6  audit relative slab difference (schur vs single) = "
-            << dcm.schurAuditRel << std::endl;
-  GRID_ASSERT( dcm.schurAuditRel >= 0.0 );      // audit actually ran
-  GRID_ASSERT( dcm.schurAuditRel <  1.0e-3 );
-
-  ///////////////////////////////////////////////////////////////////////
-  // Random-vector round trip through the SCHUR inverse
-  ///////////////////////////////////////////////////////////////////////
-  CoarseVector x(Coarse5d);
-  CoarseVector y(Coarse5d);
-  CoarseVector z(Coarse5d);
-  random(CRNG, x);
-  dcm(x, y);
-  LittleDiracOp.M(y, z);
-  z = z - x;
-  RealD rel = std::sqrt(norm2(z)/norm2(x));
-  std::cout << GridLogMessage << "T6  round trip ||A Ainv x - x||/||x|| (random x) = "
-            << rel << std::endl;
-  GRID_ASSERT( rel < 1.0e-2 );
-
-  std::cout << GridLogMessage << "Test_schur_dense_coarse: T6 ALL PASS" << std::endl;
-
-  Grid_finalize();
-}

tests/debug/Test_schur_inverse.cc

@@ -1,769 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: Test_schur_inverse.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-
-//
-// Staged regression gate for RecursiveSchurInverse (distributed dense
-// inversion by recursive Schur complement) -- the laptop-side certificate
-// chain of schur_recursive_inverse_plan.txt section 4B.5.  Runs on a
-// CPU-only build (Eigen BLAS backends) under mpirun:
-//
-//   mpirun -n 1 ./Test_schur_inverse --grid 8.8.8.8  --mpi 1.1.1.1
-//   mpirun -n 2 ./Test_schur_inverse --grid 8.8.8.8  --mpi 1.1.1.2
-//   mpirun -n 3 ./Test_schur_inverse --grid 8.8.8.12 --mpi 1.1.1.3
-//   mpirun -n 4 ./Test_schur_inverse --grid 8.8.8.8  --mpi 1.1.1.4
-//
-// (n=3 exercises uneven row splits throughout.)  The lattice exists only
-// to furnish the communicator; no field is ever constructed.
-//
-// PRECISION: the inversion runs ENTIRELY in fp64 (decision 2026-08-14,
-// superseding the fp32-merge design); certificates are eps64-scaled.
-// The single terminal fp32 rounding belongs to the caller (tested at the
-// glue level, Test_schur_dense_coarse).
-//
-// Stages present (cumulative -- earlier tests are never removed):
-//   T1a : ownership tables -- CheckRowStart on synthetic uneven partitions,
-//         MakeRowStart allgather vs closed form on the live communicator.
-//   T1b : STORAGE-CONVENTION PIN -- column-major + ld + window-offset
-//         semantics fixed once via identity multiplies through the
-//         explicit-ld gemmBatched, on INTEGER-VALUED data so all three
-//         cases below are EXACT (values well within the mantissa):
-//           (1) alpha=1,beta=0 read from an input column window
-//           (2) alpha=-1,beta=1 accumulate (the S-formation case)
-//           (3) write INTO an output column window, neighbours untouched
-//         No later failure can be a transposition/convention ambiguity.
-//   T2  : GatherGemm vs naive fp64 oracle (owner sub-ranges, alpha-beta
-//         cases, tiny+huge panels, half-participation call shape).
-//   T3  : LeafInvert in-place residual certificate.
-//   T4  : full recursive Invert vs Eigen fp64 oracle, growth-scaled
-//         certification, adversarial near-singular-A11 family with
-//         telemetry-spike assertion.
-//
-// Hard asserts throughout; thresholds pre-registered in the plan.
-//
-#include <Grid/Grid.h>
-#include <Grid/Grid_Eigen_Dense.h>
-#include <Grid/algorithms/multigrid/RecursiveSchurInverse.h>
-
-using namespace std;
-using namespace Grid;
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  GridCartesian Comm(GridDefaultLatt(),
-                     GridDefaultSimd(Nd,vComplex::Nsimd()),
-                     GridDefaultMpi());
-  GridBase *grid = &Comm;
-
-  ////////////////////////////////////////////////////////////////
-  // T1a : ownership tables
-  ////////////////////////////////////////////////////////////////
-  {
-    // Synthetic partitions of N=97 (prime: every P>1 is uneven)
-    const int64_t N = 97;
-    for(int P=1; P<=4; P++)
-    {
-      std::vector<int64_t> table(P+1);
-      table[0] = 0;
-      for(int r=0; r<P; r++)
-      {
-        int64_t nr = N/P + ( (r < (int)(N%P)) ? 1 : 0 );
-        table[r+1] = table[r] + nr;
-      }
-      RecursiveSchurInverse::CheckRowStart(table, N);
-    }
-
-    // Live allgather: deliberately uneven local counts, closed-form oracle
-    int P  = grid->ProcessorCount();
-    int me = grid->ThisRank();
-
-    int64_t myNrows = 3 + me;
-    std::vector<int64_t> table = RecursiveSchurInverse::MakeRowStart(grid, myNrows);
-
-    std::vector<int64_t> expect(P+1);
-    expect[0] = 0;
-    for(int r=0; r<P; r++)
-    {
-      expect[r+1] = expect[r] + (3 + r);
-    }
-    GRID_ASSERT( (int)table.size() == P+1 );
-    for(int r=0; r<=P; r++)
-    {
-      GRID_ASSERT( table[r] == expect[r] );
-    }
-
-    // Constructor smoke: derived ownership matches
-    RecursiveSchurInverse RSI(grid, table[P], table, 1024*1024);
-    GRID_ASSERT( RSI.P       == P );
-    GRID_ASSERT( RSI.me      == me );
-    GRID_ASSERT( RSI.myRow0  == expect[me] );
-    GRID_ASSERT( RSI.myNrows == myNrows );
-
-    std::cout << GridLogMessage
-      << "T1a ownership tables (synthetic P=1..4, live allgather, ctor)   PASS"
-      << std::endl;
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T1b : storage-convention pin (every rank, local, exact)
-  ////////////////////////////////////////////////////////////////
-  {
-    const int64_t rows = 5;
-    const int64_t cols = 13;
-    const int64_t col0 = 6;   // input window start
-    const int64_t w    = 4;   // window width
-
-    // f(i,j): integer-valued, unique per element
-    auto f = [](int64_t i, int64_t j) -> ComplexD
-    {
-      return ComplexD( (RealD)(1 + i + 10*j), (RealD)(i - j) );
-    };
-
-    BlockRows A;
-    A.Resize(rows, cols);
-    {
-      std::vector<ComplexD> Ahost((uint64_t)rows*cols);
-      for(int64_t j=0; j<cols; j++)
-      {
-        for(int64_t i=0; i<rows; i++)
-        {
-          Ahost[(uint64_t)(i + j*rows)] = f(i,j);
-        }
-      }
-      acceleratorCopyToDevice(&Ahost[0], &A.data[0], (uint64_t)rows*cols*sizeof(ComplexD));
-    }
-
-    // Identity I_w, column major
-    deviceVector<ComplexD> Idev((uint64_t)w*w);
-    {
-      std::vector<ComplexD> Ihost((uint64_t)w*w, ComplexD(0.0,0.0));
-      for(int64_t d=0; d<w; d++)
-      {
-        Ihost[(uint64_t)(d + d*w)] = ComplexD(1.0,0.0);
-      }
-      acceleratorCopyToDevice(&Ihost[0], &Idev[0], (uint64_t)w*w*sizeof(ComplexD));
-    }
-
-    GridBLAS BLAS;
-    ComplexD one    ( 1.0,0.0);
-    ComplexD minus  (-1.0,0.0);
-    ComplexD zero   ( 0.0,0.0);
-
-    deviceVector<ComplexD*> Ap(1);
-    deviceVector<ComplexD*> Bp(1);
-    deviceVector<ComplexD*> Cp(1);
-    std::vector<ComplexD*>  ptr_h(1);
-
-    auto setptr = [&](deviceVector<ComplexD*> &d, ComplexD *p)
-    {
-      ptr_h[0] = p;
-      acceleratorCopyToDevice(&ptr_h[0], &d[0], sizeof(ComplexD*));
-    };
-
-    ////////////////////////////////////////////////////////////
-    // Case 1: C = A(:, col0:col0+w) . I_w      (alpha=1, beta=0)
-    ////////////////////////////////////////////////////////////
-    {
-      deviceVector<ComplexD> Cdev((uint64_t)rows*w);
-      setptr(Ap, A.ColumnWindow(col0));
-      setptr(Bp, &Idev[0]);
-      setptr(Cp, &Cdev[0]);
-
-      BLAS.gemmBatched(GridBLAS_OP_N, GridBLAS_OP_N,
-                       (int)rows, (int)w, (int)w,
-                       one,  Ap, (int)A.ld,
-                             Bp, (int)w,
-                       zero, Cp, (int)rows);
-      BLAS.synchronise();
-
-      std::vector<ComplexD> Chost((uint64_t)rows*w);
-      acceleratorCopyFromDevice(&Cdev[0], &Chost[0], (uint64_t)rows*w*sizeof(ComplexD));
-      for(int64_t j=0; j<w; j++)
-      {
-        for(int64_t i=0; i<rows; i++)
-        {
-          GRID_ASSERT( Chost[(uint64_t)(i + j*rows)] == f(i, col0+j) );
-        }
-      }
-    }
-
-    ////////////////////////////////////////////////////////////
-    // Case 2: C = C0 - A(:, col0:col0+w) . I_w (alpha=-1, beta=1)
-    // -- the S-formation accumulate; exact on integer data
-    ////////////////////////////////////////////////////////////
-    {
-      auto g = [](int64_t i, int64_t j) -> ComplexD
-      {
-        return ComplexD( (RealD)(100 + i + j), (RealD)7 );
-      };
-      deviceVector<ComplexD> Cdev((uint64_t)rows*w);
-      {
-        std::vector<ComplexD> Chost((uint64_t)rows*w);
-        for(int64_t j=0; j<w; j++)
-        {
-          for(int64_t i=0; i<rows; i++)
-          {
-            Chost[(uint64_t)(i + j*rows)] = g(i,j);
-          }
-        }
-        acceleratorCopyToDevice(&Chost[0], &Cdev[0], (uint64_t)rows*w*sizeof(ComplexD));
-      }
-      setptr(Ap, A.ColumnWindow(col0));
-      setptr(Bp, &Idev[0]);
-      setptr(Cp, &Cdev[0]);
-
-      BLAS.gemmBatched(GridBLAS_OP_N, GridBLAS_OP_N,
-                       (int)rows, (int)w, (int)w,
-                       minus, Ap, (int)A.ld,
-                              Bp, (int)w,
-                       one,   Cp, (int)rows);
-      BLAS.synchronise();
-
-      std::vector<ComplexD> Chost((uint64_t)rows*w);
-      acceleratorCopyFromDevice(&Cdev[0], &Chost[0], (uint64_t)rows*w*sizeof(ComplexD));
-      for(int64_t j=0; j<w; j++)
-      {
-        for(int64_t i=0; i<rows; i++)
-        {
-          ComplexD expect = g(i,j) - f(i, col0+j);
-          GRID_ASSERT( Chost[(uint64_t)(i + j*rows)] == expect );
-        }
-      }
-    }
-
-    ////////////////////////////////////////////////////////////
-    // Case 3: write INTO a column window of a wider C;
-    // columns outside the window must be untouched
-    ////////////////////////////////////////////////////////////
-    {
-      const int64_t ccols = 6;
-      const int64_t cw0   = 2;   // output window start
-      BlockRows C;
-      C.Resize(rows, ccols);
-      {
-        std::vector<ComplexD> Chost((uint64_t)rows*ccols, ComplexD(-999.0, 999.0));
-        acceleratorCopyToDevice(&Chost[0], &C.data[0], (uint64_t)rows*ccols*sizeof(ComplexD));
-      }
-      setptr(Ap, A.ColumnWindow(col0));
-      setptr(Bp, &Idev[0]);
-      setptr(Cp, C.ColumnWindow(cw0));
-
-      BLAS.gemmBatched(GridBLAS_OP_N, GridBLAS_OP_N,
-                       (int)rows, (int)w, (int)w,
-                       one,  Ap, (int)A.ld,
-                             Bp, (int)w,
-                       zero, Cp, (int)C.ld);
-      BLAS.synchronise();
-
-      std::vector<ComplexD> Chost((uint64_t)rows*ccols);
-      acceleratorCopyFromDevice(&C.data[0], &Chost[0], (uint64_t)rows*ccols*sizeof(ComplexD));
-      for(int64_t j=0; j<ccols; j++)
-      {
-        for(int64_t i=0; i<rows; i++)
-        {
-          ComplexD got = Chost[(uint64_t)(i + j*rows)];
-          if ( (j >= cw0) && (j < cw0+w) )
-          {
-            GRID_ASSERT( got == f(i, col0 + (j-cw0)) );
-          }
-          else
-          {
-            GRID_ASSERT( got == ComplexD(-999.0, 999.0) );
-          }
-        }
-      }
-    }
-
-    std::cout << GridLogMessage
-      << "T1b storage-convention pin (window read / S-accumulate / window write, exact)   PASS"
-      << std::endl;
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T2 : GatherGemm vs naive double-precision oracle.
-  //
-  // Every rank generates the SAME full N x N random fp64 operands
-  // from a fixed seed (no comms needed for the oracle), keeps only
-  // its own rows in BlockRows form, and after each GatherGemm call
-  // checks its output window element-by-element against a plain
-  // triple-loop ComplexD accumulation over the same entries.
-  //
-  // Sweep: N in {8, 96, 97}; owner ranges full/upper-half/single;
-  // (alpha,beta) in {(1,0), (-1,1)}; panelBytes tiny (ragged
-  // many-chunk gathers) and huge (single panel).  Sentinel columns
-  // outside the output window must be untouched.  Finally, a
-  // HALF-PARTICIPATION case rehearses the recursion call pattern:
-  // lower ranks own B but pass EMPTY A/C (collectives only).
-  ////////////////////////////////////////////////////////////////
-  {
-    int P  = grid->ProcessorCount();
-    int me = grid->ThisRank();
-
-    std::mt19937 rng(777);
-    std::uniform_real_distribution<double> dist(-1.0,1.0);
-
-    const int64_t nout = 5;   // output width
-    const int64_t colB = 3;   // B window offset
-    const int64_t colC = 2;   // C window offset
-
-    for(int64_t N : {8L, 96L, 97L})
-    {
-      // Ownership: uneven for any P not dividing N
-      std::vector<int64_t> table(P+1);
-      table[0] = 0;
-      for(int r=0; r<P; r++)
-      {
-        int64_t nr = N/P + ( (r < (int)(N%P)) ? 1 : 0 );
-        table[r+1] = table[r] + nr;
-      }
-      int64_t r0   = table[me];
-      int64_t myNr = table[me+1] - table[me];
-
-      // Identical full operands on every rank
-      std::vector<ComplexD> Aglob((uint64_t)N*N);
-      std::vector<ComplexD> Bglob((uint64_t)N*N);
-      for(uint64_t i=0; i<(uint64_t)N*N; i++) Aglob[i] = ComplexD(dist(rng),dist(rng));
-      for(uint64_t i=0; i<(uint64_t)N*N; i++) Bglob[i] = ComplexD(dist(rng),dist(rng));
-
-      // My rows of a full-matrix operand as a BlockRows
-      auto fillRows = [&](BlockRows &X, std::vector<ComplexD> &glob,
-                          int64_t row0, int64_t nr)
-      {
-        X.Resize(nr, N);
-        if ( nr == 0 ) return;
-        std::vector<ComplexD> h((uint64_t)nr*N);
-        for(int64_t j=0; j<N; j++)
-        {
-          for(int64_t i=0; i<nr; i++)
-          {
-            h[(uint64_t)(i + j*nr)] = glob[(uint64_t)((row0+i) + j*N)];
-          }
-        }
-        acceleratorCopyToDevice(&h[0], &X.data[0], (uint64_t)nr*N*sizeof(ComplexD));
-      };
-
-      // Owner-range cases: full span, upper half, single interior rank
-      std::vector<std::pair<int,int> > ranges;
-      ranges.push_back(std::make_pair(0, P));
-      if ( P > 1 ) ranges.push_back(std::make_pair(P/2, P));
-      if ( P > 1 ) ranges.push_back(std::make_pair(1, 2));
-
-      for(auto range : ranges)
-      {
-        int rB0 = range.first;
-        int rB1 = range.second;
-        int64_t ka0 = table[rB0];       // A-column window start = B row span
-        int64_t k   = table[rB1] - table[rB0];
-
-        for(int acase=0; acase<2; acase++)
-        {
-          ComplexD alpha = ( acase==0 ) ? ComplexD( 1.0,0.0) : ComplexD(-1.0,0.0);
-          ComplexD beta  = ( acase==0 ) ? ComplexD( 0.0,0.0) : ComplexD( 1.0,0.0);
-
-          for(int64_t panelBytes : {64L, 1L<<30})
-          {
-            RecursiveSchurInverse RSI(grid, N, table, panelBytes);
-
-            BlockRows A;
-            BlockRows B;
-            BlockRows C;
-            fillRows(A, Aglob, r0, myNr);
-            fillRows(B, Bglob, r0, myNr);
-
-            // Output: sentinel-filled, window at colC
-            const ComplexD sentinel(-999.0, 999.0);
-            const int64_t  ccols = colC + nout + 2;
-            C.Resize(myNr, ccols);
-            std::vector<ComplexD> C0((uint64_t)myNr*ccols, sentinel);
-            if ( acase == 1 )
-            {
-              // beta=1 needs defined window content: g(i,j), integer-valued
-              for(int64_t j=0; j<nout; j++)
-              {
-                for(int64_t i=0; i<myNr; i++)
-                {
-                  C0[(uint64_t)(i + (colC+j)*myNr)] = ComplexD((RealD)(50+i+j), (RealD)-3);
-                }
-              }
-            }
-            if ( myNr > 0 )
-            {
-              acceleratorCopyToDevice(&C0[0], &C.data[0], (uint64_t)myNr*ccols*sizeof(ComplexD));
-            }
-
-            RSI.GatherGemm(alpha, A, ka0, k,
-                           rB0, rB1,
-                           B, colB, nout,
-                           beta, C, colC);
-
-            std::vector<ComplexD> Chost((uint64_t)myNr*ccols);
-            if ( myNr > 0 )
-            {
-              acceleratorCopyFromDevice(&C.data[0], &Chost[0], (uint64_t)myNr*ccols*sizeof(ComplexD));
-            }
-
-            double tol = 1.0e-14 * (double)k;
-            for(int64_t j=0; j<ccols; j++)
-            {
-              for(int64_t i=0; i<myNr; i++)
-              {
-                ComplexD got = Chost[(uint64_t)(i + j*myNr)];
-                if ( (j >= colC) && (j < colC+nout) )
-                {
-                  int64_t jj = j - colC;
-                  ComplexD acc(0.0,0.0);
-                  if ( acase == 1 )
-                  {
-                    acc = C0[(uint64_t)(i + j*myNr)];
-                  }
-                  for(int64_t t=0; t<k; t++)
-                  {
-                    acc += alpha
-                         * Aglob[(uint64_t)((r0+i)  + (ka0+t)*N)]
-                         * Bglob[(uint64_t)((ka0+t) + (colB+jj)*N)];
-                  }
-                  GRID_ASSERT( abs(got - acc) < tol );
-                }
-                else
-                {
-                  GRID_ASSERT( got == sentinel );
-                }
-              }
-            }
-          }
-        }
-      }
-
-      ////////////////////////////////////////////////////////////
-      // Half-participation: owners = [0,ph) hold B; participants
-      // = [ph,P) hold A/C; owners pass EMPTY A/C and column
-      // offset 0 (collectives only) -- the recursion call shape.
-      ////////////////////////////////////////////////////////////
-      if ( P > 1 )
-      {
-        int ph = ( P+1 ) / 2;
-        int64_t ka0 = table[0];
-        int64_t k   = table[ph] - table[0];
-        int participant = ( me >= ph );
-
-        RecursiveSchurInverse RSI(grid, N, table, 64);
-
-        BlockRows A;
-        BlockRows B;
-        BlockRows C;
-        fillRows(B, Bglob, r0, myNr);
-        if ( participant )
-        {
-          fillRows(A, Aglob, r0, myNr);
-          C.Resize(myNr, nout);
-        }
-
-        ComplexD one (1.0,0.0);
-        ComplexD zero(0.0,0.0);
-        int64_t  cA = participant ? ka0 : 0;
-        RSI.GatherGemm(one, A, cA, k,
-                       0, ph,
-                       B, colB, nout,   // owners deposit from their B window
-                       zero, C, 0);
-
-        if ( participant )
-        {
-          std::vector<ComplexD> Chost((uint64_t)myNr*nout);
-          acceleratorCopyFromDevice(&C.data[0], &Chost[0], (uint64_t)myNr*nout*sizeof(ComplexD));
-          double tol = 1.0e-14 * (double)k;
-          for(int64_t j=0; j<nout; j++)
-          {
-            for(int64_t i=0; i<myNr; i++)
-            {
-              ComplexD acc(0.0,0.0);
-              for(int64_t t=0; t<k; t++)
-              {
-                acc += Aglob[(uint64_t)((r0+i)  + (ka0+t)*N)]
-                     * Bglob[(uint64_t)((ka0+t) + (colB+j)*N)];
-              }
-              GRID_ASSERT( abs(Chost[(uint64_t)(i + j*myNr)] - acc) < tol );
-            }
-          }
-        }
-      }
-    }
-
-    std::cout << GridLogMessage
-      << "T2  GatherGemm vs oracle (N=8/96/97, 3 owner ranges, 2 alpha-beta, tiny+huge panels, half-participation)   PASS"
-      << std::endl;
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T3 : LeafInvert -- in-place fp64 inversion of the contiguous
-  // leaf window.  Purely local, every rank runs its own
-  // uneven-size leaf; residual certificate in ComplexD.
-  ////////////////////////////////////////////////////////////////
-  {
-    int P  = grid->ProcessorCount();
-    int me = grid->ThisRank();
-
-    int64_t  w   = 17 + 3*me;
-    uint64_t len = (uint64_t)w*w;
-
-    std::vector<int64_t> table = RecursiveSchurInverse::MakeRowStart(grid, w);
-    RecursiveSchurInverse RSI(grid, table[P], table, 1<<20);
-
-    // A = w I + R : well conditioned
-    std::mt19937 rng(31 + me);
-    std::uniform_real_distribution<double> dist(-1.0,1.0);
-    std::vector<ComplexD> Ahost(len);
-    for(uint64_t i=0; i<len; i++) Ahost[i] = ComplexD(dist(rng),dist(rng));
-    for(int64_t d=0; d<w; d++)  Ahost[(uint64_t)(d + d*w)] += ComplexD((RealD)w, 0.0);
-
-    BlockRows Ar;
-    Ar.Resize(w, w);
-    acceleratorCopyToDevice(&Ahost[0], &Ar.data[0], len*sizeof(ComplexD));
-    RSI.LeafInvert(0, w, Ar);
-    std::vector<ComplexD> X(len);
-    acceleratorCopyFromDevice(&Ar.data[0], &X[0], len*sizeof(ComplexD));
-
-    double maxdev = 0.0;
-    for(int64_t j=0; j<w; j++)
-    {
-      for(int64_t i=0; i<w; i++)
-      {
-        ComplexD acc(0.0,0.0);
-        for(int64_t t=0; t<w; t++)
-        {
-          acc += Ahost[(uint64_t)(i + t*w)] * X[(uint64_t)(t + j*w)];
-        }
-        if ( i==j ) acc -= ComplexD(1.0,0.0);
-        maxdev = std::max(maxdev, abs(acc));
-      }
-    }
-    GRID_ASSERT( maxdev < 1.0e-13 );
-
-    std::cout << GridLogMessage
-      << "T3  LeafInvert in-place fp64 (residual " << maxdev << ")   PASS" << std::endl;
-  }
-
-  ////////////////////////////////////////////////////////////////
-  // T4 : full recursive Invert vs Eigen fp64 oracle.
-  //
-  // Every rank builds the SAME N x N fp64 matrix from a fixed seed,
-  // keeps its rows, inverts through the full SPMD recursion, then
-  // the test gathers the complete inverse (zero-fill GlobalSum) and
-  // checks BOTH certificates:
-  //     cert1 = || A X - I ||_max          (ComplexD accumulation)
-  //     cert2 = max|X - Xref| / max|Xref|  (Xref = Eigen fp64 inverse)
-  //
-  // Families (eps64-scaled tolerances; the fp32-era growth data
-  // rescales by eps64/eps32 ~ 1.9e-9):
-  //   kappa-moderate : A = R + 3 sqrt(N) I
-  //   kappa-large    : A = R + 0.3 sqrt(N) I
-  //   adversarial    : leading block (rank 0's whole leaf) REPLACED by
-  //                    1e-2 * (R' + 3 sqrt(b) I) inside a well-conditioned
-  //                    A -- the growth spike must REGISTER in telemetry
-  //                    (asserted > 10 when P > 1); at fp64 the certificate
-  //                    barely notices it: that insensitivity IS the point
-  //                    of the fp64 conversion.
-  //
-  // N=64 runs with panelBytes=128 (ragged many-chunk gathers inside
-  // the recursion); larger N with 1 MB panels.
-  ////////////////////////////////////////////////////////////////
-  {
-    int P  = grid->ProcessorCount();
-    int me = grid->ThisRank();
-
-    std::mt19937 rng(2026);
-    std::uniform_real_distribution<double> dist(-1.0,1.0);
-
-    for(int64_t N : {64L, 200L, 513L})
-    {
-      std::vector<int64_t> table(P+1);
-      table[0] = 0;
-      for(int r=0; r<P; r++)
-      {
-        int64_t nr = N/P + ( (r < (int)(N%P)) ? 1 : 0 );
-        table[r+1] = table[r] + nr;
-      }
-      int64_t r0   = table[me];
-      int64_t myNr = table[me+1] - table[me];
-
-      for(int fam=0; fam<3; fam++)
-      {
-        const char *famname = (fam==0) ? "kappa-moderate" :
-                              (fam==1) ? "kappa-large"    : "adversarial-A11";
-        double shift = (fam==1) ? 0.3*std::sqrt((double)N) : 3.0*std::sqrt((double)N);
-        double tol   = (fam==0) ? 1.0e-12 :
-                       (fam==1) ? 1.0e-11 : 5.0e-11;
-
-        // Identical operand on every rank (all draws rank-independent)
-        std::vector<ComplexD> Aglob((uint64_t)N*N);
-        for(uint64_t i=0; i<(uint64_t)N*N; i++) Aglob[i] = ComplexD(dist(rng),dist(rng));
-        for(int64_t d=0; d<N; d++) Aglob[(uint64_t)(d + d*N)] += ComplexD(shift, 0.0);
-        if ( fam == 2 )
-        {
-          // Leading block = rank 0's whole leaf, scaled down 100x but
-          // internally well conditioned (shift scales as sqrt(b): a
-          // FIXED shift makes A11 itself near-singular at large b).
-          int64_t b = ( P > 1 ) ? table[1] : N/4;
-          for(int64_t j=0; j<b; j++)
-          {
-            for(int64_t i=0; i<b; i++)
-            {
-              Aglob[(uint64_t)(i + j*N)] = ComplexD(0.01,0.0)*ComplexD(dist(rng),dist(rng));
-            }
-          }
-          RealD bshift = (RealD)(0.03*std::sqrt((double)b));
-          for(int64_t d=0; d<b; d++) Aglob[(uint64_t)(d + d*N)] += ComplexD(bshift,0.0);
-        }
-
-        // Eigen fp64 oracle.  Explicit re/im conversion at the boundary:
-        // on HIP builds ComplexD is thrust::complex, which has no
-        // operators against Eigen's std::complex.
-        auto toStd = [](const ComplexD &z) -> std::complex<double>
-        {
-          return std::complex<double>(z.real(), z.imag());
-        };
-        Eigen::MatrixXcd eA(N,N);
-        for(int64_t j=0; j<N; j++)
-        {
-          for(int64_t i=0; i<N; i++)
-          {
-            eA(i,j) = toStd(Aglob[(uint64_t)(i + j*N)]);
-          }
-        }
-        Eigen::MatrixXcd Xref = eA.inverse();
-
-        // Distribute, invert
-        int64_t panelBytes = ( N == 64 ) ? 128 : (1<<20);
-        RecursiveSchurInverse RSI(grid, N, table, panelBytes);
-
-        BlockRows Arows;
-        Arows.Resize(myNr, N);
-        {
-          std::vector<ComplexD> h((uint64_t)myNr*N);
-          for(int64_t j=0; j<N; j++)
-          {
-            for(int64_t i=0; i<myNr; i++)
-            {
-              h[(uint64_t)(i + j*myNr)] = Aglob[(uint64_t)((r0+i) + j*N)];
-            }
-          }
-          acceleratorCopyToDevice(&h[0], &Arows.data[0], (uint64_t)myNr*N*sizeof(ComplexD));
-        }
-
-        RSI.Invert(Arows);
-
-        // Gather the full inverse: zero-fill + GlobalSum
-        std::vector<ComplexD> Xfull((uint64_t)N*N, ComplexD(0.0,0.0));
-        {
-          std::vector<ComplexD> h((uint64_t)myNr*N);
-          acceleratorCopyFromDevice(&Arows.data[0], &h[0], (uint64_t)myNr*N*sizeof(ComplexD));
-          for(int64_t j=0; j<N; j++)
-          {
-            for(int64_t i=0; i<myNr; i++)
-            {
-              Xfull[(uint64_t)((r0+i) + j*N)] = h[(uint64_t)(i + j*myNr)];
-            }
-          }
-        }
-        grid->GlobalSumVector(&Xfull[0], (int)(N*N));
-
-        // cert1 = ||A X - I||_max
-        double cert1 = 0.0;
-        for(int64_t j=0; j<N; j++)
-        {
-          for(int64_t i=0; i<N; i++)
-          {
-            ComplexD acc(0.0,0.0);
-            for(int64_t t=0; t<N; t++)
-            {
-              acc += Aglob[(uint64_t)(i + t*N)] * Xfull[(uint64_t)(t + j*N)];
-            }
-            if ( i==j ) acc -= ComplexD(1.0,0.0);
-            cert1 = std::max(cert1, abs(acc));
-          }
-        }
-
-        // cert2 = max|X - Xref| / max|Xref|
-        double maxref = 0.0;
-        double maxdif = 0.0;
-        for(int64_t j=0; j<N; j++)
-        {
-          for(int64_t i=0; i<N; i++)
-          {
-            maxref = std::max(maxref, std::abs(Xref(i,j)));
-            maxdif = std::max(maxdif, std::abs(toStd(Xfull[(uint64_t)(i + j*N)]) - Xref(i,j)));
-          }
-        }
-        double cert2 = maxdif / maxref;
-
-        double maxNormB = 0.0;
-        for(uint64_t i=0; i<RSI.telNormB.size(); i++)
-        {
-          maxNormB = std::max(maxNormB, RSI.telNormB[i]);
-        }
-
-        // GROWTH-SCALED certification, eps64 (the fp32-era model with
-        // eps swapped: cert2 ~ (10-12) ||B||_F sqrt(N) eps; threshold =
-        // 3x margin, floored at the family tolerance).  ||B||_F capped
-        // per family so growth cannot silently excuse a logic error.
-        // cert1 remains a loose absolute bound (an O(1) logic error
-        // gives cert1 ~ 1e2-1e3; fp64 rounding gives ~1e-10).
-        double eps64    = 2.3e-16;
-        double tolModel = 30.0 * std::max(1.0, maxNormB) * std::sqrt((double)N) * eps64;
-        double tolEff   = std::max(tol, tolModel);
-        double capB     = (fam==0) ? 100.0 : (fam==1) ? 2000.0 : 10000.0;
-
-        std::cout << GridLogMessage
-          << "T4  N=" << N << " " << famname
-          << "  ||AX-I||_max " << cert1
-          << "  |X-Xref|/|Xref| " << cert2
-          << "  max||B||_F " << maxNormB
-          << "  tolEff " << tolEff
-          << ( (cert2 < tolEff) && (cert1 < 1.0e-6) ? "   PASS" : "   FAIL" )
-          << std::endl;
-
-        GRID_ASSERT( cert2 < tolEff );
-        GRID_ASSERT( cert1 < 1.0e-6 );
-        GRID_ASSERT( maxNormB < capB );
-        if ( (fam == 2) && (P > 1) )
-        {
-          GRID_ASSERT( maxNormB > 10.0 );   // the spike must REGISTER
-        }
-      }
-    }
-
-    std::cout << GridLogMessage
-      << "T4  recursive Invert vs Eigen oracle (N=64/200/513, 3 families)   PASS"
-      << std::endl;
-  }
-
-  std::cout << GridLogMessage
-    << "Test_schur_inverse: ALL STAGES PASS" << std::endl;
-
-  Grid_finalize();
-}

tests/forces/Test_dwf_ratio_4dpf_force.cc

@@ -1,279 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./tests/forces/Test_dwf_ratio_4dpf_force.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-/*  END LEGAL */
-
-//
-// Validation of TwoFlavourRatio4DPseudoFermionAction (non-EO, LinearFunction
-// solver slots).  Three tests, run for BOTH wall conventions:
-//
-//  T1 HeatbathIdentityTest : refresh then S; PASS iff S == 0.5*|eta4|^2 to
-//     solver tolerance.  This adjudicates the 4D effective-operator
-//     composition identity [P M^-1 V Pdag][P V^-1 M Pdag] = 1 for the chosen
-//     (P,Pdag) wall pair.  NO PREDICTION is made about which convention
-//     passes -- that is what the test decides.
-//  T2 ForceTest (idiom from Test_double_ratio.cc) : midpoint-derivative
-//     check of deriv against S.  Should PASS for BOTH conventions (S and
-//     deriv use the same literal-adjoint pair by construction).
-//  T3 Trivial-ratio control (V == M) : T1 with NumOp = DenOp.  The solve
-//     cancels against the multiply, so S = 0.5|eta4|^2 requires only
-//     P Pdag = 1_4d.  Should PASS for BOTH conventions; isolates plumbing
-//     from the composition identity.
-//
-// Solvers here are plain CG on the normal equations (CGNR), tolerance 1e-12,
-// so every defect above ~1e-10 is structural, not solver noise.  Run small,
-// e.g.:  ./Test_dwf_ratio_4dpf_force --grid 8.8.8.8
-//
-#include <Grid/Grid.h>
-#include <Grid/qcd/action/pseudofermion/TwoFlavourRatio4DPseudoFermion.h>
-
-using namespace std;
-using namespace Grid;
-
-////////////////////////////////////////////////////////////////////
-// LinearFunction wrappers: direct M^-1 and M^-dag via CG on the
-// normal equations.  These stand in for the MG-GCR stack in this
-// test; the action class sees only LinearFunction.
-////////////////////////////////////////////////////////////////////
-template<class Matrix,class Field>
-class CGNRLinearFunction : public LinearFunction<Field> {   // out = M^-1 in
-public:
-  using LinearFunction<Field>::operator();
-  Matrix &_Mat; RealD tol; Integer maxit;
-  CGNRLinearFunction(Matrix &Mat,RealD _tol,Integer _maxit) : _Mat(Mat), tol(_tol), maxit(_maxit) {};
-  void operator()(const Field &in, Field &out) {
-    MdagMLinearOperator<Matrix,Field> MdagM(_Mat);
-    Field src(in.Grid());
-    _Mat.Mdag(in,src);                 // src = Mdag in
-    ConjugateGradient<Field> CG(tol,maxit);
-    out = Zero();
-    CG(MdagM,src,out);                 // out = (MdagM)^-1 Mdag in = M^-1 in
-  }
-};
-template<class Matrix,class Field>
-class CGNRDagLinearFunction : public LinearFunction<Field> { // out = M^-dag in
-public:
-  using LinearFunction<Field>::operator();
-  Matrix &_Mat; RealD tol; Integer maxit;
-  CGNRDagLinearFunction(Matrix &Mat,RealD _tol,Integer _maxit) : _Mat(Mat), tol(_tol), maxit(_maxit) {};
-  void operator()(const Field &in, Field &out) {
-    MdagMLinearOperator<Matrix,Field> MdagM(_Mat);
-    Field tmp(in.Grid());
-    tmp = Zero();
-    ConjugateGradient<Field> CG(tol,maxit);
-    CG(MdagM,in,tmp);                  // tmp = (MdagM)^-1 in
-    _Mat.M(tmp,out);                   // out = M (MdagM)^-1 in = M^-dag in
-  }
-};
-
-////////////////////////////////////////////////////////////////////
-// T1 / T3 : heatbath composition-identity test.
-// Twin-seeded RNG reproduces the eta4 drawn inside refresh.
-////////////////////////////////////////////////////////////////////
-template<class Impl>
-RealD HeatbathIdentityTest(TwoFlavourRatio4DPseudoFermionAction<Impl> &action,
-			   LatticeGaugeField &U,
-			   GridCartesian *UGrid,
-			   const std::string &tag)
-{
-  typedef typename Impl::FermionField FermionField;
-
-  std::vector<int> seeds({9,11,13,17});
-  GridSerialRNG   sRNG;                 sRNG.SeedFixedIntegers(seeds);
-  GridParallelRNG RNG4(UGrid);          RNG4.SeedFixedIntegers(seeds);
-  GridParallelRNG RNG4check(UGrid);     RNG4check.SeedFixedIntegers(seeds);
-
-  FermionField eta4check(UGrid);
-  gaussian(RNG4check,eta4check);        // identical to the draw inside refresh
-
-  action.refresh(U,sRNG,RNG4);
-  RealD S       = action.S(U);
-  RealD Sexpect = 0.5*norm2(eta4check);
-  RealD defect  = std::abs(S-Sexpect)/Sexpect;
-
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-  std::cout << GridLogMessage << " HeatbathIdentityTest ["<<tag<<"]" << std::endl;
-  std::cout << GridLogMessage << "   S              = " << S << std::endl;
-  std::cout << GridLogMessage << "   0.5|eta4|^2    = " << Sexpect << std::endl;
-  std::cout << GridLogMessage << "   relative defect = " << defect
-	    << ( defect < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-  return defect;
-}
-
-////////////////////////////////////////////////////////////////////
-// T2 : ForceTest idiom from Test_double_ratio.cc (midpoint derivative)
-////////////////////////////////////////////////////////////////////
-template<class Gimpl>
-void ForceTest(Action<LatticeGaugeField> &action,LatticeGaugeField & U,MomentumFilterBase<LatticeGaugeField> &Filter)
-{
-  GridBase *UGrid = U.Grid();
-
-  std::vector<int> seeds({1,2,3,5});
-  GridSerialRNG            sRNG;         sRNG.SeedFixedIntegers(seeds);
-  GridParallelRNG          RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds);
-
-  LatticeColourMatrix Pmu(UGrid);
-  LatticeGaugeField P(UGrid);
-  LatticeGaugeField UdSdU(UGrid);
-
-  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
-  std::cout << GridLogMessage << " Force test for "<<action.action_name()<<std::endl;
-  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
-
-  RealD eps=0.005;
-
-  Gimpl::generate_momenta(P,sRNG,RNG4);
-  Filter.applyFilter(P);
-
-  action.refresh(U,sRNG,RNG4);
-
-  RealD S1 = action.S(U);
-
-  Gimpl::update_field(P,U,eps);
-
-  action.deriv(U,UdSdU);
-  UdSdU = Ta(UdSdU);
-  Filter.applyFilter(UdSdU);
-
-  DumpSliceNorm("Force",UdSdU,Nd-1);
-
-  Gimpl::update_field(P,U,eps);
-
-  RealD S2 = action.S(U);
-
-  // Use the derivative
-  LatticeComplex dS(UGrid); dS = Zero();
-  for(int mu=0;mu<Nd;mu++){
-    auto UdSdUmu = PeekIndex<LorentzIndex>(UdSdU,mu);
-    Pmu= PeekIndex<LorentzIndex>(P,mu);
-    dS = dS - trace(Pmu*UdSdUmu)*eps*2.0*2.0;
-  }
-  ComplexD dSpred    = sum(dS);
-  RealD diff =  S2-S1-dSpred.real();
-
-  std::cout<< GridLogMessage << "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++"<<std::endl;
-  std::cout<< GridLogMessage << "S1 : "<< S1    <<std::endl;
-  std::cout<< GridLogMessage << "S2 : "<< S2    <<std::endl;
-  std::cout<< GridLogMessage << "dS : "<< S2-S1 <<std::endl;
-  std::cout<< GridLogMessage << "dSpred : "<< dSpred.real() <<std::endl;
-  std::cout<< GridLogMessage << "diff : "<< diff<<std::endl;
-  std::cout<< GridLogMessage << "diff/dS : "<< diff/(S2-S1)<<std::endl;
-  std::cout<< GridLogMessage << "*********************************************************"<<std::endl;
-  //  GRID_ASSERT(diff<1.0);
-  std::cout<< GridLogMessage << "Done" <<std::endl;
-  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  std::cout << std::setprecision(14);
-
-  const int Ls=8;
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  std::vector<int> seeds4({1,2,3,4});
-  GridParallelRNG RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeGaugeField U(UGrid);
-  SU<Nc>::HotConfiguration(RNG4,U);
-
-  ////////////////////////////////////////////////////////////////
-  // Operators: Mobius, campaign-like b,c; heavyish masses so CGNR
-  // is fast and well-conditioned even on a hot configuration.
-  ////////////////////////////////////////////////////////////////
-  RealD mden = 0.2;
-  RealD mnum = 0.5;
-  RealD M5   = 1.8;
-  RealD b    = 1.5;
-  RealD c    = 0.5;
-
-  WilsonImplParams p;
-  p.boundary_phases[0] = 1.0;
-  p.boundary_phases[1] = 1.0;
-  p.boundary_phases[2] = 1.0;
-  p.boundary_phases[3] = -1.0;
-
-  MobiusFermionD DenOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mden,M5,b,c,p);
-  MobiusFermionD NumOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mnum,M5,b,c,p);
-
-  RealD tol   = 1.0e-12;
-  Integer maxit = 20000;
-
-  typedef WilsonImplD::FermionField FermionField;
-  CGNRLinearFunction<MobiusFermionD,FermionField>    MinvSolver   (DenOp,tol,maxit);
-  CGNRDagLinearFunction<MobiusFermionD,FermionField> MdagInvSolver(DenOp,tol,maxit);
-  CGNRLinearFunction<MobiusFermionD,FermionField>    VinvSolver   (NumOp,tol,maxit);
-
-  ////////////////////////////////////////////////////////////////
-  // Actions: both wall conventions, plus the V==M trivial control
-  ////////////////////////////////////////////////////////////////
-  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActSol(NumOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,VinvSolver,1);
-  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActSrc(NumOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,VinvSolver,0);
-  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActTrivSol(DenOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,MinvSolver,1);
-  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActTrivSrc(DenOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,MinvSolver,0);
-
-  ////////////////////////////////////////////////////////////////
-  // T3 controls first (must both pass; isolates plumbing)
-  ////////////////////////////////////////////////////////////////
-  RealD d3s = HeatbathIdentityTest(ActTrivSol,U,UGrid,"T3 trivial V==M, solution walls");
-  RealD d3q = HeatbathIdentityTest(ActTrivSrc,U,UGrid,"T3 trivial V==M, source walls");
-
-  ////////////////////////////////////////////////////////////////
-  // T1 : the composition-identity adjudication
-  ////////////////////////////////////////////////////////////////
-  RealD d1s = HeatbathIdentityTest(ActSol,U,UGrid,"T1 ratio, solution walls");
-  RealD d1q = HeatbathIdentityTest(ActSrc,U,UGrid,"T1 ratio, source walls");
-
-  ////////////////////////////////////////////////////////////////
-  // T2 : force consistency (expected PASS for both conventions)
-  ////////////////////////////////////////////////////////////////
-  MomentumFilterNone<LatticeGaugeField> FilterNone;
-  ForceTest<GimplTypesR>(ActSol,U,FilterNone);
-  ForceTest<GimplTypesR>(ActSrc,U,FilterNone);
-
-  ////////////////////////////////////////////////////////////////
-  // Summary
-  ////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-  std::cout << GridLogMessage << " SUMMARY (relative heatbath defects)" << std::endl;
-  std::cout << GridLogMessage << "   T3 trivial  solution walls : " << d3s << std::endl;
-  std::cout << GridLogMessage << "   T3 trivial  source   walls : " << d3q << std::endl;
-  std::cout << GridLogMessage << "   T1 ratio    solution walls : " << d1s << std::endl;
-  std::cout << GridLogMessage << "   T1 ratio    source   walls : " << d1q << std::endl;
-  std::cout << GridLogMessage << " T3 must pass for both; T1 selects the wall convention." << std::endl;
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-
-  GRID_ASSERT(d3s < 1.0e-8);
-  GRID_ASSERT(d3q < 1.0e-8);
-
-  Grid_finalize();
-}

tests/forces/Test_dwf_ratio_leftprec.cc

@@ -1,341 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./tests/forces/Test_dwf_ratio_leftprec.cc
-
-    Copyright (C) 2026
-
-Author: Peter Boyle <pboyle@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-/*  END LEGAL */
-
-//
-// Correctness of TwoFlavourRatioLeftPrecPseudoFermionAction against the
-// decades-proven TwoFlavourRatioPseudoFermionAction.  Both classes compute
-// the SAME action S = phi^dag V (MdagM)^-1 Vdag phi through different solve
-// chains (normal-equations vs left-preconditioned F = Vdag M), so with
-// twin-seeded refreshes and 1e-12 solvers they must agree to solver
-// tolerance.  Tests:
-//
-//   E0a/E0b : heatbath identity, S == 0.5|eta|^2 after RNG refresh, for
-//             BOTH classes (E0a also validates the twin-eta capture).
-//   E1      : S_classic == S_leftprec       (relative, ~1e-8)
-//   E2      : deriv_classic == deriv_leftprec (pointwise field norm, ~1e-8)
-//   F1      : ForceTest (Test_double_ratio.cc idiom) on the LeftPrec class.
-//
-// All asserts are hard: this is the regression gate for the new class.
-// Run small, e.g.:  ./Test_dwf_ratio_leftprec --grid 8.8.8.8
-//
-#include <Grid/Grid.h>
-#include <Grid/qcd/action/pseudofermion/TwoFlavourRatioLeftPrec.h>
-
-using namespace std;
-using namespace Grid;
-
-////////////////////////////////////////////////////////////////////
-// Minimal LinearOperator for the composite F = Vdag M, exposing the
-// Hermitian normal operator FdagF for CG.  Stencil entries assert.
-////////////////////////////////////////////////////////////////////
-template<class Impl>
-class VdagMNormalOperator : public LinearOperatorBase<typename Impl::FermionField> {
-public:
-  typedef typename Impl::FermionField Field;
-  FermionOperator<Impl> &VOp;
-  FermionOperator<Impl> &MOp;
-  VdagMNormalOperator(FermionOperator<Impl> &V,FermionOperator<Impl> &M) : VOp(V), MOp(M) {};
-
-  void Fapply(const Field &in, Field &out) {          // out = Vdag M in
-    Field tmp(in.Grid());
-    MOp.M(in,tmp);
-    VOp.Mdag(tmp,out);
-  }
-  void FdagApply(const Field &in, Field &out) {       // out = Mdag V in
-    Field tmp(in.Grid());
-    VOp.M(in,tmp);
-    MOp.Mdag(tmp,out);
-  }
-  virtual void Op     (const Field &in, Field &out) { Fapply(in,out); }
-  virtual void AdjOp  (const Field &in, Field &out) { FdagApply(in,out); }
-  virtual void HermOp (const Field &in, Field &out) {
-    Field tmp(in.Grid());
-    Fapply(in,tmp);
-    FdagApply(tmp,out);
-  }
-  virtual void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2) {
-    HermOp(in,out);
-    ComplexD dot = innerProduct(in,out);
-    n1 = real(dot);
-    n2 = norm2(out);
-  }
-  virtual void OpDiag (const Field &in, Field &out)                  { GRID_ASSERT(0); }
-  virtual void OpDir  (const Field &in, Field &out,int dir,int disp) { GRID_ASSERT(0); }
-  virtual void OpDirAll(const Field &in, std::vector<Field> &out)    { GRID_ASSERT(0); }
-};
-
-////////////////////////////////////////////////////////////////////
-// F-contract LinearFunctions for the test, both via CG on FdagF:
-//   forward :  F x = b     ==>  x = (FdagF)^-1 Fdag b
-//   adjoint :  Fdag z = b  ==>  z = F (FdagF)^-1 b
-////////////////////////////////////////////////////////////////////
-template<class Impl>
-class ForwardFSolve : public LinearFunction<typename Impl::FermionField> {
-public:
-  typedef typename Impl::FermionField Field;
-  using LinearFunction<Field>::operator();
-  VdagMNormalOperator<Impl> &FdagF; RealD tol; Integer maxit;
-  ForwardFSolve(VdagMNormalOperator<Impl> &Op,RealD _tol,Integer _maxit) : FdagF(Op), tol(_tol), maxit(_maxit) {};
-  void operator()(const Field &in, Field &out) {
-    Field src(in.Grid());
-    FdagF.FdagApply(in,src);
-    ConjugateGradient<Field> CG(tol,maxit);
-    out = Zero();
-    CG(FdagF,src,out);
-  }
-};
-template<class Impl>
-class AdjointFSolve : public LinearFunction<typename Impl::FermionField> {
-public:
-  typedef typename Impl::FermionField Field;
-  using LinearFunction<Field>::operator();
-  VdagMNormalOperator<Impl> &FdagF; RealD tol; Integer maxit;
-  AdjointFSolve(VdagMNormalOperator<Impl> &Op,RealD _tol,Integer _maxit) : FdagF(Op), tol(_tol), maxit(_maxit) {};
-  void operator()(const Field &in, Field &out) {
-    Field y(in.Grid());
-    y = Zero();
-    ConjugateGradient<Field> CG(tol,maxit);
-    CG(FdagF,in,y);
-    FdagF.Fapply(y,out);
-  }
-};
-template<class Matrix,class Field>
-class NormalEqSolve : public LinearFunction<Field> {   // out = (MdagM)^-1 in
-public:
-  using LinearFunction<Field>::operator();
-  Matrix &_Mat; RealD tol; Integer maxit;
-  NormalEqSolve(Matrix &Mat,RealD _tol,Integer _maxit) : _Mat(Mat), tol(_tol), maxit(_maxit) {};
-  void operator()(const Field &in, Field &out) {
-    MdagMLinearOperator<Matrix,Field> MdagM(_Mat);
-    ConjugateGradient<Field> CG(tol,maxit);
-    out = Zero();
-    CG(MdagM,in,out);
-  }
-};
-
-////////////////////////////////////////////////////////////////////
-// ForceTest idiom from Test_double_ratio.cc (midpoint derivative)
-////////////////////////////////////////////////////////////////////
-template<class Gimpl>
-void ForceTest(Action<LatticeGaugeField> &action,LatticeGaugeField & U,MomentumFilterBase<LatticeGaugeField> &Filter)
-{
-  GridBase *UGrid = U.Grid();
-
-  std::vector<int> seeds({1,2,3,5});
-  GridSerialRNG            sRNG;         sRNG.SeedFixedIntegers(seeds);
-  GridParallelRNG          RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds);
-
-  LatticeColourMatrix Pmu(UGrid);
-  LatticeGaugeField P(UGrid);
-  LatticeGaugeField UdSdU(UGrid);
-
-  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
-  std::cout << GridLogMessage << " Force test for "<<action.action_name()<<std::endl;
-  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
-
-  RealD eps=0.005;
-
-  Gimpl::generate_momenta(P,sRNG,RNG4);
-  Filter.applyFilter(P);
-
-  action.refresh(U,sRNG,RNG4);
-
-  RealD S1 = action.S(U);
-
-  Gimpl::update_field(P,U,eps);
-
-  action.deriv(U,UdSdU);
-  UdSdU = Ta(UdSdU);
-  Filter.applyFilter(UdSdU);
-
-  DumpSliceNorm("Force",UdSdU,Nd-1);
-
-  Gimpl::update_field(P,U,eps);
-
-  RealD S2 = action.S(U);
-
-  LatticeComplex dS(UGrid); dS = Zero();
-  for(int mu=0;mu<Nd;mu++){
-    auto UdSdUmu = PeekIndex<LorentzIndex>(UdSdU,mu);
-    Pmu= PeekIndex<LorentzIndex>(P,mu);
-    dS = dS - trace(Pmu*UdSdUmu)*eps*2.0*2.0;
-  }
-  ComplexD dSpred    = sum(dS);
-  RealD diff =  S2-S1-dSpred.real();
-
-  std::cout<< GridLogMessage << "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++"<<std::endl;
-  std::cout<< GridLogMessage << "S1 : "<< S1    <<std::endl;
-  std::cout<< GridLogMessage << "S2 : "<< S2    <<std::endl;
-  std::cout<< GridLogMessage << "dS : "<< S2-S1 <<std::endl;
-  std::cout<< GridLogMessage << "dSpred : "<< dSpred.real() <<std::endl;
-  std::cout<< GridLogMessage << "diff : "<< diff<<std::endl;
-  std::cout<< GridLogMessage << "diff/dS : "<< diff/(S2-S1)<<std::endl;
-  std::cout<< GridLogMessage << "*********************************************************"<<std::endl;
-  std::cout<< GridLogMessage << "Done" <<std::endl;
-  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  std::cout << std::setprecision(14);
-
-  const int Ls=8;
-  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  std::vector<int> seeds4({1,2,3,4});
-  GridParallelRNG RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeGaugeField U(UGrid);
-  SU<Nc>::HotConfiguration(RNG4,U);
-
-  ////////////////////////////////////////////////////////////////
-  // Operators: quotient pair (V = PV mass 1, M light-ish), Mobius,
-  // campaign b,c.  Heavyish M so CG is quick on a hot configuration.
-  ////////////////////////////////////////////////////////////////
-  RealD mden = 0.2;
-  RealD mnum = 1.0;
-  RealD M5   = 1.8;
-  RealD b    = 1.5;
-  RealD c    = 0.5;
-
-  WilsonImplParams p;
-  p.boundary_phases[0] = 1.0;
-  p.boundary_phases[1] = 1.0;
-  p.boundary_phases[2] = 1.0;
-  p.boundary_phases[3] = -1.0;
-
-  MobiusFermionD DenOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mden,M5,b,c,p);
-  MobiusFermionD NumOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mnum,M5,b,c,p);
-
-  RealD tol     = 1.0e-12;
-  Integer maxit = 30000;
-
-  typedef WilsonImplD::FermionField FermionField;
-
-  ////////////////////////////////////////////////////////////////
-  // Solvers.  Classic: CG as OperatorFunction on the MdagM linop the
-  // action supplies.  LeftPrec: F-contract solves via CG on FdagF.
-  ////////////////////////////////////////////////////////////////
-  ConjugateGradient<FermionField> CG(tol,maxit);
-
-  VdagMNormalOperator<WilsonImplD> FdagF(NumOp,DenOp);
-  ForwardFSolve<WilsonImplD>       Ffwd (FdagF,tol,maxit);
-  AdjointFSolve<WilsonImplD>       Fadj (FdagF,tol,maxit);
-  NormalEqSolve<MobiusFermionD,FermionField> VdagVinv(NumOp,tol,maxit);
-
-  TwoFlavourRatioPseudoFermionAction<WilsonImplD>         Classic (NumOp,DenOp,CG,CG);
-  TwoFlavourRatioLeftPrecPseudoFermionAction<WilsonImplD> LeftPrec(NumOp,DenOp,Ffwd,Fadj,Fadj,VdagVinv);
-
-  ////////////////////////////////////////////////////////////////
-  // Twin-seeded refreshes: identical eta into both classes.
-  ////////////////////////////////////////////////////////////////
-  std::vector<int> seedsR({9,11,13,17});
-  GridSerialRNG   sRNGa;         sRNGa.SeedFixedIntegers(seedsR);
-  GridSerialRNG   sRNGb;         sRNGb.SeedFixedIntegers(seedsR);
-  GridParallelRNG RNG5a(FGrid);  RNG5a.SeedFixedIntegers(seedsR);
-  GridParallelRNG RNG5b(FGrid);  RNG5b.SeedFixedIntegers(seedsR);
-  GridParallelRNG RNG5c(FGrid);  RNG5c.SeedFixedIntegers(seedsR);
-
-  FermionField etaTwin(FGrid);
-  gaussian(RNG5c,etaTwin);                    // identical to both refresh draws
-
-  Classic.refresh (U,sRNGa,RNG5a);
-  LeftPrec.refresh(U,sRNGb,RNG5b);
-
-  ////////////////////////////////////////////////////////////////
-  // E0 : heatbath identity for both classes
-  ////////////////////////////////////////////////////////////////
-  RealD Sexpect = 0.5*norm2(etaTwin);
-  RealD Sc = Classic.S(U);
-  RealD Sl = LeftPrec.S(U);
-
-  RealD e0a = std::abs(Sc-Sexpect)/Sexpect;
-  RealD e0b = std::abs(Sl-Sexpect)/Sexpect;
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-  std::cout << GridLogMessage << " E0 heatbath identity: 0.5|eta|^2 = " << Sexpect << std::endl;
-  std::cout << GridLogMessage << "   classic  S = " << Sc << "  rel defect " << e0a
-	    << ( e0a < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
-  std::cout << GridLogMessage << "   leftprec S = " << Sl << "  rel defect " << e0b
-	    << ( e0b < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
-
-  ////////////////////////////////////////////////////////////////
-  // E1 : action equivalence
-  ////////////////////////////////////////////////////////////////
-  RealD e1 = std::abs(Sc-Sl)/std::abs(Sc);
-  std::cout << GridLogMessage << " E1 action equivalence: rel diff = " << e1
-	    << ( e1 < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
-
-  ////////////////////////////////////////////////////////////////
-  // E2 : derivative equivalence (pointwise field comparison)
-  ////////////////////////////////////////////////////////////////
-  LatticeGaugeField dSdUc(UGrid);
-  LatticeGaugeField dSdUl(UGrid);
-  LatticeGaugeField dDiff(UGrid);
-
-  Classic.deriv (U,dSdUc);
-  LeftPrec.deriv(U,dSdUl);
-  dDiff = dSdUc - dSdUl;
-
-  RealD e2 = std::sqrt( norm2(dDiff) / norm2(dSdUc) );
-  std::cout << GridLogMessage << " E2 deriv equivalence: |diff|/|classic| = " << e2
-	    << ( e2 < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
-  std::cout << GridLogMessage << "   |dSdU classic |^2 = " << norm2(dSdUc) << std::endl;
-  std::cout << GridLogMessage << "   |dSdU leftprec|^2 = " << norm2(dSdUl) << std::endl;
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-
-  ////////////////////////////////////////////////////////////////
-  // F1 : standalone force test on the LeftPrec class
-  ////////////////////////////////////////////////////////////////
-  MomentumFilterNone<LatticeGaugeField> FilterNone;
-  ForceTest<GimplTypesR>(LeftPrec,U,FilterNone);
-
-  ////////////////////////////////////////////////////////////////
-  // Summary + hard asserts (this is the regression gate)
-  ////////////////////////////////////////////////////////////////
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-  std::cout << GridLogMessage << " SUMMARY" << std::endl;
-  std::cout << GridLogMessage << "   E0a classic  heatbath defect : " << e0a << std::endl;
-  std::cout << GridLogMessage << "   E0b leftprec heatbath defect : " << e0b << std::endl;
-  std::cout << GridLogMessage << "   E1  action   equivalence     : " << e1  << std::endl;
-  std::cout << GridLogMessage << "   E2  deriv    equivalence     : " << e2  << std::endl;
-  std::cout << GridLogMessage << "=========================================================" << std::endl;
-
-  GRID_ASSERT(e0a < 1.0e-8);
-  GRID_ASSERT(e0b < 1.0e-8);
-  GRID_ASSERT(e1  < 1.0e-8);
-  GRID_ASSERT(e2  < 1.0e-8);
-
-  std::cout << GridLogMessage << "All equivalence tests PASSED" << std::endl;
-
-  Grid_finalize();
-}