Scaling perf

Worth an effort to understand if the different stochastic estimator is cleaner.

CLAUDE.md

@@ -31,25 +31,53 @@ Key configure options:
 | `--with-gmp=`, `--with-mpfr=`, `--with-fftw=`, `--with-lime=` | paths to libs |
 | `--enable-hdf5`, `--enable-mkl`, `--enable-lapack` | optional features |
 
+GPU builds additionally need `--enable-gen-simd-width=64` (sets 512-bit SIMD width for GPU warp/wavefront sizing) and `--enable-unified=no --enable-shm=nvlink` for multi-GPU runs.
+
+To speed up compilation, `--disable-fermion-reps --disable-gparity` skips instantiating G-parity and higher-representation fermion operators.
+
 Platform recipes from `README.md`:
 - **KNL**: `--enable-simd=KNL --enable-comms=mpi3-auto --enable-mkl`
 - **Skylake/Haswell**: `--enable-simd=AVX512` or `AVX2` + `--enable-comms=mpi3-auto`
 - **AMD EPYC**: `--enable-simd=AVX2 --enable-comms=mpi3`
 - **A64FX (Fugaku)**: `--enable-simd=A64FX --enable-comms=mpi3 --enable-shm=shmget` (see `SVE_README.txt`)
 
+Complete, working `configure` invocations for specific HPC systems (Frontier/ROCm, Perlmutter/CUDA, Summit, SDCC-A100, etc.) live in `systems/<platform>/config-command`. These are the canonical references for production builds.
+
 Required external libs: GMP, MPFR, OpenSSL, zlib.
 
-## Running Tests
+### Use `systems/` for real machines
+
+`systems/<machine>/` holds the known-good build for each production platform (`Frontier`, `Aurora`, `Perlmutter`, `Summit`, `Tursa`, `Lumi`, `Booster`, `Crusher`, `SDCC-*`, `mac-arm`, …). Each contains a `config-command` (the exact `../../configure` invocation) and a `sourceme.sh` (module loads and env). **Prefer copying/adapting these over hand-rolling configure flags** — they encode compiler workarounds, `LDFLAGS`, and shared-memory settings that are easy to get wrong. `systems/WorkArounds.txt` records known vendor bugs.
+
+Note the GPU builds use `--enable-simd=GPU --enable-gen-simd-width=64`, so `Nsimd` is *not* 1 on device (it is `64/sizeof(scalar)`).
+
+### Regenerating `Make.inc` — required after adding or deleting source files
+
+`Make.inc` files are generated, not tracked in git (`.gitignore`d). `scripts/filelist` walks `Grid/`, `tests/*`, `benchmarks/`, `examples/`, and `HMC/` and writes the file lists and per-test `bin_PROGRAMS` rules. Every new `.cc`/`.h` in `Grid/`, and every new `Test_*.cc` / `Benchmark_*.cc` / `Example_*.cc`, is invisible to the build until you run:
+
+```bash
+./scripts/filelist    # from the source root, then re-run configure/make
+```
+
+`bootstrap.sh` runs it for you on the first setup.
+
+## Running Tests and Benchmarks
 
 ```bash
 # From build directory
 make check                          # root-level tests (Test_simd, Test_cshift, etc.)
 make -C tests/<subdir> tests        # build tests in a subdirectory
+make tests                          # build all tests across all subdirectories
 ./tests/core/Test_simd              # run a single test binary directly
+mpirun -n 4 ./tests/core/Test_cshift --grid 16.16.16.16 --mpi 1.1.1.4
 ```
 
+`make check` is a thin smoke test — building a subdirectory with `make -C tests/<subdir> tests` and running the relevant binaries directly is the normal development loop. Test binaries take Grid's standard command-line arguments (`--grid`, `--mpi`, `--accelerator-threads`, `--threads`, `--debug-signals`, `--log`); see `Grid/util/Init.cc`.
+
 Test subdirectories and their focus: `core` (SIMD, stencil, comms), `solver` (CG, GMRES, eigensolvers), `hmc` (MD integrators), `forces` (fermion forces), `lanczos`, `IO`, `smearing`, `sp2n`, `debug`.
 
+Tests and benchmarks that need optional fermion representations are guarded by `disable_tests_without_instantiations.h` / `disable_benchmarks_without_instantiations.h`, so a `--disable-fermion-reps --disable-gparity` build silently compiles them to no-ops.
+
 ## Architecture
 
 ### Layer stack (bottom to top)
@@ -62,7 +90,7 @@ Test subdirectories and their focus: `core` (SIMD, stencil, comms), `solver` (CG
 
 4. **Cartesian/comms layer** (`Grid/cartesian/`, `Grid/communicator/`) — `GridCartesian` holds the MPI topology and local/global geometry. `Grid/cshift/` implements nearest-neighbour halo exchange; `Grid/stencil/` is the optimised multi-hop stencil used by Dirac operators.
 
-5. **Algorithm layer** (`Grid/algorithms/`) — iterative solvers (CG, GMRES, BiCGSTAB, mixed-precision), eigensolvers (Lanczos, LAPACK), FFT, smearing.
+5. **Algorithm layer** (`Grid/algorithms/`) — iterative solvers (CG, GMRES, BiCGSTAB, mixed-precision), eigensolvers (Lanczos, LAPACK), FFT, smearing, and multigrid.
 
 6. **QCD layer** (`Grid/qcd/`) — gauge and fermion actions, HMC integrators, observables.
 
@@ -75,9 +103,54 @@ Test subdirectories and their focus: `core` (SIMD, stencil, comms), `solver` (CG
 - `smearing/` — APE, Stout, HEX, gradient flow
 - `observables/` — Polyakov loop, plaquette, topological charge
 
+### GPU acceleration and the view/memory-manager discipline
+### Multigrid (`Grid/algorithms/multigrid/`)
+
+Aggregation-based algebraic multigrid for Wilson-type fermions. Key files: `CoarsenedMatrix.h` (coarse operator), `GeneralCoarsenedMatrix.h` and `GeneralCoarsenedMatrixMultiRHS.h` (general coarsening supporting multi-RHS solves), `Aggregates.h` (near-null vector construction), `Geometry.h` (coarse-grid geometry). `MultiGrid.h` is the top-level include.
+
 ### GPU acceleration
 
-GPU support is injected via macros (`accelerator_for`, `accelerator_for2dNB`). The `Grid/simd/` SIMD types map to scalar on GPU device code; host code paths remain vectorised. Unified virtual memory is on by default (`--enable-unified=yes`); device-aware MPI (`--enable-accelerator-aware-mpi`) avoids device→host copies on transfers.
+GPU support is injected via macros in `Grid/threads/Accelerator.h` — `accelerator_for(i, n, nsimd, {...})`, `accelerator_forNB` (non-blocking, must be followed by `accelerator_barrier()`), `accelerator_for2dNB`, and `accelerator_inline`. On a CPU build these degrade to `thread_for` (OpenMP). Unified virtual memory is on by default (`--enable-unified=yes`); device-aware MPI (`--enable-accelerator-aware-mpi`) avoids device→host copies on transfers.
+
+Lattice data is **not** directly addressable inside a kernel. You must open a view with the correct access mode so `Grid/allocator/MemoryManager.h` can move/mark the data:
+
+```cpp
+autoView(out_v, out, AcceleratorWriteDiscard);   // RAII; closes at end of scope
+autoView(in_v,  in,  AcceleratorRead);
+accelerator_for(ss, grid->oSites(), Nsimd, {
+    coalescedWrite(out_v[ss], coalescedRead(in_v[ss]));
+});
+```
+
+Modes are `AcceleratorRead/Write/WriteDiscard` and `CpuRead/Write/WriteDiscard`. Getting the mode wrong (e.g. `AcceleratorRead` on a field you write) produces stale-data bugs that only appear on GPU builds. Inside kernels use `coalescedRead`/`coalescedWrite` rather than raw `operator[]` — they map the SIMD lane onto `threadIdx.x` so accesses stay coalesced.
+
+### Repo-local debugging skills (`skills/`)
+
+`skills/` contains hard-won, Grid-specific playbooks written as invocable skill files. Consult them before debugging in these areas rather than reasoning from first principles:
+
+| File | Covers |
+|---|---|
+| `gpu-memory-performance.md` | `acceleratorThreads()`, LambdaApply thread mapping, `coalescedRead` idiom, fused vs staged HBM access |
+| `gpu-runtime-correctness.md` | GPU runtime returning early from sync, silent wrong answers |
+| `communication-overlap.md` | 7-phase halo pipeline, per-packet events, host-staging vs GPU-direct RDMA |
+| `mpi-heterogeneous.md` | `MPI_Sendrecv` device-buffer aliasing, deterministic reductions |
+| `compiler-validation.md` | Isolating GPU compiler codegen bugs, minimal reproducers |
+| `correctness-verification.md` | Double-run fingerprinting, per-packet checksums, flight recorder |
+| `hang-diagnosis.md` | Diagnosing MPI/accelerator hangs |
+
+The key loop macros (defined in `Grid/threads/Accelerator.h`) are:
+- `accelerator_for(iter, num, nsimd, {...})` — maps to CUDA/HIP kernel or OpenMP loop; `nsimd` is the innermost SIMD lane count
+- `accelerator_forNB(...)` — non-blocking variant (no implicit barrier)
+- `accelerator_for2dNB(iter1, num1, iter2, num2, nsimd, {...})` — 2D kernel launch
+- `thread_for(iter, num, {...})` — CPU OpenMP loop (never dispatches to GPU)
+
+On CPU builds, `accelerator_for` aliases to `thread_for`.
+
+### Solver patterns
+
+`SchurRedBlack` (`Grid/algorithms/iterative/SchurRedBlack.h`) implements red-black (even/odd) preconditioning for fermion operators. Most production fermion solves use `SchurRedBlackDiagMooeeSolve` or similar wrappers that internally call a `ConjugateGradient` on the Schur complement.
+
+Mixed-precision solvers (`ConjugateGradientMixedPrec`, `BiCGSTABMixedPrec`) drive a double-precision outer loop with single-precision inner solves.
 
 ### Memory and I/O
 
@@ -85,14 +158,26 @@ GPU support is injected via macros (`accelerator_for`, `accelerator_for2dNB`). T
 - `Grid/parallelIO/` — distributed parallel reader/writer for ILDG (via LIME), SciDAC, and native binary formats
 - `Grid/serialisation/` — text, binary, HDF5, XML/JSON serialisation of arbitrary Grid objects
 
-### HMC applications
+### Executables
 
-`HMC/` contains production-ready HMC driver programmes (e.g. `Mobius2p1f.cc`, `DWF_plus_DSDR_nf2plus1_Shamir_Gparity.cc`). These are built separately from the library tests.
+- `HMC/` — production HMC driver programmes (e.g. `Mobius2p1f.cc`, `DWF_plus_DSDR_nf2plus1_Shamir_Gparity.cc`)
+- `benchmarks/` — `Benchmark_dwf`, `Benchmark_ITT`, `Benchmark_comms`, `Benchmark_memory_bandwidth`, … used to qualify a new machine
+- `examples/` — small, readable programmes (`Example_plaquette.cc`, `Example_Mobius_spectrum.cc`) that are the best starting point for learning the API
+
+Each of these directories auto-builds every top-level `.cc` as its own binary via `scripts/filelist`.
+
+Every programme is wrapped in `Grid_init(&argc, &argv)` / `Grid_finalize()` (`Grid/util/Init.h`).
 
 ## Key Conventions
 
 - **C++17** is required throughout.
 - Template structure: most classes are templated on `<_FImpl>` (fermion impl) or `<Gimpl>` (gauge impl), which encode the representation and precision. Instantiation is controlled by `--enable-fermion-instantiations`.
+- **Tensor indices are positional, not labelled.** The `Grid/tensors/` arithmetic recurses structurally over the `iScalar`/`iVector`/`iMatrix` nest: each level defines only the {scalar,vector,matrix}² products at its own level, with element types resolved by automatic type deduction, so every colour/spin/lorentz combination composes from ~200 lines (versus the pre-C++11 QDP++/PETE approach of machine-generating every case). An index's meaning derives entirely from its nesting depth counted from the outside; `iScalar` is the identity/broadcast case at every level. Never insert or remove a nesting level casually — the multiplication tables contract by position.
+- **Multigrid coarsening deepens the tensor nest by one level.** A coarse site vector is `iVector<CComplex,nbasis>`, and `innerProduct` on it returns `iScalar<CComplex>` — one level deeper than the fine block scalar. So the block-inner-product scalar type gains one `iScalar` wrapper per MG level (fine: `vTComplex`; level 2: `iScalar<vTComplex>`; see `examples/Example_pvdagm_3level.cc`). When calling `blockInnerProduct`/`blockZAXPY`/`blockOrthogonalise` on coarse fields, the coarse scalar type must match `decltype(innerProduct(siteVector(),siteVector()))` exactly; a wrong depth fails to compile (no viable `operator=` deep in the instantiation chain) rather than mis-contracting.
 - The `RealD`/`RealF`/`ComplexD`/`ComplexF` typedefs are used everywhere; avoid raw `double`/`float`.
-- Logging uses `Grid_log`, `Grid_error` macros (from `Grid/log/`); performance-critical paths use the `GRID_TRACE` / timer macros from `Grid/perfmon/`.
+- Use `GRID_ASSERT(cond)` (defined in `Grid/GridStd.h`), not bare `assert` — it prints a Grid-formatted message and aborts cleanly under MPI.
+- Logging is stream-based, not macro-based: `std::cout << GridLogMessage << ... << std::endl;`. Channels declared in `Grid/log/Log.h` include `GridLogError`, `GridLogWarning`, `GridLogDebug`, `GridLogPerformance`, `GridLogIterative`, `GridLogSolver`, `GridLogHMC`, `GridLogComms`, `GridLogMemory`, `GridLogDslash`, `GridLogIRL`, `GridLogMG`. A subset is switched on at runtime with e.g. `--log Error,Warning,Message,Performance,Iterative,Integrator,Debug,Colours` (names given without the `GridLog` prefix).
+- Performance-critical paths use `GRID_TRACE(name)` from `Grid/perfmon/Tracing.h` (compiled out unless `--enable-tracing` selects a backend) and the `GridStopWatch` timers in `Grid/perfmon/Timer.h`.
 - Reductions across MPI ranks go through `GridBase::GlobalSum` / `GlobalMax`; never reduce with bare MPI calls inside library code.
+- Everything lives in `NAMESPACE_BEGIN(Grid)` / `NAMESPACE_END(Grid)` macros; follow the surrounding file rather than writing `namespace Grid { }`.
+- British spelling is used in identifiers and comments (`colour`, `neighbour`, `serialisation`).

Grid/algorithms/FFT.h

@@ -1,6 +1,6 @@
 /*************************************************************************************
 
-    Grid physics library, www.github.com/paboyle/Grid 
+    Grid physics library, www.github.com/paboyle/Grid
 
     Source file: ./lib/Cshift.h
 
@@ -65,17 +65,16 @@ public:
   typedef hipfftDoubleComplex FFTW_scalar;
   typedef hipfftHandle        FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
+				      FFTW_scalar *in, int *inembed,
-				      int istride, int idist,		
+				      int istride, int idist,
-				      FFTW_scalar *out, int *onembed,		
+				      FFTW_scalar *out, int *onembed,
-				      int ostride, int odist,		
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     auto rv = hipfftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,HIPFFT_Z2Z,howmany);
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
     return p;
-  }	  
+  }
-    
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     hipfftResult rv;
     if ( sign == forward ) rv =hipfftExecZ2Z(p,in,out,HIPFFT_FORWARD);
@@ -83,29 +82,25 @@ public:
     accelerator_barrier();
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
+  inline static void fftw_destroy_plan(const FFTW_plan p) { hipfftDestroy(p); }
-    hipfftDestroy(p);
-  }
 };
 template<> struct FFTW<ComplexF> {
 public:
   static const int forward=FFTW_FORWARD;
   static const int backward=FFTW_BACKWARD;
-  typedef hipfftComplex      FFTW_scalar;
+  typedef hipfftComplex FFTW_scalar;
-  typedef hipfftHandle        FFTW_plan;
+  typedef hipfftHandle  FFTW_plan;
-
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
+				      FFTW_scalar *in, int *inembed,
-				      int istride, int idist,		
+				      int istride, int idist,
-				      FFTW_scalar *out, int *onembed,		
+				      FFTW_scalar *out, int *onembed,
-				      int ostride, int odist,		
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     auto rv = hipfftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,HIPFFT_C2C,howmany);
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
     return p;
-  }	  
+  }
-    
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     hipfftResult rv;
     if ( sign == forward ) rv =hipfftExecC2C(p,in,out,HIPFFT_FORWARD);
@@ -113,9 +108,7 @@ public:
     accelerator_barrier();
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
+  inline static void fftw_destroy_plan(const FFTW_plan p) { hipfftDestroy(p); }
-    hipfftDestroy(p);
-  }
 };
 #endif
 
@@ -126,53 +119,45 @@ public:
   static const int backward=FFTW_BACKWARD;
   typedef cufftDoubleComplex FFTW_scalar;
   typedef cufftHandle        FFTW_plan;
-
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
+				      FFTW_scalar *in, int *inembed,
-				      int istride, int idist,		
+				      int istride, int idist,
-				      FFTW_scalar *out, int *onembed,		
+				      FFTW_scalar *out, int *onembed,
-				      int ostride, int odist,		
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     cufftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,CUFFT_Z2Z,howmany);
     return p;
-  }	  
+  }
-    
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     if ( sign == forward ) cufftExecZ2Z(p,in,out,CUFFT_FORWARD);
     else                   cufftExecZ2Z(p,in,out,CUFFT_INVERSE);
     accelerator_barrier();
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
+  inline static void fftw_destroy_plan(const FFTW_plan p) { cufftDestroy(p); }
-    cufftDestroy(p);
-  }
 };
 template<> struct FFTW<ComplexF> {
 public:
   static const int forward=FFTW_FORWARD;
   static const int backward=FFTW_BACKWARD;
   typedef cufftComplex FFTW_scalar;
-  typedef cufftHandle        FFTW_plan;
+  typedef cufftHandle  FFTW_plan;
-
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
+				      FFTW_scalar *in, int *inembed,
-				      int istride, int idist,		
+				      int istride, int idist,
-				      FFTW_scalar *out, int *onembed,		
+				      FFTW_scalar *out, int *onembed,
-				      int ostride, int odist,		
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     cufftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,CUFFT_C2C,howmany);
     return p;
-  }	  
+  }
-    
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     if ( sign == forward ) cufftExecC2C(p,in,out,CUFFT_FORWARD);
     else                   cufftExecC2C(p,in,out,CUFFT_INVERSE);
     accelerator_barrier();
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
+  inline static void fftw_destroy_plan(const FFTW_plan p) { cufftDestroy(p); }
-    cufftDestroy(p);
-  }
 };
 #endif
 
@@ -183,313 +168,325 @@ public:
   typedef fftw_complex FFTW_scalar;
   typedef fftw_plan    FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
+				      FFTW_scalar *in, int *inembed,
-				      int istride, int idist,		
+				      int istride, int idist,
-				      FFTW_scalar *out, int *onembed,		
+				      FFTW_scalar *out, int *onembed,
-				      int ostride, int odist,		
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     return ::fftw_plan_many_dft(rank,n,howmany,in,inembed,istride,idist,out,onembed,ostride,odist,sign,flags);
-  }	  
+  }
-    
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     ::fftw_execute_dft(p,in,out);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
+  inline static void fftw_destroy_plan(const FFTW_plan p) { ::fftw_destroy_plan(p); }
-    ::fftw_destroy_plan(p);
-  }
 };
 template<> struct FFTW<ComplexF> {
 public:
   typedef fftwf_complex FFTW_scalar;
   typedef fftwf_plan    FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
+				      FFTW_scalar *in, int *inembed,
-				      int istride, int idist,		
+				      int istride, int idist,
-				      FFTW_scalar *out, int *onembed,		
+				      FFTW_scalar *out, int *onembed,
-				      int ostride, int odist,		
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     return ::fftwf_plan_many_dft(rank,n,howmany,in,inembed,istride,idist,out,onembed,ostride,odist,sign,flags);
-  }	  
+  }
-    
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     ::fftwf_execute_dft(p,in,out);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
+  inline static void fftw_destroy_plan(const FFTW_plan p) { ::fftwf_destroy_plan(p); }
-    ::fftwf_destroy_plan(p);
-  }
 };
 #endif
 #endif
 
-class FFT {
+struct FFTbase {
-private:
+  double        flops;
-    
+  double        flops_call;
-  double flops;
+  uint64_t      usec;
-  double flops_call;
+  GridCartesian *_grid;
-  uint64_t usec;
-    
-public:
-    
-  static const int forward=FFTW_FORWARD;
-  static const int backward=FFTW_BACKWARD;
-    
-  double Flops(void) {return flops;}
-  double MFlops(void) {return flops/usec;}
-  double USec(void)   {return (double)usec;}    
 
-  FFT ( GridCartesian * grid ) 
+  static const int forward  = FFTW_FORWARD;
-  {
+  static const int backward = FFTW_BACKWARD;
-    flops=0;
+
-    usec =0;
+  double Flops(void)  { return flops; }
-  };
+  double MFlops(void) { return flops / usec; }
-    
+  double USec(void)   { return (double)usec; }
-  ~FFT ( void)  {
+
-    //    delete sgrid;
+  FFTbase(GridCartesian *grid) : _grid(grid), flops(0), flops_call(0), usec(0) {}
+};
+
+// Barrel-shift gather, FFT execute, and insert. Called by both FFT and PlannedFFT.
+// The caller is responsible for plan acquisition and destruction.
+template<class vobj>
+static void FFT_dim_execute(
+    Lattice<vobj>       &result,
+    const Lattice<vobj> &source,
+    int dim, int sign,
+    typename FFTW<typename vobj::scalar_type>::FFTW_plan p,
+    GridCartesian *grid,
+    double &flops, double &flops_call, uint64_t &usec)
+{
+  typedef typename vobj::scalar_type   scalar;
+  typedef typename vobj::scalar_object sobj;
+  typedef typename vobj::scalar_type   scalar_type;
+  typedef typename vobj::vector_type   vector_type;
+  typedef typename FFTW<scalar>::FFTW_scalar FFTW_scalar;
+
+  const int Ndim = grid->Nd();
+  int L = grid->_ldimensions[dim];
+  int G = grid->_fdimensions[dim];
+  int Ncomp = sizeof(sobj) / sizeof(scalar);
+  int64_t Nlow = 1, Nhigh = 1;
+  for (int d = 0; d < dim;      d++) Nlow  *= grid->_ldimensions[d];
+  for (int d = dim+1; d < Ndim; d++) Nhigh *= grid->_ldimensions[d];
+  int64_t Nperp = Nlow * Nhigh;
+
+  deviceVector<scalar> pgbuf(Nperp * Ncomp * G);
+  scalar *pgbuf_v = &pgbuf[0];
+  int howmany = Ncomp * Nperp;
+
+  scalar div;
+  if      (sign == FFTW_BACKWARD) div = 1.0 / G;
+  else if (sign == FFTW_FORWARD)  div = 1.0;
+  else GRID_ASSERT(0);
+
+  double t_pencil = 0, t_fft = 0, t_copy = 0, t_shift = 0;
+  double t_total  = -usecond();
+
+  result = source;
+  int pc = grid->_processor_coor[dim];
+
+  const Coordinate ldims      = grid->_ldimensions;
+  const Coordinate rdims      = grid->_rdimensions;
+  const Coordinate sdims      = grid->_simd_layout;
+  const Coordinate processors = grid->_processors;
+
+  Coordinate pgdims(Ndim);
+  pgdims[0] = G;
+  for (int d = 0, dd = 1; d < Ndim; d++)
+    if (d != dim) pgdims[dd++] = ldims[d];
+  int64_t pgvol = 1;
+  for (int d = 0; d < Ndim; d++) pgvol *= pgdims[d];
+
+  const int Nsimd = vobj::Nsimd();
+  t_pencil = -usecond();
+  for (int p_idx = 0; p_idx < processors[dim]; p_idx++) {
+    t_copy -= usecond();
+    autoView(r_v, result, AcceleratorRead);
+    accelerator_for(idx, grid->oSites(), vobj::Nsimd(), {
+#ifdef GRID_SIMT
+    {
+      int lane = acceleratorSIMTlane(Nsimd);
+#else
+    for (int lane = 0; lane < Nsimd; lane++) {
+#endif
+      Coordinate icoor, ocoor, pgcoor;
+      Lexicographic::CoorFromIndex(icoor, lane, sdims);
+      Lexicographic::CoorFromIndex(ocoor, idx,  rdims);
+      pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + ((pc+p_idx)%processors[dim])*L;
+      for (int d = 0, dd = 1; d < Ndim; d++)
+        if (d != dim) { pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d]; dd++; }
+      int64_t pgidx;
+      Lexicographic::IndexFromCoor(pgcoor, pgidx, pgdims);
+      vector_type *from = (vector_type *)&r_v[idx];
+      scalar_type  stmp;
+      for (int w = 0; w < Ncomp; w++) {
+        stmp = getlane(from[w], lane);
+        pgbuf_v[pgidx + w*pgvol] = stmp;
+      }
+#ifdef GRID_SIMT
+    }
+#else
+    }
+#endif
+    });
+    t_copy += usecond();
+    if (p_idx != processors[dim] - 1) {
+      Lattice<vobj> temp(grid);
+      t_shift -= usecond();
+      temp = Cshift(result, dim, L); result = temp;
+      t_shift += usecond();
+    }
   }
-    
+  t_pencil += usecond();
-  template<class vobj>
-  void FFT_dim_mask(Lattice<vobj> &result,const Lattice<vobj> &source,Coordinate mask,int sign){
 
-    //    vgrid=result.Grid();
+  FFTW_scalar *in  = (FFTW_scalar *)pgbuf_v;
-    //    conformable(result.Grid(),vgrid);
+  FFTW_scalar *out = (FFTW_scalar *)pgbuf_v;
-    //    conformable(source.Grid(),vgrid);
+  t_fft = -usecond();
-    const int Ndim = source.Grid()->Nd();
+  FFTW<scalar>::fftw_execute_dft(p, in, out, sign);
+  t_fft += usecond();
+
+  flops_call = 5.0 * howmany * G * log2(G);
+  usec  = t_fft;
+  flops = flops_call;
+
+  result = Zero();
+  double t_insert = -usecond();
+  {
+    autoView(r_v, result, AcceleratorWrite);
+    accelerator_for(idx, grid->oSites(), Nsimd, {
+#ifdef GRID_SIMT
+    {
+      int lane = acceleratorSIMTlane(Nsimd);
+#else
+    for (int lane = 0; lane < Nsimd; lane++) {
+#endif
+      Coordinate icoor(Ndim), ocoor(Ndim), pgcoor(Ndim);
+      Lexicographic::CoorFromIndex(icoor, lane, sdims);
+      Lexicographic::CoorFromIndex(ocoor, idx,  rdims);
+      pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + pc*L;
+      for (int d = 0, dd = 1; d < Ndim; d++)
+        if (d != dim) { pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d]; dd++; }
+      int64_t pgidx;
+      Lexicographic::IndexFromCoor(pgcoor, pgidx, pgdims);
+      vector_type *to  = (vector_type *)&r_v[idx];
+      scalar_type  stmp;
+      for (int w = 0; w < Ncomp; w++) {
+        stmp = pgbuf_v[pgidx + w*pgvol];
+        putlane(to[w], stmp, lane);
+      }
+#ifdef GRID_SIMT
+    }
+#else
+    }
+#endif
+    });
+  }
+  result = result * div;
+  t_insert += usecond();
+  t_total  += usecond();
+
+  std::cout << GridLogPerformance << " FFT took   " << t_total/1.0e6  << " s" << std::endl;
+  std::cout << GridLogPerformance << " FFT pencil " << t_pencil/1.0e6 << " s" << std::endl;
+  std::cout << GridLogPerformance << "  of which copy " << t_copy/1.0e6  << " s" << std::endl;
+  std::cout << GridLogPerformance << "  of which shift" << t_shift/1.0e6 << " s" << std::endl;
+  std::cout << GridLogPerformance << " FFT kernels " << t_fft/1.0e6    << " s" << std::endl;
+  std::cout << GridLogPerformance << " FFT insert  " << t_insert/1.0e6 << " s" << std::endl;
+}
+
+class FFT : public FFTbase {
+public:
+  FFT(GridCartesian *grid) : FFTbase(grid) {}
+  ~FFT() {}
+
+  template<class vobj>
+  void FFT_dim_mask(Lattice<vobj> &result, const Lattice<vobj> &source, Coordinate mask, int sign) {
+    const int Ndim = _grid->Nd();
     Lattice<vobj> tmp = source;
-    for(int d=0;d<Ndim;d++){
+    for (int d = 0; d < Ndim; d++) {
-      if( mask[d] ) {
+      if (mask[d]) {
-	FFT_dim(result,tmp,d,sign);
+        FFT_dim(result, tmp, d, sign);
-	tmp=result;
+        tmp = result;
       }
     }
   }
 
   template<class vobj>
-  void FFT_all_dim(Lattice<vobj> &result,const Lattice<vobj> &source,int sign){
+  void FFT_all_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int sign) {
-    const int Ndim = source.Grid()->Nd();
+    Coordinate mask(_grid->Nd(), 1);
-    Coordinate mask(Ndim,1);
+    FFT_dim_mask(result, source, mask, sign);
-    FFT_dim_mask(result,source,mask,sign);
   }
 
-
   template<class vobj>
-  void FFT_dim(Lattice<vobj> &result,const Lattice<vobj> &source,int dim, int sign){
+  void FFT_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int dim, int sign) {
-    const int Ndim = source.Grid()->Nd();
+    GRID_ASSERT(source.Grid() == _grid);
-    GridBase *grid = source.Grid();
+    GRID_ASSERT(result.Grid() == _grid);
-    conformable(result.Grid(),source.Grid());
+    conformable(result.Grid(), source.Grid());
 
-    int L = grid->_ldimensions[dim];
-    int G = grid->_fdimensions[dim];
-      
-    Coordinate layout(Ndim,1);
-    
-    // Construct pencils
-    typedef typename vobj::scalar_object sobj;
     typedef typename vobj::scalar_type   scalar;
-    typedef typename vobj::scalar_type   scalar_type;
+    typedef typename vobj::scalar_object sobj;
-    typedef typename vobj::vector_type   vector_type;
-      
-    //std::cout << "CPU view" << std::endl;
-    
     typedef typename FFTW<scalar>::FFTW_scalar FFTW_scalar;
     typedef typename FFTW<scalar>::FFTW_plan   FFTW_plan;
-      
-    int Ncomp = sizeof(sobj)/sizeof(scalar);
-    int64_t Nlow  = 1;
-    int64_t Nhigh = 1;
 
-    for(int d=0;d<dim;d++){
+    const int Ndim = _grid->Nd();
-      Nlow*=grid->_ldimensions[d];
+    int G     = _grid->_fdimensions[dim];
-    }
+    int Ncomp = sizeof(sobj) / sizeof(scalar);
-    for(int d=dim+1;d<Ndim;d++){
+    int64_t Nperp = 1;
-      Nhigh*=grid->_ldimensions[d];
+    for (int d = 0; d < Ndim; d++)
-    }
+      if (d != dim) Nperp *= _grid->_ldimensions[d];
-    int64_t Nperp=Nlow*Nhigh;
+    int n[]     = {G};
-    
-    deviceVector<scalar> pgbuf; // Layout is [perp][component][dim]
-    pgbuf.resize(Nperp*Ncomp*G);
-    scalar *pgbuf_v = &pgbuf[0];
-      
-    int rank = 1;  /* 1d transforms */
-    int n[] = {G}; /* 1d transforms of length G */
     int howmany = Ncomp * Nperp;
-    int odist,idist,istride,ostride;
-    idist   = odist   = G;            /* Distance between consecutive FT's */
-    istride = ostride = 1;            /* Distance between two elements in the same FT */
-    int *inembed = n, *onembed = n;
-      
-    scalar div;
-    if ( sign == backward ) div = 1.0/G;
-    else if ( sign == forward ) div = 1.0;
-    else GRID_ASSERT(0);
 
-    double t_pencil=0;
+    deviceVector<scalar> dummy(2);
-    double t_fft   =0;
+    FFTW_scalar *buf = (FFTW_scalar *)&dummy[0];
-    double t_total =-usecond();
+    FFTW_plan p = FFTW<scalar>::fftw_plan_many_dft(1, n, howmany,
-    //    std::cout << GridLogPerformance<<"Making FFTW plan" << std::endl;
+                                                    buf, n, 1, G,
-    /*
+                                                    buf, n, 1, G,
-     *
+                                                    sign, FFTW_ESTIMATE);
-     */
+    FFT_dim_execute(result, source, dim, sign, p, _grid, flops, flops_call, usec);
-    FFTW_plan p;
-    {
-      FFTW_scalar *in = (FFTW_scalar *)&pgbuf_v[0];
-      FFTW_scalar *out= (FFTW_scalar *)&pgbuf_v[0];
-      p = FFTW<scalar>::fftw_plan_many_dft(rank,n,howmany,
-					   in,inembed,
-					   istride,idist,
-					   out,onembed,
-					   ostride, odist,
-					   sign,FFTW_ESTIMATE);
-    }
-      
-    // Barrel shift and collect global pencil
-    //    std::cout << GridLogPerformance<<"Making pencil" << std::endl;
-    Coordinate lcoor(Ndim), gcoor(Ndim);
-    double t_copy=0;
-    double t_shift=0;
-    t_pencil = -usecond();
-    result = source;
-    int pc = grid->_processor_coor[dim];
-
-    const Coordinate ldims = grid->_ldimensions;
-    const Coordinate rdims = grid->_rdimensions;
-    const Coordinate sdims = grid->_simd_layout;
-
-    Coordinate processors = grid->_processors;
-    Coordinate pgdims(Ndim);
-    pgdims[0] = G;
-    for(int d=0, dd=1;d<Ndim;d++){
-      if ( d!=dim ) pgdims[dd++] = ldims[d];
-    }
-    int64_t pgvol=1;
-    for(int d=0;d<Ndim;d++) pgvol*=pgdims[d];
-    
-    const int Nsimd = vobj::Nsimd();
-    for(int p=0;p<processors[dim];p++) {
-      t_copy-=usecond();
-      autoView(r_v,result,AcceleratorRead);
-      accelerator_for(idx, grid->oSites(), vobj::Nsimd(), {
-#ifdef GRID_SIMT
-      {
-	int lane=acceleratorSIMTlane(Nsimd); // buffer lane
-#else
-      for(int lane=0;lane<Nsimd;lane++) {
-#endif
-	Coordinate icoor;
-	Coordinate ocoor;
-	Coordinate pgcoor;
-
-	Lexicographic::CoorFromIndex(icoor,lane,sdims);
-	Lexicographic::CoorFromIndex(ocoor,idx,rdims);
-
-	pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + ((pc+p)%processors[dim])*L;
-	for(int d=0,dd=1;d<Ndim;d++){
-	  if ( d!=dim ) {
-	    pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d];
-	    dd++;
-	  }
-	}
-
-	// Map coordinates in lattice layout to FFTW index
-	int64_t pgidx;
-	Lexicographic::IndexFromCoor(pgcoor,pgidx,pgdims);
-
-	vector_type *from = (vector_type *)&r_v[idx];
-	scalar_type stmp;
-	for(int w=0;w<Ncomp;w++){
-	  int64_t pg_idx = pgidx + w*pgvol;
-	  stmp = getlane(from[w], lane);
-	  pgbuf_v[pg_idx] = stmp;
-	}
-#ifdef GRID_SIMT
-      }
-#else
-      }
-#endif
-      });
-
-      t_copy+=usecond();
-      if (p != processors[dim] - 1) {
-	Lattice<vobj> temp(grid);
-	t_shift-=usecond();
-	temp = Cshift(result,dim,L); result = temp;
-	t_shift+=usecond();
-      }
-    }
-    t_pencil += usecond();
-      
-    FFTW_scalar *in = (FFTW_scalar *)pgbuf_v;
-    FFTW_scalar *out= (FFTW_scalar *)pgbuf_v;
-    t_fft = -usecond();
-    FFTW<scalar>::fftw_execute_dft(p,in,out,sign);
-    t_fft += usecond();
-    
-    // performance counting
-    flops_call = 5.0*howmany*G*log2(G);
-    usec = t_fft;
-    flops= flops_call;
-
-    result = Zero();
-    
-    double t_insert = -usecond();
-    {
-      autoView(r_v,result,AcceleratorWrite);
-      accelerator_for(idx,grid->oSites(),Nsimd,{
-#ifdef GRID_SIMT
-      {
-	int lane=acceleratorSIMTlane(Nsimd); // buffer lane
-#else
-      for(int lane=0;lane<Nsimd;lane++) {
-#endif
-	Coordinate icoor(Ndim);
-	Coordinate ocoor(Ndim);
-	Coordinate pgcoor(Ndim);
-
-	Lexicographic::CoorFromIndex(icoor,lane,sdims);
-	Lexicographic::CoorFromIndex(ocoor,idx,rdims);
-
-	pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + pc*L;
-	for(int d=0,dd=1;d<Ndim;d++){
-	  if ( d!=dim ) {
-	    pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d];
-	    dd++;
-	  }
-	}
-	// Map coordinates in lattice layout to FFTW index
-	int64_t pgidx;
-	Lexicographic::IndexFromCoor(pgcoor,pgidx,pgdims);
-
-	vector_type *to = (vector_type *)&r_v[idx];
-	scalar_type stmp;
-	for(int w=0;w<Ncomp;w++){
-	  int64_t pg_idx = pgidx + w*pgvol;
-	  stmp = pgbuf_v[pg_idx];
-	  putlane(to[w], stmp, lane);
-	}
-	
-#ifdef GRID_SIMT
-      }
-#else
-      }
-#endif
-      });
-    }
-
-    result = result*div;
-
-    t_insert +=usecond();
-    
-    // destroying plan
     FFTW<scalar>::fftw_destroy_plan(p);
+  }
+};
 
-    t_total +=usecond();
+template<class vobj>
+class PlannedFFT : public FFTbase {
+private:
+  typedef typename vobj::scalar_type   scalar;
+  typedef typename vobj::scalar_object sobj;
+  typedef typename vobj::vector_type   vector_type;
+  typedef typename FFTW<scalar>::FFTW_scalar FFTW_scalar;
+  typedef typename FFTW<scalar>::FFTW_plan   FFTW_plan;
 
-    std::cout <<GridLogPerformance<< " FFT took   "<<t_total/1.0e6 <<" s" << std::endl;
+  std::vector<FFTW_plan> forward_plans;
-    std::cout <<GridLogPerformance<< " FFT pencil "<<t_pencil/1.0e6 <<" s" << std::endl;
+  std::vector<FFTW_plan> backward_plans;
-    std::cout <<GridLogPerformance<< "  of which copy "<<t_copy/1.0e6 <<" s" << std::endl;
+
-    std::cout <<GridLogPerformance<< "  of which shift"<<t_shift/1.0e6 <<" s" << std::endl;
+  void PlanCreate() {
-    std::cout <<GridLogPerformance<< " FFT kernels "<<t_fft/1.0e6 <<" s" << std::endl;
+    const int Ndim = _grid->Nd();
-    std::cout <<GridLogPerformance<< " FFT insert  "<<t_insert/1.0e6 <<" s" << std::endl;
+    forward_plans.resize(Ndim);
-    
+    backward_plans.resize(Ndim);
+
+    for (int d = 0; d < Ndim; d++) {
+      int G       = _grid->_fdimensions[d];
+      int Ncomp   = sizeof(sobj) / sizeof(scalar);
+      int64_t Nperp = 1;
+      for (int dd = 0; dd < Ndim; dd++)
+        if (dd != d) Nperp *= _grid->_ldimensions[dd];
+      int howmany = Ncomp * (int)Nperp;
+      int n[]     = {G};
+
+      deviceVector<scalar> dummy(2);
+      FFTW_scalar *buf = (FFTW_scalar *)&dummy[0];
+
+      forward_plans[d]  = FFTW<scalar>::fftw_plan_many_dft(1, n, howmany, buf, n, 1, G, buf, n, 1, G, FFTW_FORWARD,  FFTW_ESTIMATE);
+      backward_plans[d] = FFTW<scalar>::fftw_plan_many_dft(1, n, howmany, buf, n, 1, G, buf, n, 1, G, FFTW_BACKWARD, FFTW_ESTIMATE);
+    }
+  }
+
+  void PlanDestroy() {
+    for (auto p : forward_plans)  FFTW<scalar>::fftw_destroy_plan(p);
+    for (auto p : backward_plans) FFTW<scalar>::fftw_destroy_plan(p);
+    forward_plans.clear();
+    backward_plans.clear();
+  }
+
+public:
+  PlannedFFT(GridCartesian *grid) : FFTbase(grid) { PlanCreate(); }
+  ~PlannedFFT() { PlanDestroy(); }
+
+  void FFT_dim_mask(Lattice<vobj> &result, const Lattice<vobj> &source, Coordinate mask, int sign) {
+    const int Ndim = _grid->Nd();
+    Lattice<vobj> tmp = source;
+    for (int d = 0; d < Ndim; d++) {
+      if (mask[d]) {
+        FFT_dim(result, tmp, d, sign);
+        tmp = result;
+      }
+    }
+  }
+
+  void FFT_all_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int sign) {
+    Coordinate mask(_grid->Nd(), 1);
+    FFT_dim_mask(result, source, mask, sign);
+  }
+
+  void FFT_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int dim, int sign) {
+    GRID_ASSERT(source.Grid() == _grid);
+    GRID_ASSERT(result.Grid() == _grid);
+    GRID_ASSERT((int)forward_plans.size() == _grid->Nd());
+    conformable(result.Grid(), source.Grid());
+    FFTW_plan p = (sign == forward ? forward_plans : backward_plans)[dim];
+    FFT_dim_execute(result, source, dim, sign, p, _grid, flops, flops_call, usec);
   }
 };
 

Grid/algorithms/blas/BatchedBlas.h

@@ -68,6 +68,59 @@ NAMESPACE_BEGIN(Grid);
 enum GridBLASOperation_t { GridBLAS_OP_N, GridBLAS_OP_T, GridBLAS_OP_C } ;
 enum GridBLASPrecision_t { GridBLAS_PRECISION_DEFAULT, GridBLAS_PRECISION_16F, GridBLAS_PRECISION_16BF, GridBLAS_PRECISION_TF32 };
 
+///////////////////////////////////////////////////////////////////////////
+// BLAS scalar constants: the put() wrapper OWNS the residency policy so
+// call sites just pass values (scalars-live-on-the-host rule).
+//
+// Policy per backend:
+//  - CUDA / HIP : handles are put in HOST pointer mode at Init (cuBLAS docs
+//    2.2.7: host mode is the documented default; 2.1.5: host-mode scalars
+//    are consumed AT CALL TIME, "can be freed just after the return of the
+//    call even though the kernel launch is asynchronous").  put() stores
+//    the value in persistent host memory and returns its address: ZERO
+//    host->device copies.
+//  - SYCL : the oneMKL group-API alpha/beta arrays are dereferenced
+//    USM-side (spec is silent for the group API; implementation observed
+//    to require USM-accessible storage -- host stack pointers fault).
+//    put() keeps a device-resident copy with VALUE CACHING: the copy is
+//    issued only when the value changes (accumulation pattern
+//    beta = (p==0 ? 0 : 1) costs two copies per Mult instead of npoint).
+//
+// Motivation (rocprof, Frontier, 2026-08-13): per-call alpha/beta device
+// staging generated ~92k tiny staged hipMemcpys in a 12s solve window
+// (~26% of host API time) at the latency-bound coarse level.
+// NB not thread safe -- matches the single-threaded host BLAS call
+// pattern of the per-call staging it replaces.
+///////////////////////////////////////////////////////////////////////////
+template<class T>
+class GridBLASDeviceConstant {
+#ifdef GRID_SYCL
+  deviceVector<T> dev;
+  T               host;
+  int             valid;
+public:
+  GridBLASDeviceConstant() : dev(1), valid(0) {};
+  T * put(T v) {
+    if ( (!valid) || (v != host) ) {
+      acceleratorCopyToDevice((void *)&v,(void *)&dev[0],sizeof(T));
+      host  = v;
+      valid = 1;
+    }
+    return &dev[0];
+  }
+#else
+  // CUDA / HIP in HOST pointer mode (and CPU/Eigen, where the pointer is
+  // unused): persistent host storage, no copies ever.
+  T host;
+public:
+  GridBLASDeviceConstant() {};
+  T * put(T v) {
+    host = v;
+    return &host;
+  }
+#endif
+};
+
 class GridBLAS {
 public:
 
@@ -81,11 +134,31 @@ public:
 #ifdef GRID_CUDA
       std::cout << "cublasCreate"<<std::endl;
       cublasCreate(&gridblasHandle);
-      cublasSetPointerMode(gridblasHandle, CUBLAS_POINTER_MODE_DEVICE);
+      // HOST pointer mode: scalars consumed at call time from host memory
+      // (cuBLAS docs 2.1.5/2.2.7) -- no device staging of alpha/beta.
+      // DEVICE mode would be a deliberate opt-in for device-produced
+      // scalars (e.g. a future graph-captured solver).
+      cublasSetPointerMode(gridblasHandle, CUBLAS_POINTER_MODE_HOST);
+      {
+        cublasPointerMode_t pm;
+        cublasGetPointerMode(gridblasHandle,&pm);
+        std::cout << "GridBLAS: cuBLAS pointer mode "
+                  << ((pm==CUBLAS_POINTER_MODE_DEVICE)?"DEVICE":"HOST") <<std::endl;
+      }
 #endif
 #ifdef GRID_HIP
       std::cout << "hipblasCreate"<<std::endl;
       hipblasCreate(&gridblasHandle);
+      // Explicit HOST mode: the hipBLAS default is UNDOCUMENTED in the
+      // headers (enum 0 == HOST by cuBLAS-mirroring convention only);
+      // set it and print it so every log carries the ground truth.
+      hipblasSetPointerMode(gridblasHandle, HIPBLAS_POINTER_MODE_HOST);
+      {
+        hipblasPointerMode_t pm;
+        hipblasGetPointerMode(gridblasHandle,&pm);
+        std::cout << "GridBLAS: hipBLAS pointer mode "
+                  << ((pm==HIPBLAS_POINTER_MODE_DEVICE)?"DEVICE":"HOST") <<std::endl;
+      }
 #endif
 #ifdef GRID_SYCL
       gridblasHandle = theGridAccelerator;
@@ -240,11 +313,11 @@ public:
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
     
-    static deviceVector<ComplexD> alpha_p(1);
+    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static deviceVector<ComplexD> beta_p(1);
+    static GridBLASDeviceConstant<ComplexD> alpha_c;
-    // can prestore the 1 and the zero on device
+    static GridBLASDeviceConstant<ComplexD> beta_c;
-    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(ComplexD));
+    ComplexD *alpha_p = alpha_c.put(alpha);
-    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(ComplexD));
+    ComplexD *beta_p  = beta_c.put(beta);
     RealD t0=usecond();
     //    std::cout << "ZgemmBatched mnk  "<<m<<","<<n<<","<<k<<" count "<<batchCount<<std::endl;
 #ifdef GRID_HIP
@@ -498,11 +571,11 @@ public:
       lda = k;
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
-    static deviceVector<ComplexF> alpha_p(1);
+    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static deviceVector<ComplexF> beta_p(1);
+    static GridBLASDeviceConstant<ComplexF> alpha_c;
-    // can prestore the 1 and the zero on device
+    static GridBLASDeviceConstant<ComplexF> beta_c;
-    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(ComplexF));
+    ComplexF *alpha_p = alpha_c.put(alpha);
-    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(ComplexF));
+    ComplexF *beta_p  = beta_c.put(beta);
     RealD t0=usecond();
 
     GRID_ASSERT(Bkn.size()==batchCount);
@@ -695,6 +768,229 @@ public:
      RealD bytes = 1.0*sizeof(ComplexF)*(m*k+k*n+m*n)*batchCount;
   }
   
+  ///////////////////////////////////////////////////////////////////////////////////
+  // Explicit-leading-dimension complex single GEMM.
+  //
+  // A,B,C may be SLICES of larger parent allocations: lda/ldb/ldc are the
+  // PARENT strides (>= the compact values the ten-argument overload derives).
+  // Motivating use: software split-K for tiny-output/huge-K dense multiplies
+  // (arXiv:2409.03904 fig 11; cf MultiRHSBlockCGLinalg) -- batch over K-chunks
+  // of a dense slab by pointer offset j*Kchunk with lda = the full K extent,
+  // then reduce the partial C's.  Backends pass lda straight through; only
+  // the compact overload invented them.
+  ///////////////////////////////////////////////////////////////////////////////////
+  void gemmBatched(GridBLASOperation_t OpA,
+		   GridBLASOperation_t OpB,
+		   int m,int n, int k,
+		   ComplexF alpha,
+		   deviceVector<ComplexF*> &Amk, int lda,
+		   deviceVector<ComplexF*> &Bkn, int ldb,
+		   ComplexF beta,
+		   deviceVector<ComplexF*> &Cmn, int ldc,
+		   GridBLASPrecision_t precision = GridBLAS_PRECISION_DEFAULT)
+  {
+    RealD t2=usecond();
+    int32_t batchCount = Amk.size();
+
+    GRID_ASSERT( lda >= ((OpA==GridBLAS_OP_N) ? m : k) );
+    GRID_ASSERT( ldb >= ((OpB==GridBLAS_OP_N) ? k : n) );
+    GRID_ASSERT( ldc >= m );
+
+    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
+    static GridBLASDeviceConstant<ComplexF> alpha_c;
+    static GridBLASDeviceConstant<ComplexF> beta_c;
+    ComplexF *alpha_p = alpha_c.put(alpha);
+    ComplexF *beta_p  = beta_c.put(beta);
+    RealD t0=usecond();
+
+    GRID_ASSERT(Bkn.size()==batchCount);
+    GRID_ASSERT(Cmn.size()==batchCount);
+#ifdef GRID_HIP
+    GRID_ASSERT(precision == GridBLAS_PRECISION_DEFAULT);
+    hipblasOperation_t hOpA;
+    hipblasOperation_t hOpB;
+    if ( OpA == GridBLAS_OP_N ) hOpA = HIPBLAS_OP_N;
+    if ( OpA == GridBLAS_OP_T ) hOpA = HIPBLAS_OP_T;
+    if ( OpA == GridBLAS_OP_C ) hOpA = HIPBLAS_OP_C;
+    if ( OpB == GridBLAS_OP_N ) hOpB = HIPBLAS_OP_N;
+    if ( OpB == GridBLAS_OP_T ) hOpB = HIPBLAS_OP_T;
+    if ( OpB == GridBLAS_OP_C ) hOpB = HIPBLAS_OP_C;
+#if defined(HIP_VERSION_MAJOR) && (HIP_VERSION_MAJOR >=7)
+    auto err = hipblasCgemmBatched(gridblasHandle,
+				   hOpA,
+				   hOpB,
+				   m,n,k,
+				   (hipComplex *) &alpha_p[0],
+				   (hipComplex **)&Amk[0], lda,
+				   (hipComplex **)&Bkn[0], ldb,
+				   (hipComplex *) &beta_p[0],
+				   (hipComplex **)&Cmn[0], ldc,
+				   batchCount);
+#else
+    auto err = hipblasCgemmBatched(gridblasHandle,
+                                   hOpA,
+                                   hOpB,
+                                   m,n,k,
+                                   (hipblasComplex *) &alpha_p[0],
+                                   (hipblasComplex **)&Amk[0], lda,
+                                   (hipblasComplex **)&Bkn[0], ldb,
+                                   (hipblasComplex *) &beta_p[0],
+                                   (hipblasComplex **)&Cmn[0], ldc,
+                                   batchCount);
+#endif
+    GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
+#endif
+#ifdef GRID_CUDA
+    cublasOperation_t hOpA;
+    cublasOperation_t hOpB;
+    if ( OpA == GridBLAS_OP_N ) hOpA = CUBLAS_OP_N;
+    if ( OpA == GridBLAS_OP_T ) hOpA = CUBLAS_OP_T;
+    if ( OpA == GridBLAS_OP_C ) hOpA = CUBLAS_OP_C;
+    if ( OpB == GridBLAS_OP_N ) hOpB = CUBLAS_OP_N;
+    if ( OpB == GridBLAS_OP_T ) hOpB = CUBLAS_OP_T;
+    if ( OpB == GridBLAS_OP_C ) hOpB = CUBLAS_OP_C;
+    cublasStatus_t err;
+    if (precision == GridBLAS_PRECISION_DEFAULT) {
+      err = cublasCgemmBatched(gridblasHandle,
+			       hOpA,
+			       hOpB,
+			       m,n,k,
+			       (cuComplex *) &alpha_p[0],
+			       (cuComplex **)&Amk[0], lda,
+			       (cuComplex **)&Bkn[0], ldb,
+			       (cuComplex *) &beta_p[0],
+			       (cuComplex **)&Cmn[0], ldc,
+			       batchCount);
+    } else {
+      cublasComputeType_t compute_precision = toDataType(precision);
+      err = cublasGemmBatchedEx(gridblasHandle,
+				hOpA,
+				hOpB,
+				m,n,k,
+				(void *) &alpha_p[0],
+				(void **)&Amk[0], CUDA_C_32F, lda,
+				(void **)&Bkn[0], CUDA_C_32F, ldb,
+				(void *) &beta_p[0],
+				(void **)&Cmn[0], CUDA_C_32F, ldc,
+				batchCount, compute_precision, CUBLAS_GEMM_DEFAULT);
+    }
+    GRID_ASSERT(err==CUBLAS_STATUS_SUCCESS);
+#endif
+#ifdef GRID_SYCL
+    GRID_ASSERT(precision == GridBLAS_PRECISION_DEFAULT);
+    int64_t m64=m;
+    int64_t n64=n;
+    int64_t k64=k;
+    int64_t lda64=lda;
+    int64_t ldb64=ldb;
+    int64_t ldc64=ldc;
+    int64_t batchCount64=batchCount;
+
+    oneapi::mkl::transpose iOpA;
+    oneapi::mkl::transpose iOpB;
+
+    if ( OpA == GridBLAS_OP_N ) iOpA = oneapi::mkl::transpose::N;
+    if ( OpA == GridBLAS_OP_T ) iOpA = oneapi::mkl::transpose::T;
+    if ( OpA == GridBLAS_OP_C ) iOpA = oneapi::mkl::transpose::C;
+    if ( OpB == GridBLAS_OP_N ) iOpB = oneapi::mkl::transpose::N;
+    if ( OpB == GridBLAS_OP_T ) iOpB = oneapi::mkl::transpose::T;
+    if ( OpB == GridBLAS_OP_C ) iOpB = oneapi::mkl::transpose::C;
+
+    oneapi::mkl::blas::column_major::gemm_batch(*gridblasHandle,
+						&iOpA,
+						&iOpB,
+						&m64,&n64,&k64,
+						(ComplexF *) &alpha_p[0],
+						(const ComplexF **)&Amk[0], (const int64_t *)&lda64,
+						(const ComplexF **)&Bkn[0], (const int64_t *)&ldb64,
+						(ComplexF *) &beta_p[0],
+						(ComplexF **)&Cmn[0], (const int64_t *)&ldc64,
+						(int64_t)1,&batchCount64,std::vector<sycl::event>());
+    synchronise();
+#endif
+#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
+    GRID_ASSERT(precision == GridBLAS_PRECISION_DEFAULT);
+    // Reference implementation: Eigen with explicit outer stride
+    typedef Eigen::Map<Eigen::MatrixXcf,0,Eigen::OuterStride<> > eMat;
+    if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_N) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk * eBkn ;
+	else
+	  eCmn = alpha * eAmk * eBkn ;
+	});
+    } else if ( (OpA == GridBLAS_OP_C ) && (OpB == GridBLAS_OP_N) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk.adjoint() * eBkn ;
+	else
+	  eCmn = alpha * eAmk.adjoint() * eBkn ;
+	});
+    } else if ( (OpA == GridBLAS_OP_T ) && (OpB == GridBLAS_OP_N) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],k,n,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk.transpose() * eBkn ;
+	else
+	  eCmn = alpha * eAmk.transpose() * eBkn ;
+	});
+    } else if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_C) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk * eBkn.adjoint() ;
+	else
+	  eCmn = alpha * eAmk * eBkn.adjoint() ;
+	});
+    } else if ( (OpA == GridBLAS_OP_N ) && (OpB == GridBLAS_OP_T) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],m,k,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk * eBkn.transpose() ;
+	else
+	  eCmn = alpha * eAmk * eBkn.transpose() ;
+	});
+    } else if ( (OpA == GridBLAS_OP_C ) && (OpB == GridBLAS_OP_C) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk.adjoint() * eBkn.adjoint() ;
+	else
+	  eCmn = alpha * eAmk.adjoint() * eBkn.adjoint() ;
+	} );
+    } else if ( (OpA == GridBLAS_OP_T ) && (OpB == GridBLAS_OP_T) ) {
+      thread_for (p, batchCount, {
+	eMat eAmk(Amk[p],k,m,Eigen::OuterStride<>(lda));
+	eMat eBkn(Bkn[p],n,k,Eigen::OuterStride<>(ldb));
+	eMat eCmn(Cmn[p],m,n,Eigen::OuterStride<>(ldc));
+	if (std::abs(beta) != 0.0)
+	  eCmn = beta * eCmn + alpha * eAmk.transpose() * eBkn.transpose() ;
+	else
+	  eCmn = alpha * eAmk.transpose() * eBkn.transpose() ;
+	} );
+    } else {
+      assert(0);
+    }
+#endif
+    RealD t1=usecond();
+    RealD flops = 8.0*m*n*k*batchCount;
+    RealD bytes = 1.0*sizeof(ComplexF)*(m*k+k*n+m*n)*batchCount;
+  }
+
   ///////////////////////////////////////////////////////////////////////////
   // Single precision real GEMM
   ///////////////////////////////////////////////////////////////////////////
@@ -721,11 +1017,11 @@ public:
       lda = k;
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
-    static deviceVector<RealF> alpha_p(1);
+    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static deviceVector<RealF> beta_p(1);
+    static GridBLASDeviceConstant<RealF> alpha_c;
-    // can prestore the 1 and the zero on device
+    static GridBLASDeviceConstant<RealF> beta_c;
-    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(RealF));
+    RealF *alpha_p = alpha_c.put(alpha);
-    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(RealF));
+    RealF *beta_p  = beta_c.put(beta);
     RealD t0=usecond();
 
     GRID_ASSERT(Bkn.size()==batchCount);
@@ -882,11 +1178,11 @@ public:
     if(OpB!=GridBLAS_OP_N)
       ldb = n;
     
-    static deviceVector<RealD> alpha_p(1);
+    // Cached device constants: copy only on value change (see GridBLASDeviceConstant)
-    static deviceVector<RealD> beta_p(1);
+    static GridBLASDeviceConstant<RealD> alpha_c;
-    // can prestore the 1 and the zero on device
+    static GridBLASDeviceConstant<RealD> beta_c;
-    acceleratorCopyToDevice((void *)&alpha,(void *)&alpha_p[0],sizeof(RealD));
+    RealD *alpha_p = alpha_c.put(alpha);
-    acceleratorCopyToDevice((void *)&beta ,(void *)&beta_p[0],sizeof(RealD));
+    RealD *beta_p  = beta_c.put(beta);
     RealD t0=usecond();
 
     GRID_ASSERT(Bkn.size()==batchCount);

Grid/algorithms/blas/BatchedInverse.h

@@ -0,0 +1,282 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: BatchedInverse.h
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#pragma once
+
+#include <Grid/algorithms/blas/BatchedBlas.h>
+
+#ifdef GRID_HIP
+#include <rocsolver/rocsolver.h>
+#endif
+// GRID_CUDA: batched LU inversion lives in cuBLAS (getrfBatched/getriBatched);
+//            cublas_v2.h already included via BatchedBlas.h.
+// GRID_SYCL: oneapi/mkl.hpp already included via BatchedBlas.h (lapack::getrf/getri).
+
+NAMESPACE_BEGIN(Grid);
+
+///////////////////////////////////////////////////////////////////////////////
+// GridBLASInverse: cross-platform batched dense matrix inversion.
+//
+// HIGH LEVEL contract (deliberately NOT a getrf/getrs interface): invert a
+// batch of dense N x N matrices IN PLACE,
+//
+//     A[i] <- A[i]^{-1}       i = 0 .. batchCount-1
+//
+// Layout: column major, lda = N, contiguous per batch element; pointer list
+// exactly as GridBLAS::gemmBatched (deviceVector<T*> of device pointers).
+// Each backend chooses HOW:
+//   HIP  : rocSOLVER getrf_batched + getri_batched
+//   CUDA : cuBLAS getrfBatched + getriBatched (out-of-place getri; workspace
+//          hidden here, result copied back so the surface stays in-place)
+//   SYCL : oneMKL LAPACK getrf + getri per batch element (USM, in-order queue)
+//   CPU  : Eigen PartialPivLU (the correctness oracle for all of the above)
+//
+// The int32 vendor-batched entry points bound N < 2^31 (asserted); the huge
+// single-matrix ILP64 path (getrf_64 + blocked identity-getrs harvest, proven
+// in the dense coarse-coarse setup at N=69120) migrates here as a batch==1
+// large-N dispatch in a follow-up -- the recursive Schur leaves are the
+// batched consumers this surface is shaped for.
+//
+// NB GPU-backend call signatures are written to vendor documentation but the
+// air-gapped development loop compiles only the CPU/Eigen path; verify the
+// rocSOLVER/cuBLAS/oneMKL calls against headers on first device compile.
+// Semantics are locked by the CPU unit test (Test_batched_blas).
+///////////////////////////////////////////////////////////////////////////////
+class GridBLASInverse {
+public:
+
+#ifdef GRID_HIP
+  // rocSOLVER runs on a rocblas_handle (distinct type from hipblasHandle_t)
+  static rocblas_handle & Handle(void) {
+    static rocblas_handle h;
+    static int init = 0;
+    if ( !init ) {
+      auto st = rocblas_create_handle(&h);
+      GRID_ASSERT(st == rocblas_status_success);
+      init = 1;
+    }
+    return h;
+  }
+#endif
+#ifdef GRID_CUDA
+  // cuBLAS batched LU shares the GridBLAS handle
+  static cublasHandle_t & Handle(void) {
+    GridBLAS::Init();
+    return GridBLAS::gridblasHandle;
+  }
+#endif
+#ifdef GRID_SYCL
+  static sycl::queue * & Handle(void) {
+    GridBLAS::Init();
+    return GridBLAS::gridblasHandle;
+  }
+#endif
+
+  GridBLASInverse() {};
+  ~GridBLASInverse() {};
+
+  void inverseBatched(int64_t N, deviceVector<ComplexF*> &Amat)
+  {
+    int32_t batchCount = Amat.size();
+    GRID_ASSERT(batchCount > 0);
+
+#ifdef GRID_HIP
+    GRID_ASSERT( N < 2147483647L );
+    rocblas_int n   = (rocblas_int)N;
+    rocblas_int lda = (rocblas_int)N;
+
+    deviceVector<rocblas_int> ipiv((uint64_t)batchCount*N);
+    deviceVector<rocblas_int> info(batchCount);
+
+    auto st1 = rocsolver_cgetrf_batched(Handle(), n, n,
+					(rocblas_float_complex *const *)&Amat[0], lda,
+					&ipiv[0], (rocblas_stride)N,
+					&info[0], batchCount);
+    GRID_ASSERT(st1 == rocblas_status_success);
+    auto st2 = rocsolver_cgetri_batched(Handle(), n,
+					(rocblas_float_complex *const *)&Amat[0], lda,
+					&ipiv[0], (rocblas_stride)N,
+					&info[0], batchCount);
+    GRID_ASSERT(st2 == rocblas_status_success);
+    accelerator_barrier();
+    std::vector<rocblas_int> info_h(batchCount);
+    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(rocblas_int));
+    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0); // singular pivot => abort loudly
+#endif
+#ifdef GRID_CUDA
+    GRID_ASSERT( N < 2147483647L );
+    int n = (int)N;
+
+    deviceVector<int> ipiv((uint64_t)batchCount*N);
+    deviceVector<int> info(batchCount);
+
+    auto st1 = cublasCgetrfBatched(Handle(), n,
+				   (cuComplex **)&Amat[0], n,
+				   &ipiv[0], &info[0], batchCount);
+    GRID_ASSERT(st1 == CUBLAS_STATUS_SUCCESS);
+
+    // getri is OUT of place: hidden workspace keeps the surface in-place
+    deviceVector<ComplexF>  work((uint64_t)batchCount*N*N);
+    deviceVector<ComplexF*> Cptr(batchCount);
+    std::vector<ComplexF*>  Cptr_h(batchCount);
+    std::vector<ComplexF*>  Aptr_h(batchCount);
+    for(int i=0;i<batchCount;i++) Cptr_h[i] = &work[(uint64_t)i*N*N];
+    acceleratorCopyToDevice(&Cptr_h[0],&Cptr[0],batchCount*sizeof(ComplexF*));
+    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexF*));
+
+    auto st2 = cublasCgetriBatched(Handle(), n,
+				   (const cuComplex *const *)&Amat[0], n,
+				   &ipiv[0],
+				   (cuComplex **)&Cptr[0], n,
+				   &info[0], batchCount);
+    GRID_ASSERT(st2 == CUBLAS_STATUS_SUCCESS);
+    accelerator_barrier();
+    std::vector<int> info_h(batchCount);
+    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(int));
+    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0);
+    for(int i=0;i<batchCount;i++)
+      acceleratorCopyDeviceToDevice(Cptr_h[i],Aptr_h[i],(uint64_t)N*N*sizeof(ComplexF));
+#endif
+#ifdef GRID_SYCL
+    // Per-element oneMKL LAPACK on the in-order queue; group API optimisation later.
+    sycl::queue *q = Handle();
+    std::vector<ComplexF*> Aptr_h(batchCount);
+    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexF*));
+
+    int64_t lwf = oneapi::mkl::lapack::getrf_scratchpad_size<std::complex<float> >(*q,N,N,N);
+    int64_t lwi = oneapi::mkl::lapack::getri_scratchpad_size<std::complex<float> >(*q,N,N);
+    deviceVector<ComplexF> scratchf(lwf);
+    deviceVector<ComplexF> scratchi(lwi);
+    deviceVector<int64_t>  ipiv(N);
+    for(int i=0;i<batchCount;i++){
+      oneapi::mkl::lapack::getrf(*q,N,N,(std::complex<float>*)Aptr_h[i],N,&ipiv[0],
+				 (std::complex<float>*)&scratchf[0],lwf);
+      oneapi::mkl::lapack::getri(*q,N,  (std::complex<float>*)Aptr_h[i],N,&ipiv[0],
+				 (std::complex<float>*)&scratchi[0],lwi);
+    }
+    q->wait();
+#endif
+#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
+    // Reference implementation; the oracle the unit test locks semantics with.
+    thread_for (p, batchCount, {
+      Eigen::Map<Eigen::MatrixXcf> eA(Amat[p],N,N);
+      Eigen::PartialPivLU<Eigen::MatrixXcf> lu(eA);
+      eA = lu.inverse();
+      });
+#endif
+  }
+
+  void inverseBatched(int64_t N, deviceVector<ComplexD*> &Amat)
+  {
+    int32_t batchCount = Amat.size();
+    GRID_ASSERT(batchCount > 0);
+
+#ifdef GRID_HIP
+    GRID_ASSERT( N < 2147483647L );
+    rocblas_int n   = (rocblas_int)N;
+    rocblas_int lda = (rocblas_int)N;
+
+    deviceVector<rocblas_int> ipiv((uint64_t)batchCount*N);
+    deviceVector<rocblas_int> info(batchCount);
+
+    auto st1 = rocsolver_zgetrf_batched(Handle(), n, n,
+					(rocblas_double_complex *const *)&Amat[0], lda,
+					&ipiv[0], (rocblas_stride)N,
+					&info[0], batchCount);
+    GRID_ASSERT(st1 == rocblas_status_success);
+    auto st2 = rocsolver_zgetri_batched(Handle(), n,
+					(rocblas_double_complex *const *)&Amat[0], lda,
+					&ipiv[0], (rocblas_stride)N,
+					&info[0], batchCount);
+    GRID_ASSERT(st2 == rocblas_status_success);
+    accelerator_barrier();
+    std::vector<rocblas_int> info_h(batchCount);
+    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(rocblas_int));
+    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0);
+#endif
+#ifdef GRID_CUDA
+    GRID_ASSERT( N < 2147483647L );
+    int n = (int)N;
+
+    deviceVector<int> ipiv((uint64_t)batchCount*N);
+    deviceVector<int> info(batchCount);
+
+    auto st1 = cublasZgetrfBatched(Handle(), n,
+				   (cuDoubleComplex **)&Amat[0], n,
+				   &ipiv[0], &info[0], batchCount);
+    GRID_ASSERT(st1 == CUBLAS_STATUS_SUCCESS);
+
+    deviceVector<ComplexD>  work((uint64_t)batchCount*N*N);
+    deviceVector<ComplexD*> Cptr(batchCount);
+    std::vector<ComplexD*>  Cptr_h(batchCount);
+    std::vector<ComplexD*>  Aptr_h(batchCount);
+    for(int i=0;i<batchCount;i++) Cptr_h[i] = &work[(uint64_t)i*N*N];
+    acceleratorCopyToDevice(&Cptr_h[0],&Cptr[0],batchCount*sizeof(ComplexD*));
+    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexD*));
+
+    auto st2 = cublasZgetriBatched(Handle(), n,
+				   (const cuDoubleComplex *const *)&Amat[0], n,
+				   &ipiv[0],
+				   (cuDoubleComplex **)&Cptr[0], n,
+				   &info[0], batchCount);
+    GRID_ASSERT(st2 == CUBLAS_STATUS_SUCCESS);
+    accelerator_barrier();
+    std::vector<int> info_h(batchCount);
+    acceleratorCopyFromDevice(&info[0],&info_h[0],batchCount*sizeof(int));
+    for(int i=0;i<batchCount;i++) GRID_ASSERT(info_h[i]==0);
+    for(int i=0;i<batchCount;i++)
+      acceleratorCopyDeviceToDevice(Cptr_h[i],Aptr_h[i],(uint64_t)N*N*sizeof(ComplexD));
+#endif
+#ifdef GRID_SYCL
+    sycl::queue *q = Handle();
+    std::vector<ComplexD*> Aptr_h(batchCount);
+    acceleratorCopyFromDevice(&Amat[0],&Aptr_h[0],batchCount*sizeof(ComplexD*));
+
+    int64_t lwf = oneapi::mkl::lapack::getrf_scratchpad_size<std::complex<double> >(*q,N,N,N);
+    int64_t lwi = oneapi::mkl::lapack::getri_scratchpad_size<std::complex<double> >(*q,N,N);
+    deviceVector<ComplexD> scratchf(lwf);
+    deviceVector<ComplexD> scratchi(lwi);
+    deviceVector<int64_t>  ipiv(N);
+    for(int i=0;i<batchCount;i++){
+      oneapi::mkl::lapack::getrf(*q,N,N,(std::complex<double>*)Aptr_h[i],N,&ipiv[0],
+				 (std::complex<double>*)&scratchf[0],lwf);
+      oneapi::mkl::lapack::getri(*q,N,  (std::complex<double>*)Aptr_h[i],N,&ipiv[0],
+				 (std::complex<double>*)&scratchi[0],lwi);
+    }
+    q->wait();
+#endif
+#if !defined(GRID_SYCL) && !defined(GRID_CUDA) && !defined(GRID_HIP)
+    thread_for (p, batchCount, {
+      Eigen::Map<Eigen::MatrixXcd> eA(Amat[p],N,N);
+      Eigen::PartialPivLU<Eigen::MatrixXcd> lu(eA);
+      eA = lu.inverse();
+      });
+#endif
+  }
+};
+
+NAMESPACE_END(Grid);

Grid/algorithms/iterative/AdefGeneric.h

@@ -53,7 +53,22 @@ class TwoLevelCG : public LinearFunction<Field>
   // Fine operator, Smoother, CoarseSolver
   LinearOperatorBase<Field>   &_FineLinop;
   LinearFunction<Field>   &_Smoother;
-  
+
+  GridStopWatch ProjectTimer;
+  GridStopWatch PromoteTimer;
+  GridStopWatch CoarseTimer;
+  GridStopWatch SmoothTimer;
+  GridStopWatch MatrixTimer;
+  GridStopWatch M3Timer;
+  GridStopWatch LinalgTimer;
+
+  int64_t M3Calls;
+  int64_t SmoothCalls;
+  int64_t MatrixCalls;
+  int64_t ProjectCalls;
+  int64_t CoarseCalls;
+  int64_t PromoteCalls;
+
   // more most opertor functions
   TwoLevelCG(RealD tol,
 	     Integer maxit,
@@ -103,12 +118,20 @@ class TwoLevelCG : public LinearFunction<Field>
     RealD tn;
     
     GridStopWatch HDCGTimer;
+    ProjectTimer.Reset();
+    PromoteTimer.Reset();
+    CoarseTimer.Reset();
+    SmoothTimer.Reset();
+    MatrixTimer.Reset();
+    M3Timer.Reset();
+    LinalgTimer.Reset();
+    M3Calls = SmoothCalls = MatrixCalls = ProjectCalls = CoarseCalls = PromoteCalls = 0;
     HDCGTimer.Start();
     //////////////////////////
     // x0 = Vstart -- possibly modify guess
     //////////////////////////
     Vstart(x,src);
-    
+
     // r0 = b -A x0
     _FineLinop.HermOp(x,mmp[0]);
     axpy (r, -1.0,mmp[0], src);    // Recomputes r=src-Ax0
@@ -145,33 +168,40 @@ class TwoLevelCG : public LinearFunction<Field>
       int peri_kp = (k+1) % mmax;
 
       rtz=rtzp;
+      M3Timer.Start();
       d= PcgM3(p[peri_k],mmp[peri_k]);
+      M3Timer.Stop();
+      M3Calls++;
       a = rtz/d;
-    
+
       // Memorise this
       pAp[peri_k] = d;
-      
+
+      LinalgTimer.Start();
       axpy(x,a,p[peri_k],x);
       RealD rn = axpy_norm(r,-a,mmp[peri_k],r);
+      LinalgTimer.Stop();
 
       // Compute z = M x
       PcgM1(r,z);
-      
+
       {
 	RealD n1,n2;
 	n1=norm2(r);
 	n2=norm2(z);
 	std::cout << GridLogMessage<<"HDCG::fPcg iteration "<<k<<" : vector r,z "<<n1<<" "<<n2<<"\n";
       }
+      LinalgTimer.Start();
       rtzp =real(innerProduct(r,z));
+      LinalgTimer.Stop();
       std::cout << GridLogMessage<<"HDCG::fPcg iteration "<<k<<" : inner rtzp "<<rtzp<<"\n";
 
       //    PcgM2(z,p[0]);
       PcgM2(z,mu); // ADEF-2 this is identity. Axpy possible to eliminate
-      
+
       p[peri_kp]=mu;
 
-      // Standard search direction  p -> z + b p    
+      // Standard search direction  p -> z + b p
       b = (rtzp)/rtz;
       
       int northog;
@@ -202,8 +232,25 @@ class TwoLevelCG : public LinearFunction<Field>
       if ( rn <= rsq ) { 
 
 	HDCGTimer.Stop();
-	std::cout<<GridLogMessage<<"HDCG: fPcg converged in "<<k<<" iterations and "<<HDCGTimer.Elapsed()<<std::endl;;
+	std::cout<<GridLogMessage<<"HDCG: fPcg converged in "<<k<<" iterations and "<<HDCGTimer.Elapsed()<<std::endl;
-	
+	std::cout<<GridLogMessage<<"HDCG: fPcg breakdown"<<std::endl;
+	auto mspc = [](GridStopWatch &sw, int64_t n) -> double {
+	  return (n > 0) ? sw.useconds() * 1e-3 / n : 0.0;
+	};
+	std::cout<<GridLogMessage<<"HDCG: fPcg M3 (fine MVM)  "<<M3Timer.Elapsed()
+		 <<"  "<<M3Calls<<" calls  "<<mspc(M3Timer,M3Calls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg linalg         "<<LinalgTimer.Elapsed()<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg smoother       "<<SmoothTimer.Elapsed()
+		 <<"  "<<SmoothCalls<<" calls  "<<mspc(SmoothTimer,SmoothCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg matrix (in M1) "<<MatrixTimer.Elapsed()
+		 <<"  "<<MatrixCalls<<" calls  "<<mspc(MatrixTimer,MatrixCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg project        "<<ProjectTimer.Elapsed()
+		 <<"  "<<ProjectCalls<<" calls  "<<mspc(ProjectTimer,ProjectCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg coarse         "<<CoarseTimer.Elapsed()
+		 <<"  "<<CoarseCalls<<" calls  "<<mspc(CoarseTimer,CoarseCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg promote        "<<PromoteTimer.Elapsed()
+		 <<"  "<<PromoteCalls<<" calls  "<<mspc(PromoteTimer,PromoteCalls)<<" ms/call"<<std::endl;
+
 	_FineLinop.HermOp(x,mmp[0]);			  
 	axpy(tmp,-1.0,src,mmp[0]);
 	
@@ -475,35 +522,29 @@ class TwoLevelADEF2 : public TwoLevelCG<Field>
     CoarseField PleftProj(this->coarsegrid);
     CoarseField PleftMss_proj(this->coarsegrid);
 
-    GridStopWatch SmootherTimer;
+    this->SmoothTimer.Start();
-    GridStopWatch MatrixTimer;
-    SmootherTimer.Start();
     this->_Smoother(in,Min);
-    SmootherTimer.Stop();
+    this->SmoothTimer.Stop();
+    this->SmoothCalls++;
 
-    MatrixTimer.Start();
+    this->MatrixTimer.Start();
     this->_FineLinop.HermOp(Min,out);
-    MatrixTimer.Stop();
+    this->MatrixTimer.Stop();
+    this->MatrixCalls++;
     axpy(tmp,-1.0,out,in);          // tmp  = in - A Min
 
-    GridStopWatch ProjTimer;
+    this->ProjectTimer.Start();
-    GridStopWatch CoarseTimer;
+    this->_Aggregates.ProjectToSubspace(PleftProj,tmp);
-    GridStopWatch PromTimer;
+    this->ProjectTimer.Stop();
-    ProjTimer.Start();
+    this->ProjectCalls++;
-    this->_Aggregates.ProjectToSubspace(PleftProj,tmp);     
+    this->CoarseTimer.Start();
-    ProjTimer.Stop();
-    CoarseTimer.Start();
     this->_CoarseSolver(PleftProj,PleftMss_proj); // Ass^{-1} [in - A Min]_s
-    CoarseTimer.Stop();
+    this->CoarseTimer.Stop();
-    PromTimer.Start();
+    this->CoarseCalls++;
-    this->_Aggregates.PromoteFromSubspace(PleftMss_proj,tmp);// tmp = Q[in - A Min]  
+    this->PromoteTimer.Start();
-    PromTimer.Stop();
+    this->_Aggregates.PromoteFromSubspace(PleftMss_proj,tmp);// tmp = Q[in - A Min]
-    std::cout << GridLogPerformance << "PcgM1 breakdown "<<std::endl;
+    this->PromoteTimer.Stop();
-    std::cout << GridLogPerformance << "\tSmoother   " << SmootherTimer.Elapsed() <<std::endl;
+    this->PromoteCalls++;
-    std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tProj       " << ProjTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tCoarse     " << CoarseTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tProm       " << PromTimer.Elapsed() <<std::endl;
 
     axpy(out,1.0,Min,tmp); // Min+tmp
   }

Grid/algorithms/iterative/ConjugateGradient.h

@@ -212,13 +212,17 @@ public:
 		  << "\tTarget " << Tolerance << std::endl;
 
 	//	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tSolver Elapsed    " << SolverTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "Time breakdown "<<std::endl;
+	std::cout << GridLogMessage << "\tConstruct  " << ConstructTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tNorm       " << NormTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tAssign     " << AssignTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tSolver     " << SolverTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "Solver breakdown "<<std::endl;
-	std::cout << GridLogPerformance << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
 
 	std::cout << GridLogDebug << "\tMobius flop rate " << DwfFlops/ usecs<< " Gflops " <<std::endl;
 

Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h

@@ -236,4 +236,5 @@ public:
   }
 };
 NAMESPACE_END(Grid);
+#undef GCRLogLevel
 #endif

Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h

@@ -38,13 +38,14 @@ Author: Peter Boyle <paboyle@ph.ed.ac.uk>
 ///////////////////////////////////////////////////////////////////////////////////////////////////////
 NAMESPACE_BEGIN(Grid);
 
-#define GCRLogLevel std::cout << GridLogMessage <<std::string(level,'\t')<< " Level "<<level<<" " 
+#define GCRLogLevel std::cout << GridLogMessage <<std::string(level,'\t')<< name<<" "
 
 template<class Field>
 class PrecGeneralisedConjugateResidualNonHermitian : public LinearFunction<Field> {
 public:                                                
   using LinearFunction<Field>::operator();
   RealD   Tolerance;
+  RealD   SSQ;
   Integer MaxIterations;
   int verbose;
   int mmax;
@@ -54,11 +55,18 @@ public:
   GridStopWatch PrecTimer;
   GridStopWatch MatTimer;
   GridStopWatch LinalgTimer;
+  std::string name;
+  int ZeroGuess  = 0;  // caller contract: guess is always zero => first-cycle r0 = src, skip the apply
+  int FirstCycle = 0;
 
   LinearFunction<Field>     &Preconditioner;
   LinearOperatorBase<Field> &Linop;
 
-  void Level(int lv) { level=lv; };
+  void Name(std::string _name) { name = _name; };
+
+  void Level(int n) { Name("Level " + std::to_string(n)); level = n; }
+
+  void SetZeroGuess(int z) { ZeroGuess = z; };
 
   PrecGeneralisedConjugateResidualNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,LinearFunction<Field> &Prec,int _mmax,int _nstep) : 
     Tolerance(tol), 
@@ -67,8 +75,8 @@ public:
     Preconditioner(Prec),
     mmax(_mmax),
     nstep(_nstep)
-  { 
+  {
-    level=1;
+    Level(1);
     verbose=1;
   };
 
@@ -77,6 +85,7 @@ public:
     //    psi=Zero();
     RealD cp, ssq,rsq;
     ssq=norm2(src);
+    SSQ=ssq;
     rsq=Tolerance*Tolerance*ssq;
       
     Field r(src.Grid());
@@ -89,11 +98,12 @@ public:
     SolverTimer.Start();
 
     steps=0;
+    FirstCycle=1;
     for(int k=0;k<MaxIterations;k++){
 
       cp=GCRnStep(src,psi,rsq);
 
-      GCRLogLevel <<"PGCR("<<mmax<<","<<nstep<<") "<< steps <<" steps cp = "<<cp<<" target "<<rsq <<std::endl;
+      GCRLogLevel <<"PGCR("<<mmax<<","<<nstep<<") "<< steps <<" steps cp = "<<sqrt(cp/ssq)<<" target "<<sqrt(rsq/ssq) <<std::endl;
 
       if(cp<rsq) {
 
@@ -142,21 +152,25 @@ public:
     GCRLogLevel<< "PGCR nStep("<<nstep<<")"<<std::endl;
 
     //////////////////////////////////
-    // initial guess x0 is taken as nonzero.
+    // r0 = src - A x0.  ZeroGuess: on the first cycle x0==0 by caller
-    // r0=src-A x0 = src
+    // contract (enforced here), so r0 = src exactly; skip the apply.
+    // Restart cycles (psi!=0) always do the full computation.
     //////////////////////////////////
-    MatTimer.Start();
+    if (ZeroGuess && FirstCycle) {
-    Linop.Op(psi,Az);
+      psi = Zero();
-    //    zAz = innerProduct(Az,psi);
+      LinalgTimer.Start();
-    zAAz= norm2(Az);
+      r = src;
-    MatTimer.Stop();
+      LinalgTimer.Stop();
-    
+    } else {
+      MatTimer.Start();
+      Linop.Op(psi,Az);
+      MatTimer.Stop();
+      LinalgTimer.Start();
+      r=src-Az;
+      LinalgTimer.Stop();
+    }
+    FirstCycle=0;
 
-    LinalgTimer.Start();
-    r=src-Az;
-    LinalgTimer.Stop();
-    GCRLogLevel<< "PGCR true residual r = src - A psi   "<<norm2(r) <<std::endl;
-    
     /////////////////////
     // p = Prec(r)
     /////////////////////
@@ -181,6 +195,7 @@ public:
     
     cp =norm2(r);
     LinalgTimer.Stop();
+    GCRLogLevel<< "PGCR true residual "<< sqrt(cp/SSQ)     <<std::endl;
 
     for(int k=0;k<nstep;k++){
 
@@ -199,13 +214,12 @@ public:
       cp = axpy_norm(r,-a,q[peri_k],r);
       LinalgTimer.Stop();
 
-      GCRLogLevel<< "PGCR step["<<steps<<"]  resid " << cp << " target " <<rsq<<std::endl; 
+      GCRLogLevel<< "PGCR step["<<steps<<"]  resid " << sqrt(cp/SSQ)<<std::endl;
 
       if((k==nstep-1)||(cp<rsq)){
 	return cp;
       }
 
-
       PrecTimer.Start();
       Preconditioner(r,z);// solve Az = r
       PrecTimer.Stop();
@@ -239,4 +253,6 @@ public:
   }
 };
 NAMESPACE_END(Grid);
+
+#undef GCRLogLevel
 #endif

Grid/algorithms/multigrid/Aggregates.h

@@ -66,7 +66,21 @@ public:
   {
   };
   
-  
+  void GlobalOrthonormalise(void)
+  {
+    // Normalise all vectors
+    for(int i=0;i<nbasis; i++){
+      RealD scale = std::pow(norm2(subspace[i]),-0.5); 
+      subspace[i] = subspace[i]*scale;
+    }
+    for(int i=0;i<nbasis; i++){
+      for(int j=0;j<i; j++){
+	basisOrthogonalize(subspace,subspace[i],j);
+      }
+      RealD scale = std::pow(norm2(subspace[i]),-0.5); 
+      subspace[i] = subspace[i]*scale;
+    }
+  }  
   void Orthogonalise(void){
     CoarseScalar InnerProd(CoarseGrid); 
     //    std::cout << GridLogMessage <<" Block Gramm-Schmidt pass 1"<<std::endl;
@@ -110,14 +124,16 @@ public:
       
       hermop.Op(noise,Mn); std::cout<<GridLogMessage << "noise   ["<<b<<"] <n|MdagM|n> "<<norm2(Mn)<<std::endl;
 
-      for(int i=0;i<4;i++){
+      for(int i=0;i<2;i++){
 
 	CG(hermop,noise,subspace[b]);
 
 	noise = subspace[b];
 	scale = std::pow(norm2(noise),-0.5); 
 	noise=noise*scale;
-
+	
+	hermop.Op(noise,Mn); std::cout<<GridLogMessage << "intermediate["<<i<<"] <i|MdagM|i> "<<norm2(Mn)<<std::endl;
+	
       }
 
       hermop.Op(noise,Mn); std::cout<<GridLogMessage << "filtered["<<b<<"] <f|MdagM|f> "<<norm2(Mn)<<std::endl;
@@ -134,7 +150,8 @@ public:
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,10,DiracOp,simple_fine,30,30);
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,10,DiracOp,simple_fine,12,12);
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,30,DiracOp,simple_fine,12,12);
-    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,30,DiracOp,simple_fine,10,10);
+    //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.0005,30,DiracOp,simple_fine,20,20);
+    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.0005,30,DiracOp,simple_fine,10,10);
     FineField noise(FineGrid);
     FineField src(FineGrid);
     FineField guess(FineGrid);
@@ -167,6 +184,8 @@ public:
 	noise = subspace[b];
 	scale = std::pow(norm2(noise),-0.5); 
 	noise=noise*scale;
+	
+	DiracOp.Op(noise,Mn); std::cout<<GridLogMessage << "intermediate["<<i<<"] <f|Op|f> "<<innerProduct(noise,Mn)<<" <f|OpDagOp|f>"<<norm2(Mn)<<std::endl;
 
       }
 
@@ -174,6 +193,7 @@ public:
       subspace[b]   = noise;
 
     }
+    GlobalOrthonormalise();
   }
 
   ////////////////////////////////////////////////////////////////////////////////////////////////

Grid/algorithms/multigrid/DenseCoarseMatrix.h

@@ -0,0 +1,706 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./lib/algorithms/multigrid/DenseCoarseMatrix.h
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#pragma once
+
+#include <Grid/algorithms/blas/BatchedBlas.h>
+#include <Grid/algorithms/blas/BatchedInverse.h>
+
+#include <unordered_map>
+
+NAMESPACE_BEGIN(Grid);
+
+//////////////////////////////////////////////////////////////////////////////////////
+// DenseCoarseMatrix: a coarsened operator treated as a DENSE matrix -- explicit,
+// row-distributed A^{-1} of a GeneralCoarsenedMatrix.  Library-grade successor of
+// the example-local DistributedDenseInverse (Example_pvdagm_mrhs_3level_dense.cc,
+// FROZEN as the regression baseline).
+//
+// What is new versus the example class:
+//  - Stencil -> dense DIRECT IMPORT.  The coarse operator IS the dense matrix
+//    unrolled: Dense[(s,a),(s+shift_p,b)] += A[p][s]_{a,b}.  Rows of my sites are
+//    assembled from purely LOCAL _A[p] data: no operator applies, no comms -- the
+//    O(N) probe assembly (93 s at N=69120) is retired.  ACCUMULATE (+=) because on
+//    short axes distinct shifts wrap to the same neighbour.  An IMPORT CERTIFICATE
+//    compares the dense apply against Op.M on a NON-CONSTANT vector (a constant one
+//    cannot see a shift-sign error); DENSE_IMPORT_SIGN=-1 flips the convention
+//    without recompiling.
+//  - Split-K apply through GridBLAS.gemmBatched with EXPLICIT leading dimensions
+//    (arXiv:2409.03904 fig 11): the tiny-output/huge-K GEMM Y = slab^T X becomes
+//    DENSE_SPLITK chunk-GEMMs by pointer offset into the resident slab (lda = N),
+//    partials reduced in one accelerator_for.  Platform-agnostic: deviceVector +
+//    GridBLAS run the SAME code on HIP/CUDA/SYCL and CPU(Eigen).
+//  - deviceVector everywhere in the apply path; the ONE surviving naked-HIP block
+//    is the boss inversion buffer (quarantined below, documented).
+//
+// Setup: SLAB_FILE=<stem> loads per-rank <stem>.<rank> (header-guarded N/nrows/
+// nbasis -- the interchange format shared with the frozen example; the STEM must
+// encode cfg/mass/blocking/nbasis, only the header is guarded).  Absent: direct
+// import -> import certificate -> chunked zero-fill+GlobalSum gather streamed to
+// the boss GCD -> cgetrf_64 (ILP64) -> rows of A^{-1} via blocked identity
+// cgetrs_64 + broadcast, each rank keeping the rows of its own sites -> save.
+// VERIFY ||A Ainv x - x||/||x|| runs in BOTH paths (and now certifies the DEVICE
+// slab + split-K path, since the single-RHS apply routes through the same core).
+//
+// Env: SLAB_FILE  DENSE_SPLITK (default 32, snapped to a divisor of N)
+//      DENSE_DEVICE_SUM  DENSE_IMPORT_SIGN  DENSE_APPLY_PROFILE  DENSE_CC_CHECK
+//
+// Eventual internal upgrade (unchanged surface): RecursiveSchur distributed
+// factorisation replacing the single-GCD gather/invert, lifting BOTH the fp32
+// N ~ 90k boss-HBM ceiling AND the CC-grid 256-rank SIMD cap; leaves land on
+// GridBLASInverse::inverseBatched.
+//
+// Tensor-depth agnostic: site scalar objects treated as contiguous ComplexD
+// (iScalar wrappers add no data), so any MG level's coarse operator imports.
+//////////////////////////////////////////////////////////////////////////////////////
+template<class Fobj,class CComplex,int nbasis>
+class DenseCoarseMatrix : public LinearFunction<typename GeneralCoarsenedMatrix<Fobj,CComplex,nbasis>::CoarseVector> {
+public:
+  typedef GeneralCoarsenedMatrix<Fobj,CComplex,nbasis> GeneralCoarseOp;
+  typedef typename GeneralCoarseOp::CoarseVector       Field;
+  typedef typename GeneralCoarseOp::CoarseMatrix       CoarseMatrix;
+  using LinearFunction<Field>::operator();
+  typedef typename Field::vector_object vobj;
+  typedef typename vobj::scalar_object  sobj;
+  typedef typename CoarseMatrix::vector_object Mvobj;
+  typedef typename Mvobj::scalar_object        Msobj;
+
+  GeneralCoarseOp &_Op;             // the coarse operator: stencil source + certificate oracle
+  GridBase *grid;
+  int      nd;
+  int64_t  N;                       // dense rank = gSites * nbasis
+  int      lsites;                  // my local sites
+  int64_t  nrows;                   // my rows = lsites * nbasis
+  std::vector<Coordinate> myLcoor;  // local coordinate of my site ss
+  std::vector<int64_t>    myGsite;  // global lex site index of my site ss
+  std::vector<ComplexF>   slab;     // nrows x N row-major: A during setup, rows of A^{-1} after
+
+  static const int64_t CHUNKROWS = 1024;  // getrs harvest block (trsm efficiency + fewer broadcasts)
+  static const int MRHS_MAX = 32;
+
+  // Apply machinery: resident slab + persistent buffers + AOT split-K pointers.
+  GridBLAS BLAS;
+  deviceVector<ComplexF>  dSlab;
+  deviceVector<ComplexF>  dX;       // N x MRHS_MAX
+  deviceVector<ComplexF>  dY;       // nrows x MRHS_MAX
+  deviceVector<ComplexF>  dPartial; // NK x (nrows x MRHS_MAX)
+  deviceVector<ComplexF*> aptrs;    // slab K-chunk pointers   (lda = N)
+  deviceVector<ComplexF*> xptrs;    // X    K-chunk pointers   (ldb = N)
+  deviceVector<ComplexF*> cptrs;    // partial buffers         (ldc = nrows)
+  std::vector<ComplexF>   hX;
+  std::vector<ComplexF>   hY;
+  int NK;                           // split-K chunk count (divides N)
+  int devSum;
+
+  DenseCoarseMatrix(GeneralCoarseOp &Op, GridBase *g)
+    : _Op(Op), grid(g)
+  {
+    GRID_ASSERT( sizeof(sobj)  == nbasis*sizeof(ComplexD) );
+    GRID_ASSERT( sizeof(Msobj) == nbasis*nbasis*sizeof(ComplexD) );
+    GRID_ASSERT( grid == Op.Grid() );
+    nd     = grid->_ndimension;
+    N      = grid->gSites() * nbasis;
+    lsites = grid->lSites();
+    nrows  = (int64_t)lsites * nbasis;
+
+    std::cout << GridLogMessage << "DenseCoarseMatrix: N = " << N
+              << " (" << grid->gSites() << " sites x " << nbasis << ")"
+              << "  rows/rank = " << nrows
+              << "  slab = " << (double)nrows*N*sizeof(ComplexF)/1024./1024. << " MB/rank"
+              << std::endl;
+
+    ////////////////////////////////////////////////////////////////////
+    // Enumerate my sites: local coords and global lexicographic indices
+    ////////////////////////////////////////////////////////////////////
+    Coordinate ldims = grid->LocalDimensions();
+    Coordinate gdims = grid->GlobalDimensions();
+    myLcoor.resize(lsites);
+    myGsite.resize(lsites);
+    for(int ss=0; ss<lsites; ss++){
+      Coordinate lcoor(nd);
+      Lexicographic::CoorFromIndex(lcoor, ss, ldims);
+      Coordinate gcoor(nd);
+      for(int d=0; d<nd; d++) gcoor[d] = grid->_lstart[d] + lcoor[d];
+      int64_t gsite;
+      Lexicographic::IndexFromCoor(gcoor, gsite, gdims);
+      myLcoor[ss] = lcoor;
+      myGsite[ss] = gsite;
+    }
+
+    slab.resize((uint64_t)nrows * N);
+
+    double t0 = usecond();
+    ////////////////////////////////////////////////////////////////////
+    // 0. Slab cache: SLAB_FILE=<stem> -> per-rank raw file <stem>.<rank>.
+    //    SAME format as the frozen example (interchange compatible).
+    ////////////////////////////////////////////////////////////////////
+    bool loaded = false;
+    char *sfile = getenv("SLAB_FILE");
+    std::string slabfile;
+    if (sfile) {
+      slabfile = std::string(sfile) + "." + std::to_string(grid->ThisRank());
+      FILE *f = fopen(slabfile.c_str(),"rb");
+      if (f) {
+        int64_t hdr[4] = {0,0,0,0};
+        GRID_ASSERT( fread(hdr,sizeof(int64_t),4,f) == 4 );
+        GRID_ASSERT( hdr[0] == (int64_t)0x44454E5345 );  // magic "DENSE"
+        GRID_ASSERT( hdr[1] == N && hdr[2] == (int64_t)nrows && hdr[3] == (int64_t)nbasis );
+        uint64_t nelem = (uint64_t)nrows * N;
+        GRID_ASSERT( fread(&slab[0], sizeof(ComplexF), nelem, f) == nelem );
+        fclose(f);
+        loaded = true;
+        std::cout << GridLogMessage << "DenseCoarseMatrix: slab loaded from "
+                  << slabfile << " -- skipping import/factor/solve" << std::endl;
+      } else {
+        std::cout << GridLogMessage << "DenseCoarseMatrix: slab cache " << slabfile
+                  << " absent -- full setup, will write it" << std::endl;
+      }
+    }
+    if (!loaded) {
+      ImportDense();          // slab <- my rows of A   (LOCAL, no comms)
+      ImportCertificate();    // dense apply == Op.M on a non-constant vector
+      InvertDense();          // slab <- my rows of A^{-1}
+      double t1 = usecond();
+      if (sfile) {
+        FILE *f = fopen(slabfile.c_str(),"wb");
+        GRID_ASSERT(f != nullptr);
+        int64_t hdr[4] = { (int64_t)0x44454E5345, N, (int64_t)nrows, (int64_t)nbasis };
+        GRID_ASSERT( fwrite(hdr,sizeof(int64_t),4,f) == 4 );
+        uint64_t nelem = (uint64_t)nrows * N;
+        GRID_ASSERT( fwrite(&slab[0], sizeof(ComplexF), nelem, f) == nelem );
+        fclose(f);
+        std::cout << GridLogMessage << "DenseCoarseMatrix: slab written to " << slabfile << std::endl;
+      }
+      std::cout << GridLogMessage << "DenseCoarseMatrix: import+invert took "
+                << (t1-t0)/1.0e6 << " s" << std::endl;
+    }
+
+    ////////////////////////////////////////////////////////////////////
+    // Device residency + persistent apply buffers + AOT split-K pointers
+    ////////////////////////////////////////////////////////////////////
+    {
+      uint64_t sbytes = (uint64_t)nrows * N * sizeof(ComplexF);
+      dSlab.resize((uint64_t)nrows*N);
+      acceleratorCopyToDevice(&slab[0],&dSlab[0],sbytes);
+
+      // DENSE_SPLITK: requested chunk count, snapped DOWN to a divisor of N.
+      int req = getenv("DENSE_SPLITK") ? atoi(getenv("DENSE_SPLITK")) : 32;
+      if (req < 1) req = 1;
+      NK = 1;
+      for(int j=1;j<=req;j++) if ( (N % j) == 0 ) NK = j;
+      int64_t Kc = N / NK;
+
+      dX.resize((uint64_t)N*MRHS_MAX);
+      dY.resize((uint64_t)nrows*MRHS_MAX);
+      dPartial.resize((uint64_t)NK*nrows*MRHS_MAX);
+      hX.resize((uint64_t)N*MRHS_MAX);
+      hY.resize((uint64_t)nrows*MRHS_MAX);
+
+      aptrs.resize(NK); xptrs.resize(NK); cptrs.resize(NK);
+      std::vector<ComplexF*> h(NK);
+      for(int j=0;j<NK;j++) h[j] = &dSlab[0]    + (uint64_t)j*Kc;              // K-offset, lda=N
+      acceleratorCopyToDevice(&h[0],&aptrs[0],NK*sizeof(ComplexF*));
+      for(int j=0;j<NK;j++) h[j] = &dX[0]       + (uint64_t)j*Kc;              // K-offset, ldb=N
+      acceleratorCopyToDevice(&h[0],&xptrs[0],NK*sizeof(ComplexF*));
+      for(int j=0;j<NK;j++) h[j] = &dPartial[0] + (uint64_t)j*nrows*MRHS_MAX;  // compact, ldc=nrows
+      acceleratorCopyToDevice(&h[0],&cptrs[0],NK*sizeof(ComplexF*));
+
+      devSum = getenv("DENSE_DEVICE_SUM") ? 1 : 0;
+      std::cout << GridLogMessage << "DenseCoarseMatrix: slab resident on device ("
+                << sbytes/1024./1024. << " MB/rank), split-K NK=" << NK << " (Kc=" << Kc << "); "
+                << (devSum ? "DEVICE-buffer allreduce (GPU-aware MPI)" : "host allreduce")
+                << std::endl;
+    }
+
+    ////////////////////////////////////////////////////////////////////
+    // VERIFY: || A (Ainv x) - x || / ||x|| through the DEVICE split-K core.
+    ////////////////////////////////////////////////////////////////////
+    {
+      Field x(grid); Field y(grid); Field z(grid);
+      x = ComplexD(1.0,0.0);
+      double ta = usecond();
+      (*this)(x, y);
+      double tb = usecond();
+      _Op.M(y, z);
+      z = z - x;
+      RealD rel = std::sqrt(norm2(z)/norm2(x));
+      std::cout << GridLogMessage << "DenseCoarseMatrix: VERIFY ||A Ainv x - x||/||x|| = "
+                << rel << "   (one apply took " << (tb-ta)/1000.0 << " ms)" << std::endl;
+      GRID_ASSERT(rel < 1.0e-2);
+    }
+    std::cout << GridLogMessage << "DenseCoarseMatrix: setup complete, total "
+              << (usecond()-t0)/1.0e6 << " s" << std::endl;
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // 1. Direct stencil -> dense import of MY ROWS of A (no comms):
+  //      Dense[(s,a),(wrap(s+shift_p),b)] += A[p][s]_{a,b}
+  ////////////////////////////////////////////////////////////////////
+  void ImportDense(void)
+  {
+    double t = -usecond();
+    Coordinate gdims = grid->GlobalDimensions();
+    int sign = getenv("DENSE_IMPORT_SIGN") ? atoi(getenv("DENSE_IMPORT_SIGN")) : 1;
+    GRID_ASSERT( sign==1 || sign==-1 );
+
+    uint64_t nelem = (uint64_t)nrows * N;
+    thread_for(i, nelem, { slab[i] = ComplexF(0.0,0.0); });
+
+    for(int p=0; p<_Op.geom.npoint; p++){
+      Coordinate shift = _Op.geom.shifts[p];
+      // _A[p] is PADDED after ExchangeCoarseLinks (end of CoarsenOperator):
+      // extract the unpadded field before peeking with unpadded coordinates
+      // (exactly as MultiGeneralCoarsenedMatrix::CopyMatrix does).
+      CoarseMatrix Aun = _Op.Cell.Extract(_Op._A[p]);
+      autoView(Av, Aun, CpuRead);
+      thread_for(ss, lsites, {
+        Coordinate ncoor(nd);
+        for(int d=0; d<nd; d++){
+          int64_t g = grid->_lstart[d] + myLcoor[ss][d] + sign*shift[d];
+          ncoor[d] = (int)((g % gdims[d] + gdims[d]) % gdims[d]);
+        }
+        int64_t nsite;
+        Lexicographic::IndexFromCoor(ncoor, nsite, gdims);
+        Msobj m;
+        peekLocalSite(m, Av, myLcoor[ss]);
+        ComplexD *md = (ComplexD *)&m;
+        for(int a=0; a<nbasis; a++){
+          ComplexF *row = &slab[(uint64_t)(ss*nbasis+a)*N + nsite*nbasis];
+          for(int b=0; b<nbasis; b++)
+            row[b] += ComplexF(md[a*nbasis+b]);   // += : wrapped shifts may collide
+        }
+      });
+    }
+    t += usecond();
+    std::cout << GridLogMessage << "DenseCoarseMatrix: stencil->dense import took "
+              << t/1.0e6 << " s  (" << _Op.geom.npoint << " points, local, no comms)" << std::endl;
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // 2. IMPORT CERTIFICATE: dense rows vs Op.M on a NON-CONSTANT vector.
+  //    (Constant x has x[s+d]==x[s-d]: blind to a shift-sign error.)
+  ////////////////////////////////////////////////////////////////////
+  void ImportCertificate(void)
+  {
+    Field x(grid); Field Ax(grid); Field Dx(grid);
+    for(int ss=0; ss<lsites; ss++){
+      sobj s;
+      for(int b=0; b<nbasis; b++){
+        double ph = 0.37*(double)(myGsite[ss]*nbasis+b);
+        ((ComplexD *)&s)[b] = ComplexD(std::cos(ph),std::sin(0.61*ph));
+      }
+      pokeLocalSite(s, x, myLcoor[ss]);
+    }
+    // gather full x (zero-fill + exact GlobalSum), dense rows on host
+    std::vector<ComplexD> xh((uint64_t)N, ComplexD(0.0,0.0));
+    for(int ss=0; ss<lsites; ss++){
+      sobj s;
+      peekLocalSite(s, x, myLcoor[ss]);
+      for(int b=0; b<nbasis; b++) xh[ myGsite[ss]*nbasis + b ] = ((ComplexD *)&s)[b];
+    }
+    grid->GlobalSumVector(&xh[0], (int)N);
+    std::vector<ComplexD> yh(nrows);
+    thread_for(r, nrows, {
+      ComplexD acc(0.0,0.0);
+      const ComplexF *row = &slab[(uint64_t)r * N];
+      for(int64_t j=0; j<N; j++) acc += ComplexD(row[j]) * xh[j];
+      yh[r] = acc;
+    });
+    for(int ss=0; ss<lsites; ss++){
+      sobj s;
+      for(int b=0; b<nbasis; b++) ((ComplexD *)&s)[b] = yh[ss*nbasis+b];
+      pokeLocalSite(s, Dx, myLcoor[ss]);
+    }
+    _Op.M(x, Ax);
+    Field d(grid); d = Dx - Ax;
+    RealD rel = std::sqrt(norm2(d)/norm2(Ax));
+    std::cout << GridLogMessage << "DenseCoarseMatrix: IMPORT CERTIFICATE ||Dense x - A x||/||A x|| = "
+              << rel << std::endl;
+    if ( rel >= 1.0e-3 ) {
+      std::cout << GridLogMessage << "DenseCoarseMatrix: IMPORT CERTIFICATE FAILED. If O(1), the "
+                << "stencil shift-sign convention is opposite: rerun with DENSE_IMPORT_SIGN=-1"
+                << std::endl;
+    }
+    GRID_ASSERT(rel < 1.0e-3);
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // 3. Invert: chunked zero-fill+GlobalSum gather of A streamed to the
+  //    boss GCD, cgetrf_64 (ILP64), rows of A^{-1} by blocked identity
+  //    cgetrs_64 + broadcast; each rank keeps its own rows (in `slab`,
+  //    overwriting A).  Ported proven path from the frozen example.
+  ////////////////////////////////////////////////////////////////////
+  void InvertDense(void)
+  {
+    double t1 = usecond();
+    int boss = grid->IsBoss();
+    std::vector<ComplexF> Afull;
+#ifdef GRID_HIP
+    // QUARANTINED naked HIP: the boss-only N^2 inversion buffer (34GB at
+    // N=65536) must come from raw HBM; EvictAll flushes the device-copy
+    // layer to make the window.  (FreePool of the allocator free-list
+    // awaits the type-dispatched fix.)  Confined to setup; the apply path
+    // is pure Grid primitives.
+    rocblas_float_complex *dA = nullptr;
+    rocblas_float_complex *dB = nullptr;
+    int64_t *dIpiv = nullptr;
+    uint64_t Abytes = (uint64_t)N * N * sizeof(ComplexF);
+    MemoryManager::EvictAll();
+    if (boss) {
+      auto aerr = hipMalloc((void **)&dA, Abytes);
+      if (aerr != hipSuccess) {
+        std::cout << GridLogMessage << "DenseCoarseMatrix: hipMalloc of "
+                  << Abytes/1024./1024./1024. << " GB FAILED -- reduce --device-mem" << std::endl;
+        GRID_ASSERT(aerr == hipSuccess);
+      }
+      std::cout << GridLogMessage << "DenseCoarseMatrix: device inversion buffer allocated ("
+                << Abytes/1024./1024./1024. << " GB)" << std::endl;
+    }
+#else
+    if (boss) Afull.resize((uint64_t)N * N);
+#endif
+    {
+      std::unordered_map<int64_t,int> rowmap;   // global row -> my slab row
+      for(int ss=0; ss<lsites; ss++)
+        for(int a=0; a<nbasis; a++)
+          rowmap[ myGsite[ss]*nbasis + a ] = ss*nbasis + a;
+
+      std::vector<ComplexF> chunk((uint64_t)CHUNKROWS * N);
+      for(int64_t row0=0; row0<N; row0+=CHUNKROWS){
+        int64_t nrow = std::min(CHUNKROWS, N-row0);
+        uint64_t nelem = (uint64_t)nrow * N;
+        for(uint64_t i=0;i<nelem;i++) chunk[i]=ComplexF(0.0,0.0);
+        for(int64_t r=row0; r<row0+nrow; r++){
+          auto it = rowmap.find(r);
+          if (it != rowmap.end()) {
+            uint64_t src = (uint64_t)(it->second) * N;
+            uint64_t dst = (uint64_t)(r-row0) * N;
+            for(int64_t j=0;j<N;j++) chunk[dst+j] = slab[src+j];
+          }
+        }
+        grid->GlobalSumVector(&chunk[0], (int)nelem);
+        if (boss) {
+#ifdef GRID_HIP
+          GRID_ASSERT( hipMemcpy((char *)dA + (uint64_t)row0*N*sizeof(ComplexF),
+                                 &chunk[0], nelem*sizeof(ComplexF),
+                                 hipMemcpyHostToDevice) == hipSuccess );
+#else
+          uint64_t dst = (uint64_t)row0 * N;
+          for(uint64_t i=0;i<nelem;i++) Afull[dst+i] = chunk[i];
+#endif
+        }
+      }
+    }
+    double t2 = usecond();
+    std::cout << GridLogMessage << "DenseCoarseMatrix: gather to boss took "
+              << (t2-t1)/1.0e6 << " s" << std::endl;
+
+    ////////////////////////////////////////////////////////////////////
+    // Factor in place on the boss (fp32, ILP64).  Row-major buffer handed
+    // to column-major LAPACK => LU of A^T.
+    ////////////////////////////////////////////////////////////////////
+    if (boss) {
+#ifdef GRID_HIP
+      std::cout << GridLogMessage << "DenseCoarseMatrix: rocSOLVER cgetrf_64 (ILP64 LU) N=" << N
+                << " in place on resident device buffer" << std::endl;
+      rocblas_handle handle = GridBLASInverse::Handle();
+      int64_t *dInfo;
+      GRID_ASSERT( hipMalloc((void **)&dIpiv, N*sizeof(int64_t)) == hipSuccess );
+      GRID_ASSERT( hipMalloc((void **)&dInfo, sizeof(int64_t))   == hipSuccess );
+      auto st1 = rocsolver_cgetrf_64(handle, (int64_t)N, (int64_t)N, dA, (int64_t)N, dIpiv, dInfo);
+      hipDeviceSynchronize();
+      int64_t info_h = -1;
+      hipMemcpy(&info_h, dInfo, sizeof(int64_t), hipMemcpyDeviceToHost);
+      std::cout << GridLogMessage << "DenseCoarseMatrix: cgetrf_64 status " << (int)st1
+                << " info = " << (int)info_h << std::endl;
+      GRID_ASSERT(st1 == rocblas_status_success);
+      GRID_ASSERT(info_h == 0);
+      hipFree(dInfo);
+      GRID_ASSERT( hipMalloc((void **)&dB, (uint64_t)CHUNKROWS*N*sizeof(ComplexF)) == hipSuccess );
+      // dA holds the LU of A^T; rows of A^{-1} are produced blockwise below via
+      // cgetrs_64 on identity-column blocks: A^T X = E => X columns = rows of
+      // A^{-1}, in exactly the linear layout the harvest expects.
+#else
+      // Eigen fallback: small local CPU tests only.
+      std::cout << GridLogMessage << "DenseCoarseMatrix: Eigen fallback inversion N=" << N
+                << (N > 10000 ? "  (WARNING: SLOW; use the HIP/rocSOLVER path)" : "")
+                << std::endl;
+      typedef Eigen::Matrix<std::complex<float>,Eigen::Dynamic,Eigen::Dynamic,Eigen::RowMajor> MatF;
+      Eigen::Map<MatF> A(reinterpret_cast<std::complex<float>*>(&Afull[0]), N, N);
+      MatF Ainv = A.inverse();
+      A = Ainv;
+#endif
+    }
+    double t3 = usecond();
+    std::cout << GridLogMessage << "DenseCoarseMatrix: factorisation took "
+              << (t3-t2)/1.0e6 << " s" << std::endl;
+
+    ////////////////////////////////////////////////////////////////////
+    // Blocked solve + broadcast: rows of A^{-1} chunk by chunk; each
+    // rank keeps the rows of its own sites (ownership-aligned).
+    ////////////////////////////////////////////////////////////////////
+    {
+      std::unordered_map<int64_t,int> rowmap;
+      for(int ss=0; ss<lsites; ss++)
+        for(int a=0; a<nbasis; a++)
+          rowmap[ myGsite[ss]*nbasis + a ] = ss*nbasis + a;
+
+      std::vector<ComplexF> chunk((uint64_t)CHUNKROWS * N);
+      for(int64_t row0=0; row0<N; row0+=CHUNKROWS){
+        int64_t nrow = std::min(CHUNKROWS, N-row0);
+        uint64_t nelem = (uint64_t)nrow * N;
+        if (boss) {
+#ifdef GRID_HIP
+          // Identity block E: column j = e_{row0+j}; solve A^T X = E so X's
+          // columns are rows [row0,row0+nrow) of A^{-1}.
+          for(uint64_t i=0;i<nelem;i++) chunk[i] = ComplexF(0.0,0.0);
+          for(int64_t j=0;j<nrow;j++) chunk[(uint64_t)j*N + (uint64_t)(row0+j)] = ComplexF(1.0,0.0);
+          GRID_ASSERT( hipMemcpy(dB, &chunk[0], nelem*sizeof(ComplexF), hipMemcpyHostToDevice) == hipSuccess );
+          auto strs = rocsolver_cgetrs_64(GridBLASInverse::Handle(), rocblas_operation_none,
+                                          (int64_t)N, (int64_t)nrow,
+                                          dA, (int64_t)N, dIpiv, dB, (int64_t)N);
+          GRID_ASSERT(strs == rocblas_status_success);
+          hipDeviceSynchronize();
+          GRID_ASSERT( hipMemcpy(&chunk[0], dB, nelem*sizeof(ComplexF), hipMemcpyDeviceToHost) == hipSuccess );
+#else
+          uint64_t src = (uint64_t)row0 * N;
+          for(uint64_t i=0;i<nelem;i++) chunk[i] = Afull[src+i];
+#endif
+        }
+        grid->Broadcast(0, &chunk[0], nelem*sizeof(ComplexF));
+        for(int64_t r=row0; r<row0+nrow; r++){
+          auto it = rowmap.find(r);
+          if (it != rowmap.end()) {
+            uint64_t dst = (uint64_t)(it->second) * N;
+            uint64_t src = (uint64_t)(r-row0) * N;
+            for(int64_t j=0;j<N;j++) slab[dst+j] = chunk[src+j];
+          }
+        }
+      }
+    }
+#ifdef GRID_HIP
+    if (boss) {
+      if (dA)    hipFree(dA);
+      if (dB)    hipFree(dB);
+      if (dIpiv) hipFree(dIpiv);
+    }
+#endif
+    double t4 = usecond();
+    std::cout << GridLogMessage << "DenseCoarseMatrix: blocked getrs solve+scatter took "
+              << (t4-t3)/1.0e6 << " s" << std::endl;
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // CORE apply on packed data: hX[N x nr] (zero-filled, my sites only)
+  // -> allreduce -> split-K GEMM against the resident slab -> reduce
+  // partials -> hY[nrows x nr] (column major).  Platform-agnostic:
+  // deviceVector + GridBLAS (Eigen fallback on CPU builds).
+  // fp32 allreduce is EXACT: zero-fill assembly gives every element
+  // exactly one contributing rank.
+  ////////////////////////////////////////////////////////////////////
+  void SlabApplyPacked(int nr, double *tprof)
+  {
+    GRID_ASSERT(nr <= MRHS_MAX);
+    uint64_t nX = (uint64_t)N * nr;
+    uint64_t nY = (uint64_t)nrows * nr;
+    int64_t  Kc = N / NK;
+    double t1 = usecond();
+    double t2, t3;
+    if (devSum) {
+      acceleratorCopyToDevice(&hX[0],&dX[0],nX*sizeof(ComplexF));
+      t2 = usecond();
+      grid->GlobalSumVector((ComplexF *)&dX[0], (int)nX);
+      t3 = usecond();
+    } else {
+      grid->GlobalSumVector(&hX[0], (int)nX);
+      t2 = usecond();
+      acceleratorCopyToDevice(&hX[0],&dX[0],nX*sizeof(ComplexF));
+      t3 = usecond();
+    }
+    // Y = op(slab,T) . X : row-major slab (nrows x N) == col-major A^T
+    // (N x nrows, lda=N) => transpose gives the nrows x N operator.
+    // Split-K: NK chunk-GEMMs by pointer offset (AOT lists), then reduce.
+    ComplexF one (1.0,0.0);
+    ComplexF zero(0.0,0.0);
+    BLAS.gemmBatched(GridBLAS_OP_T, GridBLAS_OP_N,
+                     (int)nrows, nr, (int)Kc,
+                     one,  aptrs, (int)N,
+                           xptrs, (int)N,
+                     zero, cptrs, (int)nrows);
+    BLAS.synchronise();
+    {
+      ComplexF *pp = &dPartial[0];
+      ComplexF *py = &dY[0];
+      uint64_t stride = (uint64_t)nrows*MRHS_MAX;
+      int nk = NK;
+      accelerator_for(i, nY, 1, {
+        ComplexF acc(0.0,0.0);
+        for(int j=0;j<nk;j++) acc += pp[(uint64_t)j*stride + i];
+        py[i] = acc;
+      });
+    }
+    double t4 = usecond();
+    acceleratorCopyFromDevice(&dY[0],&hY[0],nY*sizeof(ComplexF));
+    double t5 = usecond();
+    if (tprof) {
+      tprof[0] = devSum ? (t3-t2) : (t2-t1);   // allreduce
+      tprof[1] = devSum ? (t2-t1) : (t3-t2);   // H2D
+      tprof[2] = t4-t3;                        // gemm+reduce
+      tprof[3] = t5-t4;                        // D2H
+    }
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // Single-RHS apply (also the VERIFY path => certifies device slab).
+  ////////////////////////////////////////////////////////////////////
+  virtual void operator()(const Field &src, Field &psi)
+  {
+    uint64_t nX = (uint64_t)N;
+    thread_for(i, nX, { hX[i]=ComplexF(0.0,0.0); });
+    for(int ss=0; ss<lsites; ss++){
+      sobj s;
+      peekLocalSite(s, src, myLcoor[ss]);
+      for(int b=0; b<nbasis; b++)
+        hX[ myGsite[ss]*nbasis + b ] = ComplexF(((ComplexD *)&s)[b]);
+    }
+    SlabApplyPacked(1, nullptr);
+    for(int ss=0; ss<lsites; ss++){
+      sobj s;
+      for(int b=0; b<nbasis; b++)
+        ((ComplexD *)&s)[b] = ComplexD(hY[ss*nbasis + b]);
+      pokeLocalSite(s, psi, myLcoor[ss]);
+    }
+    if ( getenv("DENSE_CC_CHECK") ) {
+      Field tmp(grid);
+      _Op.M(psi, tmp);
+      tmp = tmp - src;
+      std::cout << GridLogMessage << "DenseCoarseMatrix: apply defect ||A x - b||/||b|| = "
+                << std::sqrt(norm2(tmp)/norm2(src)) << std::endl;
+    }
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // Batched (vector-of-fields) apply.
+  ////////////////////////////////////////////////////////////////////
+  void ApplyBatch(std::vector<Field> &src, std::vector<Field> &psi)
+  {
+    int nr = src.size();
+    GRID_ASSERT(nr <= MRHS_MAX);
+    double t0 = usecond();
+    uint64_t nX = (uint64_t)N*nr;
+    thread_for(i, nX, { hX[i]=ComplexF(0.0,0.0); });
+    for(int rr=0; rr<nr; rr++){
+      for(int ss=0; ss<lsites; ss++){
+        sobj s;
+        peekLocalSite(s, src[rr], myLcoor[ss]);
+        for(int b=0; b<nbasis; b++)
+          hX[ (uint64_t)rr*N + myGsite[ss]*nbasis + b ] = ComplexF(((ComplexD *)&s)[b]);
+      }
+    }
+    SlabApplyPacked(nr, nullptr);
+    for(int rr=0; rr<nr; rr++){
+      for(int ss=0; ss<lsites; ss++){
+        sobj s;
+        for(int b=0; b<nbasis; b++)
+          ((ComplexD *)&s)[b] = ComplexD(hY[(uint64_t)rr*nrows + (ss*nbasis+b)]);
+        pokeLocalSite(s, psi[rr], myLcoor[ss]);
+      }
+    }
+    double t1 = usecond();
+    std::cout << GridLogMessage << "DenseCoarseMatrix: batched apply " << nr << " rhs took "
+              << (t1-t0)/1000.0 << " ms  (" << (t1-t0)/1000.0/nr << " ms/rhs)" << std::endl;
+    if ( getenv("DENSE_CC_CHECK") ) {
+      Field tmp(grid);
+      for(int rr=0; rr<nr; rr++){
+        _Op.M(psi[rr], tmp);
+        tmp = tmp - src[rr];
+        std::cout << GridLogMessage << "DenseCoarseMatrix: batch defect["<<rr<<"] = "
+                  << std::sqrt(norm2(tmp)/norm2(src[rr])) << std::endl;
+      }
+    }
+  }
+
+  ////////////////////////////////////////////////////////////////////
+  // 6D mrhs apply: operates DIRECTLY on the packed 6D field (rhs = dim 0).
+  ////////////////////////////////////////////////////////////////////
+  void ApplyBatch6D(const Field &in6, Field &out6, int nr)
+  {
+    GRID_ASSERT(nr <= MRHS_MAX);
+    GRID_ASSERT(in6.Grid()->_ndimension == nd+1);   // {rhs, s, x,y,z,t}
+    double t0 = usecond();
+    Field &in = const_cast<Field &>(in6);
+    uint64_t nX = (uint64_t)N * nr;
+    thread_for(i, nX, { hX[i]=ComplexF(0.0,0.0); });
+    {
+      autoView(iv, in, CpuRead);
+      Coordinate c6(nd+1);
+      for(int ss=0; ss<lsites; ss++){
+        for(int d=0; d<nd; d++) c6[d+1] = myLcoor[ss][d];
+        for(int rr=0; rr<nr; rr++){
+          c6[0] = rr;
+          sobj s;
+          peekLocalSite(s, iv, c6);
+          for(int b=0; b<nbasis; b++)
+            hX[(uint64_t)rr*N + myGsite[ss]*nbasis + b] = ComplexF(((ComplexD *)&s)[b]);
+        }
+      }
+    }
+    double t1 = usecond();
+    double tprof[4];
+    SlabApplyPacked(nr, tprof);
+    double t5 = usecond();
+    {
+      autoView(ov, out6, CpuWrite);
+      Coordinate c6(nd+1);
+      for(int ss=0; ss<lsites; ss++){
+        for(int d=0; d<nd; d++) c6[d+1] = myLcoor[ss][d];
+        for(int rr=0; rr<nr; rr++){
+          c6[0] = rr;
+          sobj s;
+          for(int b=0; b<nbasis; b++)
+            ((ComplexD *)&s)[b] = ComplexD(hY[(uint64_t)rr*nrows + (ss*nbasis+b)]);  // Y col-major
+          pokeLocalSite(s, ov, c6);
+        }
+      }
+    }
+    double t6 = usecond();
+    std::cout << GridLogMessage << "DenseCoarseMatrix: apply6D " << nr << " rhs took "
+              << (t6-t0)/1000.0 << " ms" << std::endl;
+    if ( getenv("DENSE_APPLY_PROFILE") ) {
+      std::cout << GridLogMessage << "DenseCoarseMatrix: apply6D profile:"
+                << " pack "        << (t1-t0)/1000.0
+                << "  allreduce "  << tprof[0]/1000.0
+                << "  H2D "        << tprof[1]/1000.0
+                << "  gemm+reduce "<< tprof[2]/1000.0
+                << "  D2H "        << tprof[3]/1000.0
+                << "  unpack "     << (t6-t5)/1000.0
+                << "  ms" << std::endl;
+    }
+  }
+};
+
+NAMESPACE_END(Grid);

Grid/algorithms/multigrid/GeneralCoarsenedMatrix.h

@@ -31,6 +31,7 @@ Author: Peter Boyle <pboyle@bnl.gov>
 
 #include <Grid/lattice/PaddedCell.h>
 #include <Grid/stencil/GeneralLocalStencil.h>
+#include <Grid/algorithms/deflation/MultiRHSBlockProject.h>
 
 NAMESPACE_BEGIN(Grid);
 
@@ -66,6 +67,10 @@ public:
   std::vector<CoarseMatrix> _Adag;
   std::vector<CoarseVector> MultTemporaries;
 
+  int64_t MultCalls;
+  double  MultFlopsAccum;
+  double  MultUsecAccum;
+
   ///////////////////////
   // Interface
   ///////////////////////
@@ -104,19 +109,20 @@ public:
   }
   */
   
-  GeneralCoarsenedMatrix(NonLocalStencilGeometry &_geom,GridBase *FineGrid, GridCartesian * CoarseGrid)
+  GeneralCoarsenedMatrix(NonLocalStencilGeometry &_geom,GridBase *FineGrid, GridCartesian * CoarseGrid,int _herm=1)
     : geom(_geom),
       _FineGrid(FineGrid),
       _CoarseGrid(CoarseGrid),
-      hermitian(1),
+      hermitian(_herm),
       Cell(_geom.Depth(),_CoarseGrid),
-      Stencil(Cell.grids.back(),geom.shifts)
+      Stencil(Cell.grids.back(),geom.shifts),
+      MultCalls(0), MultFlopsAccum(0.0), MultUsecAccum(0.0)
   {
     {
       int npoint = _geom.npoint;
     }
     _A.resize(geom.npoint,CoarseGrid);
-    //    _Adag.resize(geom.npoint,CoarseGrid);
+    if ( !hermitian ) _Adag.resize(geom.npoint,CoarseGrid);
   }
   void M (const CoarseVector &in, CoarseVector &out)
   {
@@ -124,10 +130,10 @@ public:
   }
   void Mdag (const CoarseVector &in, CoarseVector &out)
   {
-    GRID_ASSERT(hermitian);
+    if(hermitian)
-    Mult(_A,in,out);
+      Mult(_A,in,out);
-    //    if ( hermitian ) M(in,out);
+    else
-    //    else Mult(_Adag,in,out);
+      Mult(_Adag,in,out);
   }
   void Mult (std::vector<CoarseMatrix> &A,const CoarseVector &in, CoarseVector &out)
   {
@@ -227,29 +233,28 @@ public:
     text+=usecond();
     ttot+=usecond();
     
-    std::cout << GridLogPerformance<<"Coarse 1rhs Mult Aviews "<<tviews<<" us"<<std::endl;
+    MultCalls++;
-    std::cout << GridLogPerformance<<"Coarse Mult exch "<<texch<<" us"<<std::endl;
+    MultFlopsAccum += flops;
-    std::cout << GridLogPerformance<<"Coarse Mult mult "<<tmult<<" us"<<std::endl;
+    MultUsecAccum  += ttot;
-    std::cout << GridLogPerformance<<" of which mult2  "<<tmult2<<" us"<<std::endl;
+    std::cout << GridLogPerformance
-    std::cout << GridLogPerformance<<"Coarse Mult ext  "<<text<<" us"<<std::endl;
+	      << "Coarse Mult call " << MultCalls
-    std::cout << GridLogPerformance<<"Coarse Mult temps "<<ttemps<<" us"<<std::endl;
+	      << "  tot " << ttot << " us"
-    std::cout << GridLogPerformance<<"Coarse Mult copy  "<<tcopy<<" us"<<std::endl;
+	      << "  kernel " << tmult << " us"
-    std::cout << GridLogPerformance<<"Coarse Mult tot  "<<ttot<<" us"<<std::endl;
+	      << "  kernel " << flops/tmult*1e-3 << " GFlop/s"
-    //    std::cout << GridLogPerformance<<std::endl;
+	      << "  overall " << MultFlopsAccum/MultUsecAccum*1e-3 << " GFlop/s (cumul)"
-    std::cout << GridLogPerformance<<"Coarse Kernel flops "<< flops<<std::endl;
+	      << "  bw " << bytes/tmult*1e-3 << " GB/s"
-    std::cout << GridLogPerformance<<"Coarse Kernel flop/s "<< flops/tmult<<" mflop/s"<<std::endl;
+	      << std::endl;
-    std::cout << GridLogPerformance<<"Coarse Kernel bytes/s "<< bytes/tmult<<" MB/s"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse overall flops/s "<< flops/ttot<<" mflop/s"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse total bytes   "<< bytes/1e6<<" MB"<<std::endl;
 
   };
   
   void PopulateAdag(void)
   {
+#if 0
+    // Serial global peek/poke reference implementation
     for(int64_t bidx=0;bidx<CoarseGrid()->gSites() ;bidx++){
       Coordinate bcoor;
       CoarseGrid()->GlobalIndexToGlobalCoor(bidx,bcoor);
-      
+
       for(int p=0;p<geom.npoint;p++){
 	Coordinate scoor = bcoor;
 	for(int mu=0;mu<bcoor.size();mu++){
@@ -262,6 +267,36 @@ public:
 	pokeSite(adj(link),_Adag[pp],bcoor);
       }
     }
+#else
+    // Parallel: _Adag[pp](x) = adj( _A[p](x + s_pp) ),  pp = Reverse(p), s_pp = -s_p.
+    // The neighbour fetch reuses the same padded-cell + stencil machinery as Mult,
+    // reading one matrix element per coalesced access so no whole site matrix
+    // (230KB at nbasis=60) ever lands on a GPU thread stack (HIP limit 128KB).
+    // Halo sites compute garbage neighbours; Cell.Extract discards them.
+    // Must run on the unpadded _A, i.e. before ExchangeCoarseLinks.
+    const int Nsimd = CComplex::Nsimd();
+    for(int p=0;p<geom.npoint;p++){
+      int pp = geom.Reverse(p);
+      CoarseMatrix Apad = Cell.ExchangePeriodic(_A[p]);
+      CoarseMatrix Dpad(Apad.Grid());
+      int64_t osites = Apad.Grid()->oSites();
+      {
+	autoView( Apad_v   , Apad, AcceleratorRead);
+	autoView( Dpad_v   , Dpad, AcceleratorWriteDiscard);
+	autoView( Stencil_v, Stencil, AcceleratorRead);
+	accelerator_for(sj, osites*nbasis, Nsimd, {
+	    int32_t ss = sj/nbasis;
+	    int32_t j  = sj%nbasis;
+	    auto SE = Stencil_v.GetEntry(pp,ss);
+	    for(int i=0;i<nbasis;i++){
+	      auto z = coalescedReadGeneralPermute(Apad_v[SE->_offset](i,j),SE->_permute,Nd);
+	      coalescedWrite(Dpad_v[ss](j,i),conjugate(z));
+	    }
+	});
+      }
+      _Adag[pp] = Cell.Extract(Dpad);
+    }
+#endif
   }
   /////////////////////////////////////////////////////////////
   // 
@@ -417,10 +452,9 @@ public:
 	int osites=CoarseGrid()->oSites();
 	autoView( A_v  , _A[k], AcceleratorWrite);
 	autoView( FT_v  , FT[k], AcceleratorRead);
-	accelerator_for(sss, osites, 1, {
+	accelerator_for(sss, osites, nbasis, {
-	    for(int j=0;j<nbasis;j++){
+	    int j = acceleratorSIMTlane(nbasis);
-	      A_v[sss](i,j) = FT_v[sss](j);
+	    A_v[sss](i,j) = FT_v[sss](j);
-	    }
         });
       }
       tinv+=usecond();
@@ -428,8 +462,8 @@ public:
 
     // Only needed if nonhermitian
     if ( ! hermitian ) {
-      //      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
+      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
-      //      PopulateAdag();
+      PopulateAdag();
     }
 
     // Need to write something to populate Adag from A
@@ -517,13 +551,9 @@ public:
     // Now compute the matrix elements of linop between the orthonormal
     // set of vectors.
     ///////////////////////////////////////////////////////////////////////
-    FineField phaV(grid); // Phased block basis vector
-    FineField MphaV(grid);// Matrix applied
     std::vector<FineComplexField> phaF(npoint,grid);
     std::vector<CoarseComplexField> pha(npoint,CoarseGrid());
-    
+
-    CoarseVector coarseInner(CoarseGrid());
-    
     typedef typename CComplex::scalar_type SComplex;
     FineComplexField one(grid); one=SComplex(1.0);
     FineComplexField zz(grid); zz = Zero();
@@ -542,37 +572,52 @@ public:
       pha[p]  =exp(pha[p]*ci);
 
       blockZAXPY(phaF[p],pha[p],one,zz);
-      
+
     }
     tphase+=usecond();
-    
+
-    std::vector<CoarseVector> ComputeProj(npoint,CoarseGrid());
+    // Import basis into BLAS layout once; blockProject then reads it once per
-    std::vector<CoarseVector>          FT(npoint,CoarseGrid());
+    // basis vector rather than once per (i,p) as in scalar blockProject.
+    // Process all npoint in a single batch.
+    MultiRHSBlockProject<FineField> Projector;
+    Projector.Allocate(nbasis, grid, CoarseGrid());
+    Projector.ImportBasis(U.subspace);
+
+    std::vector<FineField>    phaV_batch(npoint, grid);
+    std::vector<FineField>    MphaV_batch(npoint, grid);
+    std::vector<CoarseVector> proj_batch(npoint, CoarseGrid());
+    std::vector<CoarseVector> ComputeProj(npoint, CoarseGrid());
+    std::vector<CoarseVector>          FT(npoint, CoarseGrid());
+
+    // Pre-allocate BLAS_F and BLAS_C to avoid repeated hipMalloc/hipFree of
+    // ~5.6 GB per blockProject call, which hangs on ROCm for large allocations.
+    Projector.BLAS_F.resize(Projector.fine_vol * Projector.words * npoint);
+    Projector.BLAS_C.resize(Projector.coarse_vol * nbasis * npoint);
+
     for(int i=0;i<nbasis;i++){// Loop over basis vectors
+      accelerator_barrier(); // ensure prior iteration's async writes are retired
       std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<"/"<<nbasis<< std::endl;
-      for(int p=0;p<npoint;p++){ // Loop over momenta in npoint
-	tphaseBZ-=usecond();
-	phaV = phaF[p]*V.subspace[i];
-	tphaseBZ+=usecond();
 
-	/////////////////////////////////////////////////////////////////////
+      tphaseBZ-=usecond();
-	// Multiple phased subspace vector by matrix and project to subspace
+      for(int p=0;p<npoint;p++)
-	// Remove local bulk phase to leave relative phases
+	phaV_batch[p] = phaF[p] * V.subspace[i];
-	/////////////////////////////////////////////////////////////////////
+      tphaseBZ+=usecond();
-	tmat-=usecond();
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" phaseBZ done"<< std::endl;
-	linop.Op(phaV,MphaV);
-	tmat+=usecond();
-	//	std::cout << i << " " <<p << " MphaV "<<norm2(MphaV)<<" "<<norm2(phaV)<<std::endl;
 
-	tproj-=usecond();
+      tmat-=usecond();
-	blockProject(coarseInner,MphaV,U.subspace);
+      for(int p=0;p<npoint;p++)
-	coarseInner = conjugate(pha[p]) * coarseInner;
+	linop.Op(phaV_batch[p], MphaV_batch[p]);
+      tmat+=usecond();
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" mat done"<< std::endl;
 
-	ComputeProj[p] = coarseInner;
+      // One batched GEMM reads BLAS_V once for all npoint vectors.
-	tproj+=usecond();
+      tproj-=usecond();
-	//	std::cout << i << " " <<p << " ComputeProj "<<norm2(ComputeProj[p])<<std::endl;
+      Projector.blockProject(MphaV_batch, proj_batch);
-
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" blockProject done"<< std::endl;
-      }
+      for(int p=0;p<npoint;p++)
+	ComputeProj[p] = conjugate(pha[p]) * proj_batch[p];
+      tproj+=usecond();
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" proj done"<< std::endl;
 
       tinv-=usecond();
       for(int k=0;k<npoint;k++){
@@ -580,14 +625,13 @@ public:
 	for(int l=0;l<npoint;l++){
 	  FT[k]= FT[k]+ invMkl(l,k)*ComputeProj[l];
 	}
-      
+
 	int osites=CoarseGrid()->oSites();
 	autoView( A_v  , _A[k], AcceleratorWrite);
 	autoView( FT_v  , FT[k], AcceleratorRead);
-	accelerator_for(sss, osites, 1, {
+	accelerator_for(sss, osites, nbasis, {
-	    for(int j=0;j<nbasis;j++){
+	    int j = acceleratorSIMTlane(nbasis);
-	      A_v[sss](i,j) = FT_v[sss](j);
+	    A_v[sss](i,j) = FT_v[sss](j);
-	    }
         });
       }
       tinv+=usecond();
@@ -595,13 +639,13 @@ public:
 
     // Only needed if nonhermitian
     if ( ! hermitian ) {
-      //      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
+      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
-      //      PopulateAdag();
+      PopulateAdag();
     }
 
-    for(int p=0;p<geom.npoint;p++){
+    //    for(int p=0;p<geom.npoint;p++){
-      std::cout << " _A["<<p<<"] "<<norm2(_A[p])<<std::endl;
+    //      std::cout << " _A["<<p<<"] "<<norm2(_A[p])<<std::endl;
-    }
+    //    }
 
     // Need to write something to populate Adag from A
     ExchangeCoarseLinks();
@@ -616,7 +660,7 @@ public:
   void ExchangeCoarseLinks(void){
     for(int p=0;p<geom.npoint;p++){
       _A[p] = Cell.ExchangePeriodic(_A[p]);
-      //      _Adag[p]= Cell.ExchangePeriodic(_Adag[p]);
+      if ( !hermitian ) _Adag[p]= Cell.ExchangePeriodic(_Adag[p]);
     }
   }
   virtual  void Mdiag    (const Field &in, Field &out){ GRID_ASSERT(0);};

Grid/allocator/MemoryManager.cc

@@ -63,12 +63,10 @@ void MemoryManager::PrintBytes(void)
   std::cout << " MemoryManager : "<<(total_device>>20)<<" accelerator Mbytes "<<std::endl;
   std::cout << " MemoryManager : "<<(total_host>>20)  <<" cpu         Mbytes "<<std::endl;
   uint64_t cacheBytes;
-  cacheBytes = CacheBytes[Cpu];
+  cacheBytes = HostCacheBytes();
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" cpu cache Mbytes "<<std::endl;
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" cpu alloc cache Mbytes "<<std::endl;
-  cacheBytes = CacheBytes[Acc];
+  cacheBytes = DeviceCacheBytes();
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" acc cache Mbytes "<<std::endl;
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" acc alloc cache Mbytes "<<std::endl;
-  cacheBytes = CacheBytes[Shared];
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" shared cache Mbytes "<<std::endl;
   
 #ifdef GRID_CUDA
   cuda_mem();

Grid/allocator/MemoryManager.h

@@ -113,7 +113,7 @@ private:
   static void *Insert(void *ptr,size_t bytes,AllocationCacheEntry *entries,int ncache,int &victim,uint64_t &cbytes) ;
   static void *Lookup(size_t bytes,AllocationCacheEntry *entries,int ncache,uint64_t &cbytes) ;
 
- public:
+public:  
   static void PrintBytes(void);
   static void Audit(std::string s);
   static void Init(void);
@@ -215,6 +215,7 @@ private:
   static void NotifyDeletion(void * CpuPtr);
   static void Print(void);
   static void PrintAll(void);
+  static void EvictAll(void);
   static void PrintState( void* CpuPtr);
   static int   isOpen   (void* CpuPtr);
   static void  ViewClose(void* CpuPtr,ViewMode mode);

Grid/allocator/MemoryManagerCache.cc

@@ -79,6 +79,25 @@ void MemoryManager::EntryErase(uint64_t CpuPtr)
   auto AccCache = EntryLookup(CpuPtr);
   AccViewTable.erase(CpuPtr);
 }
+/////////////////////////////////////////////////////////////////////////////////
+// LRU membership invariant:
+//
+//    LRU_valid == 1   <=>   AccPtr != NULL  &&  accLock == 0  &&  cpuLock == 0
+//
+// i.e. the LRU queue contains exactly the device-resident, completely unlocked
+// entries -- the evictable set.  Membership is maintained EAGERLY at the lock
+// 0<->1 edges, O(1) via the stored LRU_entry iterator:
+//
+//    AcceleratorViewOpen   lock 0->1 : LRUremove   (gated on LRU_valid)
+//    AcceleratorViewClose  accLock->0: LRUinsert   (AccPtr necessarily exists)
+//    CpuViewOpen           lock 0->1 : LRUremove   (gated on LRU_valid)
+//    CpuViewClose          cpuLock->0: LRUinsert   (iff AccPtr exists)
+//    Evict/AccDiscard                : LRUremove   (frees the device copy)
+//
+// Consequences: victims taken from LRU.back() are evictable by construction;
+// Evict() on a locked entry is an invariant violation (asserted), and the
+// eviction loops (EvictVictims/EvictAll) cannot spin.
+/////////////////////////////////////////////////////////////////////////////////
 void  MemoryManager::LRUinsert(AcceleratorViewEntry &AccCache)
 {
   GRID_ASSERT(AccCache.LRU_valid==0);
@@ -130,21 +149,21 @@ void MemoryManager::Evict(AcceleratorViewEntry &AccCache)
 {
   ///////////////////////////////////////////////////////////////////////////
   // Make CPU consistent, remove from Accelerator, remove from LRU, LEAVE CPU only entry
-  // Cannot be acclocked. If allocated must be in LRU pool.
+  // Cannot be locked. If allocated must be in LRU pool.
   //
-  // Nov 2022... Felix issue: Allocating two CpuPtrs, can have an entry in LRU-q with CPUlock.
+  // (Historical: a Nov 2022 incident (two CpuPtrs; eviction called from
-  //                          and require to evict the AccPtr copy. Eviction was a mistake in CpuViewOpen
+  //  CpuViewOpen -- since excised) could present a cpuLocked entry here, and
-  //                          but there is a weakness where CpuLock entries are attempted for erase
+  //  silent-return guards were added.  The LRU membership invariant (see
-  //                          Take these OUT LRU queue when CPU locked?
+  //  LRUinsert) now excludes ALL locked entries from the queue eagerly at the
-  //                          Cannot take out the table as cpuLock data is important.
+  //  lock edges, so a locked victim is an invariant violation: asserted.)
   ///////////////////////////////////////////////////////////////////////////
   GRID_ASSERT(AccCache.state!=Empty);
   
   mprintf("MemoryManager: Evict CpuPtr %lx AccPtr %lx cpuLock %ld accLock %ld",
 	  (uint64_t)AccCache.CpuPtr,(uint64_t)AccCache.AccPtr,
 	  (uint64_t)AccCache.cpuLock,(uint64_t)AccCache.accLock); 
-  if (AccCache.accLock!=0) return;
+  GRID_ASSERT(AccCache.accLock==0);
-  if (AccCache.cpuLock!=0) return;
+  GRID_ASSERT(AccCache.cpuLock==0);
   if(AccCache.state==AccDirty) {
     Flush(AccCache);
   }
@@ -250,6 +269,19 @@ void  MemoryManager::EvictVictims(uint64_t bytes)
     }
   }
 }
+void  MemoryManager::EvictAll(void)
+{
+  while(LRU.size()>0){
+    if ( DeviceLRUBytes > 0){
+      uint64_t victim = LRU.back(); // From the LRU
+      auto AccCacheIterator = EntryLookup(victim);
+      auto & AccCache = AccCacheIterator->second;
+      Evict(AccCache);
+    } else {
+      return;
+    }
+  }
+}
 uint64_t MemoryManager::AcceleratorViewOpen(uint64_t CpuPtr,size_t bytes,ViewMode mode,ViewAdvise hint)
 {
   ////////////////////////////////////////////////////////////////////////////
@@ -383,6 +415,13 @@ void MemoryManager::CpuViewClose(uint64_t CpuPtr)
   GRID_ASSERT(AccCache.accLock==0);
 
   AccCache.cpuLock--;
+  // Return to LRU queue when fully unlocked -- mirrors AcceleratorViewClose.
+  // Asymmetry vs the Acc side: a device copy need not exist for a host view;
+  // only device-resident entries belong in the (evictable) LRU queue.
+  if( (AccCache.cpuLock==0) && (AccCache.AccPtr!=(uint64_t)NULL) ) {
+    dprintf("CpuViewClose %lx cpuLock decremented to zero, move to LRU queue",(uint64_t)CpuPtr);
+    LRUinsert(AccCache);
+  }
 }
 /*
  *  Action  State   StateNext         Flush    Clone
@@ -449,6 +488,14 @@ uint64_t MemoryManager::CpuViewOpen(uint64_t CpuPtr,size_t bytes,ViewMode mode,V
     GRID_ASSERT(0); // should be unreachable
   }
 
+  GRID_ASSERT(AccCache.cpuLock>0);
+  // If view is opened on host must remove from LRU -- mirrors AcceleratorViewOpen.
+  // LRU_valid==1 here implies this is the 0->1 lock edge of a device-resident entry.
+  if(AccCache.LRU_valid==1){
+    dprintf("CpuViewOpen: entry removed from LRU ");
+    LRUremove(AccCache);
+  }
+
   AccCache.transient= transient? EvictNext : 0;
 
   return AccCache.CpuPtr;

Grid/communicator/Communicator_base.h

@@ -238,6 +238,16 @@ public:
   }
   void AllToAll(int dim  ,void *in,void *out,uint64_t words,uint64_t bytes);
   void AllToAll(void  *in,void *out,uint64_t words         ,uint64_t bytes);
+
+  ////////////////////////////////////////////////////////////////////////////
+  // Variable count all to all. Counts and displacements are in units of
+  // "bytes" sized words and are indexed by rank within this communicator.
+  // For exchanges that are a permutation but do not divide evenly between
+  // ranks; AllToAll above is the uniform count special case.
+  ////////////////////////////////////////////////////////////////////////////
+  void AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
+                 void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
+                 uint64_t bytes);
   
   template<class obj> void Broadcast(int root,obj &data)
   {

Grid/communicator/Communicator_mpi3.cc

@@ -945,5 +945,25 @@ void CartesianCommunicator::AllToAll(void  *in,void *out,uint64_t words,uint64_t
   MPI_Alltoall(in,iwords,object,out,iwords,object,communicator);
   MPI_Type_free(&object);
 }
+void CartesianCommunicator::AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
+                                      void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
+                                      uint64_t bytes)
+{
+  FlightRecorder::StepLog("AllToAllV");
+  GRID_ASSERT(sendcounts.size()==(size_t)_Nprocessors);
+  GRID_ASSERT(senddispls.size()==(size_t)_Nprocessors);
+  GRID_ASSERT(recvcounts.size()==(size_t)_Nprocessors);
+  GRID_ASSERT(recvdispls.size()==(size_t)_Nprocessors);
+  // MPI counts are "int"; the caller sizes the word to keep them in range
+  int ibytes = bytes;
+  GRID_ASSERT(bytes == (uint64_t)ibytes);
+  MPI_Datatype object;
+  MPI_Type_contiguous(ibytes,MPI_BYTE,&object);
+  MPI_Type_commit(&object);
+  int ierr = MPI_Alltoallv(in ,(int *)&sendcounts[0],(int *)&senddispls[0],object,
+                           out,(int *)&recvcounts[0],(int *)&recvdispls[0],object,communicator);
+  GRID_ASSERT(ierr==0);
+  MPI_Type_free(&object);
+}
 
 NAMESPACE_END(Grid);

Grid/communicator/Communicator_none.cc

@@ -113,6 +113,17 @@ void CartesianCommunicator::AllToAll(void  *in,void *out,uint64_t words,uint64_t
 {
   bcopy(in,out,bytes*words);
 }
+void CartesianCommunicator::AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
+                                      void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
+                                      uint64_t bytes)
+{
+  // Single rank: the exchange degenerates to a copy of our own segment
+  GRID_ASSERT(sendcounts.size()==1);
+  GRID_ASSERT(recvcounts.size()==1);
+  GRID_ASSERT(sendcounts[0]==recvcounts[0]);
+  bcopy((char *)in +(uint64_t)senddispls[0]*bytes,
+        (char *)out+(uint64_t)recvdispls[0]*bytes,bytes*(uint64_t)sendcounts[0]);
+}
 
 int  CartesianCommunicator::RankWorld(void){return 0;}
 void CartesianCommunicator::Barrier(void){}

Grid/lattice/Lattice_base.h

@@ -289,7 +289,7 @@ public:
   ///////////////////////////////////////////
   // move constructor
   ///////////////////////////////////////////
-  Lattice(Lattice && r){ 
+  Lattice(Lattice && r) noexcept { 
     this->_grid = r.Grid();
     this->_odata      = r._odata;
     this->_odata_size = r._odata_size;
@@ -330,7 +330,7 @@ public:
   ///////////////////////////////////////////
   // Move assignment possible if same type
   ///////////////////////////////////////////
-  inline Lattice<vobj> & operator = (Lattice<vobj> && r){
+  inline Lattice<vobj> & operator = (Lattice<vobj> && r) noexcept {
 
     resize(0); // deletes if appropriate
     this->_grid       = r.Grid();

Grid/lattice/Lattice_reduction_gpu.h

@@ -198,7 +198,7 @@ __global__ void reduceKernel(const vobj *lat, sobj *buffer, Iterator n) {
 // Possibly promote to double and sum
 /////////////////////////////////////////////////////////////////////////////////////////////////////////
 
-#define GRID_REDUCTION_TIMING
+#undef GRID_REDUCTION_TIMING
 
 template <class vobj>
 inline typename vobj::scalar_objectD sumD_gpu_small(const vobj *lat, Integer osites)
@@ -230,7 +230,7 @@ inline typename vobj::scalar_objectD sumD_gpu_small(const vobj *lat, Integer osi
   acceleratorCopyFromDevice(buffer_v,&result,sizeof(result));
 #ifdef GRID_REDUCTION_TIMING
   t_d2h += usecond();
-  std::cout << GridLogMessage << "  sumD_gpu_small"
+  std::cout << GridLogDebug << "  sumD_gpu_small"
             << " sizeof(sobj)=" << sizeof(sobj)
             << " blocks=" << numBlocks << " threads=" << numThreads
             << " kernel+barrier=" << t_kernel << " us"
@@ -362,7 +362,7 @@ inline void sumD_gpu_reduce_words(const vobj *lat, Integer osites,
   acceleratorCopyFromDevice(buffer_v, &result, sizeof(result));
 #ifdef GRID_REDUCTION_TIMING
   t_d2h += usecond();
-  std::cout << GridLogMessage << " sumD_gpu_reduce_words R=" << R
+  std::cout << GridLogDebug << " sumD_gpu_reduce_words R=" << R
             << " base=" << base
             << " kernel=" << t_kernel << " D2H=" << t_d2h << " us" << std::endl;
 #endif
@@ -391,7 +391,7 @@ inline typename vobj::scalar_objectD sumD_gpu_large(const vobj *lat, Integer osi
   while (w      <  words) { sumD_gpu_reduce_words< 1>(lat, osites, ret_p, w); w +=  1; }
 #ifdef GRID_REDUCTION_TIMING
   t_large += usecond();
-  std::cout << GridLogMessage << "sumD_gpu_large"
+  std::cout << GridLogDebug << "sumD_gpu_large"
             << " sizeof(sobjD)=" << sizeof(sobjD)
             << " words=" << words << " total=" << t_large << " us" << std::endl;
 #endif
@@ -438,5 +438,11 @@ inline typename vobj::scalar_object sum_gpu_large(const vobj *lat, Integer osite
   result = sumD_gpu_large(lat,osites);
   return result;
 }
+template<class Word> Word checksum_gpu(Word *vec,uint64_t L)
+{
+  Word w;
+  bzero(&w,sizeof(w));
+  return w;
+}
 
 NAMESPACE_END(Grid);

Grid/lattice/Lattice_slicesum_core.h

@@ -1,7 +1,6 @@
 #pragma once
 
 #if defined(GRID_CUDA)
-
 #include <cub/cub.cuh>
 #define gpucub cub
 #define gpuError_t cudaError_t
@@ -57,8 +56,13 @@ inline void sliceSumReduction_cub_small(const vobj *Data,
   //copy offsets to device
   acceleratorCopyToDeviceAsynch(&offsets[0],d_offsets,sizeof(int)*(rd+1),computeStream);
   
+#if defined(__CUDACC__) && (__CUDACC_VER_MAJOR__ >= 13)
+  #define GRID_CUB_SUM_OP ::cuda::std::plus<>{}
+#else
+  #define GRID_CUB_SUM_OP ::gpucub::Sum()
+#endif
   
-  gpuError_t gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p,d_out, rd, d_offsets, d_offsets+1, ::gpucub::Sum(), zero_init, computeStream);
+  gpuError_t gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p,d_out, rd, d_offsets, d_offsets+1, GRID_CUB_SUM_OP, zero_init, computeStream);
   if (gpuErr!=gpuSuccess) {
     std::cout << GridLogError << "Lattice_slicesum_gpu.h: Encountered error during gpucub::DeviceSegmentedReduce::Reduce (setup)! Error: " << gpuErr <<std::endl;
     exit(EXIT_FAILURE);
@@ -82,11 +86,13 @@ inline void sliceSumReduction_cub_small(const vobj *Data,
   });
   
   //issue segmented reductions in computeStream
-  gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p, d_out, rd, d_offsets, d_offsets+1,::gpucub::Sum(), zero_init, computeStream);
+  gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p, d_out, rd, d_offsets, d_offsets+1, GRID_CUB_SUM_OP, zero_init, computeStream);
   if (gpuErr!=gpuSuccess) {
     std::cout << GridLogError << "Lattice_slicesum_gpu.h: Encountered error during gpucub::DeviceSegmentedReduce::Reduce! Error: " << gpuErr <<std::endl;
     exit(EXIT_FAILURE);
   }
+
+#undef GRID_CUB_SUM_OP
   
   acceleratorCopyFromDeviceAsynch(d_out,&lvSum[0],rd*sizeof(vobj),computeStream);
   

Grid/lattice/Lattice_trace.h

@@ -124,6 +124,68 @@ Lattice<iScalar<iScalar<iMatrix<vComplexD, N> > > > Inverse(const Lattice<iScala
   return ret;
 }
 
+template<int N>
+Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > > , N> > Inverse(const Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > >, N> > &Umu)
+{
+  GridBase *grid=Umu.Grid();
+  auto lvol = grid->lSites();
+  Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > >, N > > ret(grid);
+  
+  autoView(Umu_v,Umu,CpuRead);
+  autoView(ret_v,ret,CpuWrite);
+  thread_for(site,lvol,{
+    Eigen::MatrixXcd EigenU = Eigen::MatrixXcd::Zero(N,N);
+    Coordinate lcoor;
+    grid->LocalIndexToLocalCoor(site, lcoor);
+    iMatrix<iScalar<iScalar<iScalar<ComplexD> > >, N > Us;
+    iMatrix<iScalar<iScalar<iScalar<ComplexD> > >, N > Ui;
+    peekLocalSite(Us, Umu_v, lcoor);
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	EigenU(i,j) = Us(i,j)()()();
+      }}
+    Eigen::MatrixXcd EigenUinv = EigenU.inverse();
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	Ui(i,j)()()() = EigenUinv(i,j);
+      }}
+    pokeLocalSite(Ui,ret_v,lcoor);
+  });
+  return ret;
+}
+
+template<int N>
+Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > > , N> > Inverse(const Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > >, N> > &Umu)
+{
+  GridBase *grid=Umu.Grid();
+  auto lvol = grid->lSites();
+  Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > >, N > > ret(grid);
+  
+  autoView(Umu_v,Umu,CpuRead);
+  autoView(ret_v,ret,CpuWrite);
+  thread_for(site,lvol,{
+    Eigen::MatrixXcd EigenU = Eigen::MatrixXcd::Zero(N,N);
+    Coordinate lcoor;
+    grid->LocalIndexToLocalCoor(site, lcoor);
+    iMatrix<iScalar<iScalar<iScalar<iScalar<ComplexD> > > >, N > Us;
+    iMatrix<iScalar<iScalar<iScalar<iScalar<ComplexD> > > >, N > Ui;
+    peekLocalSite(Us, Umu_v, lcoor);
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	EigenU(i,j) = Us(i,j)()()()();
+      }}
+    Eigen::MatrixXcd EigenUinv = EigenU.inverse();
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	Ui(i,j)()()()() = EigenUinv(i,j);
+      }}
+    pokeLocalSite(Ui,ret_v,lcoor);
+  });
+  return ret;
+}
+
+
+
 
 NAMESPACE_END(Grid);
 #endif

Grid/parallelIO/BinaryIO.cc

@@ -2,3 +2,7 @@
 
 int                    Grid::BinaryIO::latticeWriteMaxRetry = -1;
 Grid::BinaryIO::IoPerf Grid::BinaryIO::lastPerf;
+
+// Target size of a single contiguous file extent under BINARYIO_AGGREGATE.
+// 4MB is around the knee for Lustre; exposed so it can be swept at runtime.
+uint64_t               Grid::BinaryIO::aggregateTargetBytes = 4*1024*1024;

Grid/parallelIO/BinaryIO.h

@@ -39,6 +39,7 @@
 #endif
 
 #include <arpa/inet.h>
+#include <sys/stat.h>
 #include <algorithm>
 
 NAMESPACE_BEGIN(Grid);
@@ -87,6 +88,7 @@ class BinaryIO {
 
   static IoPerf lastPerf;
   static int latticeWriteMaxRetry;
+  static uint64_t aggregateTargetBytes;
 
   /////////////////////////////////////////////////////////////////////////////
   // more byte manipulation helpers
@@ -253,12 +255,392 @@ class BinaryIO {
   // Read or Write distributed lexico array of ANY object to a specific location in file 
   //////////////////////////////////////////////////////////////////////////////////////
 
+  static const int BINARYIO_AGGREGATE     = 0x20;
   static const int BINARYIO_MASTER_APPEND = 0x10;
   static const int BINARYIO_UNORDERED     = 0x08;
   static const int BINARYIO_LEXICOGRAPHIC = 0x04;
   static const int BINARYIO_READ          = 0x02;
   static const int BINARYIO_WRITE         = 0x01;
 
+#ifdef USE_MPI_IO
+  /////////////////////////////////////////////////////////////////////////////
+  // Aggregation: self controlled transposition onto an I/O friendly layout.
+  //
+  // Under BINARYIO_LEXICOGRAPHIC the subarray file view handed to MPI-IO has
+  // contiguous runs of only lLattice[0]*sizeof(fobj) bytes -- a few KB for
+  // typical local volumes. Rather than rely on collective buffering to repair
+  // that, redistribute the payload ourselves so every rank owns a contiguous
+  // range of the global lexicographic site ordering, then issue large plain
+  // contiguous writes.
+  //
+  // "Un-splitting" the nunsplit fastest dimensions means the row of ranks
+  // sharing the remaining process coordinates collectively owns whole global
+  // hyperplanes. All data movement is then confined to that row communicator.
+  // Every rank still owns exactly lSites() sites afterwards, so the exchange is
+  // a pure permutation and needs no divisibility condition on the process grid.
+  /////////////////////////////////////////////////////////////////////////////
+  struct AggregationPlan {
+    int      nunsplit{0};  // number of fastest dimensions un-split
+    int      rowsize{0};   // ranks in the aggregation (row) communicator
+    int      rowrank{0};   // our logical (lexicographic) index within the row
+    uint64_t lsites{0};    // sites per rank -- invariant under the permutation
+    uint64_t chunk{0};     // sites in one globally contiguous run owned by the row
+    std::unique_ptr<CartesianCommunicator> rowcomm;
+    // counts and displacements are indexed by rank within rowcomm
+    std::vector<int>      sendcounts, senddispls, recvcounts, recvdispls;
+    std::vector<uint64_t> scatter;     // recv slot -> slot in the aggregated buffer
+    std::vector<uint64_t> extentGsite; // global lex site index of extent start
+    std::vector<uint64_t> extentLocal; // offset of extent within aggregated buffer
+    std::vector<uint64_t> extentSites; // sites in this extent
+  };
+
+  static inline void BuildAggregationPlan(GridBase *grid,uint64_t fobjSize,AggregationPlan &p)
+  {
+    int ndim           = grid->Dimensions();
+    Coordinate psizes  = grid->ProcessorGrid();
+    Coordinate pcoor   = grid->ThisProcessorCoor();
+    Coordinate gLattice= grid->GlobalDimensions();
+    Coordinate lLattice= grid->LocalDimensions();
+    Coordinate lstart  = grid->LocalStarts();
+
+    uint64_t lsites = grid->lSites();
+    p.lsites = lsites;
+
+    //////////////////////////////////////////////////////////////////////////
+    // Un-splitting dims 0..k-1 gives the row a contiguous run of
+    //     chunk(k) = prod_{d<k} gLattice[d] * lLattice[k]
+    // sites, and each rank writes extents of min(chunk,lsites). Take the
+    // smallest k that reaches the target so we disturb as few dimensions --
+    // and move as little data -- as possible.
+    //////////////////////////////////////////////////////////////////////////
+    int k = ndim-1;
+    for(int trial=1; trial<ndim; trial++){
+      uint64_t chunk = lLattice[trial];
+      for(int d=0; d<trial; d++) chunk *= gLattice[d];
+      if ( std::min(chunk,lsites)*fobjSize >= aggregateTargetBytes ) { k = trial; break; }
+    }
+    p.nunsplit = k;
+
+    //////////////////////////////////////////////////////////////////////////
+    // The box the row collectively owns, expressed in global coordinates.
+    // Restricting the global lexicographic order to this box preserves the
+    // ordering, so the row index below is monotone in the global index.
+    //////////////////////////////////////////////////////////////////////////
+    Coordinate B(ndim), S(ndim);
+    for(int d=0; d<ndim; d++){
+      if ( d<k ) { B[d] = gLattice[d]; S[d] = 0;         }
+      else       { B[d] = lLattice[d]; S[d] = lstart[d]; }
+    }
+
+    uint64_t chunk = lLattice[k];
+    for(int d=0; d<k; d++) chunk *= gLattice[d];
+    p.chunk = chunk;
+
+    //////////////////////////////////////////////////////////////////////////
+    // Row communicator: the ranks sharing the process coordinates of the slow
+    // (still split) dimensions. This is the sub-division the Cartesian
+    // communicator already performs for AllToAll(dim,...), widened from one
+    // dimension to the k fastest.
+    //////////////////////////////////////////////////////////////////////////
+    Coordinate row(ndim,1);
+    for(int d=0; d<k; d++) row[d] = psizes[d];
+    int srank;
+    p.rowcomm.reset(new CartesianCommunicator(row,*grid,srank));
+    p.rowsize = p.rowcomm->ProcessorCount();
+
+    //////////////////////////////////////////////////////////////////////////
+    // Our logical index in the row is the forward lexicographic index of the
+    // un-split process coordinates, so that increasing logical index means
+    // increasing global lexicographic position in the file. The communicator
+    // numbers its own ranks by the reversed (MPI) convention, so build the map
+    // between the two rather than assuming either.
+    //////////////////////////////////////////////////////////////////////////
+    int64_t logical=0, lstride=1;
+    for(int d=0; d<k; d++){ logical += pcoor[d]*lstride; lstride *= psizes[d]; }
+    GRID_ASSERT(lstride == (int64_t)p.rowsize);
+    p.rowrank = (int)logical;
+
+    std::vector<uint64_t> commOf(p.rowsize,0);
+    commOf[p.rowrank] = (uint64_t)p.rowcomm->ThisRank();
+    p.rowcomm->GlobalSumVector(&commOf[0],p.rowsize);
+
+    uint64_t mystart = (uint64_t)p.rowrank * lsites;
+    uint64_t myend   = mystart + lsites;
+
+    Coordinate lcoor(ndim), bcoor(ndim), gcoor(ndim);
+
+    //////////////////////////////////////////////////////////////////////////
+    // Send side. Walking our local sites in local lexicographic order walks
+    // the row index monotonically, so the send buffer is iodata untouched and
+    // we need only the per destination counts.
+    //////////////////////////////////////////////////////////////////////////
+    std::vector<int> sendLogical(p.rowsize,0);
+    for(uint64_t L=0; L<lsites; L++){
+      Lexicographic::CoorFromIndex(lcoor,L,lLattice);
+      for(int d=0; d<ndim; d++) bcoor[d] = (d<k) ? (lstart[d]+lcoor[d]) : lcoor[d];
+      int64_t ri; Lexicographic::IndexFromCoor(bcoor,ri,B);
+      sendLogical[ ri/(int64_t)lsites ]++;
+    }
+    p.sendcounts.assign(p.rowsize,0);
+    p.senddispls.assign(p.rowsize,0);
+    { int64_t disp=0;
+      for(int d=0; d<p.rowsize; d++){          // send buffer is in logical order
+        int c = (int)commOf[d];
+        p.sendcounts[c] = sendLogical[d];
+        p.senddispls[c] = (int)disp;
+        disp += sendLogical[d];
+      }
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // Receive side. For each slot of our aggregated range work out which rank
+    // of the row owns it. Within one source the slots arrive in increasing row
+    // index order, which is the order the source sends them in.
+    //////////////////////////////////////////////////////////////////////////
+    std::vector<int> recvLogical(p.rowsize,0), recvDisplLogical(p.rowsize,0);
+    std::vector<int> source(lsites);
+    for(uint64_t pos=0; pos<lsites; pos++){
+      Lexicographic::CoorFromIndex(bcoor,(int64_t)(mystart+pos),B);
+      int64_t j=0, jstride=1;
+      for(int d=0; d<k; d++){ j += (bcoor[d]/lLattice[d])*jstride; jstride *= psizes[d]; }
+      source[pos] = (int)j;
+      recvLogical[j]++;
+    }
+    p.recvcounts.assign(p.rowsize,0);
+    p.recvdispls.assign(p.rowsize,0);
+    { int64_t disp=0;
+      for(int s=0; s<p.rowsize; s++){         // recv buffer is in logical order
+        int c = (int)commOf[s];
+        recvDisplLogical[s] = (int)disp;
+        p.recvcounts[c] = recvLogical[s];
+        p.recvdispls[c] = (int)disp;
+        disp += recvLogical[s];
+      }
+    }
+    p.scatter.resize(lsites);
+    {
+      std::vector<int> fill(p.rowsize,0);
+      for(uint64_t pos=0; pos<lsites; pos++){
+        int j = source[pos];
+        p.scatter[ recvDisplLogical[j] + fill[j]++ ] = pos;
+      }
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // The two sides are derived independently; make them check each other.
+    //////////////////////////////////////////////////////////////////////////
+    {
+      std::vector<uint64_t> sendc(p.rowsize),recvc(p.rowsize);
+      for(int c=0;c<p.rowsize;c++) sendc[c]=(uint64_t)p.sendcounts[c];
+      p.rowcomm->AllToAll(&sendc[0],&recvc[0],1,sizeof(uint64_t));
+      for(int c=0;c<p.rowsize;c++) GRID_ASSERT((int)recvc[c]==p.recvcounts[c]);
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // Decompose our range into globally contiguous file extents.
+    //////////////////////////////////////////////////////////////////////////
+    for(uint64_t c = mystart/chunk; c <= (myend-1)/chunk; c++){
+      uint64_t lo = std::max(mystart, c*chunk);
+      uint64_t hi = std::min(myend, (c+1)*chunk);
+      Lexicographic::CoorFromIndex(bcoor,(int64_t)(c*chunk),B);
+      for(int d=0;d<ndim;d++) gcoor[d] = (d<k) ? bcoor[d] : bcoor[d]+S[d];
+      int64_t gbase; Lexicographic::IndexFromCoor(gcoor,gbase,gLattice);
+      p.extentGsite.push_back( (uint64_t)gbase + (lo - c*chunk) );
+      p.extentLocal.push_back( lo - mystart );
+      p.extentSites.push_back( hi - lo );
+    }
+  }
+
+  static inline void ReportAggregationPlan(GridBase *grid,const AggregationPlan &p,uint64_t fobjSize,const char *what)
+  {
+    if ( !grid->IsBoss() ) return;
+    std::cout << GridLogMessage << "IOobject: aggregate " << what
+              << " un-splitting " << p.nunsplit << " fastest dimensions, row of "
+              << p.rowsize << " ranks" << std::endl;
+    std::cout << GridLogMessage << "IOobject: aggregate " << p.extentSites.size()
+              << " extent(s)/rank, first " << p.extentSites[0]*fobjSize/1024./1024. << " MB"
+              << " (target " << aggregateTargetBytes/1024./1024. << " MB)" << std::endl;
+    std::cout << GridLogMessage << "IOobject: aggregate buffer overhead "
+              << p.lsites*fobjSize/1024./1024. << " MB/rank" << std::endl;
+  }
+
+  ////////////////////////////////////////////////////////////////////////////
+  // Stage timings.  The interesting quantity is the slowest rank, since every
+  // stage is followed sooner or later by a synchronisation, so reduce with
+  // GlobalMax rather than reporting whatever the boss happened to see.
+  ////////////////////////////////////////////////////////////////////////////
+  static inline void ReportStages(GridBase *grid,const char *what,
+                                  const std::vector<const char *> &names,
+                                  std::vector<RealD> &useconds)
+  {
+    GRID_ASSERT(names.size()==useconds.size());
+    for(uint64_t i=0;i<useconds.size();i++) grid->GlobalMax(useconds[i]);
+    if ( grid->IsBoss() ) {
+      std::cout << GridLogMessage << "IOobject: aggregate " << what << " stages (max over ranks, s):";
+      for(uint64_t i=0;i<names.size();i++)
+        std::cout << " " << names[i] << " " << useconds[i]/1.0e6;
+      std::cout << std::endl;
+    }
+  }
+
+  template<class fobj>
+  static inline void AggregateExchange(GridBase *grid,AggregationPlan &p,std::vector<fobj> &iodata,
+                                       std::vector<fobj> &aggregated,int forward)
+  {
+    uint64_t lsites = p.lsites;
+    GridStopWatch talloc,tperm,tcomm;
+
+    talloc.Start();
+    std::vector<fobj> tmp(lsites);
+    talloc.Stop();
+
+    if ( forward ) { // iodata (local order) -> aggregated (lexicographic order)
+      tcomm.Start();
+      p.rowcomm->AllToAllV(&iodata[0],p.sendcounts,p.senddispls,
+                           &tmp[0],   p.recvcounts,p.recvdispls,sizeof(fobj));
+      tcomm.Stop();
+      tperm.Start();
+      thread_for(s,lsites,{ aggregated[p.scatter[s]] = tmp[s]; });
+      tperm.Stop();
+    } else {         // aggregated -> iodata, the exact mirror
+      tperm.Start();
+      thread_for(s,lsites,{ tmp[s] = aggregated[p.scatter[s]]; });
+      tperm.Stop();
+      tcomm.Start();
+      p.rowcomm->AllToAllV(&tmp[0],   p.recvcounts,p.recvdispls,
+                           &iodata[0],p.sendcounts,p.senddispls,sizeof(fobj));
+      tcomm.Stop();
+    }
+
+    std::vector<RealD> us = { (RealD)talloc.useconds(), (RealD)tperm.useconds(), (RealD)tcomm.useconds() };
+    ReportStages(grid,forward?"exchange (write)":"exchange (read)",
+                 {"alloc","permute","alltoallv"},us);
+  }
+
+  template<class fobj>
+  static inline void AggregateWrite(GridBase *grid,AggregationPlan &p,std::vector<fobj> &aggregated,
+                                    std::string file,uint64_t offset)
+  {
+    //////////////////////////////////////////////////////////////////////////
+    // All ranks write concurrently into a shared file, so the file must exist
+    // before any of them open it for update, but it does NOT have to be the
+    // right length first: the extents tile the record exactly, so writing them
+    // extends a short file to precisely offset+payload.
+    //
+    // Records are created in sequence, so this payload ends the file: the
+    // length must end up precisely offset+payload. Anything beyond is left
+    // over from whatever the file previously held and must not survive -- a
+    // shorter new record written over a longer old one would otherwise leave
+    // a trailing fragment of the previous contents masquerading as data.
+    // That is the only case needing a truncate, so stat first and truncate
+    // afterwards only when the size actually came out wrong. Measured on
+    // Frontier, an unconditional truncate up front cost 0.22 to 5.4 s per
+    // record -- 15 to 25% of a 19 GB write and 100% of a small one -- while
+    // create, open and close together cost a few milliseconds. It is per
+    // record, so multi record files do not amortise it away.
+    //
+    // ::truncate is used because the C++ standard library cannot express this.
+    // std::filebuf has no length operation at all; ios::trunc only truncates to
+    // zero; seeking past the end and writing a byte can grow a file but never
+    // shrink one; and there is no portable way to recover a descriptor from a
+    // stream in order to call ftruncate. C++17 does finally offer
+    // std::filesystem::resize_file, but that would be Grid's first <filesystem>
+    // dependency and needs -lstdc++fs on the older toolchains still in use.
+    //////////////////////////////////////////////////////////////////////////
+    GridStopWatch tcreate,ttrunc,tbar,topen,twrite,tclose,tskew;
+    uint64_t need = offset + (uint64_t)grid->_gsites*sizeof(fobj);
+
+    tcreate.Start();
+    if ( grid->IsBoss() ) {
+      // opening for update needs the file to exist; create one only if not
+      std::fstream probe(file,std::ios::binary|std::ios::out|std::ios::in);
+      if ( !probe.is_open() ) {
+        std::ofstream create(file,std::ios::binary|std::ios::out);
+        create.close();
+      }
+    }
+    tcreate.Stop();
+
+    tbar.Start();
+    grid->Barrier();
+    tbar.Stop();
+
+    std::ofstream fout;
+    fout.exceptions( std::fstream::failbit | std::fstream::badbit );
+    try {
+      topen.Start();
+      fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
+      topen.Stop();
+      twrite.Start();
+      for(uint64_t e=0;e<p.extentSites.size();e++){
+        fout.seekp(offset + p.extentGsite[e]*sizeof(fobj));
+        fout.write((char *)&aggregated[p.extentLocal[e]],p.extentSites[e]*sizeof(fobj));
+      }
+      twrite.Stop();
+      tclose.Start();
+      fout.close();   // flushes the stream buffer; does not force writeback
+      tclose.Stop();
+    } catch (const std::fstream::failure& exc) {
+      std::cout << GridLogError << "Error in aggregate write to " << file << std::endl;
+      std::cout << GridLogError << "Exception description: " << exc.what() << std::endl;
+      GridAbort();
+    }
+
+    ////////////////////////////////////////////////////////////////////////
+    // Timed apart from the truncate that follows it.  seek+write above is the
+    // slowest rank; this barrier is what the fastest rank then waits, so the
+    // pair separates the write cost from the spread across ranks. Folding it
+    // into the truncate makes a millisecond stat look like a second.
+    ////////////////////////////////////////////////////////////////////////
+    tskew.Start();
+    grid->Barrier();                 // every extent must be on its way first
+    tskew.Stop();
+
+    ttrunc.Start();
+    if ( grid->IsBoss() ) {
+      struct stat sb;
+      int ierr = ::stat(file.c_str(),&sb);
+      GRID_ASSERT(ierr==0);
+      if ( (uint64_t)sb.st_size != need ) {   // only when a longer record preceded us
+        ierr = ::truncate(file.c_str(),(off_t)need);
+        GRID_ASSERT(ierr==0);
+      }
+    }
+    grid->Barrier();
+    ttrunc.Stop();
+
+    std::vector<RealD> us = { (RealD)tcreate.useconds(), (RealD)tbar.useconds(),
+                              (RealD)topen.useconds(),   (RealD)twrite.useconds(),
+                              (RealD)tclose.useconds(),  (RealD)tskew.useconds(),
+                              (RealD)ttrunc.useconds() };
+    ReportStages(grid,"write",{"create","barrier","open","seek+write","close","skew","stat+truncate"},us);
+  }
+
+  template<class fobj>
+  static inline void AggregateRead(GridBase *grid,AggregationPlan &p,std::vector<fobj> &aggregated,
+                                   std::string file,uint64_t offset)
+  {
+    GridStopWatch topen,tread,tclose;
+    std::ifstream fin;
+    topen.Start();
+    fin.open(file,std::ios::binary|std::ios::in);
+    topen.Stop();
+    tread.Start();
+    for(uint64_t e=0;e<p.extentSites.size();e++){
+      fin.seekg(offset + p.extentGsite[e]*sizeof(fobj));
+      fin.read((char *)&aggregated[p.extentLocal[e]],p.extentSites[e]*sizeof(fobj));
+      GRID_ASSERT(fin.fail()==0);
+    }
+    tread.Stop();
+    tclose.Start();
+    fin.close();
+    tclose.Stop();
+
+    std::vector<RealD> us = { (RealD)topen.useconds(), (RealD)tread.useconds(), (RealD)tclose.useconds() };
+    ReportStages(grid,"read",{"open","seek+read","close"},us);
+  }
+#endif
+
   template<class word,class fobj>
   static inline void IOobject(word w,
 			      GridBase *grid,
@@ -302,6 +684,18 @@ class BinaryIO {
       lStart[d] = 0;
     }
 
+    //////////////////////////////////////////////////////////////////////////////
+    // Aggregate the lexicographic layout onto contiguous per rank extents
+    // ourselves rather than leaving it to MPI-IO collective buffering
+    //////////////////////////////////////////////////////////////////////////////
+    int aggregate = (control & BINARYIO_AGGREGATE)
+                 && (control & BINARYIO_LEXICOGRAPHIC)
+                 && !(control & BINARYIO_MASTER_APPEND)
+                 && (nrank > 1);
+#ifndef USE_MPI_IO
+    GRID_ASSERT(aggregate==0); // BINARYIO_AGGREGATE requires MPI
+#endif
+
 #ifdef USE_MPI_IO
     std::vector<int> distribs(ndim,MPI_DISTRIBUTE_BLOCK);
     std::vector<int> dargs   (ndim,MPI_DISTRIBUTE_DFLT_DARG);
@@ -329,6 +723,8 @@ class BinaryIO {
     ierr = MPI_Type_contiguous(numword,mpiword,&mpiObject);    GRID_ASSERT(ierr==0);
     ierr = MPI_Type_commit(&mpiObject);
 
+    // The subarray view is what aggregation exists to avoid; do not build it
+    if ( !aggregate ) {
     //////////////////////////////////////////////////////////////////////////////
     // File global array data type
     //////////////////////////////////////////////////////////////////////////////
@@ -340,6 +736,7 @@ class BinaryIO {
     //////////////////////////////////////////////////////////////////////////////
     ierr=MPI_Type_create_subarray(ndim,&lLattice[0],&lLattice[0],&lStart[0],MPI_ORDER_FORTRAN, mpiObject,&localArray);    GRID_ASSERT(ierr==0);
     ierr=MPI_Type_commit(&localArray);    GRID_ASSERT(ierr==0);
+    }
 #endif
 
     //////////////////////////////////////////////////////////////////////////////
@@ -358,7 +755,19 @@ class BinaryIO {
 
       timer.Start();
 
-      if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
+      if ( aggregate ) {
+#ifdef USE_MPI_IO
+        std::cout<< GridLogMessage<<"IOobject: aggregate read I/O "<< file<< std::endl;
+        AggregationPlan plan;
+        BuildAggregationPlan(grid,sizeof(fobj),plan);
+        ReportAggregationPlan(grid,plan,sizeof(fobj),"read");
+        std::vector<fobj> aggregated(lsites);
+        AggregateRead(grid,plan,aggregated,file,offset);
+        AggregateExchange(grid,plan,iodata,aggregated,0);
+#else
+        GRID_ASSERT(0);
+#endif
+      } else if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
 #ifdef USE_MPI_IO
 	std::cout<< GridLogMessage<<"IOobject: MPI read I/O "<< file<< std::endl;
 	ierr=MPI_File_open(grid->communicator,(char *) file.c_str(), MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);    GRID_ASSERT(ierr==0);
@@ -416,7 +825,25 @@ class BinaryIO {
       grid->Barrier();
 
       timer.Start();
-      if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
+      if ( aggregate ) {
+#ifdef USE_MPI_IO
+        std::cout << GridLogMessage <<"IOobject: aggregate write I/O " << file << std::endl;
+        AggregationPlan plan;
+        BuildAggregationPlan(grid,sizeof(fobj),plan);
+        ReportAggregationPlan(grid,plan,sizeof(fobj),"write");
+        std::vector<fobj> aggregated(lsites);
+        AggregateExchange(grid,plan,iodata,aggregated,1);
+        AggregateWrite(grid,plan,aggregated,file,offset);
+        ////////////////////////////////////////////////////////////////////////
+        // Not every rank ends at the end of the payload, so the position can
+        // not be recovered from a file handle. Callers (Lime record chaining)
+        // rely on this being the first byte past the record.
+        ////////////////////////////////////////////////////////////////////////
+        offset = offset + (uint64_t)grid->_gsites*sizeof(fobj);
+#else
+        GRID_ASSERT(0);
+#endif
+      } else if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
 #ifdef USE_MPI_IO
         std::cout << GridLogMessage <<"IOobject: MPI write I/O " << file << std::endl;
         ierr = MPI_File_open(grid->communicator, (char *)file.c_str(), MPI_MODE_RDWR | MPI_MODE_CREATE, MPI_INFO_NULL, &fh);
@@ -461,12 +888,26 @@ class BinaryIO {
         
 	std::ofstream fout; 
 	fout.exceptions ( std::fstream::failbit | std::fstream::badbit );
+
+        ////////////////////////////////////////////////////////////////////
+        // Grid's model is that the boss rank performs the metadata
+        // operations and every other rank only seeks and writes into a file
+        // that already exists. Opening with ios::out on all ranks broke that:
+        // it is O_TRUNC, so a rank opening late truncated the file back to
+        // zero after an earlier rank had written its segment, leaving a hole
+        // in its place. The barriers around this block are outside it and do
+        // not order the opens against the writes. Let the boss create and
+        // empty the file, then everyone opens for update only. Same resulting
+        // length, one metadata operation instead of one per rank, no race.
+        ////////////////////////////////////////////////////////////////////
+        if ( !offset && grid->IsBoss() ) { // offset zero: this record starts the file
+          std::ofstream create(file,std::ios::binary|std::ios::out);
+          create.close();
+        }
+        grid->Barrier();
+
 	try {
-	  if (offset) { // Must already exist and contain data
+	  fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
-	    fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
-	  } else {     // Allow create
-	    fout.open(file,std::ios::binary|std::ios::out);
-	  }
 	} catch (const std::fstream::failure& exc) {
 	  std::cout << GridLogError << "Error in opening the file " << file << " for output" <<std::endl;
 	  std::cout << GridLogError << "Exception description: " << exc.what() << std::endl;
@@ -477,7 +918,7 @@ class BinaryIO {
 	  exit(1);
 #endif
 	}
-	
+
 	if ( control & BINARYIO_MASTER_APPEND )  {
 	  try {
 	    fout.seekp(0,fout.end);
@@ -548,7 +989,7 @@ class BinaryIO {
 				       uint32_t &nersc_csum,
 				       uint32_t &scidac_csuma,
 				       uint32_t &scidac_csumb,
-				       int control=BINARYIO_LEXICOGRAPHIC
+				       int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE
 				       )
   {
     typedef typename vobj::scalar_object sobj;
@@ -587,7 +1028,7 @@ class BinaryIO {
 					  uint32_t &nersc_csum,
 					  uint32_t &scidac_csuma,
 					  uint32_t &scidac_csumb,
-					  int control=BINARYIO_LEXICOGRAPHIC)
+					  int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     typedef typename vobj::scalar_object sobj;
     typedef typename vobj::Realified::scalar_type word;    word w=0;
@@ -674,7 +1115,7 @@ class BinaryIO {
     std::cout << GridLogMessage << "RNG read I/O on file " << file << std::endl;
 
     std::vector<RNGstate> iodata(lsites);
-    IOobject(w,grid,iodata,file,offset,format,BINARYIO_READ|BINARYIO_LEXICOGRAPHIC,
+    IOobject(w,grid,iodata,file,offset,format,BINARYIO_READ|BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE,
 	     nersc_csum,scidac_csuma,scidac_csumb);
 
     timer.Start();
@@ -753,7 +1194,7 @@ class BinaryIO {
     });
     timer.Stop();
 
-    IOobject(w,grid,iodata,file,offset,format,BINARYIO_WRITE|BINARYIO_LEXICOGRAPHIC,
+    IOobject(w,grid,iodata,file,offset,format,BINARYIO_WRITE|BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE,
 	     nersc_csum,scidac_csuma,scidac_csumb);
     iodata.resize(1);
     {

Grid/parallelIO/IldgIO.h

@@ -212,7 +212,7 @@ class GridLimeReader : public BinaryIO {
   // Read a generic lattice field and verify checksum
   ////////////////////////////////////////////
   template<class vobj>
-  void readLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC)
+  void readLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     typedef typename vobj::scalar_object sobj;
     scidacChecksum scidacChecksum_;
@@ -414,7 +414,7 @@ class GridLimeWriter : public BinaryIO
   // in communicator used by the field.Grid()
   ////////////////////////////////////////////////////
   template<class vobj>
-  void writeLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC)
+  void writeLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     ////////////////////////////////////////////////////////////////////
     // NB: FILE and iostream are jointly writing disjoint sequences in the
@@ -519,7 +519,7 @@ class ScidacWriter : public GridLimeWriter {
   template <class vobj, class userRecord>
   void writeScidacFieldRecord(Lattice<vobj> &field,userRecord _userRecord,
                               const unsigned int recordScientificPrec = 0,
-			      int control=BINARYIO_LEXICOGRAPHIC)
+			      int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     GridBase * grid = field.Grid();
 
@@ -561,7 +561,7 @@ class ScidacReader : public GridLimeReader {
   ////////////////////////////////////////////////
   template <class vobj, class userRecord>
   void readScidacFieldRecord(Lattice<vobj> &field,userRecord &_userRecord,
-			     int control=BINARYIO_LEXICOGRAPHIC) 
+			     int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE) 
   {
     typedef typename vobj::scalar_object sobj;
     GridBase * grid = field.Grid();

Grid/perfmon/PerfCount.h

@@ -51,8 +51,8 @@ Author: paboyle <paboyle@ph.ed.ac.uk>
 #endif
 #ifdef __x86_64__
 #ifdef GRID_CUDA
-//accelerator_inline uint64_t __rdtsc(void) {  return 0; }
+accelerator_inline uint64_t __rdtsc(void) {  return 0; }
-//accelerator_inline uint64_t __rdpmc(int ) {  return 0; }
+accelerator_inline uint64_t __rdpmc(int ) {  return 0; }
 #else
 #include <x86intrin.h>
 #endif
@@ -93,8 +93,7 @@ inline uint64_t cyclecount(void){
 }
 #elif defined __x86_64__
 inline uint64_t cyclecount(void){ 
-  uint64_t ret = __rdtsc();
+  return (uint64_t)0;
-  return (uint64_t)ret;
 }
 #else
 

Grid/qcd/action/pseudofermion/TwoFlavourBosonPseudoFermion.h

@@ -0,0 +1,133 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./lib/qcd/action/pseudofermion/TwoFlavourBosonPseudoFermion.h
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#pragma once
+
+NAMESPACE_BEGIN(Grid);
+
+///////////////////////////////////////////////////////////////////////////////
+// Two flavour BOSON (wrong-sign) pseudofermion for any FermionOperator B:
+//
+//    S2 = chi^dag Bdag B chi = |B chi|^2
+//
+//    integral  ==>  det( Bdag B )^-1 = |det B|^-2
+//
+// A compensator monomial: supplies an INVERSE determinant with NO solve in
+// the force or the action -- both are matrix multiplies.  The only solve is
+// the heatbath  chi = B^-1 eta,  once per trajectory (for B = the
+// Pauli-Villars operator this is a mass-one solve, trivially cheap).
+//
+// Primary use: two instances with B = PV cancel the |det PV|^2 excess of
+// TwoFlavourPVdagMPseudoFermionAction down to the DWF quotient
+// |det M|^2/|det PV|^2 (two unsquared instances rather than one squared
+// kernel: first powers of PV in the force, milder).  Being generic in B it
+// also serves Hasenbusch-chain compensation at intermediate masses, or any
+// future inverse-det bookkeeping.  (Sibling of the domain-decomposed boson
+// in DomainDecomposedBoundaryTwoFlavourBosonPseudoFermion.h, without the
+// boundary machinery.)
+//
+// Heatbath exact by construction: S2 after refresh = |B B^-1 eta|^2 = |eta|^2.
+///////////////////////////////////////////////////////////////////////////////
+template<class Impl>
+class TwoFlavourBosonPseudoFermionAction : public Action<typename Impl::GaugeField> {
+public:
+  INHERIT_IMPL_TYPES(Impl);
+
+private:
+  FermionOperator<Impl> & BOp;  // the operator whose |det|^-2 is supplied
+
+  LinearFunction<FermionField> &HeatbathSolver; // b -> B^-1 b (heatbath only)
+
+  FermionField Chi; // the pseudo fermion field for this trajectory
+
+public:
+  TwoFlavourBosonPseudoFermionAction(FermionOperator<Impl>  &_BOp,
+				     LinearFunction<FermionField> & HS
+				     ) : BOp(_BOp),
+					 HeatbathSolver(HS),
+					 Chi(_BOp.FermionGrid())
+  {};
+
+  virtual std::string action_name(){return "TwoFlavourBosonPseudoFermionAction";}
+
+  virtual std::string LogParameters(){
+    std::stringstream sstream;
+    sstream << GridLogMessage << "["<<action_name()<<"] has no parameters" << std::endl;
+    return sstream.str();
+  }
+
+  virtual void refresh(const GaugeField &U, GridSerialRNG &sRNG, GridParallelRNG& pRNG) {
+    // P(chi) = e^{- chi^dag BdagB chi} ; chi = B^-1 eta ; P(eta) = e^{-eta^dag eta}
+    // e^{-x^2/2 sig^2} => sig^2 = 0.5 ; eta enters with width 1/sqrt(2).
+    RealD scale = std::sqrt(0.5);
+    FermionField eta(BOp.FermionGrid());
+    gaussian(pRNG,eta);
+    eta = eta * scale;
+    refresh(U,eta);
+  }
+
+  // Deterministic-noise variant (test hook):
+  // after this,  S(U) == norm2(eta)  exactly (to solver tolerance).
+  void refresh(const GaugeField &U, const FermionField &eta) {
+    BOp.ImportGauge(U);
+    Chi = Zero();
+    HeatbathSolver(eta,Chi);       // Chi = B^-1 eta : the ONLY solve
+    std::cout << GridLogMessage << action_name() << " refresh |Chi|^2 = "<< norm2(Chi)<<std::endl;
+  }
+
+  //////////////////////////////////////////////////////
+  // S2 = |B chi|^2  -- matrix multiply only
+  //////////////////////////////////////////////////////
+  virtual RealD S(const GaugeField &U) {
+    BOp.ImportGauge(U);
+
+    FermionField w(BOp.FermionGrid());
+    BOp.M(Chi,w);                  // w = B chi
+    RealD action = norm2(w);
+    return action;
+  }
+
+  //////////////////////////////////////////////////////
+  // dS2 = chi^dag dBdag w + w^dag dB chi ,   w = B chi
+  // NO solves.
+  //////////////////////////////////////////////////////
+  virtual void deriv(const GaugeField &U,GaugeField & dSdU) {
+    BOp.ImportGauge(U);
+
+    FermionField w(BOp.FermionGrid());
+    GaugeField force(BOp.GaugeGrid());
+
+    BOp.M(Chi,w);                  // w = B chi
+
+    BOp.MDeriv(force, Chi, w, DaggerYes); dSdU =      force;
+    BOp.MDeriv(force, w, Chi, DaggerNo ); dSdU = dSdU+force;
+
+    dSdU *= -1.0;  // Grid action sign convention (cf TwoFlavourRatio.h)
+  };
+};
+
+NAMESPACE_END(Grid);

Grid/qcd/action/pseudofermion/TwoFlavourRatio4DPseudoFermion.h

@@ -0,0 +1,264 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./lib/qcd/action/pseudofermion/TwoFlavourRatio4DPseudoFermion.h
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#pragma once
+
+NAMESPACE_BEGIN(Grid);
+
+///////////////////////////////////////////////////////////////////////////////
+// Two flavour ratio with FOUR dimensional pseudofermion, UNpreconditioned
+// (full grid) solves.
+//
+// Companion to TwoFlavourRatioEO4DPseudoFermion.h but with the solver
+// plumbing exposed as LinearFunction<FermionField> objects that already
+// know their operator -- the natural interface for the non-Hermitian
+// multigrid GCR stack (PVdagM), which solves M and Mdag DIRECTLY rather
+// than through SchurRedBlack normal equations.
+//
+// Why: with 5D pseudofermions the squared-operator formulation hands
+// normal-equation solvers (MdagM)^-1 phi AND Mdag^-1 phi from ONE Krylov
+// space; a direct solver must solve twice, halving its per-solve gain.
+// The 4D pseudofermion action needs one M^-1 and one M^-dag solve per
+// force evaluation FOR BOTH solver families, so the direct-solver gain
+// carries through undiluted.  In addition phi4 is Ls-agnostic, so the
+// force can be evaluated with a reduced-Ls operator pair while the
+// accept/reject uses full Ls (inexact force, exact action).
+//
+// Solver slots (all full-grid 5D LinearFunctions, solution overwritten,
+// zero guess imposed internally):
+//   DerivMinvSolver     : x = M^-1 b        (DenOp)
+//   DerivMdagInvSolver  : x = M^-dag b      (DenOp).  For G5R5-hermitian
+//                         actions this may be implemented by the caller as
+//                         G5R5 . DerivMinvSolver . G5R5 -- no adjoint
+//                         multigrid needed.
+//   ActionMinvSolver    : x = M^-1 b        (DenOp, accept/reject tolerance)
+//   HeatbathVinvSolver  : x = V^-1 b        (NumOp)
+//
+// 4D <-> 5D wall maps: the action is S = | P (M^-1 V) Pdag phi4 |^2 where
+// (P,Pdag) MUST be a mutually adjoint pair for S and deriv to be
+// consistent.  Two candidate conventions, selected by solution_walls:
+//   true  : P = P_- psi(0) + P_+ psi(Ls-1)   (solution walls, matches
+//           ExportPhysicalFermionSolution) and Pdag its literal adjoint.
+//   false : P = P_+ psi(0) + P_- psi(Ls-1)   (source walls, Pdag matches
+//           ImportUnphysicalFermion).
+// The heatbath is exact iff [P M^-1 V Pdag][P V^-1 M Pdag] = 1 (the 4D
+// effective-operator composition identity); which convention satisfies it
+// is settled numerically by the refresh test  S == 0.5*|eta4|^2  exactly.
+///////////////////////////////////////////////////////////////////////////////
+template<class Impl>
+class TwoFlavourRatio4DPseudoFermionAction : public Action<typename Impl::GaugeField> {
+public:
+  INHERIT_IMPL_TYPES(Impl);
+
+private:
+  typedef FermionOperator<Impl> FermOp;
+  FermionOperator<Impl> & NumOp;// the basic operator (V)
+  FermionOperator<Impl> & DenOp;// the basic operator (M)
+
+  LinearFunction<FermionField> &DerivMinvSolver;
+  LinearFunction<FermionField> &DerivMdagInvSolver;
+  LinearFunction<FermionField> &ActionMinvSolver;
+  LinearFunction<FermionField> &HeatbathVinvSolver;
+
+  FermionField phi4; // the pseudo fermion field for this trajectory
+
+  int solution_walls; // wall convention for the (P,Pdag) pair; see header
+
+  ////////////////////////////////////////////////////////////////////
+  // The mutually adjoint 4D <-> 5D pair.
+  //   Wall4D    : q4  = P psi5      (extract)
+  //   Wall4DAdj : psi5 = Pdag q4    (insert; literal adjoint of Wall4D)
+  ////////////////////////////////////////////////////////////////////
+  void Wall4D(const FermionField &psi5, FermionField &q4)
+  {
+    int Ls = NumOp.FermionGrid()->_fdimensions[0];
+    FermionField tmp(NumOp.FermionGrid());
+    if ( solution_walls ) {
+      // q4 = P_- psi(0) + P_+ psi(Ls-1)
+      axpby_ssp_pminus(tmp, 0., psi5, 1., psi5, 0, 0);
+      axpby_ssp_pplus (tmp, 1., tmp , 1., psi5, 0, Ls-1);
+    } else {
+      // q4 = P_+ psi(0) + P_- psi(Ls-1)
+      axpby_ssp_pplus (tmp, 0., psi5, 1., psi5, 0, 0);
+      axpby_ssp_pminus(tmp, 1., tmp , 1., psi5, 0, Ls-1);
+    }
+    ExtractSlice(q4, tmp, 0, 0);
+  }
+  void Wall4DAdj(const FermionField &q4, FermionField &psi5)
+  {
+    int Ls = NumOp.FermionGrid()->_fdimensions[0];
+    FermionField tmp(NumOp.FermionGrid());
+    tmp = Zero();
+    InsertSlice(q4, tmp, 0   , 0);
+    InsertSlice(q4, tmp, Ls-1, 0);
+    if ( solution_walls ) {
+      // psi(0) = P_- q4 ; psi(Ls-1) = P_+ q4
+      axpby_ssp_pminus(tmp, 0., tmp, 1., tmp, 0   , 0);
+      axpby_ssp_pplus (tmp, 0., tmp, 1., tmp, Ls-1, Ls-1);
+    } else {
+      // psi(0) = P_+ q4 ; psi(Ls-1) = P_- q4
+      axpby_ssp_pplus (tmp, 0., tmp, 1., tmp, 0   , 0);
+      axpby_ssp_pminus(tmp, 0., tmp, 1., tmp, Ls-1, Ls-1);
+    }
+    psi5 = tmp;
+  }
+
+public:
+  TwoFlavourRatio4DPseudoFermionAction(FermionOperator<Impl>  &_NumOp,
+				       FermionOperator<Impl>  &_DenOp,
+				       LinearFunction<FermionField> & DMS,
+				       LinearFunction<FermionField> & DMDS,
+				       LinearFunction<FermionField> & AMS,
+				       LinearFunction<FermionField> & HVS,
+				       int _solution_walls = 1
+				       ) : NumOp(_NumOp),
+					   DenOp(_DenOp),
+					   DerivMinvSolver(DMS),
+					   DerivMdagInvSolver(DMDS),
+					   ActionMinvSolver(AMS),
+					   HeatbathVinvSolver(HVS),
+					   phi4(_NumOp.GaugeGrid()),
+					   solution_walls(_solution_walls)
+  {};
+
+  virtual std::string action_name(){return "TwoFlavourRatio4DPseudoFermionAction";}
+
+  virtual std::string LogParameters(){
+    std::stringstream sstream;
+    sstream << GridLogMessage << "["<<action_name()<<"] solution_walls " << solution_walls << std::endl;
+    return sstream.str();
+  }
+
+  virtual void refresh(const GaugeField &U, GridSerialRNG &sRNG, GridParallelRNG& pRNG) {
+
+    // P(phi4) = e^{- phi4^dag Beff^dag Beff phi4}   ;  Beff = P M^-1 V Pdag
+    //
+    // NumOp == V
+    // DenOp == M
+    //
+    // Take phi4 = P V^-1 M Pdag eta4   ( = Beff^-1 eta4 by the composition
+    // identity; verified numerically by S == 0.5 |eta4|^2 after refresh )
+    //
+    // P(eta) = e^{- eta^dag eta} ;  e^{-x^2/2 sig^2} => sig^2 = 0.5
+    // so eta enters with width 1/sqrt(2).
+    //
+    RealD scale = std::sqrt(0.5);
+
+    FermionField eta4(NumOp.GaugeGrid());
+    FermionField eta5(NumOp.FermionGrid());
+    FermionField tmp (NumOp.FermionGrid());
+    FermionField phi5(NumOp.FermionGrid());
+
+    gaussian(pRNG,eta4);
+
+    NumOp.ImportGauge(U);
+    DenOp.ImportGauge(U);
+
+    Wall4DAdj(eta4,eta5);            // eta5 = Pdag eta4
+    DenOp.M(eta5,tmp);               // tmp  = M eta5
+    phi5 = Zero();
+    HeatbathVinvSolver(tmp,phi5);    // phi5 = V^-1 M eta5
+    Wall4D(phi5,phi4);               // phi4 = P phi5
+    phi4 = phi4*scale;
+
+    std::cout << GridLogMessage << "4d pf (non-EO) refresh "<< norm2(phi4)<<"\n";
+  };
+
+  //////////////////////////////////////////////////////
+  // S = phi4^dag (Pdag^dag V^dag M^-dag P^dag) (P M^-1 V Pdag) phi4
+  //   = | P M^-1 V Pdag phi4 |^2
+  //////////////////////////////////////////////////////
+  virtual RealD S(const GaugeField &U) {
+
+    NumOp.ImportGauge(U);
+    DenOp.ImportGauge(U);
+
+    FermionField Y4  (NumOp.GaugeGrid());
+    FermionField phi5(NumOp.FermionGrid());
+    FermionField X   (NumOp.FermionGrid());
+    FermionField Y   (NumOp.FermionGrid());
+
+    Wall4DAdj(phi4,phi5);            // phi5 = Pdag phi4
+    NumOp.M(phi5,X);                 // X    = V phi5
+    Y = Zero();
+    ActionMinvSolver(X,Y);           // Y    = M^-1 V phi5
+    Wall4D(Y,Y4);                    // Y4   = P Y
+
+    RealD action = norm2(Y4);
+
+    return action;
+  };
+
+  //////////////////////////////////////////////////////
+  // dS/du = 2 Re [ (M^-dag Pdag w4)^dag dV Pdag phi4 ]
+  //       - 2 Re [ (M^-dag Pdag w4)^dag dM (M^-1 V Pdag phi4) ]
+  // with w4 = P M^-1 V Pdag phi4.
+  // Two first-power solves: one M^-1, one M^-dag.
+  //////////////////////////////////////////////////////
+  virtual void deriv(const GaugeField &U,GaugeField & dSdU) {
+
+    NumOp.ImportGauge(U);
+    DenOp.ImportGauge(U);
+
+    FermionField  phi5           (NumOp.FermionGrid());
+    FermionField  Vphi           (NumOp.FermionGrid());
+    FermionField  MinvVphi       (NumOp.FermionGrid());
+    FermionField  w4             (NumOp.GaugeGrid());
+    FermionField  Y              (NumOp.FermionGrid());
+    FermionField  MdagInvPdagW   (NumOp.FermionGrid());
+
+    GaugeField   force(NumOp.GaugeGrid());
+
+    Wall4DAdj(phi4,phi5);              // phi5     = Pdag phi4
+    NumOp.M(phi5,Vphi);                // Vphi     = V phi5
+    MinvVphi = Zero();
+    DerivMinvSolver(Vphi,MinvVphi);    // MinvVphi = M^-1 V phi5
+    std::cout << GridLogMessage << "4d pf (non-EO) deriv solve "<< norm2(MinvVphi)<<"\n";
+
+    // Project onto the physical 4D subspace and back: Y = Pdag P MinvVphi.
+    // Pdag here MUST be the literal adjoint of the P used in S, else the
+    // force is inconsistent with the action.
+    Wall4D(MinvVphi,w4);               // w4 = P MinvVphi
+    Wall4DAdj(w4,Y);                   // Y  = Pdag w4
+
+    MdagInvPdagW = Zero();
+    DerivMdagInvSolver(Y,MdagInvPdagW);   // = M^-dag Pdag w4  (adjoint solve)
+    std::cout << GridLogMessage << "4d pf (non-EO) deriv solve dag "<< norm2(MdagInvPdagW)<<"\n";
+
+    // phi^dag (Pdag' Vdag Mdag^-1 P') (dV) Pdag phi   + h.c.
+    NumOp.MDeriv(force, MdagInvPdagW, phi5, DaggerNo );  dSdU=force;
+    NumOp.MDeriv(force, phi5, MdagInvPdagW, DaggerYes);  dSdU=dSdU+force;
+
+    // - phi^dag ( ... Mdag^-1 ) dM ( M^-1 V ... ) phi  + h.c.
+    DenOp.MDeriv(force, MdagInvPdagW, MinvVphi, DaggerNo );  dSdU=dSdU-force;
+    DenOp.MDeriv(force, MinvVphi, MdagInvPdagW, DaggerYes);  dSdU=dSdU-force;
+
+    dSdU *= -1.0;
+  };
+};
+
+NAMESPACE_END(Grid);

Grid/qcd/action/pseudofermion/TwoFlavourRatioLeftPrec.h

@@ -0,0 +1,206 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./lib/qcd/action/pseudofermion/TwoFlavourRatioLeftPrec.h
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#pragma once
+
+NAMESPACE_BEGIN(Grid);
+
+///////////////////////////////////////////////////////////////////////////////
+// Two flavour ratio with LEFT-PRECONDITIONED solves.
+//
+// Same action content as TwoFlavourRatio.h:
+//
+//    S = phi^dag V (Mdag M)^-1 Vdag phi     ==>   det[ Mdag M / Vdag V ]
+//
+// (V = NumOp the heavier / Pauli-Villars operator, M = DenOp the lighter),
+// but organised around the composite
+//
+//    F = Vdag M
+//
+// which is the 2-hop-coarsenable operator the non-Hermitian multigrid
+// serves.  Solving M X = b as F X = Vdag b is LEFT PRECONDITIONING by
+// Vdag; the determinant/action layer is the standard quotient, and all
+// novelty is confined to the solver contract.
+//
+// TwoFlavourRatio.h is tied to a normal-equations solver: one (MdagM)^-1
+// solve, then Y = M X gives Mdag^-1 Vdag phi almost free.  The left-
+// preconditioned idiom is DIFFERENT: the chain
+//
+//    b = Vdag phi
+//    z : Fdag z = b          (adjoint  F solve)
+//    Y = V z                 (= Mdag^-1 Vdag phi -- harvested from solve 1)
+//    s = Vdag Y              (= Vdag V z)
+//    X : F X = s             (forward F solve;  X = (MdagM)^-1 Vdag phi)
+//
+// yields Y BEFORE X (so S(U) needs only the adjoint solve), with Y's
+// accuracy independent of the second solve.  Force terms are then the
+// standard four MDeriv insertions of TwoFlavourRatio.
+//
+// Solver slots are LinearFunctions with the F-SOLVE contract (solution
+// overwritten, zero guess imposed internally):
+//    ForwardSolver(b,x)  :  F x    = b
+//    AdjointSolver(b,z)  :  Fdag z = b
+// implemented in production by the multigrid-GCR stack (forward cycle and
+// adjoint cycle); in tests by CG on the composite normal equations.
+// HeatbathSolver(b,x) : x = (Vdag V)^-1 b -- heavy operator, plain CG.
+//
+// Heatbath is exact by operator algebra:  phi = V (VdagV)^-1 Mdag eta
+// ==>  S = | Mdag^-1 Vdag phi |^2 = |eta|^2  (to solver tolerance); the
+// deterministic refresh(U,eta) hook below is the test point.
+//
+// Hasenbusch: nothing requires V to have mass one; any (heavier,lighter)
+// pair works, F(V,M) = Vdag M coarsenable by the same machinery, rungs'
+// solves are F-family (mrhs-batchable, mass-shared coarse space).
+///////////////////////////////////////////////////////////////////////////////
+template<class Impl>
+class TwoFlavourRatioLeftPrecPseudoFermionAction : public Action<typename Impl::GaugeField> {
+public:
+  INHERIT_IMPL_TYPES(Impl);
+
+private:
+  FermionOperator<Impl> & NumOp;// V
+  FermionOperator<Impl> & DenOp;// M
+
+  LinearFunction<FermionField> &DerivForwardSolver;  // F x = b, MD tolerance
+  LinearFunction<FermionField> &DerivAdjointSolver;  // Fdag z = b, MD tolerance
+  LinearFunction<FermionField> &ActionAdjointSolver; // Fdag z = b, accept/reject tolerance
+  LinearFunction<FermionField> &HeatbathSolver;      // (VdagV)^-1 b, heavy op
+
+  FermionField Phi; // the pseudo fermion field for this trajectory
+
+public:
+  TwoFlavourRatioLeftPrecPseudoFermionAction(FermionOperator<Impl>  &_NumOp,
+					     FermionOperator<Impl>  &_DenOp,
+					     LinearFunction<FermionField> & DFS,
+					     LinearFunction<FermionField> & DAS,
+					     LinearFunction<FermionField> & AAS,
+					     LinearFunction<FermionField> & HS
+					     ) : NumOp(_NumOp),
+						 DenOp(_DenOp),
+						 DerivForwardSolver(DFS),
+						 DerivAdjointSolver(DAS),
+						 ActionAdjointSolver(AAS),
+						 HeatbathSolver(HS),
+						 Phi(_NumOp.FermionGrid())
+  {};
+
+  virtual std::string action_name(){return "TwoFlavourRatioLeftPrecPseudoFermionAction";}
+
+  virtual std::string LogParameters(){
+    std::stringstream sstream;
+    sstream << GridLogMessage << "["<<action_name()<<"] has no parameters" << std::endl;
+    return sstream.str();
+  }
+
+  virtual void refresh(const GaugeField &U, GridSerialRNG &sRNG, GridParallelRNG& pRNG) {
+    // P(phi) = e^{- phi^dag V (MdagM)^-1 Vdag phi} ; phi = Vdag^-1 Mdag eta
+    // e^{-x^2/2 sig^2} => sig^2 = 0.5 ; eta enters with width 1/sqrt(2).
+    RealD scale = std::sqrt(0.5);
+    FermionField eta(NumOp.FermionGrid());
+    gaussian(pRNG,eta);
+    eta = eta * scale;
+    refresh(U,eta);
+  }
+
+  // Deterministic-noise variant (test hook):
+  // after this,  S(U) == norm2(eta)  exactly (to solver tolerance).
+  void refresh(const GaugeField &U, const FermionField &eta) {
+    NumOp.ImportGauge(U);
+    DenOp.ImportGauge(U);
+
+    FermionField tmp(NumOp.FermionGrid());
+    FermionField w  (NumOp.FermionGrid());
+
+    DenOp.Mdag(eta,tmp);           // tmp = Mdag eta
+    w = Zero();
+    HeatbathSolver(tmp,w);         // w   = (VdagV)^-1 Mdag eta
+    NumOp.M(w,Phi);                // Phi = V (VdagV)^-1 Mdag eta = Vdag^-1 Mdag eta
+    std::cout << GridLogMessage << action_name() << " refresh |Phi|^2 = "<< norm2(Phi)<<std::endl;
+  }
+
+  //////////////////////////////////////////////////////
+  // S = phi^dag V (MdagM)^-1 Vdag phi = | Mdag^-1 Vdag phi |^2
+  // ONE adjoint F solve:  Y = V Fdag^-1 Vdag phi = Mdag^-1 Vdag phi
+  //////////////////////////////////////////////////////
+  virtual RealD S(const GaugeField &U) {
+    NumOp.ImportGauge(U);
+    DenOp.ImportGauge(U);
+
+    FermionField b(NumOp.FermionGrid());
+    FermionField z(NumOp.FermionGrid());
+    FermionField Y(NumOp.FermionGrid());
+
+    NumOp.Mdag(Phi,b);             // b = Vdag phi
+    z = Zero();
+    ActionAdjointSolver(b,z);      // Fdag z = b
+    NumOp.M(z,Y);                  // Y = V z = Mdag^-1 Vdag phi
+
+    RealD action = norm2(Y);
+    return action;
+  }
+
+  //////////////////////////////////////////////////////
+  // dS/du = phi^dag dV (MdagM)^-1 Vdag phi
+  //       - phi^dag V (MdagM)^-1 [ Mdag dM + dMdag M ] (MdagM)^-1 Vdag phi
+  //       + phi^dag V (MdagM)^-1 dVdag phi
+  // Identical force insertions to TwoFlavourRatio.h; X and Y from the
+  // left-preconditioned chain (Y harvested from the adjoint solve).
+  //////////////////////////////////////////////////////
+  virtual void deriv(const GaugeField &U,GaugeField & dSdU) {
+    NumOp.ImportGauge(U);
+    DenOp.ImportGauge(U);
+
+    FermionField b(NumOp.FermionGrid());
+    FermionField z(NumOp.FermionGrid());
+    FermionField Y(NumOp.FermionGrid());
+    FermionField s(NumOp.FermionGrid());
+    FermionField X(NumOp.FermionGrid());
+
+    GaugeField force(NumOp.GaugeGrid());
+
+    NumOp.Mdag(Phi,b);             // b = Vdag phi
+    z = Zero();
+    DerivAdjointSolver(b,z);       // Fdag z = b
+    NumOp.M(z,Y);                  // Y = V z = Mdag^-1 Vdag phi   (solve-1 harvest)
+    NumOp.Mdag(Y,s);               // s = Vdag V z
+    X = Zero();
+    DerivForwardSolver(s,X);       // F X = s  ==>  X = (MdagM)^-1 Vdag phi
+
+    // phi^dag V (MdagM)^-1 dVdag phi
+    NumOp.MDeriv(force , X, Phi, DaggerYes);  dSdU  =      force;
+    // phi^dag dV (MdagM)^-1 Vdag phi
+    NumOp.MDeriv(force , Phi, X, DaggerNo );  dSdU  = dSdU+force;
+    //  - phi^dag V (MdagM)^-1 Mdag dM (MdagM)^-1 Vdag phi
+    //  - phi^dag V (MdagM)^-1 dMdag M (MdagM)^-1 Vdag phi
+    DenOp.MDeriv(force, Y, X, DaggerNo );     dSdU  = dSdU-force;
+    DenOp.MDeriv(force, X, Y, DaggerYes);     dSdU  = dSdU-force;
+
+    dSdU *= -1.0;
+  };
+};
+
+NAMESPACE_END(Grid);

Grid/simd/Simd.h

@@ -113,6 +113,14 @@ accelerator_inline RealD    adj(const RealD  & r){ return r; }
 accelerator_inline ComplexD adj(const ComplexD& r){ return(conjugate(r)); }
 accelerator_inline ComplexF adj(const ComplexF& r ){ return(conjugate(r)); }
 
+#if defined(GRID_CUDA) || defined(GRID_HIP)
+//Provide for convenience
+inline std::complex<double> conjugate(const std::complex<double>& r){ return(conj(r)); }
+inline std::complex<float>  conjugate(const std::complex<float>& r) { return(conj(r)); }
+inline std::complex<double> adj(const std::complex<double>& r)      { return(conj(r)); }
+inline std::complex<float>  adj(const std::complex<float>& r)       { return(conj(r)); }
+#endif
+
 accelerator_inline RealF real(const RealF  & r){ return r; }
 accelerator_inline RealD real(const RealD  & r){ return r; }
 accelerator_inline RealF real(const ComplexF  & r){ return r.real(); }

Grid/threads/Accelerator.cc

@@ -3,7 +3,7 @@
 NAMESPACE_BEGIN(Grid);
 int      world_rank; // Use to control world rank for print guarding
 int      acceleratorAbortOnGpuError=1;
-uint32_t accelerator_threads=16;
+uint32_t accelerator_threads=8;
 uint32_t acceleratorThreads(void)       {return accelerator_threads;};
 void     acceleratorThreads(uint32_t t) {accelerator_threads = t;};
 

Grid/threads/Accelerator.h

@@ -96,7 +96,9 @@ void     acceleratorInit(void);
 
 #ifdef GRID_CUDA
 
+NAMESPACE_END(Grid);
 #include <cuda.h>
+NAMESPACE_BEGIN(Grid);
 
 #ifdef __CUDA_ARCH__
 #define GRID_SIMT

MPI_benchmark/compile-command

@@ -1 +1,2 @@
-mpicxx  -fsycl halo_mpi.cc -o halo_mpi
+mpicxx  -fsycl halo_mpi.cc -o halo_mpi
+mpicxx -O2 -std=c++11 io_mpi.cc -o io_mpi

MPI_benchmark/io_aurora.pbs

@@ -0,0 +1,137 @@
+#!/bin/bash
+
+# Standalone MPI-only I/O reproducer on Aurora.  Host only -- no SYCL, no
+# gpu_tile_compact.sh -- so unlike halo_mpi this needs nothing but MPI:
+#
+#   mpicxx -O2 -std=c++11 io_mpi.cc -o io_mpi
+#
+# 12 ranks per node, one per tile, which is what the machine is.  That is a
+# deliberate difference from io_frontier.slurm (8 per node, one per GCD), so
+# read the comparison carefully -- see WHAT IS AND IS NOT COMPARABLE below.
+
+# Run io_aurora_debug.pbs first.  If the cross validation fails there, this
+# scan is 2 hours of 128 nodes producing numbers for a broken file.
+
+#PBS -q prod
+#PBS -l filesystems=flare
+#PBS -l filesystems=home
+#PBS -l select=128
+#PBS -l walltime=02:00:00
+#PBS -A 15479
+
+cd $PBS_O_WORKDIR
+cp $PBS_NODEFILE nodefile
+
+# Only if mpiexec is not already in the environment.  io_mpi is host only
+# and needs no part of the Grid build environment.
+#source ../../sourceme.sh
+
+##########################################################################
+# Environment.  io_mpi has no OpenMP and never touches a GPU, so one thread
+# per rank and a NUMA NIC policy rather than a GPU one.
+#
+# The MPICH_DBG_* variables are deliberately absent: at 1536 ranks they
+# produce gigabytes of log and perturb the timings they would explain.
+# MPICH_MPIIO_STATS/TIMERS are also off here -- they are per collective and
+# 1536 ranks x 6 rungs x 3 reps is unreadable.  Get them from the debug run.
+##########################################################################
+export OMP_NUM_THREADS=1
+export MPICH_CH4_SHM=XPMEM
+export MPICH_OFI_NIC_POLICY=NUMA
+
+##########################################################################
+# WHAT IS AND IS NOT COMPARABLE WITH THE FRONTIER SCAN
+#
+# Held identical at every rung of both scans:
+#   local volume  8.8.32.128 = 262144 sites x 576 B = 151 MB/rank
+#   file view     32768 contiguous runs of 4608 B per rank
+#   aggregation   k=2, row of 16, 8 extents of 18 MB
+# (verified: 4.4.3.1 at 48 ranks and 4.4.3.2 at 96 ranks give exactly the
+# same plan as Frontier's 4.4.2.1 at 32 ranks.)
+#
+# NOT identical, because 12 ranks/node is 1.5x the clients per node:
+#   record size at a given NODE count is 1.5x Frontier's
+#   client count at a given NODE count is 1.5x Frontier's
+#
+# So compare the two machines at equal RANK count (Aurora 4 nodes vs
+# Frontier 6, and so on) if what you want is equal client count and equal
+# record size; compare at equal NODE count if what you want is each machine
+# used as it is meant to be used.  Both are legitimate, they answer
+# different questions, and a table that does not say which one it is
+# reporting is worthless.  The quantity that carries the MPI-IO pathology --
+# per rank local volume and the resulting file view -- is invariant either
+# way, which is the point.
+##########################################################################
+
+##########################################################################
+# WHICH FILESYSTEM.  Point this at Lustre for the like-for-like comparison
+# with Frontier's Orion.  DAOS is a different architecture -- its numbers
+# are interesting but they are NOT a reproduction of the Frontier result,
+# and mixing them into one table would misrepresent both.  Label every set
+# of numbers with the filesystem it came from.
+##########################################################################
+# PROJECT is the flare project DIRECTORY name, which is not the -A account
+# number.  Set it once; the mkdir below fails loudly rather than writing
+# somewhere unintended.
+PROJECT=LatticeQCD_aesp_CNDA
+WORK=/lus/flare/projects/$PROJECT/$USER/iompi.$PBS_JOBID
+mkdir -p $WORK || { echo "cannot create $WORK -- set PROJECT correctly"; exit 1; }
+cd       $WORK
+
+# Match Frontier's default: no explicit striping.  Record what was inherited.
+lfs getstripe -d $WORK 2>/dev/null || echo "(no lfs getstripe -- not Lustre?)"
+
+# The largest rung writes three files of 232 GB, so budget ~700 GB and check
+# the quota before submitting.  Each run unlinks the three files first, so
+# that is peak usage, not cumulative.
+
+NRANKS=12          # one per tile
+
+BIN=$PBS_O_WORKDIR/io_mpi
+ARGS="--target 4194304 --reps 3"
+
+run () {   # run <nodes> <grid> <mpi> <comment> [extra args...]
+  local nodes=$1 gr=$2 mp=$3 note=$4
+  local ntot=$(( nodes * NRANKS ))
+  shift 4
+  echo
+  echo "==================================================================="
+  echo "=== nodes=$nodes ranks=$ntot grid=$gr mpi=$mp   $note"
+  echo "=== extra: $@"
+  echo "==================================================================="
+  mpiexec -np $ntot -ppn $NRANKS -envall $BIN --grid $gr --mpi $mp $ARGS "$@"
+  echo "=== exit $?"
+}
+
+#####################################################################
+# Phase 0.  Correctness.  Both count branches of MPI_Alltoallv are
+# covered; the labels below were checked, not assumed.  The whole-file
+# crc32 is serial, so keep these small.
+#####################################################################
+run 1 16.16.16.24  2.2.1.3  "correctness, UNIFORM counts, row of 4"      --reps 0 --serial-crc
+run 2 24.12.8.8    3.2.2.2  "correctness, NON-UNIFORM counts, row of 12" --reps 0 --serial-crc
+run 4 16.16.32.24  2.2.4.3  "correctness, NON-UNIFORM, non-zero offset"  --reps 0 --serial-crc --offset 1024
+
+#####################################################################
+# Phase 1.  Weak scan at 151 MB/rank.  Identical plan at every rung:
+# k=2, row of 16, 8 extents of 18 MB, 32768 runs of 4608 B in the view.
+#####################################################################
+#          nodes  global lattice    mpi        record
+run    4     32.32.96.128    4.4.3.1   "7.2 GB"    --no-validate
+run    8     32.32.96.256    4.4.3.2   "14.5 GB"   --no-validate
+run   16     32.32.96.512    4.4.3.4   "29.0 GB"   --no-validate
+run   32     32.32.192.512   4.4.6.4   "58.0 GB"   --no-validate
+run   64     32.32.192.1024  4.4.6.8   "116.0 GB"  --no-validate
+run  128     32.32.384.1024  4.4.12.8  "231.9 GB"  --no-validate
+
+#####################################################################
+# Phase 2.  The three questions a reviewer asks immediately.
+#####################################################################
+run  128     32.32.384.1024  4.4.12.8  "231.9 GB, durable"  --no-validate --fsync --drop-cache
+run  128     32.32.384.1024  4.4.12.8  "231.9 GB, cb hints" --no-validate \
+     --hints romio_cb_write=enable,romio_cb_read=enable,cb_nodes=128,cb_buffer_size=16777216
+run  128     32.32.384.1024  4.4.12.8  "231.9 GB, mem subarray" --no-validate --mem-subarray
+
+echo
+echo "=== done.  Files left in $WORK"
+ls -l $WORK

MPI_benchmark/io_frontier.slurm

@@ -0,0 +1,109 @@
+#!/bin/bash -l
+
+# Standalone MPI-only I/O reproducer on Frontier.  No Grid, no accelerator,
+# so no GCD/NUMA wrapper is needed -- the point of the exercise is that this
+# depends on nothing but an MPI installation and a filesystem.
+#
+#   mpicxx -O2 -std=c++11 io_mpi.cc -o io_mpi
+#
+# Weak scan: the local volume, and therefore the file view structure, is held
+# identical at every rung and only the number of Lustre clients changes:
+#
+#   local volume  8.8.32.128 = 262144 sites x 576 B = 151 MB/rank
+#   file view     32768 contiguous runs of 4608 B per rank, at every rung
+#   aggregate     k=2, row of 16, 8 extents of 18 MB, at every rung
+#
+# so any change in the relative bandwidth of the two lexicographic paths is a
+# property of the client count alone.
+#
+# The PERF lines are MiB/s (bytes/1024/1024/s), which is what BinaryIO.h
+# computes for lastPerf.mbytesPerSecond and prints as "MB/s", so the two
+# tools can be compared directly.  Grid's timed region is used here too:
+# barrier, start, [plan build + exchange + I/O], barrier, stop, quoting the
+# boss rank's stopwatch.  --reuse-plan hoists the plan build out, which is
+# how to show it is not where the time goes; do not use it when comparing
+# against Grid's own numbers.
+#
+# io_aurora.pbs runs 12 ranks per node, one per tile, because that is what
+# that machine is.  The per rank local volume and the file view are the same
+# there as here, but the record size and client count at a given NODE count
+# are 1.5x.  See the header of that script before tabulating the two
+# together.
+
+#SBATCH --job-name=ioMPI
+#SBATCH --nodes=128
+#SBATCH --ntasks-per-node=8
+#SBATCH --cpus-per-task=7
+#SBATCH --time=02:00:00
+#SBATCH --account=phy157_dwf
+#SBATCH --exclusive
+#SBATCH --mem=0
+
+module load cce/21.0.0
+module load cpe/26.03
+
+WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/iompi.$SLURM_JOB_ID
+mkdir -p $WORK
+cd       $WORK
+
+# Do NOT stripe by default.  Wide striping is what rescues the collective at
+# scale and costs every other path a factor of 1.2-2.2; the default layout is
+# what a user gets without knowing to ask.  Uncomment to reproduce that
+# interaction, and record which one you ran.
+#lfs setstripe -c -1 -S 8M $WORK
+lfs getstripe -d $WORK
+
+BIN=$SLURM_SUBMIT_DIR/io_mpi
+ARGS="--target 4194304 --reps 3"
+
+# ROMIO's own view of what it did.  Verbose, but the first thing anyone
+# reading the report will ask for.
+# export MPICH_MPIIO_STATS=1
+# export MPICH_MPIIO_TIMERS=1
+
+run () {   # run <nodes> <grid> <mpi> <comment> [extra args...]
+  local nodes=$1 gr=$2 mp=$3 note=$4
+  local nranks=$(( nodes * 8 ))
+  shift 4
+  echo
+  echo "==================================================================="
+  echo "=== nodes=$nodes ranks=$nranks grid=$gr mpi=$mp   $note"
+  echo "=== extra: $@"
+  echo "==================================================================="
+  srun -N$nodes -n$nranks --ntasks-per-node=8 $BIN --grid $gr --mpi $mp $ARGS "$@"
+  echo "=== exit $?"
+}
+
+#####################################################################
+# Phase 0.  Correctness, including the non-uniform Alltoallv branch
+# (odd process factor in an un-split dimension).  Small, and the
+# whole-file crc32 is serial, so keep the volume down here.
+#####################################################################
+run 1 12.12.8.8   2.2.2.1  "correctness, uniform counts"     --reps 0 --serial-crc
+run 3 24.12.8.8   3.2.2.2  "correctness, NON-uniform counts" --reps 0 --serial-crc
+run 4 16.16.16.32 2.2.2.4  "correctness, non-zero offset"    --reps 0 --serial-crc --offset 1024
+
+#####################################################################
+# Phase 1.  Weak scan, 151 MB/rank.  Timing only.
+#####################################################################
+run    4     32.32.64.128    4.4.2.1   "4.8 GB"    --no-validate
+run    8     32.32.64.256    4.4.2.2   "9.7 GB"    --no-validate
+run   16     32.32.64.512    4.4.2.4   "19.3 GB"   --no-validate
+run   32     32.32.128.512   4.4.4.4   "38.6 GB"   --no-validate
+run   64     32.32.128.1024  4.4.4.8   "77.3 GB"   --no-validate
+run  128     32.32.256.1024  4.4.8.8   "154.6 GB"  --no-validate
+
+#####################################################################
+# Phase 2.  Answer the two questions a reviewer will ask immediately.
+#####################################################################
+# Is the gap an artefact of measuring cache rather than the filesystem?
+run  128     32.32.256.1024  4.4.8.8   "154.6 GB, durable"  --no-validate --fsync --drop-cache
+# Does the collective recover if it is given the hints it wants?
+run  128     32.32.256.1024  4.4.8.8   "154.6 GB, cb hints" --no-validate \
+     --hints romio_cb_write=enable,romio_cb_read=enable,cb_nodes=128,cb_buffer_size=16777216
+# Does the degenerate memory subarray matter?
+run  128     32.32.256.1024  4.4.8.8   "154.6 GB, mem subarray" --no-validate --mem-subarray
+
+echo
+echo "=== done.  Files left in $WORK"
+ls -l $WORK

examples/Example_pvdagm.cc

@@ -0,0 +1,803 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid 
+
+    Source file: ./tests/Test_padded_cell.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+RealD FineSmootherShift = 0.1;
+int   FineSmootherOrder = 8;
+int   FineSmootherTol   = 0;
+//RealD CoarseSmootherShift = 0.1;
+//int   CoarseSmootherOrder = 8;
+//int   CoarseSmootherTol   = 0;
+RealD CoarseSolverShift   = 0.002;
+RealD CoarseSolverTol     = 0.03;
+int   CoarseSolverOrder   = 200;
+int   CoarseMmax          = 20;   // coarse GCR restart length (was hardcoded 20)
+RealD mass=0.00078;
+void ParseEnvironment(void)
+{
+    
+  if(getenv("MASS") ) mass = atof(getenv("MASS"));
+  if(getenv("FineSmootherShift"))   FineSmootherShift   = atof(getenv("FineSmootherShift"));
+  if(getenv("FineSmootherOrder"))   FineSmootherOrder   = atoi(getenv("FineSmootherOrder"));
+  if(getenv("CoarseSolverShift"))   CoarseSolverShift   = atof(getenv("CoarseSolverShift"));
+  if(getenv("CoarseSolverTol"))     CoarseSolverTol     = atof(getenv("CoarseSolverTol"));
+  if(getenv("CoarseSolverOrder"))   CoarseSolverOrder   = atoi(getenv("CoarseSolverOrder"));
+  if(getenv("CoarseMmax"))          CoarseMmax          = atoi(getenv("CoarseMmax"));
+  if(getenv("DiagInvPrec"))
+  {
+    std::cout << GridLogMessage << "WARNING: DiagInvPrec option REMOVED (diagonal-inverse preconditioning wrecks fine->coarse null-vector inheritance); IGNORED" << std::endl;
+  }
+
+  //  if(getenv("CoarseSmootherShift")) CoarseSmootherShift = atof(getenv("CoarseSmootherShift"));
+  //  if(getenv("CoarseSmootherOrder")) CoarseSmootherOrder = atoi(getenv("CoarseSmootherOrder"));
+  
+  std::cout << GridLogMessage << "PARAM: FineSmootherShift   "<<FineSmootherShift<<std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherOrder   "<<FineSmootherOrder<<std::endl;
+  //  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift "<<CoarseSmootherShift<<std::endl;
+  //  std::cout << GridLogMessage << "PARAM: CoarseSmootherOrder "<<CoarseSmootherOrder<<std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverShift   "<<CoarseSolverShift<<std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverTol     "<<CoarseSolverTol<<std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder   "<<CoarseSolverOrder<<std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseMmax          "<<CoarseMmax<<std::endl;
+
+  std::cout << GridLogMessage << "PARAM: MASS "<<mass<<std::endl;
+}
+
+template <class T> void readFile(T& out, std::string const fname){  
+  #ifdef HAVE_LIME
+    // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
+    std::cout << Grid::GridLogMessage << "Reads at: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    // Grid::ScidacReader SR(out.Grid()->IsBoss());
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    //    std::cout << GridLogMessage<< "Op: PVdag M "<<std::endl;
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    //    std::cout << GridLogMessage<<"AdjOp: Mdag PV "<<std::endl;
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+
+  void clear() {
+    nApp = 0;
+    nAppDag = 0;
+  }
+
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    //    std::cout <<GridLogMessage<< "HermOp: Mdag PV PVdag M"<<std::endl;
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
+  }
+};
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    //    std::cout <<GridLogMessage<< "Op: PVdag M "<<std::endl;
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    //    std::cout <<GridLogMessage<< "AdjOp: Mdag PV "<<std::endl;
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    //    std::cout << GridLogMessage<<"HermOp: PVdag M Mdag PV "<<std::endl;
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
+  }
+};
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  RealD shift;
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV) , nApp(0), nAppDag(0){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    //    std::cout << "Op: PVdag M "<<std::endl;
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    //    std::cout << "AdjOp: Mdag PV "<<std::endl;
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    nAppDag++;
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
+  void HermOp(const Field &in, Field &out){
+    //    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+  void clear() {
+    nApp = 0;
+    nAppDag = 0;
+  }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of ShiftedPVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of ShiftedPVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+};
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  std::string name;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   std::string _name = std::string("unnamed"))
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      name(_name),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out) 
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    //    auto CoarseGrid = _CoarseOperator.Grid();
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    std::cout<<GridLogMessage << "Calling PreSmoother " <<std::endl;
+
+    //    std::cout<<GridLogMessage << "Calling PreSmoother input residual "<<norm2(in) <<std::endl;
+    double t;
+    // Fine Smoother
+    //    out = in;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    // Update the residual
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);   
+    //    std::cout<<GridLogMessage <<"Residual-1 now " <<norm2(vec1)<<std::endl;
+
+    // Fine to Coarse 
+    t=-usecond();
+    _Aggregates.ProjectToSubspace  (Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    // Coarse correction
+    t=-usecond();
+    Csol = Zero();
+    _CoarseSolve(Csrc,Csol);
+    //Csol=Zero();
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    // Coarse to Fine
+    t=-usecond();  
+    //    _CoarseOperator.PromoteFromSubspace(_Aggregates,Csol,vec1);
+    _Aggregates.PromoteFromSubspace(Csol,vec1); 
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    // Residual
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);  
+    //    std::cout<<GridLogMessage <<"Residual-2 now " <<norm2(vec1)<<std::endl;
+
+    // Fine Smoother
+    t=-usecond();
+    //    vec2=vec1;
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add( out,out,vec2);
+    std::cout<<GridLogMessage << "Done " <<std::endl;
+  }
+};
+
+template<class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d, 
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t PVdagM,
+  ShiftedPVdagM_t ShiftedPVdagM,
+  // std::vector<LatticeFermion> subspace
+  Subspace AggregatesPD
+) {
+
+  // typedef Aggregation<vSpinColourVector,vTComplex,nbasis> Subspace;
+  // typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
+  // typedef LittleDiracOperator::CoarseVector CoarseVector;
+  ParseEnvironment();
+
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  int nbasis = subspace.size();
+  const int cb = 0 ;
+
+  LatticeFermion err(FGrid);
+  LatticeFermion prom(FGrid);
+  LatticeFermion tmp(FGrid);
+
+  CoarseVector c_src (Coarse5d);
+  CoarseVector c_res (Coarse5d);
+  CoarseVector c_proj(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+  // typedef MGPreconditioner<vSpinColourVector,  vTComplex,nbasis> TwoLevelMG;
+  TrivialPrecon<CoarseVector> simple;
+  TrivialPrecon<LatticeFermionD> simple_fine;
+
+  // Subspace AggregatesPD(Coarse5d,FGrid,cb);
+
+  // Orthonormalize subspace and compute nulliness
+
+  ShiftedPVdagM.shift = CoarseSolverShift;
+  int nonherm = 0;
+  LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d,nonherm);
+  LittleDiracOpPV.CoarsenOperator(ShiftedPVdagM, AggregatesPD);
+  ShiftedPVdagM.shift = FineSmootherShift;
+
+  std::cout<<GridLogMessage<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<std::endl;
+  std::cout<<GridLogMessage<<"Testing coarsened operator "<<std::endl;
+
+  c_src = one;  // 1 in every element for vector 1.
+  blockPromote(c_src,err,subspace);
+
+  prom=Zero();
+  for(int b=0;b<nbasis;b++){
+    prom=prom+subspace[b];
+  }
+  err=err-prom; 
+  std::cout<<GridLogMessage<<"Promoted back from subspace: err "<<norm2(err)<<std::endl;
+  std::cout<<GridLogMessage<<"c_src "<<norm2(c_src)<<std::endl;
+  std::cout<<GridLogMessage<<"prom  "<<norm2(prom)<<std::endl;
+
+  // PVdagM.Op(prom,tmp);
+  // blockProject(c_proj,tmp,subspace);
+  // std::cout<<GridLogMessage<<" Called Big Dirac Op "<<norm2(tmp)<<std::endl;
+
+  // LittleDiracOpPV.M(c_src,c_res);
+  // std::cout<<GridLogMessage<<" Called Little Dirac Op c_src "<< norm2(c_src) << "  c_res "<< norm2(c_res) <<std::endl;
+
+  // std::cout<<GridLogMessage<<"Little dop : "<<norm2(c_res)<<std::endl;
+  // //  std::cout<<GridLogMessage<<" Little "<< c_res<<std::endl;
+  // std::cout<<GridLogMessage<<"Big dop in subspace : "<<norm2(c_proj)<<std::endl;
+  // //  std::cout<<GridLogMessage<<" Big "<< c_proj<<std::endl;
+  // c_proj = c_proj - c_res;
+  // std::cout<<GridLogMessage<<" ldop error: "<<norm2(c_proj)<<std::endl;
+  // //  std::cout<<GridLogMessage<<" error "<< c_proj<<std::endl;
+
+  ///////////////////////////////////////
+  // Coarse grid solver test
+  ///////////////////////////////////////
+
+  std::cout<<GridLogMessage<<"******************* "<<std::endl;
+  std::cout<<GridLogMessage<<" Coarse Grid Solve -- Level 2 "<<std::endl;
+  std::cout<<GridLogMessage<<"******************* "<<std::endl;
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+  // DiagonalInverse preconditioning REMOVED (library support withdrawn: it
+  // wrecks the collinearity that makes fine->coarse null-vector inheritance
+  // free).  TrivialPrecon reproduces the former DiagInvPrec=0 path exactly.
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>  L2PGCR(CoarseSolverTol, (CoarseSolverOrder+CoarseMmax-1)/CoarseMmax, LinOpCoarse,simple,CoarseMmax,CoarseMmax);
+  L2PGCR.SetZeroGuess(1);              // callers zero Csol / c_res
+  L2PGCR.Level(2);
+  L2PGCR.Name("Couter");
+  c_res=Zero();
+  L2PGCR(c_src,c_res);
+
+
+  ////////////////////////////////////////
+  // Fine grid smoother
+  ////////////////////////////////////////
+  //  NonHermitianLinearOperator<PVdagM_t,LatticeFermionD> LinOpSmooth(PVdagM);
+  
+  //  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.05,1,ShiftedPVdagM,simple_fine,8,8);
+  // Force 10 iters exactly, no early termination
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(FineSmootherTol,1,
+									    ShiftedPVdagM,simple_fine,
+									    FineSmootherOrder,FineSmootherOrder);
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+  SmootherGCR.SetZeroGuess(1);         // pre/post slots + direct call all zero their guess
+
+  f_src = one;  // 1 in every element for vector 1.
+  f_res=Zero();
+  SmootherGCR(f_src,f_res);
+
+  TwoLevelMG TwoLevelPrecon(AggregatesPD,
+			    PVdagM,
+			    simple_fine,
+			    SmootherGCR,
+			    LinOpCoarse,
+			    L2PGCR,
+			    "PVdagM");
+  
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,TwoLevelPrecon,32,32);
+  L1PGCR.SetZeroGuess(1);              // f_res=Zero() before the solve
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  std::cout<<GridLogMessage<<"******************* "<<std::endl;
+  std::cout<<GridLogMessage<<" Running Multi Grid Solver "<<std::endl;
+  std::cout<<GridLogMessage<<"******************* "<<std::endl;
+  f_res=Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Fine Grid Smoother -- Level 2 operator uses: " << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+  ShiftedPVdagM.getApplications();
+  ShiftedPVdagM.clear();
+
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  // TODO read in more parameters: nbasis, GCR iters, smoother order, m
+  // Might be impossible because nbasis needs to be a constant to be a template parameter
+  // Usage : $ ./Example_pvdagm <nbasis> <smooth> <outerIters> <m>
+  // std::string nbasisStr  = argv[1];
+  // std::string smoothStr  = argv[2];
+  // std::string outerStr   = argv[3];
+  // std::string mStr       = argv[4];
+  // int nbasis = std::stoi(nbasisStr);
+  // int smooth = std::stoi(smoothStr);
+
+  const int Ls=24;
+  RealD M5=1.8;
+
+  
+  //  const int nbasis = 40;
+  const int nbasis = 60;
+  
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", running Mobius kernel with b=1.5, c=0.5" << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Construct a coarsened grid
+  // Coordinate clatt = GridDefaultLatt();
+  Coordinate clatt = lat_size;
+  Coordinate Block({4,4,4,4});
+  std::cout << GridLogMessage << "Lattice size: " << lat_size << std::endl;
+  for(int d=0;d<clatt.size();d++){
+    clatt[d] = lat_size[d]/Block[d];
+  }
+
+  std::cout << GridLogMessage << "constructing coarse grid" << std::endl;
+  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  std::vector<int> cseeds({5,6,7,8});
+  GridParallelRNG          RNG5(FGrid);   RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG          RNG4(UGrid);   RNG4.SeedFixedIntegers(seeds4);
+  GridParallelRNG          CRNG(Coarse5d);CRNG.SeedFixedIntegers(cseeds);
+
+  LatticeFermion    src(FGrid); random(RNG5,src);
+  LatticeFermion result(FGrid); result=Zero();
+  LatticeFermion    ref(FGrid); ref=Zero();
+  LatticeFermion    tmp(FGrid);
+  LatticeFermion    err(FGrid);
+  LatticeGaugeField Umu(UGrid);
+
+
+  std::cout << GridLogMessage << "Reading in gauge field" << std::endl;
+  FieldMetaData header;
+  // std::string file("/sdcc/u/poare/PETSc-Grid/ckpoint_lat.4000");
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  /*
+
+  // DWF, m=0.01
+  // std::string eigenPath = "/hpcgpfs01/work/lqcd/staging/RBC/ckpoint_lat.4000/ks_evecs/PVdagM_Nm80_Nk40_Niter5000_337342/";
+
+  // DWF, m=0.001
+  // std::string eigenPath = "/hpcgpfs01/work/lqcd/staging/RBC/ckpoint_lat.4000/ks_evecs/PVdagM_Nm80_Nk40_Niter5000_m0p001_339143/";
+
+  // Mobius, m=0.001
+  // std::string eigenPath = "/hpcgpfs01/work/lqcd/staging/RBC/ckpoint_lat.4000/ks_evecs/PVdagM_Nm80_Nk40_Niter5000_346851/";
+
+  // Frontier path
+  std::string eigenPath = "/ccs/home/poare/lqcd/multigrid/spectra/ckpoint_lat.1000/...";
+  
+
+  std::cout << GridLogMessage << "Loading eigenvalues" << std::endl;
+  std::ifstream evalFile(eigenPath + "evals.txt");
+  std::string str;
+  std::vector<ComplexD> evals;
+  while (std::getline(evalFile, str)) {
+      std::cout << GridLogMessage << "Reading line: " << str << std::endl;
+      int i1 = str.find("(") + 1;
+      int i2 = str.find(",") + 1;
+      int i3 = str.find(")");
+      std::cout << "i1,i2,i3 = " << i1 << "," << i2 << "," << i3 << std::endl;
+      std::string reStr = str.substr(i1, i2 - i1);
+      std::string imStr = str.substr(i2, i3 - i2);
+      std::cout << GridLogMessage << "Parsed re = " << reStr << " and im = " << imStr << std::endl;
+      // ComplexD z (std::stof(reStr), std::stof(imStr));
+      ComplexD z (std::stod(reStr), std::stod(imStr));
+      evals.push_back(z);
+  }
+  std::cout << GridLogMessage << "Eigenvalues: " << evals << std::endl;
+
+  int Nevecs = 20;
+  std::vector<LatticeFermion> evecs;
+  LatticeFermion evec (FGrid);
+  for (int i = 0; i < Nevecs; i++) {
+    std::string evecPath = eigenPath + "evec" + std::to_string(i);
+    readFile(evec, evecPath);
+    evecs.push_back(evec);
+  }
+  std::cout << GridLogMessage << "Evecs loaded" << std::endl;
+
+  */
+  // TODO uncomment when evecs are computed!
+
+  // DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
+  // DomainWallFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5);
+
+  // Mobius
+  RealD b=1.5;// Scale factor b+c=2, b-c=1
+  RealD c=0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
+  MobiusFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5,b,c);
+
+  const int cb = 0 ;
+  LatticeFermion prom(FGrid);
+
+  // assert(nbasis <= Nevecs);     // need to have enough evecs
+
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector CoarseVector;
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  std::cout<<GridLogMessage<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<std::endl;
+
+  // typedef PVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> PVdagM_t;
+  // typedef MdagPVLinearOperator<DomainWallFermionD,LatticeFermionD> MdagPV_t;
+  // typedef ShiftedPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD> PVdagM_t;
+  typedef MdagPVLinearOperator<MobiusFermionD,LatticeFermionD> MdagPV_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+
+  PVdagM_t PVdagM(Ddwf,Dpv);
+  MdagPV_t MdagPV(Ddwf,Dpv);
+  //  ShiftedPVdagM_t ShiftedPVdagM(2.0,Ddwf,Dpv); // 355
+  //  ShiftedPVdagM_t ShiftedPVdagM(1.0,Ddwf,Dpv); // 246
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.5,Ddwf,Dpv); // 183
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // 145
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 134
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 127 -- NULL space via inverse iteration
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 57 -- NULL space via inverse iteration; 3 iterations
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // 57 , tighter inversion
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // nbasis 20 -- 49 iters
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // nbasis 20 -- 70 iters; asymmetric 
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.25,Ddwf,Dpv); // 58; Loosen coarse, tighten fine
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 56 ... 
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 51 ...  with 24 vecs
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 31 ...  with 24 vecs and 2^4 blocking
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 43 ...  with 16 vecs and 2^4 blocking, sloppier
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 35  ...  with 20 vecs and 2^4 blocking
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 35  ...  with 20 vecs and 2^4 blocking, looser coarse
+  //  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv); // 64  ...  with 20 vecs, Christoph setup, and 2^4 blocking, looser coarse
+
+  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv); // 
+
+  // Run power method on HOA??
+  PowerMethod<LatticeFermion>       PM;
+
+  CoarseVector c_src (Coarse5d);
+  CoarseVector c_res (Coarse5d);
+  CoarseVector c_proj(Coarse5d);
+  Complex one(1.0);
+
+  std::vector<LatticeFermion> subspace(nbasis,FGrid);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+  typedef MGPreconditioner<vSpinColourVector,  vTComplex,nbasis> TwoLevelMG;
+  TrivialPrecon<CoarseVector> simple;
+  TrivialPrecon<LatticeFermionD> simple_fine;
+
+  // Warning: This routine calls PVdagM.Op, not PVdagM.HermOp
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis> Subspace;
+
+  // Breeds right singular vectors with call to HermOp (V)
+  // int chebyOrd = 500;
+  // V.CreateSubspaceChebyshev(RNG5,PVdagM,
+	// 		    nbasis,
+	// 		    4000.0,0.003,
+	// 		    chebyOrd);
+  // AggregatesPD.CreateSubspaceChebyshev(RNG5,
+	// 			       PVdagM,
+	// 			       nbasis,
+	// 			       4000.0,
+	// 			       0.003,
+	// 			       chebyOrd);
+  
+  // Subspace testing (uncomment blocks when needed)
+
+  // - nbasis = 20, m=0.01, 35 outer iterations
+  // - nbasis = 40, m=0.01, 23 outer iterations
+  std::cout << GridLogMessage << "*** GCR setup ***" << std::endl;
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, matching what
+    // CreateSubspaceGCR applies to generated subspaces (Aggregates.h:196), so a
+    // reloaded file ends in the same state.  This replaces the block
+    // Orthogonalise() previously called here -- that is redundant (CoarsenOperator
+    // block-GS's the subspace internally) and would leave a loaded file block-
+    // orthonormal while a generated one is globally orthonormal.  Global GS is
+    // span-preserving, so the coarse operator is unchanged.
+    AggregatesGCR.GlobalOrthonormalise();
+    std::cout << GridLogMessage << "Subspace loaded and globally orthonormalised." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+  
+  std::cout << GridLogMessage << "Basis construction operator uses: " << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  std::cout << GridLogMessage << "Calling runMG " << std::endl;
+  runMG<PVdagM_t, ShiftedPVdagM_t, Subspace, LittleDiracOperator, CoarseVector, TwoLevelMG>(
+    FGrid, 
+    Coarse5d, 
+    geom, 
+    PVdagM, 
+    ShiftedPVdagM, 
+    AggregatesGCR
+  );
+
+  //////////////////////////////////
+  // Standard CG
+  //////////////////////////////////
+#if 0
+  {
+  std::cout << "**************************************"<<std::endl;
+  std::cout << "Calling red black CG"<<std::endl;
+  std::cout << "**************************************"<<std::endl;
+    ConjugateGradient<LatticeFermionD>  CGfine(1.0e-8,30000,false);
+    SchurDiagMooeeOperator<MobiusFermionD, LatticeFermion> HermOpEO(Ddwf);
+      
+    LatticeFermion result(FrbGrid); result=Zero();
+    LatticeFermion    src(FrbGrid); random(RNG5,src);
+    result=Zero();
+
+    CGfine(HermOpEO, src, result);
+  }
+  {
+    std::cout << "**************************************"<<std::endl;
+    std::cout << "Calling MdagM CG"<<std::endl;
+    std::cout << "**************************************"<<std::endl;
+      
+    LatticeFermion result(FGrid); result=Zero();
+    LatticeFermion    src(FGrid); random(RNG5,src);
+    result=Zero();
+
+    MdagMLinearOperator<MobiusFermionD, LatticeFermionD> HermOp(Ddwf);
+    ConjugateGradient<LatticeFermionD>  CGfine(1.0e-8,100000,false);
+    CGfine(HermOp, src, result);
+  }
+  {
+    std::cout << "**************************************"<<std::endl;
+    std::cout << "Calling PVdagM GCR"<<std::endl;
+    std::cout << "**************************************"<<std::endl;
+
+    LatticeFermion result(FGrid); result=Zero();
+    LatticeFermion    src(FGrid); random(RNG5,src);
+    result=Zero();
+    PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> GCR(1.0e-8,3000,PVdagM,simple_fine,50,50);
+    GCR.Name("Fbaseline");
+    GCR.SetZeroGuess(1);               // result=Zero() above
+    GCR(src,result);
+  }
+#endif  
+
+  
+  std::cout<<GridLogMessage<<std::endl;
+  std::cout<<GridLogMessage << "Done "<< std::endl;
+
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_3level.cc

@@ -0,0 +1,699 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_3level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
+// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
+template<class Field>
+class LuscherGuesser : public LinearFunction<Field> {
+  const std::vector<Field> &psi;
+  Eigen::MatrixXcd          C_inv;
+public:
+  using LinearFunction<Field>::operator();
+  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
+    : psi(psi_), C_inv(Cinv_) {}
+  virtual void operator()(const Field &src, Field &guess) {
+    int N = psi.size();
+    Eigen::VectorXcd b(N);
+    for (int t = 0; t < N; t++)
+      b(t) = TensorRemove(innerProduct(psi[t], src));
+    Eigen::VectorXcd c = C_inv * b;
+    guess = Zero();
+    for (int s = 0; s < N; s++)
+      guess += ComplexD(c(s)) * psi[s];
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_)
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  L2PGCR_baseline.Name("Cbaseline");
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Lüscher deflation guesser for L3PGCR.
+  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
+  //         CoarseCoarseVector space — these cover the zero-momentum
+  //         component of the near-null space of LinOpCC.
+  // Step 2: breed Nextra additional null vectors directly on LinOpCC
+  //         using GCR with random sources — these pick up near-null
+  //         modes at all spatial frequencies not spanned by step 1.
+  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
+  //         full augmented basis and invert directly via Eigen LU.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+
+  {
+    int Nextra = nbasis;  // breed as many extra as we have promoted ones
+    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
+    GridParallelRNG RNG_CC(CoarseCoarse5d);
+    RNG_CC.SeedFixedIntegers({11,13,17,19});
+    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
+    CoarseCoarseVector tmp(CoarseCoarse5d);
+    for (int k = 0; k < Nextra; k++) {
+      CoarseCoarseVector src(CoarseCoarse5d);
+      gaussian(RNG_CC, src);
+      tmp = Zero();
+      nullGCR(src, tmp);
+      psi_cc.push_back(tmp);
+    }
+    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
+              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
+  }
+
+  const int Naug = psi_cc.size();
+  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
+  {
+    CoarseCoarseVector Acc(CoarseCoarse5d);
+    for (int l = 0; l < Naug; l++) {
+      LinOpCC.Op(psi_cc[l], Acc);
+      for (int k = 0; k < Naug; k++)
+        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
+    }
+  }
+  {
+    RealD normCcc      = Ccc.norm();
+    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
+  }
+  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
+  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2 solver: plain GCR, no further coarsening
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3PGCR(1.0e-1,200,LinOpCC,simpleCC,16,16);
+  L3PGCR.Level(3);
+  L3PGCR.Name("CCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.01;
+  int   coarse_smoother_nstep = 12;
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+                       LinOpCoarse,
+                       simpleC,            // no pre-smoother (matches fine-grid setup)
+                       CoarseSmootherGCR,  // post-smoother: 12 GCR steps
+                       LinOpCC,
+                       L3PGCR,
+                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  L2MGsolver.Name("Couter");
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full three-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+                               PVdagM,
+                               simple_fine,
+                               SmootherGCR,
+                               LinOpCoarse,
+                               L2MGsolver,
+                               simpleC);
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, matching CreateSubspaceGCR
+    // (Aggregates.h:196), in case the cached file predates it.  Span-preserving
+    // and globally orthonormal -- NOT the block Orthogonalise() below, which would
+    // defeat the raw-null discipline (runMG promotes the RAW subspace to build L2;
+    // block-GS here -> psi_coarse = e_k).  The raw copy in runMG happens AFTER this.
+    AggregatesGCR.GlobalOrthonormalise();
+    //    AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_3level_madj.cc

@@ -0,0 +1,625 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_3level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+  std::string      _name;
+  
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_,
+		   std::string name = std::string("unnamed"))
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      _name(name),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << _name << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << _name << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << _name <<"PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2 solver: plain GCR, no further coarsening
+  //////////////////////////////////////////////////////////////////////
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+  // L3PGCR is an inner solver inside the L1→2 V-cycle; does not need to converge
+  // to fine-grid precision. Loose tolerance (3e-2) and large restart (64) to allow
+  // the Krylov space to span enough of the near-null spectrum of LinOpCC per cycle.
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3PGCR(1.0e-4,5,LinOpCC,simpleCC,64,64);
+  L3PGCR.Level(3);
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.0;
+  int   coarse_smoother_nstep = 8;
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.0,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.Level(2);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+		      LinOpCoarse,
+		      simpleC,            // no pre-smoother (matches fine-grid setup)
+		      CoarseSmootherGCR,  // post-smoother: 12 GCR steps
+		      LinOpCC,
+		      L3PGCR,
+		      simpleCC,
+		      std::string("LinOpC"));
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full three-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.00,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.Level(1);
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+			      PVdagM,
+			      simple_fine,
+			      SmootherGCR,
+			      LinOpCoarse,
+			      L2MGsolver,
+			      simpleC,
+			      std::string("PVdagM"));
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+
+  RealD madj = 1.0;
+  if ( getenv("MADJ") ) madj=atof(getenv("MADJ"));
+  std::cout << "PV mass set to "<<madj<<std::endl;
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,madj, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.00,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Re-orthogonalise after loading to ensure block-GS condition holds.
+    //    AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded and re-orthogonalised." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_4level.cc

@@ -0,0 +1,798 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_3level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
+// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
+template<class Field>
+class LuscherGuesser : public LinearFunction<Field> {
+  const std::vector<Field> &psi;
+  Eigen::MatrixXcd          C_inv;
+public:
+  using LinearFunction<Field>::operator();
+  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
+    : psi(psi_), C_inv(Cinv_) {}
+  virtual void operator()(const Field &src, Field &guess) {
+    int N = psi.size();
+    Eigen::VectorXcd b(N);
+    for (int t = 0; t < N; t++)
+      b(t) = TensorRemove(innerProduct(psi[t], src));
+    Eigen::VectorXcd c = C_inv * b;
+    guess = Zero();
+    for (int s = 0; s < N; s++)
+      guess += ComplexD(c(s)) * psi[s];
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_)
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  GridCartesian *CoarseCoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  L2PGCR_baseline.Name("Cbaseline");
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Lüscher deflation guesser for L3PGCR.
+  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
+  //         CoarseCoarseVector space — these cover the zero-momentum
+  //         component of the near-null space of LinOpCC.
+  // Step 2: breed Nextra additional null vectors directly on LinOpCC
+  //         using GCR with random sources — these pick up near-null
+  //         modes at all spatial frequencies not spanned by step 1.
+  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
+  //         full augmented basis and invert directly via Eigen LU.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+
+  {
+    int Nextra = nbasis;  // breed as many extra as we have promoted ones
+    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
+    GridParallelRNG RNG_CC(CoarseCoarse5d);
+    RNG_CC.SeedFixedIntegers({11,13,17,19});
+    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
+    CoarseCoarseVector tmp(CoarseCoarse5d);
+    for (int k = 0; k < Nextra; k++) {
+      CoarseCoarseVector src(CoarseCoarse5d);
+      gaussian(RNG_CC, src);
+      tmp = Zero();
+      nullGCR(src, tmp);
+      psi_cc.push_back(tmp);
+    }
+    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
+              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
+  }
+
+  const int Naug = psi_cc.size();
+  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
+  {
+    CoarseCoarseVector Acc(CoarseCoarse5d);
+    for (int l = 0; l < Naug; l++) {
+      LinOpCC.Op(psi_cc[l], Acc);
+      for (int k = 0; k < Naug; k++)
+        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
+    }
+  }
+  {
+    RealD normCcc      = Ccc.norm();
+    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
+  }
+  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
+  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3: coarsen LinOpCC using the RAW promoted psi_cc as aggregation
+  // to build the Level 4 (coarse-coarse-coarse) operator.
+  //   psi_cc[0..nbasis-1] are the coarse-coarse near-null vectors, projected
+  //   from the RAW psi_coarse (themselves projected from the RAW fine null
+  //   vectors) -- the pre-block-GS chain the whole construction depends on.
+  //   CoarsenOperator block-GS orthogonalises AggregatesL3.subspace IN PLACE,
+  //   so assign COPIES of psi_cc and keep psi_cc itself raw.
+  //
+  // Tensor depth deepens once more: innerProduct(CoarseCoarseSiteObj,...) returns
+  // iScalar<vTTComplex>, so CComplex for the L2→L3 level is iScalar<iScalar<vTComplex>>.
+  //////////////////////////////////////////////////////////////////////
+  typedef typename CoarseCoarseVector::vector_object                     CoarseCoarseSiteObj;
+  typedef iScalar<vTTComplex>                                            vTTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseCoarseSiteObj,vTTTComplex,NB>     LittleDiracOperatorL3;
+  typedef typename LittleDiracOperatorL3::CoarseVector                   CoarseCoarseCoarseVector;
+  typedef Aggregation<CoarseCoarseSiteObj,vTTTComplex,NB>               SubspaceL3;
+  typedef MGPreconditioner<CoarseCoarseSiteObj,vTTTComplex,NB>          L2to3MG;
+
+  SubspaceL3 AggregatesL3(CoarseCoarseCoarse5d, CoarseCoarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL3.subspace[k] = psi_cc[k];          // raw promoted; COPY, keeps psi_cc raw
+
+  NextToNearestStencilGeometry5D geom3(CoarseCoarseCoarse5d);
+  LittleDiracOperatorL3 LittleDiracOpL3(geom3, CoarseCoarse5d, CoarseCoarseCoarse5d);
+  LittleDiracOpL3.CoarsenOperator(LinOpCC, AggregatesL3);   // block-GS's AggregatesL3.subspace in place
+
+  NonHermitianLinearOperator<LittleDiracOperatorL3,CoarseCoarseCoarseVector> LinOpCCC(LittleDiracOpL3);
+  TrivialPrecon<CoarseCoarseCoarseVector> simpleCCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 4 bottom solve: GCR on a SHIFTED LinOpCCC.  This is the one level
+  // with no IRS shift, and it is the most non-normal (coarsest) operator, so
+  // the bare bottom GCR wanders in a field of values that wraps the origin and
+  // its iteration count blows out (observed 5..54 iters to hit 0.2).  Solving
+  // (A_ccc + l4_shift) instead slides the FoV off the origin; the correction is
+  // only ever a loose 0.2 approximation anyway, so the detuning is free.
+  // l4_shift defaults to 0.0 => bare LinOpCCC, baseline unchanged until opted in.
+  //////////////////////////////////////////////////////////////////////
+  RealD l4_shift = 0.0;
+  if(getenv("l4_shift")) l4_shift = atof(getenv("l4_shift"));
+  std::cout << GridLogMessage << "PARAM l4_shift = " << l4_shift << std::endl;
+
+  ShiftedLinearOperator<CoarseCoarseCoarseVector> ShiftedLinOpCCC(l4_shift, LinOpCCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector> L4PGCR(1.0e-1,200,ShiftedLinOpCCC,simpleCCC,16,16);
+  L4PGCR.Level(4);
+  L4PGCR.Name("CCCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3 V-cycle: depth-2 SHIFTED smoother on LinOpCC + L4 bottom.
+  // The shift slides the coarse-coarse field of values off the origin so a
+  // 2-step smoother has something to bite on a non-normal operator (IRS idea).
+  //////////////////////////////////////////////////////////////////////
+  RealD cc_smoother_shift = 0.01;
+  int   cc_smoother_nstep = 2;
+  if(getenv("cc_smoother_shift")) cc_smoother_shift = atof(getenv("cc_smoother_shift"));
+  if(getenv("cc_smoother_nstep")) cc_smoother_nstep = atoi(getenv("cc_smoother_nstep"));
+
+  ShiftedLinearOperator<CoarseCoarseVector> ShiftedLinOpCC(cc_smoother_shift, LinOpCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    CoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCC,simpleCC,cc_smoother_nstep,cc_smoother_nstep);
+  CoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
+  CoarseCoarseSmootherGCR.Level(3);
+  CoarseCoarseSmootherGCR.Name("CCsmoother");
+
+  L2to3MG L2to3Precon(AggregatesL3,
+                      LinOpCC,
+                      simpleCC,                  // no pre-smoother
+                      CoarseCoarseSmootherGCR,   // post-smoother: depth-2 shifted GCR
+                      LinOpCCC,
+                      L4PGCR,
+                      simpleCCC);                // trivial guesser at the bottom
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3 (coarse-coarse) solve: GCR preconditioned by the L2→L3 V-cycle.
+  // Replaces the plain L3PGCR of the 3-level build -- the coarse-coarse level
+  // is now smoothed shallowly and recursed rather than solved deeply.
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3MGsolver(1.0e-1,200,LinOpCC,L2to3Precon,16,16);
+  L3MGsolver.Level(3);
+  L3MGsolver.Name("CCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.01;
+  int   coarse_smoother_nstep = 2;   // depth-2 smoother on the coarse level
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+                       LinOpCoarse,
+                       simpleC,            // no pre-smoother (matches fine-grid setup)
+                       CoarseSmootherGCR,  // post-smoother: depth-2 shifted GCR
+                       LinOpCC,
+                       L3MGsolver,         // coarse-coarse solve is now the L2→L3 V-cycle
+                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  L2MGsolver.Name("Couter");
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full three-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+                               PVdagM,
+                               simple_fine,
+                               SmootherGCR,
+                               LinOpCoarse,
+                               L2MGsolver,
+                               simpleC);
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // Level 3 coarse-coarse-coarse grid: block clatt2 = {12,12,8,16} -> {6,12,8,8}.
+  // GEOMETRY (mpi 3.6.4.4, Nsimd=4 => SIMD layout {1,1,2,2}, factor 2 on z and t):
+  //   every grid needs z-local and t-local EVEN.  clatt2-local is {4,2,2,4}, so
+  //   z-local=2 is already at its minimum even value and CANNOT be blocked (2->1
+  //   is odd and trips the SIMD assertion); y-local=2 would go to 1 (degenerate).
+  //   Only x and t have room, so block {2,1,1,2}: clatt3 {6,12,8,8}, L4-local
+  //   {2,2,2,2} -- all dims even and >=2.  z stays unblocked by construction.
+  Coordinate clatt3 = clatt2;
+  clatt3[0] /= 2;   // x: 12 -> 6   (x-local 4 -> 2)
+  // clatt3[1] (y) unblocked: y-local 2 -> blocking gives 1 (degenerate)
+  // clatt3[2] (z) unblocked: z-local 2 is SIMD-pinned even, cannot halve
+  clatt3[3] /= 2;   // t: 16 -> 8   (t-local 4 -> 2)
+  std::cout << GridLogMessage << "Level 3 coarse-coarse-coarse lattice: " << clatt3 << std::endl;
+
+  GridCartesian *CoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt3, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, in case the cached file
+    // predates the GlobalOrthonormalise() that CreateSubspaceGCR now applies
+    // (Aggregates.h:196).  It is span-preserving and makes the vectors globally
+    // orthonormal -- it is NOT the block Orthogonalise() below, so it does NOT
+    // cause the psi_coarse->e_k trap.  The RAW subspace copy in runMG happens
+    // AFTER this call, so the raw-null (pre-block-GS) discipline is preserved.
+    AggregatesGCR.GlobalOrthonormalise();
+    // DO NOT block-orthogonalise here: runMG copies subspace[] as the RAW
+    // (pre-block-GS) basis and CoarsenOperator block-GS's it in place later.
+    // Orthogonalising now defeats the raw-null discipline (psi_coarse -> e_k)
+    // and poisons L2/L3/L4.  See project_block_orthogonalise_leak.
+    // AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    CoarseCoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_5level.cc

@@ -0,0 +1,954 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_5level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
+// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
+template<class Field>
+class LuscherGuesser : public LinearFunction<Field> {
+  const std::vector<Field> &psi;
+  Eigen::MatrixXcd          C_inv;
+public:
+  using LinearFunction<Field>::operator();
+  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
+    : psi(psi_), C_inv(Cinv_) {}
+  virtual void operator()(const Field &src, Field &guess) {
+    int N = psi.size();
+    Eigen::VectorXcd b(N);
+    for (int t = 0; t < N; t++)
+      b(t) = TensorRemove(innerProduct(psi[t], src));
+    Eigen::VectorXcd c = C_inv * b;
+    guess = Zero();
+    for (int s = 0; s < N; s++)
+      guess += ComplexD(c(s)) * psi[s];
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_)
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  GridCartesian *CoarseCoarseCoarse5d,
+  GridCartesian *CoarseCoarseCoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  L2PGCR_baseline.Name("Cbaseline");
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Lüscher deflation guesser for L3PGCR.
+  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
+  //         CoarseCoarseVector space — these cover the zero-momentum
+  //         component of the near-null space of LinOpCC.
+  // Step 2: breed Nextra additional null vectors directly on LinOpCC
+  //         using GCR with random sources — these pick up near-null
+  //         modes at all spatial frequencies not spanned by step 1.
+  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
+  //         full augmented basis and invert directly via Eigen LU.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+
+  {
+    int Nextra = nbasis;  // breed as many extra as we have promoted ones
+    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
+    GridParallelRNG RNG_CC(CoarseCoarse5d);
+    RNG_CC.SeedFixedIntegers({11,13,17,19});
+    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
+    CoarseCoarseVector tmp(CoarseCoarse5d);
+    for (int k = 0; k < Nextra; k++) {
+      CoarseCoarseVector src(CoarseCoarse5d);
+      gaussian(RNG_CC, src);
+      tmp = Zero();
+      nullGCR(src, tmp);
+      psi_cc.push_back(tmp);
+    }
+    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
+              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
+  }
+
+  const int Naug = psi_cc.size();
+  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
+  {
+    CoarseCoarseVector Acc(CoarseCoarse5d);
+    for (int l = 0; l < Naug; l++) {
+      LinOpCC.Op(psi_cc[l], Acc);
+      for (int k = 0; k < Naug; k++)
+        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
+    }
+  }
+  {
+    RealD normCcc      = Ccc.norm();
+    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
+  }
+  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
+  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3: coarsen LinOpCC using the RAW promoted psi_cc as aggregation
+  // to build the Level 4 (coarse-coarse-coarse) operator.
+  //   psi_cc[0..nbasis-1] are the coarse-coarse near-null vectors, projected
+  //   from the RAW psi_coarse (themselves projected from the RAW fine null
+  //   vectors) -- the pre-block-GS chain the whole construction depends on.
+  //   CoarsenOperator block-GS orthogonalises AggregatesL3.subspace IN PLACE,
+  //   so assign COPIES of psi_cc and keep psi_cc itself raw.
+  //
+  // Tensor depth deepens once more: innerProduct(CoarseCoarseSiteObj,...) returns
+  // iScalar<vTTComplex>, so CComplex for the L2→L3 level is iScalar<iScalar<vTComplex>>.
+  //////////////////////////////////////////////////////////////////////
+  typedef typename CoarseCoarseVector::vector_object                     CoarseCoarseSiteObj;
+  typedef iScalar<vTTComplex>                                            vTTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseCoarseSiteObj,vTTTComplex,NB>     LittleDiracOperatorL3;
+  typedef typename LittleDiracOperatorL3::CoarseVector                   CoarseCoarseCoarseVector;
+  typedef Aggregation<CoarseCoarseSiteObj,vTTTComplex,NB>               SubspaceL3;
+  typedef MGPreconditioner<CoarseCoarseSiteObj,vTTTComplex,NB>          L2to3MG;
+
+  SubspaceL3 AggregatesL3(CoarseCoarseCoarse5d, CoarseCoarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL3.subspace[k] = psi_cc[k];          // raw promoted; COPY, keeps psi_cc raw
+
+  NextToNearestStencilGeometry5D geom3(CoarseCoarseCoarse5d);
+  LittleDiracOperatorL3 LittleDiracOpL3(geom3, CoarseCoarse5d, CoarseCoarseCoarse5d);
+  LittleDiracOpL3.CoarsenOperator(LinOpCC, AggregatesL3);   // block-GS's AggregatesL3.subspace in place
+
+  NonHermitianLinearOperator<LittleDiracOperatorL3,CoarseCoarseCoarseVector> LinOpCCC(LittleDiracOpL3);
+  TrivialPrecon<CoarseCoarseCoarseVector> simpleCCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3→4: coarsen LinOpCCC to build the Level 5 operator, using a
+  // TRUNCATED basis of only the first NB5 (< nbasis) raw promoted null vectors.
+  //   psi_ccc[k] = raw psi_cc projected through the (block-GS'd) L3 aggregation
+  //   -- the pre-block-GS chain continued one level deeper.  We keep only the
+  //   leading NB5: after the global orthogonalisation of the original fine null
+  //   vectors the early indices retain the most-null content (shared low-mode
+  //   components are peeled in first), so the leading NB5 are the crudely-most-
+  //   null slice.  This is the cheap "first 30" truncation test; a principled
+  //   sigma-ordered rotation of psi_ccc would replace the slice, not the idea.
+  //   NB: a positive result is conservative (sigma-ordering can only help); a
+  //   negative one is inconclusive until the sigma-ordered NB5 is tried.
+  //
+  // Tensor depth deepens once more: CComplex for the L3→L4 level is
+  // iScalar<vTTTComplex>.  NB5 (the coarse dimension) is independent of the
+  // depth -- it just makes the coarsest site vector NB5-dimensional.
+  //////////////////////////////////////////////////////////////////////
+  const int NB5 = 30;   // compile-time: changing it re-instantiates the L4/L5 tensors
+  std::cout << GridLogMessage << "PARAM NB5 (truncated coarsest basis) = " << NB5 << std::endl;
+  assert(NB5 <= nbasis);
+
+  std::vector<CoarseCoarseCoarseVector> psi_ccc(nbasis, CoarseCoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL3.ProjectToSubspace(psi_ccc[k], psi_cc[k]);   // raw psi_cc -> L4 null vectors
+
+  //////////////////////////////////////////////////////////////////////
+  // Optional sigma-ordering of psi_ccc (SVD_REORDER set): replace the crude
+  // first-NB5 slice with the NB5 genuinely-most-null directions of span(psi_ccc)
+  // under LinOpCCC.  For a NON-NORMAL operator the nullness measure is the
+  // singular value of A restricted to the span -- eig of Q†A†AQ -- NOT the
+  // numerical range Q†AQ (which non-normality contaminates).  Robust route:
+  // whiten by the Gram (drop near-dependent directions), Hermitian-eig the
+  // whitened A†A, rotate.  The printed singular spectrum IS the SVD study: where
+  // it falls off tells you the natural NB5, and the same numbers illuminate why
+  // the earlier singular-subspace deflation re-entered.  Safe here because we
+  // ORDER vectors that then feed a Galerkin projection, not REMOVE a subspace.
+  // Default (unset) leaves psi_ccc in raw order == the "first 30" test.
+  //////////////////////////////////////////////////////////////////////
+  if ( getenv("SVD_REORDER") ) {
+    std::cout << GridLogMessage << "SVD_REORDER: sigma-ordering psi_ccc under LinOpCCC" << std::endl;
+
+    Eigen::MatrixXcd G(nbasis,nbasis);                    // Gram  = Psi^dag Psi
+    for (int i=0;i<nbasis;i++)
+    for (int j=0;j<nbasis;j++)
+      G(i,j) = TensorRemove(innerProduct(psi_ccc[i],psi_ccc[j]));
+
+    std::vector<CoarseCoarseCoarseVector> Apsi(nbasis, CoarseCoarseCoarse5d);
+    for (int j=0;j<nbasis;j++) LinOpCCC.Op(psi_ccc[j], Apsi[j]);
+
+    Eigen::MatrixXcd M(nbasis,nbasis);                    // A^dagA = Psi^dag A^dag A Psi
+    for (int i=0;i<nbasis;i++)
+    for (int j=0;j<nbasis;j++)
+      M(i,j) = TensorRemove(innerProduct(Apsi[i],Apsi[j]));
+
+    // Whiten by the Gram: G = Ug diag(g) Ug^dag; keep g > tol*max; T = Ug diag(1/sqrt g).
+    // Q = Psi T is then orthonormal (Q^dag Q = T^dag G T = I).
+    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> esG(G);
+    Eigen::VectorXd g = esG.eigenvalues();               // ascending, real
+    RealD gmax = g(nbasis-1);
+    RealD gtol = 1.0e-9 * gmax;
+    int keep = 0; for (int i=0;i<nbasis;i++) if (g(i) > gtol) keep++;
+    std::cout << GridLogMessage << "  Gram spectrum: min=" << g(0) << " max=" << gmax
+              << " cond=" << gmax/std::max(g(0),1.0e-300) << " keep=" << keep << "/" << nbasis << std::endl;
+    assert(keep >= NB5);
+
+    Eigen::MatrixXcd T(nbasis, keep);                    // whitening (largest-g first)
+    { int c=0;
+      for (int i=nbasis-1;i>=0;i--) if (g(i) > gtol) { T.col(c) = esG.eigenvectors().col(i)/std::sqrt(g(i)); c++; }
+    }
+
+    Eigen::MatrixXcd Mw = T.adjoint() * M * T;           // whitened A^dagA (keep x keep, Hermitian)
+    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> esM(Mw);
+    Eigen::VectorXd s2 = esM.eigenvalues();              // ascending sigma^2 (most-null first)
+    std::cout << GridLogMessage << "  Singular spectrum sigma_k (most-null first):" << std::endl;
+    for (int k=0;k<keep;k++)
+      std::cout << GridLogMessage << "    sigma[" << k << "] = " << std::sqrt(std::max(s2(k),0.0)) << std::endl;
+
+    Eigen::MatrixXcd R = T * esM.eigenvectors();         // coeffs over Psi, sigma-ordered orthonormal dirs
+    std::vector<CoarseCoarseCoarseVector> phi(keep, CoarseCoarseCoarse5d);
+    for (int k=0;k<keep;k++) {
+      phi[k] = Zero();
+      for (int j=0;j<nbasis;j++)
+        phi[k] = phi[k] + ComplexD(R(j,k)) * psi_ccc[j];
+    }
+    for (int k=0;k<keep;k++) psi_ccc[k] = phi[k];        // psi_ccc[0..NB5-1] now = most-null dirs
+    std::cout << GridLogMessage << "SVD_REORDER: psi_ccc replaced by sigma-ordered directions" << std::endl;
+  }
+
+  typedef typename CoarseCoarseCoarseVector::vector_object              CoarseCoarseCoarseSiteObj;
+  typedef iScalar<vTTTComplex>                                          vTTTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5> LittleDiracOperatorL4;
+  typedef typename LittleDiracOperatorL4::CoarseVector                  CoarseCoarseCoarseCoarseVector;
+  typedef Aggregation<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5>       SubspaceL4;
+  typedef MGPreconditioner<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5>  L3to4MG;
+
+  SubspaceL4 AggregatesL4(CoarseCoarseCoarseCoarse5d, CoarseCoarseCoarse5d, cb);
+  for (int k = 0; k < NB5; k++)
+    AggregatesL4.subspace[k] = psi_ccc[k];         // FIRST NB5 raw promoted vectors (truncation)
+
+  NextToNearestStencilGeometry5D geom4(CoarseCoarseCoarseCoarse5d);
+  LittleDiracOperatorL4 LittleDiracOpL4(geom4, CoarseCoarseCoarse5d, CoarseCoarseCoarseCoarse5d);
+  LittleDiracOpL4.CoarsenOperator(LinOpCCC, AggregatesL4);   // block-GS's AggregatesL4.subspace in place
+
+  NonHermitianLinearOperator<LittleDiracOperatorL4,CoarseCoarseCoarseCoarseVector> LinOpCCCC(LittleDiracOpL4);
+  TrivialPrecon<CoarseCoarseCoarseCoarseVector> simpleCCCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 5 bottom solve: GCR on a SHIFTED LinOpCCCC (the coarsest, most
+  // non-normal operator).  l5_shift slides its field of values off the origin;
+  // defaults to 0.0 (bare LinOpCCCC) until opted in.  This is the level a dense
+  // direct inverse would eventually replace: rank = NB5 * sites(clatt4).
+  //////////////////////////////////////////////////////////////////////
+  RealD l5_shift = 0.0;
+  if(getenv("l5_shift")) l5_shift = atof(getenv("l5_shift"));
+  std::cout << GridLogMessage << "PARAM l5_shift = " << l5_shift << std::endl;
+
+  ShiftedLinearOperator<CoarseCoarseCoarseCoarseVector> ShiftedLinOpCCCC(l5_shift, LinOpCCCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseCoarseVector> L5PGCR(1.0e-1,200,ShiftedLinOpCCCC,simpleCCCC,16,16);
+  L5PGCR.Level(5);
+  L5PGCR.Name("CCCCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3→4 V-cycle: depth-2 SHIFTED smoother on LinOpCCC + Level 5 bottom.
+  // Level 4 is no longer the bottom -- it is smoothed shallowly and recursed to
+  // Level 5, mirroring how Level 3 recurses to Level 4.
+  //////////////////////////////////////////////////////////////////////
+  RealD ccc_smoother_shift = 0.05;
+  int   ccc_smoother_nstep = 2;
+  if(getenv("ccc_smoother_shift")) ccc_smoother_shift = atof(getenv("ccc_smoother_shift"));
+  if(getenv("ccc_smoother_nstep")) ccc_smoother_nstep = atoi(getenv("ccc_smoother_nstep"));
+
+  ShiftedLinearOperator<CoarseCoarseCoarseVector> ShiftedLinOpCCC(ccc_smoother_shift, LinOpCCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector>
+    CoarseCoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCCC,simpleCCC,ccc_smoother_nstep,ccc_smoother_nstep);
+  CoarseCoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
+  CoarseCoarseCoarseSmootherGCR.Level(4);
+  CoarseCoarseCoarseSmootherGCR.Name("CCCsmoother");
+
+  L3to4MG L3to4Precon(AggregatesL4,
+                      LinOpCCC,
+                      simpleCCC,                     // no pre-smoother
+                      CoarseCoarseCoarseSmootherGCR, // post-smoother: depth-2 shifted GCR
+                      LinOpCCCC,
+                      L5PGCR,
+                      simpleCCCC);                   // trivial guesser at the bottom
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 4 (coarse-coarse-coarse) solve: GCR preconditioned by the L3→L4 V-cycle.
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector> L4MGsolver(1.0e-1,200,LinOpCCC,L3to4Precon,16,16);
+  L4MGsolver.Level(4);
+  L4MGsolver.Name("CCCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3 V-cycle: depth-2 SHIFTED smoother on LinOpCC + Level 4 solve.
+  // The shift slides the coarse-coarse field of values off the origin so a
+  // 2-step smoother has something to bite on a non-normal operator (IRS idea).
+  //////////////////////////////////////////////////////////////////////
+  RealD cc_smoother_shift = 0.01;
+  int   cc_smoother_nstep = 2;
+  if(getenv("cc_smoother_shift")) cc_smoother_shift = atof(getenv("cc_smoother_shift"));
+  if(getenv("cc_smoother_nstep")) cc_smoother_nstep = atoi(getenv("cc_smoother_nstep"));
+
+  ShiftedLinearOperator<CoarseCoarseVector> ShiftedLinOpCC(cc_smoother_shift, LinOpCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    CoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCC,simpleCC,cc_smoother_nstep,cc_smoother_nstep);
+  CoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
+  CoarseCoarseSmootherGCR.Level(3);
+  CoarseCoarseSmootherGCR.Name("CCsmoother");
+
+  L2to3MG L2to3Precon(AggregatesL3,
+                      LinOpCC,
+                      simpleCC,                  // no pre-smoother
+                      CoarseCoarseSmootherGCR,   // post-smoother: depth-2 shifted GCR
+                      LinOpCCC,
+                      L4MGsolver,                // coarse solve is now the L3→L4 V-cycle
+                      simpleCCC);                // trivial guesser
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3 (coarse-coarse) solve: GCR preconditioned by the L2→L3 V-cycle.
+  // Replaces the plain L3PGCR of the 3-level build -- the coarse-coarse level
+  // is now smoothed shallowly and recursed rather than solved deeply.
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3MGsolver(1.0e-1,200,LinOpCC,L2to3Precon,16,16);
+  L3MGsolver.Level(3);
+  L3MGsolver.Name("CCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.01;
+  int   coarse_smoother_nstep = 2;   // depth-2 smoother on the coarse level
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+                       LinOpCoarse,
+                       simpleC,            // no pre-smoother (matches fine-grid setup)
+                       CoarseSmootherGCR,  // post-smoother: depth-2 shifted GCR
+                       LinOpCC,
+                       L3MGsolver,         // coarse-coarse solve is now the L2→L3 V-cycle
+                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  L2MGsolver.Name("Couter");
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full five-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Five-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+                               PVdagM,
+                               simple_fine,
+                               SmootherGCR,
+                               LinOpCoarse,
+                               L2MGsolver,
+                               simpleC);
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Five-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // Level 3 coarse-coarse-coarse grid: block clatt2 = {12,12,8,16} -> {6,12,8,8}.
+  // GEOMETRY (mpi 3.6.4.4, Nsimd=4 => SIMD layout {1,1,2,2}, factor 2 on z and t):
+  //   every grid needs z-local and t-local EVEN.  clatt2-local is {4,2,2,4}, so
+  //   z-local=2 is already at its minimum even value and CANNOT be blocked (2->1
+  //   is odd and trips the SIMD assertion); y-local=2 would go to 1 (degenerate).
+  //   Only x and t have room, so block {2,1,1,2}: clatt3 {6,12,8,8}, L4-local
+  //   {2,2,2,2} -- all dims even and >=2.  z stays unblocked by construction.
+  Coordinate clatt3 = clatt2;
+  clatt3[0] /= 2;   // x: 12 -> 6   (x-local 4 -> 2)
+  // clatt3[1] (y) unblocked: y-local 2 -> blocking gives 1 (degenerate)
+  // clatt3[2] (z) unblocked: z-local 2 is SIMD-pinned even, cannot halve
+  clatt3[3] /= 2;   // t: 16 -> 8   (t-local 4 -> 2)
+  std::cout << GridLogMessage << "Level 3 coarse-coarse-coarse lattice: " << clatt3 << std::endl;
+
+  GridCartesian *CoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt3, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarse4d);
+
+  // Level 4 coarse^4 grid: block clatt3 = {6,12,8,8} -> {3,6,8,8} via {2,2,1,1}.
+  //   mpi 3.6.4.4 => clatt4-local {1,1,2,2}: z-local=2, t-local=2 stay EVEN (SIMD
+  //   factor 2 pins them), so z,t are unblocked; x,y (SIMD factor 1) halve to
+  //   local 1 -- fully distributed but legal for the halo-depth-1 NextToNearest
+  //   stencil.  1152 sites; with NB5=30 that is the 34,560-rank coarsest operator
+  //   a dense direct inverse would target.
+  Coordinate clatt4 = clatt3;
+  clatt4[0] /= 2;   // x: 6  -> 3   (x-local 2 -> 1)
+  clatt4[1] /= 2;   // y: 12 -> 6   (y-local 2 -> 1)
+  // clatt4[2] (z) unblocked: z-local 2 is SIMD-pinned even
+  // clatt4[3] (t) unblocked: t-local 2 is SIMD-pinned even
+  std::cout << GridLogMessage << "Level 4 coarse^4 lattice: " << clatt4 << std::endl;
+
+  GridCartesian *CoarseCoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt4, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, in case the cached file
+    // predates the GlobalOrthonormalise() that CreateSubspaceGCR now applies
+    // (Aggregates.h:196).  It is span-preserving and makes the vectors globally
+    // orthonormal -- it is NOT the block Orthogonalise() below, so it does NOT
+    // cause the psi_coarse->e_k trap.  It also (re)establishes the weak nullness
+    // gradient (shared most-null components peeled into the early indices) that
+    // the "first NB5" truncation relies on.  Idempotent if the file was already
+    // globally orthonormal.  The RAW subspace copy in runMG happens AFTER this
+    // call, so the raw-null (pre-block-GS) discipline is preserved.
+    AggregatesGCR.GlobalOrthonormalise();
+    // DO NOT block-orthogonalise here: runMG copies subspace[] as the RAW
+    // (pre-block-GS) basis and CoarsenOperator block-GS's it in place later.
+    // Orthogonalising now defeats the raw-null discipline (psi_coarse -> e_k)
+    // and poisons L2/L3/L4.  See project_block_orthogonalise_leak.
+    // AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    CoarseCoarseCoarse5d,
+    CoarseCoarseCoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_census.cc

@@ -0,0 +1,717 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_census.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// Spectral census of the coarsened PVdagM operator A_c.
+//
+// Measures the three sets that discriminate between the candidate explanations
+// for slow coarse-grid Krylov convergence:
+//
+//   0. Fine Ritz diagonal of RAW subspace vectors (pre-block-orthog).
+//      NB CoarsenOperator block-orthogonalises subspace[] IN PLACE; all
+//      nulliness/deflation bases must be built from a raw copy.
+//   1. Adjoint correctness check    <y|A x> == <A^dag y|x>      (fail-fast)
+//   2. Raw-vector coarse images vs A_c: RQ (must equal CENSUS 0 by Galerkin),
+//      ||A_c psi_c||/||psi_c||, and representability error
+//   3. sigma_max^2 = lambda_max(A_c^dag A_c)  via power method
+//   4. Low singular values          Chebyshev-filtered IRL on A_c^dag A_c
+//      -> sigma_min census = pseudospectrum of A_c evaluated at the origin
+//   5. Half-plane margin            lambda_min/max of H = (A_c + A_c^dag)/2
+//      -> min Re W(A_c);  positive-real check (Eisenstat-Elman-Schultz bound)
+//
+// Interpretation:
+//   sigma_min ~ min|lambda|, ~nbasis tiny then gap : effectively normal, bipartite
+//   sigma_min ~ min|lambda|, dense low tail        : normal but rank-starved
+//   sigma_min << min|lambda|                       : non-normal near origin
+//   lambda_min(H) < 0                              : half-plane condition violated
+//
+// Requires the dagger code path in GeneralCoarsenedMatrix:
+//   _Adag allocated, PopulateAdag active, _Adag exchanged, hermitian=0.
+//
+// Env vars:
+//   MASS               fermion mass                     (default 0.00078)
+//   SUBSPACE_FILE      subspace cache path
+//   CoarseSolverShift  shift baked into coarsening      (default 0.0: pure Galerkin)
+//   CENSUS_NSTOP       converged low modes wanted       (default 60)
+//   CENSUS_NK          Lanczos Nk                       (default 96)
+//   CENSUS_NM          Lanczos Nm                       (default 192)
+//   CENSUS_TOL         Lanczos residual                 (default 1e-5)
+//   CENSUS_MAXIT       Lanczos max restarts             (default 50)
+//   CHEBY_LO           filter low edge in sigma^2       (default 4.0)
+//   CHEBY_HI           filter high edge; 0 = auto from power method x1.1
+//   CHEBY_ORDER        filter order                     (default 401)
+//                      filter gain at 0 ~ cosh(order*2*sqrt(lo/hi)); with
+//                      hi~2200, lo=4, order=401 => gain ~ 1e14. lo=0.01 at
+//                      order 201 gives gain ~1.4 (stagnation).
+
+#include <Grid/Grid.h>
+#include <Grid/Grid_Eigen_Dense.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+using namespace std;
+using namespace Grid;
+
+RealD mass              = 0.00078;
+RealD CoarseSolverShift = 0.0;
+int   CensusNstop       = 60;
+int   CensusNk          = 96;
+int   CensusNm          = 192;
+RealD CensusTol         = 1.0e-5;
+int   CensusMaxIt       = 50;
+RealD ChebyLo           = 4.0;   // sigma^2 cutoff: amplifies sigma < 2. Filter gain ~ cosh(order*2*sqrt(lo/hi))
+RealD ChebyHi           = 0.0;   // 0 => auto: 1.1 * power-method sigma_max^2
+int   ChebyOrder        = 401;
+RealD CGdeflTol         = 1.0e-8; // CENSUS 6 deflated-CG tolerance
+int   CGdeflMaxIt       = 4000;   // CENSUS 6 deflated-CG max iterations
+int   DeflRank          = 0;      // CENSUS 6 deflation rank; 0 => all available per basis
+
+void ParseEnvironment(void)
+{
+  if(getenv("MASS"))              mass              = atof(getenv("MASS"));
+  if(getenv("CoarseSolverShift")) CoarseSolverShift = atof(getenv("CoarseSolverShift"));
+  if(getenv("CENSUS_NSTOP"))      CensusNstop       = atoi(getenv("CENSUS_NSTOP"));
+  if(getenv("CENSUS_NK"))         CensusNk          = atoi(getenv("CENSUS_NK"));
+  if(getenv("CENSUS_NM"))         CensusNm          = atoi(getenv("CENSUS_NM"));
+  if(getenv("CENSUS_TOL"))        CensusTol         = atof(getenv("CENSUS_TOL"));
+  if(getenv("CENSUS_MAXIT"))      CensusMaxIt       = atoi(getenv("CENSUS_MAXIT"));
+  if(getenv("CHEBY_LO"))          ChebyLo           = atof(getenv("CHEBY_LO"));
+  if(getenv("CHEBY_HI"))          ChebyHi           = atof(getenv("CHEBY_HI"));
+  if(getenv("CHEBY_ORDER"))       ChebyOrder        = atoi(getenv("CHEBY_ORDER"));
+  if(getenv("CGDEFL_TOL"))        CGdeflTol         = atof(getenv("CGDEFL_TOL"));
+  if(getenv("CGDEFL_MAXIT"))      CGdeflMaxIt       = atoi(getenv("CGDEFL_MAXIT"));
+  if(getenv("DEFL_RANK"))         DeflRank          = atoi(getenv("DEFL_RANK"));
+
+  std::cout << GridLogMessage << "PARAM: MASS              " << mass              << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverShift " << CoarseSolverShift << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_NSTOP      " << CensusNstop       << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_NK         " << CensusNk          << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_NM         " << CensusNm          << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_TOL        " << CensusTol         << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_MAXIT      " << CensusMaxIt       << std::endl;
+  std::cout << GridLogMessage << "PARAM: CHEBY_LO          " << ChebyLo           << std::endl;
+  std::cout << GridLogMessage << "PARAM: CHEBY_HI          " << ChebyHi           << std::endl;
+  std::cout << GridLogMessage << "PARAM: CHEBY_ORDER       " << ChebyOrder        << std::endl;
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+  Grid::emptyUserRecord record;
+  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+  SW.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++)
+    SW.writeScidacFieldRecord(subspace[k], record);
+  SW.close();
+#endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+  Grid::emptyUserRecord record;
+  Grid::ScidacReader SR;
+  SR.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++)
+    SR.readScidacFieldRecord(subspace[k], record);
+  SR.close();
+#endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  RealD shift;
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// H = (A + A^dag)/2 : Hermitian part of the coarse operator.
+// lambda_min(H) = min Re W(A) is the half-plane margin; the EES GCR
+// convergence theorem requires it positive.
+template<class Matrix,class Field>
+class HermitianPartOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+public:
+  HermitianPartOperator(Matrix &Mat): _Mat(Mat) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,out);
+    _Mat.Mdag(in,tmp);
+    out = 0.5*(out + tmp);
+  }
+};
+
+// s*I - Op : power method on this gives s - lambda_min(Op) for Hermitian Op.
+template<class Field>
+class ShiftedNegatedOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD s;
+public:
+  ShiftedNegatedOperator(RealD _s, LinearOperatorBase<Field> &Op): _Op(Op), s(_s) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    _Op.HermOp(in,out);
+    out = s*in - out;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+  ParseEnvironment();
+
+  const int Ls=24;
+  RealD M5=1.8;
+  RealD b=1.5;
+  RealD c=0.5;
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Census of coarse PVdagM: mass=" << mass << " Ls=" << Ls << " nbasis=" << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Blocking: default matches Example_pvdagm.cc; override with e.g. BLOCK=2.2.2.2
+  Coordinate clatt = lat_size;
+  Coordinate Block({4,4,6,4});
+  if ( getenv("BLOCK") ) {
+    GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block);
+    GRID_ASSERT(Block.size()==4);
+  }
+  for(int d=0;d<clatt.size();d++){
+    GRID_ASSERT(lat_size[d] % Block[d] == 0);
+    clatt[d] = lat_size[d]/Block[d];
+  }
+  std::cout << GridLogMessage << "Block " << Block << "  coarse lattice " << clatt << std::endl;
+
+  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  GridParallelRNG RNG5(FGrid);    RNG5.SeedFixedIntegers({5,6,7,8});
+  GridParallelRNG RNG4(UGrid);    RNG4.SeedFixedIntegers({1,2,3,4});
+  GridParallelRNG CRNG(Coarse5d); CRNG.SeedFixedIntegers({5,6,7,8});
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(CoarseSolverShift,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // Subspace: load from cache or generate
+  //////////////////////////////////////////////////////////////////////
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Keep the RAW (pre-block-orthogonalisation) near-null vectors.
+  // CoarsenOperator block-orthogonalises subspace[] IN PLACE, after which
+  // subspace[k] is the orthonormal basis phi_k and Project(phi_k) = e_k,
+  // the block-constant unit vector -- NOT a near-null direction.
+  // All nulliness measurements and any deflation basis must use raw[].
+  //////////////////////////////////////////////////////////////////////
+  std::vector<LatticeFermionD> raw(nbasis,FGrid);
+  for(int k=0;k<nbasis;k++) raw[k] = AggregatesGCR.subspace[k];
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 0: fine-grid Ritz diagonal on the loaded/generated raw vectors.
+  // Expect Re <psi|A|psi>/<psi|psi> ~ the nulliness achieved at generation
+  // (~2e-3). O(0.1-10) values mean the cache holds orthogonalised vectors
+  // and must be regenerated.
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 0: fine Ritz diagonal of raw subspace vectors" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  {
+    LatticeFermionD Ap(FGrid);
+    for(int k=0;k<nbasis;k++){
+      PVdagM.Op(raw[k],Ap);
+      RealD    n2psi = norm2(raw[k]);
+      ComplexD rq    = innerProduct(raw[k],Ap)/n2psi;
+      std::cout << GridLogMessage << "CENSUS: raw[" << k << "]  <psi|A|psi>/<psi|psi> = " << rq
+                << "   ||A psi||/||psi|| = " << std::sqrt(norm2(Ap)/n2psi) << std::endl;
+    }
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarsen. hermitian=0 is REQUIRED: enables PopulateAdag so that
+  // Mdag applies A^dag rather than silently aliasing to A.
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d,0);
+  if ( CoarseSolverShift != 0.0 ) {
+    std::cout << GridLogMessage << "Coarsening SHIFTED operator, shift=" << CoarseSolverShift << std::endl;
+    LittleDiracOpPV.CoarsenOperator(ShiftedPVdagM, AggregatesGCR);
+  } else {
+    std::cout << GridLogMessage << "Coarsening pure Galerkin operator (no shift)" << std::endl;
+    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);
+  }
+
+  CoarseVector c_x(Coarse5d);
+  CoarseVector c_y(Coarse5d);
+  CoarseVector c_t1(Coarse5d);
+  CoarseVector c_t2(Coarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 1: adjoint correctness (fail fast)
+  //   <y|A x> == <A^dag y|x>  for random x,y
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 1: adjoint correctness of dagger code path" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  random(CRNG,c_x);
+  random(CRNG,c_y);
+  LittleDiracOpPV.M   (c_x,c_t1);       // A x
+  LittleDiracOpPV.Mdag(c_y,c_t2);       // A^dag y
+  ComplexD ip1 = innerProduct(c_y,c_t1);   // <y|A x>
+  ComplexD ip2 = innerProduct(c_t2,c_x);   // <A^dag y|x>
+  RealD reldiff = abs(ip1-ip2)/abs(ip1);
+  std::cout << GridLogMessage << "CENSUS: <y|Ax>       = " << ip1 << std::endl;
+  std::cout << GridLogMessage << "CENSUS: <Adag y|x>   = " << ip2 << std::endl;
+  std::cout << GridLogMessage << "CENSUS: rel diff     = " << reldiff << "  (expect ~1e-14; FAIL if O(1))" << std::endl;
+  GRID_ASSERT(reldiff < 1.0e-8);
+
+  // Coarse near-null ("global") vectors psi_c[k] = P^dag raw[k], stored for the
+  // Ritz-matrix + deflation study in CENSUS 6 (filled in CENSUS 2's projection
+  // loop below, before raw[]/subspace[] are freed).
+  std::vector<CoarseVector> psi_c(nbasis,Coarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 2: nulliness of the RAW vectors' coarse images against A_c.
+  //   psi_c[k] = P^dag raw[k].  Galerkin guarantees the Rayleigh quotient
+  //   equals CENSUS 0's fine value exactly (raw[k] is in span of its own
+  //   chopped pieces) -- agreement is a machine-precision validation of
+  //   the coarsening.  ||A_c psi_c||/||psi_c|| is the sigma-relevant norm.
+  //   The representability column ||raw - P psi_c||/||raw|| must be ~eps.
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 2: raw-vector coarse images against coarse operator" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  {
+    LatticeFermionD back(FGrid);
+    for(int k=0;k<nbasis;k++){
+      AggregatesGCR.ProjectToSubspace(c_x, raw[k]);
+      psi_c[k] = c_x;                       // store coarse near-null vector for CENSUS 6
+      AggregatesGCR.PromoteFromSubspace(c_x, back);
+      back = back - raw[k];
+      RealD represent = std::sqrt(norm2(back)/norm2(raw[k]));
+      LittleDiracOpPV.M(c_x, c_t1);
+      RealD    n2psi = norm2(c_x);
+      RealD    n2Apsi= norm2(c_t1);
+      ComplexD rq    = innerProduct(c_x,c_t1) / n2psi;
+      std::cout << GridLogMessage << "CENSUS: psi_c[" << k << "]  <psi|A|psi>/<psi|psi> = " << rq
+                << "   ||A psi||/||psi|| = " << std::sqrt(n2Apsi/n2psi)
+                << "   represent_err = " << represent << std::endl;
+    }
+  }
+
+  // Fine subspace + raw copy are needed only through CENSUS 2; CENSUS 3-5 are
+  // entirely coarse (LittleDiracOpPV only), and the CENSUS 4 evec save writes the
+  // coarse vectors directly.  Release the ~2*nbasis fine 5D fields (~14 GB/GCD at
+  // 2^4) HERE, before the order-ChebyOrder Lanczos whose padded coarse temporaries
+  // otherwise push host memory over the top on top of _A + _Adag (the AccCache
+  // CpuPtr!=NULL abort seen mid-iteration).
+  // Direct orthonormality check of the fine near-null vectors (GlobalOrthonormalise
+  // in CreateSubspaceGCR).  raw is freed just below, so this runs here, not CENSUS 6.
+  // If this is ~0 but the coarse Gram S (CENSUS 6) is not, the gap is representability,
+  // not orthonormality.
+  {
+    Eigen::MatrixXcd Gfine(nbasis,nbasis);
+    for(int i=0;i<nbasis;i++){
+      for(int j=i;j<nbasis;j++){
+        ComplexD g = innerProduct(raw[i],raw[j]);
+        Gfine(i,j) = std::complex<double>(g.real(),g.imag());
+        Gfine(j,i) = std::conj(Gfine(i,j));
+      }
+    }
+    double GmI = (Gfine - Eigen::MatrixXcd::Identity(nbasis,nbasis)).norm();
+    std::cout << GridLogMessage << "CENSUS 2b: fine Gram ||<raw_i|raw_j> - I||_F = " << GmI
+              << "   (expect ~0 if fine vectors orthonormal)" << std::endl;
+  }
+
+  raw.clear();                    raw.shrink_to_fit();
+  AggregatesGCR.subspace.clear(); AggregatesGCR.subspace.shrink_to_fit();
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 3: sigma_max^2 = lambda_max( A_c^dag A_c ) by power method
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 3: power method for sigma_max" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  MdagMLinearOperator<LittleDiracOperator,CoarseVector> HermOpAdagA(LittleDiracOpPV);
+  random(CRNG,c_x);
+  PowerMethod<CoarseVector> PM;
+  RealD sigmax2 = PM(HermOpAdagA,c_x);
+  std::cout << GridLogMessage << "CENSUS: lambda_max(AdagA) = " << sigmax2
+            << "   sigma_max = " << std::sqrt(sigmax2) << std::endl;
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 4: low singular values via Chebyshev-filtered IRL on A^dag A
+  //   The low end of sigma(A_c) is the pseudospectrum of A_c at z=0.
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 4: Chebyshev-filtered Lanczos, low sigma^2" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  RealD cheby_hi = (ChebyHi > 0.0) ? ChebyHi : 1.1*sigmax2;
+  std::cout << GridLogMessage << "Chebyshev filter [" << ChebyLo << "," << cheby_hi << "] order " << ChebyOrder << std::endl;
+
+  // eval/evec/Nconv hoisted out of the block so CENSUS 6 can deflate with them.
+  std::vector<RealD>        eval(CensusNm);
+  std::vector<CoarseVector> evec(CensusNm,Coarse5d);
+  int Nconv=0;
+  {
+    Chebyshev<CoarseVector>      Cheby(ChebyLo,cheby_hi,ChebyOrder);
+    FunctionHermOp<CoarseVector> OpCheby(Cheby,HermOpAdagA);
+    PlainHermOp<CoarseVector>    Op     (HermOpAdagA);
+
+    ImplicitlyRestartedLanczos<CoarseVector> IRL(OpCheby,Op,CensusNstop,CensusNk,CensusNm,CensusTol,CensusMaxIt);
+
+    random(CRNG,c_x);
+    IRL.calc(eval,evec,c_x,Nconv);
+
+    std::cout << GridLogMessage << "CENSUS: converged " << Nconv << " modes of AdagA" << std::endl;
+    for(int i=0;i<Nconv;i++){
+      std::cout << GridLogMessage << "CENSUS: sigma[" << i << "]^2 = " << eval[i]
+                << "   sigma = " << std::sqrt(std::max(eval[i],0.0)) << std::endl;
+    }
+
+    // Optionally persist the low right-singular-vector basis: this IS the
+    // deflation basis for the coarse solve (ADEF1 / MultiRHSDeflation).
+    // Set CENSUS_EVEC_FILE to enable.
+    if ( getenv("CENSUS_EVEC_FILE") && Nconv>0 ) {
+#ifdef HAVE_LIME
+      std::string evec_file(getenv("CENSUS_EVEC_FILE"));
+      std::string eval_file = evec_file + ".evals.xml";
+      std::cout << GridLogMessage << "CENSUS: saving " << Nconv << " singular vectors to " << evec_file << std::endl;
+      emptyUserRecord record;
+      ScidacWriter WR(evec[0].Grid()->IsBoss());
+      WR.open(evec_file);
+      for(int i=0;i<Nconv;i++) WR.writeScidacFieldRecord(evec[i],record);
+      WR.close();
+      XmlWriter WRx(eval_file);
+      std::vector<RealD> eval_out(eval.begin(),eval.begin()+Nconv); // don't shrink shared eval
+      write(WRx,"evals",eval_out);
+#endif
+    }
+  }
+
+  // NB: evec/eval stay sized CensusNm (Lattice has no default ctor, so
+  // std::vector<CoarseVector>::resize won't instantiate).  They match in size,
+  // which is all DeflatedGuesser asserts; CENSUS 6 only ever indexes [0,Nconv).
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 5: half-plane margin from the Hermitian part
+  //   lambda_min(H) = min Re W(A_c) > 0 <=> positive-real (EES applies)
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 5: Hermitian part H=(A+Adag)/2, half-plane margin" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  HermitianPartOperator<LittleDiracOperator,CoarseVector> HermPart(LittleDiracOpPV);
+
+  random(CRNG,c_x);
+  RealD lamHmax = PM(HermPart,c_x);
+  std::cout << GridLogMessage << "CENSUS: lambda_max(H) = " << lamHmax << std::endl;
+
+  // lambda_min(H): most-negative eigenvalue via Chebyshev-filtered IRL on H.
+  // A shifted power method cannot separate it from the dense low tail (which is
+  // why the earlier -0.006 is suspect); Cheby(lo, hi>=lambda_max) amplifies the
+  // most-negative mode hardest so IRL isolates the true bottom of the spectrum.
+  RealD hpLo    = getenv("HPLANE_CHEBY_LO")    ? atof(getenv("HPLANE_CHEBY_LO"))    : 0.1;
+  RealD hpHi    = getenv("HPLANE_CHEBY_HI")    ? atof(getenv("HPLANE_CHEBY_HI"))    : 1.1*lamHmax;
+  int   hpOrder = getenv("HPLANE_CHEBY_ORDER") ? atoi(getenv("HPLANE_CHEBY_ORDER")) : 61;
+  // Grid's Chebyshev filter MUST be odd order (positive for x < -1, where the low/
+  // negative modes map); an even order flips the sign there and the IRL blows up.
+  if(hpOrder%2==0){ hpOrder++;
+    std::cout<<GridLogMessage<<"HPLANE_CHEBY_ORDER forced odd -> "<<hpOrder<<std::endl; }
+  int   hpNstop = getenv("HPLANE_NSTOP")       ? atoi(getenv("HPLANE_NSTOP"))       : 8;
+  int   hpNk    = getenv("HPLANE_NK")          ? atoi(getenv("HPLANE_NK"))          : 24;
+  int   hpNm    = getenv("HPLANE_NM")          ? atoi(getenv("HPLANE_NM"))          : 48;
+  RealD hpTol   = getenv("HPLANE_TOL")         ? atof(getenv("HPLANE_TOL"))         : 1.0e-4;
+  int   hpMaxIt = getenv("HPLANE_MAXIT")       ? atoi(getenv("HPLANE_MAXIT"))       : 20;
+
+  Chebyshev<CoarseVector>      HCheby(hpLo,hpHi,hpOrder);
+  FunctionHermOp<CoarseVector> HOpCheby(HCheby,HermPart);
+  PlainHermOp<CoarseVector>    HOpPlain(HermPart);
+  ImplicitlyRestartedLanczos<CoarseVector> HIRL(HOpCheby,HOpPlain,hpNstop,hpNk,hpNm,hpTol,hpMaxIt);
+  std::vector<RealD>        heval(hpNm);
+  std::vector<CoarseVector> hevec(hpNm,Coarse5d);
+  int hNconv=0;
+  random(CRNG,c_x);
+  HIRL.calc(heval,hevec,c_x,hNconv);
+  RealD lamHmin = (hNconv>0) ? heval[0] : 9.99e99;
+  for(int kk=0;kk<hNconv;kk++) lamHmin = std::min(lamHmin, heval[kk]);
+  std::cout << GridLogMessage << "CENSUS: IRL H-bottom converged " << hNconv
+            << " eigenvalues; most-negative = " << lamHmin << std::endl;
+  std::cout << GridLogMessage << "CENSUS: lambda_min(H) = " << lamHmin
+            << "  (positive-real / half-plane margin; NEGATIVE => GCR unguaranteed)" << std::endl;
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 6: Ritz matrix of the coarse near-null basis + deflated-CG study
+  //
+  //   C_ij = <psi_c^i | A^dag A | psi_c^j>,   S_ij = <psi_c^i | psi_c^j>.
+  //   psi_c are NOT orthonormal (raw near-null projected to coarse), so the
+  //   Rayleigh-Ritz problem is the GENERALISED Hermitian one  C v = theta S v.
+  //   Its eigenpairs (theta_i, g_i = sum_j V(j,i) psi_c^j) are the best approximate
+  //   eigenpairs of A^dag A available from span{psi_c}; Eigen normalises so that
+  //   V^dag S V = I, hence <g_i|g_j> = delta_ij and the g_i are an orthonormal
+  //   DeflatedGuesser basis.  Compare theta_i to the Lanczos sigma_i^2, then run
+  //   three CG solves on A^dag A: [1] no deflation, [2] Lanczos-eigenvector
+  //   deflated guess, [3] Ritz global-vector deflated guess (g_i treated as pure
+  //   eigenvectors with eigenvalue theta_i).
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 6: Ritz matrix C_ij = <psi_c^i|AdagA|psi_c^j> + deflated CG" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+
+  std::vector<CoarseVector> Apsi(nbasis,Coarse5d);
+  for(int j=0;j<nbasis;j++) HermOpAdagA.HermOp(psi_c[j],Apsi[j]);   // A^dag A psi_c^j
+
+  Eigen::MatrixXcd Cmat(nbasis,nbasis);
+  Eigen::MatrixXcd Smat(nbasis,nbasis);
+  for(int i=0;i<nbasis;i++){
+    for(int j=0;j<nbasis;j++){
+      ComplexD cij = innerProduct(psi_c[i],Apsi[j]);
+      ComplexD sij = innerProduct(psi_c[i],psi_c[j]);
+      Cmat(i,j) = std::complex<double>(cij.real(),cij.imag());
+      Smat(i,j) = std::complex<double>(sij.real(),sij.imag());
+    }
+  }
+
+  {
+    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> ses(Smat);
+    Eigen::MatrixXcd Id = Eigen::MatrixXcd::Identity(nbasis,nbasis);
+    double SmI = (Smat - Id).norm();   // ||S - I||_F : ~0 iff psi_c orthonormal
+    std::cout << GridLogMessage << "CENSUS 6: Gram S eig range [" << ses.eigenvalues()(0)
+              << ", " << ses.eigenvalues()(nbasis-1)
+              << "]   ||S - I||_F = " << SmI
+              << "   (expect ~0: fine vectors are GlobalOrthonormalise'd => psi_c orthonormal)" << std::endl;
+  }
+  Eigen::GeneralizedSelfAdjointEigenSolver<Eigen::MatrixXcd> ges(Cmat,Smat);
+  Eigen::VectorXd  theta = ges.eigenvalues();    // ascending, real
+  Eigen::MatrixXcd Vr    = ges.eigenvectors();   // columns; V^dag S V = I
+
+  int ncmp = std::min((int)nbasis,Nconv);
+  std::cout << GridLogMessage << "CENSUS 6: Ritz theta vs Lanczos sigma^2 (both ascending):" << std::endl;
+  for(int i=0;i<nbasis;i++){
+    if(i<ncmp)
+      std::cout << GridLogMessage << "CENSUS: theta["<<i<<"] = "<<theta(i)
+                <<"   sqrt = "<<std::sqrt(std::max(theta(i),0.0))
+                <<"   | sigma^2 = "<<eval[i]<<"   theta/sigma^2 = "<<theta(i)/eval[i] << std::endl;
+    else
+      std::cout << GridLogMessage << "CENSUS: theta["<<i<<"] = "<<theta(i)
+                <<"   sqrt = "<<std::sqrt(std::max(theta(i),0.0)) << std::endl;
+  }
+
+  // Ritz global vectors g_i = sum_j V(j,i) psi_c^j  (S-orthonormal), eigenvalue theta_i
+  std::vector<CoarseVector> gvec(nbasis,Coarse5d);
+  std::vector<RealD>        gval(nbasis);
+  for(int i=0;i<nbasis;i++){
+    gvec[i] = Zero();
+    for(int j=0;j<nbasis;j++){
+      ComplexD coeff(Vr(j,i).real(),Vr(j,i).imag());
+      axpy(gvec[i],coeff,psi_c[j],gvec[i]);
+    }
+    gval[i] = theta(i);
+  }
+
+  // How good are the diagonalised global vectors as actual eigenvectors of A^dag A?
+  {
+    CoarseVector Ag(Coarse5d), rr(Coarse5d);
+    int nchk = std::min((int)nbasis,16);
+    for(int i=0;i<nchk;i++){
+      HermOpAdagA.HermOp(gvec[i],Ag);
+      axpy(rr,-gval[i],gvec[i],Ag);          // rr = A^dag A g - theta g
+      RealD rn = std::sqrt(norm2(rr));
+      std::cout << GridLogMessage << "CENSUS 6: Ritz resid ["<<i<<"] ||AdagA g - theta g||/theta = "
+                << rn/std::max(gval[i],1.0e-30) << "   (theta="<<gval[i]<<")" << std::endl;
+    }
+  }
+
+  // --- Three CG solves on A^dag A, common random source ---
+  int rankLanc = (DeflRank>0) ? std::min(DeflRank,Nconv)       : Nconv;
+  int rankRitz = (DeflRank>0) ? std::min(DeflRank,(int)nbasis) : (int)nbasis;
+  std::cout << GridLogMessage << "CENSUS 6: CG tol "<<CGdeflTol<<" maxit "<<CGdeflMaxIt
+            << " ; deflation ranks -- Lanczos "<<rankLanc<<", Ritz "<<rankRitz << std::endl;
+
+  CoarseVector cg_src(Coarse5d);  random(CRNG,cg_src);
+  CoarseVector cg_x  (Coarse5d);
+  ConjugateGradient<CoarseVector> CGdefl(CGdeflTol,CGdeflMaxIt,false);
+
+  cg_x = Zero();
+  CGdefl(HermOpAdagA,cg_src,cg_x);
+  std::cout << GridLogMessage << "CENSUS 6: [1] no deflation          : iters = "
+            << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
+
+  if(rankLanc>0){
+    DeflatedGuesser<CoarseVector> guessL(evec,eval,rankLanc);
+    guessL(cg_src,cg_x);
+    CGdefl(HermOpAdagA,cg_src,cg_x);
+    std::cout << GridLogMessage << "CENSUS 6: [2] Lanczos-evec deflation : iters = "
+              << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
+  }
+
+  {
+    DeflatedGuesser<CoarseVector> guessR(gvec,gval,rankRitz);
+    guessR(cg_src,cg_x);
+    CGdefl(HermOpAdagA,cg_src,cg_x);
+    std::cout << GridLogMessage << "CENSUS 6: [3] Ritz-vector deflation  : iters = "
+              << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Summary
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS SUMMARY" << std::endl;
+  std::cout << GridLogMessage << "  sigma_max        = " << std::sqrt(sigmax2) << std::endl;
+  std::cout << GridLogMessage << "  lambda_max(H)    = " << lamHmax  << std::endl;
+  std::cout << GridLogMessage << "  lambda_min(H)    = " << lamHmin  << std::endl;
+  std::cout << GridLogMessage << "  low sigma census : see CENSUS 4 table above" << std::endl;
+  std::cout << GridLogMessage << "  Compare min sigma with |lambda| from Krylov-Schur (Patrick):" << std::endl;
+  std::cout << GridLogMessage << "    sigma_min ~ min|lambda| : effectively normal; deflation rank is the issue" << std::endl;
+  std::cout << GridLogMessage << "    sigma_min << min|lambda|: non-normal; need two-sided/singular-vector deflation" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_halfplane.cc

@@ -0,0 +1,278 @@
+/*
+ * Example_pvdagm_halfplane.cc
+ *
+ * Standalone fine-operator diagnostic: the EES half-plane margin of the
+ * (non-Hermitian) PV-preconditioned Mobius DWF operator
+ *
+ *     A(m_adj) = D_adj^dag D_light      (D_adj plays the Pauli-Villars role)
+ *
+ * as a function of the adjoint mass m_adj, dialled from the light quark mass
+ * up to the Pauli-Villars mass (=1).  No coarse grid, no subspace, no Lanczos
+ * -- pure power-method spectral tests on the fine grid.
+ *
+ * Purpose: A is the LEFT preconditioner for inverting the light operator.
+ * To solve  D_light X = B  we iterate the preconditioned system
+ *     (D_adj^dag D_light) X = D_adj^dag B ,
+ * whose solution X is independent of m_adj -- only the conditioning and the
+ * iterative convergence change.  m_adj = m_light is the usual CGNR (symmetric
+ * normal equations); m_adj = 1 is the Pauli-Villars preconditioned system.
+ * The sweep asks which m_adj keeps the preconditioned operator well-behaved
+ * (positive-real / EES-guaranteed) while buying the wider spectral range.
+ *
+ * For the Hermitian part  H(A) = (A + A^dag)/2  we measure, per m_adj:
+ *
+ *   lambda_max(H)   -- power method on H
+ *   lambda_min(H)   -- power method on (sI - H) => min Re W(A), the half-plane
+ *                      margin.  EES (Eisenstat-Elman-Schultz 1983, Thm 3.3)
+ *                      GUARANTEES GCR convergence with rate
+ *                          [ 1 - lambda_min(H)^2 / sigma_max^2 ]^{1/2}
+ *                      ONLY when lambda_min(H) > 0 (positive-real / A's field
+ *                      of values in the open right half-plane).  A negative
+ *                      value means the guarantee is lost (not that GCR
+ *                      diverges); the magnitude is then the distance-to-
+ *                      positive-realness, i.e. the shift/deflation needed to
+ *                      recover it.
+ *   sigma_max       -- power method on A^dag A (= A.HermOp)
+ *
+ * Endpoints:
+ *   m_adj = m_light  =>  A = M^dag M, Hermitian PD, positive-real by
+ *                        construction, lambda_min(H) = sigma_min^2 > 0 (the
+ *                        squared / CGNR operator).
+ *   m_adj = 1        =>  A = PV^dag M, the standard PVdagM operator.
+ *
+ * Env: MASS, M5, MOBIUS_B, MOBIUS_C, LS, CONFIG,
+ *      MADJ_LIST (comma separated) OR MADJ_MIN / MADJ_MAX / MADJ_N (geometric).
+ *
+ * Caveat: lambda_min(H) via a shifted power method can be soft when it sits
+ * near zero over a dense low spectrum.  The SIGN and the TREND across m_adj
+ * are the robust signal; confirm an individual near-zero value with a proper
+ * shifted Lanczos if it is load-bearing.
+ */
+
+#include <Grid/Grid.h>
+
+using namespace std;
+using namespace Grid;
+
+//////////////////////////////////////////////////////////////////////
+// A = PV^dag M : Op = _PV.Mdag . _Mat.M ,  AdjOp = _Mat.Mdag . _PV.M
+//////////////////////////////////////////////////////////////////////
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){ // A^dag A
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// H = (A + A^dag)/2 for a general non-Hermitian LinearOperator A.
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class HermitianPartLinOp : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_A;
+public:
+  HermitianPartLinOp(LinearOperatorBase<Field> &A): _A(A) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _A.Op(in,out);      // A in
+    _A.AdjOp(in,tmp);   // A^dag in
+    out = 0.5*(out + tmp);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// s*I - Op : power method on this gives s - lambda_min(Op) for Hermitian Op.
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class ShiftedNegatedOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD s;
+public:
+  ShiftedNegatedOperator(RealD _s, LinearOperatorBase<Field> &Op): _Op(Op), s(_s) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    _Op.HermOp(in,out);
+    out = s*in - out;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  RealD mass = 0.00078;
+  RealD M5   = 1.8;
+  RealD b    = 1.5;
+  RealD c    = 0.5;
+  int   Ls   = 24;
+  std::string config("ckpoint_lat.1000");
+
+  if(getenv("MASS"))     mass = atof(getenv("MASS"));
+  if(getenv("M5"))       M5   = atof(getenv("M5"));
+  if(getenv("MOBIUS_B")) b    = atof(getenv("MOBIUS_B"));
+  if(getenv("MOBIUS_C")) c    = atof(getenv("MOBIUS_C"));
+  if(getenv("LS"))       Ls   = atoi(getenv("LS"));
+  if(getenv("CONFIG"))   config = std::string(getenv("CONFIG"));
+
+  // Adjoint-mass sweep: explicit list, or geometric MADJ_MIN..MADJ_MAX in MADJ_N steps.
+  std::vector<RealD> madj_list;
+  if(getenv("MADJ_LIST")){
+    std::stringstream ss(getenv("MADJ_LIST"));
+    std::string tok;
+    while(std::getline(ss,tok,',')) if(tok.size()) madj_list.push_back(std::stod(tok));
+  } else {
+    int   N  = getenv("MADJ_N")   ? atoi(getenv("MADJ_N"))   : 6;
+    RealD lo = getenv("MADJ_MIN") ? atof(getenv("MADJ_MIN")) : mass;
+    RealD hi = getenv("MADJ_MAX") ? atof(getenv("MADJ_MAX")) : 1.0;
+    GRID_ASSERT(N>=1);
+    for(int i=0;i<N;i++)
+      madj_list.push_back( (N==1) ? lo : lo*std::pow(hi/lo, double(i)/double(N-1)) );
+  }
+
+  // lambda_min(H) is the most-negative eigenvalue; resolved by Chebyshev-filtered
+  // Lanczos on H (a shifted power method cannot separate it from the dense low tail).
+  RealD HalfChebyLo    = getenv("HALF_CHEBY_LO")    ? atof(getenv("HALF_CHEBY_LO"))    : 0.1;
+  RealD HalfChebyHi    = getenv("HALF_CHEBY_HI")    ? atof(getenv("HALF_CHEBY_HI"))    : 0.0; // 0 => auto
+  int   HalfChebyOrder = getenv("HALF_CHEBY_ORDER") ? atoi(getenv("HALF_CHEBY_ORDER")) : 61;
+  // Grid's Chebyshev filter MUST be odd order: only then is the polynomial positive
+  // for x < -1, the region the low/negative modes map to.  An even order flips the
+  // sign there, the filtered operator explodes negative, and the IRL never converges.
+  if(HalfChebyOrder%2==0){ HalfChebyOrder++;
+    std::cout<<GridLogMessage<<"HALF_CHEBY_ORDER forced odd -> "<<HalfChebyOrder<<std::endl; }
+  int   HalfNstop      = getenv("HALF_NSTOP")       ? atoi(getenv("HALF_NSTOP"))       : 8;
+  int   HalfNk         = getenv("HALF_NK")          ? atoi(getenv("HALF_NK"))          : 24;
+  int   HalfNm         = getenv("HALF_NM")          ? atoi(getenv("HALF_NM"))          : 48;
+  RealD HalfTol        = getenv("HALF_TOL")         ? atof(getenv("HALF_TOL"))         : 1.0e-4;
+  int   HalfMaxIt      = getenv("HALF_MAXIT")       ? atoi(getenv("HALF_MAXIT"))       : 20;
+
+  std::vector<int> lat = {48,48,48,96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
+
+  std::cout << GridLogMessage << "PARAM: MASS(light) " << mass << "  M5 " << M5
+            << "  b " << b << "  c " << c << "  Ls " << Ls << std::endl;
+  std::cout << GridLogMessage << "PARAM: CONFIG " << config << std::endl;
+
+  LatticeGaugeField Umu(UGrid);
+  FieldMetaData header;
+  std::cout << GridLogMessage << "Reading gauge field " << config << std::endl;
+  NerscIO::readConfiguration(Umu,header,config);
+
+  // Fixed light operator (never changes across the sweep).
+  MobiusFermionD Dlight(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid, mass, M5, b, c);
+
+  LatticeFermionD x(FGrid);
+
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "FINE HALF-PLANE SWEEP  A(m_adj) = D_adj^dag D_light" << std::endl;
+  std::cout << GridLogMessage << "  m_adj = " << mass << " => M^dag M (positive-real); m_adj = 1 => PVdagM" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+
+  for(auto madj : madj_list){
+
+    MobiusFermionD Dadj(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid, madj, M5, b, c);
+
+    PVdagMLinearOperator<MobiusFermionD,LatticeFermionD> A(Dlight,Dadj); // A = Dadj^dag Dlight
+    HermitianPartLinOp<LatticeFermionD>                  H(A);
+
+    PowerMethod<LatticeFermionD> PM;
+
+    random(RNG5,x); RealD lamHmax = PM(H,x);
+
+    // lambda_min(H): most-negative eigenvalue via Chebyshev-filtered IRL on H.
+    // Cheby(lo,hi) amplifies eigenvalues below lo; with hi>=lambda_max(H) the most
+    // negative mode is amplified hardest, so IRL isolates the true bottom of the
+    // (possibly indefinite) spectrum where the shifted power method could not.
+    RealD fhi = (HalfChebyHi>0.0)? HalfChebyHi : 1.1*lamHmax;
+    Chebyshev<LatticeFermionD>      Cheby(HalfChebyLo,fhi,HalfChebyOrder);
+    FunctionHermOp<LatticeFermionD> OpCheby(Cheby,H);
+    PlainHermOp<LatticeFermionD>    OpPlain(H);
+    ImplicitlyRestartedLanczos<LatticeFermionD> IRL(OpCheby,OpPlain,HalfNstop,HalfNk,HalfNm,HalfTol,HalfMaxIt);
+    std::vector<RealD>           heval(HalfNm);
+    std::vector<LatticeFermionD> hevec(HalfNm,FGrid);
+    int hNconv=0;
+    random(RNG5,x);
+    IRL.calc(heval,hevec,x,hNconv);
+    RealD lamHmin = (hNconv>0) ? heval[0] : 9.99e99;
+    for(int kk=0;kk<hNconv;kk++) lamHmin = std::min(lamHmin, heval[kk]);
+    std::cout << GridLogMessage << "  (IRL H-bottom: " << hNconv
+              << " converged, most-negative eval " << lamHmin << ")" << std::endl;
+
+    random(RNG5,x); RealD sigmax2 = PM(A,x);   // A.HermOp = A^dag A
+    RealD sigmax  = std::sqrt(sigmax2);
+
+    bool  posreal = (lamHmin > 0.0);
+    RealD ratefac = posreal ? std::sqrt(1.0 - lamHmin*lamHmin/sigmax2) : 0.0; // EES per-iter
+    RealD iters8  = (posreal && ratefac < 1.0) ? std::log(1.0e-8)/std::log(ratefac) : 0.0;
+
+    std::cout << GridLogMessage << "HALFPLANE: m_adj " << madj
+              << "  lambda_min(H) " << lamHmin
+              << "  lambda_max(H) " << lamHmax
+              << "  sigma_max " << sigmax
+              << "  positive_real " << (posreal ? "YES" : "NO ")
+              << (posreal
+                    ? ("  EES_rate " + std::to_string(ratefac) + "  EES_iters(1e-8) " + std::to_string(iters8))
+                    : ("  margin_below_zero " + std::to_string(-lamHmin) + " (EES guarantee lost)"))
+              << std::endl;
+  }
+
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "Reading: lambda_min(H) > 0 => EES guarantees GCR at the quoted rate." << std::endl;
+  std::cout << GridLogMessage << "         crossing to < 0 as m_adj -> 1 marks loss of positive-realness." << std::endl;
+  std::cout << GridLogMessage << "         (non-normality: eigenvalues may still be right-half-plane.)" << std::endl;
+  std::cout << GridLogMessage << "Done" << std::endl;
+
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_mrhs_3level.cc

@@ -0,0 +1,613 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_mrhs_3level.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// MultiRHS (valence) THREE-level multigrid for PVdagM.
+//
+// This is exactly the plain three-level algorithm of Example_pvdagm_3level_SVDdefl.cc
+// with L3_DEFL=0 (NO deflation), applied to the enlarged block-diagonal mRHS system:
+// the coarse and coarse-coarse levels run a SINGLE Krylov (one GCR polynomial, inner
+// products summed over rhs) on the packed 6D mrhs fields, so both coarse levels batch
+// through GEMM (MultiGeneralCoarsenedMatrix) -- the valence throughput win at BOTH levels.
+//
+// Level structure (each coarse level is a single-field PGCR on a packed 6D mrhs field):
+//   L1 (fine)         : std::vector<LatticeFermionD>, MrhsPGCRNonHermitian on PVdagM,
+//                       preconditioned by the L1->L2 mrhs V-cycle (MrhsTwoLevelMG).
+//   L2 (coarse)       : 6D mrhs coarse field, PGCR, preconditioned by the L2->L3 mrhs
+//                       V-cycle (MrhsCoarseThreeLevelPrec) -- coarse-coarse correction + coarse smoother.
+//   L3 (coarse-coarse): 6D mrhs coarse-coarse field, PGCR (the innermost solve).
+//
+// RAW-NULL DISCIPLINE (critical -- see project_block_orthogonalise_leak): the L2->L3
+// aggregation MUST be built from RAW fine near-null vectors (pre block-GS).  We take a
+// raw copy of the loaded subspace BEFORE the L1->L2 CoarsenOperator (which block-
+// orthonormalises in place) and project THAT.  Guards print ||<psi|psi> - I||: ~0.23 =
+// content preserved, ~N_coarse = the e_k leak is back.
+//
+// Env: MASS SUBSPACE_FILE NRHS
+//      BLOCK  (dotted, default 2.2.2.2)  BLOCK2 (dotted, default 2.2.3.3)
+//      FineSmootherShift FineSmootherOrder
+//      CoarseSmootherShift CoarseSmootherNstep
+//      CoarseSolverTol CoarseSolverOrder
+//      L3_TOL L3_MAXIT L3_NSTEP
+//      OuterMmax OuterNstep OuterTol
+
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+
+using namespace std;
+using namespace Grid;
+
+RealD FineSmootherShift    = 0.1;
+int   FineSmootherOrder    = 16;
+RealD CoarseSmootherShift  = 0.1;
+int   CoarseSmootherNstep  = 4;
+RealD CoarseSolverTol      = 0.03;
+int   CoarseSolverOrder    = 200;
+RealD L3Tol                = 2.5e-1;
+int   L3MaxIt              = 50;
+int   L3Nstep              = 50;
+RealD OuterTol             = 1.0e-8;
+int   OuterMmax            = 8;
+int   OuterNstep           = 8;
+int   Nrhs                 = 12;
+RealD mass                 = 0.00078;
+
+void ParseEnvironment(void)
+{
+  if(getenv("MASS"))               mass               = atof(getenv("MASS"));
+  if(getenv("FineSmootherShift"))  FineSmootherShift  = atof(getenv("FineSmootherShift"));
+  if(getenv("FineSmootherOrder"))  FineSmootherOrder  = atoi(getenv("FineSmootherOrder"));
+  if(getenv("CoarseSmootherShift"))CoarseSmootherShift= atof(getenv("CoarseSmootherShift"));
+  if(getenv("CoarseSmootherNstep"))CoarseSmootherNstep= atoi(getenv("CoarseSmootherNstep"));
+  if(getenv("CoarseSolverTol"))    CoarseSolverTol    = atof(getenv("CoarseSolverTol"));
+  if(getenv("CoarseSolverOrder"))  CoarseSolverOrder  = atoi(getenv("CoarseSolverOrder"));
+  if(getenv("L3_TOL"))             L3Tol              = atof(getenv("L3_TOL"));
+  if(getenv("L3_MAXIT"))           L3MaxIt            = atoi(getenv("L3_MAXIT"));
+  if(getenv("L3_NSTEP"))           L3Nstep            = atoi(getenv("L3_NSTEP"));
+  if(getenv("OuterTol"))           OuterTol           = atof(getenv("OuterTol"));
+  if(getenv("OuterMmax"))          OuterMmax          = atoi(getenv("OuterMmax"));
+  if(getenv("OuterNstep"))         OuterNstep         = atoi(getenv("OuterNstep"));
+  if(getenv("NRHS"))               Nrhs               = atoi(getenv("NRHS"));
+
+  std::cout << GridLogMessage << "PARAM: MASS               " << mass               << std::endl;
+  std::cout << GridLogMessage << "PARAM: NRHS               " << Nrhs               << std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherShift  " << FineSmootherShift  << std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherOrder  " << FineSmootherOrder  << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift" << CoarseSmootherShift<< std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSmootherNstep" << CoarseSmootherNstep<< std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverTol    " << CoarseSolverTol    << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder  " << CoarseSolverOrder  << std::endl;
+  std::cout << GridLogMessage << "PARAM: L3_TOL             " << L3Tol              << std::endl;
+  std::cout << GridLogMessage << "PARAM: OuterMmax          " << OuterMmax          << std::endl;
+  std::cout << GridLogMessage << "PARAM: OuterNstep         " << OuterNstep         << std::endl;
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  Grid::emptyUserRecord record;
+  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+  SW.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++) SW.writeScidacFieldRecord(subspace[k], record);
+  SW.close();
+#endif
+}
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  Grid::emptyUserRecord record;
+  Grid::ScidacReader SR;
+  SR.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++) SR.readScidacFieldRecord(subspace[k], record);
+  SR.close();
+#endif
+}
+
+//////////////////////////////////////////////////////////////////////
+// A = PV^dag M (non-Hermitian), and shifted variant for smoothers.
+//////////////////////////////////////////////////////////////////////
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat; Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) { assert(0); }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
+  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); }
+  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(in,tmp); _Mat.Mdag(tmp,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ HermOp(in,out); ComplexD d=innerProduct(in,out); n1=real(d); n2=norm2(out); }
+  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat; Matrix &_PV;
+public:
+  RealD shift;
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+  void OpDiag (const Field &in, Field &out) { assert(0); }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
+  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); out = out + shift*in; }
+  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(tmp,out); _Mat.Mdag(in,tmp); out = out + shift*in; }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+// Generic shift wrapper (for the coarse-level smoother on the 6D mrhs coarse operator).
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op; RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : _Op(Op), shift(_shift) {}
+  void OpDiag  (const Field &in, Field &out) { assert(0); }
+  void OpDir   (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out) { assert(0); }
+  void Op      (const Field &in, Field &out) { _Op.Op(in,out);    out = out + shift*in; }
+  void AdjOp   (const Field &in, Field &out) { _Op.AdjOp(in,out); out = out + shift*in; }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp  (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+//////////////////////////////////////////////////////////////////////
+// mrhs interfaces + single-polynomial mrhs PGCR (verbatim from Example_pvdagm_mrhs.cc):
+// reductions summed over rhs -> one alpha/beta per step for the enlarged system.
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class MrhsLinearFunction {
+public:
+  virtual void operator()(std::vector<Field> &in, std::vector<Field> &out) = 0;
+};
+
+template<class Field>
+class MrhsPGCRNonHermitian {
+public:
+  RealD Tolerance; Integer MaxIterations; int mmax,nstep,steps,level;
+  int ZeroGuess = 0; int FirstCycle = 0;   // caller contract: zero guess => first-cycle r0 = src
+  std::string name = "Level 1";
+  LinearOperatorBase<Field> &Linop;
+  MrhsLinearFunction<Field> &Preconditioner;
+  void Level(int lv){ name = "Level " + std::to_string(lv); level=lv; }
+  void Name(std::string n){ name = n; }
+  void SetZeroGuess(int z){ ZeroGuess=z; }
+  MrhsPGCRNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,MrhsLinearFunction<Field> &Prec,int _mmax,int _nstep)
+    : Tolerance(tol),MaxIterations(maxit),Linop(_Linop),Preconditioner(Prec),mmax(_mmax),nstep(_nstep){ level=1; }
+  static RealD vnorm2(std::vector<Field> &x){ RealD s=0; for(auto &f:x) s+=norm2(f); return s; }
+  static ComplexD vinnerProduct(std::vector<Field> &x,std::vector<Field> &y){ ComplexD s(0); for(int r=0;r<(int)x.size();r++) s+=innerProduct(x[r],y[r]); return s; }
+  static void vaxpy(std::vector<Field> &z,ComplexD a,std::vector<Field> &x,std::vector<Field> &y){ for(int r=0;r<(int)z.size();r++) axpy(z[r],a,x[r],y[r]); }
+  void vOp(std::vector<Field> &in,std::vector<Field> &out){ for(int r=0;r<(int)in.size();r++) Linop.Op(in[r],out[r]); }
+  void operator()(std::vector<Field> &src,std::vector<Field> &psi){
+    RealD cp,ssq,rsq; int nrhs=src.size(); GridBase *grid=src[0].Grid();
+    ssq=vnorm2(src); rsq=Tolerance*Tolerance*ssq;
+    std::vector<Field> r(nrhs,grid);
+    GridStopWatch T; T.Start(); steps=0; FirstCycle=1;
+    for(int k=0;k<MaxIterations;k++){
+      cp=GCRnStep(src,psi,rsq);
+      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR("<<mmax<<","<<nstep<<") "<<steps<<" steps cp = "<<cp<<" target "<<rsq<<std::endl;
+      if(cp<rsq){
+        T.Stop(); vOp(psi,r); for(int rr=0;rr<nrhs;rr++) axpy(r[rr],-1.0,src[rr],r[rr]);
+        RealD tr=vnorm2(r);
+        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR: Converged on iteration "<<steps
+                 <<" computed residual "<<std::sqrt(cp/ssq)<<" true residual "<<std::sqrt(tr/ssq)<<" target "<<Tolerance<<std::endl;
+        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR Time elapsed: Total "<<T.Elapsed()<<std::endl;
+        for(int rr=0;rr<nrhs;rr++){ RealD rn=std::sqrt(norm2(r[rr])/norm2(src[rr])); std::cout<<GridLogMessage<<"MrhsPGCR per-rhs true residual["<<rr<<"] = "<<rn<<std::endl; }
+        return;
+      }
+    }
+    std::cout<<GridLogMessage<<"MrhsPGCR: did not converge"<<std::endl;
+  }
+  RealD GCRnStep(std::vector<Field> &src,std::vector<Field> &psi,RealD rsq){
+    RealD cp; ComplexD a,b,rq; RealD zAAz; int nrhs=src.size(); GridBase *grid=src[0].Grid();
+    std::vector<Field> r(nrhs,grid),z(nrhs,grid),Az(nrhs,grid);
+    std::vector< std::vector<Field> > q(mmax,std::vector<Field>(nrhs,grid));
+    std::vector< std::vector<Field> > p(mmax,std::vector<Field>(nrhs,grid));
+    std::vector<RealD> qq(mmax);
+    std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR nStep("<<nstep<<")"<<std::endl;
+    if (ZeroGuess && FirstCycle) { for(int rr=0;rr<nrhs;rr++){ psi[rr]=Zero(); r[rr]=src[rr]; } }
+    else                         { vOp(psi,Az); for(int rr=0;rr<nrhs;rr++) r[rr]=src[rr]-Az[rr]; }
+    FirstCycle=0;
+    Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
+    p[0]=z; q[0]=Az; qq[0]=zAAz; cp=vnorm2(r);
+    for(int k=0;k<nstep;k++){
+      steps++; int kp=k+1, peri_k=k%mmax, peri_kp=kp%mmax;
+      rq=vinnerProduct(q[peri_k],r); a=rq/qq[peri_k];
+      vaxpy(psi,a,p[peri_k],psi); vaxpy(r,-a,q[peri_k],r); cp=vnorm2(r);
+      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR step["<<steps<<"]  resid "<<cp<<" target "<<rsq<<std::endl;
+      if((k==nstep-1)||(cp<rsq)) return cp;
+      Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
+      q[peri_kp]=Az; p[peri_kp]=z;
+      int northog=((kp)>(mmax-1))?(mmax-1):(kp);
+      for(int back=0;back<northog;back++){ int peri_back=(k-back)%mmax; GRID_ASSERT((k-back)>=0);
+        b=-real(vinnerProduct(q[peri_back],Az))/qq[peri_back];
+        vaxpy(p[peri_kp],b,p[peri_back],p[peri_kp]); vaxpy(q[peri_kp],b,q[peri_back],q[peri_kp]); }
+      qq[peri_kp]=vnorm2(q[peri_kp]);
+    }
+    GRID_ASSERT(0); return cp;
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// L2->L3 mrhs V-cycle: a LinearFunction on the 6D mrhs COARSE field.
+// Mirrors Example_pvdagm_mrhs.cc's MrhsTwoLevelMG one level down, and the
+// single-RHS MGPreconditioner of Example_pvdagm_3level_SVDdefl.cc:
+//   out = in (trivial pre)
+//   r = in - A_coarse out
+//   restrict (unpack 6D coarse -> blockProject -> pack 6D coarse-coarse)
+//   ONE coarse-coarse solve (L3, GEMM)
+//   prolong (unpack -> blockPromote -> pack); out += correction
+//   r = in - A_coarse out
+//   coarse smoother (shifted 6D coarse op); out += smooth(r)
+//////////////////////////////////////////////////////////////////////
+template<class CoarseField, class CoarseCoarseField>
+class MrhsCoarseThreeLevelPrec : public LinearFunction<CoarseField> {
+public:
+  LinearOperatorBase<CoarseField>          &_CoarseOp;          // mrhs coarse op (6D)
+  LinearFunction<CoarseField>              &_CoarseSmoother;    // shifted 6D coarse smoother
+  MultiRHSBlockProject<CoarseField>        &_Projector;         // L2->L3 (vector-based)
+  LinearFunction<CoarseCoarseField>        &_CoarseCoarseSolve; // L3 solve (6D cc)
+  GridBase *_Coarse5d, *_CoarseCoarse5d, *_CoarseCoarseMrhs;
+  int _nrhs;
+
+  MrhsCoarseThreeLevelPrec(LinearOperatorBase<CoarseField> &CoarseOp,
+                           LinearFunction<CoarseField> &CoarseSmoother,
+                           MultiRHSBlockProject<CoarseField> &Projector,
+                           LinearFunction<CoarseCoarseField> &CoarseCoarseSolve,
+                           GridBase *Coarse5d, GridBase *CoarseCoarse5d, GridBase *CoarseCoarseMrhs, int nrhs)
+    : _CoarseOp(CoarseOp), _CoarseSmoother(CoarseSmoother), _Projector(Projector),
+      _CoarseCoarseSolve(CoarseCoarseSolve),
+      _Coarse5d(Coarse5d), _CoarseCoarse5d(CoarseCoarse5d), _CoarseCoarseMrhs(CoarseCoarseMrhs), _nrhs(nrhs) {}
+
+  using LinearFunction<CoarseField>::operator();
+  virtual void operator()(const CoarseField &in, CoarseField &out) {
+    int nrhs=_nrhs; double t;
+    CoarseField vec1(in.Grid());
+    CoarseField vec2(in.Grid());
+
+    // trivial pre-smoother
+    out = in;
+
+    // residual (6D coarse)
+    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
+
+    // restrict: unpack 6D coarse -> vector<Coarse> -> blockProject -> vector<CoarseCoarse> -> pack 6D cc
+    std::vector<CoarseField>       csplit(nrhs,_Coarse5d);
+    std::vector<CoarseCoarseField> ccsplit(nrhs,_CoarseCoarse5d);
+    CoarseCoarseField CCsrc(_CoarseCoarseMrhs);
+    CoarseCoarseField CCsol(_CoarseCoarseMrhs);
+
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(csplit[r],vec1,r,0);
+    _Projector.blockProject(csplit,ccsplit);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(ccsplit[r],CCsrc,r,0);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L2->L3 restrict took "<<t/1000.0<<"ms"<<std::endl;
+
+    // L3 solve (6D coarse-coarse, GEMM)
+    t=-usecond();
+    CCsol=Zero();
+    _CoarseCoarseSolve(CCsrc,CCsol);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L3 coarse-coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+
+    // prolong: unpack 6D cc -> blockPromote -> pack 6D coarse; add correction
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(ccsplit[r],CCsol,r,0);
+    _Projector.blockPromote(csplit,ccsplit);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(csplit[r],vec1,r,0);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L2->L3 prolong took "<<t/1000.0<<"ms"<<std::endl;
+
+    // residual + coarse smoother (6D coarse)
+    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
+    vec2=Zero();
+    _CoarseSmoother(vec1,vec2);
+    add(out,out,vec2);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// L1->L2 mrhs V-cycle (verbatim from Example_pvdagm_mrhs.cc):
+// per-rhs fine smoother + batched restriction + ONE coarse solve + batched prolong.
+// The coarse solve passed in is now itself three-level (preconditioned by L2->L3).
+//////////////////////////////////////////////////////////////////////
+template<class FineField, class MrhsCoarseVector, class FineSmoother>
+class MrhsTwoLevelMG : public MrhsLinearFunction<FineField> {
+public:
+  typedef MrhsCoarseVector CoarseVector;
+  LinearOperatorBase<FineField>   &_FineOperator;
+  FineSmoother                    &_PostSmoother;
+  MultiRHSBlockProject<FineField> &_Projector;
+  LinearFunction<CoarseVector>    &_CoarseSolve;
+  GridBase *_CoarseGrid, *_CoarseGridMrhs;
+  MrhsTwoLevelMG(LinearOperatorBase<FineField> &FineOp, FineSmoother &Post,
+                 MultiRHSBlockProject<FineField> &Projector, LinearFunction<CoarseVector> &CoarseSolve,
+                 GridBase *CoarseGrid, GridBase *CoarseGridMrhs)
+    : _FineOperator(FineOp),_PostSmoother(Post),_Projector(Projector),_CoarseSolve(CoarseSolve),
+      _CoarseGrid(CoarseGrid),_CoarseGridMrhs(CoarseGridMrhs){}
+  virtual void operator()(std::vector<FineField> &in, std::vector<FineField> &out){
+    int nrhs=in.size(); GridBase *fgrid=in[0].Grid(); double t;
+    std::vector<FineField> vec1(nrhs,fgrid),vec2(nrhs,fgrid);
+    for(int r=0;r<nrhs;r++) out[r]=in[r];
+    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
+    std::vector<CoarseVector> Csrc_split(nrhs,_CoarseGrid), Csol_split(nrhs,_CoarseGrid);
+    CoarseVector CsrcMrhs(_CoarseGridMrhs), CsolMrhs(_CoarseGridMrhs);
+    t=-usecond();
+    _Projector.blockProject(vec1,Csrc_split);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(Csrc_split[r],CsrcMrhs,r,0);
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs project+pack took "<<t/1000.0<<"ms"<<std::endl;
+    t=-usecond(); CsolMrhs=Zero(); _CoarseSolve(CsrcMrhs,CsolMrhs); t+=usecond();
+    std::cout<<GridLogMessage<<"Mrhs coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(Csol_split[r],CsolMrhs,r,0);
+    _Projector.blockPromote(vec1,Csol_split);
+    for(int r=0;r<nrhs;r++) add(out[r],out[r],vec1[r]);
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs unpack+promote took "<<t/1000.0<<"ms"<<std::endl;
+    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
+    t=-usecond();
+    for(int r=0;r<nrhs;r++){ vec2[r]=Zero(); _PostSmoother(vec1[r],vec2[r]); add(out[r],out[r],vec2[r]); }
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs post-smooth took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+  ParseEnvironment();
+
+  const int Ls=24; RealD M5=1.8, b=1.5, c=0.5;
+  const int nbasis=60; const int nrhs=Nrhs;
+  GRID_ASSERT(nrhs % vComplex::Nsimd() == 0);
+
+  std::vector<int> lat_size {48,48,48,96};
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 blocking (default 2^4)
+  Coordinate clatt = lat_size;
+  Coordinate Block({2,2,2,2});
+  if ( getenv("BLOCK") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block); GRID_ASSERT(Block.size()==4); }
+  for(int d=0;d<4;d++){ GRID_ASSERT(lat_size[d]%Block[d]==0); clatt[d]=lat_size[d]/Block[d]; }
+  std::cout << GridLogMessage << "Block  " << Block  << "  coarse lattice       " << clatt << std::endl;
+
+  // Level 2 blocking (default 2,2,3,3)  -- matches Example_pvdagm_3level_SVDdefl
+  Coordinate cclatt = clatt;
+  Coordinate Block2({2,2,3,3});
+  if ( getenv("BLOCK2") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK2")),Block2); GRID_ASSERT(Block2.size()==4); }
+  for(int d=0;d<4;d++){ GRID_ASSERT(clatt[d]%Block2[d]==0); cclatt[d]=clatt[d]/Block2[d]; }
+  std::cout << GridLogMessage << "Block2 " << Block2 << "  coarse-coarse lattice " << cclatt << std::endl;
+
+  GridCartesian *Coarse4d = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(cclatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // 6D mrhs grids: rhs is dim 0, SIMD across rhs (pattern: Test_general_coarse_hdcg_phys48.cc)
+  Coordinate mpi=GridDefaultMpi();
+  Coordinate rhMpi ({1,1,mpi[0],mpi[1],mpi[2],mpi[3]});
+  Coordinate rhSimd({vComplex::Nsimd(),1,1,1,1,1});
+  Coordinate rhLatt ({nrhs,1,clatt[0], clatt[1], clatt[2], clatt[3]});
+  Coordinate rhLatt2({nrhs,1,cclatt[0],cclatt[1],cclatt[2],cclatt[3]});
+  GridCartesian *CoarseMrhs       = new GridCartesian(rhLatt, rhSimd,rhMpi);
+  GridCartesian *CoarseCoarseMrhs = new GridCartesian(rhLatt2,rhSimd,rhMpi);
+
+  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+
+  // Level 1 tensor types
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>       LittleDiracOperator;
+  typedef MultiGeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  MrhsLittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                                CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>                  Subspace;
+
+  // Level 2 tensor types (coarsening deepens the nest by one iScalar -- see CLAUDE.md)
+  typedef CoarseVector::vector_object                                      CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>          LittleDiracOperatorL2;
+  typedef MultiGeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>     MrhsLittleDiracOperatorL2;
+  typedef LittleDiracOperatorL2::CoarseVector                              CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,nbasis>                     SubspaceL2;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom (Coarse5d);
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);   // 33-point at L2->L3, matching SVDdefl
+
+  //////////////////////////////////////////////////////////////////////
+  // Subspace: load RAW (no Orthogonalise!), or generate.
+  //////////////////////////////////////////////////////////////////////
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                            + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+  uint64_t file_exists=0;
+  if ( UGrid->IsBoss() ){ std::ifstream f(subspace_file); file_exists=f.good()?1:0; }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb=0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+  if ( file_exists ){
+    std::cout << GridLogMessage << "*** Loading subspace from disk (kept RAW) ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+  }
+
+  // RAW copy of the fine null vectors BEFORE CoarsenOperator block-orthonormalises in place.
+  std::vector<LatticeFermionD> rawNull(nbasis,FGrid);
+  for(int k=0;k<nbasis;k++) rawNull[k]=AggregatesGCR.subspace[k];
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarsen L1->L2 and L2->L3 with SINGLE-RHS machinery, import into the mrhs
+  // operators via CopyMatrix.  The single-RHS L1->L2 coarse operator must stay
+  // alive to be the "fine" operator for the L2->L3 coarsening, so BOTH single-RHS
+  // ops (and their padded _A) live in one scope and free together.  [MEMORY: this
+  // is the setup peak -- L1->L2 padded _A (~large at 2^4) + L2->L3 padded _A.]
+  //////////////////////////////////////////////////////////////////////
+  MrhsLittleDiracOperator   mrhsLittleDiracOpPV(geom,  CoarseMrhs);
+  MrhsLittleDiracOperatorL2 mrhsLittleDiracOpL2(geom2, CoarseCoarseMrhs);
+  MultiRHSBlockProject<LatticeFermionD> MrhsProjector;
+  MultiRHSBlockProject<CoarseVector>    MrhsProjectorL2;
+  {
+    // --- L1->L2 single-RHS coarse operator (kept alive for the L2->L3 coarsening) ---
+    LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d);
+    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);   // orthonormalises AggregatesGCR.subspace in place
+    mrhsLittleDiracOpPV.CopyMatrix(LittleDiracOpPV);
+    MrhsProjector.Allocate(nbasis,FGrid,Coarse5d);
+    MrhsProjector.ImportBasis(AggregatesGCR.subspace);       // orthonormalised, matches the coarse op
+    NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+    // --- psi_coarse = P^dag (RAW fine null) -> Galerkin images, NOT e_k ---
+    std::vector<CoarseVector> psi_coarse(nbasis,Coarse5d);
+    for(int k=0;k<nbasis;k++) AggregatesGCR.ProjectToSubspace(psi_coarse[k], rawNull[k]);
+    rawNull.clear(); rawNull.shrink_to_fit();
+    {
+      RealD s2=0.0;
+      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
+        ComplexD sij=TensorRemove(innerProduct(psi_coarse[i],psi_coarse[j]));
+        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
+      }
+      std::cout<<GridLogMessage<<"GUARD: ||<psi_coarse|psi_coarse> - I||_F = "<<std::sqrt(s2)
+               <<"   (~0.23 good; ~sqrt(N_coarse)="<<std::sqrt((double)Coarse5d->gSites())<<" = e_k leak)"<<std::endl;
+    }
+
+    // --- L2->L3 single-RHS coarsening (coarsen the single-RHS LinOpCoarse) ---
+    SubspaceL2 AggregatesL2(CoarseCoarse5d,Coarse5d,cb);
+    for(int k=0;k<nbasis;k++) AggregatesL2.subspace[k]=psi_coarse[k];
+    LittleDiracOperatorL2 LittleDiracOpL2(geom2,Coarse5d,CoarseCoarse5d);
+    LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+    mrhsLittleDiracOpL2.CopyMatrix(LittleDiracOpL2);
+    MrhsProjectorL2.Allocate(nbasis,Coarse5d,CoarseCoarse5d);
+    MrhsProjectorL2.ImportBasis(AggregatesL2.subspace);      // orthonormalised by CoarsenOperator
+
+    // --- guard psi_cc (RAW psi_coarse; AggregatesL2 holds a separate orthonormalised copy) ---
+    {
+      std::vector<CoarseCoarseVector> psi_cc(nbasis,CoarseCoarse5d);
+      for(int k=0;k<nbasis;k++) AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+      RealD s2=0.0;
+      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
+        ComplexD sij=TensorRemove(innerProduct(psi_cc[i],psi_cc[j]));
+        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
+      }
+      std::cout<<GridLogMessage<<"GUARD: ||<psi_cc|psi_cc> - I||_F = "<<std::sqrt(s2)
+               <<"   (~0.23 good; ~sqrt(N_cc)="<<std::sqrt((double)CoarseCoarse5d->gSites())<<" = e_k leak)"<<std::endl;
+    }
+  } // both single-RHS ops + padded _A + AggregatesL2 + psi_coarse freed here
+
+  NonHermitianLinearOperator<MrhsLittleDiracOperator,CoarseVector>         mrhsLinOpCoarse(mrhsLittleDiracOpPV);
+  NonHermitianLinearOperator<MrhsLittleDiracOperatorL2,CoarseCoarseVector> mrhsLinOpCC(mrhsLittleDiracOpL2);
+
+  //////////////////////////////////////////////////////////////////////
+  // Solvers, innermost first.
+  //////////////////////////////////////////////////////////////////////
+  TrivialPrecon<CoarseVector>       simpleC;
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+  TrivialPrecon<LatticeFermionD>    simple_fine;
+
+  // L3 (coarse-coarse) solve: PGCR on the 6D cc operator
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    L3PGCR(L3Tol,L3MaxIt,mrhsLinOpCC,simpleCC,L3Nstep,L3Nstep);
+  L3PGCR.Level(3);
+  L3PGCR.Name("CCouter");
+  L3PGCR.SetZeroGuess(1);              // caller zeroes CCsol
+
+  // L2 coarse smoother: shifted 6D coarse op, fixed nstep
+  ShiftedLinearOperator<CoarseVector> ShiftedMrhsCoarse(CoarseSmootherShift, mrhsLinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
+    CoarseSmootherGCR(0.01,1,ShiftedMrhsCoarse,simpleC,CoarseSmootherNstep,CoarseSmootherNstep);
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+  CoarseSmootherGCR.SetZeroGuess(1);   // caller zeroes vec2
+
+  // L2->L3 V-cycle preconditioner (operates on 6D coarse field)
+  MrhsCoarseThreeLevelPrec<CoarseVector,CoarseCoarseVector>
+    L2to3Precon(mrhsLinOpCoarse, CoarseSmootherGCR, MrhsProjectorL2, L3PGCR,
+                Coarse5d, CoarseCoarse5d, CoarseCoarseMrhs, nrhs);
+
+  // L2 coarse solve: PGCR on 6D coarse op, preconditioned by the L2->L3 V-cycle
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
+    L2PGCR(CoarseSolverTol, CoarseSolverOrder/16, mrhsLinOpCoarse, L2to3Precon, 16, 16);
+  L2PGCR.Level(2);
+  L2PGCR.Name("Couter");
+  L2PGCR.SetZeroGuess(1);              // caller zeroes CsolMrhs
+
+  // Fine smoother (per-rhs, looped in the L1->L2 V-cycle)
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD>
+    SmootherGCR(0.0,1,ShiftedPVdagM,simple_fine,FineSmootherOrder,FineSmootherOrder);
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+  SmootherGCR.SetZeroGuess(1);         // caller zeroes vec2[r]
+
+  // L1->L2 V-cycle (fine); its coarse solve is the three-level L2PGCR
+  typedef PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> FineSmoother_t;
+  MrhsTwoLevelMG<LatticeFermionD,CoarseVector,FineSmoother_t>
+    ThreeLevelPrecon(PVdagM, SmootherGCR, MrhsProjector, L2PGCR, Coarse5d, CoarseMrhs);
+
+  // Outer mrhs solve
+  MrhsPGCRNonHermitian<LatticeFermionD>
+    L1PGCR(OuterTol,1000,PVdagM,ThreeLevelPrecon,OuterMmax,OuterNstep);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+  L1PGCR.SetZeroGuess(1);              // sol[r]=Zero() at source setup
+
+  //////////////////////////////////////////////////////////////////////
+  // Sources and solve
+  //////////////////////////////////////////////////////////////////////
+  std::vector<LatticeFermionD> src(nrhs,FGrid), sol(nrhs,FGrid);
+  for(int r=0;r<nrhs;r++){ gaussian(RNG5,src[r]); sol[r]=Zero(); }
+
+  std::cout << GridLogMessage << "**********************************************" << std::endl;
+  std::cout << GridLogMessage << " MultiRHS THREE-level solve: " << nrhs << " RHS " << std::endl;
+  std::cout << GridLogMessage << "**********************************************" << std::endl;
+
+  GridStopWatch w; w.Start();
+  L1PGCR(src,sol);
+  w.Stop();
+  std::cout << GridLogMessage << "MultiRHS 3-level solve total " << w.Elapsed()
+            << "  (per RHS: " << w.useconds()/1.0e6/nrhs << " s)" << std::endl;
+
+  { LatticeFermionD Ax(FGrid); RealD worst=0.0;
+    for(int r=0;r<nrhs;r++){ PVdagM.Op(sol[r],Ax); Ax=Ax-src[r];
+      RealD rn=std::sqrt(norm2(Ax)/norm2(src[r]));
+      std::cout << GridLogMessage << "FINAL: rhs["<<r<<"] true residual = " << rn << std::endl;
+      worst=std::max(worst,rn); }
+    std::cout << GridLogMessage << "FINAL: worst-case residual = " << worst << std::endl;
+  }
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_mrhs_3level_DenseCoarseMatrix.cc

@@ -0,0 +1,663 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_mrhs_3level_DenseCoarseMatrix.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// MultiRHS (valence) THREE-level multigrid for PVdagM with a DENSE, EXACT,
+// non-iterative coarse-coarse bottom -- the LIBRARY-CLASS successor of
+// Example_pvdagm_mrhs_3level_dense.cc, which is FROZEN as the regression
+// baseline / champion-provenance artifact (21.7 s/RHS at BLOCK=2.2.3.3,
+// BLOCK2=8.4.2.4, nb60, CSO3/FSO6/CST0.04 on 36 Frontier nodes).
+//
+// The dense bottom is now Grid/algorithms/multigrid/DenseCoarseMatrix.h:
+//  - stencil -> dense DIRECT import (no probe assembly: rows are local data)
+//    + IMPORT CERTIFICATE  (DENSE_IMPORT_SIGN=-1 flips convention, no rebuild)
+//  - split-K apply via GridBLAS.gemmBatched with explicit leading dimensions
+//    (DENSE_SPLITK chunks, default 32) -- the fig-11 software split-K
+//  - deviceVector / GridBLAS throughout the apply: platform-agnostic
+//
+// INTERCHANGE: same SLAB_FILE per-rank format as the frozen example (stem MUST
+// encode cfg/mass/blocking/nbasis; the header guards only N/nrows/nbasis) and
+// the same env-var set, so existing sbatch scripts drive either binary.
+//
+// A/B acceptance (old binary = control):
+//   slab-cached  : outer counts match EXACTLY (identical apply data; split-K
+//                  changes only fp32 reduction order); wall delta = split-K gain.
+//   fresh setup  : outer equal-or-+-1 (import vs probe = rounding); VERIFY
+//                  ~7e-4 both; setup delta = import gain (~93 s probe retired).
+//
+// Level structure, solvers, and tuning knobs are UNCHANGED from the frozen
+// example.  Env: MASS SUBSPACE_FILE NRHS BLOCK BLOCK2 FineSmootherShift/Order
+// CoarseSmootherShift/Nstep CoarseSolverTol/Order DENSE_CC DENSE_CC_CHECK
+// DENSE_SPLITK DENSE_DEVICE_SUM DENSE_IMPORT_SIGN DENSE_APPLY_PROFILE
+// L3_TOL L3_MAXIT L3_NSTEP OuterMmax OuterNstep OuterTol
+
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/multigrid/DenseCoarseMatrix.h>
+
+#include <memory>
+
+using namespace std;
+using namespace Grid;
+
+RealD FineSmootherShift    = 0.1;
+int   FineSmootherOrder    = 16;
+RealD CoarseSmootherShift  = 0.1;
+int   CoarseSmootherNstep  = 4;
+RealD CoarseSolverTol      = 0.03;
+int   CoarseSolverOrder    = 200;
+RealD L3Tol                = 2.5e-1;
+int   L3MaxIt              = 50;
+int   L3Nstep              = 50;
+RealD OuterTol             = 1.0e-8;
+int   OuterMmax            = 8;
+int   OuterNstep           = 8;
+int   Nrhs                 = 12;
+int   UseDenseCC           = 1;
+RealD mass                 = 0.00078;
+
+void ParseEnvironment(void)
+{
+  if(getenv("MASS"))               mass               = atof(getenv("MASS"));
+  if(getenv("FineSmootherShift"))  FineSmootherShift  = atof(getenv("FineSmootherShift"));
+  if(getenv("FineSmootherOrder"))  FineSmootherOrder  = atoi(getenv("FineSmootherOrder"));
+  if(getenv("CoarseSmootherShift"))CoarseSmootherShift= atof(getenv("CoarseSmootherShift"));
+  if(getenv("CoarseSmootherNstep"))CoarseSmootherNstep= atoi(getenv("CoarseSmootherNstep"));
+  if(getenv("CoarseSolverTol"))    CoarseSolverTol    = atof(getenv("CoarseSolverTol"));
+  if(getenv("CoarseSolverOrder"))  CoarseSolverOrder  = atoi(getenv("CoarseSolverOrder"));
+  if(getenv("L3_TOL"))             L3Tol              = atof(getenv("L3_TOL"));
+  if(getenv("L3_MAXIT"))           L3MaxIt            = atoi(getenv("L3_MAXIT"));
+  if(getenv("L3_NSTEP"))           L3Nstep            = atoi(getenv("L3_NSTEP"));
+  if(getenv("OuterTol"))           OuterTol           = atof(getenv("OuterTol"));
+  if(getenv("OuterMmax"))          OuterMmax          = atoi(getenv("OuterMmax"));
+  if(getenv("OuterNstep"))         OuterNstep         = atoi(getenv("OuterNstep"));
+  if(getenv("NRHS"))               Nrhs               = atoi(getenv("NRHS"));
+  if(getenv("DENSE_CC"))           UseDenseCC         = atoi(getenv("DENSE_CC"));
+
+  std::cout << GridLogMessage << "PARAM: MASS               " << mass               << std::endl;
+  std::cout << GridLogMessage << "PARAM: NRHS               " << Nrhs               << std::endl;
+  std::cout << GridLogMessage << "PARAM: DENSE_CC           " << UseDenseCC         << std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherShift  " << FineSmootherShift  << std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherOrder  " << FineSmootherOrder  << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift" << CoarseSmootherShift<< std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSmootherNstep" << CoarseSmootherNstep<< std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverTol    " << CoarseSolverTol    << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder  " << CoarseSolverOrder  << std::endl;
+  std::cout << GridLogMessage << "PARAM: L3_TOL             " << L3Tol              << std::endl;
+  std::cout << GridLogMessage << "PARAM: OuterMmax          " << OuterMmax          << std::endl;
+  std::cout << GridLogMessage << "PARAM: OuterNstep         " << OuterNstep         << std::endl;
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  Grid::emptyUserRecord record;
+  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+  SW.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++) SW.writeScidacFieldRecord(subspace[k], record);
+  SW.close();
+#endif
+}
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  Grid::emptyUserRecord record;
+  Grid::ScidacReader SR;
+  SR.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++) SR.readScidacFieldRecord(subspace[k], record);
+  SR.close();
+#endif
+}
+
+//////////////////////////////////////////////////////////////////////
+// A = PV^dag M (non-Hermitian), and shifted variant for smoothers.
+//////////////////////////////////////////////////////////////////////
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat; Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) { assert(0); }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
+  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); }
+  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(in,tmp); _Mat.Mdag(tmp,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ HermOp(in,out); ComplexD d=innerProduct(in,out); n1=real(d); n2=norm2(out); }
+  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat; Matrix &_PV;
+public:
+  RealD shift;
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+  void OpDiag (const Field &in, Field &out) { assert(0); }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
+  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); out = out + shift*in; }
+  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(tmp,out); _Mat.Mdag(in,tmp); out = out + shift*in; }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+// Generic shift wrapper (for the coarse-level smoother on the 6D mrhs coarse operator).
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op; RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : _Op(Op), shift(_shift) {}
+  void OpDiag  (const Field &in, Field &out) { assert(0); }
+  void OpDir   (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out) { assert(0); }
+  void Op      (const Field &in, Field &out) { _Op.Op(in,out);    out = out + shift*in; }
+  void AdjOp   (const Field &in, Field &out) { _Op.AdjOp(in,out); out = out + shift*in; }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp  (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+//////////////////////////////////////////////////////////////////////
+// Dense CC solve on the PACKED 6D mrhs coarse-coarse field: drop-in
+// for the L3 PGCR, delegating to the library DenseCoarseMatrix.
+//////////////////////////////////////////////////////////////////////
+template<class DenseType, class CoarseCoarseField>
+class MrhsDenseCCSolve : public LinearFunction<CoarseCoarseField> {
+public:
+  DenseType &_Dense;
+  GridBase *_CoarseCoarse5d;
+  int _nrhs;
+  MrhsDenseCCSolve(DenseType &D, GridBase *cc5d, int nrhs)
+    : _Dense(D), _CoarseCoarse5d(cc5d), _nrhs(nrhs) {}
+  using LinearFunction<CoarseCoarseField>::operator();
+  virtual void operator()(const CoarseCoarseField &in, CoarseCoarseField &out){
+    if ( getenv("DENSE_CC_CHECK") ) {
+      // Audit path: per-rhs 5D unpack so ApplyBatch can run the _Op defect
+      // check per rhs.  ~50ms/call of slice/split overhead -- audit only.
+      CoarseCoarseField tmp(in.Grid());
+      tmp = in;
+      std::vector<CoarseCoarseField> split_in (_nrhs,_CoarseCoarse5d);
+      std::vector<CoarseCoarseField> split_out(_nrhs,_CoarseCoarse5d);
+      for(int r=0;r<_nrhs;r++) ExtractSliceFast(split_in[r], tmp, r, 0);
+      _Dense.ApplyBatch(split_in, split_out);
+      for(int r=0;r<_nrhs;r++) InsertSliceFast(split_out[r], out, r, 0);
+    } else {
+      _Dense.ApplyBatch6D(in, out, _nrhs);
+    }
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// mrhs interfaces + single-polynomial mrhs PGCR (verbatim from the
+// frozen Example_pvdagm_mrhs_3level_dense.cc)
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class MrhsLinearFunction {
+public:
+  virtual void operator()(std::vector<Field> &in, std::vector<Field> &out) = 0;
+};
+
+template<class Field>
+class MrhsPGCRNonHermitian {
+public:
+  RealD Tolerance; Integer MaxIterations; int mmax,nstep,steps,level;
+  int ZeroGuess = 0; int FirstCycle = 0;   // caller contract: zero guess => first-cycle r0 = src
+  std::string name = "Level 1";
+  LinearOperatorBase<Field> &Linop;
+  MrhsLinearFunction<Field> &Preconditioner;
+  void Level(int lv){ name = "Level " + std::to_string(lv); level=lv; }
+  void Name(std::string n){ name = n; }
+  void SetZeroGuess(int z){ ZeroGuess=z; }
+  MrhsPGCRNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,MrhsLinearFunction<Field> &Prec,int _mmax,int _nstep)
+    : Tolerance(tol),MaxIterations(maxit),Linop(_Linop),Preconditioner(Prec),mmax(_mmax),nstep(_nstep){ level=1; }
+  static RealD vnorm2(std::vector<Field> &x){ RealD s=0; for(auto &f:x) s+=norm2(f); return s; }
+  static ComplexD vinnerProduct(std::vector<Field> &x,std::vector<Field> &y){ ComplexD s(0); for(int r=0;r<(int)x.size();r++) s+=innerProduct(x[r],y[r]); return s; }
+  static void vaxpy(std::vector<Field> &z,ComplexD a,std::vector<Field> &x,std::vector<Field> &y){ for(int r=0;r<(int)z.size();r++) axpy(z[r],a,x[r],y[r]); }
+  void vOp(std::vector<Field> &in,std::vector<Field> &out){ for(int r=0;r<(int)in.size();r++) Linop.Op(in[r],out[r]); }
+  void operator()(std::vector<Field> &src,std::vector<Field> &psi){
+    RealD cp,ssq,rsq; int nrhs=src.size(); GridBase *grid=src[0].Grid();
+    ssq=vnorm2(src); rsq=Tolerance*Tolerance*ssq;
+    std::vector<Field> r(nrhs,grid);
+    GridStopWatch T; T.Start(); steps=0; FirstCycle=1;
+    for(int k=0;k<MaxIterations;k++){
+      cp=GCRnStep(src,psi,rsq);
+      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR("<<mmax<<","<<nstep<<") "<<steps<<" steps cp = "<<cp<<" target "<<rsq<<std::endl;
+      if(cp<rsq){
+        T.Stop(); vOp(psi,r); for(int rr=0;rr<nrhs;rr++) axpy(r[rr],-1.0,src[rr],r[rr]);
+        RealD tr=vnorm2(r);
+        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR: Converged on iteration "<<steps
+                 <<" computed residual "<<std::sqrt(cp/ssq)<<" true residual "<<std::sqrt(tr/ssq)<<" target "<<Tolerance<<std::endl;
+        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR Time elapsed: Total "<<T.Elapsed()<<std::endl;
+        for(int rr=0;rr<nrhs;rr++){ RealD rn=std::sqrt(norm2(r[rr])/norm2(src[rr])); std::cout<<GridLogMessage<<"MrhsPGCR per-rhs true residual["<<rr<<"] = "<<rn<<std::endl; }
+        return;
+      }
+    }
+    std::cout<<GridLogMessage<<"MrhsPGCR: did not converge"<<std::endl;
+  }
+  RealD GCRnStep(std::vector<Field> &src,std::vector<Field> &psi,RealD rsq){
+    RealD cp; ComplexD a,b,rq; RealD zAAz; int nrhs=src.size(); GridBase *grid=src[0].Grid();
+    std::vector<Field> r(nrhs,grid),z(nrhs,grid),Az(nrhs,grid);
+    std::vector< std::vector<Field> > q(mmax,std::vector<Field>(nrhs,grid));
+    std::vector< std::vector<Field> > p(mmax,std::vector<Field>(nrhs,grid));
+    std::vector<RealD> qq(mmax);
+    std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR nStep("<<nstep<<")"<<std::endl;
+    if (ZeroGuess && FirstCycle) { for(int rr=0;rr<nrhs;rr++){ psi[rr]=Zero(); r[rr]=src[rr]; } }
+    else                         { vOp(psi,Az); for(int rr=0;rr<nrhs;rr++) r[rr]=src[rr]-Az[rr]; }
+    FirstCycle=0;
+    Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
+    p[0]=z; q[0]=Az; qq[0]=zAAz; cp=vnorm2(r);
+    for(int k=0;k<nstep;k++){
+      steps++; int kp=k+1, peri_k=k%mmax, peri_kp=kp%mmax;
+      rq=vinnerProduct(q[peri_k],r); a=rq/qq[peri_k];
+      vaxpy(psi,a,p[peri_k],psi); vaxpy(r,-a,q[peri_k],r); cp=vnorm2(r);
+      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR step["<<steps<<"]  resid "<<cp<<" target "<<rsq<<std::endl;
+      if((k==nstep-1)||(cp<rsq)) return cp;
+      Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
+      q[peri_kp]=Az; p[peri_kp]=z;
+      int northog=((kp)>(mmax-1))?(mmax-1):(kp);
+      for(int back=0;back<northog;back++){ int peri_back=(k-back)%mmax; GRID_ASSERT((k-back)>=0);
+        b=-real(vinnerProduct(q[peri_back],Az))/qq[peri_back];
+        vaxpy(p[peri_kp],b,p[peri_back],p[peri_kp]); vaxpy(q[peri_kp],b,q[peri_back],q[peri_kp]); }
+      qq[peri_kp]=vnorm2(q[peri_kp]);
+    }
+    GRID_ASSERT(0); return cp;
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// L2->L3 mrhs V-cycle: LinearFunction on the 6D mrhs COARSE field.
+// The coarse-coarse solve slot takes EITHER the dense mrhs solve
+// (DENSE_CC=1) or the L3 PGCR (DENSE_CC=0).
+//////////////////////////////////////////////////////////////////////
+template<class CoarseField, class CoarseCoarseField>
+class MrhsCoarseThreeLevelPrec : public LinearFunction<CoarseField> {
+public:
+  LinearOperatorBase<CoarseField>          &_CoarseOp;          // mrhs coarse op (6D)
+  LinearFunction<CoarseField>              &_CoarseSmoother;    // shifted 6D coarse smoother
+  MultiRHSBlockProject<CoarseField>        &_Projector;         // L2->L3 (vector-based)
+  LinearFunction<CoarseCoarseField>        &_CoarseCoarseSolve; // L3 solve (6D cc)
+  GridBase *_Coarse5d, *_CoarseCoarse5d, *_CoarseCoarseMrhs;
+  int _nrhs;
+
+  MrhsCoarseThreeLevelPrec(LinearOperatorBase<CoarseField> &CoarseOp,
+                           LinearFunction<CoarseField> &CoarseSmoother,
+                           MultiRHSBlockProject<CoarseField> &Projector,
+                           LinearFunction<CoarseCoarseField> &CoarseCoarseSolve,
+                           GridBase *Coarse5d, GridBase *CoarseCoarse5d, GridBase *CoarseCoarseMrhs, int nrhs)
+    : _CoarseOp(CoarseOp), _CoarseSmoother(CoarseSmoother), _Projector(Projector),
+      _CoarseCoarseSolve(CoarseCoarseSolve),
+      _Coarse5d(Coarse5d), _CoarseCoarse5d(CoarseCoarse5d), _CoarseCoarseMrhs(CoarseCoarseMrhs), _nrhs(nrhs) {}
+
+  using LinearFunction<CoarseField>::operator();
+  virtual void operator()(const CoarseField &in, CoarseField &out) {
+    int nrhs=_nrhs; double t;
+    CoarseField vec1(in.Grid());
+    CoarseField vec2(in.Grid());
+
+    // trivial pre-smoother
+    out = in;
+
+    // residual (6D coarse)
+    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
+
+    // restrict: unpack 6D coarse -> vector<Coarse> -> blockProject -> vector<CoarseCoarse> -> pack 6D cc
+    std::vector<CoarseField>       csplit(nrhs,_Coarse5d);
+    std::vector<CoarseCoarseField> ccsplit(nrhs,_CoarseCoarse5d);
+    CoarseCoarseField CCsrc(_CoarseCoarseMrhs);
+    CoarseCoarseField CCsol(_CoarseCoarseMrhs);
+
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(csplit[r],vec1,r,0);
+    _Projector.blockProject(csplit,ccsplit);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(ccsplit[r],CCsrc,r,0);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L2->L3 restrict took "<<t/1000.0<<"ms"<<std::endl;
+
+    // L3 solve (dense mrhs GEMM, or PGCR)
+    t=-usecond();
+    CCsol=Zero();
+    _CoarseCoarseSolve(CCsrc,CCsol);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L3 coarse-coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+
+    // prolong: unpack 6D cc -> blockPromote -> pack 6D coarse; add correction
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(ccsplit[r],CCsol,r,0);
+    _Projector.blockPromote(csplit,ccsplit);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(csplit[r],vec1,r,0);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L2->L3 prolong took "<<t/1000.0<<"ms"<<std::endl;
+
+    // residual + coarse smoother (6D coarse)
+    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
+    vec2=Zero();
+    _CoarseSmoother(vec1,vec2);
+    add(out,out,vec2);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// L1->L2 mrhs V-cycle (verbatim from the frozen example)
+//////////////////////////////////////////////////////////////////////
+template<class FineField, class MrhsCoarseVector, class FineSmoother>
+class MrhsTwoLevelMG : public MrhsLinearFunction<FineField> {
+public:
+  typedef MrhsCoarseVector CoarseVector;
+  LinearOperatorBase<FineField>   &_FineOperator;
+  FineSmoother                    &_PostSmoother;
+  MultiRHSBlockProject<FineField> &_Projector;
+  LinearFunction<CoarseVector>    &_CoarseSolve;
+  GridBase *_CoarseGrid, *_CoarseGridMrhs;
+  MrhsTwoLevelMG(LinearOperatorBase<FineField> &FineOp, FineSmoother &Post,
+                 MultiRHSBlockProject<FineField> &Projector, LinearFunction<CoarseVector> &CoarseSolve,
+                 GridBase *CoarseGrid, GridBase *CoarseGridMrhs)
+    : _FineOperator(FineOp),_PostSmoother(Post),_Projector(Projector),_CoarseSolve(CoarseSolve),
+      _CoarseGrid(CoarseGrid),_CoarseGridMrhs(CoarseGridMrhs){}
+  virtual void operator()(std::vector<FineField> &in, std::vector<FineField> &out){
+    int nrhs=in.size(); GridBase *fgrid=in[0].Grid(); double t;
+    std::vector<FineField> vec1(nrhs,fgrid),vec2(nrhs,fgrid);
+    for(int r=0;r<nrhs;r++) out[r]=in[r];
+    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
+    std::vector<CoarseVector> Csrc_split(nrhs,_CoarseGrid), Csol_split(nrhs,_CoarseGrid);
+    CoarseVector CsrcMrhs(_CoarseGridMrhs), CsolMrhs(_CoarseGridMrhs);
+    t=-usecond();
+    _Projector.blockProject(vec1,Csrc_split);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(Csrc_split[r],CsrcMrhs,r,0);
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs project+pack took "<<t/1000.0<<"ms"<<std::endl;
+    t=-usecond(); CsolMrhs=Zero(); _CoarseSolve(CsrcMrhs,CsolMrhs); t+=usecond();
+    std::cout<<GridLogMessage<<"Mrhs coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(Csol_split[r],CsolMrhs,r,0);
+    _Projector.blockPromote(vec1,Csol_split);
+    for(int r=0;r<nrhs;r++) add(out[r],out[r],vec1[r]);
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs unpack+promote took "<<t/1000.0<<"ms"<<std::endl;
+    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
+    t=-usecond();
+    for(int r=0;r<nrhs;r++){ vec2[r]=Zero(); _PostSmoother(vec1[r],vec2[r]); add(out[r],out[r],vec2[r]); }
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs post-smooth took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+  ParseEnvironment();
+
+  const int Ls=24; RealD M5=1.8, b=1.5, c=0.5;
+  const int nbasis=60; const int nrhs=Nrhs;
+  GRID_ASSERT(nrhs % vComplex::Nsimd() == 0);
+
+  std::vector<int> lat_size {48,48,48,96};
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 blocking (default 2^4)
+  Coordinate clatt = lat_size;
+  Coordinate Block({2,2,2,2});
+  if ( getenv("BLOCK") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block); GRID_ASSERT(Block.size()==4); }
+  for(int d=0;d<4;d++){ GRID_ASSERT(lat_size[d]%Block[d]==0); clatt[d]=lat_size[d]/Block[d]; }
+  std::cout << GridLogMessage << "Block  " << Block  << "  coarse lattice       " << clatt << std::endl;
+
+  // Level 2 blocking: SUPERCOARSE default 8,4,3,6 -> CC [3,6,8,8], the dense floor.
+  Coordinate cclatt = clatt;
+  Coordinate Block2({8,4,3,6});
+  if ( getenv("BLOCK2") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK2")),Block2); GRID_ASSERT(Block2.size()==4); }
+  for(int d=0;d<4;d++){ GRID_ASSERT(clatt[d]%Block2[d]==0); cclatt[d]=clatt[d]/Block2[d]; }
+  std::cout << GridLogMessage << "Block2 " << Block2 << "  coarse-coarse lattice " << cclatt << std::endl;
+
+  GridCartesian *Coarse4d = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(cclatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // 6D mrhs grids: rhs is dim 0, SIMD across rhs
+  Coordinate mpi=GridDefaultMpi();
+  Coordinate rhMpi ({1,1,mpi[0],mpi[1],mpi[2],mpi[3]});
+  Coordinate rhSimd({vComplex::Nsimd(),1,1,1,1,1});
+  Coordinate rhLatt ({nrhs,1,clatt[0], clatt[1], clatt[2], clatt[3]});
+  Coordinate rhLatt2({nrhs,1,cclatt[0],cclatt[1],cclatt[2],cclatt[3]});
+  GridCartesian *CoarseMrhs       = new GridCartesian(rhLatt, rhSimd,rhMpi);
+  GridCartesian *CoarseCoarseMrhs = new GridCartesian(rhLatt2,rhSimd,rhMpi);
+
+  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+
+  // Level 1 tensor types
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>       LittleDiracOperator;
+  typedef MultiGeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  MrhsLittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                                CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>                  Subspace;
+
+  // Level 2 tensor types (coarsening deepens the nest by one iScalar)
+  typedef CoarseVector::vector_object                                      CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>          LittleDiracOperatorL2;
+  typedef MultiGeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>     MrhsLittleDiracOperatorL2;
+  typedef LittleDiracOperatorL2::CoarseVector                              CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,nbasis>                     SubspaceL2;
+
+  // The library dense bottom over the L2 coarse operator
+  typedef DenseCoarseMatrix<CoarseSiteObj,vTTComplex,nbasis>               DenseCC_t;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom (Coarse5d);
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // Subspace: load RAW (no Orthogonalise!), or generate.
+  //////////////////////////////////////////////////////////////////////
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                            + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+  uint64_t file_exists=0;
+  if ( UGrid->IsBoss() ){ std::ifstream f(subspace_file); file_exists=f.good()?1:0; }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb=0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+  if ( file_exists ){
+    std::cout << GridLogMessage << "*** Loading subspace from disk (kept RAW) ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+  }
+
+  // RAW copy of the fine null vectors BEFORE CoarsenOperator block-orthonormalises in place.
+  std::vector<LatticeFermionD> rawNull(nbasis,FGrid);
+  for(int k=0;k<nbasis;k++) rawNull[k]=AggregatesGCR.subspace[k];
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarsen L1->L2 and L2->L3 with SINGLE-RHS machinery; import to mrhs via
+  // CopyMatrix.  The L2 (coarse-coarse) single-RHS operator is HOISTED to
+  // main scope: DenseCoarseMatrix imports its stencil and uses its M for
+  // certificates, so it must stay alive for the whole run.
+  //////////////////////////////////////////////////////////////////////
+  MrhsLittleDiracOperator   mrhsLittleDiracOpPV(geom,  CoarseMrhs);
+  MrhsLittleDiracOperatorL2 mrhsLittleDiracOpL2(geom2, CoarseCoarseMrhs);
+  MultiRHSBlockProject<LatticeFermionD> MrhsProjector;
+  MultiRHSBlockProject<CoarseVector>    MrhsProjectorL2;
+
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2,Coarse5d,CoarseCoarse5d);
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC5d(LittleDiracOpL2);
+
+  {
+    // --- L1->L2 single-RHS coarse operator (scoped: its padded _A is the memory peak) ---
+    LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d);
+    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);   // orthonormalises AggregatesGCR.subspace in place
+    mrhsLittleDiracOpPV.CopyMatrix(LittleDiracOpPV);
+    MrhsProjector.Allocate(nbasis,FGrid,Coarse5d);
+    MrhsProjector.ImportBasis(AggregatesGCR.subspace);
+    NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+    // --- psi_coarse = P^dag (RAW fine null) -> Galerkin images, NOT e_k ---
+    std::vector<CoarseVector> psi_coarse(nbasis,Coarse5d);
+    for(int k=0;k<nbasis;k++) AggregatesGCR.ProjectToSubspace(psi_coarse[k], rawNull[k]);
+    rawNull.clear(); rawNull.shrink_to_fit();
+    {
+      RealD s2=0.0;
+      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
+        ComplexD sij=TensorRemove(innerProduct(psi_coarse[i],psi_coarse[j]));
+        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
+      }
+      std::cout<<GridLogMessage<<"GUARD: ||<psi_coarse|psi_coarse> - I||_F = "<<std::sqrt(s2)
+               <<"   (~0.23 good; ~sqrt(N_coarse)="<<std::sqrt((double)Coarse5d->gSites())<<" = e_k leak)"<<std::endl;
+    }
+
+    // --- L2->L3 single-RHS coarsening ---
+    SubspaceL2 AggregatesL2(CoarseCoarse5d,Coarse5d,cb);
+    for(int k=0;k<nbasis;k++) AggregatesL2.subspace[k]=psi_coarse[k];
+    LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+    mrhsLittleDiracOpL2.CopyMatrix(LittleDiracOpL2);
+    MrhsProjectorL2.Allocate(nbasis,Coarse5d,CoarseCoarse5d);
+    MrhsProjectorL2.ImportBasis(AggregatesL2.subspace);
+
+    // --- guard psi_cc ---
+    {
+      std::vector<CoarseCoarseVector> psi_cc(nbasis,CoarseCoarse5d);
+      for(int k=0;k<nbasis;k++) AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+      RealD s2=0.0;
+      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
+        ComplexD sij=TensorRemove(innerProduct(psi_cc[i],psi_cc[j]));
+        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
+      }
+      std::cout<<GridLogMessage<<"GUARD: ||<psi_cc|psi_cc> - I||_F = "<<std::sqrt(s2)
+               <<"   (~0.23 good; ~sqrt(N_cc)="<<std::sqrt((double)CoarseCoarse5d->gSites())<<" = e_k leak)"<<std::endl;
+    }
+  } // single-RHS FINE op + padded _A + AggregatesL2 + psi_coarse freed here
+
+  NonHermitianLinearOperator<MrhsLittleDiracOperator,CoarseVector>         mrhsLinOpCoarse(mrhsLittleDiracOpPV);
+  NonHermitianLinearOperator<MrhsLittleDiracOperatorL2,CoarseCoarseVector> mrhsLinOpCC(mrhsLittleDiracOpL2);
+
+  //////////////////////////////////////////////////////////////////////
+  // DENSE coarse-coarse bottom: the LIBRARY class, constructed AFTER the
+  // fine coarsening frees its memory peak.  Imports the stencil of the
+  // hoisted single-RHS LittleDiracOpL2 directly (no probing).
+  //////////////////////////////////////////////////////////////////////
+  std::unique_ptr<DenseCC_t> DenseCC;
+  std::unique_ptr<MrhsDenseCCSolve<DenseCC_t,CoarseCoarseVector>> MrhsDenseCC;
+  if (UseDenseCC) {
+    std::cout << GridLogMessage << "**********************************************" << std::endl;
+    std::cout << GridLogMessage << " Dense CC inverse setup (library DenseCoarseMatrix)" << std::endl;
+    std::cout << GridLogMessage << "**********************************************" << std::endl;
+    DenseCC.reset(new DenseCC_t(LittleDiracOpL2, CoarseCoarse5d));
+    MrhsDenseCC.reset(new MrhsDenseCCSolve<DenseCC_t,CoarseCoarseVector>(*DenseCC, CoarseCoarse5d, nrhs));
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Solvers, innermost first.
+  //////////////////////////////////////////////////////////////////////
+  TrivialPrecon<CoarseVector>       simpleC;
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+  TrivialPrecon<LatticeFermionD>    simple_fine;
+
+  // L3 (coarse-coarse) iterative solve: PGCR on the 6D cc operator (DENSE_CC=0 branch)
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    L3PGCR(L3Tol,L3MaxIt,mrhsLinOpCC,simpleCC,L3Nstep,L3Nstep);
+  L3PGCR.Level(3);
+  L3PGCR.Name("CCouter");
+
+  LinearFunction<CoarseCoarseVector> *ccSolve;
+  if (UseDenseCC) ccSolve = MrhsDenseCC.get();
+  else            ccSolve = &L3PGCR;
+
+  // L2 coarse smoother: shifted 6D coarse op, fixed nstep
+  ShiftedLinearOperator<CoarseVector> ShiftedMrhsCoarse(CoarseSmootherShift, mrhsLinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
+    CoarseSmootherGCR(0.01,1,ShiftedMrhsCoarse,simpleC,CoarseSmootherNstep,CoarseSmootherNstep);
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+  CoarseSmootherGCR.SetZeroGuess(1);   // caller zeroes vec2: skip r0 apply every L2 iteration
+
+  // L2->L3 V-cycle preconditioner (operates on 6D coarse field)
+  MrhsCoarseThreeLevelPrec<CoarseVector,CoarseCoarseVector>
+    L2to3Precon(mrhsLinOpCoarse, CoarseSmootherGCR, MrhsProjectorL2, *ccSolve,
+                Coarse5d, CoarseCoarse5d, CoarseCoarseMrhs, nrhs);
+
+  // L2 coarse solve: PGCR on 6D coarse op, preconditioned by the L2->L3 V-cycle
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
+    L2PGCR(CoarseSolverTol, CoarseSolverOrder/16, mrhsLinOpCoarse, L2to3Precon, 16, 16);
+  L2PGCR.Level(2);
+  L2PGCR.Name("Couter");
+  L2PGCR.SetZeroGuess(1);              // caller zeroes CsolMrhs; restarts still recompute r
+
+  // Fine smoother (per-rhs, looped in the L1->L2 V-cycle)
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD>
+    SmootherGCR(0.0,1,ShiftedPVdagM,simple_fine,FineSmootherOrder,FineSmootherOrder);
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+  SmootherGCR.SetZeroGuess(1);         // caller zeroes vec2[r]: saves 12 fine mults/outer
+
+  // L1->L2 V-cycle (fine); its coarse solve is the three-level L2PGCR
+  typedef PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> FineSmoother_t;
+  MrhsTwoLevelMG<LatticeFermionD,CoarseVector,FineSmoother_t>
+    ThreeLevelPrecon(PVdagM, SmootherGCR, MrhsProjector, L2PGCR, Coarse5d, CoarseMrhs);
+
+  // Outer mrhs solve
+  MrhsPGCRNonHermitian<LatticeFermionD>
+    L1PGCR(OuterTol,1000,PVdagM,ThreeLevelPrecon,OuterMmax,OuterNstep);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+  L1PGCR.SetZeroGuess(1);              // sol[r]=Zero() below; restarts recompute r as always
+
+  //////////////////////////////////////////////////////////////////////
+  // Sources and solve
+  //////////////////////////////////////////////////////////////////////
+  std::vector<LatticeFermionD> src(nrhs,FGrid), sol(nrhs,FGrid);
+  for(int r=0;r<nrhs;r++){ gaussian(RNG5,src[r]); sol[r]=Zero(); }
+
+  std::cout << GridLogMessage << "**********************************************" << std::endl;
+  std::cout << GridLogMessage << " MultiRHS THREE-level solve (DenseCoarseMatrix bottom): " << nrhs << " RHS " << std::endl;
+  std::cout << GridLogMessage << "**********************************************" << std::endl;
+
+  GridStopWatch w; w.Start();
+  L1PGCR(src,sol);
+  w.Stop();
+  std::cout << GridLogMessage << "MultiRHS 3-level dense solve total " << w.Elapsed()
+            << "  (per RHS: " << w.useconds()/1.0e6/nrhs << " s)" << std::endl;
+
+  { LatticeFermionD Ax(FGrid); RealD worst=0.0;
+    for(int r=0;r<nrhs;r++){ PVdagM.Op(sol[r],Ax); Ax=Ax-src[r];
+      RealD rn=std::sqrt(norm2(Ax)/norm2(src[r]));
+      std::cout << GridLogMessage << "FINAL: rhs["<<r<<"] true residual = " << rn << std::endl;
+      worst=std::max(worst,rn); }
+    std::cout << GridLogMessage << "FINAL: worst-case residual = " << worst << std::endl;
+  }
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

scripts/filelist

@@ -11,7 +11,7 @@ CCFILES=`find . -name '*.cc' -not -path '*/instantiation/*/*' -not -path '*/gamm
 
 ZWILS_FERMION_FILES=` find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/ZWilsonImpl*' `
 WILS_FERMION_FILES=`  find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/WilsonImpl*' `
-STAG_FERMION_FILES=`  find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/Staggered*' `
+STAG_FERMION_FILES=`  find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/StaggeredImpl*' `
 GP_FERMION_FILES=`    find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/Gparity*' `
 ADJ_FERMION_FILES=`   find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/WilsonAdj*' `
 TWOIND_FERMION_FILES=`find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/WilsonTwoIndex*'`

skills/gpu-memory-performance.md

@@ -9,13 +9,26 @@ allowed-tools:
 
 # GPU Memory Performance in Grid
 
+## Nsimd on GPU builds
+
+With `GEN_SIMD_WIDTH=64B` (the typical production setting), `Nsimd` is **not 1**:
+
+| Scalar type | `sizeof` | `Nsimd = 64 / sizeof` |
+|---|---|---|
+| `ComplexD` | 16 B | **4** |
+| `ComplexF` | 8 B  | **8** |
+| `RealD`    | 8 B  | **8** |
+| `RealF`    | 4 B  | **16** |
+
+So for `LatticePropagatorD` (scalar type `ComplexD`), `Nsimd=4` and the SIMD lane runs `threadIdx.x ∈ {0,1,2,3}`.  True `Nsimd=1` scalar-GPU builds are the exception, not the rule.
+
 ## The acceleratorThreads() Trap
 
-`acceleratorThreads()` is a runtime-settable global (default **2**) that controls the `blockDim.y` of every `accelerator_for` launch.  It is NOT the SIMD width — it is the number of sites processed per block in the y-dimension.
+`acceleratorThreads()` is a runtime-settable global (default **8**) that controls the `blockDim.y` of every `accelerator_for` launch.  It is NOT the SIMD width — it is the number of sites processed per block in the y-dimension.
 
 ```cpp
 // Grid/threads/Accelerator.cc
-uint32_t accelerator_threads = 2;   // <-- default
+uint32_t accelerator_threads = 8;
 ```
 
 With `accelerator_for(ss, osites, nsimd, ...)`, the launch is:
@@ -25,15 +38,27 @@ dim3 threads(nsimd, acceleratorThreads(), 1)
 dim3 blocks ((osites + acceleratorThreads() - 1) / acceleratorThreads(), 1, 1)
 ```
 
-For `nsimd=1` and the default `acceleratorThreads()=2`:
+Total threads per block = `Nsimd × acceleratorThreads()`.  With `GEN_SIMD_WIDTH=64B` and `Nsimd=8` (fp32 / ComplexF):
-- **2 threads per block** on a 64-thread AMD wavefront → **3% occupancy**
-- Expected bandwidth ≈ peak × 3% ≈ 50 GB/s on MI250X
 
-**Diagnostic**: observed bandwidth << peak, kernel time >> expected from data volume.  Check with `--accelerator-threads 16` or `--accelerator-threads 32` at runtime.  A large speedup confirms occupancy starvation.
+| `acceleratorThreads()` | threads/block | AMD wavefront | note |
+|---|---|---|---|
+| 2 (original default) | 16 | 25% | sub-wavefront, poor |
+| 4 | 32 | 50% | half wavefront |
+| **8 (current default)** | **64** | **100%** | **one full wavefront — Dslash sweet spot** |
+| 16 | 128 | 200% | two wavefronts/block; register-pressure cliff for heavy kernels |
+| 32 | 256 | 400% | severe register spill for stencil kernels |
+
+**Why 8 and not higher?**  Compute-heavy kernels like the Domain Wall Dslash carry many live registers per thread (spinors + gauge links + projections).  Doubling `acceleratorThreads` from 8→16 doubles the register demand per block, which on AMD GFX90A triggers a hard occupancy cliff: `Benchmark_dwf_fp32` drops from 1.7 TF/s (nt=8) to ~300 GF/s (nt=16).  The sweet spot is one full wavefront per block, which with `Nsimd=8` (fp32) means `nt=8`.
+
+For fp64 work (`Nsimd=4`), `nt=8` gives 32 threads = half a wavefront (AMD pads to 64 with idle lanes).  Kernels that are not register-limited (e.g. simple lattice arithmetic) can benefit from `--accelerator-threads 16` at runtime.  Reduction kernels bypass `acceleratorThreads()` entirely via `getNumBlocksAndThreads`.
+
+**Why was the default ever 2?**  Before the `threadIdx.x`/`threadIdx.y` remap (see LambdaApply section below), the site index lived in `threadIdx.x` — the fast, coalescing dimension.  Increasing `acceleratorThreads` widened the block in the *site* direction, so adjacent threads in a warp hit adjacent sites, each stride `sizeof(vobj)` apart in AoS memory — breaking coalescing.  On early NVIDIA ports with `Nsimd≈8`, `nt=2` gave 16 threads = 50% of a 32-thread warp; NVIDIA recovers this via multiple concurrent blocks per SM, so occupancy was barely tolerable.  AMD has no such multiplier when blocks are already sub-wavefront.  After the remap put the site index in `threadIdx.y` and the SIMD lane in `threadIdx.x`, coalescing became independent of `acceleratorThreads`, removing the constraint.
+
+**Diagnostic**: observed bandwidth << peak, kernel time >> expected from data volume.  Check with `--accelerator-threads 32` at runtime.  A large speedup confirms occupancy starvation.
 
 **Fix options** (in order of preference):
 1. Kernel needs its own thread count — use `getNumBlocksAndThreads` and launch a `__global__` kernel directly (see below).
-2. Temporarily acceptable: set `--accelerator-threads 16` or 32 at the application level.  Note this affects every `accelerator_for` site in the binary.
+2. Temporarily acceptable: set `--accelerator-threads 32` at the application level.  Note this affects every `accelerator_for` site in the binary.
 
 ## LambdaApply Thread Mapping
 
@@ -49,7 +74,7 @@ Lambda(x, y, z);
 
 `threadIdx.x` is the **fast** (lane) dimension — consecutive thread IDs within a warp/wavefront correspond to consecutive lane values on the **same** site, not consecutive sites.
 
-Consequence: for coalesced access from a `vobj` array (AoS layout, stride = `sizeof(vobj)` between adjacent sites), adjacent threads in a wavefront address the **same** site at different lanes, not adjacent sites.  With `Nsimd=1` (GPU scalar build), `threadIdx.x` is always 0 and provides no coalescing benefit at all.
+With `GEN_SIMD_WIDTH=64B` and `Nsimd=4` (PropagatorD), a 64-thread AMD wavefront contains 64/4 = 16 sites, each processed by 4 lanes.  Adjacent threads within the wavefront read different lanes of the same site — this is a broadcast pattern (hardware handles this efficiently), not a stride.  The stride between consecutive *sites* (`sizeof(vobj)`) only appears between groups of `Nsimd` threads, spaced `Nsimd` apart in threadIdx.x — not between adjacent threads within a warp.
 
 ## coalescedRead / coalescedWrite
 
@@ -57,7 +82,7 @@ These are Grid's canonical way to read/write one SIMD lane from a vector type in
 
 ```cpp
 // accelerator_for(ss, osites, Nsimd, {
-//   lane = acceleratorSIMTlane(Nsimd) = threadIdx.x
+//   lane = acceleratorSIMTlane(Nsimd) = threadIdx.x  ∈ {0..Nsimd-1}
 auto scalar_val = coalescedRead(field[ss]);       // extractLane(lane, field[ss])
 coalescedWrite(field[ss], scalar_val);             // insertLane(lane, field[ss], scalar_val)
 ```
@@ -66,8 +91,6 @@ For `vobj` aggregate types, `coalescedRead` calls `extractLane(lane, vobj)` whic
 
 For `vsimd` (raw SIMD vector) types, it casts to `scalar_type*` and indexes with `lane`.
 
-**When Nsimd=1** (GPU scalar build): `lane=0` always, so `coalescedRead`/`coalescedWrite` are effectively no-ops (direct read/write).  Coalescing must be achieved through the iteration structure instead.
-
 ## Coalescing the Iteration Structure
 
 For an AoS input array where each site is `words` 16-byte elements, adjacent threads reading the same site's consecutive words achieve coalesced access:
@@ -108,12 +131,12 @@ Pattern: use `getNumBlocksAndThreads` to pick `numThreads` and `numBlocks`:
 Integer numThreads, numBlocks;
 int ok = getNumBlocksAndThreads(n, sizeof(sobj), numThreads, numBlocks);
 // starts at warpSize (32/64), doubles while 2*threads*sizeof(sobj) < sharedMemPerBlock
-// gives 64–256 threads/block → near-100% wavefront occupancy
+// gives 64–256 threads/block → correct occupancy independent of acceleratorThreads()
 Integer smemSize = numThreads * sizeof(sobj);
 myKernel<<<numBlocks, numThreads, smemSize, computeStream>>>(args...);
 ```
 
-This gives 64–256 threads/block regardless of `acceleratorThreads()`.  Grid's `reduceKernel` uses this pattern and achieves ~400 GB/s on MI250X.
+Grid's `reduceKernel` uses this pattern and achieves ~400 GB/s on MI250X.
 
 ## Fused vs Staged HBM Access
 
@@ -161,21 +184,22 @@ __device__ void packReduceBlocks(
 }
 ```
 
-Launched with `getNumBlocksAndThreads` → 128–256 threads/block → correct occupancy without depending on `acceleratorThreads()`.
+Launched with `getNumBlocksAndThreads` → 128 threads/block for R=12 (`BundleScalarD`=192 B, sharedMem=64 KB) → correct occupancy without depending on `acceleratorThreads()`.
 
-## Observed Numbers on MI250X (32^4 LatticePropagatorD, Nsimd=1)
+## Observed Numbers on MI250X (32^4 LatticePropagatorD, Nsimd=4, GEN_SIMD_WIDTH=64B)
 
 | Configuration | pack µs/group | reduce µs/group | total µs | GB/s |
 |---|---|---|---|---|
-| acceleratorThreads=2, staged | 10,080 | 470 | 126,909 | 50 |
+| acceleratorThreads=2 (8 threads/block), staged | 10,080 | 470 | 126,909 | 50 |
-| acceleratorThreads=16, staged | 342 | 310 | 8,251 | 297 |
+| acceleratorThreads=16 (64 threads/block), staged | 342 | 310 | 8,251 | 297 |
-| acceleratorThreads=16, fused | — | 349 | 4,584 | 546 |
+| acceleratorThreads=16, fused (128 threads/block via getNumBlocksAndThreads) | — | 349 | 4,584 | 546 |
 
 The fused kernel at 349 µs/group reads 201 MB at 576 GB/s — 36% of MI250X HBM peak.  The remaining gap from peak is the in-kernel serial loop over R=12 words and the 12 serial kernel launches.
 
 ## Quick Checklist When a Kernel Is Slow
 
-1. Check threads per block: `accelerator_for(ss, N, 1, ...)` with default `acceleratorThreads()=2` = 2 threads/block = 3% occupancy on AMD.  Try `--accelerator-threads 16` at runtime; if it helps a lot, occupancy is the problem.
+1. **Check Nsimd**: `GEN_SIMD_WIDTH=64B` → Nsimd=4 (ComplexD), 8 (ComplexF).  Total threads/block = `Nsimd × acceleratorThreads()`.  With old default nt=2 and Nsimd=4: 8 threads = 12.5% of AMD wavefront.
-2. Check for bulk struct accumulation in registers (`Bundle b; for(...) b._internal[k] = ...;`).  Replace with per-element writes via `coalescedWrite`.
+2. Check threads per block: for `accelerator_for` kernels use `--accelerator-threads 32` and measure; a large speedup confirms occupancy starvation.
-3. Check for staged HBM access (pack → buffer → reduce).  Count the passes; fuse if ≥ 2 passes over the same data.
+3. Check for bulk struct accumulation in registers (`Bundle b; for(...) b._internal[k] = ...;`).  Replace with per-element writes via `coalescedWrite`.
-4. For reduction kernels, always use `getNumBlocksAndThreads` rather than `accelerator_for` so thread count is independent of `acceleratorThreads()`.
+4. Check for staged HBM access (pack → buffer → reduce).  Count the passes; fuse if ≥ 2 passes over the same data.
+5. For reduction kernels, always use `getNumBlocksAndThreads` rather than `accelerator_for` so thread count is independent of `acceleratorThreads()`.

skills/mpi-heterogeneous.md

@@ -1,6 +1,6 @@
 ---
 name: mpi-heterogeneous
-description: Diagnose and work around MPI correctness bugs on heterogeneous (CPU+GPU) systems — device buffer aliasing in MPI_Sendrecv, AARCH64 PLT corruption from libfabric, topology-dependent allreduce hangs, and deterministic point-to-point reduction trees as a replacement for MPI_Allreduce.
+description: Diagnose and work around MPI correctness bugs on heterogeneous (CPU+GPU) systems — device buffer aliasing in MPI_Sendrecv, AARCH64 PLT corruption from libfabric, topology-dependent allreduce hangs, mixed-ABI HIP runtime from wrong GTL library (Frontier/ROCm), and deterministic point-to-point reduction trees as a replacement for MPI_Allreduce.
 user-invocable: true
 allowed-tools:
   - Read
@@ -110,6 +110,51 @@ void GlobalSumP2P(double *data, int count, MPI_Comm comm) {
 
 Grid reference: `USE_GRID_REDUCTION` macro in `Grid/communicator/Communicator_mpi3.cc`.
 
+## Bug Class 4: Mixed HIP ABI from Wrong GTL Library (Frontier / ROCm)
+
+**Symptom**: `HIPFFT_PARSE_ERROR` (error code 12) returned by `hipfftPlanMany` / `hipfftMakePlanMany` / `hipfftPlan1d` for FFT sizes G < 32, but G ≥ 32 succeeds. The failure only occurs with an empty rocFFT kernel cache (`~/.cache/rocfft`); a warm cache may mask it. Host-side operations and GPU kernels that do not invoke rocFFT JIT work correctly.
+
+**Root cause — mixed HIP ABI**: rocFFT uses JIT compilation (via `libamd_comgr`) for small transforms (G < 32); for G ≥ 32 it uses pre-compiled device code bundled in the library, so the JIT path is never exercised. When two HIP runtime versions are loaded in the same process — e.g. `libamdhip64.so.7` (ROCm 7) and `libamdhip64.so.6` (ROCm 6) — the rocFFT JIT cannot complete successfully.
+
+The hidden source of the old library is the Cray MPI GPU Transport Layer. On Frontier, `cray-mpich`'s `libmpi_gtl_hsa.so` may be compiled against `libamdhip64.so.6` (ROCm 6 ABI) even when the loaded ROCm module is 7.0.2. Because `LD_LIBRARY_PATH` picks up the GTL directory before the ROCm 7 library directory, `libamdhip64.so.6` is pulled in first, and both ABI versions end up resident in the process.
+
+**Diagnosis**:
+```bash
+# Check which libamdhip64 versions are actually linked into your binary at runtime
+ldd --verbose ./your_binary 2>&1 | grep amdhip
+# Bad output — two different .so versions:
+#   libamdhip64.so.6 => /opt/rocm-6.4.2/lib/libamdhip64.so.6
+#   libamdhip64.so.7 => /opt/rocm-7.0.2/lib/libamdhip64.so.7
+# Good output — only one:
+#   libamdhip64.so.7 => /opt/rocm-7.0.2/lib/libamdhip64.so.7
+```
+
+If two versions appear, the problem is the GTL/LD_LIBRARY_PATH ordering.
+
+**Fix — correct module stack and LD_LIBRARY_PATH ordering (Frontier)**:
+```bash
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.0.2
+# Prepend CRAY_LD_LIBRARY_PATH so the ROCm-7-aware GTL is found first
+export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
+# Ensure ROCm 7 LLVM libs (needed by libamd_comgr JIT) are on the path
+export LD_LIBRARY_PATH=/opt/rocm-7.0.2/lib/llvm/lib/:$LD_LIBRARY_PATH
+```
+
+The critical step is prepending `CRAY_LD_LIBRARY_PATH`: this ensures the GTL library built against the ROCm 7 ABI is resolved before any older version that may appear further down `LD_LIBRARY_PATH`. Without this step, a stale symlink or directory ordering can silently load the wrong `libmpi_gtl_hsa.so`.
+
+**Reproducer**: `tests/debug/Test_hipfft_repro.cc` — standalone hipFFT test (no Grid headers) that sweeps G and howmany values matching realistic Grid lattice geometries. Compile with:
+```bash
+hipcc -o Test_hipfft_repro Test_hipfft_repro.cc -lhipfft
+rm -rf ~/.cache/rocfft   # empty cache required to trigger JIT path
+./Test_hipfft_repro
+```
+
+**Reference**: `systems/WorkArounds.txt`, Frontier section — GPU mapping, XPMEM, and `FI_MR_CACHE_MONITOR=disabled` settings for Frontier are documented there.
+
+**Systems affected**: Frontier (ORNL, MI250X). Likely applies to any Cray PE system where the loaded `cray-mpich` GTL was compiled against an older ROCm ABI than the runtime ROCm module. LumiG (CSC, MI250X) uses the same Cray PE and may exhibit the same issue.
+
 ## Compile-Time Guard Structure
 
 Recommended macro structure to switch between the workaround paths:

systems/Aurora/aggregate_io.pbs

@@ -0,0 +1,144 @@
+#!/bin/bash
+
+# Grid's tests/IO/Test_aggregate_io on Aurora, 12 ranks per node (one per
+# tile).  This is the Grid-side counterpart of MPI_benchmark/io_aurora.pbs:
+# same three paths, same comparison, but exercising the real BinaryIO.h
+# code that production uses, with Grid's checksums and its munge/endian
+# conversion in the loop.  Run both -- if they disagree, the difference is
+# in Grid, not in MPI-IO.
+#
+# Submit from the BUILD directory: BIN below is $PBS_O_WORKDIR relative.
+
+#PBS -q debug-scaling
+#PBS -l filesystems=flare
+#PBS -l filesystems=home
+#PBS -l select=4
+#PBS -l walltime=01:00:00
+#PBS -A 15479
+##PBS -A LatticeQCD_aesp_CNDA
+
+cd $PBS_O_WORKDIR
+cp $PBS_NODEFILE nodefile
+
+##########################################################################
+# Grid source tree.  Unlike io_mpi this IS a Grid binary -- SYCL, c-lime,
+# the lot -- so sourceme.sh is required, and so is the tile wrapper.
+##########################################################################
+root=$HOME/Grid/systems/Aurora            # <-- adjust to your source tree
+source ${root}/sourceme.sh
+TILE=${root}/benchmarks/gpu_tile.sh
+
+BIN=$PBS_O_WORKDIR/tests/IO/Test_aggregate_io
+[ -x $BIN   ] || { echo "no $BIN -- submit from the build directory"; exit 1; }
+[ -x $TILE  ] || { echo "no $TILE -- check \$root"; exit 1; }
+
+export OMP_NUM_THREADS=4
+export MPICH_OFI_NIC_POLICY=GPU
+# The aggregate path's MPI_Alltoallv runs on HOST buffers, so NIC policy
+# NUMA is arguably the better match here even though GPU is the standard
+# for Grid on this machine.  Worth one comparison run; do not change it
+# silently, the rest of the Grid numbers on Aurora were taken with GPU.
+#export MPICH_OFI_NIC_POLICY=NUMA
+
+##########################################################################
+# Test_aggregate_io writes ref.bin / agg.bin / raw.bin / trunc.bin with
+# RELATIVE paths, so they land in the cwd.  Home is not the point of the
+# exercise.
+##########################################################################
+# PROJECT is the flare project DIRECTORY name, not the -A account number.
+PROJECT=LatticeQCD_aesp_CNDA
+WORK=/lus/flare/projects/$PROJECT/$USER/aggio.$PBS_JOBID
+mkdir -p $WORK || { echo "cannot create $WORK -- set PROJECT correctly"; exit 1; }
+cd       $WORK
+
+# Leave striping at the directory default, as the Frontier runs did.  At
+# 128 nodes there the aggregate path was 2.2x faster on the default layout
+# than on -c -1, while MPI-IO was 2.4x slower on it; the two paths want
+# opposite things.  Set it only to reproduce that interaction, and say so.
+#lfs setstripe -c -1 -S 8M $WORK
+lfs getstripe -d $WORK 2>/dev/null || echo "(no lfs getstripe -- not Lustre?)"
+
+# ROMIO's own account of what the reference path did.  Verbose; switch on
+# deliberately.
+#export MPICH_MPIIO_STATS=1
+#export MPICH_MPIIO_TIMERS=1
+
+AT=8        # accelerator threads; immaterial to an I/O test, kept canonical
+
+run () {   # run <nodes> <grid> <mpi> <comment> [extra args...]
+  local nodes=$1 gr=$2 mp=$3 note=$4
+  local ntot=$(( nodes * 12 ))
+  shift 4
+  echo
+  echo "==================================================================="
+  echo "=== nodes=$nodes ranks=$ntot grid=$gr mpi=$mp   $note"
+  echo "=== extra: $@"
+  echo "==================================================================="
+  # Deliberately NOT removing ref.bin/agg.bin/raw.bin here.  The test does
+  # it itself (Test_aggregate_io.cc:203 and :370) for exactly the stale-tail
+  # reason, and doing it in the script would destroy the files a later
+  # --io-read-only job needs.
+  mpiexec -np $ntot -ppn 12 -envall $TILE $BIN --mpi $mp --grid $gr \
+          --shm-mpi 0 --shm 2048 --device-mem 32000 --accelerator-threads $AT "$@"
+  echo "=== exit $?"
+}
+
+#####################################################################
+# Phase 0.  Logic corners, correctness only, no bandwidth claimed.
+#
+# These run the FULL default target sweep {1, 1024, 64K, 4M}, which walks
+# k from 1 to Nd-1 and so exercises every shape of aggregation plan on a
+# volume small enough that the serial byte-for-byte file comparison is
+# affordable.  1.3 MB/rank means nothing for bandwidth and is not meant to.
+#
+# Both branches of MPI_Alltoallv are covered; the labels were checked with
+# MPI_benchmark/io_mpi, not assumed.
+#####################################################################
+run 1 16.16.16.24 2.2.1.3 "UNIFORM counts, row of 4"      --io-reps 0
+run 1 12.12.24.8  2.2.3.1 "NON-UNIFORM counts, row of 12" --io-reps 0
+
+#####################################################################
+# Phase 1.  151 MB/rank, 3.6 GB record, 24 ranks.  Correctness AND
+# performance at the production 4 MB target only -- the byte-for-byte
+# comparison reads the whole file on one rank, so the four-target sweep
+# is not affordable here.
+#####################################################################
+run 2 16.16.96.256 2.2.3.2 "3.6 GB, row of 4, 32 extents, correctness + perf" \
+    --aggregate-target 4194304
+
+# Same volume per rank, but an odd process factor in an un-split dimension
+# puts the variable-count branch of Alltoallv where the bandwidth is real.
+run 2 24.16.64.256 3.2.2.2 "3.6 GB, NON-UNIFORM counts, row of 6, 22 extents" \
+    --aggregate-target 4194304
+
+#####################################################################
+# Phase 2.  604 MB/rank, 29.0 GB record, 48 ranks.  Correctness off.
+#
+# Per-rank volume is held at 16.16.32.128 in ALL FOUR, so the only thing
+# changing is how much work the aggregation has to do:
+#
+#   4.4.3.1   row 16    8 extents of 72 MB   <- old path: 9 KB runs
+#   4.2.3.2   row  8   16 extents of 36 MB
+#   2.2.12.1  row  4   32 extents of 18 MB
+#   1.1.3.16  row  1  128 extents of 4.5 MB  <- no exchange at all
+#
+# The last is the interesting control: with the fast dimensions unsplit
+# the aggregate path degenerates to plain large-extent POSIX and the
+# Alltoallv is trivial, so it separates "the transposition is cheap" from
+# "the writes are large".  Note the test's own header warns that 1.1.X.Y
+# makes the CORRECTNESS section vacuous -- rows of size one exchange
+# nothing.  That is fine here because correctness is off; it would not be
+# fine as a correctness run.
+#####################################################################
+run 4 64.64.96.128   4.4.3.1  "29.0 GB, row of 16"  --aggregate-target 4194304 --io-no-correctness
+run 4 64.32.96.256   4.2.3.2  "29.0 GB, row of 8"   --aggregate-target 4194304 --io-no-correctness
+run 4 32.32.384.128  2.2.12.1 "29.0 GB, row of 4"   --aggregate-target 4194304 --io-no-correctness
+run 4 16.16.96.2048  1.1.3.16 "29.0 GB, row of 1, no exchange" \
+    --aggregate-target 4194304 --io-no-correctness
+
+echo
+echo "=== done.  Output left in $WORK"
+echo "=== For COLD read numbers, submit a second job on a fresh allocation"
+echo "=== with WORK pointed at this directory and --io-read-only added."
+echo "=== Reading back what this job just wrote only measures page cache."
+ls -l $WORK

systems/Aurora/aggregate_io_scaling.pbs

@@ -0,0 +1,117 @@
+#!/bin/bash
+
+# Node count scan for Grid's tests/IO/Test_aggregate_io on Aurora, 12 ranks
+# per node.  The Aurora counterpart of systems/Frontier/aggregate_io_scaling.slurm.
+#
+# Everything that could confound the comparison is held fixed and only the
+# number of Lustre clients changes:
+#
+#   local volume   8.8.32.128        = 262144 sites x 576 B = 151 MB / rank
+#   aggregation    k=2, row of 16, 8 extents of 18 MB        (identical at
+#                                                             every node count)
+#   old path       contiguous run = lLattice[0]*576 = 4.6 KB (identical too)
+#
+# So the ratio of contiguous extent between the two layout preserving paths
+# is a constant 4000x across the whole scan, and any change in their relative
+# bandwidth is a property of the client count alone.
+#
+# The mpi decompositions differ from the Frontier scan because 12 ranks per
+# node is not 8, but they were chosen to give the SAME aggregation plan --
+# verified with MPI_benchmark/io_mpi, which uses the identical plan algorithm
+# and the same 576 B site object:
+#
+#   Frontier   4.4.2.1  at   32 ranks  ->  k=2, row 16, 8 extents
+#   Aurora     4.4.3.1  at   48 ranks  ->  k=2, row 16, 8 extents
+#   Aurora     4.4.12.8 at 1536 ranks  ->  k=2, row 16, 8 extents
+#
+# Consequence of 12 rather than 8: at a given NODE count the record and the
+# client count are 1.5x Frontier's.  Compare the machines at equal RANK count
+# for equal record size, or at equal NODE count for each machine used as it
+# is meant to be used.  Say which one any table is reporting.
+#
+# Trim the tail of the run list for a smaller allocation; the geometries are
+# independent.
+#
+# Submit from the BUILD directory.
+
+#PBS -q prod
+#PBS -l filesystems=flare
+#PBS -l filesystems=home
+#PBS -l select=128
+#PBS -l walltime=02:00:00
+#PBS -A 15479
+##PBS -A LatticeQCD_aesp_CNDA
+
+cd $PBS_O_WORKDIR
+cp $PBS_NODEFILE nodefile
+
+root=$HOME/Grid/systems/Aurora            # <-- adjust to your source tree
+source ${root}/sourceme.sh
+TILE=${root}/benchmarks/gpu_tile.sh
+
+BIN=$PBS_O_WORKDIR/tests/IO/Test_aggregate_io
+[ -x $BIN  ] || { echo "no $BIN -- submit from the build directory"; exit 1; }
+[ -x $TILE ] || { echo "no $TILE -- check \$root"; exit 1; }
+
+export OMP_NUM_THREADS=4
+export MPICH_OFI_NIC_POLICY=GPU
+
+# MPICH_DBG_* and MPICH_MPIIO_STATS are deliberately absent: at 1536 ranks
+# they produce gigabytes of log and perturb the timings they would explain.
+# Get those from systems/Aurora/aggregate_io.pbs on the debug queue.
+
+# PROJECT is the flare project DIRECTORY name, not the -A account number.
+PROJECT=LatticeQCD_aesp_CNDA
+WORK=/lus/flare/projects/$PROJECT/$USER/aggscale.$PBS_JOBID
+mkdir -p $WORK || { echo "cannot create $WORK -- set PROJECT correctly"; exit 1; }
+cd       $WORK
+
+# The largest rung writes three files of 232 GB, so budget ~700 GB for the
+# scan and check the quota before submitting.  The test unlinks its three
+# files at the start of each run, so that is peak, not cumulative.
+#
+# Do NOT stripe by default.  Measured on FRONTIER at 128 nodes / 1024 ranks,
+# 154.6 GB record, best of 3 (MB/s) -- these are Frontier numbers, quoted for
+# what to look for, not a prediction for Aurora:
+#
+#                     default layout    lfs setstripe -c -1 -S 8M
+#   raw                    97673                81728
+#   MPI-IO lexicographic   31675                77534
+#   aggregate             189535                86063
+#
+# The two layout preserving paths want opposite things.  Wide striping is
+# what rescues MPI-IO at scale, 32 -> 78 GB/s; it costs the aggregate path a
+# factor of 2.2 and costs raw as well.  With the default layout MPI-IO
+# plateaued at ~30 GB/s from 64 nodes upward while the aggregate path kept
+# scaling.  Uncomment only to reproduce that interaction, and record which.
+#lfs setstripe -c -1 -S 8M $WORK
+lfs getstripe -d $WORK 2>/dev/null || echo "(no lfs getstripe -- not Lustre?)"
+
+ARGS="--aggregate-target 4194304 --io-no-correctness --io-reps 3"
+AT=8
+
+run () {   # run <nodes> <grid> <mpi> <comment>
+  local nodes=$1 nranks=$(( $1 * 12 ))
+  echo
+  echo "==================================================================="
+  echo "=== SCAN nodes=$nodes ranks=$nranks  grid=$2  mpi=$3   $4"
+  echo "==================================================================="
+  mpiexec -np $nranks -ppn 12 -envall $TILE $BIN --grid $2 --mpi $3 \
+          --shm-mpi 0 --shm 2048 --device-mem 32000 --accelerator-threads $AT $ARGS
+  echo "=== exit $?"
+}
+
+#            nodes  global lattice      decomposition   record
+run    4     32.32.96.128    4.4.3.1   "7.2 GB"
+run    8     32.32.96.256    4.4.3.2   "14.5 GB"
+run   16     32.32.96.512    4.4.3.4   "29.0 GB"
+run   32     32.32.192.512   4.4.6.4   "58.0 GB"
+run   64     32.32.192.1024  4.4.6.8   "116.0 GB"
+run  128     32.32.384.1024  4.4.12.8  "231.9 GB"
+
+echo
+echo "=== done.  Files left in $WORK"
+echo "=== For cold read numbers, submit a SECOND job on a fresh allocation"
+echo "=== with WORK pointed at this directory and --io-read-only added to"
+echo "=== ARGS.  Reading back what this job wrote only measures page cache."
+ls -l $WORK

systems/Booster/config-command

@@ -8,7 +8,6 @@ LIME=/p/home/jusers/boyle2/juwels/gm2dwf/boyle/
     --disable-gparity \
     --disable-fermion-reps \
     --with-lime=$LIME \
-    --enable-accelerator-cshift \
     --disable-unified \
     CXX=nvcc \
     LDFLAGS="-cudart shared " \

systems/Frontier/aggregate_io.slurm

@@ -0,0 +1,152 @@
+#!/bin/bash -l
+
+#SBATCH --job-name=runIO
+#SBATCH --nodes=4
+#SBATCH --ntasks-per-node=8
+#SBATCH --cpus-per-task=7
+#SBATCH --gpus-per-node=8
+#SBATCH --time=02:00:00
+#SBATCH --account=phy157_dwf
+#SBATCH --gpu-bind=none
+#SBATCH --exclusive
+#SBATCH --mem=0
+#SBATCH -q debug
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3 7 6 5 4)
+export NUMA_MAP=(3 3 1 1 2 2 0 0)
+export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
+export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
+unset ROCR_VISIBLE_DEVICES
+export HIP_VISIBLE_DEVICES=\$GPU
+
+rank=\$SLURM_PROCID
+CMD="numactl -m \$NUMA -N \$NUMA  \$*"
+exec \$CMD
+EOF
+
+chmod +x ./select_gpu
+
+# GPU_MAP/NUMA_MAP above describe a full node.  At fewer than 8 ranks per node
+# Slurm hands each task cpus-per-task cores block-wise from core 0, so the NUMA
+# domain NUMA_MAP names is not in the task's allowed cpu mask and numactl fails
+# with EINVAL before Grid starts.  The GPU choice stays valid; only the memory
+# binding does not.  For the small correctness runs the binding is irrelevant,
+# so use a wrapper that picks the GCD and leaves placement to Slurm.
+cat << EOF > select_gcd
+#!/bin/bash
+export GPU_MAP=(0 1 2 3 7 6 5 4)
+export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
+unset ROCR_VISIBLE_DEVICES
+export HIP_VISIBLE_DEVICES=\$GPU
+exec \$*
+EOF
+
+chmod +x ./select_gcd
+SELECT=$SLURM_SUBMIT_DIR/select_gpu
+SELECT_NONUMA=$SLURM_SUBMIT_DIR/select_gcd
+
+root=$HOME/ParallelIO/systems/Frontier/
+source ${root}/sourceme-rocm7.2.sh
+
+export OMP_NUM_THREADS=7
+export MPICH_GPU_SUPPORT_ENABLED=1
+
+# Test_aggregate_io writes ref.bin / agg.bin / trunc.bin with *relative* paths,
+# so they land in the cwd of each rank, which is the submission directory unless
+# we move.  Home is NFS; the whole point of the exercise is Lustre.
+WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/aggio.$SLURM_JOB_ID
+mkdir -p $WORK
+cd       $WORK
+
+# Striping has to be set on the directory before the files are created, and
+# all three paths inherit it, so the A/B comparison stays fair either way.
+# Left unset: at 128 nodes the aggregate path is 2.2x faster on the default
+# layout than on -c -1, while MPI-IO is 2.4x slower on it.  See the table in
+# aggregate_io_scaling.slurm.  Set it only to reproduce that interaction.
+#lfs setstripe -c 32 -S 8M $WORK
+lfs getstripe -d $WORK      # record the inherited layout; needed to read the numbers
+
+BIN=$SLURM_SUBMIT_DIR/tests/IO/Test_aggregate_io
+
+# ROMIO / Cray MPICH collective buffering report for the reference path.
+# Verbose; switch on when you want to see what the old path is doing.
+# export MPICH_MPIIO_STATS=1
+# export MPICH_MPIIO_TIMERS=1
+
+# Always state ntasks-per-node explicitly.  Letting srun derive it from -n
+# gives an uneven packing (3,2,2,2 for -n9 over 4 nodes) which no fixed
+# localid -> GCD map can describe.  Ranks per node must not exceed 8.
+run () {   # run <nodes> <ranks-per-node> <grid> <mpi> <comment> [extra args...]
+  local nodes=$1 rpn=$2 gr=$3 mp=$4 note=$5 wrap=$SELECT
+  local nranks=$(( $1 * $2 ))
+  shift 5
+  if [ $rpn -ne 8 ]; then wrap=$SELECT_NONUMA; fi   # NUMA_MAP only valid at 8
+  echo
+  echo "==================================================================="
+  echo "=== N=$nodes x $rpn = $nranks  grid=$gr  mpi=$mp   $note"
+  echo "=== extra: $@"
+  echo "==================================================================="
+  srun -N$nodes -n$nranks --ntasks-per-node=$rpn $wrap $BIN --grid $gr --mpi $mp \
+       --accelerator-threads 8 --shm 2048 --device-mem 32000 "$@"
+  echo "=== exit $?"
+}
+
+#####################################################################
+# Phase 0.  Non-uniform AllToAllV, correctness only.
+#
+# The variable-count branch of CartesianCommunicator::AllToAllV only runs
+# when the send counts differ between destinations, which needs an odd
+# process factor in an un-split (fast) dimension.  It does NOT need a toy
+# volume -- see phase 1b, which hits it at 226 MB/rank.  The one small run
+# kept here is the only way to reach the non-uniform case at k=1 with many
+# extents per rank; it is a logic corner, carries --io-reps 0, and reports
+# no bandwidth.  2.2.4.4 per rank is 9 KB and means nothing for I/O.
+#####################################################################
+run 3 3 6.6.4.4  3.3.1.1  "non-uniform at all four targets, k=1 and k=3" --io-reps 0
+
+#####################################################################
+# Phase 1.  Medium volume, 4.8 GB record, 151 MB/rank.
+# Correctness plus performance at the production 4MB target only; the
+# byte-for-byte comparison reads the whole file on one rank, so the
+# full four target sweep is not affordable above a few GB.
+#####################################################################
+run 4 8 32.32.64.128  4.4.2.1  "4.8 GB, correctness + perf" --aggregate-target 4194304
+
+#####################################################################
+# Phase 1b.  Non-uniform AllToAllV at a volume that means something.
+# 24 ranks = 3 nodes x 8, so the NUMA map stays valid.
+#   48.48.64.64 / 3.2.2.2  k=2 row=6  <=6 extents of 13.5 MB  NON-UNIFORM
+# 5.4 GB record, 226 MB/rank.  Correctness and performance together, so
+# the variable-count path is covered where the bandwidth is also real.
+#####################################################################
+run 3 8 48.48.64.64  3.2.2.2  "5.4 GB, NON-UNIFORM counts, correctness + perf" --aggregate-target 4194304
+
+#####################################################################
+# Phase 2.  Production volume, 19.3 GB record, 604 MB/rank.
+# Correctness off.  Modelled extents at the 4MB target (fobj = 576 B):
+#   1.1.4.8   k=2  row=1   16 extents of 36 MB   <- old path gives 36 KB runs
+#   4.4.2.1   k=2  row=16   8 extents of 72 MB   <- old path gives  9 KB runs
+#   2.2.2.4   k=2  row=4    8 extents
+#   8.4.1.1   k=2  row=32   4 extents
+#####################################################################
+run 4 8 64.64.64.128  1.1.4.8  "19.3 GB, baseline decomposition" --aggregate-target 4194304 --io-no-correctness
+run 4 8 64.64.64.128  4.4.2.1  "19.3 GB, row of 32"             --aggregate-target 4194304 --io-no-correctness
+run 4 8 64.64.64.128  2.2.2.4  "19.3 GB, row of 4"              --aggregate-target 4194304 --io-no-correctness
+run 4 8 64.64.64.128  8.4.1.1  "19.3 GB, row of 32, x heavy"    --aggregate-target 4194304 --io-no-correctness
+
+# 10.9 GB with non-uniform counts, 453 MB/rank, 27 MB extents, 24 ranks.
+run 3 8 48.96.64.64   3.4.2.1  "10.9 GB, NON-UNIFORM counts"    --aggregate-target 4194304 --io-no-correctness
+
+#####################################################################
+# Phase 3.  Large volume, 43.5 GB record, 1.36 GB/rank.
+# Host side buffering is 3 copies of the local slice, so ~4 GB/rank
+# and ~33 GB/node.  Two repetitions only.
+#####################################################################
+run 4 8 64.64.96.192  1.1.4.8  "43.5 GB, baseline"  --aggregate-target 4194304 --io-no-correctness --io-reps 2
+run 4 8 64.64.96.192  4.4.2.1  "43.5 GB, row of 32" --aggregate-target 4194304 --io-no-correctness --io-reps 2
+
+echo
+echo "=== done.  Output left in $WORK"
+ls -l $WORK

systems/Frontier/aggregate_io_scaling.slurm

@@ -0,0 +1,104 @@
+#!/bin/bash -l
+
+# Node count scan.  Everything that could confound the comparison is held
+# fixed and only the number of Lustre clients changes:
+#
+#   local volume   8.8.32.128        = 262144 sites x 576 B = 151 MB / rank
+#   aggregation    k=2, row of 16, 8 extents of 18 MB        (identical at
+#                                                             every node count)
+#   old path       contiguous run = lLattice[0]*576 = 4.6 KB (identical too)
+#
+# So the ratio of contiguous extent between the two layout preserving paths is
+# a constant 4000x across the whole scan, and any change in their relative
+# bandwidth is a property of the client count alone.  That is the variable the
+# 4 node runs could not move.
+#
+# Trim the tail of the run list if you want a smaller allocation; the
+# geometries are independent.
+
+#SBATCH --job-name=ioScale
+#SBATCH --nodes=128
+#SBATCH --ntasks-per-node=8
+#SBATCH --cpus-per-task=7
+#SBATCH --gpus-per-node=8
+#SBATCH --time=02:00:00
+#SBATCH --account=phy157_dwf
+#SBATCH --gpu-bind=none
+#SBATCH --exclusive
+#SBATCH --mem=0
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3 7 6 5 4)
+export NUMA_MAP=(3 3 1 1 2 2 0 0)
+export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
+export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
+unset ROCR_VISIBLE_DEVICES
+export HIP_VISIBLE_DEVICES=\$GPU
+
+rank=\$SLURM_PROCID
+CMD="numactl -m \$NUMA -N \$NUMA  \$*"
+exec \$CMD
+EOF
+
+chmod +x ./select_gpu
+SELECT=$SLURM_SUBMIT_DIR/select_gpu
+
+root=$HOME/ParallelIO/systems/Frontier/
+source ${root}/sourceme-rocm7.2.sh
+
+export OMP_NUM_THREADS=7
+export MPICH_GPU_SUPPORT_ENABLED=1
+
+WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/aggscale.$SLURM_JOB_ID
+mkdir -p $WORK
+cd       $WORK
+
+# The largest rung writes three files of 154 GB, so budget ~700 GB for the
+# whole scan and check the quota before submitting.
+#
+# Do NOT stripe by default.  Measured on Frontier at 128 nodes / 1024 ranks,
+# 154.6 GB record, best of 3 (MB/s):
+#
+#                     default layout    lfs setstripe -c -1 -S 8M
+#   raw                    97673                81728
+#   MPI-IO lexicographic   31675                77534
+#   aggregate             189535                86063
+#
+# The two layout preserving paths want opposite things.  Wide striping is what
+# rescues MPI-IO at scale, 32 -> 78 GB/s, which is presumably what careful
+# striping has been buying in production.  It costs the aggregate path a factor
+# of 2.2, and costs raw as well.  With the default layout MPI-IO plateaus at
+# ~30 GB/s from 64 nodes upward -- the 20-40 GB/s seen in production -- while
+# the aggregate path keeps scaling.  Uncomment only to reproduce that.
+#lfs setstripe -c -1 -S 8M $WORK
+lfs getstripe -d $WORK      # record the inherited layout; needed to read the numbers
+
+BIN=$SLURM_SUBMIT_DIR/tests/IO/Test_aggregate_io
+ARGS="--aggregate-target 4194304 --io-no-correctness --io-reps 3"
+
+run () {   # run <nodes> <grid> <mpi> <comment>
+  local nodes=$1 nranks=$(( $1 * 8 ))
+  echo
+  echo "==================================================================="
+  echo "=== SCAN nodes=$nodes ranks=$nranks  grid=$2  mpi=$3   $4"
+  echo "==================================================================="
+  srun -N$nodes -n$nranks --ntasks-per-node=8 $SELECT $BIN --grid $2 --mpi $3 \
+       --accelerator-threads 8 --shm 2048 --device-mem 32000 $ARGS
+  echo "=== exit $?"
+}
+
+#            nodes  global lattice      decomposition   record
+run    4     32.32.64.128    4.4.2.1   "4.8 GB"
+run    8     32.32.64.256    4.4.2.2   "9.7 GB"
+run   16     32.32.64.512    4.4.2.4   "19.3 GB"
+run   32     32.32.128.512   4.4.4.4   "38.6 GB"
+run   64     32.32.128.1024  4.4.4.8   "77.3 GB"
+run  128     32.32.256.1024  4.4.8.8   "154.6 GB"
+
+echo
+echo "=== done.  Files left in $WORK"
+echo "=== For cold read numbers, submit a SECOND job on a fresh allocation"
+echo "=== with WORK pointed at this directory and --io-read-only added to"
+echo "=== ARGS.  Reading back what this job wrote only measures page cache."
+ls -l $WORK

systems/Frontier/config-command

@@ -1,4 +1,5 @@
 CLIME=`spack find --paths c-lime@2-3-9 | grep c-lime| cut -c 15-`
+
 ../../configure --enable-comms=mpi-auto \
 --with-lime=$CLIME \
 --enable-unified=no \
@@ -9,12 +10,13 @@ CLIME=`spack find --paths c-lime@2-3-9 | grep c-lime| cut -c 15-`
 --disable-gparity \
 --disable-fermion-reps \
 --enable-simd=GPU \
---with-gmp=$OLCF_GMP_ROOT \
+--with-openssl=$OPENSSL \
---with-mpfr=/opt/cray/pe/gcc/mpfr/3.1.4/ \
+--with-gmp=$GMP \
+--with-mpfr=$MPFR \
 --disable-fermion-reps \
 CXX=hipcc MPICXX=mpicxx \
-CXXFLAGS="-fPIC -I${ROCM_PATH}/include/ -I${MPICH_DIR}/include -L/lib64 " \
+CXXFLAGS="-fPIC -I${ROCM_PATH}/include/ -I${MPICH_DIR}/include " \
- LDFLAGS="-L/lib64 -L${ROCM_PATH}/lib -L${MPICH_DIR}/lib -lmpi -L${CRAY_MPICH_ROOTDIR}/gtl/lib -lmpi_gtl_hsa -lhipblas -lrocblas -lhipfft"
+ LDFLAGS="-L${ROCM_PATH}/lib -L${MPICH_DIR}/lib -lmpi -lmpi_gtl_hsa -lhipblas -lrocsolver -lrocblas -lhipfft -lamdhip64"
 
 
 

systems/Frontier/sourceme-rocm6.4.sh

@@ -0,0 +1,15 @@
+
+echo spack
+. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
+
+export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
+export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
+export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
+export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
+
+module load cce/20.0.0
+module load cpe/25.09
+module load rocm/6.4.0
+export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/rocm-6.4.0/lib/llvm/lib/:$LD_LIBRARY_PATH
+module load emacs

systems/Frontier/sourceme-rocm7.0.sh

@@ -0,0 +1,13 @@
+
+echo spack
+. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
+
+
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.0.2
+
+export LD_LIBRARY_PATH=/opt/rocm-7.0.2/lib/llvm/lib/:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CRAY_LD_LIBRARY_PATH
+
+module load emacs

systems/Frontier/sourceme-rocm7.2.sh

@@ -0,0 +1,15 @@
+
+echo spack
+. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
+
+export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
+export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
+export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
+export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
+
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.2.0
+export LD_LIBRARY_PATH=/opt/rocm-7.2.0/lib/llvm/lib/:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CRAY_LD_LIBRARY_PATH
+module load emacs

systems/Frontier/sourceme.sh

@@ -1,28 +0,0 @@
-
-echo spack
-. /autofs/nccs-svm1_home1/paboyle/Crusher/Grid/spack/share/spack/setup-env.sh
-
-module load cce/15.0.1
-module load amd/7.0.2
-#module load amd/7.1.1
-#module load rocm/7.2.0
-#module load rocm/6.4.2
-module load cray-fftw
-module load craype-accel-amd-gfx90a
-
-#Ugly hacks to get down level software working on current system
-export LD_LIBRARY_PATH=/opt/cray/libfabric/1.20.1/lib64/:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=/opt/gcc/mpfr/3.1.4/lib:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=`pwd`/:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/LD_PATH/
-
-#echo spack load c-lime
-#spack load c-lime
-#module load emacs 
-##module load PrgEnv-gnu
-##module load cray-mpich
-##module load cray-fftw
-##module load craype-accel-amd-gfx90a
-##export LD_LIBRARY_PATH=/opt/gcc/mpfr/3.1.4/lib:$LD_LIBRARY_PATH
-#Hack for lib
-##export LD_LIBRARY_PATH=`pwd`/:$LD_LIBRARY_PATH

systems/Perlmutter/config-command

@@ -1,12 +1,12 @@
 DIR=`pwd`
-PREFIX=$HOME/DDHMC/Grid/systems/Prerequisites/install/
+
 ../../configure \
     --enable-comms=mpi \
     --enable-simd=GPU \
     --enable-shm=nvlink \
     --enable-gen-simd-width=64 \
-    --with-gmp=$PREFIX \
+    --with-gmp=$GMP \
-    --with-mpfr=$PREFIX \
+    --with-mpfr=$MPFR \
     --enable-accelerator=cuda \
     --disable-fermion-reps \
     --disable-unified \

systems/Perlmutter/sourceme.sh

@@ -1,4 +1,6 @@
-
 export CRAY_ACCEL_TARGET=nvidia80
+source /global/homes/p/pboyle/spack/share/spack/setup-env.sh
+export MPFR=`spack find --paths mpfr | grep mpfr | cut -c 13-`
+export GMP=`spack find --paths gmp   | grep gmp  | cut -c 12-`
 
-module load PrgEnv-gnu cpe-cuda cudatoolkit/11.4
+module load PrgEnv-gnu cpe-cuda cudatoolkit/12.0

systems/mac-arm/config-command-mpi

@@ -3,7 +3,10 @@
 CXX=mpicxx ../../configure \
 	   --enable-simd=GEN \
 	   --enable-comms=mpi-auto \
-	   --enable-Sp=yes \
+	   --enable-Sp=no \
+	   --disable-fermion-reps \
+	   --disable-gparity \
+	   --with-fftw=$FFTW \
 	   --enable-unified=yes \
 	   --prefix /Users/peterboyle/QCD/vtk/Grid/install \
 	   --with-lime=$CLIME \

systems/mac-arm/sourceme.sh

@@ -0,0 +1,11 @@
+source /Users/peterboyle/QCD//Spack/spack//share/spack/setup-env.sh
+
+export FFTW=`spack find --paths fftw    | grep ^fftw   | awk '{print $2}' `
+#export HDF5=`spack find --paths hdf5+cxx   | grep ^hdf5   | awk '{print $2}' `
+export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
+export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
+export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
+export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
+
+export LD_LIBRARY_PATH=$MPFR/lib:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$GMP/lib:$LD_LIBRARY_PATH

tests/IO/Test_aggregate_io.cc

@@ -0,0 +1,458 @@
+    /*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/IO/Test_aggregate_io.cc
+
+    Copyright (C) 2015
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// Correctness and performance test for BINARYIO_AGGREGATE.
+//
+// Correctness, per aggregateTargetBytes:
+//   1. write via the MPI-IO lexicographic path      -> ref.bin
+//   2. write via the aggregate path                 -> agg.bin
+//   3. the two files must be byte identical         <- proves the layout matches
+//   4. read agg.bin back through the aggregate path <- proves the mirror inverts
+//   5. write and read the non-lexicographic path    -> raw.bin.  Its layout is
+//      different by construction (each rank owns one contiguous segment in
+//      rank order) so it cannot be compared byte for byte, but the NERSC and
+//      SciDAC checksums are computed from the global site index and are
+//      therefore layout independent: they must match the other two paths.
+//   6. a record written over a longer pre-existing file must leave the file at
+//      exactly offset+payload, with no trailing fragment of the old contents
+//
+// Performance: three paths, both directions, timed with the client page cache
+// dropped before every read so that a read back reports filesystem bandwidth
+// rather than memory bandwidth.  The non-lexicographic path is the zero
+// overhead reference: no transposition, no layout independence, one disjoint
+// contiguous segment per rank, which is the arrangement that reaches full
+// filesystem bandwidth on a leadership machine.  It is the upper bound the
+// other two are trying to approach.
+//
+// Options:
+//   --aggregate-target <bytes>   sweep this one target only (default: sweep
+//                                1, 1024, 64K, 4M)
+//   --io-reps <n>                repetitions in the performance section
+//                                (default 3; 0 disables it)
+//   --io-no-correctness          skip the correctness section, which reads the
+//                                whole file on one rank and is not affordable
+//                                at very large volume
+//   --io-read-only               time reads only, of files left in place by an
+//                                earlier job.  Reading back what this job just
+//                                wrote measures the client page cache; a fresh
+//                                allocation pointed at the same directory is
+//                                the only way to get a cold read without root.
+//
+// The exchange is only meaningfully exercised when the fast dimensions are
+// split across ranks; --mpi 1.1.X.Y leaves the rows of size one and the test
+// then passes vacuously.  Non-uniform AllToAllV counts additionally need an
+// odd process factor in a fast dimension and a small local volume.
+
+#include <Grid/Grid.h>
+#include <fcntl.h>
+
+using namespace Grid;
+
+/////////////////////////////////////////////////////////////////////////////
+// Compare in chunks.  Slurping both files into memory is fine for a few MB
+// and fatal for the multi-GB records this test is meant to reach.
+/////////////////////////////////////////////////////////////////////////////
+static bool FilesIdentical(std::string a,std::string b)
+{
+  std::ifstream fa(a,std::ios::binary), fb(b,std::ios::binary);
+  if ( !fa.good() || !fb.good() ) {
+    std::cout<<GridLogMessage<<"  could not open "<<a<<" and/or "<<b<<std::endl;
+    return false;
+  }
+  fa.seekg(0,std::ios::end); fb.seekg(0,std::ios::end);
+  uint64_t sa = (uint64_t)fa.tellg(), sb = (uint64_t)fb.tellg();
+  if ( sa != sb ) {
+    std::cout<<GridLogMessage<<"  size mismatch "<<sa<<" vs "<<sb<<std::endl;
+    return false;
+  }
+  fa.seekg(0,std::ios::beg); fb.seekg(0,std::ios::beg);
+
+  const uint64_t chunk = 8*1024*1024;
+  std::vector<char> va(chunk), vb(chunk);
+  uint64_t done=0;
+  while ( done < sa ) {
+    uint64_t n = std::min(chunk,sa-done);
+    fa.read(&va[0],n);
+    fb.read(&vb[0],n);
+    for(uint64_t i=0;i<n;i++){
+      if ( va[i]!=vb[i] ) {
+        std::cout<<GridLogMessage<<"  first differing byte at "<<done+i<<" of "<<sa<<std::endl;
+        return false;
+      }
+    }
+    done += n;
+  }
+  return true;
+}
+
+/////////////////////////////////////////////////////////////////////////////
+// Reading back a file we have just written measures the client page cache,
+// not the filesystem: the earlier runs of this test reported 8 GB/s on reads
+// and ~1 GB/s on writes for the same data.  POSIX_FADV_DONTNEED asks the
+// kernel to drop the cached pages for the file.  It is advisory and every
+// rank must do it, since each client caches independently, so treat this as
+// best effort rather than a guarantee of a cold read.
+/////////////////////////////////////////////////////////////////////////////
+static void DropCache(GridBase *grid,std::string file)
+{
+  grid->Barrier();
+  int fd = ::open(file.c_str(),O_RDONLY);
+  if ( fd >= 0 ) {
+#ifdef POSIX_FADV_DONTNEED
+    ::posix_fadvise(fd,0,0,POSIX_FADV_DONTNEED);
+#endif
+    ::close(fd);
+  }
+  grid->Barrier();
+}
+
+static uint64_t OptionU64(int argc,char **argv,const char *opt,uint64_t def)
+{
+  if ( GridCmdOptionExists(argv,argv+argc,opt) ) {
+    std::string arg = GridCmdOptionPayload(argv,argv+argc,opt);
+    return (uint64_t)std::stoull(arg);
+  }
+  return def;
+}
+
+int main(int argc,char **argv)
+{
+  Grid_init(&argc,&argv);
+
+  Coordinate latt = GridDefaultLatt();
+  Coordinate simd = GridDefaultSimd(Nd,vComplexD::Nsimd());
+  Coordinate mpi  = GridDefaultMpi();
+  GridCartesian grid(latt,simd,mpi);
+
+  typedef vLorentzColourMatrixD vobj;
+  typedef LorentzColourMatrixD  sobj;
+
+  GridParallelRNG pRNG(&grid);
+  pRNG.SeedFixedIntegers(std::vector<int>({1,2,3,4}));
+  LatticeGaugeFieldD Umu(&grid);
+  random(pRNG,Umu);
+
+  BinarySimpleMunger<sobj,sobj> munge;
+  const std::string format("IEEE64BIG");
+
+  const int lex = BinaryIO::BINARYIO_LEXICOGRAPHIC;
+  const int agg = BinaryIO::BINARYIO_LEXICOGRAPHIC|BinaryIO::BINARYIO_AGGREGATE;
+  const int raw = 0;   // no BINARYIO_LEXICOGRAPHIC: contiguous segment per rank
+
+  uint64_t payload = (uint64_t)grid._gsites*sizeof(sobj);
+
+  std::vector<uint64_t> targets = {1, 1024, 64*1024, 4*1024*1024};
+  if ( GridCmdOptionExists(argv,argv+argc,"--aggregate-target") ) {
+    targets.clear();
+    targets.push_back(OptionU64(argc,argv,"--aggregate-target",4*1024*1024));
+  }
+  uint64_t reps        = OptionU64(argc,argv,"--io-reps",3);
+  bool     correctness = !GridCmdOptionExists(argv,argv+argc,"--io-no-correctness");
+  // Read only: time reads of files left by an earlier job.  The only way to
+  // get a cold client cache without root is to read on an allocation that did
+  // not write the data, so run one job to write and a second, pointed at the
+  // same directory, with this flag.
+  bool     readonly    = GridCmdOptionExists(argv,argv+argc,"--io-read-only");
+  if ( readonly ) correctness = false;
+
+  std::cout<<GridLogMessage<<"Record payload "<<payload<<" bytes = "
+           <<payload/1024./1024.<<" MB, "
+           <<payload/(RealD)grid.ProcessorCount()/1024./1024.<<" MB/rank"<<std::endl;
+
+  int failures=0;
+
+  //////////////////////////////////////////////////////////////////////////
+  // Correctness
+  //////////////////////////////////////////////////////////////////////////
+  if ( correctness ) for(auto target : targets){
+
+    std::cout<<GridLogMessage<<"=== correctness, aggregateTargetBytes = "<<target<<" ==="<<std::endl;
+
+    uint32_t n1,a1,b1, n2,a2,b2, n3,a3,b3;
+    uint64_t off;
+
+    // Start from a clean slate.  The aggregate path sets the file length to
+    // exactly offset+payload; the MPI-IO path (MPI_MODE_CREATE) leaves any
+    // pre-existing tail in place.  Comparing stale files would therefore
+    // report a size mismatch that says nothing about the payload.
+    if ( grid.IsBoss() ) { ::unlink("ref.bin"); ::unlink("agg.bin"); ::unlink("raw.bin"); }
+    grid.Barrier();
+
+    off=0;
+    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"ref.bin",munge,off,format,n1,a1,b1,lex);
+
+    BinaryIO::aggregateTargetBytes = target;
+    off=0;
+    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"agg.bin",munge,off,format,n2,a2,b2,agg);
+
+    grid.Barrier();
+
+    if ( grid.IsBoss() ) {
+      if ( !FilesIdentical("ref.bin","agg.bin") ) {
+        std::cout<<GridLogError<<"  FAIL: aggregate file differs from lexicographic file"<<std::endl;
+        failures++;
+      } else {
+        std::cout<<GridLogMessage<<"  files byte identical"<<std::endl;
+      }
+    }
+    if ( (n1!=n2)||(a1!=a2)||(b1!=b2) ) {
+      std::cout<<GridLogError<<"  FAIL: checksum mismatch between paths"<<std::endl;
+      failures++;
+    }
+    // writeLatticeObject takes offset by value, so the out-parameter that
+    // IOobject sets never reaches us here and cannot be checked directly.
+    // The observable equivalent is the file length: both paths must leave the
+    // record ending at exactly offset+payload.
+    if ( grid.IsBoss() ) {
+      for(auto f : {std::string("ref.bin"),std::string("agg.bin")}){
+        std::ifstream fs(f,std::ios::binary|std::ios::ate);
+        uint64_t sz = (uint64_t)fs.tellg();
+        if ( sz != payload ) {
+          std::cout<<GridLogError<<"  FAIL: "<<f<<" is "<<sz<<" bytes, expected "<<payload<<std::endl;
+          failures++;
+        }
+      }
+    }
+
+    LatticeGaugeFieldD Uchk(&grid);
+    DropCache(&grid,"agg.bin");
+    off=0;
+    BinaryIO::readLatticeObject<vobj,sobj>(Uchk,"agg.bin",munge,off,format,n3,a3,b3,agg);
+    if ( (n3!=n1)||(a3!=a1)||(b3!=b1) ) {
+      std::cout<<GridLogError<<"  FAIL: read back checksum mismatch"<<std::endl;
+      failures++;
+    }
+    Uchk = Uchk - Umu;
+    RealD residual = norm2(Uchk);
+    std::cout<<GridLogMessage<<"  read back residual "<<residual<<std::endl;
+    if ( residual != 0.0 ) {
+      std::cout<<GridLogError<<"  FAIL: read back does not reproduce the field"<<std::endl;
+      failures++;
+    }
+
+    ////////////////////////////////////////////////////////////////////////
+    // Non-lexicographic.  Different file layout by construction, so compare
+    // by checksum and by round trip rather than by bytes.
+    ////////////////////////////////////////////////////////////////////////
+    uint32_t n4,a4,b4, n5,a5,b5;
+    off=0;
+    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"raw.bin",munge,off,format,n4,a4,b4,raw);
+    grid.Barrier();
+    if ( (n4!=n1)||(a4!=a1)||(b4!=b1) ) {
+      std::cout<<GridLogError<<"  FAIL: raw path checksum differs; the NERSC and"
+               <<" SciDAC checksums are layout independent and must agree"<<std::endl;
+      failures++;
+    }
+    DropCache(&grid,"raw.bin");
+    off=0;
+    BinaryIO::readLatticeObject<vobj,sobj>(Uchk,"raw.bin",munge,off,format,n5,a5,b5,raw);
+    if ( (n5!=n1)||(a5!=a1)||(b5!=b1) ) {
+      std::cout<<GridLogError<<"  FAIL: raw read back checksum mismatch"<<std::endl;
+      failures++;
+    }
+    Uchk = Uchk - Umu;
+    RealD rawresidual = norm2(Uchk);
+    std::cout<<GridLogMessage<<"  raw read back residual "<<rawresidual<<std::endl;
+    if ( rawresidual != 0.0 ) {
+      std::cout<<GridLogError<<"  FAIL: raw read back does not reproduce the field"<<std::endl;
+      failures++;
+    }
+  }
+
+  //////////////////////////////////////////////////////////////////////////
+  // Truncation.  offset!=0 is the case that matters: it is what ILDG and
+  // NERSC use, and it is the branch that used to only ever grow the file.
+  // The oversized starting file is made by extending a sparse one rather
+  // than writing padding from a single rank, which does not scale.
+  //////////////////////////////////////////////////////////////////////////
+  if ( correctness ) {
+    BinaryIO::aggregateTargetBytes = 4*1024*1024;
+    for(uint64_t testOffset : {(uint64_t)0, (uint64_t)1024}){
+
+      uint64_t expect = testOffset + payload;
+
+      if ( grid.IsBoss() ) {
+        { std::ofstream create("trunc.bin",std::ios::binary|std::ios::out); create.close(); }
+        int ierr = ::truncate("trunc.bin",(off_t)(expect+65536));
+        GRID_ASSERT(ierr==0);
+      }
+      grid.Barrier();
+
+      uint32_t n,a,b;
+      uint64_t off = testOffset;
+      BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"trunc.bin",munge,off,format,n,a,b,agg);
+      grid.Barrier();
+
+      if ( grid.IsBoss() ) {
+        std::ifstream f("trunc.bin",std::ios::binary|std::ios::ate);
+        uint64_t sz = (uint64_t)f.tellg();
+        f.close();
+        if ( sz != expect ) {
+          std::cout<<GridLogError<<"  FAIL: offset "<<testOffset<<" left file "<<sz
+                   <<" bytes, expected "<<expect<<std::endl;
+          failures++;
+        } else {
+          std::cout<<GridLogMessage<<"  truncation ok at offset "<<testOffset
+                   <<": file is exactly "<<sz<<" bytes"<<std::endl;
+        }
+      }
+    }
+  }
+
+  //////////////////////////////////////////////////////////////////////////
+  // Performance.  Four numbers per repetition: write and read, old path and
+  // new.  Reads are preceded by a cache drop; writes are not, so a write
+  // number is "time to hand the data to the client cache and close", the
+  // same convention for both paths.
+  //////////////////////////////////////////////////////////////////////////
+  if ( reps ) {
+
+    LatticeGaugeFieldD Uio(&grid);
+    uint32_t n,a,b;
+
+    // The NERSC and SciDAC checksums are computed from the global site index,
+    // so all three layouts must produce the same values.  This costs nothing
+    // and is the only correctness check available at a volume where the byte
+    // for byte comparison (single rank, whole file) is unaffordable.
+    uint32_t cn[6],ca[6],cb[6];
+    auto agreeing = [&](const char *what,int lo,int hi){
+      for(int i=lo+1;i<=hi;i++){
+        if ( (cn[i]!=cn[lo])||(ca[i]!=ca[lo])||(cb[i]!=cb[lo]) ) {
+          std::cout<<GridLogError<<"  FAIL: "<<what<<" checksums disagree between paths"<<std::endl;
+          return false;
+        }
+      }
+      return true;
+    };
+
+    for(auto target : targets){
+
+      BinaryIO::aggregateTargetBytes = target;
+      std::cout<<GridLogMessage<<"=== performance, aggregateTargetBytes = "<<target<<" ==="<<std::endl;
+
+      std::vector<RealD> wref,wagg,wraw,rref,ragg,rraw;
+
+      for(uint64_t rep=0;rep<reps;rep++){
+
+        uint64_t off;
+
+        // Unlink only before the first repetition.  Lustre metadata cost is
+        // per file, not per byte, so rep 0 reports "create the file and write
+        // it" and the later reps report the steady state of overwriting an
+        // existing file -- which is what a multi record file does for every
+        // record after the first, and what production actually looks like.
+        if ( (rep==0) && !readonly && grid.IsBoss() ) {
+          ::unlink("ref.bin"); ::unlink("agg.bin"); ::unlink("raw.bin");
+        }
+        grid.Barrier();
+
+        if ( !readonly ) {
+
+        off=0;
+        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"ref.bin",munge,off,format,n,a,b,lex);
+        wref.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[0]=n; ca[0]=a; cb[0]=b;
+
+        off=0;
+        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"agg.bin",munge,off,format,n,a,b,agg);
+        wagg.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[1]=n; ca[1]=a; cb[1]=b;
+
+        off=0;
+        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"raw.bin",munge,off,format,n,a,b,raw);
+        wraw.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[2]=n; ca[2]=a; cb[2]=b;
+        if ( !agreeing("write",0,2) ) failures++;
+
+        }  // !readonly
+
+        DropCache(&grid,"ref.bin");
+        off=0;
+        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"ref.bin",munge,off,format,n,a,b,lex);
+        rref.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[3]=n; ca[3]=a; cb[3]=b;
+
+        DropCache(&grid,"agg.bin");
+        off=0;
+        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"agg.bin",munge,off,format,n,a,b,agg);
+        ragg.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[4]=n; ca[4]=a; cb[4]=b;
+
+        DropCache(&grid,"raw.bin");
+        off=0;
+        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"raw.bin",munge,off,format,n,a,b,raw);
+        rraw.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[5]=n; ca[5]=a; cb[5]=b;
+        if ( !agreeing("read back",readonly?3:0,5) ) failures++;
+      }
+
+      if ( grid.IsBoss() ) {
+        auto report = [&](const char *name,std::vector<RealD> &v){
+          if ( v.empty() ) return;
+          RealD best=0, sum=0;
+          for(auto x : v){ if(x>best) best=x; sum+=x; }
+          // First sample includes file creation, later ones do not; quote both
+          // rather than a mean that mixes the two.
+          std::cout<<GridLogMessage<<"  PERF target="<<target<<" "<<name
+                   <<" best "<<best<<" MB/s, mean "<<sum/v.size()
+                   <<" MB/s, first(cold create) "<<v[0]<<" MB/s, samples";
+          for(auto x : v) std::cout<<" "<<x;
+          std::cout<<std::endl;
+        };
+        report("write raw       ",wraw);   // zero overhead reference
+        report("write MPI-IO    ",wref);
+        report("write aggregate ",wagg);
+        report("read  raw       ",rraw);
+        report("read  MPI-IO    ",rref);
+        report("read  aggregate ",ragg);
+
+        // Fraction of the zero overhead reference that each layout preserving
+        // path achieves.  This is the number the whole exercise is about.
+        auto best = [](std::vector<RealD> &v){ RealD m=0; for(auto x:v) if(x>m) m=x; return m; };
+        if ( !wraw.empty() && best(wraw) > 0 ) {
+          std::cout<<GridLogMessage<<"  PERF target="<<target
+                   <<" write fraction of raw: MPI-IO "<<best(wref)/best(wraw)
+                   <<"  aggregate "<<best(wagg)/best(wraw)<<std::endl;
+        }
+        if ( !rraw.empty() && best(rraw) > 0 ) {
+          std::cout<<GridLogMessage<<"  PERF target="<<target
+                   <<" read  fraction of raw: MPI-IO "<<best(rref)/best(rraw)
+                   <<"  aggregate "<<best(ragg)/best(rraw)<<std::endl;
+        }
+      }
+    }
+  }
+
+  if ( grid.IsBoss() ) {
+    if ( failures ) std::cout<<GridLogError  <<failures<<" FAILURE(S)"<<std::endl;
+    else            std::cout<<GridLogMessage<<"ALL AGGREGATE IO TESTS PASSED"<<std::endl;
+  }
+
+  Grid_finalize();
+  return failures!=0;
+}

tests/Test_fft_memory.cc

@@ -0,0 +1,261 @@
+/*************************************************************************************
+
+ Test_fft_memory.cc
+
+ Memory growth test for PlannedFFT on a spin-colour matrix (propagator) field.
+
+ The test creates a single PlannedFFT object (which allocates FFTW plans once),
+ then repeatedly applies FFT_all_dim to the same propagator 400 times.
+
+ If PlannedFFT is working correctly the RSS should remain flat after the first
+ iteration — no new plans, no new deviceVector allocations beyond the per-call
+ pencil buffer which is freed at the end of each FFT_dim_execute call.
+
+ Build exactly like any other Grid test, e.g.:
+   make Test_fft_memory
+ or compile manually:
+   $(CXX) $(CXXFLAGS) Test_fft_memory.cc -o Test_fft_memory $(LDFLAGS)
+
+*************************************************************************************/
+
+#include <Grid/Grid.h>
+using namespace Grid;
+
+// --------------------------------------------------------------------------
+// Helper: read RSS (resident set size) in kB from /proc/self/status.
+// Returns 0 on platforms where /proc is unavailable.
+// --------------------------------------------------------------------------
+static long getCPURSSKb()
+{
+  long rss = 0;
+  FILE *fp = fopen("/proc/self/status", "r");
+  if (!fp) return -1;
+  char line[256];
+  while (fgets(line, sizeof(line), fp)) {
+    if (strncmp(line, "VmRSS:", 6) == 0) {
+      sscanf(line + 6, "%ld", &rss);
+      break;
+    }
+  }
+  fclose(fp);
+  return rss;
+}
+
+static long getGPUUsedMb()
+{
+#if defined(GRID_CUDA)
+  size_t free_bytes  = 0;
+  size_t total_bytes = 0;
+  cudaError_t err = cudaMemGetInfo(&free_bytes, &total_bytes);
+  if (err != cudaSuccess) return -1;
+  return (long)((total_bytes - free_bytes) / (1024 * 1024));
+ 
+#elif defined(GRID_HIP)
+  size_t free_bytes  = 0;
+  size_t total_bytes = 0;
+  hipError_t err = hipMemGetInfo(&free_bytes, &total_bytes);
+  if (err != hipSuccess) return -1;
+  return (long)((total_bytes - free_bytes) / (1024 * 1024));
+ 
+#else
+  return -1;   // CPU-only build: no GPU to query
+#endif
+}
+
+// ============================================================
+//  Convenience struct — one snapshot of both sides
+// ============================================================
+struct MemSnapshot {
+  long cpu_rss_kb;   // host RSS in kB  (-1 if unavailable)
+  long gpu_used_mb;  // device used in MB (-1 if no GPU)
+};
+ 
+static MemSnapshot takeSnapshot()
+{
+  MemSnapshot s;
+  s.cpu_rss_kb  = getCPURSSKb();
+  s.gpu_used_mb = getGPUUsedMb();
+  return s;
+}
+ 
+// ============================================================
+//  Pretty-print one row of the monitoring table
+// ============================================================
+static void printRow(int iter,
+                     const MemSnapshot &now,
+                     const MemSnapshot &prev)
+{
+  long cpu_delta = (now.cpu_rss_kb  >= 0 && prev.cpu_rss_kb  >= 0)
+                 ? now.cpu_rss_kb  - prev.cpu_rss_kb  : 0;
+  long gpu_delta = (now.gpu_used_mb >= 0 && prev.gpu_used_mb >= 0)
+                 ? now.gpu_used_mb - prev.gpu_used_mb : 0;
+ 
+  // Sign prefix so deltas are unambiguous
+  auto sign = [](long v) -> const char* { return v >= 0 ? "+" : ""; };
+ 
+  std::cout << GridLogMessage
+            << std::setw(6)  << iter
+            << "  CPU: " << std::setw(10) << now.cpu_rss_kb  << " kB"
+            << "  (" << sign(cpu_delta) << std::setw(7) << cpu_delta << " kB)"
+            << "  GPU: " << std::setw(7)  << now.gpu_used_mb << " MB"
+            << "  (" << sign(gpu_delta) << std::setw(5) << gpu_delta << " MB)"
+            << "\n";
+}
+
+// ============================================================
+
+int main(int argc, char **argv)
+{
+  Grid_init(&argc, &argv);
+
+  int threads = GridThread::GetThreads();
+  std::cout << GridLogMessage
+            << "Grid is setup to use " << threads << " threads" << std::endl;
+
+  // ------------------------------------------------------------------
+  // Grid setup — use whatever lattice/mpi/simd was passed on the CLI,
+  // e.g.  --grid 8.8.8.8 --mpi 1.1.1.1
+  // ------------------------------------------------------------------
+  Coordinate latt_size   = GridDefaultLatt();
+  Coordinate simd_layout = GridDefaultSimd(Nd, vComplexD::Nsimd());
+  Coordinate mpi_layout  = GridDefaultMpi();
+
+  GridCartesian GRID(latt_size, simd_layout, mpi_layout);
+
+  int vol = 1;
+  for (int d = 0; d < (int)latt_size.size(); d++) vol *= latt_size[d];
+
+  std::cout << GridLogMessage << "Lattice : ";
+  for (int d = 0; d < Nd; d++) std::cout << latt_size[d] << " ";
+  std::cout << std::endl;
+
+  // ------------------------------------------------------------------
+  // Propagator field: SpinColourMatrix = 12x12 complex, i.e.
+  // LatticePropagatorD (= Lattice<iSpinColourMatrix<vComplexD>>).
+  // This is the standard QCD quark propagator type.
+  // ------------------------------------------------------------------
+  LatticePropagatorD prop(&GRID);
+  
+  // ------------------------------------------------------------------
+  // Fill the propagator with a momentum-space plane wave,
+  // following the pattern from Test_fft.cc.
+  // We set each spin-colour component (a,b) to exp(i * sum_mu p_mu x_mu)
+  // with a fixed momentum p = (1,2,1,2).
+  // ------------------------------------------------------------------
+  Coordinate pvec({1, 2, 1, 2});
+
+  LatticeComplexD phase(&GRID);
+  LatticeComplexD coor(&GRID);
+  ComplexD ci(0.0, 1.0);
+
+  phase = Zero();
+  for (int mu = 0; mu < Nd; mu++) {
+    RealD TwoPiL = M_PI * 2.0 / latt_size[mu];
+    LatticeCoordinate(coor, mu);
+    phase = phase + (TwoPiL * pvec[mu]) * coor;
+  }
+  phase = exp(phase * ci);   // e^{i p.x}
+
+  // Broadcast the phase into every spin-colour matrix entry
+  prop = Zero();
+  prop = prop + phase;
+
+  std::cout << GridLogMessage
+            << "Propagator norm2 = " << norm2(prop) << std::endl;
+
+  // ------------------------------------------------------------------
+  // Baseline snapshot BEFORE PlannedFFT construction
+  // ------------------------------------------------------------------
+  MemSnapshot snap_before_plan = takeSnapshot();
+  std::cout << GridLogMessage
+            << "[mem] Before PlannedFFT construction"
+            << "  CPU: " << snap_before_plan.cpu_rss_kb  << " kB"
+            << "  GPU: " << snap_before_plan.gpu_used_mb << " MB"
+            << std::endl;
+ 
+  // ------------------------------------------------------------------
+  // Create the PlannedFFT — plans are allocated here ONCE for all
+  // dimensions and stored inside the object.
+  // ------------------------------------------------------------------
+  PlannedFFT<iSpinColourMatrix<vComplexD>> plannedFFT(&GRID);
+
+  // ------------------------------------------------------------------
+  // Snapshot AFTER plan construction — this is the true baseline
+  // for the loop, because cufftPlanMany itself grabs device memory.
+  // ------------------------------------------------------------------
+  MemSnapshot snap_after_plan = takeSnapshot();
+  std::cout << GridLogMessage
+            << "[mem] After  PlannedFFT construction"
+            << "  CPU: " << snap_after_plan.cpu_rss_kb  << " kB"
+            << "  GPU: " << snap_after_plan.gpu_used_mb << " MB"
+            << "  (plan overhead:"
+            << "  CPU +" << snap_after_plan.cpu_rss_kb - snap_before_plan.cpu_rss_kb << " kB"
+            << "  GPU +" << snap_after_plan.gpu_used_mb - snap_before_plan.gpu_used_mb << " MB)"
+            << std::endl;
+
+  MemoryManager::Print();
+  // ------------------------------------------------------------------
+  // 400-iteration loop.
+  // Each iteration computes the full 4d forward FFT of `prop`.
+  // We deliberately do NOT cache the result — we always start from
+  // the same `prop` so the FFT is recomputed identically each time.
+  // The point is to watch memory, not correctness.
+  // ------------------------------------------------------------------
+  const int Niter = 40;
+  const int Niter2 = 32;
+
+  // Print header for the memory table
+  std::cout << GridLogMessage
+              << "\n"
+              << std::setw(6)  << "iter"
+              << "  CPU: " << std::setw(10) << "RSS[kB]"
+              << "  (  delta  )"
+              << "  GPU: " << std::setw(7)  << "used[MB]"
+              << "  (delta)"
+              << "\n";
+ 
+  MemSnapshot snap_prev = snap_after_plan;
+
+  for (int i = 0; i < Niter; i++) {
+    std::vector<LatticePropagatorD> G;
+
+    for (int j = 0; j < Niter2; j++) {
+      LatticePropagatorD prop_fft(&GRID);
+
+      // Full 4d forward FFT using the pre-built plans
+      plannedFFT.FFT_all_dim(prop_fft, prop, FFT::forward);
+
+      G.push_back(prop_fft);
+    }
+ 
+    // cudaMemGetInfo reflects the state *after* any pooled frees have
+    // been committed, so this is accurate without an explicit sync —
+    // FFT_dim_execute already calls accelerator_barrier() internally.
+    MemSnapshot snap_now = takeSnapshot();
+    printRow(i, snap_now, snap_prev);
+    MemoryManager::Print();
+    snap_prev = snap_now;
+  }
+
+  // ------------------------------------------------------------------
+  // Summary
+  // ------------------------------------------------------------------
+  MemSnapshot snap_final = takeSnapshot();
+ 
+  long cpu_growth = snap_final.cpu_rss_kb  - snap_after_plan.cpu_rss_kb;
+  long gpu_growth = snap_final.gpu_used_mb - snap_after_plan.gpu_used_mb;
+ 
+  std::cout << GridLogMessage
+            << "\n==== Memory summary (baseline = after plan construction) ====\n"
+            << "  CPU RSS growth over " << Niter << " FFTs : "
+            << cpu_growth << " kB"
+            << (cpu_growth == 0 ? "  OK" : "  *** GROWING ***") << "\n"
+            << "  GPU used growth over " << Niter << " FFTs : "
+            << gpu_growth << " MB"
+            << (gpu_growth == 0 ? "  OK" : "  *** GROWING ***") << "\n"
+            << "  Note: first-call watermark from pool fill is expected and benign.\n"
+            << "        A leak shows as continuous growth beyond iter ~2-3.\n";
+ 
+  Grid_finalize();
+  return 0;
+}

tests/core/Test_planned_fft.cc

@@ -0,0 +1,321 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/core/Test_planned_fft.cc
+
+    Copyright (C) 2015
+
+Author: Azusa Yamaguchi <ayamaguc@staffmail.ed.ac.uk>
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#include <Grid/Grid.h>
+
+using namespace Grid;
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  int threads = GridThread::GetThreads();
+  std::cout<<GridLogMessage << "Grid is setup to use "<<threads<<" threads"<<std::endl;
+
+  Coordinate latt_size   = GridDefaultLatt();
+  Coordinate simd_layout = GridDefaultSimd(Nd,vComplexD::Nsimd());
+  Coordinate mpi_layout  = GridDefaultMpi();
+
+  int vol = 1;
+  for(int d=0;d<latt_size.size();d++) vol *= latt_size[d];
+
+  GridCartesian         GRID(latt_size,simd_layout,mpi_layout);
+  GridRedBlackCartesian RBGRID(&GRID);
+
+  LatticeComplexD     one(&GRID);
+  LatticeComplexD      zz(&GRID);
+  LatticeComplexD       C(&GRID);
+  LatticeComplexD  Ctilde(&GRID);
+  LatticeComplexD  Cref  (&GRID);
+  LatticeComplexD  Csav  (&GRID);
+  LatticeComplexD    coor(&GRID);
+
+  LatticeSpinMatrixD    S(&GRID);
+  LatticeSpinMatrixD    Stilde(&GRID);
+
+  Coordinate p({1,3,2,3});
+
+  one = ComplexD(1.0,0.0);
+  zz  = ComplexD(0.0,0.0);
+  ComplexD ci(0.0,1.0);
+
+  std::cout<<"*************************************************"<<std::endl;
+  std::cout<<"Testing Fourier form of known plane wave         "<<std::endl;
+  std::cout<<"*************************************************"<<std::endl;
+  C=Zero();
+  for(int mu=0;mu<4;mu++){
+    RealD TwoPiL =  M_PI * 2.0/ latt_size[mu];
+    LatticeCoordinate(coor,mu);
+    C = C + (TwoPiL * p[mu]) * coor;
+  }
+  C = exp(C*ci);
+  Csav = C;
+  S=Zero();
+  S = S+C;
+
+  // PlannedFFT is templated on the lattice element type (vector_object), not the Lattice<> itself.
+  PlannedFFT<LatticeComplexD::vector_object>    theFFT(&GRID);
+  PlannedFFT<LatticeSpinMatrixD::vector_object> theFFT_spin(&GRID);
+
+  Ctilde=C;
+  std::cout<<" Benchmarking PlannedFFT of LatticeComplex  "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,0,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,1,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,2,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,3,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+
+  TComplexD cVol;
+  cVol()()() = vol;
+
+  Cref=Zero();
+  pokeSite(cVol,Cref,p);
+
+  Cref=Cref-Ctilde;
+  std::cout << "diff scalar "<<norm2(Cref) << std::endl;
+
+  C=Csav;
+  theFFT.FFT_all_dim(Ctilde,C,FFTbase::forward);
+  theFFT.FFT_all_dim(Cref,Ctilde,FFTbase::backward);
+
+  std::cout << norm2(C) << " " << norm2(Ctilde) << " " << norm2(Cref)<< " vol " << vol<< std::endl;
+
+  Cref= Cref - C;
+  std::cout << " invertible check " << norm2(Cref)<<std::endl;
+
+  Stilde=S;
+  std::cout<<" Benchmarking PlannedFFT of LatticeSpinMatrix  "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,0,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,1,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,2,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,3,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+
+  SpinMatrixD Sp;
+  Sp = Zero(); Sp = Sp+cVol;
+
+  S=Zero();
+  pokeSite(Sp,S,p);
+
+  S= S-Stilde;
+  std::cout << "diff FT[SpinMat] "<<norm2(S) << std::endl;
+
+  std::vector<int> seeds({1,2,3,4});
+  GridSerialRNG          sRNG;  sRNG.SeedFixedIntegers(seeds);
+  GridParallelRNG          pRNG(&GRID);
+  pRNG.SeedFixedIntegers(seeds);
+
+  LatticeGaugeFieldD Umu(&GRID);
+  SU<Nc>::ColdConfiguration(pRNG,Umu);
+
+  ////////////////////////////////////////////////////
+  // Wilson test
+  ////////////////////////////////////////////////////
+  {
+    LatticeFermionD    src(&GRID); gaussian(pRNG,src);
+    LatticeFermionD    tmp(&GRID);
+    LatticeFermionD    ref(&GRID);
+
+    RealD mass=0.01;
+    WilsonFermionD Dw(Umu,GRID,RBGRID,mass);
+
+    Dw.M(src,tmp);
+
+    std::cout << "Dw src = " <<norm2(src)<<std::endl;
+    std::cout << "Dw tmp = " <<norm2(tmp)<<std::endl;
+
+    Dw.FreePropagator(tmp,ref,mass);
+
+    std::cout << "Dw ref = " <<norm2(ref)<<std::endl;
+
+    ref = ref - src;
+    std::cout << "Dw ref-src = " <<norm2(ref)<<std::endl;
+  }
+
+  ////////////////////////////////////////////////////
+  // Dwf matrix — verify Fourier representation using PlannedFFT<LatticeFermionD>
+  ////////////////////////////////////////////////////
+  {
+    std::cout<<"****************************************"<<std::endl;
+    std::cout<<"Testing Fourier representation of Ddwf"<<std::endl;
+    std::cout<<"****************************************"<<std::endl;
+
+    const int Ls=16;
+    const int sdir=0;
+    RealD mass=0.01;
+    RealD M5  =1.0;
+    Gamma G5(Gamma::Algebra::Gamma5);
+
+    GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,&GRID);
+    GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,&GRID);
+
+    DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,GRID,RBGRID,mass,M5);
+
+    GridParallelRNG          RNG5(FGrid);      RNG5.SeedFixedIntegers(seeds);
+    LatticeFermionD    src5(FGrid); gaussian(RNG5,src5);
+    LatticeFermionD    src5_p(FGrid);
+    LatticeFermionD    result5(FGrid);
+    LatticeFermionD    ref5(FGrid);
+    LatticeFermionD    tmp5(FGrid);
+
+    Ddwf.M(src5,tmp5);
+    ref5 = tmp5;
+
+    PlannedFFT<LatticeFermionD::vector_object> theFFT5(FGrid);
+
+    theFFT5.FFT_dim(result5,tmp5,1,FFTbase::forward); tmp5 = result5;
+    std::cout<<"Fourier xformed Ddwf 1 "<<norm2(result5)<<std::endl;
+    theFFT5.FFT_dim(result5,tmp5,2,FFTbase::forward); tmp5 = result5;
+    std::cout<<"Fourier xformed Ddwf 2 "<<norm2(result5)<<std::endl;
+    theFFT5.FFT_dim(result5,tmp5,3,FFTbase::forward); tmp5 = result5;
+    std::cout<<"Fourier xformed Ddwf 3 "<<norm2(result5)<<std::endl;
+    theFFT5.FFT_dim(result5,tmp5,4,FFTbase::forward);
+    std::cout<<"Fourier xformed Ddwf 4 "<<norm2(result5)<<std::endl;
+    result5 = result5*ComplexD(::sqrt(1.0/vol),0.0);
+
+    std::cout<<"Fourier xformed Ddwf "<<norm2(result5)<<std::endl;
+
+    tmp5 = src5;
+    theFFT5.FFT_dim(src5_p,tmp5,1,FFTbase::forward); tmp5 = src5_p;
+    theFFT5.FFT_dim(src5_p,tmp5,2,FFTbase::forward); tmp5 = src5_p;
+    theFFT5.FFT_dim(src5_p,tmp5,3,FFTbase::forward); tmp5 = src5_p;
+    theFFT5.FFT_dim(src5_p,tmp5,4,FFTbase::forward); src5_p = src5_p*ComplexD(::sqrt(1.0/vol),0.0);
+
+    std::cout<<"Fourier xformed src5"<< norm2(src5)<<" -> "<<norm2(src5_p)<<std::endl;
+
+    Gamma::Algebra Gmu [] = {
+      Gamma::Algebra::GammaX,
+      Gamma::Algebra::GammaY,
+      Gamma::Algebra::GammaZ,
+      Gamma::Algebra::GammaT,
+      Gamma::Algebra::Gamma5
+    };
+    LatticeFermionD    Kinetic(FGrid); Kinetic = Zero();
+    LatticeComplexD    kmu(FGrid);
+    LatticeInteger     scoor(FGrid);
+    LatticeComplexD    sk (FGrid); sk = Zero();
+    LatticeComplexD    sk2(FGrid); sk2= Zero();
+    LatticeComplexD    W(FGrid); W= Zero();
+    LatticeComplexD    one5(FGrid); one5 =ComplexD(1.0,0.0);
+
+    for(int mu=0;mu<Nd;mu++) {
+      LatticeCoordinate(kmu,mu+1);
+      RealD TwoPiL =  M_PI * 2.0/ latt_size[mu];
+      kmu = TwoPiL * kmu;
+      sk2 = sk2 + 2.0*sin(kmu*0.5)*sin(kmu*0.5);
+      sk  = sk  +     sin(kmu)    *sin(kmu);
+      Kinetic = Kinetic + sin(kmu)*ci*(Gamma(Gmu[mu])*src5_p);
+    }
+    std::cout << " src5    "<<norm2(src5_p)<<std::endl;
+    std::cout << " Kinetic "<<norm2(Kinetic)<<std::endl;
+
+    W = one5 - M5 + sk2;
+    std::cout << " W "<<norm2(W)<<std::endl;
+    Kinetic = Kinetic + W * src5_p;
+    std::cout << " Kinetic "<<norm2(Kinetic)<<std::endl;
+
+    LatticeCoordinate(scoor,sdir);
+
+    tmp5 = Cshift(src5_p,sdir,+1);
+    tmp5 = (tmp5 - G5*tmp5)*0.5;
+    tmp5 = where(scoor==Integer(Ls-1),mass*tmp5,-tmp5);
+    Kinetic = Kinetic + tmp5;
+
+    tmp5 = Cshift(src5_p,sdir,-1);
+    tmp5 = (tmp5 + G5*tmp5)*0.5;
+    tmp5 = where(scoor==Integer(0),mass*tmp5,-tmp5);
+    Kinetic = Kinetic + tmp5;
+
+    std::cout<<"Momentum space Ddwf  "<< norm2(Kinetic)<<std::endl;
+    std::cout<<"Stencil Ddwf         "<< norm2(result5)<<std::endl;
+
+    result5 = result5 - Kinetic;
+    std::cout<<"diff "<< norm2(result5)<<std::endl;
+    GRID_ASSERT(norm2(result5)<1.0e-4);
+  }
+
+  ////////////////////////////////////////////////////
+  // Dwf prop
+  ////////////////////////////////////////////////////
+  {
+    std::cout<<"****************************************"<<std::endl;
+    std::cout << "Testing Ddwf Ht Mom space 4d propagator \n";
+    std::cout<<"****************************************"<<std::endl;
+
+    LatticeFermionD    src(&GRID); gaussian(pRNG,src);
+    LatticeFermionD    tmp(&GRID);
+    LatticeFermionD    ref(&GRID);
+    LatticeFermionD    diff(&GRID);
+
+    Coordinate point(4,0);
+    src=Zero();
+    SpinColourVectorD ferm; gaussian(sRNG,ferm);
+    pokeSite(ferm,src,point);
+
+    const int Ls=32;
+    GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,&GRID);
+    GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,&GRID);
+
+    RealD mass=0.01;
+    RealD M5  =0.8;
+    DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,GRID,RBGRID,mass,M5);
+
+    std::cout << " Solving by FFT and Feynman rules" <<std::endl;
+    bool fiveD = false;
+    Ddwf.FreePropagator(src,ref,mass,fiveD);
+
+    Gamma G5(Gamma::Algebra::Gamma5);
+
+    LatticeFermionD    src5(FGrid); src5=Zero();
+    LatticeFermionD    tmp5(FGrid);
+    LatticeFermionD    result5(FGrid); result5=Zero();
+    LatticeFermionD    result4(&GRID);
+    const int sdir=0;
+
+    tmp =   (src + G5*src)*0.5;      InsertSlice(tmp,src5,   0,sdir);
+    tmp =   (src - G5*src)*0.5;      InsertSlice(tmp,src5,Ls-1,sdir);
+
+    std::cout << " Solving by Conjugate Gradient (CGNE)" <<std::endl;
+    Ddwf.Mdag(src5,tmp5);
+    src5=tmp5;
+    MdagMLinearOperator<DomainWallFermionD,LatticeFermionD> HermOp(Ddwf);
+    ConjugateGradient<LatticeFermionD> CG(1.0e-8,10000);
+    CG(HermOp,src5,result5);
+
+    ExtractSlice(tmp,result5,0   ,sdir);   result4 =         (tmp-G5*tmp)*0.5;
+    ExtractSlice(tmp,result5,Ls-1,sdir);   result4 = result4+(tmp+G5*tmp)*0.5;
+
+    std::cout << " Taking difference" <<std::endl;
+    std::cout << "Ddwf result4 "<<norm2(result4)<<std::endl;
+    std::cout << "Ddwf ref     "<<norm2(ref)<<std::endl;
+
+    diff = ref - result4;
+    std::cout << "result - ref     "<<norm2(diff)<<std::endl;
+    GRID_ASSERT(norm2(diff)<1.0e-4);
+  }
+
+  Grid_finalize();
+}

tests/debug/Test_batched_blas.cc

@@ -0,0 +1,256 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/debug/Test_batched_blas.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+/*  END LEGAL */
+
+//
+// Unit tests for the blas-layer dense primitives:
+//
+//  T1 : GridBLASInverse::inverseBatched (ComplexF and ComplexD) --
+//       || A A^{-1} - I ||_max over a random well-conditioned batch.
+//       On a CPU build this exercises the Eigen reference (the semantic
+//       oracle); the SAME binary exercises rocSOLVER/cuBLAS/oneMKL on a
+//       device build.
+//  T2 : explicit-leading-dimension gemmBatched == SOFTWARE SPLIT-K in
+//       miniature.  Y = op(A,T).X computed (a) in one compact batch=1 call
+//       and (b) as NK K-chunks by POINTER OFFSET with lda = full K, partials
+//       reduced on the host.  (a)==(b) validates the strided overload AND
+//       the production dense-slab apply pattern (arXiv:2409.03904 fig 11).
+//
+// Hard asserts throughout: regression gate for Grid/algorithms/blas.
+//
+#include <Grid/Grid.h>
+#include <Grid/algorithms/blas/BatchedInverse.h>
+
+#include <random>
+
+using namespace std;
+using namespace Grid;
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  GridBLAS        blas;
+  GridBLASInverse inverse;
+
+  std::mt19937 rng(12345);
+  std::uniform_real_distribution<double> dist(-1.0,1.0);
+
+  ////////////////////////////////////////////////////////////////
+  // T1a : batched inversion, ComplexF
+  ////////////////////////////////////////////////////////////////
+  {
+    const int64_t N = 64;
+    const int  batch = 4;
+    const uint64_t elems = (uint64_t)batch*N*N;
+
+    // Random diagonally-dominant batch: A = N*I + R, |R_ij| <= 1
+    std::vector<ComplexF> Ahost(elems);
+    for(uint64_t i=0;i<elems;i++) Ahost[i] = ComplexF(dist(rng),dist(rng));
+    for(int b=0;b<batch;b++)
+      for(int64_t d=0;d<N;d++)
+	Ahost[(uint64_t)b*N*N + d*N + d] += ComplexF((RealF)N,0.0);
+
+    deviceVector<ComplexF> Adev(elems);   // gets inverted in place
+    deviceVector<ComplexF> Aorig(elems);  // untouched copy for the residual
+    deviceVector<ComplexF> Cdev(elems);
+    acceleratorCopyToDevice(&Ahost[0],&Adev[0], elems*sizeof(ComplexF));
+    acceleratorCopyToDevice(&Ahost[0],&Aorig[0],elems*sizeof(ComplexF));
+
+    deviceVector<ComplexF*> Ap(batch);
+    deviceVector<ComplexF*> Op(batch);
+    deviceVector<ComplexF*> Cp(batch);
+    std::vector<ComplexF*> ptr_h(batch);
+    for(int b=0;b<batch;b++) ptr_h[b] = &Adev[(uint64_t)b*N*N];
+    acceleratorCopyToDevice(&ptr_h[0],&Ap[0],batch*sizeof(ComplexF*));
+    for(int b=0;b<batch;b++) ptr_h[b] = &Aorig[(uint64_t)b*N*N];
+    acceleratorCopyToDevice(&ptr_h[0],&Op[0],batch*sizeof(ComplexF*));
+    for(int b=0;b<batch;b++) ptr_h[b] = &Cdev[(uint64_t)b*N*N];
+    acceleratorCopyToDevice(&ptr_h[0],&Cp[0],batch*sizeof(ComplexF*));
+
+    inverse.inverseBatched(N,Ap);                      // A <- A^{-1}
+
+    ComplexF one (1.0,0.0);
+    ComplexF zero(0.0,0.0);
+    blas.gemmBatched(GridBLAS_OP_N,GridBLAS_OP_N,
+		     (int)N,(int)N,(int)N,
+		     one, Op, Ap, zero, Cp);           // C = A A^{-1}
+    blas.synchronise();
+
+    std::vector<ComplexF> Chost(elems);
+    acceleratorCopyFromDevice(&Cdev[0],&Chost[0],elems*sizeof(ComplexF));
+
+    double maxdev = 0.0;
+    for(int b=0;b<batch;b++)
+      for(int64_t j=0;j<N;j++)
+	for(int64_t i=0;i<N;i++){
+	  ComplexF expect = (i==j) ? ComplexF(1.0,0.0) : ComplexF(0.0,0.0);
+	  ComplexF got = Chost[(uint64_t)b*N*N + j*N + i];
+	  maxdev = std::max(maxdev,(double)abs(got-expect));
+	}
+    std::cout << GridLogMessage << "T1a inverseBatched ComplexF  ||A Ainv - I||_max = "
+	      << maxdev << ( maxdev < 1.0e-4 ? "   PASS" : "   FAIL" ) << std::endl;
+    GRID_ASSERT(maxdev < 1.0e-4);
+  }
+
+  ////////////////////////////////////////////////////////////////
+  // T1b : batched inversion, ComplexD
+  ////////////////////////////////////////////////////////////////
+  {
+    const int64_t N = 48;
+    const int  batch = 3;
+    const uint64_t elems = (uint64_t)batch*N*N;
+
+    std::vector<ComplexD> Ahost(elems);
+    for(uint64_t i=0;i<elems;i++) Ahost[i] = ComplexD(dist(rng),dist(rng));
+    for(int b=0;b<batch;b++)
+      for(int64_t d=0;d<N;d++)
+	Ahost[(uint64_t)b*N*N + d*N + d] += ComplexD((RealD)N,0.0);
+
+    deviceVector<ComplexD> Adev(elems);
+    deviceVector<ComplexD> Aorig(elems);
+    deviceVector<ComplexD> Cdev(elems);
+    acceleratorCopyToDevice(&Ahost[0],&Adev[0], elems*sizeof(ComplexD));
+    acceleratorCopyToDevice(&Ahost[0],&Aorig[0],elems*sizeof(ComplexD));
+
+    deviceVector<ComplexD*> Ap(batch);
+    deviceVector<ComplexD*> Op(batch);
+    deviceVector<ComplexD*> Cp(batch);
+    std::vector<ComplexD*> ptr_h(batch);
+    for(int b=0;b<batch;b++) ptr_h[b] = &Adev[(uint64_t)b*N*N];
+    acceleratorCopyToDevice(&ptr_h[0],&Ap[0],batch*sizeof(ComplexD*));
+    for(int b=0;b<batch;b++) ptr_h[b] = &Aorig[(uint64_t)b*N*N];
+    acceleratorCopyToDevice(&ptr_h[0],&Op[0],batch*sizeof(ComplexD*));
+    for(int b=0;b<batch;b++) ptr_h[b] = &Cdev[(uint64_t)b*N*N];
+    acceleratorCopyToDevice(&ptr_h[0],&Cp[0],batch*sizeof(ComplexD*));
+
+    inverse.inverseBatched(N,Ap);
+
+    ComplexD one (1.0,0.0);
+    ComplexD zero(0.0,0.0);
+    blas.gemmBatched(GridBLAS_OP_N,GridBLAS_OP_N,
+		     (int)N,(int)N,(int)N,
+		     one, Op, Ap, zero, Cp);
+    blas.synchronise();
+
+    std::vector<ComplexD> Chost(elems);
+    acceleratorCopyFromDevice(&Cdev[0],&Chost[0],elems*sizeof(ComplexD));
+
+    double maxdev = 0.0;
+    for(int b=0;b<batch;b++)
+      for(int64_t j=0;j<N;j++)
+	for(int64_t i=0;i<N;i++){
+	  ComplexD expect = (i==j) ? ComplexD(1.0,0.0) : ComplexD(0.0,0.0);
+	  ComplexD got = Chost[(uint64_t)b*N*N + j*N + i];
+	  maxdev = std::max(maxdev,(double)abs(got-expect));
+	}
+    std::cout << GridLogMessage << "T1b inverseBatched ComplexD  ||A Ainv - I||_max = "
+	      << maxdev << ( maxdev < 1.0e-10 ? "   PASS" : "   FAIL" ) << std::endl;
+    GRID_ASSERT(maxdev < 1.0e-10);
+  }
+
+  ////////////////////////////////////////////////////////////////
+  // T2 : explicit-ld gemmBatched == software split-K miniature.
+  // Slab layout as in the dense coarse-coarse apply: A is K x nrows
+  // column major (lda=K); Y = op(A,T).X with X K x nrhs (ldb=K).
+  ////////////////////////////////////////////////////////////////
+  {
+    const int nrows = 8;
+    const int nrhs  = 4;
+    const int K     = 256;
+    const int NK    = 8;        // split-K chunks
+    const int Kc    = K/NK;
+
+    std::vector<ComplexF> Ahost((uint64_t)K*nrows);
+    std::vector<ComplexF> Xhost((uint64_t)K*nrhs);
+    for(auto &z : Ahost) z = ComplexF(dist(rng),dist(rng));
+    for(auto &z : Xhost) z = ComplexF(dist(rng),dist(rng));
+
+    deviceVector<ComplexF> Adev(Ahost.size());
+    deviceVector<ComplexF> Xdev(Xhost.size());
+    deviceVector<ComplexF> Yref((uint64_t)nrows*nrhs);
+    deviceVector<ComplexF> Ypart((uint64_t)NK*nrows*nrhs);
+    acceleratorCopyToDevice(&Ahost[0],&Adev[0],Ahost.size()*sizeof(ComplexF));
+    acceleratorCopyToDevice(&Xhost[0],&Xdev[0],Xhost.size()*sizeof(ComplexF));
+
+    ComplexF one (1.0,0.0);
+    ComplexF zero(0.0,0.0);
+
+    // (a) reference: one compact batch=1 call (compact lda == K for OP_T)
+    {
+      deviceVector<ComplexF*> Ap(1), Xp(1), Yp(1);
+      std::vector<ComplexF*> h(1);
+      h[0]=&Adev[0]; acceleratorCopyToDevice(&h[0],&Ap[0],sizeof(ComplexF*));
+      h[0]=&Xdev[0]; acceleratorCopyToDevice(&h[0],&Xp[0],sizeof(ComplexF*));
+      h[0]=&Yref[0]; acceleratorCopyToDevice(&h[0],&Yp[0],sizeof(ComplexF*));
+      blas.gemmBatched(GridBLAS_OP_T,GridBLAS_OP_N,
+		       nrows,nrhs,K,
+		       one, Ap, Xp, zero, Yp);
+      blas.synchronise();
+    }
+
+    // (b) split-K: NK chunk-pointers into the SAME allocations, lda/ldb = K
+    {
+      deviceVector<ComplexF*> Ap(NK), Xp(NK), Yp(NK);
+      std::vector<ComplexF*> h(NK);
+      for(int j=0;j<NK;j++) h[j] = &Adev[(uint64_t)j*Kc];        // K-offset slice
+      acceleratorCopyToDevice(&h[0],&Ap[0],NK*sizeof(ComplexF*));
+      for(int j=0;j<NK;j++) h[j] = &Xdev[(uint64_t)j*Kc];
+      acceleratorCopyToDevice(&h[0],&Xp[0],NK*sizeof(ComplexF*));
+      for(int j=0;j<NK;j++) h[j] = &Ypart[(uint64_t)j*nrows*nrhs];
+      acceleratorCopyToDevice(&h[0],&Yp[0],NK*sizeof(ComplexF*));
+      blas.gemmBatched(GridBLAS_OP_T,GridBLAS_OP_N,
+		       nrows,nrhs,Kc,
+		       one, Ap, /*lda*/ K,
+		            Xp, /*ldb*/ K,
+		       zero,Yp, /*ldc*/ nrows);
+      blas.synchronise();
+    }
+
+    std::vector<ComplexF> Yref_h((uint64_t)nrows*nrhs);
+    std::vector<ComplexF> Ypart_h((uint64_t)NK*nrows*nrhs);
+    acceleratorCopyFromDevice(&Yref[0], &Yref_h[0], Yref_h.size()*sizeof(ComplexF));
+    acceleratorCopyFromDevice(&Ypart[0],&Ypart_h[0],Ypart_h.size()*sizeof(ComplexF));
+
+    double maxdev = 0.0;
+    double maxval = 0.0;
+    for(int i=0;i<nrows*nrhs;i++){
+      ComplexF sum(0.0,0.0);
+      for(int j=0;j<NK;j++) sum = sum + Ypart_h[(uint64_t)j*nrows*nrhs + i];
+      maxdev = std::max(maxdev,(double)abs(sum-Yref_h[i]));
+      maxval = std::max(maxval,(double)abs(Yref_h[i]));
+    }
+    double rel = maxdev/maxval;
+    std::cout << GridLogMessage << "T2  split-K strided gemmBatched  max rel dev vs compact = "
+	      << rel << ( rel < 1.0e-4 ? "   PASS" : "   FAIL" ) << std::endl;
+    GRID_ASSERT(rel < 1.0e-4);
+  }
+
+  std::cout << GridLogMessage << "All batched-blas tests PASSED" << std::endl;
+
+  Grid_finalize();
+}

tests/debug/Test_hipfft_bug_fail.cc

@@ -0,0 +1,76 @@
+/*
+ * Isolating the hipfft HIPFFT_PARSE_ERROR on ROCm 7 / hipFFT 1.0.20.
+ *
+ * Tests three orderings with an empty rocFFT cache to find which GPU
+ * operation before plan creation triggers the failure:
+ *   A) hipMalloc only            — hypothesis: passes (no async GPU work)
+ *   B) hipMalloc + hipMemset     — hypothesis: fails  (async GPU work in flight)
+ *   C) hipMalloc + hipMemset     — hypothesis: passes (work completed before plan)
+ *      + hipDeviceSynchronize
+ *
+ * Compile:
+ *   hipcc -o Test_hipfft_bug_fail Test_hipfft_bug_fail.cc -lhipfft
+ *
+ * Run with empty cache:
+ *   rm -rf ~/.cache/
+ *   ./Test_hipfft_bug_fail
+ */
+
+#include <cstdio>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+static const char *res(hipfftResult rv) {
+  return rv == HIPFFT_SUCCESS ? "SUCCESS" : "PARSE_ERROR";
+}
+
+static hipfftResult makePlan(int G, int howmany) {
+  int n[] = {G};
+  hipfftHandle p;
+  size_t workSize = 0;
+  hipfftCreate(&p);
+  hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                       nullptr, 1, G, nullptr, 1, G,
+                                       HIPFFT_Z2Z, howmany, &workSize);
+  hipfftDestroy(p);
+  return rv;
+}
+
+int main(void) {
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, 0);
+  printf("Device: %s\n", prop.name);
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  for (int G : {4, 8, 16, 32}) {
+    int howmany = 512;
+    long nelems = (long)G * howmany;
+    hipfftDoubleComplex *buf = nullptr;
+    hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+
+    // Tests ordered so each runs before a prior success can populate the cache.
+
+    // B first: hipMalloc + hipMemset (async GPU work in flight)
+    // If this fails, A (no hipMemset) will pass, confirming hipMemset is the trigger.
+    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+    hipfftResult rvB = makePlan(G, howmany);
+    printf("G=%-4d  B) hipMalloc + hipMemset       : %s\n", G, res(rvB));
+
+    // C: hipMalloc + hipMemset + sync — does syncing before plan creation fix it?
+    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+    hipDeviceSynchronize();
+    hipfftResult rvC = makePlan(G, howmany);
+    printf("G=%-4d  C) hipMalloc + hipMemset + sync: %s\n", G, res(rvC));
+
+    // A last: hipMalloc only, no async GPU work — should always pass
+    hipfftResult rvA = makePlan(G, howmany);
+    printf("G=%-4d  A) hipMalloc only             : %s\n\n", G, res(rvA));
+
+    hipFree(buf);
+  }
+
+  return 0;
+}

tests/debug/Test_hipfft_bug_pass.cc

@@ -0,0 +1,61 @@
+/*
+ * Minimal program demonstrating the workaround for the hipfft ROCm 7 bug.
+ *
+ * Workaround: create the hipfft plan BEFORE any hipMalloc.  Plan creation
+ * for G < 32 then succeeds even with an empty rocFFT cache.
+ *
+ * Compile:
+ *   hipcc -o Test_hipfft_bug_pass Test_hipfft_bug_pass.cc -lhipfft
+ *
+ * Run:
+ *   rm -rf ~/.cache/rocfft
+ *   ./Test_hipfft_bug_pass
+ *
+ * Expected: all G values succeed.
+ * Compare with Test_hipfft_bug_fail.cc which uses the opposite ordering.
+ */
+
+#include <cstdio>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+int main(void) {
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, 0);
+  printf("Device: %s\n", prop.name);
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  for (int G : {8, 16, 32}) {
+    int howmany = 512;
+    int n[] = {G};
+    long nelems = (long)G * howmany;
+
+    // Plan created BEFORE hipMalloc — succeeds for all G
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftCreate(&p);
+    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                         nullptr, 1, G, nullptr, 1, G,
+                                         HIPFFT_Z2Z, howmany, &workSize);
+    printf("G=%-4d  plan-then-hipMalloc: %d (%s)\n",
+           G, (int)rv, rv == HIPFFT_SUCCESS ? "HIPFFT_SUCCESS" : "HIPFFT_PARSE_ERROR");
+
+    if (rv == HIPFFT_SUCCESS) {
+      hipfftDoubleComplex *buf = nullptr;
+      hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+      hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("G=%-4d  execFwd:             %d (%s)\n",
+             G, (int)rv, rv == HIPFFT_SUCCESS ? "HIPFFT_SUCCESS" : "FAILED");
+      hipFree(buf);
+    }
+
+    hipfftDestroy(p);
+  }
+
+  return 0;
+}

tests/debug/Test_hipfft_minimal.cc

@@ -0,0 +1,142 @@
+/*
+ * Minimal reproducer for hipfftMakePlanMany / hipfftPlanMany failures.
+ *
+ * Compile on Frontier (no Grid headers needed):
+ *   hipcc -o Test_hipfft_minimal Test_hipfft_minimal.cc -lhipfft
+ *
+ * Run:
+ *   ./Test_hipfft_minimal
+ */
+
+#include <cstdio>
+#include <cstdlib>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+static const char *hipfftResultString(hipfftResult r) {
+  switch (r) {
+  case HIPFFT_SUCCESS:                  return "HIPFFT_SUCCESS";
+  case HIPFFT_INVALID_PLAN:             return "HIPFFT_INVALID_PLAN";
+  case HIPFFT_ALLOC_FAILED:             return "HIPFFT_ALLOC_FAILED";
+  case HIPFFT_INVALID_TYPE:             return "HIPFFT_INVALID_TYPE";
+  case HIPFFT_INVALID_VALUE:            return "HIPFFT_INVALID_VALUE";
+  case HIPFFT_INTERNAL_ERROR:           return "HIPFFT_INTERNAL_ERROR";
+  case HIPFFT_EXEC_FAILED:              return "HIPFFT_EXEC_FAILED";
+  case HIPFFT_SETUP_FAILED:             return "HIPFFT_SETUP_FAILED";
+  case HIPFFT_INVALID_SIZE:             return "HIPFFT_INVALID_SIZE";
+  case HIPFFT_UNALIGNED_DATA:           return "HIPFFT_UNALIGNED_DATA";
+  case HIPFFT_INCOMPLETE_PARAMETER_LIST:return "HIPFFT_INCOMPLETE_PARAMETER_LIST";
+  case HIPFFT_INVALID_DEVICE:           return "HIPFFT_INVALID_DEVICE";
+  case HIPFFT_PARSE_ERROR:              return "HIPFFT_PARSE_ERROR";
+  case HIPFFT_NO_WORKSPACE:             return "HIPFFT_NO_WORKSPACE";
+  case HIPFFT_NOT_IMPLEMENTED:          return "HIPFFT_NOT_IMPLEMENTED";
+  case HIPFFT_NOT_SUPPORTED:            return "HIPFFT_NOT_SUPPORTED";
+  default:                              return "UNKNOWN";
+  }
+}
+
+// Plan creation + execution for (G, howmany).
+// Tests two orderings to isolate whether a prior hipMalloc poisons hipfft
+// plan creation for small G on ROCm 7:
+//   A) plan BEFORE hipMalloc  — hypothesis: succeeds
+//   B) hipMalloc BEFORE plan  — hypothesis: fails for G < 32
+static void tryPlanAndExec(int G, long howmany) {
+  int n[] = {G};
+  long nelems = (long)G * howmany;
+
+  printf("--- G=%-4d  howmany=%-10ld  total_elems=%-12ld ---\n",
+         G, howmany, nelems);
+
+  // --- A: create plan first, allocate buffer afterwards ---
+  {
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftCreate(&p);
+    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                         nullptr, 1, G, nullptr, 1, G,
+                                         HIPFFT_Z2Z, (int)howmany, &workSize);
+    printf("  plan-first  create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      hipfftDoubleComplex *buf = nullptr;
+      hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+      hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  plan-first  execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipFree(buf);
+    }
+    hipfftDestroy(p);
+  }
+
+  // --- B: hipMalloc first, create plan afterwards ---
+  {
+    hipfftDoubleComplex *buf = nullptr;
+    hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftCreate(&p);
+    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                         nullptr, 1, G, nullptr, 1, G,
+                                         HIPFFT_Z2Z, (int)howmany, &workSize);
+    printf("  malloc-first create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  malloc-first execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+    }
+    hipfftDestroy(p);
+    hipFree(buf);
+  }
+
+  printf("\n");
+}
+
+int main(void) {
+  // Print HIP device info
+  int device = 0;
+  hipGetDevice(&device);
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, device);
+  printf("Device %d: %s  warpSize=%d\n\n", device, prop.name, prop.warpSize);
+
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  // Original sweep with small howmany (these passed first time)
+  printf("=== Small howmany (original sweep) ===\n\n");
+  for (int G : {4, 8, 12, 16, 24, 32, 48, 64})
+    tryPlanAndExec(G, 512);
+
+  // Grid-realistic howmany values derived from actual lattice geometries.
+  // howmany = Ncomp * product(ldimensions[d] for d != dim)
+  // For LatticeComplexD: Ncomp=1.
+  printf("=== Grid-realistic parameters ===\n\n");
+
+  // --grid 16.16.16.16  4D FFT  (KNOWN TO FAIL in Grid)
+  // Each dim: G=16, Nperp=16^3=4096
+  tryPlanAndExec(16, 4096);
+
+  // --grid 32.32.32.32  4D FFT  (KNOWN TO SUCCEED in Grid)
+  // Each dim: G=32, Nperp=32^3=32768
+  tryPlanAndExec(32, 32768);
+
+  // --grid 32.32.32.32 Ls=8  5D DWF FFT  (KNOWN TO FAIL on dim 0 in Grid)
+  // dim 0: G=8,  Nperp=32^4=1048576
+  tryPlanAndExec(8, 1048576);
+  // dim 1-4: G=32, Nperp=8*32^3=262144
+  tryPlanAndExec(32, 262144);
+
+  // Extra intermediate cases to bracket the failure
+  tryPlanAndExec(16, 1024);
+  tryPlanAndExec(16, 2048);
+  tryPlanAndExec(16, 8192);
+  tryPlanAndExec(8,  4096);
+  tryPlanAndExec(8,  65536);
+  tryPlanAndExec(8,  262144);
+
+  return 0;
+}

tests/debug/Test_hipfft_repro.cc

@@ -0,0 +1,168 @@
+/*
+ * Reproducer for HIPFFT_PARSE_ERROR (error 12) from hipfftMakePlanMany on
+ * ROCm 7 / hipFFT 1.0.20 (Frontier, MI210 login and MI250X compute nodes).
+ *
+ * Observed failure: G < 32 returns HIPFFT_PARSE_ERROR from all three plan
+ * creation APIs (hipfftPlanMany, hipfftMakePlanMany, hipfftPlan1d) when a
+ * device buffer is allocated and zeroed with hipMalloc+hipMemset before the
+ * plan creation call.  G >= 32 succeeds.
+ *
+ * Contrast with Test_hipfft_minimal.cc (plan-first ordering) which passes
+ * for all G even with an empty rocFFT cache.
+ *
+ * Compile on Frontier (no Grid headers needed):
+ *   hipcc -o Test_hipfft_repro Test_hipfft_repro.cc -lhipfft
+ *
+ * Run with empty cache to reproduce the failure:
+ *   rm -rf ~/.cache/rocfft
+ *   ./Test_hipfft_repro
+ */
+
+#include <cstdio>
+#include <cstdlib>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+static const char *hipfftResultString(hipfftResult r) {
+  switch (r) {
+  case HIPFFT_SUCCESS:                  return "HIPFFT_SUCCESS";
+  case HIPFFT_INVALID_PLAN:             return "HIPFFT_INVALID_PLAN";
+  case HIPFFT_ALLOC_FAILED:             return "HIPFFT_ALLOC_FAILED";
+  case HIPFFT_INVALID_TYPE:             return "HIPFFT_INVALID_TYPE";
+  case HIPFFT_INVALID_VALUE:            return "HIPFFT_INVALID_VALUE";
+  case HIPFFT_INTERNAL_ERROR:           return "HIPFFT_INTERNAL_ERROR";
+  case HIPFFT_EXEC_FAILED:              return "HIPFFT_EXEC_FAILED";
+  case HIPFFT_SETUP_FAILED:             return "HIPFFT_SETUP_FAILED";
+  case HIPFFT_INVALID_SIZE:             return "HIPFFT_INVALID_SIZE";
+  case HIPFFT_UNALIGNED_DATA:           return "HIPFFT_UNALIGNED_DATA";
+  case HIPFFT_INCOMPLETE_PARAMETER_LIST:return "HIPFFT_INCOMPLETE_PARAMETER_LIST";
+  case HIPFFT_INVALID_DEVICE:           return "HIPFFT_INVALID_DEVICE";
+  case HIPFFT_PARSE_ERROR:              return "HIPFFT_PARSE_ERROR";
+  case HIPFFT_NO_WORKSPACE:             return "HIPFFT_NO_WORKSPACE";
+  case HIPFFT_NOT_IMPLEMENTED:          return "HIPFFT_NOT_IMPLEMENTED";
+  case HIPFFT_NOT_SUPPORTED:            return "HIPFFT_NOT_SUPPORTED";
+  default:                              return "UNKNOWN";
+  }
+}
+
+// Plan creation + execution for (G, howmany) using hipfftCreate+hipfftMakePlanMany.
+// This is the path Grid's FFT.h now uses.
+static void tryPlanAndExec(int G, long howmany) {
+  int n[] = {G};
+  long nelems = (long)G * howmany;
+
+  printf("--- G=%-4d  howmany=%-10ld  total_elems=%-12ld ---\n",
+         G, howmany, nelems);
+
+  // Allocate device buffer (hipfftDoubleComplex = 16 bytes each)
+  hipfftDoubleComplex *dbuf = nullptr;
+  hipError_t herr = hipMalloc(&dbuf, nelems * sizeof(hipfftDoubleComplex));
+  if (herr != hipSuccess) {
+    printf("  hipMalloc failed (%d) for %ld elems — skipping\n\n", (int)herr, nelems);
+    return;
+  }
+  hipMemset(dbuf, 0, nelems * sizeof(hipfftDoubleComplex));
+
+  // 1. hipfftPlanMany (one-step, nullptr embed) — current Grid path
+  {
+    hipfftHandle p;
+    hipfftResult rv = hipfftPlanMany(&p, 1, n,
+                                     nullptr, 1, G,
+                                     nullptr, 1, G,
+                                     HIPFFT_Z2Z, (int)howmany);
+    printf("  hipfftPlanMany   create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  hipfftPlanMany   execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipfftDestroy(p);
+    }
+  }
+
+  // 2. hipfftCreate + hipfftMakePlanMany (two-step) — also current Grid path
+  {
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftResult rc = hipfftCreate(&p);
+    if (rc == HIPFFT_SUCCESS) {
+      hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                           nullptr, 1, G,
+                                           nullptr, 1, G,
+                                           HIPFFT_Z2Z, (int)howmany, &workSize);
+      printf("  hipfftMakePlanMany      : %d (%s)  workSize=%zu\n",
+             (int)rv, hipfftResultString(rv), workSize);
+      if (rv == HIPFFT_SUCCESS) {
+        rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+        hipDeviceSynchronize();
+        printf("  hipfftMakePlanMany exec : %d (%s)\n", (int)rv, hipfftResultString(rv));
+      }
+      hipfftDestroy(p);
+    } else {
+      printf("  hipfftCreate            : %d (%s)\n", (int)rc, hipfftResultString(rc));
+    }
+  }
+
+  // 3. hipfftPlan1d (simplest API, batch = howmany)
+  {
+    hipfftHandle p;
+    hipfftResult rv = hipfftPlan1d(&p, G, HIPFFT_Z2Z, (int)howmany);
+    printf("  hipfftPlan1d     create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  hipfftPlan1d     execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipfftDestroy(p);
+    }
+  }
+
+  hipFree(dbuf);
+  printf("\n");
+}
+
+int main(void) {
+  // Print HIP device info
+  int device = 0;
+  hipGetDevice(&device);
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, device);
+  printf("Device %d: %s  warpSize=%d\n\n", device, prop.name, prop.warpSize);
+
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  // Original sweep with small howmany (these passed first time)
+  printf("=== Small howmany (original sweep) ===\n\n");
+  for (int G : {4, 8, 12, 16, 24, 32, 48, 64})
+    tryPlanAndExec(G, 512);
+
+  // Grid-realistic howmany values derived from actual lattice geometries.
+  // howmany = Ncomp * product(ldimensions[d] for d != dim)
+  // For LatticeComplexD: Ncomp=1.
+  printf("=== Grid-realistic parameters ===\n\n");
+
+  // --grid 16.16.16.16  4D FFT  (KNOWN TO FAIL in Grid)
+  // Each dim: G=16, Nperp=16^3=4096
+  tryPlanAndExec(16, 4096);
+
+  // --grid 32.32.32.32  4D FFT  (KNOWN TO SUCCEED in Grid)
+  // Each dim: G=32, Nperp=32^3=32768
+  tryPlanAndExec(32, 32768);
+
+  // --grid 32.32.32.32 Ls=8  5D DWF FFT  (KNOWN TO FAIL on dim 0 in Grid)
+  // dim 0: G=8,  Nperp=32^4=1048576
+  tryPlanAndExec(8, 1048576);
+  // dim 1-4: G=32, Nperp=8*32^3=262144
+  tryPlanAndExec(32, 262144);
+
+  // Extra intermediate cases to bracket the failure
+  tryPlanAndExec(16, 1024);
+  tryPlanAndExec(16, 2048);
+  tryPlanAndExec(16, 8192);
+  tryPlanAndExec(8,  4096);
+  tryPlanAndExec(8,  65536);
+  tryPlanAndExec(8,  262144);
+
+  return 0;
+}

tests/forces/Test_dwf_ratio_4dpf_force.cc

@@ -0,0 +1,279 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/forces/Test_dwf_ratio_4dpf_force.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+/*  END LEGAL */
+
+//
+// Validation of TwoFlavourRatio4DPseudoFermionAction (non-EO, LinearFunction
+// solver slots).  Three tests, run for BOTH wall conventions:
+//
+//  T1 HeatbathIdentityTest : refresh then S; PASS iff S == 0.5*|eta4|^2 to
+//     solver tolerance.  This adjudicates the 4D effective-operator
+//     composition identity [P M^-1 V Pdag][P V^-1 M Pdag] = 1 for the chosen
+//     (P,Pdag) wall pair.  NO PREDICTION is made about which convention
+//     passes -- that is what the test decides.
+//  T2 ForceTest (idiom from Test_double_ratio.cc) : midpoint-derivative
+//     check of deriv against S.  Should PASS for BOTH conventions (S and
+//     deriv use the same literal-adjoint pair by construction).
+//  T3 Trivial-ratio control (V == M) : T1 with NumOp = DenOp.  The solve
+//     cancels against the multiply, so S = 0.5|eta4|^2 requires only
+//     P Pdag = 1_4d.  Should PASS for BOTH conventions; isolates plumbing
+//     from the composition identity.
+//
+// Solvers here are plain CG on the normal equations (CGNR), tolerance 1e-12,
+// so every defect above ~1e-10 is structural, not solver noise.  Run small,
+// e.g.:  ./Test_dwf_ratio_4dpf_force --grid 8.8.8.8
+//
+#include <Grid/Grid.h>
+#include <Grid/qcd/action/pseudofermion/TwoFlavourRatio4DPseudoFermion.h>
+
+using namespace std;
+using namespace Grid;
+
+////////////////////////////////////////////////////////////////////
+// LinearFunction wrappers: direct M^-1 and M^-dag via CG on the
+// normal equations.  These stand in for the MG-GCR stack in this
+// test; the action class sees only LinearFunction.
+////////////////////////////////////////////////////////////////////
+template<class Matrix,class Field>
+class CGNRLinearFunction : public LinearFunction<Field> {   // out = M^-1 in
+public:
+  using LinearFunction<Field>::operator();
+  Matrix &_Mat; RealD tol; Integer maxit;
+  CGNRLinearFunction(Matrix &Mat,RealD _tol,Integer _maxit) : _Mat(Mat), tol(_tol), maxit(_maxit) {};
+  void operator()(const Field &in, Field &out) {
+    MdagMLinearOperator<Matrix,Field> MdagM(_Mat);
+    Field src(in.Grid());
+    _Mat.Mdag(in,src);                 // src = Mdag in
+    ConjugateGradient<Field> CG(tol,maxit);
+    out = Zero();
+    CG(MdagM,src,out);                 // out = (MdagM)^-1 Mdag in = M^-1 in
+  }
+};
+template<class Matrix,class Field>
+class CGNRDagLinearFunction : public LinearFunction<Field> { // out = M^-dag in
+public:
+  using LinearFunction<Field>::operator();
+  Matrix &_Mat; RealD tol; Integer maxit;
+  CGNRDagLinearFunction(Matrix &Mat,RealD _tol,Integer _maxit) : _Mat(Mat), tol(_tol), maxit(_maxit) {};
+  void operator()(const Field &in, Field &out) {
+    MdagMLinearOperator<Matrix,Field> MdagM(_Mat);
+    Field tmp(in.Grid());
+    tmp = Zero();
+    ConjugateGradient<Field> CG(tol,maxit);
+    CG(MdagM,in,tmp);                  // tmp = (MdagM)^-1 in
+    _Mat.M(tmp,out);                   // out = M (MdagM)^-1 in = M^-dag in
+  }
+};
+
+////////////////////////////////////////////////////////////////////
+// T1 / T3 : heatbath composition-identity test.
+// Twin-seeded RNG reproduces the eta4 drawn inside refresh.
+////////////////////////////////////////////////////////////////////
+template<class Impl>
+RealD HeatbathIdentityTest(TwoFlavourRatio4DPseudoFermionAction<Impl> &action,
+			   LatticeGaugeField &U,
+			   GridCartesian *UGrid,
+			   const std::string &tag)
+{
+  typedef typename Impl::FermionField FermionField;
+
+  std::vector<int> seeds({9,11,13,17});
+  GridSerialRNG   sRNG;                 sRNG.SeedFixedIntegers(seeds);
+  GridParallelRNG RNG4(UGrid);          RNG4.SeedFixedIntegers(seeds);
+  GridParallelRNG RNG4check(UGrid);     RNG4check.SeedFixedIntegers(seeds);
+
+  FermionField eta4check(UGrid);
+  gaussian(RNG4check,eta4check);        // identical to the draw inside refresh
+
+  action.refresh(U,sRNG,RNG4);
+  RealD S       = action.S(U);
+  RealD Sexpect = 0.5*norm2(eta4check);
+  RealD defect  = std::abs(S-Sexpect)/Sexpect;
+
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+  std::cout << GridLogMessage << " HeatbathIdentityTest ["<<tag<<"]" << std::endl;
+  std::cout << GridLogMessage << "   S              = " << S << std::endl;
+  std::cout << GridLogMessage << "   0.5|eta4|^2    = " << Sexpect << std::endl;
+  std::cout << GridLogMessage << "   relative defect = " << defect
+	    << ( defect < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+  return defect;
+}
+
+////////////////////////////////////////////////////////////////////
+// T2 : ForceTest idiom from Test_double_ratio.cc (midpoint derivative)
+////////////////////////////////////////////////////////////////////
+template<class Gimpl>
+void ForceTest(Action<LatticeGaugeField> &action,LatticeGaugeField & U,MomentumFilterBase<LatticeGaugeField> &Filter)
+{
+  GridBase *UGrid = U.Grid();
+
+  std::vector<int> seeds({1,2,3,5});
+  GridSerialRNG            sRNG;         sRNG.SeedFixedIntegers(seeds);
+  GridParallelRNG          RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds);
+
+  LatticeColourMatrix Pmu(UGrid);
+  LatticeGaugeField P(UGrid);
+  LatticeGaugeField UdSdU(UGrid);
+
+  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
+  std::cout << GridLogMessage << " Force test for "<<action.action_name()<<std::endl;
+  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
+
+  RealD eps=0.005;
+
+  Gimpl::generate_momenta(P,sRNG,RNG4);
+  Filter.applyFilter(P);
+
+  action.refresh(U,sRNG,RNG4);
+
+  RealD S1 = action.S(U);
+
+  Gimpl::update_field(P,U,eps);
+
+  action.deriv(U,UdSdU);
+  UdSdU = Ta(UdSdU);
+  Filter.applyFilter(UdSdU);
+
+  DumpSliceNorm("Force",UdSdU,Nd-1);
+
+  Gimpl::update_field(P,U,eps);
+
+  RealD S2 = action.S(U);
+
+  // Use the derivative
+  LatticeComplex dS(UGrid); dS = Zero();
+  for(int mu=0;mu<Nd;mu++){
+    auto UdSdUmu = PeekIndex<LorentzIndex>(UdSdU,mu);
+    Pmu= PeekIndex<LorentzIndex>(P,mu);
+    dS = dS - trace(Pmu*UdSdUmu)*eps*2.0*2.0;
+  }
+  ComplexD dSpred    = sum(dS);
+  RealD diff =  S2-S1-dSpred.real();
+
+  std::cout<< GridLogMessage << "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++"<<std::endl;
+  std::cout<< GridLogMessage << "S1 : "<< S1    <<std::endl;
+  std::cout<< GridLogMessage << "S2 : "<< S2    <<std::endl;
+  std::cout<< GridLogMessage << "dS : "<< S2-S1 <<std::endl;
+  std::cout<< GridLogMessage << "dSpred : "<< dSpred.real() <<std::endl;
+  std::cout<< GridLogMessage << "diff : "<< diff<<std::endl;
+  std::cout<< GridLogMessage << "diff/dS : "<< diff/(S2-S1)<<std::endl;
+  std::cout<< GridLogMessage << "*********************************************************"<<std::endl;
+  //  GRID_ASSERT(diff<1.0);
+  std::cout<< GridLogMessage << "Done" <<std::endl;
+  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  std::cout << std::setprecision(14);
+
+  const int Ls=8;
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  std::vector<int> seeds4({1,2,3,4});
+  GridParallelRNG RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField U(UGrid);
+  SU<Nc>::HotConfiguration(RNG4,U);
+
+  ////////////////////////////////////////////////////////////////
+  // Operators: Mobius, campaign-like b,c; heavyish masses so CGNR
+  // is fast and well-conditioned even on a hot configuration.
+  ////////////////////////////////////////////////////////////////
+  RealD mden = 0.2;
+  RealD mnum = 0.5;
+  RealD M5   = 1.8;
+  RealD b    = 1.5;
+  RealD c    = 0.5;
+
+  WilsonImplParams p;
+  p.boundary_phases[0] = 1.0;
+  p.boundary_phases[1] = 1.0;
+  p.boundary_phases[2] = 1.0;
+  p.boundary_phases[3] = -1.0;
+
+  MobiusFermionD DenOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mden,M5,b,c,p);
+  MobiusFermionD NumOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mnum,M5,b,c,p);
+
+  RealD tol   = 1.0e-12;
+  Integer maxit = 20000;
+
+  typedef WilsonImplD::FermionField FermionField;
+  CGNRLinearFunction<MobiusFermionD,FermionField>    MinvSolver   (DenOp,tol,maxit);
+  CGNRDagLinearFunction<MobiusFermionD,FermionField> MdagInvSolver(DenOp,tol,maxit);
+  CGNRLinearFunction<MobiusFermionD,FermionField>    VinvSolver   (NumOp,tol,maxit);
+
+  ////////////////////////////////////////////////////////////////
+  // Actions: both wall conventions, plus the V==M trivial control
+  ////////////////////////////////////////////////////////////////
+  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActSol(NumOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,VinvSolver,1);
+  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActSrc(NumOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,VinvSolver,0);
+  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActTrivSol(DenOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,MinvSolver,1);
+  TwoFlavourRatio4DPseudoFermionAction<WilsonImplD> ActTrivSrc(DenOp,DenOp,MinvSolver,MdagInvSolver,MinvSolver,MinvSolver,0);
+
+  ////////////////////////////////////////////////////////////////
+  // T3 controls first (must both pass; isolates plumbing)
+  ////////////////////////////////////////////////////////////////
+  RealD d3s = HeatbathIdentityTest(ActTrivSol,U,UGrid,"T3 trivial V==M, solution walls");
+  RealD d3q = HeatbathIdentityTest(ActTrivSrc,U,UGrid,"T3 trivial V==M, source walls");
+
+  ////////////////////////////////////////////////////////////////
+  // T1 : the composition-identity adjudication
+  ////////////////////////////////////////////////////////////////
+  RealD d1s = HeatbathIdentityTest(ActSol,U,UGrid,"T1 ratio, solution walls");
+  RealD d1q = HeatbathIdentityTest(ActSrc,U,UGrid,"T1 ratio, source walls");
+
+  ////////////////////////////////////////////////////////////////
+  // T2 : force consistency (expected PASS for both conventions)
+  ////////////////////////////////////////////////////////////////
+  MomentumFilterNone<LatticeGaugeField> FilterNone;
+  ForceTest<GimplTypesR>(ActSol,U,FilterNone);
+  ForceTest<GimplTypesR>(ActSrc,U,FilterNone);
+
+  ////////////////////////////////////////////////////////////////
+  // Summary
+  ////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+  std::cout << GridLogMessage << " SUMMARY (relative heatbath defects)" << std::endl;
+  std::cout << GridLogMessage << "   T3 trivial  solution walls : " << d3s << std::endl;
+  std::cout << GridLogMessage << "   T3 trivial  source   walls : " << d3q << std::endl;
+  std::cout << GridLogMessage << "   T1 ratio    solution walls : " << d1s << std::endl;
+  std::cout << GridLogMessage << "   T1 ratio    source   walls : " << d1q << std::endl;
+  std::cout << GridLogMessage << " T3 must pass for both; T1 selects the wall convention." << std::endl;
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+
+  GRID_ASSERT(d3s < 1.0e-8);
+  GRID_ASSERT(d3q < 1.0e-8);
+
+  Grid_finalize();
+}

tests/forces/Test_dwf_ratio_leftprec.cc

@@ -0,0 +1,341 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/forces/Test_dwf_ratio_leftprec.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+/*  END LEGAL */
+
+//
+// Correctness of TwoFlavourRatioLeftPrecPseudoFermionAction against the
+// decades-proven TwoFlavourRatioPseudoFermionAction.  Both classes compute
+// the SAME action S = phi^dag V (MdagM)^-1 Vdag phi through different solve
+// chains (normal-equations vs left-preconditioned F = Vdag M), so with
+// twin-seeded refreshes and 1e-12 solvers they must agree to solver
+// tolerance.  Tests:
+//
+//   E0a/E0b : heatbath identity, S == 0.5|eta|^2 after RNG refresh, for
+//             BOTH classes (E0a also validates the twin-eta capture).
+//   E1      : S_classic == S_leftprec       (relative, ~1e-8)
+//   E2      : deriv_classic == deriv_leftprec (pointwise field norm, ~1e-8)
+//   F1      : ForceTest (Test_double_ratio.cc idiom) on the LeftPrec class.
+//
+// All asserts are hard: this is the regression gate for the new class.
+// Run small, e.g.:  ./Test_dwf_ratio_leftprec --grid 8.8.8.8
+//
+#include <Grid/Grid.h>
+#include <Grid/qcd/action/pseudofermion/TwoFlavourRatioLeftPrec.h>
+
+using namespace std;
+using namespace Grid;
+
+////////////////////////////////////////////////////////////////////
+// Minimal LinearOperator for the composite F = Vdag M, exposing the
+// Hermitian normal operator FdagF for CG.  Stencil entries assert.
+////////////////////////////////////////////////////////////////////
+template<class Impl>
+class VdagMNormalOperator : public LinearOperatorBase<typename Impl::FermionField> {
+public:
+  typedef typename Impl::FermionField Field;
+  FermionOperator<Impl> &VOp;
+  FermionOperator<Impl> &MOp;
+  VdagMNormalOperator(FermionOperator<Impl> &V,FermionOperator<Impl> &M) : VOp(V), MOp(M) {};
+
+  void Fapply(const Field &in, Field &out) {          // out = Vdag M in
+    Field tmp(in.Grid());
+    MOp.M(in,tmp);
+    VOp.Mdag(tmp,out);
+  }
+  void FdagApply(const Field &in, Field &out) {       // out = Mdag V in
+    Field tmp(in.Grid());
+    VOp.M(in,tmp);
+    MOp.Mdag(tmp,out);
+  }
+  virtual void Op     (const Field &in, Field &out) { Fapply(in,out); }
+  virtual void AdjOp  (const Field &in, Field &out) { FdagApply(in,out); }
+  virtual void HermOp (const Field &in, Field &out) {
+    Field tmp(in.Grid());
+    Fapply(in,tmp);
+    FdagApply(tmp,out);
+  }
+  virtual void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2) {
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1 = real(dot);
+    n2 = norm2(out);
+  }
+  virtual void OpDiag (const Field &in, Field &out)                  { GRID_ASSERT(0); }
+  virtual void OpDir  (const Field &in, Field &out,int dir,int disp) { GRID_ASSERT(0); }
+  virtual void OpDirAll(const Field &in, std::vector<Field> &out)    { GRID_ASSERT(0); }
+};
+
+////////////////////////////////////////////////////////////////////
+// F-contract LinearFunctions for the test, both via CG on FdagF:
+//   forward :  F x = b     ==>  x = (FdagF)^-1 Fdag b
+//   adjoint :  Fdag z = b  ==>  z = F (FdagF)^-1 b
+////////////////////////////////////////////////////////////////////
+template<class Impl>
+class ForwardFSolve : public LinearFunction<typename Impl::FermionField> {
+public:
+  typedef typename Impl::FermionField Field;
+  using LinearFunction<Field>::operator();
+  VdagMNormalOperator<Impl> &FdagF; RealD tol; Integer maxit;
+  ForwardFSolve(VdagMNormalOperator<Impl> &Op,RealD _tol,Integer _maxit) : FdagF(Op), tol(_tol), maxit(_maxit) {};
+  void operator()(const Field &in, Field &out) {
+    Field src(in.Grid());
+    FdagF.FdagApply(in,src);
+    ConjugateGradient<Field> CG(tol,maxit);
+    out = Zero();
+    CG(FdagF,src,out);
+  }
+};
+template<class Impl>
+class AdjointFSolve : public LinearFunction<typename Impl::FermionField> {
+public:
+  typedef typename Impl::FermionField Field;
+  using LinearFunction<Field>::operator();
+  VdagMNormalOperator<Impl> &FdagF; RealD tol; Integer maxit;
+  AdjointFSolve(VdagMNormalOperator<Impl> &Op,RealD _tol,Integer _maxit) : FdagF(Op), tol(_tol), maxit(_maxit) {};
+  void operator()(const Field &in, Field &out) {
+    Field y(in.Grid());
+    y = Zero();
+    ConjugateGradient<Field> CG(tol,maxit);
+    CG(FdagF,in,y);
+    FdagF.Fapply(y,out);
+  }
+};
+template<class Matrix,class Field>
+class NormalEqSolve : public LinearFunction<Field> {   // out = (MdagM)^-1 in
+public:
+  using LinearFunction<Field>::operator();
+  Matrix &_Mat; RealD tol; Integer maxit;
+  NormalEqSolve(Matrix &Mat,RealD _tol,Integer _maxit) : _Mat(Mat), tol(_tol), maxit(_maxit) {};
+  void operator()(const Field &in, Field &out) {
+    MdagMLinearOperator<Matrix,Field> MdagM(_Mat);
+    ConjugateGradient<Field> CG(tol,maxit);
+    out = Zero();
+    CG(MdagM,in,out);
+  }
+};
+
+////////////////////////////////////////////////////////////////////
+// ForceTest idiom from Test_double_ratio.cc (midpoint derivative)
+////////////////////////////////////////////////////////////////////
+template<class Gimpl>
+void ForceTest(Action<LatticeGaugeField> &action,LatticeGaugeField & U,MomentumFilterBase<LatticeGaugeField> &Filter)
+{
+  GridBase *UGrid = U.Grid();
+
+  std::vector<int> seeds({1,2,3,5});
+  GridSerialRNG            sRNG;         sRNG.SeedFixedIntegers(seeds);
+  GridParallelRNG          RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds);
+
+  LatticeColourMatrix Pmu(UGrid);
+  LatticeGaugeField P(UGrid);
+  LatticeGaugeField UdSdU(UGrid);
+
+  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
+  std::cout << GridLogMessage << " Force test for "<<action.action_name()<<std::endl;
+  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
+
+  RealD eps=0.005;
+
+  Gimpl::generate_momenta(P,sRNG,RNG4);
+  Filter.applyFilter(P);
+
+  action.refresh(U,sRNG,RNG4);
+
+  RealD S1 = action.S(U);
+
+  Gimpl::update_field(P,U,eps);
+
+  action.deriv(U,UdSdU);
+  UdSdU = Ta(UdSdU);
+  Filter.applyFilter(UdSdU);
+
+  DumpSliceNorm("Force",UdSdU,Nd-1);
+
+  Gimpl::update_field(P,U,eps);
+
+  RealD S2 = action.S(U);
+
+  LatticeComplex dS(UGrid); dS = Zero();
+  for(int mu=0;mu<Nd;mu++){
+    auto UdSdUmu = PeekIndex<LorentzIndex>(UdSdU,mu);
+    Pmu= PeekIndex<LorentzIndex>(P,mu);
+    dS = dS - trace(Pmu*UdSdUmu)*eps*2.0*2.0;
+  }
+  ComplexD dSpred    = sum(dS);
+  RealD diff =  S2-S1-dSpred.real();
+
+  std::cout<< GridLogMessage << "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++"<<std::endl;
+  std::cout<< GridLogMessage << "S1 : "<< S1    <<std::endl;
+  std::cout<< GridLogMessage << "S2 : "<< S2    <<std::endl;
+  std::cout<< GridLogMessage << "dS : "<< S2-S1 <<std::endl;
+  std::cout<< GridLogMessage << "dSpred : "<< dSpred.real() <<std::endl;
+  std::cout<< GridLogMessage << "diff : "<< diff<<std::endl;
+  std::cout<< GridLogMessage << "diff/dS : "<< diff/(S2-S1)<<std::endl;
+  std::cout<< GridLogMessage << "*********************************************************"<<std::endl;
+  std::cout<< GridLogMessage << "Done" <<std::endl;
+  std::cout << GridLogMessage << "*********************************************************"<<std::endl;
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  std::cout << std::setprecision(14);
+
+  const int Ls=8;
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  std::vector<int> seeds4({1,2,3,4});
+  GridParallelRNG RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField U(UGrid);
+  SU<Nc>::HotConfiguration(RNG4,U);
+
+  ////////////////////////////////////////////////////////////////
+  // Operators: quotient pair (V = PV mass 1, M light-ish), Mobius,
+  // campaign b,c.  Heavyish M so CG is quick on a hot configuration.
+  ////////////////////////////////////////////////////////////////
+  RealD mden = 0.2;
+  RealD mnum = 1.0;
+  RealD M5   = 1.8;
+  RealD b    = 1.5;
+  RealD c    = 0.5;
+
+  WilsonImplParams p;
+  p.boundary_phases[0] = 1.0;
+  p.boundary_phases[1] = 1.0;
+  p.boundary_phases[2] = 1.0;
+  p.boundary_phases[3] = -1.0;
+
+  MobiusFermionD DenOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mden,M5,b,c,p);
+  MobiusFermionD NumOp(U,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mnum,M5,b,c,p);
+
+  RealD tol     = 1.0e-12;
+  Integer maxit = 30000;
+
+  typedef WilsonImplD::FermionField FermionField;
+
+  ////////////////////////////////////////////////////////////////
+  // Solvers.  Classic: CG as OperatorFunction on the MdagM linop the
+  // action supplies.  LeftPrec: F-contract solves via CG on FdagF.
+  ////////////////////////////////////////////////////////////////
+  ConjugateGradient<FermionField> CG(tol,maxit);
+
+  VdagMNormalOperator<WilsonImplD> FdagF(NumOp,DenOp);
+  ForwardFSolve<WilsonImplD>       Ffwd (FdagF,tol,maxit);
+  AdjointFSolve<WilsonImplD>       Fadj (FdagF,tol,maxit);
+  NormalEqSolve<MobiusFermionD,FermionField> VdagVinv(NumOp,tol,maxit);
+
+  TwoFlavourRatioPseudoFermionAction<WilsonImplD>         Classic (NumOp,DenOp,CG,CG);
+  TwoFlavourRatioLeftPrecPseudoFermionAction<WilsonImplD> LeftPrec(NumOp,DenOp,Ffwd,Fadj,Fadj,VdagVinv);
+
+  ////////////////////////////////////////////////////////////////
+  // Twin-seeded refreshes: identical eta into both classes.
+  ////////////////////////////////////////////////////////////////
+  std::vector<int> seedsR({9,11,13,17});
+  GridSerialRNG   sRNGa;         sRNGa.SeedFixedIntegers(seedsR);
+  GridSerialRNG   sRNGb;         sRNGb.SeedFixedIntegers(seedsR);
+  GridParallelRNG RNG5a(FGrid);  RNG5a.SeedFixedIntegers(seedsR);
+  GridParallelRNG RNG5b(FGrid);  RNG5b.SeedFixedIntegers(seedsR);
+  GridParallelRNG RNG5c(FGrid);  RNG5c.SeedFixedIntegers(seedsR);
+
+  FermionField etaTwin(FGrid);
+  gaussian(RNG5c,etaTwin);                    // identical to both refresh draws
+
+  Classic.refresh (U,sRNGa,RNG5a);
+  LeftPrec.refresh(U,sRNGb,RNG5b);
+
+  ////////////////////////////////////////////////////////////////
+  // E0 : heatbath identity for both classes
+  ////////////////////////////////////////////////////////////////
+  RealD Sexpect = 0.5*norm2(etaTwin);
+  RealD Sc = Classic.S(U);
+  RealD Sl = LeftPrec.S(U);
+
+  RealD e0a = std::abs(Sc-Sexpect)/Sexpect;
+  RealD e0b = std::abs(Sl-Sexpect)/Sexpect;
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+  std::cout << GridLogMessage << " E0 heatbath identity: 0.5|eta|^2 = " << Sexpect << std::endl;
+  std::cout << GridLogMessage << "   classic  S = " << Sc << "  rel defect " << e0a
+	    << ( e0a < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
+  std::cout << GridLogMessage << "   leftprec S = " << Sl << "  rel defect " << e0b
+	    << ( e0b < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
+
+  ////////////////////////////////////////////////////////////////
+  // E1 : action equivalence
+  ////////////////////////////////////////////////////////////////
+  RealD e1 = std::abs(Sc-Sl)/std::abs(Sc);
+  std::cout << GridLogMessage << " E1 action equivalence: rel diff = " << e1
+	    << ( e1 < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
+
+  ////////////////////////////////////////////////////////////////
+  // E2 : derivative equivalence (pointwise field comparison)
+  ////////////////////////////////////////////////////////////////
+  LatticeGaugeField dSdUc(UGrid);
+  LatticeGaugeField dSdUl(UGrid);
+  LatticeGaugeField dDiff(UGrid);
+
+  Classic.deriv (U,dSdUc);
+  LeftPrec.deriv(U,dSdUl);
+  dDiff = dSdUc - dSdUl;
+
+  RealD e2 = std::sqrt( norm2(dDiff) / norm2(dSdUc) );
+  std::cout << GridLogMessage << " E2 deriv equivalence: |diff|/|classic| = " << e2
+	    << ( e2 < 1.0e-8 ? "   PASS" : "   FAIL" ) << std::endl;
+  std::cout << GridLogMessage << "   |dSdU classic |^2 = " << norm2(dSdUc) << std::endl;
+  std::cout << GridLogMessage << "   |dSdU leftprec|^2 = " << norm2(dSdUl) << std::endl;
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+
+  ////////////////////////////////////////////////////////////////
+  // F1 : standalone force test on the LeftPrec class
+  ////////////////////////////////////////////////////////////////
+  MomentumFilterNone<LatticeGaugeField> FilterNone;
+  ForceTest<GimplTypesR>(LeftPrec,U,FilterNone);
+
+  ////////////////////////////////////////////////////////////////
+  // Summary + hard asserts (this is the regression gate)
+  ////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+  std::cout << GridLogMessage << " SUMMARY" << std::endl;
+  std::cout << GridLogMessage << "   E0a classic  heatbath defect : " << e0a << std::endl;
+  std::cout << GridLogMessage << "   E0b leftprec heatbath defect : " << e0b << std::endl;
+  std::cout << GridLogMessage << "   E1  action   equivalence     : " << e1  << std::endl;
+  std::cout << GridLogMessage << "   E2  deriv    equivalence     : " << e2  << std::endl;
+  std::cout << GridLogMessage << "=========================================================" << std::endl;
+
+  GRID_ASSERT(e0a < 1.0e-8);
+  GRID_ASSERT(e0b < 1.0e-8);
+  GRID_ASSERT(e1  < 1.0e-8);
+  GRID_ASSERT(e2  < 1.0e-8);
+
+  std::cout << GridLogMessage << "All equivalence tests PASSED" << std::endl;
+
+  Grid_finalize();
+}