3 level dense

calculation used by GeneralCoarsenedMatrixMultiRHS

CLAUDE.md

@@ -0,0 +1,156 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What This Is
+
+Grid is a data-parallel C++ library for lattice QCD. It provides SIMD-vectorised lattice containers, MPI-based domain decomposition, GPU acceleration (CUDA/HIP/SYCL), and a full suite of QCD algorithms including HMC.
+
+## Build
+
+Uses GNU Autotools. The bootstrap step only needs to run once (or after `configure.ac` changes).
+
+```bash
+./bootstrap.sh                    # downloads Eigen 3.4.0, generates configure
+mkdir build && cd build
+../configure [options]
+make -j$(nproc)
+make check                        # run root-level tests
+make install
+```
+
+Key configure options:
+
+| Option | Common values |
+|--------|---------------|
+| `--enable-simd=` | `AVX2`, `AVX512`, `KNL`, `A64FX`, `NEONv8`, `GPU` |
+| `--enable-comms=` | `mpi-auto`, `mpi3-auto`, `none` |
+| `--enable-accelerator=` | `cuda`, `hip`, `sycl` |
+| `--enable-shm=` | `shmopen`, `hugetlbfs`, `nvlink` |
+| `--enable-Nc=` | `3` (default), `2`, `4`, `5` |
+| `--with-gmp=`, `--with-mpfr=`, `--with-fftw=`, `--with-lime=` | paths to libs |
+| `--enable-hdf5`, `--enable-mkl`, `--enable-lapack` | optional features |
+
+Platform recipes from `README.md`:
+- **KNL**: `--enable-simd=KNL --enable-comms=mpi3-auto --enable-mkl`
+- **Skylake/Haswell**: `--enable-simd=AVX512` or `AVX2` + `--enable-comms=mpi3-auto`
+- **AMD EPYC**: `--enable-simd=AVX2 --enable-comms=mpi3`
+- **A64FX (Fugaku)**: `--enable-simd=A64FX --enable-comms=mpi3 --enable-shm=shmget` (see `SVE_README.txt`)
+
+Required external libs: GMP, MPFR, OpenSSL, zlib.
+
+### Use `systems/` for real machines
+
+`systems/<machine>/` holds the known-good build for each production platform (`Frontier`, `Aurora`, `Perlmutter`, `Summit`, `Tursa`, `Lumi`, `Booster`, `Crusher`, `SDCC-*`, `mac-arm`, …). Each contains a `config-command` (the exact `../../configure` invocation) and a `sourceme.sh` (module loads and env). **Prefer copying/adapting these over hand-rolling configure flags** — they encode compiler workarounds, `LDFLAGS`, and shared-memory settings that are easy to get wrong. `systems/WorkArounds.txt` records known vendor bugs.
+
+Note the GPU builds use `--enable-simd=GPU --enable-gen-simd-width=64`, so `Nsimd` is *not* 1 on device (it is `64/sizeof(scalar)`).
+
+### Regenerating `Make.inc` — required after adding or deleting source files
+
+`Make.inc` files are generated, not tracked in git (`.gitignore`d). `scripts/filelist` walks `Grid/`, `tests/*`, `benchmarks/`, `examples/`, and `HMC/` and writes the file lists and per-test `bin_PROGRAMS` rules. Every new `.cc`/`.h` in `Grid/`, and every new `Test_*.cc` / `Benchmark_*.cc` / `Example_*.cc`, is invisible to the build until you run:
+
+```bash
+./scripts/filelist    # from the source root, then re-run configure/make
+```
+
+`bootstrap.sh` runs it for you on the first setup.
+
+## Running Tests and Benchmarks
+
+```bash
+# From build directory
+make check                          # runs only Test_simd, Test_cshift, Test_stencil, Test_dwf_mixedcg_prec
+make -C tests/<subdir> tests        # build (not run) the tests in a subdirectory
+make bench                          # build benchmarks
+./tests/core/Test_simd              # run a single test binary directly
+mpirun -n 4 ./tests/core/Test_cshift --grid 16.16.16.16 --mpi 1.1.1.4
+```
+
+`make check` is a thin smoke test — building a subdirectory with `make -C tests/<subdir> tests` and running the relevant binaries directly is the normal development loop. Test binaries take Grid's standard command-line arguments (`--grid`, `--mpi`, `--accelerator-threads`, `--threads`, `--debug-signals`, `--log`); see `Grid/util/Init.cc`.
+
+Test subdirectories and their focus: `core` (SIMD, stencil, comms), `solver` (CG, GMRES, eigensolvers), `hmc` (MD integrators), `forces` (fermion forces), `lanczos`, `IO`, `smearing`, `sp2n`, `debug`.
+
+Tests and benchmarks that need optional fermion representations are guarded by `disable_tests_without_instantiations.h` / `disable_benchmarks_without_instantiations.h`, so a `--disable-fermion-reps --disable-gparity` build silently compiles them to no-ops.
+
+## Architecture
+
+### Layer stack (bottom to top)
+
+1. **SIMD layer** (`Grid/simd/`) — platform-specific intrinsics wrapped into `vRealF`, `vComplexD`, etc. The SIMD width and layout are compile-time constants controlled by `--enable-simd`.
+
+2. **Tensor layer** (`Grid/tensors/`) — Lorentz/colour/spin tensor algebra built on top of SIMD types. `iMatrix`, `iVector`, `iScalar` templates compose into QCD types like `ColourMatrix`, `SpinColourVector`.
+
+3. **Lattice layer** (`Grid/lattice/`) — `Lattice<T>` container: a site-local tensor replicated across a distributed Cartesian grid. All arithmetic is site-parallel and expression-template-fused.
+
+4. **Cartesian/comms layer** (`Grid/cartesian/`, `Grid/communicator/`) — `GridCartesian` holds the MPI topology and local/global geometry. `Grid/cshift/` implements nearest-neighbour halo exchange; `Grid/stencil/` is the optimised multi-hop stencil used by Dirac operators.
+
+5. **Algorithm layer** (`Grid/algorithms/`) — iterative solvers (CG, GMRES, BiCGSTAB, mixed-precision), eigensolvers (Lanczos, LAPACK), FFT, smearing.
+
+6. **QCD layer** (`Grid/qcd/`) — gauge and fermion actions, HMC integrators, observables.
+
+### QCD subsystem (`Grid/qcd/`)
+
+- `action/fermion/` — Wilson, Clover, DWF (Mobius), Staggered, twisted-mass, G-parity variants
+- `action/gauge/` — Wilson gauge, Symanzik, Iwasaki, DBW2, plaquette+rect
+- `representations/` — Fundamental, Adjoint, Two-index, Sp(2n)
+- `hmc/` — Leapfrog, OMF2/OMF4 integrators; pseudofermion refreshment; Metropolis accept/reject
+- `smearing/` — APE, Stout, HEX, gradient flow
+- `observables/` — Polyakov loop, plaquette, topological charge
+
+### GPU acceleration and the view/memory-manager discipline
+
+GPU support is injected via macros in `Grid/threads/Accelerator.h` — `accelerator_for(i, n, nsimd, {...})`, `accelerator_forNB` (non-blocking, must be followed by `accelerator_barrier()`), `accelerator_for2dNB`, and `accelerator_inline`. On a CPU build these degrade to `thread_for` (OpenMP). Unified virtual memory is on by default (`--enable-unified=yes`); device-aware MPI (`--enable-accelerator-aware-mpi`) avoids device→host copies on transfers.
+
+Lattice data is **not** directly addressable inside a kernel. You must open a view with the correct access mode so `Grid/allocator/MemoryManager.h` can move/mark the data:
+
+```cpp
+autoView(out_v, out, AcceleratorWriteDiscard);   // RAII; closes at end of scope
+autoView(in_v,  in,  AcceleratorRead);
+accelerator_for(ss, grid->oSites(), Nsimd, {
+    coalescedWrite(out_v[ss], coalescedRead(in_v[ss]));
+});
+```
+
+Modes are `AcceleratorRead/Write/WriteDiscard` and `CpuRead/Write/WriteDiscard`. Getting the mode wrong (e.g. `AcceleratorRead` on a field you write) produces stale-data bugs that only appear on GPU builds. Inside kernels use `coalescedRead`/`coalescedWrite` rather than raw `operator[]` — they map the SIMD lane onto `threadIdx.x` so accesses stay coalesced.
+
+### Repo-local debugging skills (`skills/`)
+
+`skills/` contains hard-won, Grid-specific playbooks written as invocable skill files. Consult them before debugging in these areas rather than reasoning from first principles:
+
+| File | Covers |
+|---|---|
+| `gpu-memory-performance.md` | `acceleratorThreads()`, LambdaApply thread mapping, `coalescedRead` idiom, fused vs staged HBM access |
+| `gpu-runtime-correctness.md` | GPU runtime returning early from sync, silent wrong answers |
+| `communication-overlap.md` | 7-phase halo pipeline, per-packet events, host-staging vs GPU-direct RDMA |
+| `mpi-heterogeneous.md` | `MPI_Sendrecv` device-buffer aliasing, deterministic reductions |
+| `compiler-validation.md` | Isolating GPU compiler codegen bugs, minimal reproducers |
+| `correctness-verification.md` | Double-run fingerprinting, per-packet checksums, flight recorder |
+| `hang-diagnosis.md` | Diagnosing MPI/accelerator hangs |
+
+### Memory and I/O
+
+- `Grid/allocator/` — aligned/NUMA-aware allocators; caching allocator via `--enable-alloc-cache`
+- `Grid/parallelIO/` — distributed parallel reader/writer for ILDG (via LIME), SciDAC, and native binary formats
+- `Grid/serialisation/` — text, binary, HDF5, XML/JSON serialisation of arbitrary Grid objects
+
+### Executables
+
+- `HMC/` — production HMC driver programmes (e.g. `Mobius2p1f.cc`, `DWF_plus_DSDR_nf2plus1_Shamir_Gparity.cc`)
+- `benchmarks/` — `Benchmark_dwf`, `Benchmark_ITT`, `Benchmark_comms`, `Benchmark_memory_bandwidth`, … used to qualify a new machine
+- `examples/` — small, readable programmes (`Example_plaquette.cc`, `Example_Mobius_spectrum.cc`) that are the best starting point for learning the API
+
+Each of these directories auto-builds every top-level `.cc` as its own binary via `scripts/filelist`.
+
+Every programme is wrapped in `Grid_init(&argc, &argv)` / `Grid_finalize()` (`Grid/util/Init.h`).
+
+## Key Conventions
+
+- **C++17** is required throughout.
+- Template structure: most classes are templated on `<_FImpl>` (fermion impl) or `<Gimpl>` (gauge impl), which encode the representation and precision. Instantiation is controlled by `--enable-fermion-instantiations`; the explicit instantiation `.cc` files live under `Grid/qcd/action/fermion/instantiation/<Impl>/` and are selected by `scripts/filelist`.
+- The `RealD`/`RealF`/`ComplexD`/`ComplexF` typedefs are used everywhere; avoid raw `double`/`float`.
+- Use `GRID_ASSERT(cond)` (defined in `Grid/GridStd.h`), not bare `assert` — it prints a Grid-formatted message and aborts cleanly under MPI.
+- Logging is stream-based, not macro-based: `std::cout << GridLogMessage << ... << std::endl;`. Channels declared in `Grid/log/Log.h` include `GridLogError`, `GridLogWarning`, `GridLogDebug`, `GridLogPerformance`, `GridLogIterative`, `GridLogSolver`, `GridLogHMC`, `GridLogComms`, `GridLogMemory`, `GridLogDslash`, `GridLogIRL`, `GridLogMG`. A subset is switched on at runtime with e.g. `--log Error,Warning,Message,Performance,Iterative,Integrator,Debug,Colours` (names given without the `GridLog` prefix).
+- Performance-critical paths use `GRID_TRACE(name)` from `Grid/perfmon/Tracing.h` (compiled out unless `--enable-tracing` selects a backend) and the `GridStopWatch` timers in `Grid/perfmon/Timer.h`.
+- Reductions across MPI ranks go through `GridBase::GlobalSum` / `GlobalMax`; never reduce with bare MPI calls inside library code.
+- Everything lives in `NAMESPACE_BEGIN(Grid)` / `NAMESPACE_END(Grid)` macros; follow the surrounding file rather than writing `namespace Grid { }`.
+- British spelling is used in identifiers and comments (`colour`, `neighbour`, `serialisation`).

Grid/algorithms/Algorithms.h

@@ -55,7 +55,6 @@ NAMESPACE_CHECK(approx);
 #include <Grid/algorithms/blas/MomentumProject.h>
 NAMESPACE_CHECK(deflation);
 #include <Grid/algorithms/iterative/ConjugateGradient.h>
-//#include <Grid/algorithms/iterative/EigCG.h>
 NAMESPACE_CHECK(ConjGrad);
 #include <Grid/algorithms/iterative/BiCGSTAB.h>
 NAMESPACE_CHECK(BiCGSTAB);
@@ -76,7 +75,6 @@ NAMESPACE_CHECK(BiCGSTAB);
 #include <Grid/algorithms/iterative/FlexibleCommunicationAvoidingGeneralisedMinimalResidual.h>
 #include <Grid/algorithms/iterative/MixedPrecisionFlexibleGeneralisedMinimalResidual.h>
 #include <Grid/algorithms/iterative/ImplicitlyRestartedLanczos.h>
-#include <Grid/algorithms/iterative/SimpleLanczos.h>
 #include <Grid/algorithms/iterative/PowerMethod.h>
 #include <Grid/algorithms/iterative/AdefGeneric.h>
 #include <Grid/algorithms/iterative/AdefMrhs.h>
@@ -85,17 +83,4 @@ NAMESPACE_CHECK(PowerMethod);
 NAMESPACE_CHECK(multigrid);
 #include <Grid/algorithms/FFT.h>
 
-#include <Grid/algorithms/iterative/KrylovSchur.h>
-#include <Grid/algorithms/iterative/BlockKrylovSchur.h>
-#include <Grid/algorithms/iterative/SplitGridBlockKrylovSchur.h>
-#include <Grid/algorithms/iterative/HarmonicBlockKrylovSchur.h>
-#include <Grid/algorithms/iterative/TrueHarmonicBlockKrylovSchur.h>
-#include <Grid/algorithms/iterative/Gamma5BlockLanczos.h>
-//#include <Grid/algorithms/iterative/Gamma5ScalarLanczos.h>
-#include <Grid/algorithms/iterative/Arnoldi.h>
-#include <Grid/algorithms/iterative/LanczosBidiagonalization.h>
-#include <Grid/algorithms/iterative/RestartedLanczosBidiagonalization.h>
-//#include <Grid/algorithms/iterative/GCR.h>
-#include <Grid/algorithms/iterative/MultiSplittingPreconditionedCG.h>
-
 #endif

Grid/algorithms/FFT.h

@@ -1,6 +1,6 @@
 /*************************************************************************************
 
-    Grid physics library, www.github.com/paboyle/Grid 
+    Grid physics library, www.github.com/paboyle/Grid
 
     Source file: ./lib/Cshift.h
 
@@ -65,17 +65,16 @@ public:
   typedef hipfftDoubleComplex FFTW_scalar;
   typedef hipfftHandle        FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
-				      int istride, int idist,		
-				      FFTW_scalar *out, int *onembed,		
-				      int ostride, int odist,		
+				      FFTW_scalar *in, int *inembed,
+				      int istride, int idist,
+				      FFTW_scalar *out, int *onembed,
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     auto rv = hipfftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,HIPFFT_Z2Z,howmany);
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
     return p;
-  }	  
-    
+  }
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     hipfftResult rv;
     if ( sign == forward ) rv =hipfftExecZ2Z(p,in,out,HIPFFT_FORWARD);
@@ -83,29 +82,25 @@ public:
     accelerator_barrier();
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
-    hipfftDestroy(p);
-  }
+  inline static void fftw_destroy_plan(const FFTW_plan p) { hipfftDestroy(p); }
 };
 template<> struct FFTW<ComplexF> {
 public:
   static const int forward=FFTW_FORWARD;
   static const int backward=FFTW_BACKWARD;
-  typedef hipfftComplex      FFTW_scalar;
-  typedef hipfftHandle        FFTW_plan;
-
+  typedef hipfftComplex FFTW_scalar;
+  typedef hipfftHandle  FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
-				      int istride, int idist,		
-				      FFTW_scalar *out, int *onembed,		
-				      int ostride, int odist,		
+				      FFTW_scalar *in, int *inembed,
+				      int istride, int idist,
+				      FFTW_scalar *out, int *onembed,
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     auto rv = hipfftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,HIPFFT_C2C,howmany);
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
     return p;
-  }	  
-    
+  }
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     hipfftResult rv;
     if ( sign == forward ) rv =hipfftExecC2C(p,in,out,HIPFFT_FORWARD);
@@ -113,9 +108,7 @@ public:
     accelerator_barrier();
     GRID_ASSERT(rv==HIPFFT_SUCCESS);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
-    hipfftDestroy(p);
-  }
+  inline static void fftw_destroy_plan(const FFTW_plan p) { hipfftDestroy(p); }
 };
 #endif
 
@@ -126,53 +119,45 @@ public:
   static const int backward=FFTW_BACKWARD;
   typedef cufftDoubleComplex FFTW_scalar;
   typedef cufftHandle        FFTW_plan;
-
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
-				      int istride, int idist,		
-				      FFTW_scalar *out, int *onembed,		
-				      int ostride, int odist,		
+				      FFTW_scalar *in, int *inembed,
+				      int istride, int idist,
+				      FFTW_scalar *out, int *onembed,
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     cufftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,CUFFT_Z2Z,howmany);
     return p;
-  }	  
-    
+  }
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     if ( sign == forward ) cufftExecZ2Z(p,in,out,CUFFT_FORWARD);
     else                   cufftExecZ2Z(p,in,out,CUFFT_INVERSE);
     accelerator_barrier();
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
-    cufftDestroy(p);
-  }
+  inline static void fftw_destroy_plan(const FFTW_plan p) { cufftDestroy(p); }
 };
 template<> struct FFTW<ComplexF> {
 public:
   static const int forward=FFTW_FORWARD;
   static const int backward=FFTW_BACKWARD;
   typedef cufftComplex FFTW_scalar;
-  typedef cufftHandle        FFTW_plan;
-
+  typedef cufftHandle  FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
-				      int istride, int idist,		
-				      FFTW_scalar *out, int *onembed,		
-				      int ostride, int odist,		
+				      FFTW_scalar *in, int *inembed,
+				      int istride, int idist,
+				      FFTW_scalar *out, int *onembed,
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     FFTW_plan p;
     cufftPlanMany(&p,rank,n,n,istride,idist,n,ostride,odist,CUFFT_C2C,howmany);
     return p;
-  }	  
-    
+  }
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     if ( sign == forward ) cufftExecC2C(p,in,out,CUFFT_FORWARD);
     else                   cufftExecC2C(p,in,out,CUFFT_INVERSE);
     accelerator_barrier();
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
-    cufftDestroy(p);
-  }
+  inline static void fftw_destroy_plan(const FFTW_plan p) { cufftDestroy(p); }
 };
 #endif
 
@@ -183,313 +168,325 @@ public:
   typedef fftw_complex FFTW_scalar;
   typedef fftw_plan    FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
-				      int istride, int idist,		
-				      FFTW_scalar *out, int *onembed,		
-				      int ostride, int odist,		
+				      FFTW_scalar *in, int *inembed,
+				      int istride, int idist,
+				      FFTW_scalar *out, int *onembed,
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     return ::fftw_plan_many_dft(rank,n,howmany,in,inembed,istride,idist,out,onembed,ostride,odist,sign,flags);
-  }	  
-    
+  }
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     ::fftw_execute_dft(p,in,out);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
-    ::fftw_destroy_plan(p);
-  }
+  inline static void fftw_destroy_plan(const FFTW_plan p) { ::fftw_destroy_plan(p); }
 };
 template<> struct FFTW<ComplexF> {
 public:
   typedef fftwf_complex FFTW_scalar;
   typedef fftwf_plan    FFTW_plan;
   static FFTW_plan fftw_plan_many_dft(int rank, int *n,int howmany,
-				      FFTW_scalar *in, int *inembed,		
-				      int istride, int idist,		
-				      FFTW_scalar *out, int *onembed,		
-				      int ostride, int odist,		
+				      FFTW_scalar *in, int *inembed,
+				      int istride, int idist,
+				      FFTW_scalar *out, int *onembed,
+				      int ostride, int odist,
 				      int sign, unsigned flags) {
     return ::fftwf_plan_many_dft(rank,n,howmany,in,inembed,istride,idist,out,onembed,ostride,odist,sign,flags);
-  }	  
-    
+  }
   inline static void fftw_execute_dft(const FFTW_plan p,FFTW_scalar *in,FFTW_scalar *out, int sign) {
     ::fftwf_execute_dft(p,in,out);
   }
-  inline static void fftw_destroy_plan(const FFTW_plan p) {
-    ::fftwf_destroy_plan(p);
-  }
+  inline static void fftw_destroy_plan(const FFTW_plan p) { ::fftwf_destroy_plan(p); }
 };
 #endif
 #endif
 
-class FFT {
-private:
-    
-  double flops;
-  double flops_call;
-  uint64_t usec;
-    
-public:
-    
-  static const int forward=FFTW_FORWARD;
-  static const int backward=FFTW_BACKWARD;
-    
-  double Flops(void) {return flops;}
-  double MFlops(void) {return flops/usec;}
-  double USec(void)   {return (double)usec;}    
+struct FFTbase {
+  double        flops;
+  double        flops_call;
+  uint64_t      usec;
+  GridCartesian *_grid;
 
-  FFT ( GridCartesian * grid ) 
-  {
-    flops=0;
-    usec =0;
-  };
-    
-  ~FFT ( void)  {
-    //    delete sgrid;
+  static const int forward  = FFTW_FORWARD;
+  static const int backward = FFTW_BACKWARD;
+
+  double Flops(void)  { return flops; }
+  double MFlops(void) { return flops / usec; }
+  double USec(void)   { return (double)usec; }
+
+  FFTbase(GridCartesian *grid) : _grid(grid), flops(0), flops_call(0), usec(0) {}
+};
+
+// Barrel-shift gather, FFT execute, and insert. Called by both FFT and PlannedFFT.
+// The caller is responsible for plan acquisition and destruction.
+template<class vobj>
+static void FFT_dim_execute(
+    Lattice<vobj>       &result,
+    const Lattice<vobj> &source,
+    int dim, int sign,
+    typename FFTW<typename vobj::scalar_type>::FFTW_plan p,
+    GridCartesian *grid,
+    double &flops, double &flops_call, uint64_t &usec)
+{
+  typedef typename vobj::scalar_type   scalar;
+  typedef typename vobj::scalar_object sobj;
+  typedef typename vobj::scalar_type   scalar_type;
+  typedef typename vobj::vector_type   vector_type;
+  typedef typename FFTW<scalar>::FFTW_scalar FFTW_scalar;
+
+  const int Ndim = grid->Nd();
+  int L = grid->_ldimensions[dim];
+  int G = grid->_fdimensions[dim];
+  int Ncomp = sizeof(sobj) / sizeof(scalar);
+  int64_t Nlow = 1, Nhigh = 1;
+  for (int d = 0; d < dim;      d++) Nlow  *= grid->_ldimensions[d];
+  for (int d = dim+1; d < Ndim; d++) Nhigh *= grid->_ldimensions[d];
+  int64_t Nperp = Nlow * Nhigh;
+
+  deviceVector<scalar> pgbuf(Nperp * Ncomp * G);
+  scalar *pgbuf_v = &pgbuf[0];
+  int howmany = Ncomp * Nperp;
+
+  scalar div;
+  if      (sign == FFTW_BACKWARD) div = 1.0 / G;
+  else if (sign == FFTW_FORWARD)  div = 1.0;
+  else GRID_ASSERT(0);
+
+  double t_pencil = 0, t_fft = 0, t_copy = 0, t_shift = 0;
+  double t_total  = -usecond();
+
+  result = source;
+  int pc = grid->_processor_coor[dim];
+
+  const Coordinate ldims      = grid->_ldimensions;
+  const Coordinate rdims      = grid->_rdimensions;
+  const Coordinate sdims      = grid->_simd_layout;
+  const Coordinate processors = grid->_processors;
+
+  Coordinate pgdims(Ndim);
+  pgdims[0] = G;
+  for (int d = 0, dd = 1; d < Ndim; d++)
+    if (d != dim) pgdims[dd++] = ldims[d];
+  int64_t pgvol = 1;
+  for (int d = 0; d < Ndim; d++) pgvol *= pgdims[d];
+
+  const int Nsimd = vobj::Nsimd();
+  t_pencil = -usecond();
+  for (int p_idx = 0; p_idx < processors[dim]; p_idx++) {
+    t_copy -= usecond();
+    autoView(r_v, result, AcceleratorRead);
+    accelerator_for(idx, grid->oSites(), vobj::Nsimd(), {
+#ifdef GRID_SIMT
+    {
+      int lane = acceleratorSIMTlane(Nsimd);
+#else
+    for (int lane = 0; lane < Nsimd; lane++) {
+#endif
+      Coordinate icoor, ocoor, pgcoor;
+      Lexicographic::CoorFromIndex(icoor, lane, sdims);
+      Lexicographic::CoorFromIndex(ocoor, idx,  rdims);
+      pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + ((pc+p_idx)%processors[dim])*L;
+      for (int d = 0, dd = 1; d < Ndim; d++)
+        if (d != dim) { pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d]; dd++; }
+      int64_t pgidx;
+      Lexicographic::IndexFromCoor(pgcoor, pgidx, pgdims);
+      vector_type *from = (vector_type *)&r_v[idx];
+      scalar_type  stmp;
+      for (int w = 0; w < Ncomp; w++) {
+        stmp = getlane(from[w], lane);
+        pgbuf_v[pgidx + w*pgvol] = stmp;
+      }
+#ifdef GRID_SIMT
+    }
+#else
+    }
+#endif
+    });
+    t_copy += usecond();
+    if (p_idx != processors[dim] - 1) {
+      Lattice<vobj> temp(grid);
+      t_shift -= usecond();
+      temp = Cshift(result, dim, L); result = temp;
+      t_shift += usecond();
+    }
   }
-    
-  template<class vobj>
-  void FFT_dim_mask(Lattice<vobj> &result,const Lattice<vobj> &source,Coordinate mask,int sign){
+  t_pencil += usecond();
 
-    //    vgrid=result.Grid();
-    //    conformable(result.Grid(),vgrid);
-    //    conformable(source.Grid(),vgrid);
-    const int Ndim = source.Grid()->Nd();
+  FFTW_scalar *in  = (FFTW_scalar *)pgbuf_v;
+  FFTW_scalar *out = (FFTW_scalar *)pgbuf_v;
+  t_fft = -usecond();
+  FFTW<scalar>::fftw_execute_dft(p, in, out, sign);
+  t_fft += usecond();
+
+  flops_call = 5.0 * howmany * G * log2(G);
+  usec  = t_fft;
+  flops = flops_call;
+
+  result = Zero();
+  double t_insert = -usecond();
+  {
+    autoView(r_v, result, AcceleratorWrite);
+    accelerator_for(idx, grid->oSites(), Nsimd, {
+#ifdef GRID_SIMT
+    {
+      int lane = acceleratorSIMTlane(Nsimd);
+#else
+    for (int lane = 0; lane < Nsimd; lane++) {
+#endif
+      Coordinate icoor(Ndim), ocoor(Ndim), pgcoor(Ndim);
+      Lexicographic::CoorFromIndex(icoor, lane, sdims);
+      Lexicographic::CoorFromIndex(ocoor, idx,  rdims);
+      pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + pc*L;
+      for (int d = 0, dd = 1; d < Ndim; d++)
+        if (d != dim) { pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d]; dd++; }
+      int64_t pgidx;
+      Lexicographic::IndexFromCoor(pgcoor, pgidx, pgdims);
+      vector_type *to  = (vector_type *)&r_v[idx];
+      scalar_type  stmp;
+      for (int w = 0; w < Ncomp; w++) {
+        stmp = pgbuf_v[pgidx + w*pgvol];
+        putlane(to[w], stmp, lane);
+      }
+#ifdef GRID_SIMT
+    }
+#else
+    }
+#endif
+    });
+  }
+  result = result * div;
+  t_insert += usecond();
+  t_total  += usecond();
+
+  std::cout << GridLogPerformance << " FFT took   " << t_total/1.0e6  << " s" << std::endl;
+  std::cout << GridLogPerformance << " FFT pencil " << t_pencil/1.0e6 << " s" << std::endl;
+  std::cout << GridLogPerformance << "  of which copy " << t_copy/1.0e6  << " s" << std::endl;
+  std::cout << GridLogPerformance << "  of which shift" << t_shift/1.0e6 << " s" << std::endl;
+  std::cout << GridLogPerformance << " FFT kernels " << t_fft/1.0e6    << " s" << std::endl;
+  std::cout << GridLogPerformance << " FFT insert  " << t_insert/1.0e6 << " s" << std::endl;
+}
+
+class FFT : public FFTbase {
+public:
+  FFT(GridCartesian *grid) : FFTbase(grid) {}
+  ~FFT() {}
+
+  template<class vobj>
+  void FFT_dim_mask(Lattice<vobj> &result, const Lattice<vobj> &source, Coordinate mask, int sign) {
+    const int Ndim = _grid->Nd();
     Lattice<vobj> tmp = source;
-    for(int d=0;d<Ndim;d++){
-      if( mask[d] ) {
-	FFT_dim(result,tmp,d,sign);
-	tmp=result;
+    for (int d = 0; d < Ndim; d++) {
+      if (mask[d]) {
+        FFT_dim(result, tmp, d, sign);
+        tmp = result;
       }
     }
   }
 
   template<class vobj>
-  void FFT_all_dim(Lattice<vobj> &result,const Lattice<vobj> &source,int sign){
-    const int Ndim = source.Grid()->Nd();
-    Coordinate mask(Ndim,1);
-    FFT_dim_mask(result,source,mask,sign);
+  void FFT_all_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int sign) {
+    Coordinate mask(_grid->Nd(), 1);
+    FFT_dim_mask(result, source, mask, sign);
   }
 
-
   template<class vobj>
-  void FFT_dim(Lattice<vobj> &result,const Lattice<vobj> &source,int dim, int sign){
-    const int Ndim = source.Grid()->Nd();
-    GridBase *grid = source.Grid();
-    conformable(result.Grid(),source.Grid());
+  void FFT_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int dim, int sign) {
+    GRID_ASSERT(source.Grid() == _grid);
+    GRID_ASSERT(result.Grid() == _grid);
+    conformable(result.Grid(), source.Grid());
 
-    int L = grid->_ldimensions[dim];
-    int G = grid->_fdimensions[dim];
-      
-    Coordinate layout(Ndim,1);
-    
-    // Construct pencils
-    typedef typename vobj::scalar_object sobj;
     typedef typename vobj::scalar_type   scalar;
-    typedef typename vobj::scalar_type   scalar_type;
-    typedef typename vobj::vector_type   vector_type;
-      
-    //std::cout << "CPU view" << std::endl;
-    
+    typedef typename vobj::scalar_object sobj;
     typedef typename FFTW<scalar>::FFTW_scalar FFTW_scalar;
     typedef typename FFTW<scalar>::FFTW_plan   FFTW_plan;
-      
-    int Ncomp = sizeof(sobj)/sizeof(scalar);
-    int64_t Nlow  = 1;
-    int64_t Nhigh = 1;
 
-    for(int d=0;d<dim;d++){
-      Nlow*=grid->_ldimensions[d];
-    }
-    for(int d=dim+1;d<Ndim;d++){
-      Nhigh*=grid->_ldimensions[d];
-    }
-    int64_t Nperp=Nlow*Nhigh;
-    
-    deviceVector<scalar> pgbuf; // Layout is [perp][component][dim]
-    pgbuf.resize(Nperp*Ncomp*G);
-    scalar *pgbuf_v = &pgbuf[0];
-      
-    int rank = 1;  /* 1d transforms */
-    int n[] = {G}; /* 1d transforms of length G */
+    const int Ndim = _grid->Nd();
+    int G     = _grid->_fdimensions[dim];
+    int Ncomp = sizeof(sobj) / sizeof(scalar);
+    int64_t Nperp = 1;
+    for (int d = 0; d < Ndim; d++)
+      if (d != dim) Nperp *= _grid->_ldimensions[d];
+    int n[]     = {G};
     int howmany = Ncomp * Nperp;
-    int odist,idist,istride,ostride;
-    idist   = odist   = G;            /* Distance between consecutive FT's */
-    istride = ostride = 1;            /* Distance between two elements in the same FT */
-    int *inembed = n, *onembed = n;
-      
-    scalar div;
-    if ( sign == backward ) div = 1.0/G;
-    else if ( sign == forward ) div = 1.0;
-    else GRID_ASSERT(0);
 
-    double t_pencil=0;
-    double t_fft   =0;
-    double t_total =-usecond();
-    //    std::cout << GridLogPerformance<<"Making FFTW plan" << std::endl;
-    /*
-     *
-     */
-    FFTW_plan p;
-    {
-      FFTW_scalar *in = (FFTW_scalar *)&pgbuf_v[0];
-      FFTW_scalar *out= (FFTW_scalar *)&pgbuf_v[0];
-      p = FFTW<scalar>::fftw_plan_many_dft(rank,n,howmany,
-					   in,inembed,
-					   istride,idist,
-					   out,onembed,
-					   ostride, odist,
-					   sign,FFTW_ESTIMATE);
-    }
-      
-    // Barrel shift and collect global pencil
-    //    std::cout << GridLogPerformance<<"Making pencil" << std::endl;
-    Coordinate lcoor(Ndim), gcoor(Ndim);
-    double t_copy=0;
-    double t_shift=0;
-    t_pencil = -usecond();
-    result = source;
-    int pc = grid->_processor_coor[dim];
-
-    const Coordinate ldims = grid->_ldimensions;
-    const Coordinate rdims = grid->_rdimensions;
-    const Coordinate sdims = grid->_simd_layout;
-
-    Coordinate processors = grid->_processors;
-    Coordinate pgdims(Ndim);
-    pgdims[0] = G;
-    for(int d=0, dd=1;d<Ndim;d++){
-      if ( d!=dim ) pgdims[dd++] = ldims[d];
-    }
-    int64_t pgvol=1;
-    for(int d=0;d<Ndim;d++) pgvol*=pgdims[d];
-    
-    const int Nsimd = vobj::Nsimd();
-    for(int p=0;p<processors[dim];p++) {
-      t_copy-=usecond();
-      autoView(r_v,result,AcceleratorRead);
-      accelerator_for(idx, grid->oSites(), vobj::Nsimd(), {
-#ifdef GRID_SIMT
-      {
-	int lane=acceleratorSIMTlane(Nsimd); // buffer lane
-#else
-      for(int lane=0;lane<Nsimd;lane++) {
-#endif
-	Coordinate icoor;
-	Coordinate ocoor;
-	Coordinate pgcoor;
-
-	Lexicographic::CoorFromIndex(icoor,lane,sdims);
-	Lexicographic::CoorFromIndex(ocoor,idx,rdims);
-
-	pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + ((pc+p)%processors[dim])*L;
-	for(int d=0,dd=1;d<Ndim;d++){
-	  if ( d!=dim ) {
-	    pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d];
-	    dd++;
-	  }
-	}
-
-	// Map coordinates in lattice layout to FFTW index
-	int64_t pgidx;
-	Lexicographic::IndexFromCoor(pgcoor,pgidx,pgdims);
-
-	vector_type *from = (vector_type *)&r_v[idx];
-	scalar_type stmp;
-	for(int w=0;w<Ncomp;w++){
-	  int64_t pg_idx = pgidx + w*pgvol;
-	  stmp = getlane(from[w], lane);
-	  pgbuf_v[pg_idx] = stmp;
-	}
-#ifdef GRID_SIMT
-      }
-#else
-      }
-#endif
-      });
-
-      t_copy+=usecond();
-      if (p != processors[dim] - 1) {
-	Lattice<vobj> temp(grid);
-	t_shift-=usecond();
-	temp = Cshift(result,dim,L); result = temp;
-	t_shift+=usecond();
-      }
-    }
-    t_pencil += usecond();
-      
-    FFTW_scalar *in = (FFTW_scalar *)pgbuf_v;
-    FFTW_scalar *out= (FFTW_scalar *)pgbuf_v;
-    t_fft = -usecond();
-    FFTW<scalar>::fftw_execute_dft(p,in,out,sign);
-    t_fft += usecond();
-    
-    // performance counting
-    flops_call = 5.0*howmany*G*log2(G);
-    usec = t_fft;
-    flops= flops_call;
-
-    result = Zero();
-    
-    double t_insert = -usecond();
-    {
-      autoView(r_v,result,AcceleratorWrite);
-      accelerator_for(idx,grid->oSites(),Nsimd,{
-#ifdef GRID_SIMT
-      {
-	int lane=acceleratorSIMTlane(Nsimd); // buffer lane
-#else
-      for(int lane=0;lane<Nsimd;lane++) {
-#endif
-	Coordinate icoor(Ndim);
-	Coordinate ocoor(Ndim);
-	Coordinate pgcoor(Ndim);
-
-	Lexicographic::CoorFromIndex(icoor,lane,sdims);
-	Lexicographic::CoorFromIndex(ocoor,idx,rdims);
-
-	pgcoor[0] = ocoor[dim] + icoor[dim]*rdims[dim] + pc*L;
-	for(int d=0,dd=1;d<Ndim;d++){
-	  if ( d!=dim ) {
-	    pgcoor[dd] = ocoor[d] + icoor[d]*rdims[d];
-	    dd++;
-	  }
-	}
-	// Map coordinates in lattice layout to FFTW index
-	int64_t pgidx;
-	Lexicographic::IndexFromCoor(pgcoor,pgidx,pgdims);
-
-	vector_type *to = (vector_type *)&r_v[idx];
-	scalar_type stmp;
-	for(int w=0;w<Ncomp;w++){
-	  int64_t pg_idx = pgidx + w*pgvol;
-	  stmp = pgbuf_v[pg_idx];
-	  putlane(to[w], stmp, lane);
-	}
-	
-#ifdef GRID_SIMT
-      }
-#else
-      }
-#endif
-      });
-    }
-
-    result = result*div;
-
-    t_insert +=usecond();
-    
-    // destroying plan
+    deviceVector<scalar> dummy(2);
+    FFTW_scalar *buf = (FFTW_scalar *)&dummy[0];
+    FFTW_plan p = FFTW<scalar>::fftw_plan_many_dft(1, n, howmany,
+                                                    buf, n, 1, G,
+                                                    buf, n, 1, G,
+                                                    sign, FFTW_ESTIMATE);
+    FFT_dim_execute(result, source, dim, sign, p, _grid, flops, flops_call, usec);
     FFTW<scalar>::fftw_destroy_plan(p);
+  }
+};
 
-    t_total +=usecond();
+template<class vobj>
+class PlannedFFT : public FFTbase {
+private:
+  typedef typename vobj::scalar_type   scalar;
+  typedef typename vobj::scalar_object sobj;
+  typedef typename vobj::vector_type   vector_type;
+  typedef typename FFTW<scalar>::FFTW_scalar FFTW_scalar;
+  typedef typename FFTW<scalar>::FFTW_plan   FFTW_plan;
 
-    std::cout <<GridLogPerformance<< " FFT took   "<<t_total/1.0e6 <<" s" << std::endl;
-    std::cout <<GridLogPerformance<< " FFT pencil "<<t_pencil/1.0e6 <<" s" << std::endl;
-    std::cout <<GridLogPerformance<< "  of which copy "<<t_copy/1.0e6 <<" s" << std::endl;
-    std::cout <<GridLogPerformance<< "  of which shift"<<t_shift/1.0e6 <<" s" << std::endl;
-    std::cout <<GridLogPerformance<< " FFT kernels "<<t_fft/1.0e6 <<" s" << std::endl;
-    std::cout <<GridLogPerformance<< " FFT insert  "<<t_insert/1.0e6 <<" s" << std::endl;
-    
+  std::vector<FFTW_plan> forward_plans;
+  std::vector<FFTW_plan> backward_plans;
+
+  void PlanCreate() {
+    const int Ndim = _grid->Nd();
+    forward_plans.resize(Ndim);
+    backward_plans.resize(Ndim);
+
+    for (int d = 0; d < Ndim; d++) {
+      int G       = _grid->_fdimensions[d];
+      int Ncomp   = sizeof(sobj) / sizeof(scalar);
+      int64_t Nperp = 1;
+      for (int dd = 0; dd < Ndim; dd++)
+        if (dd != d) Nperp *= _grid->_ldimensions[dd];
+      int howmany = Ncomp * (int)Nperp;
+      int n[]     = {G};
+
+      deviceVector<scalar> dummy(2);
+      FFTW_scalar *buf = (FFTW_scalar *)&dummy[0];
+
+      forward_plans[d]  = FFTW<scalar>::fftw_plan_many_dft(1, n, howmany, buf, n, 1, G, buf, n, 1, G, FFTW_FORWARD,  FFTW_ESTIMATE);
+      backward_plans[d] = FFTW<scalar>::fftw_plan_many_dft(1, n, howmany, buf, n, 1, G, buf, n, 1, G, FFTW_BACKWARD, FFTW_ESTIMATE);
+    }
+  }
+
+  void PlanDestroy() {
+    for (auto p : forward_plans)  FFTW<scalar>::fftw_destroy_plan(p);
+    for (auto p : backward_plans) FFTW<scalar>::fftw_destroy_plan(p);
+    forward_plans.clear();
+    backward_plans.clear();
+  }
+
+public:
+  PlannedFFT(GridCartesian *grid) : FFTbase(grid) { PlanCreate(); }
+  ~PlannedFFT() { PlanDestroy(); }
+
+  void FFT_dim_mask(Lattice<vobj> &result, const Lattice<vobj> &source, Coordinate mask, int sign) {
+    const int Ndim = _grid->Nd();
+    Lattice<vobj> tmp = source;
+    for (int d = 0; d < Ndim; d++) {
+      if (mask[d]) {
+        FFT_dim(result, tmp, d, sign);
+        tmp = result;
+      }
+    }
+  }
+
+  void FFT_all_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int sign) {
+    Coordinate mask(_grid->Nd(), 1);
+    FFT_dim_mask(result, source, mask, sign);
+  }
+
+  void FFT_dim(Lattice<vobj> &result, const Lattice<vobj> &source, int dim, int sign) {
+    GRID_ASSERT(source.Grid() == _grid);
+    GRID_ASSERT(result.Grid() == _grid);
+    GRID_ASSERT((int)forward_plans.size() == _grid->Nd());
+    conformable(result.Grid(), source.Grid());
+    FFTW_plan p = (sign == forward ? forward_plans : backward_plans)[dim];
+    FFT_dim_execute(result, source, dim, sign, p, _grid, flops, flops_call, usec);
   }
 };
 

Grid/algorithms/blas/BatchedBlas.h

@@ -279,7 +279,7 @@ public:
                                    (hipblasDoubleComplex **)&Cmn[0], ldc,
                                    batchCount);
 #endif
-    //	 std::cout << " hipblas return code " <<(int)err<<std::endl;
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -541,6 +541,7 @@ public:
                                    batchCount);
 
 #endif
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -577,6 +578,7 @@ public:
 				(void **)&Cmn[0], CUDA_C_32F, ldc,
 				batchCount, compute_precision, CUBLAS_GEMM_DEFAULT);
     }
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==CUBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_SYCL
@@ -747,6 +749,7 @@ public:
 				   (float *) &beta_p[0],
 				   (float **)&Cmn[0], ldc,
 				   batchCount);
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -907,6 +910,7 @@ public:
 				   (double *) &beta_p[0],
 				   (double **)&Cmn[0], ldc,
 				   batchCount);
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -1134,6 +1138,7 @@ public:
                                     (int*) &info[0],
                                     (int)batchCount);
 #endif
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -1173,6 +1178,7 @@ public:
                                     (int)batchCount);
 #endif
 
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -1261,6 +1267,7 @@ public:
                                     (int)batchCount);
 
 #endif
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA
@@ -1304,6 +1311,7 @@ public:
                                     (int*) &info[0],
                                     (int)batchCount);
 #endif
+    //    std::cout << " hipblas return code " <<(int)err<<" "<<__LINE__<<std::endl;
     GRID_ASSERT(err==HIPBLAS_STATUS_SUCCESS);
 #endif
 #ifdef GRID_CUDA

Grid/algorithms/iterative/AdefGeneric.h

@@ -53,7 +53,22 @@ class TwoLevelCG : public LinearFunction<Field>
   // Fine operator, Smoother, CoarseSolver
   LinearOperatorBase<Field>   &_FineLinop;
   LinearFunction<Field>   &_Smoother;
-  
+
+  GridStopWatch ProjectTimer;
+  GridStopWatch PromoteTimer;
+  GridStopWatch CoarseTimer;
+  GridStopWatch SmoothTimer;
+  GridStopWatch MatrixTimer;
+  GridStopWatch M3Timer;
+  GridStopWatch LinalgTimer;
+
+  int64_t M3Calls;
+  int64_t SmoothCalls;
+  int64_t MatrixCalls;
+  int64_t ProjectCalls;
+  int64_t CoarseCalls;
+  int64_t PromoteCalls;
+
   // more most opertor functions
   TwoLevelCG(RealD tol,
 	     Integer maxit,
@@ -103,12 +118,20 @@ class TwoLevelCG : public LinearFunction<Field>
     RealD tn;
     
     GridStopWatch HDCGTimer;
+    ProjectTimer.Reset();
+    PromoteTimer.Reset();
+    CoarseTimer.Reset();
+    SmoothTimer.Reset();
+    MatrixTimer.Reset();
+    M3Timer.Reset();
+    LinalgTimer.Reset();
+    M3Calls = SmoothCalls = MatrixCalls = ProjectCalls = CoarseCalls = PromoteCalls = 0;
     HDCGTimer.Start();
     //////////////////////////
     // x0 = Vstart -- possibly modify guess
     //////////////////////////
     Vstart(x,src);
-    
+
     // r0 = b -A x0
     _FineLinop.HermOp(x,mmp[0]);
     axpy (r, -1.0,mmp[0], src);    // Recomputes r=src-Ax0
@@ -145,33 +168,40 @@ class TwoLevelCG : public LinearFunction<Field>
       int peri_kp = (k+1) % mmax;
 
       rtz=rtzp;
+      M3Timer.Start();
       d= PcgM3(p[peri_k],mmp[peri_k]);
+      M3Timer.Stop();
+      M3Calls++;
       a = rtz/d;
-    
+
       // Memorise this
       pAp[peri_k] = d;
-      
+
+      LinalgTimer.Start();
       axpy(x,a,p[peri_k],x);
       RealD rn = axpy_norm(r,-a,mmp[peri_k],r);
+      LinalgTimer.Stop();
 
       // Compute z = M x
       PcgM1(r,z);
-      
+
       {
 	RealD n1,n2;
 	n1=norm2(r);
 	n2=norm2(z);
 	std::cout << GridLogMessage<<"HDCG::fPcg iteration "<<k<<" : vector r,z "<<n1<<" "<<n2<<"\n";
       }
+      LinalgTimer.Start();
       rtzp =real(innerProduct(r,z));
+      LinalgTimer.Stop();
       std::cout << GridLogMessage<<"HDCG::fPcg iteration "<<k<<" : inner rtzp "<<rtzp<<"\n";
 
       //    PcgM2(z,p[0]);
       PcgM2(z,mu); // ADEF-2 this is identity. Axpy possible to eliminate
-      
+
       p[peri_kp]=mu;
 
-      // Standard search direction  p -> z + b p    
+      // Standard search direction  p -> z + b p
       b = (rtzp)/rtz;
       
       int northog;
@@ -202,8 +232,25 @@ class TwoLevelCG : public LinearFunction<Field>
       if ( rn <= rsq ) { 
 
 	HDCGTimer.Stop();
-	std::cout<<GridLogMessage<<"HDCG: fPcg converged in "<<k<<" iterations and "<<HDCGTimer.Elapsed()<<std::endl;;
-	
+	std::cout<<GridLogMessage<<"HDCG: fPcg converged in "<<k<<" iterations and "<<HDCGTimer.Elapsed()<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg breakdown"<<std::endl;
+	auto mspc = [](GridStopWatch &sw, int64_t n) -> double {
+	  return (n > 0) ? sw.useconds() * 1e-3 / n : 0.0;
+	};
+	std::cout<<GridLogMessage<<"HDCG: fPcg M3 (fine MVM)  "<<M3Timer.Elapsed()
+		 <<"  "<<M3Calls<<" calls  "<<mspc(M3Timer,M3Calls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg linalg         "<<LinalgTimer.Elapsed()<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg smoother       "<<SmoothTimer.Elapsed()
+		 <<"  "<<SmoothCalls<<" calls  "<<mspc(SmoothTimer,SmoothCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg matrix (in M1) "<<MatrixTimer.Elapsed()
+		 <<"  "<<MatrixCalls<<" calls  "<<mspc(MatrixTimer,MatrixCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg project        "<<ProjectTimer.Elapsed()
+		 <<"  "<<ProjectCalls<<" calls  "<<mspc(ProjectTimer,ProjectCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg coarse         "<<CoarseTimer.Elapsed()
+		 <<"  "<<CoarseCalls<<" calls  "<<mspc(CoarseTimer,CoarseCalls)<<" ms/call"<<std::endl;
+	std::cout<<GridLogMessage<<"HDCG: fPcg promote        "<<PromoteTimer.Elapsed()
+		 <<"  "<<PromoteCalls<<" calls  "<<mspc(PromoteTimer,PromoteCalls)<<" ms/call"<<std::endl;
+
 	_FineLinop.HermOp(x,mmp[0]);			  
 	axpy(tmp,-1.0,src,mmp[0]);
 	
@@ -475,35 +522,29 @@ class TwoLevelADEF2 : public TwoLevelCG<Field>
     CoarseField PleftProj(this->coarsegrid);
     CoarseField PleftMss_proj(this->coarsegrid);
 
-    GridStopWatch SmootherTimer;
-    GridStopWatch MatrixTimer;
-    SmootherTimer.Start();
+    this->SmoothTimer.Start();
     this->_Smoother(in,Min);
-    SmootherTimer.Stop();
+    this->SmoothTimer.Stop();
+    this->SmoothCalls++;
 
-    MatrixTimer.Start();
+    this->MatrixTimer.Start();
     this->_FineLinop.HermOp(Min,out);
-    MatrixTimer.Stop();
+    this->MatrixTimer.Stop();
+    this->MatrixCalls++;
     axpy(tmp,-1.0,out,in);          // tmp  = in - A Min
 
-    GridStopWatch ProjTimer;
-    GridStopWatch CoarseTimer;
-    GridStopWatch PromTimer;
-    ProjTimer.Start();
-    this->_Aggregates.ProjectToSubspace(PleftProj,tmp);     
-    ProjTimer.Stop();
-    CoarseTimer.Start();
+    this->ProjectTimer.Start();
+    this->_Aggregates.ProjectToSubspace(PleftProj,tmp);
+    this->ProjectTimer.Stop();
+    this->ProjectCalls++;
+    this->CoarseTimer.Start();
     this->_CoarseSolver(PleftProj,PleftMss_proj); // Ass^{-1} [in - A Min]_s
-    CoarseTimer.Stop();
-    PromTimer.Start();
-    this->_Aggregates.PromoteFromSubspace(PleftMss_proj,tmp);// tmp = Q[in - A Min]  
-    PromTimer.Stop();
-    std::cout << GridLogPerformance << "PcgM1 breakdown "<<std::endl;
-    std::cout << GridLogPerformance << "\tSmoother   " << SmootherTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tProj       " << ProjTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tCoarse     " << CoarseTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\tProm       " << PromTimer.Elapsed() <<std::endl;
+    this->CoarseTimer.Stop();
+    this->CoarseCalls++;
+    this->PromoteTimer.Start();
+    this->_Aggregates.PromoteFromSubspace(PleftMss_proj,tmp);// tmp = Q[in - A Min]
+    this->PromoteTimer.Stop();
+    this->PromoteCalls++;
 
     axpy(out,1.0,Min,tmp); // Min+tmp
   }

Grid/algorithms/iterative/Arnoldi.h

@@ -1,433 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid 
-
-Source file: ./lib/algorithms/iterative/Arnoldi.h
-
-Copyright (C) 2015
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-Author: paboyle <paboyle@ph.ed.ac.uk>
-Author: Patrick Oare <poare@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_ARNOLDI_H
-#define GRID_ARNOLDI_H
-
-NAMESPACE_BEGIN(Grid); 
-
-//Moved to KrylovSchur
-#if 0
-/**
-<<<<<<< HEAD
- * Options for which Ritz values to keep in implicit restart.
- */
-enum RitzFilter {
-  EvalNormSmall,           // Keep evals with smallest norm
-  EvalNormLarge,           // Keep evals with largest norm
-  EvalReSmall,             // Keep evals with smallest real part
-  EvalReLarge              // Keep evals with largest real part
-};
-
-// Select comparison function from RitzFilter
-struct ComplexComparator
-{
-  RitzFilter f;
-  ComplexComparator (RitzFilter _f) : f(_f) {}
-  bool operator()(std::complex<double> z1, std::complex<double> z2) { 
-    switch (f) {
-      RealD tmp1, tmp2;
-      tmp1=std::abs(std::imag(z1));
-      tmp2=std::abs(std::imag(z2));
-      case EvalNormSmall:
-        return std::abs(z1) < std::abs(z2);
-      case EvalNormLarge:
-        return std::abs(z1) > std::abs(z2);
-// Terrible hack
-//        return std::abs(std::real(z1)) < std::abs(std::real(z2));
-//	if ( std::abs(std::real(z1))  >4.) tmp1 +=1.;
-//	if ( std::abs(std::real(z2))  >4.) tmp2 +=1.;
-      case EvalReSmall:
-	  return tmp1 < tmp2;
-//        return std::abs(std::imag(z1)) < std::abs(std::imag(z2));
-      case EvalReLarge:
-	  return tmp1 > tmp2;
-//        return std::abs(std::real(z1)) > std::abs(std::real(z2));
-      default:
-        assert(0);
-    }
-  }
-};
-
-=======
->>>>>>> 68af1bba67dd62881ead5ab1e54962a5486a0791
-#endif
-
-/**
- * Implementation of the Arnoldi algorithm.
- */
-template<class Field> 
-class Arnoldi {
-
-  private:
-  
-    std::string cname = std::string("Arnoldi");
-    int MaxIter;   // Max iterations
-    RealD Tolerance;
-    RealD ssq;
-    RealD rtol;
-    int Nm;           // Number of basis vectors to track (equals MaxIter if no restart)
-    int Nk;           // Number of basis vectors to keep every restart (equals -1 if no restart)
-    int Nstop;       // Stop after converging Nstop eigenvectors.
-
-    LinearOperatorBase<Field> &Linop;
-    GridBase *Grid;
-
-    RealD approxLambdaMax;
-    RealD beta_k;
-    Field f;
-    std::vector<Field> basis;               // orthonormal Arnoldi basis
-    Eigen::MatrixXcd Hess;                  // Hessenberg matrix of size Nbasis (after construction)
-    Eigen::MatrixXcd Qt;                    // Transpose of basis rotation which projects out high modes.
-
-    Eigen::VectorXcd evals;                 // evals of Hess
-    Eigen::MatrixXcd littleEvecs;           // Nm x Nm evecs matrix
-    std::vector<Field> evecs;               // Vector of evec fields
-
-    RitzFilter ritzFilter;                        // how to sort evals
-
-  public:       
-
-    Arnoldi(LinearOperatorBase<Field> &_Linop, GridBase *_Grid, RealD _Tolerance, RitzFilter filter = EvalReSmall)
-      : Linop(_Linop), Grid(_Grid), Tolerance(_Tolerance), ritzFilter(filter), f(_Grid), MaxIter(-1), Nm(-1), Nk(-1), 
-          Nstop (-1), evals (0), evecs (), ssq (0.0), rtol (0.0), beta_k (0.0), approxLambdaMax (0.0)
-    {
-      f = Zero();
-    };
-
-    /**
-     * Runs the Arnoldi loop with(out) implicit restarting. For each iteration:
-     *   - Runs an Arnoldi step.
-     *   - Computes the eigensystem of the Hessenberg matrix.
-     *   - Performs implicit restarting.
-     */
-    void operator()(const Field& v0, int _maxIter, int _Nm, int _Nk, int _Nstop, bool doubleOrthog = false) {
-      MaxIter = _maxIter;
-      Nm = _Nm; Nk = _Nk;
-      Nstop = _Nstop;
-      
-      ssq = norm2(v0);
-      RealD approxLambdaMax = approxMaxEval(v0);
-      rtol = Tolerance * approxLambdaMax;
-
-      ComplexComparator compareComplex (ritzFilter);
-      std::cout << GridLogMessage << "Comparing Ritz values with: " << ritzFilter << std::endl;
-
-      int start = 1;
-      Field startVec = v0;
-      littleEvecs = Eigen::MatrixXcd::Zero(Nm, Nm);
-      for (int i = 0; i < MaxIter; i++) {
-        std::cout << GridLogMessage << "Restart Iteration " << i << std::endl;
-
-        // Perform Arnoldi steps to compute Krylov basis and Rayleigh quotient (Hess)
-        arnoldiIteration(startVec, Nm, start, doubleOrthog);
-        startVec = f;
-
-        // compute eigensystem and sort evals
-        // compute_eigensystem();
-        compute_eigensystem(Hess);
-        std::cout << GridLogMessage << "Eigenvalues after Arnoldi step: " << std::endl << evals << std::endl;
-
-        std::sort(evals.begin(), evals.end(), compareComplex);
-        std::cout << GridLogMessage << "Ritz values after sorting (first Nk preserved): " << std::endl << evals << std::endl;
-        // SU(N)::tepidConfiguration
-
-        // Implicit restart to de-weight unwanted eigenvalues
-        implicitRestart(_Nm, _Nk);      // probably can delete _Nm and _Nk from function args
-        start = Nk;
-
-        // check convergence and return if needed.
-        int Nconv = converged();
-        std::cout << GridLogMessage << "Number of evecs converged: " << Nconv << std::endl;
-        if (Nconv >= Nstop || i == MaxIter - 1) {
-          std::cout << GridLogMessage << "Converged with " << Nconv << " / " << Nstop << " eigenvectors on iteration " 
-                        << i << "." << std::endl;
-          basisRotate(evecs, Qt, 0, Nk, 0, Nk, Nm);
-          std::cout << GridLogMessage << "Eigenvalues [first " << Nconv << " converged]: " << std::endl << evals << std::endl;
-          return;
-        }
-      }      
-    }
-
-    /**
-     * Approximates the maximum eigenvalue of Linop.Op to normalize the residual and test for convergence. 
-     * 
-     * Parameters
-     * ----------
-     * Field& v0
-     *  Source field to start with. Must have non-zero norm.
-     * int MAX_ITER (default = 50)
-     *  Maximum number of iterations for power approximation. 
-     * 
-     * Returns
-     * -------
-     * RealD lamApprox
-     *  Approximation of largest eigenvalue. 
-     */
-    RealD approxMaxEval(const Field& v0, int MAX_ITER = 50) {
-      assert (norm2(v0) > 1e-8);                        // must have relatively large source norm to start
-      RealD lamApprox = 0.0;
-      RealD denom = 1.0; RealD num = 1.0;
-      Field v0cp (Grid); Field tmp (Grid);
-      v0cp = v0;
-      denom = std::sqrt(norm2(v0cp));
-      for (int i = 0; i < MAX_ITER; i++) {
-        Linop.Op(v0cp, tmp);                               // CAREFUL: do not do Op(tmp, tmp)
-        v0cp = tmp;
-        num = std::sqrt(norm2(v0cp));                      // num = |A^{n+1} v0|
-        lamApprox = num / denom;                           // lam = |A^{n+1} v0| / |A^n v0|
-        std::cout << GridLogDebug << "Approx for max eval: " << lamApprox << std::endl;
-        denom = num;                                       // denom = |A^{n} v0|
-      }
-      return lamApprox;
-    }
-
-    /**
-     * Constructs the Arnoldi basis for the Krylov space K_n(D, src). (TODO make private)
-     * 
-     * Parameters
-     * ----------
-     * v0 : Field&
-     *  Source to generate Krylov basis. 
-     * Nm : int
-     *  Final size of the basis desired. If the basis becomes complete before a basis of size Nm is constructed 
-     *  (determined by relative tolerance Tolerance), stops iteration there. 
-     * doubleOrthog : bool (default = false)
-     *  Whether to double orthogonalize the basis (for numerical cancellations) or not. 
-     * start        : int (default = 0)
-     *  If non-zero, assumes part of the Arnoldi basis has already been constructed. 
-     */
-    void arnoldiIteration(const Field& v0, int Nm, int start = 1, bool doubleOrthog = false)
-    {
-
-      ComplexD coeff;
-      Field w (Grid);           // A acting on last Krylov vector. 
-
-      if (start == 1) {       // initialize everything that we need.
-        RealD v0Norm = 1 / std::sqrt(ssq);
-        basis.push_back(v0Norm * v0);                // normalized source
-
-        Hess = Eigen::MatrixXcd::Zero(Nm, Nm);
-        f = Zero();
-      } else {
-        assert( start == basis.size() );      // should be starting at the end of basis (start = Nk)
-        Eigen::MatrixXcd HessCp = Hess;
-        Hess = Eigen::MatrixXcd::Zero(Nm, Nm);
-        Hess(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk)) = HessCp;
-      }
-
-      // Construct next Arnoldi vector by normalizing w_i = Dv_i - \sum_j v_j h_{ji}
-      for (int i = start - 1; i < Nm; i++) {
-
-        Linop.Op(basis.back(), w);
-        for (int j = 0; j < basis.size(); j++) {
-          coeff = innerProduct(basis[j], w);       // coeff = h_{ij}. Note that since {vi} is ONB it's OK to subtract it off after. 
-          Hess(j, i) = coeff;
-          w -= coeff * basis[j];
-        }
-
-        if (doubleOrthog) {
-          // TODO implement
-        }
-
-        // add w_i to the pile
-        if (i < Nm - 1) {
-          coeff = std::sqrt(norm2(w));
-          Hess(i+1, i) = coeff;
-          basis.push_back(
-            (1.0/coeff) * w
-          );
-        }
-
-        // after iterations, update f and beta_k = ||f||
-        f = w;                                // make sure f is not normalized
-        beta_k = std::sqrt(norm2(f));         // beta_k = ||f_k|| determines convergence.
-      }
-
-      std::cout << GridLogMessage << "|f|^2 after Arnoldi step = " << norm2(f) << std::endl;
-      std::cout << GridLogDebug << "Computed Hessenberg matrix = " << std::endl << Hess << std::endl;
-
-      return;
-    }
-
-    /**
-     * Approximates the eigensystem of the linear operator by computing the eigensystem of 
-     * the Hessenberg matrix. Assumes that the Hessenberg matrix has already been constructed (by 
-     * calling the operator() function).
-     * 
-     * TODO implement in parent class eventually.
-     * 
-     * Parameters
-     * ----------
-     * Eigen::MatrixXcd& S
-     *  Schur matrix (upper triangular) similar to original Rayleigh quotient.
-     */
-    void compute_eigensystem(Eigen::MatrixXcd& S)
-    {
-
-      std::cout << GridLogMessage << "Computing eigenvalues." << std::endl;
-
-      evecs.clear();
-
-      Eigen::ComplexEigenSolver<Eigen::MatrixXcd> es;
-      es.compute(S);
-      evals = es.eigenvalues();
-      littleEvecs = es.eigenvectors();
-
-      // Convert evecs to lattice fields
-      for (int k = 0; k < evals.size(); k++) {
-        Eigen::VectorXcd vec = littleEvecs.col(k);
-        Field tmp (basis[0].Grid());
-        tmp = Zero();
-        for (int j = 0; j < basis.size(); j++) {
-          tmp = tmp + vec[j] * basis[j];
-        }
-        evecs.push_back(tmp);
-      }
-
-      std::cout << GridLogMessage << "Eigenvalues: " << std::endl << evals << std::endl;
-
-    }
-
-    /**
-     * Verifies the factorization DV = V^\dag H + f e^\dag with the last-computed 
-     * V, H, f. 
-     */
-    // RealD verifyFactorization() {
-    //   int k = basis.size();         // number of basis vectors, also the size of H.
-    //   std::vector<Field> factorized (k, Zero());
-    //   Field tmp (FGrid); tmp = Zero();
-    //   for (int i = 0; i < basis.size(); i++) {
-    //     Linop.Op(basis[i], tmp);
-    //   }
-    //   // basisRotate(basis, Q, 0, Nk, 0, Nk, Nm);
-    //   // Linop.Op(, )
-    // }
-
-    /* Getters */
-    Eigen::MatrixXcd    getHessenbergMat()  { return Hess; }
-    Field               getF()              { return f; }
-    std::vector<Field>  getBasis()          { return basis; }
-    Eigen::VectorXcd    getEvals()          { return evals; }
-    std::vector<Field>  getEvecs()          { return evecs; }
-
-    /**
-     * Implements implicit restarting for Arnoldi. Assumes eigenvalues are sorted. 
-     * 
-     * Parameters
-     * ----------
-     * int _Nm
-     *  Size of basis to keep (Hessenberg is MxM).
-     * int Nk
-     *  Number of basis vectors to keep at each restart.
-     */
-    void implicitRestart(int _Nm, int _Nk) {
-      assert ( _Nk <= _Nm );
-      Nm = _Nm; Nk = _Nk;
-      int Np = Nm - Nk;       // keep Nk smallest (or largest, depends on sort function) evecs
-      
-      std::cout << GridLogMessage << "Computing QR Factorizations." << std::endl;
-
-      Eigen::MatrixXcd Q = Eigen::MatrixXcd::Identity(Nm, Nm);
-      Eigen::MatrixXcd Qi (Nm, Nm);
-      Eigen::MatrixXcd R (Nm, Nm);
-
-      for (int i = Nk; i < Nm; i++) {        // keep the first Nk eigenvalues and iterate through the last Np. Should loop Np times
-
-        // Useful debugging output
-        std::cout << GridLogDebug << "Computing QR factorization for i = " << i << std::endl;
-        std::cout << GridLogDebug << "Eval shift = " << evals[i] << std::endl;
-        std::cout << GridLogDebug << "Hess before rotation: " << Hess << std::endl;
-
-        // QR factorize 
-        Eigen::HouseholderQR<Eigen::MatrixXcd> QR (Hess - evals[i] * Eigen::MatrixXcd::Identity(Nm, Nm));
-        Qi = QR.householderQ();
-        Q = Q * Qi;
-        Hess = Qi.adjoint() * Hess * Qi;
-
-        std::cout << GridLogDebug << "Qt up to i = " << Q.transpose() << std::endl;
-
-      }
-
-      std::cout << GridLogDebug << "Hess after all rotations: " << std::endl << Hess << std::endl; 
-
-      // form Arnoldi vector f: f is normal to the basis vectors and its norm \beta is used to determine the Ritz estimate. 
-      std::complex<double> beta = Hess(Nk, Nk-1);
-      std::complex<double> sigma = Q(Nm-1, Nk-1);
-      f = basis[Nk] * beta + f * sigma;
-      RealD betak = std::sqrt(norm2(f));
-      std::cout << GridLogMessage << "|f|^2 after implicit restart = " << norm2(f) << std::endl;
-
-      // Rotate basis by Qt
-      Qt = Q.transpose();
-      basisRotate(basis, Qt, 0, Nk + 1, 0, Nm, Nm);
-
-      // rotate
-      basisRotate(evecs, Qt, 0, Nk + 1, 0, Nm, Nm);
-
-      // Truncate the basis and restart
-      basis = std::vector<Field> (basis.begin(), basis.begin() + Nk);
-      // evecs = std::vector<Field> (evecs.begin(), evecs.begin() + Nk);
-      Hess = Hess(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk));
-
-      std::cout << "evecs size: " << evecs.size() << std::endl;
-
-    }
-  
-    /**
-     * Computes the number of Arnoldi eigenvectors that have converged. An eigenvector s is considered converged 
-     * for a tolerance epsilon if 
-     *    r(s) := |\beta e_m^T s| < epsilon
-     * where beta is the norm of f_{m+1}.
-     * 
-     * Parameters
-     * ----------
-     * 
-     * Returns
-     * -------
-     * int : Number of converged eigenvectors.
-     */
-    int converged() {
-      int Nconv = 0;
-      for (int k = 0; k < evecs.size(); k++) {
-        RealD emTs = std::abs(littleEvecs(Nm - 1, k));           // e_m^T s
-        RealD ritzEstimate = beta_k * emTs;
-        // TODO should be ritzEstimate < Tolerance * lambda_max
-        std::cout << GridLogMessage << "Ritz estimate for evec " << k << " = " << ritzEstimate << std::endl;
-        if (ritzEstimate < rtol) {
-          Nconv++;
-        }
-      }
-      return Nconv;
-    }
-
-};
-    
-NAMESPACE_END(Grid);
-#endif

Grid/algorithms/iterative/BlockKrylovSchur.h

@@ -1,878 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./lib/algorithms/iterative/BlockKrylovSchur.h
-
-Copyright (C) 2015
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_BLOCKED_KRYLOV_SCHUR_H
-#define GRID_BLOCKED_KRYLOV_SCHUR_H
-
-#include <iomanip>
-#include <numeric>
-
-NAMESPACE_BEGIN(Grid);
-
-/**
- * Block (block-Arnoldi) restarted Krylov-Schur eigensolver for
- * general non-Hermitian operators.
- *
- * Algorithm
- * ---------
- * Uses a block Arnoldi factorisation of block size Nblock:
- *
- *   A V_k = V_k H_k + F_k B_k^dag
- *
- * where
- *   V_k   = Nm orthonormal basis vectors (stored flat in basis[])
- *   H_k   = Nm x Nm upper block-Hessenberg Rayleigh quotient
- *   F_k   = Nblock residual vectors (the next block beyond V_k)
- *   B_k   = Nm x Nblock coupling matrix (non-zero only in last Nblock rows)
- *
- * Each block Arnoldi step applies A to each of the Nblock vectors in the
- * current block, orthogonalises against all previous basis vectors, and
- * reduces the residual block to upper-triangular form via Householder QR
- * (implemented here as modified Gram-Schmidt within the block).
- *
- * The restart is a thick restart via the Schur decomposition of H_k:
- *   H_k = Q^dag S Q
- * The leading Nk Schur vectors (chosen by RitzFilter) are retained,
- * the basis and Rayleigh quotient are truncated, and block Arnoldi continues
- * from the (Nk/Nblock)-th block.
- *
- * Parameters
- * ----------
- * Nblock  : block size p
- * Nm      : total Krylov dimension (must be divisible by Nblock)
- * Nk      : total vectors to keep after each restart (must be divisible by Nblock, Nk < Nm)
- * Nstop   : declare convergence when this many eigenpairs have converged
- * MaxIter : maximum number of outer (restart) iterations
- * Tolerance : relative convergence tolerance (||r|| < Tolerance * |lambda_max|)
- */
-template<class Field>
-class BlockKrylovSchur {
-
-protected:
-  //--------------------------------------------------------------------
-  // Types
-  //--------------------------------------------------------------------
-  typedef Eigen::MatrixXcd CMat;
-  typedef Eigen::VectorXcd CVec;
-
-  //--------------------------------------------------------------------
-  // Parameters (set by operator())
-  //--------------------------------------------------------------------
-  int   Nblock;    // block size
-  int   Nm;        // total Krylov dimension (multiple of Nblock)
-  int   Nk;        // total vectors retained after restart (multiple of Nblock)
-  int   Nstop;
-  int   MaxIter;
-  RealD Tolerance;
-
-  //--------------------------------------------------------------------
-  // Internal state
-  //--------------------------------------------------------------------
-  LinearOperatorBase<Field>& Linop;
-  GridBase*                  Grid_;
-  RitzFilter                 ritzFilter;
-
-  // Prefix used in log messages; derived classes override in their constructor
-  // (e.g. "HarmonicBlockKrylovSchur") so the shared code reports the right name.
-  std::string className = "BlockKrylovSchur";
-
-  // Flat storage: basis[s*Nblock + t] is the t-th vector of block s
-  // After construction: basis has Nm entries
-  std::vector<Field> basis;
-
-  // Rayleigh quotient  Nm x Nm
-  CMat H;
-
-  // Residual block: Nblock vectors (the (Nm/Nblock+1)-th block, unnormalised before
-  // QR; normalised and orthogonalised as part of block Arnoldi)
-  std::vector<Field> F;
-
-  // Coupling matrix B: Nm x Nblock.
-  // In exact arithmetic only the last Nblock rows are non-zero:
-  //   B(Nm - Nblock + t, s) = H_{Nm/Nblock+1, Nm/Nblock}(t, s)   (the subdiagonal block)
-  // We keep it as a full matrix for generality after restarts.
-  CMat B;
-
-  RealD beta_k;          // Frobenius norm of the last subdiagonal block
-  RealD rtol;            // absolute tolerance = Tolerance * approxLambdaMax
-
-  // Output
-  CVec               evals;
-  CMat               littleEvecs;   // Nm columns
-  std::vector<RealD> ritzEstimates;
-
-public:
-  std::vector<Field> evecs;
-  bool doEvalCheck  = false;
-  // When true (and Nblock even), only Nblock/2 starting vectors are required.
-  // The remaining Nblock/2 slots are filled by pairing each supplied vector
-  // with its parity-flipped partner (sign negated on odd checkerboard sites).
-  bool useParityFlip = false;
-  // Same pairing behaviour but the partner is produced by gamma5Func(v).
-  // gamma5Func must be set by the caller before operator() is called.
-  // It is a std::function so that BlockKrylovSchur.h does not need to know
-  // about the Grid QCD Gamma class (which is defined at a later include layer).
-  bool useGamma5     = false;
-  std::function<void(const Field&, Field&)> gamma5Func;
-
-  //--------------------------------------------------------------------
-  // Constructor
-  //--------------------------------------------------------------------
-  BlockKrylovSchur(LinearOperatorBase<Field>& _Linop, GridBase* _Grid,
-                     RealD _Tolerance, RitzFilter _rf = EvalReSmall)
-    : Linop(_Linop), Grid_(_Grid), Tolerance(_Tolerance), ritzFilter(_rf),
-      Nblock(-1), Nm(-1), Nk(-1), Nstop(-1), MaxIter(-1),
-      beta_k(0.0), rtol(0.0)
-  {}
-
-  virtual ~BlockKrylovSchur() = default;
-
-  //--------------------------------------------------------------------
-  // Main entry point
-  //--------------------------------------------------------------------
-  /**
-   * Run the blocked Krylov-Schur algorithm.
-   *
-   * Parameters
-   * ----------
-   * v0       : block of Nblock starting vectors (size >= Nblock)
-   * _maxIter : maximum outer (restart) iterations
-   * _Nm      : total Krylov dimension (must be divisible by _Nblock)
-   * _Nk      : total vectors to keep after restart (must be divisible by _Nblock, _Nk < _Nm)
-   * _Nstop   : stop after _Nstop eigenvalues converged
-   * _Nblock  : block size
-   */
-  virtual void operator()(const std::vector<Field>& v0, int _maxIter, int _Nm, int _Nk,
-                          int _Nstop, int _Nblock = 1, bool doubleOrthog = true,
-                          bool doVerify = false)
-  {
-    MaxIter = _maxIter;
-    Nm      = _Nm;
-    Nk      = _Nk;
-    Nstop   = _Nstop;
-    Nblock  = _Nblock;
-
-    {
-      int divisor = 1;
-      if (useParityFlip) divisor *= 2;
-      if (useGamma5)     divisor *= 2;
-      assert(Nblock % divisor == 0 && (int)v0.size() >= Nblock / divisor);
-      std::cout << GridLogMessage << className << ": divisor = " << divisor << std::endl;
-    }
-    assert(Nm % Nblock == 0);
-    assert(Nk % Nblock == 0);
-    assert(Nk < Nm);
-    if (useGamma5) assert(gamma5Func && "useGamma5: gamma5Func must be set");
-    preRun();  // hook: derived classes add parameter assertions here
-
-    int N = Nm;   // total Krylov dimension
-
-    // Approximate largest eigenvalue for tolerance normalisation
-    RealD approxLambdaMax = approxMaxEval(v0[0]);
-    rtol = Tolerance * approxLambdaMax;
-    std::cout << GridLogMessage << className << ": approx max eval = "
-              << approxLambdaMax << ", rtol = " << rtol << std::endl;
-
-    // Initialise
-    H = CMat::Zero(N, N);
-    B = CMat::Zero(N, Nblock);
-
-    int start = 0;
-    std::vector<Field> startBlock = expandStartBlock(v0);
-
-    for (int iter = 0; iter < MaxIter; iter++) {
-      std::cout << GridLogMessage << className << ": restart iteration " << iter << std::endl;
-
-      // ---- Block Arnoldi: extend from block start to block Nm/Nblock ----
-      blockArnoldiIteration(startBlock, Nm/Nblock, start, doubleOrthog);
-
-      // After first full cycle start from block Nk/Nblock
-      start = Nk/Nblock;
-
-      if (doVerify) {
-        std::string lbl = "iter " + std::to_string(iter) + " after Arnoldi";
-        verify(lbl);
-      }
-
-      // ---- Restart rotation: compute Q and the rotated+reordered Rayleigh
-      //      quotient Hnew.  Base class uses the plain Schur decomposition of
-      //      H; derived classes (e.g. HarmonicBlockKrylovSchur) override this
-      //      to target a shift.
-      CMat Q, Hnew;
-      restartRotation(Q, Hnew);
-      CMat Qt = Q.adjoint();
-
-      // Rotate Krylov basis: basis_new[i] = sum_j basis[j] * Qt(j,i)
-      std::vector<Field> basis2;
-      constructUR(basis2, basis, Qt, N);
-      basis = basis2;
-
-      // Update B and H
-      B = Q * B;
-      H = Hnew;
-
-      // ---- Truncate to Nk ----
-      int Nkeep = Nk;
-
-      CMat Htmp = H(Eigen::seqN(0, Nkeep), Eigen::seqN(0, Nkeep));
-      H = CMat::Zero(N, N);
-      H(Eigen::seqN(0, Nkeep), Eigen::seqN(0, Nkeep)) = Htmp;
-
-      std::vector<Field> basisTmp(basis.begin(), basis.begin() + Nkeep);
-      basis = basisTmp;
-
-      CMat Btmp = B(Eigen::seqN(0, Nkeep), Eigen::all);
-      B = CMat::Zero(N, Nblock);
-      B(Eigen::seqN(0, Nkeep), Eigen::all) = Btmp;
-
-      // beta_k = Frobenius norm of the effective coupling
-      beta_k = Btmp.norm();
-      std::cout << GridLogMessage << className << ": beta_k = " << beta_k << std::endl;
-
-      // Restart: the new starting block is F (the residual block from Arnoldi)
-      startBlock = F;
-
-      if (doVerify) {
-        std::string lbl = "iter " + std::to_string(iter) + " after restart+truncation";
-        verify(lbl);
-      }
-
-      // ---- Compute eigensystem of truncated H for convergence check ----
-      CMat Hk = H(Eigen::seqN(0, Nkeep), Eigen::seqN(0, Nkeep));
-      computeEigensystem(Hk, Nkeep);
-
-      int Nconv = converged(Nkeep);
-      std::cout << GridLogMessage << className << ": converged " << Nconv
-                << " / " << Nstop << std::endl;
-
-      if (Nconv >= Nstop || iter == MaxIter - 1) {
-        std::cout << GridLogMessage << className << ": done after " << iter
-                  << " restarts, " << Nconv << " converged." << std::endl;
-        std::cout << GridLogMessage << "Eigenvalues: " << evals << std::endl;
-
-        if (doEvalCheck) {
-          Field w(Grid_);
-          for (int k = 0; k < (int)evecs.size(); k++) {
-            Linop.Op(evecs[k], w);
-            ComplexD eval_est = toStdCmplx(innerProduct(evecs[k], w));
-            w -= eval_est * evecs[k];
-            RealD res = std::sqrt(norm2(w));
-            std::cout << GridLogMessage << className << ": evec[" << k << "]"
-                      << "  eval_reported = " << evals[k]
-                      << "  eval_est = " << eval_est
-                      << "  || A v - eval_est * v || = " << res << std::endl;
-          }
-        }
-
-        return;
-      }
-    }
-  }
-
-  // Accessors
-  std::vector<Field>  getEvecs()         { return evecs;         }
-  CVec                getEvals()         { return evals;         }
-  std::vector<RealD>  getRitzEstimates() { return ritzEstimates; }
-
-  // Negate the field on all odd-parity (checkerboard) sites.
-  // Uses pickCheckerboard/setCheckerboard so the SIMD layout is handled
-  // correctly and the negation runs in parallel.
-  static void parityFlippedField(const Field& v, Field& out) {
-    GridCartesian* cgrid = dynamic_cast<GridCartesian*>(v.Grid());
-    assert(cgrid != nullptr);
-    GridRedBlackCartesian rbgrid(cgrid);
-    Field veven(&rbgrid), vodd(&rbgrid);
-    pickCheckerboard(Even, veven, v);
-    pickCheckerboard(Odd,  vodd,  v);
-    vodd = -vodd;
-    setCheckerboard(out, veven);
-    setCheckerboard(out, vodd);
-  }
-
-  //--------------------------------------------------------------------
-  // Verification: print H and B, check A V = V H + F B^dag explicitly
-  //--------------------------------------------------------------------
-  /**
-   * Checks the block Arnoldi / Krylov-Schur decomposition
-   *
-   *   A V = V H + F B^dag                                      (KS)
-   *
-   * by explicit operator applications.  For each basis vector j:
-   *
-   *   w_j = A basis[j]
-   *
-   * The nBasis × nBasis matrix M of inner products is computed:
-   *
-   *   M[i, j] = <basis[i] | A basis[j]>
-   *
-   * and compared column-by-column against H.  Separately, the nBasis × Nblock
-   * residual coupling matrix R is computed:
-   *
-   *   R[j, t] = <basis[j] | F[t]> * ||F[t]||   (scaled by F-block norms)
-   *
-   * but since F is already normalised, R[j,t] = <basis[j] | F[t]>.
-   *
-   * The KS relation for column j reads:
-   *   w_j = sum_i basis[i] H[i,j]  +  sum_t F[t] B[j,t]*
-   * so the deviation in column j is
-   *   dev_j = w_j - sum_i basis[i] M[i,j]   (should be zero for exact arithmetic)
-   * augmented by the F B^dag term in the last block.
-   *
-   * Prints:
-   *   - H (current Rayleigh quotient, nBasis × nBasis)
-   *   - B (coupling matrix, nBasis × Nblock)
-   *   - M (explicit inner product matrix <V | A V>)
-   *   - max |H[i,j] - M[i,j]|  (should be O(machine epsilon))
-   *   - for each basis column j: || A v_j - V H[:,j] - F B[j,:]^* ||
-   *
-   * Parameters
-   * ----------
-   * label : string printed at the start (e.g. "after restart 2")
-   */
-  void verify(const std::string& label = "")
-  {
-    int nBasis = (int)basis.size();
-    int nF     = (int)F.size();
-
-    if (nBasis == 0) {
-      std::cout << GridLogMessage
-                << className << "::verify [" << label
-                << "]: basis is empty." << std::endl;
-      return;
-    }
-
-    // Full allocated dimension of H (= Nm); may be larger than nBasis after restart+truncation
-    int Nfull = (int)H.rows();
-
-    std::cout << GridLogMessage
-              << "======== " << className << "::verify [" << label << "] ========" << std::endl;
-    std::cout << GridLogMessage
-              << "  nBasis = " << nBasis << "  Nfull = " << Nfull
-              << "  Nblock = " << Nblock << "  nF = " << nF << std::endl;
-
-    // ---- Print H (full Nm x Nm matrix) ----
-    // After restart+truncation nBasis < Nfull; entries outside [0:nBasis,0:nBasis]
-    // should be ~0 — printing the full matrix makes this visible.
-    std::cout << GridLogMessage << "H (" << Nfull << " x " << Nfull << "):" << std::endl;
-    for (int i = 0; i < Nfull; i++) {
-      for (int j = 0; j < Nfull; j++)
-        std::cout << "  " << std::setprecision(4) << std::setw(14) << H(i, j);
-      std::cout << std::endl;
-    }
-
-    // ---- Print B (full Nm x Nblock matrix) ----
-    std::cout << GridLogMessage << "B (" << Nfull << " x " << nF << "):" << std::endl;
-    for (int i = 0; i < Nfull; i++) {
-      for (int t = 0; t < nF; t++)
-        std::cout << "  " << std::setprecision(4) << std::setw(14) << B(i, t);
-      std::cout << std::endl;
-    }
-
-    // ---- Compute M[i,j] = <basis[i] | A basis[j]> ----
-    CMat M = CMat::Zero(nBasis, nBasis);
-    Field w(Grid_);
-    for (int j = 0; j < nBasis; j++) {
-      Linop.Op(basis[j], w);
-      for (int i = 0; i < nBasis; i++)
-        M(i, j) = toStdCmplx(innerProduct(basis[i], w));
-    }
-
-    std::cout << GridLogMessage << "M = <V|AV> (" << nBasis << " x " << nBasis << "):" << std::endl;
-    for (int i = 0; i < nBasis; i++) {
-      for (int j = 0; j < nBasis; j++)
-        std::cout << "  " << std::setprecision(4) << std::setw(14) << M(i, j);
-      std::cout << std::endl;
-    }
-
-    // ---- max |H - M| (for the nBasis x nBasis Rayleigh-quotient block) ----
-    RealD maxHM = 0.0;
-    for (int i = 0; i < nBasis; i++)
-      for (int j = 0; j < nBasis; j++)
-        maxHM = std::max(maxHM, std::abs(H(i,j) - M(i,j)));
-    std::cout << GridLogMessage
-              << "  max |H[i,j] - M[i,j]| (i,j < nBasis) = " << maxHM << std::endl;
-
-    // ---- Check entries of H OUTSIDE the nBasis x nBasis block ----
-    // These live in rows i >= nBasis or columns j >= nBasis.
-    // They must be ~0 after restart+truncation (before the coupling rows are
-    // re-filled by the next blockArnoldiIteration call).
-    // During a full Arnoldi run (nBasis == Nfull) this loop is a no-op.
-    RealD maxOutside = 0.0;
-    for (int i = 0; i < Nfull; i++)
-      for (int j = 0; j < Nfull; j++)
-        if (i >= nBasis || j >= nBasis)
-          maxOutside = std::max(maxOutside, std::abs(H(i,j)));
-    if (Nfull > nBasis)
-      std::cout << GridLogMessage
-                << "  max |H[i,j]| outside [0:" << nBasis << ",0:" << nBasis
-                << "] block = " << maxOutside
-                << " (should be ~0 after restart+truncation)" << std::endl;
-
-    // ---- Check orthonormality of basis ----
-    CMat G = CMat::Zero(nBasis, nBasis);
-    for (int i = 0; i < nBasis; i++)
-      for (int j = 0; j < nBasis; j++)
-        G(i, j) = toStdCmplx(innerProduct(basis[i], basis[j]));
-    CMat Gerr = G - CMat::Identity(nBasis, nBasis);
-    std::cout << GridLogMessage
-              << "  max |<V_i|V_j> - delta_ij| = " << Gerr.cwiseAbs().maxCoeff() << std::endl;
-
-    // ---- Per-column residual: || A v_j - V H[:,j] - F B[j,:]^* || ----
-    // For each basis vector j, compute A v_j then subtract V H[:,j] and F B[j,:]^*
-    RealD maxColDev = 0.0;
-    for (int j = 0; j < nBasis; j++) {
-      Linop.Op(basis[j], w);
-
-      // subtract V H[:,j]
-      for (int i = 0; i < nBasis; i++)
-        w -= basis[i] * H(i, j);
-
-      // subtract F B[j,:]^*  (F[t] * conj(B[j,t]))
-      for (int t = 0; t < nF; t++)
-        w -= F[t] * std::conj(B(j, t));
-
-      RealD dev = std::sqrt(norm2(w));
-      std::cout << GridLogMessage
-                << "  || A v[" << j << "] - V H[:,j] - F B[j,:]* || = " << dev << std::endl;
-      maxColDev = std::max(maxColDev, dev);
-    }
-    std::cout << GridLogMessage
-              << "  max column deviation = " << maxColDev << std::endl;
-
-    // ---- Check F block orthogonality against basis ----
-    if (nF > 0) {
-      RealD maxFV = 0.0;
-      for (int t = 0; t < nF; t++)
-        for (int i = 0; i < nBasis; i++) {
-          RealD ip = std::abs(toStdCmplx(innerProduct(basis[i], F[t])));
-          maxFV = std::max(maxFV, ip);
-        }
-      std::cout << GridLogMessage
-                << "  max |<V_i | F_t>| (should be ~0) = " << maxFV << std::endl;
-    }
-
-    std::cout << GridLogMessage
-              << "======== end verify ========" << std::endl;
-  }
-
-  //--------------------------------------------------------------------
-  // Extension hooks for derived classes
-  //--------------------------------------------------------------------
-
-  /**
-   * Apply the operator (or poly-filtered operator) to the current Arnoldi
-   * block and return the results in W.
-   *
-   * Base implementation: W[t] = Linop.Op(basis[kBase + t]).
-   * Derived classes (e.g. SplitGridBlockKrylovSchur) override this to
-   * apply poly(A) via Grid_split / Grid_unsplit instead.
-   */
-  virtual void applyBlock(std::vector<Field>& W, int kBase)
-  {
-    for (int t = 0; t < Nblock; t++)
-      Linop.Op(basis[kBase + t], W[t]);
-  }
-
-  /**
-   * Called once at the start of operator(), after parameters are set.
-   * Base implementation is a no-op; derived classes override to add
-   * parameter assertions (e.g. Nblock % mrhs == 0).
-   */
-  virtual void preRun() {}
-
-  /**
-   * Compute the restart rotation for the current Rayleigh quotient H.
-   *
-   * Returns Q (unitary) and Hnew = Q H Q^dag reordered so that the wanted
-   * Nk eigenvalues occupy the leading block, ready for truncation.
-   *
-   * Base implementation: the plain Schur decomposition of H, reordered by
-   * ritzFilter.  HarmonicBlockKrylovSchur overrides this to Schur-decompose
-   * the shifted quotient (H - sigma I) so that eigenvalues nearest sigma are
-   * selected.
-   */
-  virtual void restartRotation(CMat& Q, CMat& Hnew)
-  {
-    ComplexSchurDecomposition schur(H, false, ritzFilter);
-    std::cout << GridLogMessage << className << ": Schur decomposed." << std::endl;
-    schur.schurReorder(Nk);
-    std::cout << GridLogMessage << className << ": Schur reordered." << std::endl;
-    Q    = schur.getMatrixQ();
-    Hnew = schur.getMatrixS();
-  }
-
-protected:
-
-  //--------------------------------------------------------------------
-  // Starting-block expansion (parity flip / gamma5 partners)
-  //--------------------------------------------------------------------
-  /**
-   * Expand Nblock/divisor seed vectors into the full Nblock starting block by
-   * pairing each seed with its parity-flipped and/or gamma5 partners.  Order:
-   * {v, flip(v)} then doubled by {., g5(.)} giving {v, flip(v), g5(v),
-   * g5(flip(v))} when both flags are set.  Requires Nblock, Grid_ and the
-   * flags to be set.
-   */
-  std::vector<Field> expandStartBlock(const std::vector<Field>& v0)
-  {
-    int divisor = (useParityFlip ? 2 : 1) * (useGamma5 ? 2 : 1);
-    std::vector<Field> startBlock;
-    startBlock.reserve(Nblock);
-    for (int i = 0; i < Nblock / divisor; i++) {
-      std::vector<Field> group;
-      group.push_back(v0[i]);
-      if (useParityFlip) {
-        int n = (int)group.size();
-        for (int j = 0; j < n; j++) {
-          Field fp(Grid_);
-          parityFlippedField(group[j], fp);
-          group.push_back(std::move(fp));
-        }
-      }
-      if (useGamma5) {
-        int n = (int)group.size();
-        for (int j = 0; j < n; j++) {
-          Field g5v(Grid_);
-          gamma5Func(group[j], g5v);
-          group.push_back(std::move(g5v));
-        }
-      }
-      for (auto& f : group) startBlock.push_back(std::move(f));
-    }
-    return startBlock;
-  }
-
-protected:
-
-  //--------------------------------------------------------------------
-  // Block Arnoldi iteration
-  //--------------------------------------------------------------------
-  /**
-   * Extends the block Arnoldi factorisation from block index 'start' to
-   * block index 'Nm'.
-   *
-   * On entry (start > 0): basis[0..start*Nblock-1] already set,
-   *   H[0..start*Nblock-1, 0..start*Nblock-1] already set,
-   *   B[start*Nblock-1, :] set (coupling from prior residual block).
-   *   startBlock = the normalised residual block F from the previous cycle.
-   *
-   * On entry (start == 0): initialises everything from startBlock.
-   */
-  void blockArnoldiIteration(std::vector<Field>& startBlock, int endBlock,
-                             int startIdx, bool doubleOrthog)
-  {
-    int N = Nm;
-
-    if (startIdx == 0) {
-      basis.clear();
-      F.clear();
-      H = CMat::Zero(N, N);
-      B = CMat::Zero(N, Nblock);
-
-      // Orthonormalise starting block via modified Gram-Schmidt
-      std::vector<Field> V0 = startBlock;
-      blockOrthonormalise(V0);
-      for (auto& v : V0) basis.push_back(v);
-    } else {
-      // Append residual block (startBlock = F_old) to basis.
-      // The truncated KS relation after restart is:
-      //
-      //   A V_k = V_k S_k + F_old B_old^dag          (*)
-      //
-      // where V_k = basis[0:Nkeep], S_k is stored in H[0:Nkeep,0:Nkeep],
-      // B_old = B[0:Nkeep,:], F_old = startBlock.
-      //
-      // Once F_old is appended as basis[Nkeep:Nkeep+Nblock], (*) becomes
-      // a statement about the extended H matrix:
-      //
-      //   H[Nkeep+t, j] = (B_old^dag)[t,j] = conj(B_old[j,t])
-      //                   for t=0..Nblock-1, j=0..Nkeep-1
-      //
-      // These entries are the "restart coupling rows" that connect the new
-      // block to all retained Schur vectors and must be set before Arnoldi
-      // continues, otherwise A V_k = V_k H[:,0:Nkeep] would be missing the
-      // F_old B_old^dag term for those columns.
-
-      int Nkeep = startIdx * Nblock;
-      for (auto& v : startBlock) basis.push_back(v);
-
-      // Fill restart coupling rows into H
-      for (int t = 0; t < Nblock; t++)
-        for (int j = 0; j < Nkeep; j++)
-          H(Nkeep + t, j) = std::conj(B(j, t));
-
-      // Zero out B for the retained columns now that the coupling is in H
-      for (int j = 0; j < Nkeep; j++)
-        for (int t = 0; t < Nblock; t++)
-          B(j, t) = 0.0;
-    }
-
-    // Main block Arnoldi loop
-    for (int k = startIdx; k < endBlock; k++) {
-      blockArnoldiStep(k, doubleOrthog);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  /**
-   * One block Arnoldi step: extends by one block (Nblock vectors).
-   *
-   * Computes block column k of H and the next basis block V_{k+1}.
-   *
-   * Layout of basis (flat):
-   *   basis[j*Nblock + t]  = t-th vector of j-th block,  j = 0..k
-   *
-   * After this call:
-   *   H[i, k*Nblock : (k+1)*Nblock] filled for i = 0..(k+1)*Nblock - 1
-   *   basis[k*Nblock .. (k+1)*Nblock - 1] normalised (already set on entry)
-   *   F = residual block (to become V_{k+1} after this step if k < Nm-1)
-   *
-   * If k < Nm-1, also:
-   *   H[(k+1)*Nblock : (k+2)*Nblock, k*Nblock : (k+1)*Nblock] = subdiag block (from QR of residual)
-   *   basis extended by Nblock (the normalised residual vectors)
-   */
-  void blockArnoldiStep(int k, bool doubleOrthog)
-  {
-    int kBase  = k * Nblock;          // first flat index of current block
-    int prevN  = kBase + Nblock;      // number of basis vectors so far after this step
-    int N      = Nm;
-
-    // W[t] = op(basis[kBase + t])  — dispatches to applyBlock() virtual
-    std::vector<Field> W(Nblock, Field(Grid_));
-    applyBlock(W, kBase);
-
-    // Orthogonalise W against all current basis vectors (full reorthogonalisation)
-    // H[i, kBase + t] = <basis[i] | W[t]>
-    for (int pass = 0; pass < (doubleOrthog ? 2 : 1); pass++) {
-      for (int i = 0; i < prevN; i++) {
-        for (int t = 0; t < Nblock; t++) {
-          ComplexD coeff = innerProduct(basis[i], W[t]);
-          if (pass == 0)
-            H(i, kBase + t) = toStdCmplx(coeff);
-          else
-            H(i, kBase + t) += toStdCmplx(coeff);
-          W[t] -= coeff * basis[i];
-        }
-      }
-    }
-
-    // Store residual block F
-    F = W;
-
-    if (k == Nm/Nblock - 1) {
-      // Last block: compute coupling matrix B for KS decomp.
-      //
-      // blockQR modifies F in-place (F → Q orthonormal) and returns R
-      // such that  W_orig[t] = sum_s F[s] * R[s,t]  (W_orig = F_after * R).
-      //
-      // The KS relation for column j = kBase+t requires the coefficient of F[s]
-      // to be (B†)[s,j] = conj(B[j,s]).  Matching with R[s,t]:
-      //   conj(B[kBase+t, s]) = R[s,t]  →  B[kBase+t, s] = conj(R[s,t])
-      //
-      // Equivalently the last Nblock rows of B are R^H (Hermitian conjugate of R).
-      // Note: for Nblock=1, R is scalar real positive, so this reduces to B = R. ✓
-      CMat R = blockQR(F);   // F is modified in-place to become Q; returns R
-      for (int t = 0; t < Nblock; t++)
-        for (int s = 0; s < Nblock; s++)
-          B(kBase + t, s) = std::conj(R(s, t));    // B_block = R^H
-
-      beta_k = R.norm();
-      return;
-    }
-
-    // Not last block: QR-decompose residual to get V_{k+1}
-    CMat R = blockQR(F);   // F orthonormalised in-place, R is upper triangular
-
-    // Subdiagonal block of H: H[(k+1)*Nblock : (k+2)*Nblock, kBase : kBase+Nblock] = R
-    int nextBase = (k + 1) * Nblock;
-    for (int i = 0; i < Nblock; i++)
-      for (int j = 0; j < Nblock; j++)
-        H(nextBase + i, kBase + j) = R(i, j);
-
-    // Append normalised residual block to basis
-    for (int t = 0; t < Nblock; t++)
-      basis.push_back(F[t]);
-  }
-
-  //--------------------------------------------------------------------
-  // Block QR via modified Gram-Schmidt within the block
-  //--------------------------------------------------------------------
-  /**
-   * Given a block of Nblock vectors W (not necessarily orthonormal),
-   * orthonormalises them in-place and returns the upper-triangular R
-   * such that W_in = W_out * R.
-   *
-   * Handles (near-)linear dependence by zeroing vectors below threshold.
-   */
-  CMat blockQR(std::vector<Field>& W)
-  {
-    CMat R = CMat::Zero(Nblock, Nblock);
-    const RealD deflThresh = 1e-14;
-
-    for (int j = 0; j < Nblock; j++) {
-      // Orthogonalise W[j] against W[0..j-1]
-      for (int i = 0; i < j; i++) {
-        ComplexD coeff = innerProduct(W[i], W[j]);
-        R(i, j) = toStdCmplx(coeff);
-        W[j] -= coeff * W[i];
-      }
-      RealD nrm = std::sqrt(norm2(W[j]));
-      R(j, j)   = nrm;
-      if (nrm > deflThresh) {
-        W[j] *= (1.0 / nrm);
-      } else {
-        // deflation: zero this vector
-        W[j] = Zero();
-        std::cout << GridLogMessage
-                  << className << ": deflation at block column " << j
-                  << " (norm = " << nrm << ")" << std::endl;
-      }
-    }
-    return R;
-  }
-
-  //--------------------------------------------------------------------
-  // Orthonormalise a block against itself (no prior basis)
-  //--------------------------------------------------------------------
-  void blockOrthonormalise(std::vector<Field>& V)
-  {
-    for (int j = 0; j < (int)V.size(); j++) {
-      for (int i = 0; i < j; i++) {
-        ComplexD c = innerProduct(V[i], V[j]);
-        V[j] -= c * V[i];
-      }
-      RealD nrm = std::sqrt(norm2(V[j]));
-      assert(nrm > 1e-14);
-      V[j] *= (1.0 / nrm);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  // Basis rotation: UR[i] = sum_j U[j] * R(j, i)
-  //--------------------------------------------------------------------
-  void constructUR(std::vector<Field>& UR, std::vector<Field>& U,
-                   CMat& R, int N)
-  {
-    UR.clear();
-    Field tmp(Grid_);
-    for (int i = 0; i < N; i++) {
-      tmp = Zero();
-      for (int j = 0; j < N; j++)
-        tmp += U[j] * R(j, i);
-      UR.push_back(tmp);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  // Eigensystem of the truncated Rayleigh quotient
-  //--------------------------------------------------------------------
-  void computeEigensystem(CMat& Hk, int Nkeep)
-  {
-    Eigen::ComplexEigenSolver<CMat> es;
-    es.compute(Hk);
-
-    // Sort to match schurReorder ordering.
-    int n = es.eigenvalues().size();
-    ComplexComparator cComp(ritzFilter);
-    std::vector<int> idx(n);
-    std::iota(idx.begin(), idx.end(), 0);
-    std::sort(idx.begin(), idx.end(), [&](int a, int b){
-      return cComp(toStdCmplx(es.eigenvalues()(a)), toStdCmplx(es.eigenvalues()(b)));
-    });
-
-    evals.resize(n);
-    littleEvecs.resize(n, n);
-    for (int k = 0; k < n; k++) {
-      evals(k)           = es.eigenvalues()(idx[k]);
-      littleEvecs.col(k) = es.eigenvectors().col(idx[k]);
-    }
-
-    evecs.clear();
-    for (int k = 0; k < Nkeep; k++) {
-      CVec vec = littleEvecs.col(k);
-      Field tmp(Grid_);
-      tmp = Zero();
-      for (int j = 0; j < (int)basis.size() && j < Nkeep; j++)
-        tmp += vec[j] * basis[j];
-      evecs.push_back(tmp);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  // Convergence check
-  //--------------------------------------------------------------------
-  /**
-   * An eigenpair (lambda, y) is converged if the Ritz estimate
-   *   r = || B^dag y ||
-   * satisfies r < rtol.  Here B is the (Nkeep x Nblock) coupling matrix
-   * and y is the little eigenvector (Nkeep-vector) of H.
-   */
-  int converged(int Nkeep)
-  {
-    ritzEstimates.clear();
-    int Nconv = 0;
-
-    CMat Bk = B(Eigen::seqN(0, Nkeep), Eigen::all);  // Nkeep x Nblock
-
-    for (int k = 0; k < Nkeep; k++) {
-      CVec yk   = littleEvecs.col(k);                // Nkeep-vector
-      CVec Bty  = Bk.adjoint() * yk;                // Nblock-vector
-      RealD res = Bty.norm();
-      ritzEstimates.push_back(res);
-      std::cout << GridLogMessage << className << ": Ritz estimate[" << k
-                << "] = " << res << "  eval = " << evals[k] << std::endl;
-      if (res < rtol) Nconv++;
-      else break;
-    }
-    return Nconv;
-  }
-
-  //--------------------------------------------------------------------
-  // Approximate maximum eigenvalue via power iteration
-  //--------------------------------------------------------------------
-  RealD approxMaxEval(const Field& v0, int MAX_ITER = 50)
-  {
-    assert(norm2(v0) > 1e-8);
-    RealD lam = 0.0, denom = std::sqrt(norm2(v0));
-    Field vcur(Grid_), vtmp(Grid_);
-    vcur = v0;
-    for (int i = 0; i < MAX_ITER; i++) {
-      Linop.Op(vcur, vtmp);
-      vcur  = vtmp;
-      RealD num = std::sqrt(norm2(vcur));
-      lam   = num / denom;
-      denom = num;
-    }
-    return lam;
-  }
-
-};
-
-NAMESPACE_END(Grid);
-
-#endif  // GRID_BLOCKED_KRYLOV_SCHUR_H

Grid/algorithms/iterative/ConjugateGradient.h

@@ -212,13 +212,17 @@ public:
 		  << "\tTarget " << Tolerance << std::endl;
 
 	//	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-	std::cout << GridLogMessage << "\tSolver Elapsed    " << SolverTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "Time breakdown "<<std::endl;
-	std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
-	std::cout << GridLogPerformance << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tConstruct  " << ConstructTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tNorm       " << NormTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tAssign     " << AssignTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tSolver     " << SolverTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "Solver breakdown "<<std::endl;
+	std::cout << GridLogMessage << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
+	std::cout << GridLogMessage << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
 
 	std::cout << GridLogDebug << "\tMobius flop rate " << DwfFlops/ usecs<< " Gflops " <<std::endl;
 

Grid/algorithms/iterative/ConjugateGradientTimeslice.h

@@ -1,277 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./lib/algorithms/iterative/ConjugateGradientTimeslice.h
-
-Copyright (C) 2015
-
-Author: Azusa Yamaguchi <ayamaguc@staffmail.ed.ac.uk>
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-Author: paboyle <paboyle@ph.ed.ac.uk>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution
-directory
-*************************************************************************************/
-			   /*  END LEGAL */
-#ifndef GRID_CONJUGATE_GRADIENT_TIMESLICE_H
-#define GRID_CONJUGATE_GRADIENT_TIMESLICE_H
-
-NAMESPACE_BEGIN(Grid);
-
-/////////////////////////////////////////////////////////////
-// Base classes for iterative processes based on operators
-// single input vec, single output vec.
-/////////////////////////////////////////////////////////////
-
-/**
- * Simple modification of conjugate gradient that outputs the residual as a function 
- * of time, in order to study the large wavelength behavior of the solver. 
- */
-
-
-template <class Field>
-class ConjugateGradientTimeslice : public OperatorFunction<Field> {
-public:
-
-  using OperatorFunction<Field>::operator();
-  
-  bool ErrorOnNoConverge;  // throw an assert when the CG fails to converge.
-                           // Defaults true.
-  RealD Tolerance;
-  Integer MaxIterations;
-  Integer IterationsToComplete; //Number of iterations the CG took to finish. Filled in upon completion
-  RealD TrueResidual;
-  
-  ConjugateGradientTimeslice(RealD tol, Integer maxit, bool err_on_no_conv = true)
-    : Tolerance(tol),
-      MaxIterations(maxit),
-      ErrorOnNoConverge(err_on_no_conv)
-  {};
-
-  virtual void LogIteration(int k,RealD a,RealD b){
-    //    std::cout << "ConjugageGradient::LogIteration() "<<std::endl;
-  };
-  virtual void LogBegin(void){
-    std::cout << "ConjugageGradient::LogBegin() "<<std::endl;
-  };
-
-    void operator()(LinearOperatorBase<Field> &Linop, const Field &src, Field &psi) {
-
-      this->LogBegin();
-
-      GRID_TRACE("ConjugateGradientTimeslice");
-    GridStopWatch PreambleTimer;
-    GridStopWatch ConstructTimer;
-    GridStopWatch NormTimer;
-    GridStopWatch AssignTimer;
-    PreambleTimer.Start();
-    psi.Checkerboard() = src.Checkerboard();
-
-    conformable(psi, src);
-
-    RealD cp, c, a, d, b, ssq, qq;
-    //RealD b_pred;
-
-    // Was doing copies
-    ConstructTimer.Start();
-    Field p  (src.Grid());
-    Field mmp(src.Grid());
-    Field r  (src.Grid());
-    ConstructTimer.Stop();
-
-    // Initial residual computation & set up
-    NormTimer.Start();
-    ssq = norm2(src);                 // Norm of source vector ||b||^2
-
-    ssqtx = localNorm2(src);          // Norm |b(x, t)|^2 as a field
-    std::vector<RealD> ssqt;          // Norm of source not summed over time slices, ssq(t) = \sum_x |b(x, t)|^2
-    sliceSum(ssqtx, ssqt, Tdir);      // TODO make sure Tdir is globally defined
-
-    RealD guess = norm2(psi);         // Norm of initial guess ||psi||^2
-    NormTimer.Stop();
-    assert(std::isnan(guess) == 0);
-    AssignTimer.Start();
-    if ( guess == 0.0 ) {
-      r = src;
-      p = r;
-      a = ssq;
-    } else { 
-      Linop.HermOpAndNorm(psi, mmp, d, b);        // 
-      r = src - mmp;      // Initial residual r0 = b - A guess
-      p = r;              // initial conj vector p0 = r0
-      a = norm2(p);
-    }
-    cp = a;
-    AssignTimer.Stop();
-
-    // Handle trivial case of zero src
-    if (ssq == 0.){
-      psi = Zero();
-      IterationsToComplete = 1;
-      TrueResidual = 0.;
-      return;
-    }
-
-    std::cout << GridLogIterative << std::setprecision(8) << "ConjugateGradient: guess " << guess << std::endl;
-    std::cout << GridLogIterative << std::setprecision(8) << "ConjugateGradient:   src " << ssq << std::endl;
-    std::cout << GridLogIterative << std::setprecision(8) << "ConjugateGradient:    mp " << d << std::endl;
-    std::cout << GridLogIterative << std::setprecision(8) << "ConjugateGradient:   mmp " << b << std::endl;
-    std::cout << GridLogIterative << std::setprecision(8) << "ConjugateGradient:  cp,r " << cp << std::endl;
-    std::cout << GridLogIterative << std::setprecision(8) << "ConjugateGradient:     p " << a << std::endl;
-
-    RealD rsq = Tolerance * Tolerance * ssq;
-
-    // Check if guess is really REALLY good :)
-    if (cp <= rsq) {
-      TrueResidual = std::sqrt(a/ssq);
-      std::cout << GridLogMessage << "ConjugateGradient guess is converged already " << std::endl;
-      IterationsToComplete = 0;	
-      return;
-    }
-
-    std::cout << GridLogIterative << std::setprecision(8)
-              << "ConjugateGradient: k=0 residual " << cp << " target " << rsq << std::endl;
-
-    PreambleTimer.Stop();
-    GridStopWatch LinalgTimer;
-    GridStopWatch InnerTimer;
-    GridStopWatch AxpyNormTimer;
-    GridStopWatch LinearCombTimer;
-    GridStopWatch MatrixTimer;
-    GridStopWatch SolverTimer;
-
-    RealD usecs = -usecond();
-    SolverTimer.Start();
-    int k;
-    for (k = 1; k <= MaxIterations; k++) {
-
-      GridStopWatch IterationTimer;
-      IterationTimer.Start();
-      c = cp;
-
-      MatrixTimer.Start();
-      Linop.HermOp(p, mmp);         // Computes mmp = Ap
-      MatrixTimer.Stop();
-
-      LinalgTimer.Start();
-
-      InnerTimer.Start();
-      ComplexD dc  = innerProduct(p,mmp);         // p^\dagger A p
-      InnerTimer.Stop();
-      d = dc.real();
-      a = c / d;
-
-      // What is axpy? Some accelerator or something? Check Lattice_arith.h
-      AxpyNormTimer.Start();
-
-      // axpy_norm computes ax+by for vectors x and y compatible with a GPU. Here b is set to 1 (see the function in Lattice_reduction.h). 
-      // The first argument passes r by reference, so it stores r --> -a * Ap + 1 * r, i.e. it performs an update on 
-      // r_k --> r_{k+1} = r_k - \alpha_k A p_k. The function returns the norm squared of the first variable, i.e. ||r_{k+1}||^2.
-      cp = axpy_norm(r, -a, mmp, r);
-      AxpyNormTimer.Stop();
-      b = cp / c;
-
-      LinearCombTimer.Start();
-      {
-        autoView( psi_v , psi, AcceleratorWrite);
-        autoView( p_v   , p,   AcceleratorWrite);
-        autoView( r_v   , r,   AcceleratorWrite);
-        accelerator_for(ss,p_v.size(), Field::vector_object::Nsimd(),{
-            coalescedWrite(psi_v[ss], a      *  p_v(ss) + psi_v(ss));
-            coalescedWrite(p_v[ss]  , b      *  p_v(ss) + r_v  (ss));
-        });
-      }
-      LinearCombTimer.Stop();
-      LinalgTimer.Stop();
-      LogIteration(k,a,b);
-
-      IterationTimer.Stop();
-      if ( (k % 500) == 0 ) {
-        std::cout << GridLogMessage << "ConjugateGradient: Iteration " << k
-                << " residual " << sqrt(cp/ssq) << " target " << Tolerance << std::endl;
-      } else { 
-        std::cout << GridLogIterative << "ConjugateGradient: Iteration " << k
-                << " residual " << sqrt(cp/ssq) << " target " << Tolerance << " took " << IterationTimer.Elapsed() << std::endl;
-      }
-
-      // Stopping condition
-      if (cp <= rsq) {
-        usecs +=usecond();
-        SolverTimer.Stop();
-        Linop.HermOpAndNorm(psi, mmp, d, qq);
-        p = mmp - src;
-        GridBase *grid = src.Grid();
-        RealD DwfFlops = (1452. )*grid->gSites()*4*k
-   	               + (8+4+8+4+4)*12*grid->gSites()*k; // CG linear algebra
-        RealD srcnorm = std::sqrt(norm2(src));
-        RealD resnorm = std::sqrt(norm2(p));
-        RealD true_residual = resnorm / srcnorm;
-        std::cout << GridLogMessage << "ConjugateGradient Converged on iteration " << k 
-          << "\tComputed residual " << std::sqrt(cp / ssq)
-          << "\tTrue residual " << true_residual
-          << "\tTarget " << Tolerance << std::endl;
-
-        // GridLogMessage logs the message to the terminal output; GridLogPerformance probably writes to a log file?
-        //	std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-        std::cout << GridLogMessage << "\tSolver Elapsed    " << SolverTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "Time breakdown "<<std::endl;
-        std::cout << GridLogPerformance << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
-        std::cout << GridLogPerformance << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
-
-        std::cout << GridLogDebug << "\tMobius flop rate " << DwfFlops/ usecs<< " Gflops " <<std::endl;
-
-        if (ErrorOnNoConverge) assert(true_residual / Tolerance < 10000.0);
-
-        IterationsToComplete = k;	
-        TrueResidual = true_residual;
-
-        return;
-      }
-    }
-    // Failed. Calculate true residual before giving up                                                         
-    // Linop.HermOpAndNorm(psi, mmp, d, qq);
-    //    p = mmp - src;
-    //TrueResidual = sqrt(norm2(p)/ssq);
-    //    TrueResidual = 1;
-
-    std::cout << GridLogMessage << "ConjugateGradient did NOT converge "<<k<<" / "<< MaxIterations
-    	      <<" residual "<< std::sqrt(cp / ssq)<< std::endl;
-    SolverTimer.Stop();
-    std::cout << GridLogMessage << "\tPreamble   " << PreambleTimer.Elapsed() <<std::endl;
-    std::cout << GridLogMessage << "\tConstruct  " << ConstructTimer.Elapsed() <<std::endl;
-    std::cout << GridLogMessage << "\tNorm       " << NormTimer.Elapsed() <<std::endl;
-    std::cout << GridLogMessage << "\tAssign     " << AssignTimer.Elapsed() <<std::endl;
-    std::cout << GridLogMessage << "\tSolver     " << SolverTimer.Elapsed() <<std::endl;
-    std::cout << GridLogMessage << "Solver breakdown "<<std::endl;
-    std::cout << GridLogMessage << "\tMatrix     " << MatrixTimer.Elapsed() <<std::endl;
-    std::cout << GridLogMessage<< "\tLinalg     " << LinalgTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\t\tInner      " << InnerTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\t\tAxpyNorm   " << AxpyNormTimer.Elapsed() <<std::endl;
-    std::cout << GridLogPerformance << "\t\tLinearComb " << LinearCombTimer.Elapsed() <<std::endl;
-
-    if (ErrorOnNoConverge) assert(0);
-    IterationsToComplete = k;
-
-  }
-};
-
-NAMESPACE_END(Grid);
-#endif

Grid/algorithms/iterative/Gamma5ScalarLanczos.h

@@ -1,230 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    γ5-Scalar Lanczos algorithm for γ5-Hermitian operators (Wilson Dirac).
-
-    Single-vector Lanczos using the γ5-inner product (u,v)_{γ5} = u†γ5v.
-    Builds the Krylov space {q, D_W q, D_W² q, ...} and projects D_W onto it,
-    producing a real tridiagonal matrix T_m whose eigenvalues approximate those
-    of D_W.  Complex conjugate pairs of D_W eigenvalues appear as complex
-    eigenvalues of the real non-symmetric T_m.
-
-    Recurrence (G_k = sign(q_k†γ5 q_k) = ±1):
-        r_k      = D_W q_k − α_k q_k − β_sup[k-1] q_{k-1}
-        β_sub[k] = √|r_k†γ5 r_k|           (lower subdiag T_{k+1,k})
-        β_sup[k] = β_sub[k] G_{k+1}/G_k    (upper superdiag T_{k,k+1})
-        q_{k+1}  = r_k / β_sub[k]
-
-    T_m is real but not symmetric in general; eigenvalues via EigenSolver.
-
-*************************************************************************************/
-#ifndef GRID_GAMMA5_SCALAR_LANCZOS_H
-#define GRID_GAMMA5_SCALAR_LANCZOS_H
-
-#include <functional>
-#include <numeric>
-#include <iomanip>
-
-NAMESPACE_BEGIN(Grid);
-
-template<class Field>
-class Gamma5ScalarLanczos {
-public:
-  using Gamma5Func = std::function<void(const Field&, Field&)>;
-
-private:
-  typedef Eigen::MatrixXcd CMat;
-  typedef Eigen::VectorXcd CVec;
-  typedef Eigen::MatrixXd  RMat;
-
-  LinearOperatorBase<Field>& Linop;
-  GridBase*                  Grid_;
-  Gamma5Func                 applyGamma5;
-  RealD                      Tolerance;
-  int                        nSteps;
-
-  std::vector<Field>  basis;
-  std::vector<RealD>  alpha_;    // diagonal of T_m
-  std::vector<RealD>  beta_sub_; // lower subdiagonal: T_{k+1,k}
-  std::vector<RealD>  beta_sup_; // upper superdiagonal: T_{k,k+1}
-  std::vector<int>    gsign_;    // G_k = ±1
-
-  CVec               evals_;
-  std::vector<Field> evecs_;
-  std::vector<RealD> residuals_;
-
-public:
-  bool doEvalCheck = false;
-
-  Gamma5ScalarLanczos(LinearOperatorBase<Field>& op, GridBase* grid,
-                      Gamma5Func g5, RealD tol = 1e-8)
-    : Linop(op), Grid_(grid), applyGamma5(g5), Tolerance(tol), nSteps(0)
-  {}
-
-  CVec               getEvals()     { return evals_;     }
-  std::vector<Field> getEvecs()     { return evecs_;     }
-  std::vector<RealD> getResiduals() { return residuals_; }
-
-  // v0       : starting vector
-  // maxSteps : maximum Lanczos steps
-  // Nstop    : target Ritz pairs to return (0 = all)
-  // reorthog : full γ5-reorthogonalisation at each step
-  // filter   : eigenvalue selection criterion
-  void operator()(const Field& v0, int maxSteps, int Nstop,
-                  bool reorthog = false, RitzFilter filter = EvalImNormSmall)
-  {
-    basis.clear(); alpha_.clear(); beta_sub_.clear(); beta_sup_.clear(); gsign_.clear();
-    nSteps = 0;
-
-    // Normalise starting vector in γ5-norm: q_0 = v0 / √|v0†γ5 v0|
-    Field q(Grid_); q = v0;
-    Field g5q(Grid_);
-    applyGamma5(q, g5q);
-    RealD omega = std::real(toStdCmplx(innerProduct(g5q, q)));
-    RealD beta0 = std::sqrt(std::abs(omega));
-    assert(beta0 > 1e-14 && "starting vector has zero γ5-norm");
-    q *= (1.0 / beta0);
-    gsign_.push_back(omega >= 0 ? +1 : -1);
-    basis.push_back(q);
-
-    std::cout << GridLogMessage
-              << "Gamma5ScalarLanczos: start  (v0†γ5 v0)=" << omega
-              << "  G[0]=" << gsign_[0] << std::endl;
-
-    for (int k = 0; k < maxSteps; k++) {
-      // ── Apply operator ─────────────────────────────────────────────────
-      Field p(Grid_);
-      Linop.Op(basis[k], p);
-
-      // ── α_k = (q_k†γ5 D_W q_k) / G_k  (real) ─────────────────────────
-      applyGamma5(basis[k], g5q);
-      RealD a = std::real(toStdCmplx(innerProduct(g5q, p))) / RealD(gsign_[k]);
-      alpha_.push_back(a);
-
-      // ── r = D_W q_k − α_k q_k − β_sup[k-1] q_{k-1} ───────────────────
-      Field r(Grid_); r = p;
-      r -= basis[k] * a;
-      if (k > 0)
-        r -= basis[k-1] * beta_sup_[k-1];
-
-      // ── Optional γ5-reorthogonalisation ────────────────────────────────
-      if (reorthog) {
-        for (int i = 0; i <= k; i++) {
-          applyGamma5(basis[i], g5q);
-          ComplexD c = toStdCmplx(innerProduct(g5q, r)) / RealD(gsign_[i]);
-          r -= basis[i] * c;
-        }
-      }
-
-      // ── β_sub[k] = √|r†γ5 r| ───────────────────────────────────────────
-      applyGamma5(r, g5q);
-      omega = std::real(toStdCmplx(innerProduct(g5q, r)));
-      RealD beta = std::sqrt(std::abs(omega));
-
-      std::cout << GridLogMessage
-                << "Gamma5ScalarLanczos: k=" << k
-                << "  alpha=" << a
-                << "  beta=" << beta
-                << "  G[k]=" << gsign_[k]
-                << "  (r†γ5r)=" << omega << std::endl;
-
-      beta_sub_.push_back(beta);
-
-      if (beta < Tolerance) {
-        std::cout << GridLogMessage
-                  << "Gamma5ScalarLanczos: invariant subspace at step " << k << std::endl;
-        nSteps = k + 1;
-        break;
-      }
-
-      int G_next = (omega >= 0) ? +1 : -1;
-      gsign_.push_back(G_next);
-      // β_sup[k] = T_{k,k+1} = β_sub[k] * G_{k+1} / G_k
-      beta_sup_.push_back(beta * RealD(G_next) / RealD(gsign_[k]));
-
-      Field qnew(Grid_); qnew = r; qnew *= (1.0 / beta);
-      basis.push_back(qnew);
-      nSteps = k + 1;
-    }
-
-    if (nSteps == 0) return;
-    computeRitzPairs(nSteps, Nstop, filter);
-  }
-
-private:
-  void computeRitzPairs(int m, int Nstop, RitzFilter filter)
-  {
-    // Assemble real tridiagonal T_m
-    RMat Tm = RMat::Zero(m, m);
-    for (int k = 0; k < m; k++) {
-      Tm(k, k) = alpha_[k];
-      if (k + 1 < m) {
-        Tm(k+1, k) = beta_sub_[k];
-        Tm(k,   k+1) = beta_sup_[k];
-      }
-    }
-
-    std::cout << GridLogMessage
-              << "Gamma5ScalarLanczos: T_m (" << m << "×" << m << "):" << std::endl;
-    for (int i = 0; i < m; i++) {
-      for (int j = 0; j < m; j++)
-        std::cout << "  " << std::setprecision(8) << std::setw(16) << Tm(i,j);
-      std::cout << std::endl;
-    }
-
-    Eigen::EigenSolver<RMat> es(Tm);
-    CVec lambdas = es.eigenvalues();
-    CMat Y       = es.eigenvectors();
-
-    // Sort by filter criterion
-    ComplexComparator cComp(filter);
-    std::vector<int> idx(m);
-    std::iota(idx.begin(), idx.end(), 0);
-    std::sort(idx.begin(), idx.end(), [&](int a, int b){
-      return cComp(toStdCmplx(lambdas(a)), toStdCmplx(lambdas(b)));
-    });
-
-    int Nout = (Nstop > 0 && Nstop < m) ? Nstop : m;
-    evals_.resize(Nout);
-    evecs_.clear();   evecs_.reserve(Nout);
-    residuals_.clear(); residuals_.reserve(Nout);
-
-    // Trailing β for residual estimate: β_sub[m-1] * |y_j[m-1]|
-    RealD beta_trail = (m - 1 < (int)beta_sub_.size()) ? beta_sub_[m-1] : 0.0;
-
-    for (int ji = 0; ji < Nout; ji++) {
-      int  j  = idx[ji];
-      evals_(ji) = lambdas(j);
-      CVec yj = Y.col(j);
-
-      // Ritz vector: u_j = Σ_k q_k y_j[k]
-      Field uj(Grid_); uj = Zero();
-      for (int k = 0; k < m && k < (int)basis.size(); k++)
-        uj += basis[k] * toStdCmplx(yj(k));
-      evecs_.push_back(uj);
-
-      RealD res = beta_trail * std::abs(toStdCmplx(yj(m-1)));
-      if (!std::isfinite(res)) res = std::numeric_limits<RealD>::infinity();
-      residuals_.push_back(res);
-
-      std::cout << GridLogMessage
-                << "  Ritz[" << std::setw(3) << ji << "]"
-                << "  lambda=" << evals_(ji)
-                << "  |res|=" << res;
-
-      if (doEvalCheck) {
-        Field w(Grid_);
-        Linop.Op(uj, w);
-        w -= uj * evals_(ji);
-        RealD actual = std::sqrt(norm2(w));
-        std::cout << "  ||D_W u - λu||=" << actual;
-      }
-      std::cout << std::endl;
-    }
-  }
-};
-
-NAMESPACE_END(Grid);
-
-#endif // GRID_GAMMA5_SCALAR_LANCZOS_H

Grid/algorithms/iterative/HarmonicBlockKrylovSchur.h

@@ -1,719 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./lib/algorithms/iterative/HarmonicBlockKrylovSchur.h
-
-Copyright (C) 2015
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_HARMONIC_BLOCKED_KRYLOV_SCHUR_H
-#define GRID_HARMONIC_BLOCKED_KRYLOV_SCHUR_H
-
-#include <iomanip>
-#include <numeric>
-
-NAMESPACE_BEGIN(Grid);
-
-/**
- * Block shift-targeted Krylov-Schur eigensolver.
- *
- * Algorithm
- * ---------
- * Uses a block Arnoldi factorisation:
- *
- *   A V = V H + F B^dag                                         (1)
- *
- * with V orthonormal (Nm columns), H the Nm×Nm block upper-Hessenberg
- * Rayleigh quotient, F the Nblock residual vectors and B the Nm×Nblock
- * coupling matrix.
- *
- * Thick restart
- * -------------
- * To target eigenvalues near shift σ, the Schur decomposition is computed
- * for the shifted Rayleigh quotient:
- *
- *   (H - σI) = Q^dag S Q                                        (2)
- *
- * Sorting the Schur values of (H - σI) by smallest |S(i,i)| = |λ - σ| and
- * retaining the leading Nk is equivalent to selecting the Ritz values of H
- * closest to σ.  Since Q diagonalises H - σI (and hence H itself), the
- * rotated Rayleigh quotient is exactly upper triangular:
- *
- *   H_new = Q H Q^dag = S + σI  (upper triangular)              (3)
- *
- * Truncation to Nk is therefore exact: the off-diagonal coupling block
- * H_new[Nk:, :Nk] = 0 by triangularity.  The Krylov-Schur relation after
- * restart is exact and block Arnoldi resumes cleanly from F.
- *
- * Convergence
- * -----------
- * Convergence is declared when || B^H y_k || < Tolerance * approxLambdaMax
- * for each Ritz pair (λ_k, y_k) of the truncated H.
- *
- * Parameters
- * ----------
- * shift    : target shift σ (default 0.0); Schur values sorted by |λ - σ|
- * Nblock   : block size p
- * Nm       : total Krylov dimension (must be divisible by Nblock)
- * Nk       : total vectors to retain after each restart (must be divisible by Nblock, Nk < Nm)
- * Nstop    : stop when this many eigenpairs converge
- * MaxIter  : maximum outer (restart) iterations
- * Tolerance: relative convergence tolerance
- *
- * Usage
- * -----
- *   HarmonicBlockKrylovSchur<Field> hbks(LinOp, Grid, tol, shift, EvalNormSmall);
- *   std::vector<Field> v0(Nblock, Field(Grid));
- *   // fill v0 with random starting vectors
- *   hbks(v0, maxIter, Nm, Nk, Nstop, Nblock);
- *   auto evals = hbks.getEvals();
- *   auto evecs = hbks.getEvecs();
- */
-template<class Field>
-class HarmonicBlockKrylovSchur {
-
-protected:
-  typedef Eigen::MatrixXcd CMat;
-  typedef Eigen::VectorXcd CVec;
-
-  //--------------------------------------------------------------------
-  // Parameters
-  //--------------------------------------------------------------------
-  int    Nblock;
-  int    Nm;        // total Krylov dimension (multiple of Nblock)
-  int    Nk;        // total vectors retained after restart (multiple of Nblock)
-  int    Nstop;
-  int    MaxIter;
-  RealD  Tolerance;
-  ComplexD shift;       // target shift σ
-
-  //--------------------------------------------------------------------
-  // Internal state
-  //--------------------------------------------------------------------
-  LinearOperatorBase<Field>& Linop;
-  GridBase*                  Grid_;
-  RitzFilter                 ritzFilter;
-
-  std::vector<Field> basis;   // Nm*Nblock flat basis
-  CMat               H;       // (Nm*Nblock)² block-Hessenberg Rayleigh quotient
-  std::vector<Field> F;       // Nblock residual vectors
-  CMat               B;       // (Nm*Nblock) × Nblock coupling matrix
-  RealD              beta_k;
-  RealD              rtol;
-
-  CVec               evals;
-  CMat               littleEvecs;
-  std::vector<RealD> ritzEstimates;
-
-public:
-  std::vector<Field>  evecs;
-  bool doEvalCheck  = false;
-  bool useParityFlip = false;
-  bool useGamma5     = false;
-  std::function<void(const Field&, Field&)> gamma5Func;
-
-  //--------------------------------------------------------------------
-  // Constructor
-  //--------------------------------------------------------------------
-  HarmonicBlockKrylovSchur(LinearOperatorBase<Field>& _Linop, GridBase* _Grid,
-                             RealD _Tolerance, ComplexD _shift = 0.0,
-                             RitzFilter _rf = EvalNormSmall)
-    : Linop(_Linop), Grid_(_Grid), Tolerance(_Tolerance), shift(_shift),
-      ritzFilter(_rf),
-      Nblock(-1), Nm(-1), Nk(-1), Nstop(-1), MaxIter(-1),
-      beta_k(0.0), rtol(0.0)
-  {}
-
-  virtual ~HarmonicBlockKrylovSchur() = default;
-
-  //--------------------------------------------------------------------
-  // Main entry point
-  //--------------------------------------------------------------------
-  virtual void operator()(const std::vector<Field>& v0, int _maxIter, int _Nm, int _Nk,
-                          int _Nstop, int _Nblock = 1, bool doubleOrthog = true,
-                          bool doVerify = false)
-  {
-    MaxIter = _maxIter;
-    Nm      = _Nm;
-    Nk      = _Nk;
-    Nstop   = _Nstop;
-    Nblock  = _Nblock;
-
-    {
-      int divisor = 1;
-      if (useParityFlip) divisor *= 2;
-      if (useGamma5)     divisor *= 2;
-      assert(Nblock % divisor == 0 && (int)v0.size() >= Nblock / divisor);
-    std::cout << GridLogMessage << "divisor= " << divisor << std::endl;
-
-    }
-    assert(Nm % Nblock == 0);
-    assert(Nk % Nblock == 0);
-    assert(Nk < Nm);
-    if (useGamma5) assert(gamma5Func && "useGamma5: gamma5Func must be set");
-
-    int N = Nm;
-
-    RealD approxLambdaMax = approxMaxEval(v0[0]);
-    rtol = Tolerance * approxLambdaMax;
-    std::cout << GridLogMessage
-              << "HarmonicBlockKrylovSchur: approx max eval = " << approxLambdaMax
-              << ", rtol = " << rtol
-              << ", shift = " << shift << std::endl;
-
-    H = CMat::Zero(N, N);
-    B = CMat::Zero(N, Nblock);
-
-    int start = 0;
-    std::vector<Field> startBlock = expandStartBlock(v0);
-
-    for (int iter = 0; iter < MaxIter; iter++) {
-      std::cout << GridLogMessage
-                << "HarmonicBlockKrylovSchur: restart iteration " << iter << std::endl;
-
-      // ---- Block Arnoldi: extend from block 'start' to block Nm/Nblock ----
-      blockArnoldiIteration(startBlock, Nm/Nblock, start, doubleOrthog);
-      start = Nk/Nblock;
-
-      if (doVerify) {
-        std::string lbl = "iter " + std::to_string(iter) + " after Arnoldi";
-        verify(lbl);
-      }
-
-      // ---- Schur decompose (H - σI) to select Schur vectors closest to σ ----
-      // Sorting the Schur values of (H - σI) by |S(i,i)| = |λ - σ| gives the
-      // Ritz values of H nearest the target shift without any matrix inversion.
-      // Because Q diagonalises (H - σI), it also diagonalises H:
-      //   H_new = Q H Q^dag = S + σI  (upper triangular)
-      // Truncation is therefore exact.
-      CMat Hshift = H - shift * CMat::Identity(N, N);
-      ComplexSchurDecomposition schur(Hshift, false, ritzFilter);
-      schur.schurReorder(Nk);
-
-      std::cout << GridLogMessage
-                << "HarmonicBlockKrylovSchur: Ritz values nearest shift (first Nk):" << std::endl;
-      CMat S = schur.getMatrixS();
-      for (int i = 0; i < Nk; i++)
-        std::cout << GridLogMessage << "  [" << i << "] " << S(i, i) + shift << std::endl;
-
-      CMat Q  = schur.getMatrixQ();
-      CMat Qt = Q.adjoint();
-
-      // ---- Rotate Krylov basis ----
-      std::vector<Field> basis2;
-      constructUR(basis2, basis, Qt, N);
-      basis = basis2;
-
-      // ---- Update H and B ----
-      // H_new = S + σI is upper triangular; off-diagonal block H_new[Nk:,:Nk] = 0
-      H = S + shift * CMat::Identity(N, N);
-      B = Q * B;
-
-      // ---- Truncate to Nk (exact: H upper triangular) ----
-      int Nkeep = Nk;
-
-      CMat Htmp = H(Eigen::seqN(0, Nkeep), Eigen::seqN(0, Nkeep));
-      H = CMat::Zero(N, N);
-      H(Eigen::seqN(0, Nkeep), Eigen::seqN(0, Nkeep)) = Htmp;
-
-      std::vector<Field> basisTmp(basis.begin(), basis.begin() + Nkeep);
-      basis = basisTmp;
-
-      CMat Btmp = B(Eigen::seqN(0, Nkeep), Eigen::all);
-      B = CMat::Zero(N, Nblock);
-      B(Eigen::seqN(0, Nkeep), Eigen::all) = Btmp;
-
-      beta_k = Btmp.norm();
-      std::cout << GridLogMessage
-                << "HarmonicBlockKrylovSchur: beta_k = " << beta_k << std::endl;
-
-      // Restart from F (exact: no discarded-basis correction needed)
-      startBlock = F;
-
-      if (doVerify) {
-        std::string lbl = "iter " + std::to_string(iter) + " after restart+truncation";
-        verify(lbl);
-      }
-
-      // ---- Eigensystem of truncated H for convergence ----
-      CMat Hk = H(Eigen::seqN(0, Nkeep), Eigen::seqN(0, Nkeep));
-      computeEigensystem(Hk, Nkeep);
-
-      int Nconv = converged(Nkeep);
-      std::cout << GridLogMessage
-                << "HarmonicBlockKrylovSchur: converged " << Nconv
-                << " / " << Nstop << std::endl;
-
-      if (Nconv >= Nstop || iter == MaxIter - 1) {
-        std::cout << GridLogMessage
-                  << "HarmonicBlockKrylovSchur: done after " << iter
-                  << " restarts, " << Nconv << " converged." << std::endl;
-        std::cout << GridLogMessage << "Eigenvalues: " << evals.transpose() << std::endl;
-
-        if (doEvalCheck) {
-          Field w(Grid_);
-          for (int k = 0; k < (int)evecs.size(); k++) {
-            Linop.Op(evecs[k], w);
-            ComplexD eval_est = toStdCmplx(innerProduct(evecs[k], w));
-            w -= eval_est * evecs[k];
-            RealD res = std::sqrt(norm2(w));
-            std::cout << GridLogMessage << "HarmonicBlockKrylovSchur: evec[" << k << "]"
-                      << "  eval_reported = " << evals[k]
-                      << "  eval_est = " << eval_est
-                      << "  || A v - eval_est * v || = " << res << std::endl;
-          }
-        }
-
-        return;
-      }
-    }
-  }
-
-  // Accessors
-  std::vector<Field>  getEvecs()         { return evecs;         }
-  CVec                getEvals()         { return evals;         }
-  std::vector<RealD>  getRitzEstimates() { return ritzEstimates; }
-
-  static void parityFlippedField(const Field& v, Field& out) {
-    GridCartesian* cgrid = dynamic_cast<GridCartesian*>(v.Grid());
-    assert(cgrid != nullptr);
-    GridRedBlackCartesian rbgrid(cgrid);
-    Field veven(&rbgrid), vodd(&rbgrid);
-    pickCheckerboard(Even, veven, v);
-    pickCheckerboard(Odd,  vodd,  v);
-    vodd = -vodd;
-    setCheckerboard(out, veven);
-    setCheckerboard(out, vodd);
-  }
-
-  //--------------------------------------------------------------------
-  // Verification: check A V = V H + F B^dag explicitly
-  //--------------------------------------------------------------------
-  /**
-   * Checks the block Arnoldi / Krylov-Schur decomposition
-   *
-   *   A V = V H + F B^dag                                      (KS)
-   *
-   * by explicit operator applications.  H here is the standard Rayleigh
-   * quotient, so the KS relation is the same as for BlockKrylovSchur.
-   *
-   * Prints:
-   *   - H (current Rayleigh quotient, nBasis × nBasis)
-   *   - B (coupling matrix, nBasis × Nblock)
-   *   - M (explicit inner product matrix <V | A V>)
-   *   - max |H[i,j] - M[i,j]|  (should be O(machine epsilon))
-   *   - max |<V_i|V_j> - delta_ij|  (orthonormality check)
-   *   - for each basis column j: || A v_j - V H[:,j] - F B[j,:]^* ||
-   *   - max |<V_i | F_t>|  (F orthogonal to basis)
-   */
-  void verify(const std::string& label = "")
-  {
-    int nBasis = (int)basis.size();
-    int nF     = (int)F.size();
-
-    if (nBasis == 0) {
-      std::cout << GridLogMessage
-                << "HarmonicBlockKrylovSchur::verify [" << label
-                << "]: basis is empty." << std::endl;
-      return;
-    }
-
-    std::cout << GridLogMessage
-              << "======== HarmonicBlockKrylovSchur::verify [" << label << "] ========" << std::endl;
-    std::cout << GridLogMessage
-              << "  nBasis = " << nBasis << "  Nblock = " << Nblock
-              << "  nF = " << nF << std::endl;
-
-    // ---- Print H ----
-    std::cout << GridLogMessage << "H (" << nBasis << " x " << nBasis << "):" << std::endl;
-    for (int i = 0; i < nBasis; i++) {
-      for (int j = 0; j < nBasis; j++)
-        std::cout << "  " << std::setprecision(4) << std::setw(14) << H(i, j);
-      std::cout << std::endl;
-    }
-
-    // ---- Print B ----
-    std::cout << GridLogMessage << "B (" << nBasis << " x " << nF << "):" << std::endl;
-    for (int i = 0; i < nBasis; i++) {
-      for (int t = 0; t < nF; t++)
-        std::cout << "  " << std::setprecision(4) << std::setw(14) << B(i, t);
-      std::cout << std::endl;
-    }
-
-    // ---- Compute M[i,j] = <basis[i] | A basis[j]> ----
-    CMat M = CMat::Zero(nBasis, nBasis);
-    Field w(Grid_);
-    for (int j = 0; j < nBasis; j++) {
-      Linop.Op(basis[j], w);
-      for (int i = 0; i < nBasis; i++)
-        M(i, j) = toStdCmplx(innerProduct(basis[i], w));
-    }
-
-    std::cout << GridLogMessage << "M = <V|AV> (" << nBasis << " x " << nBasis << "):" << std::endl;
-    for (int i = 0; i < nBasis; i++) {
-      for (int j = 0; j < nBasis; j++)
-        std::cout << "  " << std::setprecision(4) << std::setw(14) << M(i, j);
-      std::cout << std::endl;
-    }
-
-    // ---- max |H - M| ----
-    RealD maxHM = 0.0;
-    for (int i = 0; i < nBasis; i++)
-      for (int j = 0; j < nBasis; j++)
-        maxHM = std::max(maxHM, std::abs(H(i,j) - M(i,j)));
-    std::cout << GridLogMessage
-              << "  max |H[i,j] - M[i,j]| = " << maxHM << std::endl;
-
-    // ---- Check orthonormality of basis ----
-    CMat G = CMat::Zero(nBasis, nBasis);
-    for (int i = 0; i < nBasis; i++)
-      for (int j = 0; j < nBasis; j++)
-        G(i, j) = toStdCmplx(innerProduct(basis[i], basis[j]));
-    CMat Gerr = G - CMat::Identity(nBasis, nBasis);
-    std::cout << GridLogMessage
-              << "  max |<V_i|V_j> - delta_ij| = " << Gerr.cwiseAbs().maxCoeff() << std::endl;
-
-    // ---- Per-column residual: || A v_j - V H[:,j] - F B[j,:]^* || ----
-    RealD maxColDev = 0.0;
-    for (int j = 0; j < nBasis; j++) {
-      Linop.Op(basis[j], w);
-
-      // subtract V H[:,j]
-      for (int i = 0; i < nBasis; i++)
-        w -= basis[i] * H(i, j);
-
-      // subtract F B[j,:]^*  (F[t] * conj(B[j,t]))
-      for (int t = 0; t < nF; t++)
-        w -= F[t] * std::conj(B(j, t));
-
-      RealD dev = std::sqrt(norm2(w));
-      std::cout << GridLogMessage
-                << "  || A v[" << j << "] - V H[:,j] - F B[j,:]* || = " << dev << std::endl;
-      maxColDev = std::max(maxColDev, dev);
-    }
-    std::cout << GridLogMessage
-              << "  max column deviation = " << maxColDev << std::endl;
-
-    // ---- Check F block orthogonality against basis ----
-    if (nF > 0) {
-      RealD maxFV = 0.0;
-      for (int t = 0; t < nF; t++)
-        for (int i = 0; i < nBasis; i++) {
-          RealD ip = std::abs(toStdCmplx(innerProduct(basis[i], F[t])));
-          maxFV = std::max(maxFV, ip);
-        }
-      std::cout << GridLogMessage
-                << "  max |<V_i | F_t>| (should be ~0) = " << maxFV << std::endl;
-    }
-
-    std::cout << GridLogMessage
-              << "======== end verify ========" << std::endl;
-  }
-
-protected:
-
-  //--------------------------------------------------------------------
-  // Starting-block expansion (parity flip / gamma5 partners)
-  //--------------------------------------------------------------------
-  // Expands Nblock/divisor seed vectors into the full Nblock starting
-  // block, pairing each seed with its parity-flipped and/or gamma5
-  // partners.  Requires Nblock, Grid_ and the flags to be set.
-  std::vector<Field> expandStartBlock(const std::vector<Field>& v0)
-  {
-    int divisor = (useParityFlip ? 2 : 1) * (useGamma5 ? 2 : 1);
-    std::vector<Field> startBlock;
-    startBlock.reserve(Nblock);
-    for (int i = 0; i < Nblock / divisor; i++) {
-      std::vector<Field> group;
-      group.push_back(v0[i]);
-      if (useParityFlip) {
-        int n = (int)group.size();
-        for (int j = 0; j < n; j++) {
-          Field fp(Grid_);
-          parityFlippedField(group[j], fp);
-          group.push_back(std::move(fp));
-        }
-      }
-      if (useGamma5) {
-        int n = (int)group.size();
-        for (int j = 0; j < n; j++) {
-          Field g5v(Grid_);
-          gamma5Func(group[j], g5v);
-          group.push_back(std::move(g5v));
-        }
-      }
-      for (auto& f : group) startBlock.push_back(std::move(f));
-    }
-    return startBlock;
-  }
-
-  //--------------------------------------------------------------------
-  // Block Arnoldi iteration
-  //--------------------------------------------------------------------
-  void blockArnoldiIteration(std::vector<Field>& startBlock, int endBlock,
-                             int startIdx, bool doubleOrthog)
-  {
-    int N = Nm;
-
-    if (startIdx == 0) {
-      basis.clear();
-      F.clear();
-      H = CMat::Zero(N, N);
-      B = CMat::Zero(N, Nblock);
-
-      std::vector<Field> V0 = startBlock;
-      blockOrthonormalise(V0);
-      for (auto& v : V0) basis.push_back(v);
-    } else {
-      // Append the new starting block to the retained basis.
-      //
-      // The exact truncated KS relation is  A V_k = V_k H_k + F B_k^dag,
-      // so the coupling rows are  H[Nkeep+t, j] = conj(B_k[j,t]).
-      // Since H_new = S + σI is upper triangular, the off-diagonal block
-      // H_new[Nkeep:, :Nkeep] = 0 and the restart from F is exact.
-      int Nkeep = startIdx * Nblock;
-      for (auto& v : startBlock) basis.push_back(v);
-
-      // Fill restart coupling rows into H (exact: H_new is upper triangular,
-      // so B encodes the only non-zero coupling to the new block).
-      for (int t = 0; t < Nblock; t++)
-        for (int j = 0; j < Nkeep; j++)
-          H(Nkeep + t, j) = std::conj(B(j, t));
-
-      // Zero out B for the retained columns now that the coupling is in H
-      for (int j = 0; j < Nkeep; j++)
-        for (int t = 0; t < Nblock; t++)
-          B(j, t) = 0.0;
-    }
-
-    for (int k = startIdx; k < endBlock; k++)
-      blockArnoldiStep(k, doubleOrthog);
-  }
-
-  //--------------------------------------------------------------------
-  // One block Arnoldi step
-  //--------------------------------------------------------------------
-  void blockArnoldiStep(int k, bool doubleOrthog)
-  {
-    int kBase  = k * Nblock;
-    int prevN  = kBase + Nblock;
-    int N      = Nm;
-
-    std::vector<Field> W(Nblock, Field(Grid_));
-    for (int t = 0; t < Nblock; t++)
-      Linop.Op(basis[kBase + t], W[t]);
-
-    // Full reorthogonalisation against all current basis vectors
-    for (int pass = 0; pass < (doubleOrthog ? 2 : 1); pass++) {
-      for (int i = 0; i < prevN; i++) {
-        for (int t = 0; t < Nblock; t++) {
-          ComplexD coeff = innerProduct(basis[i], W[t]);
-          if (pass == 0)
-            H(i, kBase + t) = toStdCmplx(coeff);
-          else
-            H(i, kBase + t) += toStdCmplx(coeff);
-          W[t] -= coeff * basis[i];
-        }
-      }
-    }
-
-    F = W;
-
-    if (k == Nm/Nblock - 1) {
-      // Last block: record coupling in B as R^H (Hermitian conjugate of QR factor)
-      // KS relation requires B[kBase+t, s] = conj(R[s,t])
-      CMat R = blockQR(F);
-      for (int t = 0; t < Nblock; t++)
-        for (int s = 0; s < Nblock; s++)
-          B(kBase + t, s) = std::conj(R(s, t));    // B_block = R^H
-      beta_k = R.norm();
-      return;
-    }
-
-    // Not last: QR the residual, extend basis
-    CMat R = blockQR(F);
-
-    int nextBase = (k + 1) * Nblock;
-    for (int i = 0; i < Nblock; i++)
-      for (int j = 0; j < Nblock; j++)
-        H(nextBase + i, kBase + j) = R(i, j);
-
-    for (int t = 0; t < Nblock; t++)
-      basis.push_back(F[t]);
-
-  }
-
-  //--------------------------------------------------------------------
-  // Block QR (modified Gram-Schmidt within the block)
-  //--------------------------------------------------------------------
-  CMat blockQR(std::vector<Field>& W)
-  {
-    CMat R = CMat::Zero(Nblock, Nblock);
-    const RealD deflThresh = 1e-14;
-
-    for (int j = 0; j < Nblock; j++) {
-      for (int i = 0; i < j; i++) {
-        ComplexD coeff = innerProduct(W[i], W[j]);
-        R(i, j) = toStdCmplx(coeff);
-        W[j] -= coeff * W[i];
-      }
-      RealD nrm = std::sqrt(norm2(W[j]));
-      R(j, j)   = nrm;
-      if (nrm > deflThresh) {
-        W[j] *= (1.0 / nrm);
-      } else {
-        W[j] = Zero();
-        std::cout << GridLogMessage
-                  << "HarmonicBlockKrylovSchur: deflation at block column " << j
-                  << " (norm = " << nrm << ")" << std::endl;
-      }
-    }
-    return R;
-  }
-
-  //--------------------------------------------------------------------
-  // Orthonormalise starting block
-  //--------------------------------------------------------------------
-  void blockOrthonormalise(std::vector<Field>& V)
-  {
-    for (int j = 0; j < (int)V.size(); j++) {
-      for (int i = 0; i < j; i++) {
-        ComplexD c = innerProduct(V[i], V[j]);
-        V[j] -= c * V[i];
-      }
-      RealD nrm = std::sqrt(norm2(V[j]));
-      assert(nrm > 1e-14);
-      V[j] *= (1.0 / nrm);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  // Basis rotation: UR[i] = sum_j U[j] * R(j,i)
-  //--------------------------------------------------------------------
-  void constructUR(std::vector<Field>& UR, std::vector<Field>& U,
-                   CMat& R, int N)
-  {
-    UR.clear();
-    Field tmp(Grid_);
-    for (int i = 0; i < N; i++) {
-      tmp = Zero();
-      for (int j = 0; j < N; j++)
-        tmp += U[j] * R(j, i);
-      UR.push_back(tmp);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  // Eigensystem of the truncated H (not Hhat)
-  //--------------------------------------------------------------------
-  /**
-   * Eigenvalues of H_k are the standard Ritz values in the retained
-   * subspace.  After convergence has been declared via harmonic estimates,
-   * the final reported eigenvalues and vectors come from H_k (not Hhat_k),
-   * since H_k contains the true projected operator.
-   */
-  void computeEigensystem(CMat& Hk, int Nkeep)
-  {
-    Eigen::ComplexEigenSolver<CMat> es;
-    es.compute(Hk);
-
-    // Sort to match schurReorder ordering.
-    int n = es.eigenvalues().size();
-    ComplexComparator cComp(ritzFilter);
-    std::vector<int> idx(n);
-    std::iota(idx.begin(), idx.end(), 0);
-    std::sort(idx.begin(), idx.end(), [&](int a, int b){
-      return cComp(toStdCmplx(es.eigenvalues()(a)), toStdCmplx(es.eigenvalues()(b)));
-    });
-
-    evals.resize(n);
-    littleEvecs.resize(n, n);
-    for (int k = 0; k < n; k++) {
-      evals(k)           = es.eigenvalues()(idx[k]);
-      littleEvecs.col(k) = es.eigenvectors().col(idx[k]);
-    }
-
-    evecs.clear();
-    for (int k = 0; k < Nkeep; k++) {
-      CVec vec = littleEvecs.col(k);
-      Field tmp(Grid_);
-      tmp = Zero();
-      for (int j = 0; j < (int)basis.size() && j < Nkeep; j++)
-        tmp += vec[j] * basis[j];
-      evecs.push_back(tmp);
-    }
-  }
-
-  //--------------------------------------------------------------------
-  // Convergence check
-  //--------------------------------------------------------------------
-  /**
-   * Ritz estimate for eigenpair k: || B^H y_k ||
-   * where y_k is the k-th eigenvector of the truncated H.
-   * The same bound applies whether using Ritz or harmonic Ritz restart.
-   */
-  int converged(int Nkeep)
-  {
-    ritzEstimates.clear();
-    int Nconv = 0;
-
-    CMat Bk = B(Eigen::seqN(0, Nkeep), Eigen::all);
-
-    for (int k = 0; k < Nkeep; k++) {
-      CVec yk   = littleEvecs.col(k);
-      CVec Bty  = Bk.adjoint() * yk;
-      RealD res = Bty.norm();
-      ritzEstimates.push_back(res);
-      std::cout << GridLogMessage
-                << "HarmonicBlockKrylovSchur: Ritz estimate[" << k
-                << "] = " << res << "  eval = " << evals[k] << std::endl;
-      if (res < rtol) Nconv++;
-      else break;
-    }
-    return Nconv;
-  }
-
-  //--------------------------------------------------------------------
-  // Approximate maximum eigenvalue (power iteration)
-  //--------------------------------------------------------------------
-  RealD approxMaxEval(const Field& v0, int MAX_ITER = 50)
-  {
-    assert(norm2(v0) > 1e-8);
-    RealD lam = 0.0, denom = std::sqrt(norm2(v0));
-    Field vcur(Grid_), vtmp(Grid_);
-    vcur = v0;
-    for (int i = 0; i < MAX_ITER; i++) {
-      Linop.Op(vcur, vtmp);
-      vcur  = vtmp;
-      RealD num = std::sqrt(norm2(vcur));
-      lam   = num / denom;
-      denom = num;
-    }
-    return lam;
-  }
-
-};
-
-NAMESPACE_END(Grid);
-
-#endif  // GRID_HARMONIC_BLOCKED_KRYLOV_SCHUR_H

Grid/algorithms/iterative/ImplicitlyRestartedLanczos.h

@@ -53,18 +53,6 @@ enum IRLdiagonalisation {
   IRLdiagonaliseWithEigen
 };
 
-enum IRLeigsort { 
-  IRLeigsortMax,
-  IRLeigsortSqMin
-};
-
-#if 0
-bool square_comp(RealD a, RealD b){
-	if (a*a<b*b) return true;
-	return false;
-}
-#endif
-
 template<class Field> class ImplicitlyRestartedLanczosHermOpTester  : public ImplicitlyRestartedLanczosTester<Field>
 {
  public:
@@ -131,10 +119,9 @@ class ImplicitlyRestartedLanczos {
   /////////////////////////
   // Constructor
   /////////////////////////
- public:
-  IRLeigsort EigSort;
-
   
+public:       
+
   //////////////////////////////////////////////////////////////////
   // PAB:
   //////////////////////////////////////////////////////////////////
@@ -167,7 +154,6 @@ class ImplicitlyRestartedLanczos {
     Nstop(_Nstop)  ,      Nk(_Nk),      Nm(_Nm),
     eresid(_eresid),      betastp(_betastp),
     MaxIter(_MaxIter)  ,      MinRestart(_MinRestart),
-    EigSort(IRLeigsortMax), 
     orth_period(_orth_period), diagonalisation(_diagonalisation)  { };
 
     ImplicitlyRestartedLanczos(LinearFunction<Field> & PolyOp,
@@ -184,7 +170,6 @@ class ImplicitlyRestartedLanczos {
     Nstop(_Nstop)  ,      Nk(_Nk),      Nm(_Nm),
     eresid(_eresid),      betastp(_betastp),
     MaxIter(_MaxIter)  ,      MinRestart(_MinRestart),
-    EigSort(IRLeigsortMax), 
     orth_period(_orth_period), diagonalisation(_diagonalisation)  { };
 
   ////////////////////////////////
@@ -331,12 +316,8 @@ until convergence
       // sorting
       //////////////////////////////////
       eval2_copy = eval2;
-//      if (EigSort==IRLeigsortMax)
-//      std::partial_sort(eval2.begin(),eval2.begin()+Nm,eval2.end(),square_comp);
-//      else
       std::partial_sort(eval2.begin(),eval2.begin()+Nm,eval2.end(),std::greater<RealD>());
       std::cout<<GridLogIRL <<" evals sorted "<<std::endl;
-//      eval2_copy = eval2;
       const int chunk=8;
       for(int io=0; io<k2;io+=chunk){
 	std::cout<<GridLogIRL << "eval "<< std::setw(3) << io ;
@@ -352,7 +333,6 @@ until convergence
       //////////////////////////////////
       Qt = Eigen::MatrixXd::Identity(Nm,Nm);
       for(int ip=k2; ip<Nm; ++ip){ 
-//        std::cout<<GridLogIRL <<"QR decompose "<<eval2[ip]<<std::endl;
 	QR_decomp(eval,lme,Nm,Nm,Qt,eval2[ip],k1,Nm);
       }
       std::cout<<GridLogIRL <<"QR decomposed "<<std::endl;
@@ -395,8 +375,7 @@ until convergence
 
 	//  power of two search pattern;  not every evalue in eval2 is assessed.
 	int allconv =1;
-//	for(int jj = 1; jj<=Nstop; jj*=2){
-	for(int jj = 1; jj<=Nstop; jj++){
+	for(int jj = 1; jj<=Nstop; jj*=2){
 	  int j = Nstop-jj;
 	  RealD e = eval2_copy[j]; // Discard the evalue
 	  basisRotateJ(B,evec,Qt,j,0,Nk,Nm);	    

Grid/algorithms/iterative/KrylovSchur.h

@@ -1,994 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid 
-
-Source file: ./lib/algorithms/iterative/KrylovSchur.h
-
-Copyright (C) 2015
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-Author: paboyle <paboyle@ph.ed.ac.uk>
-Author: Patrick Oare <poare@bnl.gov>
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_KRYLOVSCHUR_H
-#define GRID_KRYLOVSCHUR_H
-
-NAMESPACE_BEGIN(Grid);
-
-#if defined(GRID_CUDA) || defined(GRID_HIP)
-using thrust::abs;
-using thrust::conj;
-inline std::complex<double> toStdCmplx(const ComplexD& c) { return {c.real(), c.imag()}; }
-#else
-using std::abs;
-using std::conj;
-inline const ComplexD& toStdCmplx(const ComplexD& c) { return c; }
-#endif
-
-/**
- * Small dense coefficient matrix of ComplexD (thrust::complex on GPU builds)
- * for passing to basisRotate, whose device kernel cannot multiply Grid tensors
- * by std::complex. Must be namespace-scope: nvcc rejects function-local types
- * as template arguments to functions containing extended device lambdas.
- */
-struct KSCoeffMat {
-  std::vector<ComplexD> a; int n;
-  KSCoeffMat(int _n) : a(_n*_n), n(_n) {}
-  ComplexD&       operator()(int i, int j)       { return a[i*n+j]; }
-  const ComplexD& operator()(int i, int j) const { return a[i*n+j]; }
-};
-
-/**
- * Options for which Ritz values to keep in implicit restart. TODO move this and utilities into a new file
- */
-enum RitzFilter {
-  EvalNormSmall,           // Keep evals with smallest norm
-  EvalNormLarge,           // Keep evals with largest norm
-  EvalReSmall,             // Keep evals with smallest real part
-  EvalReLarge,             // Keep evals with largest real part
-  EvalReNormLarge,         // Keep evals with largest real part
-  EvalImSmall,             // Keep evals with smallest imaginary part
-  EvalImLarge,             // Keep evals with largest imaginary part
-  EvalImNormSmall,         // Keep evals with smallest |imaginary| part
-  EvalImNormLarge,         // Keep evals with largest |imaginary| part
-};
-
-/** Selects the RitzFilter corresponding to the input string. */
-inline RitzFilter selectRitzFilter(std::string s) {
-  if (s == "EvalNormSmall")   { return EvalNormSmall; }   else
-  if (s == "EvalNormLarge")   { return EvalNormLarge; }   else
-  if (s == "EvalReSmall")     { return EvalReSmall; }     else
-  if (s == "EvalReLarge")     { return EvalReLarge; }     else
-  if (s == "EvalReNormLarge")     { return EvalReNormLarge; }     else
-  if (s == "EvalImSmall")     { return EvalImSmall; }     else
-  if (s == "EvalImLarge")     { return EvalImLarge; }     else 
-  if (s == "EvalImNormSmall") { return EvalImNormSmall; } else
-  if (s == "EvalImNormLarge") { return EvalImNormLarge; } else 
-  { assert(0); }
-}
-
-/** Returns a string saying which RitzFilter it is. */
-inline std::string rfToString(RitzFilter RF) {
-  switch (RF) {
-    case EvalNormSmall:
-      return "EvalNormSmall";
-    case EvalNormLarge:
-      return "EvalNormLarge";
-    case EvalReSmall:
-      return "EvalReSmall";
-    case EvalReLarge:
-      return "EvalReLarge";
-    case EvalReNormLarge:
-      return "EvalReNormLarge";
-    case EvalImSmall:
-      return "EvalImSmall";
-    case EvalImLarge:
-      return "EvalImLarge";
-    case EvalImNormSmall:
-      return "EvalImNormSmall";
-    case EvalImNormLarge:
-      return "EvalImNormLarge";
-    default:
-      assert(0);
-  }
-}
-
-// Select comparison function from RitzFilter
-struct ComplexComparator
-{
-  RitzFilter RF;
-  ComplexComparator (RitzFilter _rf) : RF(_rf) {}
-  bool operator()(std::complex<double> z1, std::complex<double> z2) { 
-	  RealD tmp1=std::abs(std::imag(z1));
-	  RealD tmp2=std::abs(std::imag(z2));
-	  if ( std::abs(std::real(z1)) >2.) tmp1 += 100.;
-	  if ( std::abs(std::real(z2)) >2.) tmp2 += 100.;
-    switch (RF) {
-      case EvalNormSmall:
-        return std::abs(z1) < std::abs(z2);
-      case EvalNormLarge:
-        return std::abs(z1) > std::abs(z2);
-      case EvalReSmall:
-        return std::real(z1) < std::real(z2);     // DELETE THE ABS HERE!!!
-      case EvalReLarge:
-        return std::real(z1) > std::real(z2);
-      case EvalReNormLarge:
-        return std::abs(std::real(z1)) < std::abs(std::real(z2));
-      case EvalImSmall:
-        return std::imag(z1) < std::imag(z2);
-      case EvalImLarge:
-        return std::imag(z1) > std::imag(z2);
-      case EvalImNormSmall:
-//        return std::abs(std::imag(z1)) < std::abs(std::imag(z2));
-          return tmp1 < tmp2 ;
-      case EvalImNormLarge:
-        return std::abs(std::imag(z1)) > std::abs(std::imag(z2));
-      default:
-        assert(0);
-    }
-  }
-};
-
-/**
- * Computes a complex Schur decomposition of a complex matrix A using Eigen's matrix library. The Schur decomposition,
- *    A = Q^\dag S Q
- * factorizes A into a unitary matrix Q and an upper triangular matrix S. The eigenvalues of A lie on the diagonal of the upper triangular matrix S. 
- * The Schur decomposition is not unique: in particular, any ordering of the eigenvalues of A can be used as the diagonal of the matrix S. 
- * This class supports eigenvalue reordering by swapping two adjacent eigenvalues with a unitary transformation. 
- */
-class ComplexSchurDecomposition {
-
-  private:
-
-    typedef Eigen::MatrixXcd CMat;
-
-    CMat A;           // Matrix to decompose, A = Q^\dag S Q
-    CMat Q;           // Unitary matrix Q
-    CMat S;           // Upper triangular matrix S
-
-    // Placeholders for Givens rotation
-    CMat Givens;      // Givens rotation
-    ComplexD s;       // off-diagonal element
-    ComplexD lam1;    // First eval for swap
-    ComplexD lam2;    // Second eval for swap
-    ComplexD phi;     // phase of s
-    RealD r;          // norm of s and lam2 - lam1
-    
-    int Nm;                               // size of matrix problem
-
-    ComplexComparator cCompare;           // function to sort the Schur matrix. 
-  
-  public:
-
-    /**
-     * If the input matrix _A is in Hessenberg form (upper triangular + first subdiagonal non-zero), then the Schur 
-     * decomposition is easier to compute. 
-     */
-    ComplexSchurDecomposition(CMat _A, bool isHess, RitzFilter ritzFilter = EvalReSmall) : A(_A), Nm (_A.rows()), cCompare (ritzFilter)
-    {
-      Eigen::ComplexSchur<CMat> schur (Nm);
-      if (isHess) {
-        schur.computeFromHessenberg(_A, CMat::Identity(Nm, Nm), true);
-      } else {
-        schur.compute(_A, true);
-      }
-      S = schur.matrixT();
-      Q = schur.matrixU().adjoint();      // Eigen computes A = Q S Q^\dag, we want A = Q^\dag S Q
-    }
-
-    // Getters
-    int  getNm()      { return Nm; }         // size of matrix problem
-    CMat getMatrixA() { return A;  }         // matrix for decomposition
-    CMat getMatrixQ() { return Q;  }         // unitary matrix Q
-    CMat getMatrixS() { return S;  }         // Schur matrix (upper triangular) S
-    CMat getRitz()    { return S.diagonal(); }
-
-    /**
-     * Checks the Schur decomposition A = Q^\dag S Q holds for the computed matrices. Returns if the relative 
-     * Frobenius norm || A - Q^\dag S Q || / || A || is less than rtol. 
-     */
-    bool checkDecomposition(RealD rtol = 1e-8) {
-      RealD Anorm = A.norm();
-      if (Anorm < rtol) {
-        std::cout << GridLogMessage << "Zero matrix" << std::endl;
-        return true;
-      }
-
-      std::cout << GridLogDebug << "S = " << std::endl << S << std::endl;
-      std::cout << GridLogDebug << "Q = " << std::endl << Q << std::endl;
-
-      CMat A2 = Q.adjoint() * S * Q;
-      std::cout << GridLogDebug << "Q^dag S Q = " << std::endl << A2 << std::endl;
-      RealD dA = (A - A2).norm() / Anorm;
-      return (dA < rtol);
-    }
-
-    /**
-     * Swaps the components on the diagonal of the Schur matrix at index i with index i + 1. 
-     * Updates the orthogonal matrix Q accordingly. 
-     */
-    void swapEvals(int i) {
-
-      assert(0 <= i && i <= Nm - 1);        // can only swap blocks with upper left index between 0 and Nm - 1
-
-      // get parameters for rotation
-      s    = S(i, i+1);
-      lam1 = S(i, i);
-      lam2 = S(i+1, i+1);
-      phi  = s / abs(s);
-      r    = std::sqrt(std::pow(abs(s), 2) + std::pow(abs(lam2 - lam1), 2));
-
-      // // Original code which performs Givens rotations by manual matrix multiplication
-      // // compute Givens rotation corresponding to these parameters
-      // Givens = CMat::Identity(Nm, Nm);
-      // Givens(i, i)     = std::abs(s) / r;
-      // Givens(i+1, i+1) = Givens(i, i);
-      // Givens(i, i+1)   = (phi / r) * std::conj(lam2 - lam1);
-      // Givens(i+1, i)   = -std::conj(Givens(i, i+1));
-      // // rotate Schur matrix and unitary change of basis matrix Q
-      // S = Givens * S * Givens.adjoint();
-      // Q = Givens * Q;
-
-      // Modified code
-      Givens = CMat::Identity(2, 2);
-      Givens(0, 0)     = abs(s) / r;
-      Givens(1, 1) = Givens(0, 0);
-      Givens(0, 1)   = (phi / r) * conj(lam2 - lam1);
-      Givens(1, 0)   = -conj(Givens(0, 1));
-
-      // TODO: make sure these are correct
-      Eigen::MatrixXcd tmp;
-      tmp = Givens * S.block(i, 0, 2, Nm);
-      S.block(i, 0, 2, Nm) = tmp;
-
-      // S_{:, i:i+2} = S.block<Nm, 2>(0, i);
-      tmp = S.block(0, i, Nm, 2) * Givens.adjoint();
-      S.block(0, i, Nm, 2) = tmp;
-
-      // Q_{i:i+2, :} = Q.block<2, Nm>(i, 0);
-      tmp = Givens * Q.block(i, 0, 2, Nm);
-      Q.block(i, 0, 2, Nm) = tmp;
-
-      return;
-    }
-
-    /**
-     * Reorders a Schur matrix &Schur to have the Ritz values that we would like to keep for 
-     * restart as the first Nk elements on the diagonal. 
-     * 
-     * This algorithm is implemented as Nk iterations of a a reverse bubble sort with comparator compare.
-     * TODO: pass in compare function as an argument, default to compare with <. 
-     */
-    // void schurReorder(int Nk, std::function compare) {
-    void schurReorder(int Nk) {
-     std::cout << GridLogMessage << "schurReorder start " << std::endl;
-     int nswap=0;
-      for (int i = 0; i < Nk; i++) {
-        for (int k = 0; k <= Nm - 2; k++) {
-          int idx = Nm - 2 - k;
-          // TODO use RitzFilter enum here
-          // if (std::abs(S(idx, idx)) < std::abs(S(idx+1, idx+1))) {            // sort by largest modulus of eigenvalue
-          // if (std::real(S(idx, idx)) > std::real(S(idx+1, idx+1))) {        // sort by smallest real eigenvalue
-          if ( cCompare(S(idx+1, idx+1), S(idx, idx)) ) {            // sort by largest modulus of eigenvalue
-            swapEvals(idx);
-	    nswap++;
-          }
-        }
-      }
-     std::cout << GridLogMessage << "schurReorder end "<< nswap << std::endl;
-      for (int i = 0; i < Nk; i++) {
-      	std::cout << GridLogMessage << "schurR "<<i<<" "<<S(i,i) << std::endl;
-      }
-      return;
-    }
-
-    void schurReorderBlock() {
-      // TODO method stub
-      return;
-    }
-
-};
-
-// template<class Field>
-// inline void writeFile(const Field &field, const std::string &fname) {
-//   emptyUserRecord record;
-//   ScidacWriter WR(field.Grid()->IsBoss());
-//   WR.open(fname);
-//   WR.writeScidacFieldRecord(field, record, 0); // 0 = Lexico
-//   WR.close();
-// }
-
-/**
- * Implementation of the Krylov-Schur algorithm.
- */
-template<class Field> 
-class KrylovSchur {
-
-  private:
-  
-    std::string cname = std::string("KrylovSchur");
-    int MaxIter;   // Max iterations
-    RealD Tolerance;
-    RealD ssq;
-    RealD rtol;
-    int Nm;           // Number of basis vectors to track (equals MaxIter if no restart)
-    int Nk;           // Number of basis vectors to keep every restart (equals -1 if no restart)
-    int Nstop;       // Stop after converging Nstop eigenvectors.
-
-    LinearOperatorBase<Field> &Linop;
-    GridBase *Grid;
-
-    RealD approxLambdaMax;
-    RealD beta_k;
-    Field u;                                // Residual vector perpendicular to Krylov space (u_{k+1} in notes)
-    Eigen::VectorXcd   b;                   // b vector in Schur decomposition (e_{k+1} in Arnoldi).
-    std::vector<Field> basis;               // orthonormal Krylov basis
-    Eigen::MatrixXcd   Rayleigh;            // Rayleigh quotient of size Nbasis (after construction)
-    Eigen::MatrixXcd   Qt;                  // Transpose of basis rotation which projects out high modes.
-
-    Eigen::VectorXcd   evals;               // evals of Rayleigh quotient
-    std::vector<RealD> ritzEstimates;       // corresponding ritz estimates for evals
-    Eigen::MatrixXcd   littleEvecs;         // Nm x Nm evecs matrix
-
-    RitzFilter ritzFilter;                  // how to sort evals
-
-  public:
-    bool doEvalCheck = false;
-
-    KrylovSchur(LinearOperatorBase<Field> &_Linop, GridBase *_Grid, RealD _Tolerance, RitzFilter filter = EvalReSmall)
-      : Linop(_Linop), Grid(_Grid), Tolerance(_Tolerance), ritzFilter(filter), u(_Grid), MaxIter(-1), Nm(-1), Nk(-1), Nstop (-1),
-        evals (0), ritzEstimates (), evecs (), ssq (0.0), rtol (0.0), beta_k (0.0), approxLambdaMax (0.0)
-    {
-      u = Zero();
-    };
-    std::vector<Field> evecs;               // Vector of evec fields
-
-
-    /* Getters */
-    int                 getNk()                { return Nk;            }
-    Eigen::MatrixXcd    getRayleighQuotient()  { return Rayleigh;      }
-    Field               getU()                 { return u;             }
-    std::vector<Field>  getBasis()             { return basis;         }
-    Eigen::VectorXcd    getEvals()             { return evals;         }
-    std::vector<RealD>  getRitzEstimates()     { return ritzEstimates; }
-    std::vector<Field>  getEvecs()             { return evecs;         }
-
-    /**
-     * Runs the non-harmonic Krylov-Schur loop.
-     *   - Runs an Arnoldi step to generate the Rayleigh quotient and Krylov basis.
-     *   - Schur decompose the Rayleigh quotient.
-     *   - Permutes the Rayleigh quotient according to the eigenvalues.
-     *   - Truncate the Krylov-Schur expansion.
-     */
-    void operator()(const Field& v0, int _maxIter, int _Nm, int _Nk, int _Nstop, bool doubleOrthog = true) {
-
-      MaxIter = _maxIter;
-      Nm = _Nm; Nk = _Nk;
-      Nstop = _Nstop;
-
-      ssq = norm2(v0);
-      RealD approxLambdaMax = approxMaxEval(v0);
-      rtol = Tolerance * approxLambdaMax;
-      std::cout << GridLogMessage << "Approximate max eigenvalue: " << approxLambdaMax << std::endl;
-
-      b = Eigen::VectorXcd::Zero(Nm);       // start as e_{k+1}
-      b(Nm-1) = 1.0;
-
-      int start = 0;
-      Field startVec = v0;
-      littleEvecs = Eigen::MatrixXcd::Zero(Nm, Nm);
-      for (int i = 0; i < MaxIter; i++) {
-        std::cout << GridLogMessage << "Restart Iteration " << i << std::endl;
-
-        // Perform Arnoldi steps to compute Krylov basis and Rayleigh quotient (Hess)
-        arnoldiIteration(startVec, Nm, start, doubleOrthog);
-        startVec = u;
-        start = Nk;
-
-        std::cout << GridLogDebug << "b after Arnoldi " << b << std::endl;
-
-        // --- Restart: Schur-decompose Rayleigh, reorder by ritzFilter, rotate
-        //     the basis into the Schur vectors, and truncate to the leading Nk.
-        ComplexSchurDecomposition schur (Rayleigh, false, ritzFilter);
-        std::cout << GridLogDebug << "Schur decomp holds? " << schur.checkDecomposition() << std::endl;
-
-        // Rearrange Schur matrix so wanted evals are on top left (like MATLAB's ordschur)
-        std::cout << GridLogMessage << "Reordering Schur eigenvalues" << std::endl;
-        schur.schurReorder(Nk);
-        std::cout << GridLogMessage << "Schur eigenvalues reordered." << std::endl;
-
-        Eigen::MatrixXcd Q = schur.getMatrixQ();
-        Qt = Q.adjoint();                           // TODO should Q be real?
-
-        std::cout << GridLogMessage << "*** ROTATING TO SCHUR BASIS *** " << std::endl;
-
-        // Rotate Krylov basis, b vector, redefine Rayleigh quotient and evecs, and truncate.
-        Rayleigh = schur.getMatrixS();
-        b = Q * b;            // b^\dag = b^\dag * Q^\dag <==> b = Q*b
-
-        std::vector<Field> basis2;
-        constructUR(basis2, basis, Qt, Nm,Nk);
-
-        std::cout << GridLogMessage << "*** TRUNCATING FOR RESTART *** " << std::endl;
-        std::cout << GridLogDebug << "Rayleigh before truncation: " << std::endl << Rayleigh << std::endl;
-
-        Eigen::MatrixXcd RayTmp = Rayleigh(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk));
-        Rayleigh = RayTmp;
-
-        basis = std::vector<Field> (basis2.begin(), basis2.begin() + Nk);
-
-        b = b.head(Nk).eval();
-
-        std::cout << GridLogDebug << "Rayleigh after truncation: " << std::endl << Rayleigh << std::endl;
-
-        checkKSDecomposition();
-
-        // Compute eigensystem of Rayleigh. Note the eigenvectors correspond to the sorted eigenvalues.
-        computeEigensystem(Rayleigh);
-        std::cout << GridLogMessage << "Eigenvalues (first Nk sorted): " << std::endl << evals << std::endl;
-
-        if (checkConvergedAndReport(i)) return;
-      }
-    }
-
-    /**
-     * Runs the harmonic (shifted) Krylov-Schur loop: extracts Ritz values of a
-     * shift-augmented Rayleigh quotient so that eigenvalues near `*_shift` are
-     * reordered to the top instead of the extremal ones.
-     */
-    void operator()(const Field& v0, int _maxIter, int _Nm, int _Nk, int _Nstop, RealD *_shift, bool doubleOrthog = true) {
-
-      assert(_shift && "harmonic KrylovSchur: shift must be non-null");
-      RealD shiftVal = *_shift;
-
-      MaxIter = _maxIter;
-      Nm = _Nm; Nk = _Nk;
-      Nstop = _Nstop;
-
-      ssq = norm2(v0);
-      RealD approxLambdaMax = approxMaxEval(v0);
-      rtol = Tolerance * approxLambdaMax;
-      std::cout << GridLogMessage << "Approximate max eigenvalue: " << approxLambdaMax << std::endl;
-
-      b = Eigen::VectorXcd::Zero(Nm);       // start as e_{k+1}
-      b(Nm-1) = 1.0;
-
-      int start = 0;
-      Field startVec = v0;
-      littleEvecs = Eigen::MatrixXcd::Zero(Nm, Nm);
-      for (int i = 0; i < MaxIter; i++) {
-        std::cout << GridLogMessage << "Restart Iteration " << i << std::endl;
-
-        // Perform Arnoldi steps to compute Krylov basis and Rayleigh quotient (Hess)
-        arnoldiIteration(startVec, Nm, start, doubleOrthog);
-        startVec = u;
-        start = Nk;
-
-        std::cout << GridLogDebug << "b after Arnoldi " << b << std::endl;
-
-        // --- Harmonic restart: Schur-decompose the shift-augmented Rayleigh
-        //     quotient so Ritz values close to shiftVal are reordered to the
-        //     top, then rotate and truncate as in the non-harmonic case.
-        Eigen::MatrixXcd temp = Rayleigh;
-        for (int m=0;m<Nm;m++) temp(m,m) -= shiftVal;
-        Eigen::MatrixXcd RayleighS = temp.inverse(); // (B-tI)^-1
-        Eigen::MatrixXcd temp2 = RayleighS.adjoint(); //(B-tI)^-1*
-        Eigen::VectorXcd g  = temp2*b; //g = (B-tI)^-1* * b
-        Eigen::MatrixXcd Btilde = Rayleigh + g*(b.adjoint());
-
-        Field utilde(Grid);
-        utilde  = u;
-        for (int j = 0; j<Nm; j++){
-          utilde -= basis[j]*g(j);
-        }
-
-        ComplexSchurDecomposition schurS (Btilde, false, ritzFilter);
-        std::cout << GridLogMessage << "Shifted Schur eigenvalues shift = "<<shiftVal  << std::endl;
-        schurS.schurReorder(Nk);
-
-        Eigen::MatrixXcd Q_s = schurS.getMatrixQ();
-        Eigen::MatrixXcd Qt_s = Q_s.adjoint();                           // TODO should Q be real?
-
-        Btilde=schurS.getMatrixS();
-        Eigen::VectorXcd b_s = Q_s*b; // Q is Qt in SlepC, b_s=bhat
-
-        std::vector<Field> basis2_s;
-        constructUR(basis2_s, basis, Qt_s, Nm,Nk);
-
-        Eigen::MatrixXcd RayTmp_s = Btilde(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk));
-        Btilde = RayTmp_s;
-
-        basis2_s = std::vector<Field> (basis2_s.begin(), basis2_s.begin() + Nk);
-
-        b_s = b_s.head(Nk).eval();
-
-        Eigen::VectorXcd ghat = -Q_s * g;
-        ghat = ghat.head(Nk).eval();
-
-        Field uhat(Grid);
-        uhat  = utilde;
-        for (int j = 0; j<Nk; j++){
-          uhat -= basis2_s[j]*ghat(j);
-        }
-
-        RealD gamma = std::sqrt(norm2(uhat));
-        uhat *= 1.0/gamma;
-        std::cout << GridLogMessage << " gamma "<<gamma << std::endl;
-
-        Btilde  += ghat*(b_s.adjoint());
-        b_s *=gamma;
-
-        Rayleigh = Btilde;
-        basis= basis2_s;
-        b  = b_s;
-        beta_k = gamma;
-        u= uhat;
-
-        checkKSDecomposition();
-        computeEigensystem(Rayleigh);
-        std::cout << GridLogMessage << "Eigenvalues (first Nk sorted): " << std::endl << evals << std::endl;
-
-        if (checkConvergedAndReport(i)) return;
-      }
-    }
-
-  private:
-
-    /**
-     * Shared post-restart convergence check. On convergence (or the final
-     * iteration) reports eigenvalues and optionally the explicit residual
-     * check. Returns true if operator() should return.
-     */
-    bool checkConvergedAndReport(int i) {
-      int Nconv = converged();
-      std::cout << GridLogMessage << "Number of evecs converged: " << Nconv << std::endl;
-      if (Nconv >= Nstop || i == MaxIter - 1) {
-        std::cout << GridLogMessage << "Converged with " << Nconv << " / " << Nstop << " eigenvectors on iteration "
-                      << i << "." << std::endl;
-        std::cout << GridLogMessage << "Eigenvalues: " << std::endl << evals << std::endl;
-
-        if (doEvalCheck) {
-          Field w(Grid);
-          for (int k = 0; k < (int)evecs.size(); k++) {
-            Linop.Op(evecs[k], w);
-            ComplexD eval_est = toStdCmplx(innerProduct(evecs[k], w));
-            w -= eval_est * evecs[k];
-            RealD res = std::sqrt(norm2(w));
-            std::cout << GridLogMessage << "KrylovSchur: evec[" << k << "]"
-                      << "  eval_reported = " << evals[k]
-                      << "  eval_est = " << eval_est
-                      << "  || A v - eval_est * v || = " << res << std::endl;
-          }
-        }
-        return true;
-      }
-      return false;
-    }
-
-  public:
-
-    /**
-     * Constructs the Arnoldi basis for the Krylov space K_n(D, src). (TODO make private)
-     * 
-     * Parameters
-     * ----------
-     * v0 : Field&
-     *  Source to generate Krylov basis. 
-     * Nm : int
-     *  Final size of the basis desired. If the basis becomes complete before a basis of size Nm is constructed 
-     *  (determined by relative tolerance Tolerance), stops iteration there. 
-     * doubleOrthog : bool (default = false)
-     *  Whether to double orthogonalize the basis (for numerical cancellations) or not. 
-     * start        : int (default = 0)
-     *  If non-zero, assumes part of the Arnoldi basis has already been constructed. 
-     */
-    void arnoldiIteration(const Field& v0, int Nm, int start = 0, bool doubleOrthog = true)
-    {
-
-      ComplexD coeff;
-      Field w (Grid);           // A acting on last Krylov vector. 
-
-      // basis.reserve(Nm);
-      // for (int i = start; i < Nm; i++) {
-      //   basis.emplace_back(Grid);
-      // }
-      // basis.assign(Nm, Field(Grid));
-      // basis.resize(Nm);
-      // for (int i = start; i < Nm; i++) {
-      //   basis[i] = Field(Grid);
-      // }
-
-      if (start == 0) {       // initialize everything that we need.
-        RealD v0Norm = 1 / std::sqrt(ssq);
-        basis.push_back(v0Norm * v0);                // normalized source
-        // basis[0] = v0Norm * v0;                // normalized source
-
-        Rayleigh = Eigen::MatrixXcd::Zero(Nm, Nm);
-        u = Zero();
-      } else {
-        // assert( start == basis.size() );      // should be starting at the end of basis (start = Nk)
-        std::cout << GridLogMessage << "Resetting Rayleigh and b" << std::endl;
-        Eigen::MatrixXcd RayleighCp = Rayleigh;
-        Rayleigh = Eigen::MatrixXcd::Zero(Nm, Nm);
-        Rayleigh(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk)) = RayleighCp;
-
-        // append b^\dag to Rayleigh, add u to basis
-        Rayleigh(Nk, Eigen::seqN(0, Nk)) = b.adjoint();
-        basis.push_back(u);
-        // basis[start] = u;       // TODO make sure this is correct
-        b = Eigen::VectorXcd::Zero(Nm);
-      }
-
-      // Construct next Arnoldi vector by normalizing w_i = Dv_i - \sum_j v_j h_{ji}
-      for (int i = start; i < Nm; i++) {
-        Linop.Op(basis.back(), w);
-        // Linop.Op(basis[i], w);
-        for (int j = 0; j < basis.size(); j++) {
-          coeff = innerProduct(basis[j], w);       // coeff = h_{ij}. Note that since {vi} is ONB it's OK to subtract it off after. 
-          Rayleigh(j, i) = toStdCmplx(coeff);
-          w -= coeff * basis[j];
-        }
-
-        if (doubleOrthog) {
-          std::cout << GridLogDebug << "Double orthogonalizing." << std::endl;
-          for (int j = 0; j < basis.size(); j++) {
-            coeff = innerProduct(basis[j], w);      // see if there is any residual component in basis[j] direction
-            Rayleigh(j, i) += toStdCmplx(coeff);   // if coeff is non-zero, adjust Rayleigh
-            w -= coeff * basis[j];
-          }
-        }
-
-        // add w_i to the pile
-        if (i < Nm - 1) {
-          coeff = std::sqrt(norm2(w));
-          Rayleigh(i+1, i) = toStdCmplx(coeff);
-          basis.push_back(
-            (1.0/coeff) * w
-          );
-          // basis[i+1] = (1.0/coeff) * w;
-        }
-
-        // after iterations, update u and beta_k = ||u|| before norm
-        u = w;                                // make sure u is normalized
-        beta_k = std::sqrt(norm2(u));         // beta_k = ||f_k|| determines convergence.
-        u = (1/beta_k) * u;
-      }
-
-      b(Nm - 1) = beta_k;
-
-      // std::cout << GridLogMessage << "|f|^2 after Arnoldi step = " << norm2(f) << std::endl;
-      std::cout << GridLogMessage << "beta_k = |u| (before norm) after Arnoldi step = " << beta_k << std::endl;
-      std::cout << GridLogDebug << "Computed Rayleigh quotient = " << std::endl << Rayleigh << std::endl;
-
-      return;
-    }
-
-    /**
-     * Approximates the eigensystem of the linear operator by computing the eigensystem of 
-     * the Rayleigh quotient. Assumes that the Rayleigh quotient has already been constructed (by 
-     * calling the operator() function).
-     * 
-     * Parameters
-     * ----------
-     * Eigen::MatrixXcd& S
-     *  Schur matrix (upper triangular) similar to original Rayleigh quotient.
-     */
-    void computeEigensystem(Eigen::MatrixXcd& S)
-    {
-      std::cout << GridLogMessage << "Computing eigenvalues." << std::endl;
-
-      evecs.clear();
-
-      Eigen::ComplexEigenSolver<Eigen::MatrixXcd> es;
-      es.compute(S);
-
-      // Sort eigenvalues/evecs to match the schurReorder ordering.
-      int n = es.eigenvalues().size();
-      ComplexComparator cComp(ritzFilter);
-      std::vector<int> idx(n);
-      std::iota(idx.begin(), idx.end(), 0);
-      std::sort(idx.begin(), idx.end(), [&](int a, int b){
-        return cComp(toStdCmplx(es.eigenvalues()(a)), toStdCmplx(es.eigenvalues()(b)));
-      });
-
-      evals.resize(n);
-      littleEvecs.resize(n, n);
-      for (int k = 0; k < n; k++) {
-        evals(k)          = es.eigenvalues()(idx[k]);
-        littleEvecs.col(k) = es.eigenvectors().col(idx[k]);
-      }
-
-      // Convert evecs to lattice fields
-      for (int k = 0; k < n; k++) {
-        Eigen::VectorXcd vec = littleEvecs.col(k);
-        Field tmp (basis[0].Grid());
-        tmp = Zero();
-        for (int j = 0; j < (int)basis.size(); j++) {
-          tmp = tmp + vec[j] * basis[j];
-        }
-        evecs.push_back(tmp);
-      }
-    }
-
-    /**
-     * Approximates the maximum eigenvalue of Linop.Op to normalize the residual and test for convergence. 
-     * 
-     * TODO implement in parent class eventually
-     * 
-     * Parameters
-     * ----------
-     * Field& v0
-     *  Source field to start with. Must have non-zero norm.
-     * int MAX_ITER (default = 50)
-     *  Maximum number of iterations for power approximation. 
-     * 
-     * Returns
-     * -------
-     * RealD lamApprox
-     *  Approximation of largest eigenvalue. 
-     */
-    RealD approxMaxEval(const Field& v0, int MAX_ITER = 50) {
-      assert (norm2(v0) > 1e-8);                        // must have relatively large source norm to start
-      RealD lamApprox = 0.0;
-      RealD denom = 1.0; RealD num = 1.0;
-      Field v0cp (Grid); Field tmp (Grid);
-      v0cp = v0;
-      denom = std::sqrt(norm2(v0cp));
-      for (int i = 0; i < MAX_ITER; i++) {
-        Linop.Op(v0cp, tmp);                               // CAREFUL: do not do Op(tmp, tmp)
-        v0cp = tmp;
-        num = std::sqrt(norm2(v0cp));                      // num = |A^{n+1} v0|
-        lamApprox = num / denom;                           // lam = |A^{n+1} v0| / |A^n v0|
-        std::cout << GridLogDebug << "Approx for max eval: " << lamApprox << std::endl;
-        denom = num;                                       // denom = |A^{n} v0|
-      }
-      return lamApprox;
-    }
-
-    /**
-     * Computes the number of Krylov-Schur eigenvectors that have converged. An eigenvector s is considered converged 
-     * for a tolerance epsilon if 
-     *    r(s) := |\beta e_m^T s| < epsilon
-     * where beta is the norm of f_{m+1}.
-     * 
-     * TODO implement in parent class eventually
-     * 
-     * Parameters
-     * ----------
-     * 
-     * Returns
-     * -------
-     * int : Number of converged eigenvectors.
-     */
-    int converged() {
-      int Nconv = 0;
-      int _Nm = evecs.size();
-      std::cout << GridLogDebug << "b: " << b << std::endl;
-      Field tmp (Grid); Field fullEvec (Grid);
-      ritzEstimates.clear();
-      // ritzEstimates.resize(_Nm);
-      for (int k = 0; k < _Nm; k++) {
-        Eigen::VectorXcd evec_k = littleEvecs.col(k);
-        RealD ritzEstimate = std::abs(b.dot(evec_k));           // b^\dagger s
-        ritzEstimates.push_back(ritzEstimate);
-        std::cout << GridLogMessage << "Ritz estimate for evec " << k << " = " << ritzEstimate << std::endl;
-        if (ritzEstimate < rtol) {
-          Nconv++;
-        } else {
-          break;
-        }
-      }
-      // Check that Ritz estimate is explicitly || D (Uy) - lambda (Uy) ||
-//       checkRitzEstimate();
-      return Nconv;
-    }
-
-    /**
-     * Checks the Krylov-Schur decomposition DU = UR + f b^\dag with the last-computed 
-     * U, R, f, and b. 
-     */
-    bool checkKSDecomposition(RealD tol = 1e-8) {
-
-      std::cout << GridLogMessage << "*** CHECKING KRYLOV-SCHUR DECOMPOSITION *** " << std::endl;
-
-      int k = basis.size();         // number of basis vectors, also the size of Rayleigh.
-
-      // rotate basis by Rayleigh to construct UR
-      // std::vector<Field> rotated;
-
-      // std::cout << GridLogDebug << "Rayleigh in KSDecomposition: " << std::endl << Rayleigh << std::endl;
-
-      std::vector<Field> rotated = basis;
-      constructUR(rotated, basis, Rayleigh, k,k);             // manually rotate
-      // Eigen::MatrixXcd Rt = Rayleigh.adjoint();
-      // basisRotate(rotated, Rt, 0, k, 0, k, k);           // UR
-
-      // TODO: make a new function that I'm positive does what this is doing
-      // just take the basis U = (u1 u2 ... uNm) and form the linear combination UR from R
-      
-      // For each i, form D u(i) and subtract off (US - u b^\dag)(i)
-      RealD delta = 0.0; RealD deltaSum = 0;
-      Field tmp (Grid); tmp = Zero();
-      for (int i = 0; i < k; i++) {
-        Linop.Op(basis[i], tmp);          // tmp = D u(i)
-        delta = norm2(tmp - rotated[i] - u * std::conj(b(i)));
-        delta = delta / norm2(tmp);               // relative tolerance
-        deltaSum += delta;
-
-        // std::cout << GridLogDebug << "Iteration " << i << std::endl;
-        // std::cout << GridLogDebug << "Du = " << norm2(tmp) << std::endl;
-        // std::cout << GridLogDebug << "rotated = " << norm2(rotated[i]) << std::endl;
-        // std::cout << GridLogDebug << "b[i] = " << b(i) << std::endl;
-        std::cout << GridLogMessage << "Deviation in decomp, column " << i << ": " << delta << std::endl;
-      }
-      std::cout << GridLogMessage << "Squared sum of relative deviations in decomposition: " << deltaSum << std::endl;
-
-      // std::cout << "[DEBUG] testing basis rotate" << std::endl;
-      // std::vector<Field> rotated2;
-      // constructUR(rotated2, basis, Rayleigh, k);
-      // for (int i = 0; i < k; i++) {
-      //   std::cout << "rotated[i] - UR[i] = " << norm2(rotated[i] - rotated2[i]) << std::endl;
-      // }
-
-      return deltaSum < tol;
-    }
-
-    /**
-     * Checks the Ritz vector s was rotated correctly by explicitly recomputing the 
-     * eigenvectors of the rotated Rayleigh quotient. 
-     */
-    bool checkRitzRotation(RealD tol = 1e-8) {
-      std::cout << GridLogMessage << "*** CHECKING RITZ VECTOR ROTATION *** " << std::endl;
-
-      Eigen::ComplexEigenSolver<Eigen::MatrixXcd> es;
-      es.compute(Rayleigh);
-      Eigen::MatrixXcd littleEvecs2 = es.eigenvectors();
-      RealD dLittle = (littleEvecs2 - littleEvecs).norm() / littleEvecs.norm();
-      std::cout << GridLogMessage << "|littleEvecs2 - littleEvecs| / |littleEvecs| = " << dLittle << std::endl;
-
-      std::cout << GridLogMessage << "Forming full eigenvectors" << std::endl;
-      RealD delta = 0.0; RealD deltaSum = 0;
-      for (int k = 0; k < evals.size(); k++) {
-        Eigen::VectorXcd vec = littleEvecs.col(k);
-        Field tmpEvec (Grid);
-        tmpEvec = Zero();
-        for (int j = 0; j < basis.size(); j++) {
-          tmpEvec = tmpEvec + vec[j] * basis[j];
-        }
-        delta = norm2(tmpEvec - evecs[k]) / norm2(evecs[k]);
-        std::cout << GridLogMessage << "Deviation in evec " << k << ": " << delta << std::endl;
-        deltaSum += delta;
-      }
-      return deltaSum < tol;
-    }
-
-    /**
-     * Checks the Ritz estimate R(s) is indeed the deviation of a Ritz eigenvector from being a true eigenvector. 
-     */
-    void checkRitzEstimate(RealD tol = 1e-8) {
-      std::cout << GridLogMessage << "*** CHECKING RITZ ESTIMATE *** " << std::endl;
-
-      // The issue was that the Eigen::eigensolver occasionally returned the complex conjugate pairs in the wrong 
-      // order compared to the diagonal, which is how I was reading them out. When this happened, the Ritz estimate would 
-      // be wrong. So, just need to be more careful and actually read out the eigenvalues. 
-
-      Field tmp (Grid);
-      // std::cout << "n evecs: " << evecs.size() << std::endl;
-      for (int k = 0; k < evecs.size(); k++) {
-        tmp = Zero();
-        Linop.Op(evecs[k], tmp);        // D evec
-        RealD ritz = std::sqrt(norm2(tmp - evals[k] * evecs[k]));
-        std::cout << "Ritz estimate " << k << " = " << ritz << std::endl;
-
-        // Checking little Ritz estimate
-        // Eigen::VectorXcd littleEvec = littleEvecs.col(k);
-        // Eigen::VectorXcd dev = Rayleigh * littleEvec - evals[k] * littleEvec;
-        // std::cout << GridLogMessage << "Little Ritz estimate = " << dev.norm() << std::endl;
-      }
-      return;
-    }
-
-    /**
-     * Given a vector of fields U (equivalently, a LxN matrix, where L is the number of degrees of
-     * freedom on the lattice field) and an NxN matrix R, forms the product UR.
-     * Only the first N2 output columns are computed; columns [N2, N) are Zero.
-     *
-     * Implemented with the accelerator basisRotate kernel, which computes
-     * basis[j] <- sum_k Qt(j,k) basis[k] in place, so we copy U into UR and
-     * pass R^T. The coefficients are staged as ComplexD (thrust::complex on
-     * GPU builds) since std::complex has no device arithmetic with Grid tensors.
-     */
-    void constructUR(std::vector<Field>& UR, std::vector<Field> &U, Eigen::MatrixXcd& R, int N, int N2) {
-      KSCoeffMat Rt(N);
-      for (int i = 0; i < N; i++)
-        for (int j = 0; j < N; j++)
-          Rt(i, j) = ComplexD(R(j, i).real(), R(j, i).imag());
-
-      UR = U;
-      basisRotate(UR, Rt, 0, N2, 0, N, N);
-      for (int i = N2; i < N; i++) UR[i] = Zero();
-      return;
-    }
-
-    /**
-     * Same as constructUR but for the product order RU. 
-     */
-    void constructRU(std::vector<Field>& RU, std::vector<Field> &U, Eigen::MatrixXcd& R, int N) {
-      Field tmp (Grid);
-      RU.clear();
-      // RU.resize(N);
-      for (int i = 0; i < N; i++) {
-        tmp = Zero();
-        for (int j = 0; j < N; j++) {
-          tmp = tmp + R(i, j) * U[j];
-        }
-        RU.push_back(tmp);
-        // RU[i] = tmp;
-      }
-      return;
-    }
-
-    // void writeEvec(Field& in, std::string const fname){  
-    //   #ifdef HAVE_LIME
-    //     // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-    //     std::cout << GridLogMessage << "Writing evec to: " << fname << std::endl;
-    //     Grid::emptyUserRecord record;
-    //     Grid::ScidacWriter WR(in.Grid()->IsBoss());
-    //     WR.open(fname);
-    //     WR.writeScidacFieldRecord(in,record,0); // Lexico
-    //     WR.close();
-    //   #endif
-    //     // What is the appropriate way to throw error?
-    // }
-
-    // /**
-    //  * Writes the eigensystem of a Krylov Schur object to a directory. 
-    //  * 
-    //  * Parameters
-    //  * ----------
-    //  * std::string path
-    //  *    Directory to write to. 
-    //  */
-    // void writeEigensystem(std::string outDir) {
-    //   std::cout << GridLogMessage << "Writing output to directory: " << outDir << std::endl;
-    //   // TODO write a scidac density file so that we can easily integrate with visualization toolkit
-    //   std::string evalPath = outDir + "/evals.txt";
-    //   std::ofstream fEval;
-    //   fEval.open(evalPath);
-    //   for (int i = 0; i < Nk; i++) {
-    //     // write Eigenvalues
-    //     fEval << i << " " << evals(i);
-    //     if (i < Nk - 1) { fEval << "\n"; }
-    //   }
-    //   fEval.close();
-      
-    //   for (int i = 0; i < Nk; i++) {
-    //     std::string fName = outDir + "/evec" + std::to_string(i);
-    //     // writeFile(evecs[i], fName);     // using method from Grid/HMC/ComputeWilsonFlow.cc
-    //     // writeEvec(evecs[i], fName);
-    //   }
-      
-    // }
-
-};
-
-NAMESPACE_END(Grid);
-#endif

Grid/algorithms/iterative/LanczosBidiagonalization.h

@@ -1,276 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./Grid/algorithms/iterative/LanczosBidiagonalization.h
-
-Copyright (C) 2015
-
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_LANCZOS_BIDIAGONALIZATION_H
-#define GRID_LANCZOS_BIDIAGONALIZATION_H
-
-NAMESPACE_BEGIN(Grid);
-
-/**
- * Lanczos Bidiagonalization (Golub-Kahan)
- *
- * For a linear operator A with adjoint A^dag, constructs the bidiagonal
- * decomposition:
- *
- *   A  V_m = U_m B_m
- *   A^dag U_m = V_m B_m^T + beta_{m+1} v_{m+1} e_m^T
- *
- * where:
- *   V_m = [v_1, ..., v_m]  right Lanczos vectors (orthonormal)
- *   U_m = [u_1, ..., u_m]  left  Lanczos vectors (orthonormal)
- *   B_m is upper bidiagonal with diag(alpha_1,...,alpha_m) and
- *       superdiag(beta_2,...,beta_m)
- *
- * The singular values of A are approximated by those of B_m.
- * The singular values of B_m are the square roots of the eigenvalues of
- * the symmetric tridiagonal matrix B_m^T B_m.
- *
- * Usage:
- *   LanczosBidiagonalization<Field> lb(Linop, grid);
- *   lb.run(src, Nm, tol);
- *   // Access results via getters.
- */
-template <class Field>
-class LanczosBidiagonalization {
-
-  public: 
-  LinearOperatorBase<Field> &Linop;
-  GridBase *Grid;
-
-  int Nm;           // number of Lanczos steps taken
-  RealD Tolerance;  // convergence threshold on beta_{k+1} / alpha_k
-
-  std::vector<Field>  V;       // right Lanczos vectors v_1 ... v_m
-  std::vector<Field>  U;       // left  Lanczos vectors u_1 ... u_m
-  std::vector<RealD>  alpha;   // diagonal of bidiagonal matrix
-  std::vector<RealD>  beta;    // super-diagonal (beta[k] couples u_k and v_{k+1})
-
-  // SVD of the bidiagonal matrix (filled after computeSVD())
-  Eigen::VectorXd  singularValues;
-  Eigen::MatrixXd  leftSVecs;   // columns are left  singular vectors of B
-  Eigen::MatrixXd  rightSVecs;  // columns are right singular vectors of B
-
-public:
-
-  LanczosBidiagonalization(LinearOperatorBase<Field> &_Linop, GridBase *_Grid,
-                           RealD _tol = 1.0e-8)
-    : Linop(_Linop), Grid(_Grid), Tolerance(_tol), Nm(0)
-  {}
-
-  /**
-   * Run the Golub-Kahan Lanczos bidiagonalization.
-   *
-   * Parameters
-   * ----------
-   * src  : starting vector (need not be normalised)
-   * Nmax : maximum number of Lanczos steps
-   * reorth : if true, full reorthogonalisation of both V and U bases
-   */
-  void run(const Field &src, int Nmax, bool reorth = true)
-  {
-    assert(norm2(src) > 0.0);
-
-    V.clear(); U.clear();
-    alpha.clear(); beta.clear();
-    Nm = 0;
-
-    Field p(Grid), r(Grid);
-
-    // --- initialise: v_1 = src / ||src|| ---
-    Field v(Grid);
-    v = src;
-    RealD nrm = std::sqrt(norm2(v));
-    v = (1.0 / nrm) * v;
-    V.push_back(v);
-
-    for (int k = 0; k < Nmax; ++k) {
-
-      // p = A v_k
-      Linop.Op(V[k], p);
-
-      // p = p - beta_k * u_{k-1}   (remove previous left vector)
-      if (k > 0) {
-        p = p - beta[k-1] * U[k-1];
-      }
-
-      // alpha_k = ||p||
-      RealD ak = std::sqrt(norm2(p));
-      if (ak < 1.0e-14) {
-        std::cout << GridLogMessage
-                  << "LanczosBidiagonalization: lucky breakdown at step "
-                  << k << " (alpha = " << ak << ")" << std::endl;
-        break;
-      }
-      alpha.push_back(ak);
-
-      // u_k = p / alpha_k
-      Field u(Grid);
-      u = (1.0 / ak) * p;
-
-      // full reorthogonalisation of u against previous U
-      if (reorth) {
-        for (int j = 0; j < (int)U.size(); ++j) {
-          ComplexD ip = innerProduct(U[j], u);
-          u = u - ip * U[j];
-        }
-        RealD unrm = std::sqrt(norm2(u));
-        if (unrm > 1.0e-14) u = (1.0 / unrm) * u;
-      }
-      U.push_back(u);
-
-      // r = A^dag u_k - alpha_k * v_k
-      Linop.AdjOp(U[k], r);
-      r = r - ak * V[k];
-
-      // full reorthogonalisation of r against previous V
-      if (reorth) {
-        for (int j = 0; j < (int)V.size(); ++j) {
-          ComplexD ip = innerProduct(V[j], r);
-          r = r - ip * V[j];
-        }
-      }
-
-      // beta_{k+1} = ||r||
-      RealD bk = std::sqrt(norm2(r));
-      beta.push_back(bk);
-
-      Nm = k + 1;
-
-      std::cout << GridLogMessage
-                << "LanczosBidiagonalization step " << k
-                << "  alpha = " << ak
-                << "  beta  = " << bk << std::endl;
-
-      // convergence: residual beta / alpha small enough
-      if (bk / ak < Tolerance) {
-        std::cout << GridLogMessage
-                  << "LanczosBidiagonalization converged at step " << k
-                  << "  (beta/alpha = " << bk / ak << ")" << std::endl;
-        break;
-      }
-
-      if (k == Nmax - 1) break;   // no v_{k+2} needed after last step
-
-      // v_{k+1} = r / beta_{k+1}
-      Field vnext(Grid);
-      vnext = (1.0 / bk) * r;
-      V.push_back(vnext);
-    }
-  }
-
-  /**
-   * Compute the SVD of the bidiagonal matrix B using Eigen.
-   * Singular values are stored in descending order.
-   */
-  void computeSVD()
-  {
-    int m = Nm;
-    Eigen::MatrixXd B = Eigen::MatrixXd::Zero(m, m);
-
-    for (int k = 0; k < m; ++k) {
-      B(k, k) = alpha[k];
-      if (k + 1 < m && k < (int)beta.size())
-        B(k, k+1) = beta[k];
-    }
-
-    Eigen::JacobiSVD<Eigen::MatrixXd> svd(B,
-        Eigen::ComputeThinU | Eigen::ComputeThinV);
-
-    singularValues = svd.singularValues();   // already sorted descending
-    leftSVecs      = svd.matrixU();
-    rightSVecs     = svd.matrixV();
-
-    std::cout << GridLogMessage
-              << "LanczosBidiagonalization: singular values of B_" << m
-              << std::endl;
-    for (int k = 0; k < m; ++k)
-      std::cout << GridLogMessage << "  sigma[" << k << "] = "
-                << singularValues(k) << std::endl;
-  }
-
-  /**
-   * Return the k-th approximate left singular vector of A in the full
-   * lattice space.  computeSVD() must have been called first.
-   */
-  Field leftSingularVector(int k)
-  {
-    assert(k < (int)leftSVecs.cols());
-    Field svec(Grid);
-    svec = Zero();
-    for (int j = 0; j < Nm; ++j)
-      svec = svec + leftSVecs(j, k) * U[j];
-    return svec;
-  }
-
-  /**
-   * Return the k-th approximate right singular vector of A in the full
-   * lattice space.  computeSVD() must have been called first.
-   */
-  Field rightSingularVector(int k)
-  {
-    assert(k < (int)rightSVecs.cols());
-    Field svec(Grid);
-    svec = Zero();
-    for (int j = 0; j < Nm; ++j)
-      svec = svec + rightSVecs(j, k) * V[j];
-    return svec;
-  }
-
-  /**
-   * Verify the bidiagonalization: returns max residual
-   *   max_k || A v_k - alpha_k u_k - beta_k u_{k-1} ||
-   */
-  RealD verify()
-  {
-    Field tmp(Grid);
-    RealD maxres = 0.0;
-    for (int k = 0; k < Nm; ++k) {
-      Linop.Op(V[k], tmp);
-      tmp = tmp - alpha[k] * U[k];
-      if (k > 0 && k-1 < (int)beta.size())
-        tmp = tmp - beta[k-1] * U[k-1];
-      RealD res = std::sqrt(norm2(tmp));
-      if (res > maxres) maxres = res;
-      std::cout << GridLogMessage
-                << "LanczosBidiagonalization verify step " << k
-                << "  ||A v_k - alpha_k u_k - beta_{k-1} u_{k-1}|| = "
-                << res << std::endl;
-    }
-    return maxres;
-  }
-
-  /* Getters */
-  int                       getNm()           const { return Nm; }
-  const std::vector<Field>& getV()            const { return V; }
-  const std::vector<Field>& getU()            const { return U; }
-  const std::vector<RealD>& getAlpha()        const { return alpha; }
-  const std::vector<RealD>& getBeta()         const { return beta; }
-  Eigen::VectorXd           getSingularValues() const { return singularValues; }
-};
-
-NAMESPACE_END(Grid);
-#endif

Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h

@@ -236,4 +236,5 @@ public:
   }
 };
 NAMESPACE_END(Grid);
+#undef GCRLogLevel
 #endif

Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h

@@ -38,13 +38,14 @@ Author: Peter Boyle <paboyle@ph.ed.ac.uk>
 ///////////////////////////////////////////////////////////////////////////////////////////////////////
 NAMESPACE_BEGIN(Grid);
 
-#define GCRLogLevel std::cout << GridLogMessage <<std::string(level,'\t')<< " Level "<<level<<" " 
+#define GCRLogLevel std::cout << GridLogMessage <<std::string(level,'\t')<< name<<" "
 
 template<class Field>
 class PrecGeneralisedConjugateResidualNonHermitian : public LinearFunction<Field> {
 public:                                                
   using LinearFunction<Field>::operator();
   RealD   Tolerance;
+  RealD   SSQ;
   Integer MaxIterations;
   int verbose;
   int mmax;
@@ -54,40 +55,37 @@ public:
   GridStopWatch PrecTimer;
   GridStopWatch MatTimer;
   GridStopWatch LinalgTimer;
+  std::string name;
+  int ZeroGuess  = 0;  // caller contract: guess is always zero => first-cycle r0 = src, skip the apply
+  int FirstCycle = 0;
 
   LinearFunction<Field>     &Preconditioner;
   LinearOperatorBase<Field> &Linop;
 
-  void Level(int lv) { level=lv; };
+  void Name(std::string _name) { name = _name; };
 
-  PrecGeneralisedConjugateResidualNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,LinearFunction<Field> &Prec,int _mmax, int _nstep) : 
+  void Level(int n) { Name("Level " + std::to_string(n)); level = n; }
+
+  void SetZeroGuess(int z) { ZeroGuess = z; };
+
+  PrecGeneralisedConjugateResidualNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,LinearFunction<Field> &Prec,int _mmax,int _nstep) : 
     Tolerance(tol), 
     MaxIterations(maxit),
     Linop(_Linop),
     Preconditioner(Prec),
     mmax(_mmax),
-    nstep(_nstep)         // what is nstep vs mmax? one is the number of inner iterations
-  { 
-    level=1;
+    nstep(_nstep)
+  {
+    Level(1);
     verbose=1;
   };
 
-  // virtual method stubs for updating GCR polynomial
-  virtual void LogBegin(void){
-    std::cout << "GCR::LogBegin() "<<std::endl;
-  };
-  virtual void LogIteration(int k, ComplexD a, std::vector<ComplexD> betas){
-    std::cout << "GCR::LogIteration() "<<std::endl;
-  };
-  virtual void LogComplete(std::vector<ComplexD>& alphas, std::vector<std::vector<ComplexD>>& betas) {
-    std::cout << "GCR::LogComplete() "<<std::endl;
-  };
-
   void operator() (const Field &src, Field &psi){
 
     //    psi=Zero();
     RealD cp, ssq,rsq;
     ssq=norm2(src);
+    SSQ=ssq;
     rsq=Tolerance*Tolerance*ssq;
       
     Field r(src.Grid());
@@ -100,25 +98,27 @@ public:
     SolverTimer.Start();
 
     steps=0;
+    FirstCycle=1;
     for(int k=0;k<MaxIterations;k++){
 
       cp=GCRnStep(src,psi,rsq);
 
-      GCRLogLevel <<"PGCR("<<mmax<<","<<nstep<<") "<< steps <<" steps cp = "<<cp<<" target "<<rsq <<std::endl;
+      GCRLogLevel <<"PGCR("<<mmax<<","<<nstep<<") "<< steps <<" steps cp = "<<sqrt(cp/ssq)<<" target "<<sqrt(rsq/ssq) <<std::endl;
 
       if(cp<rsq) {
-        SolverTimer.Stop();
 
-        Linop.Op(psi,r);
-        axpy(r,-1.0,src,r);
-        RealD tr = norm2(r);
-        GCRLogLevel<<"PGCR: Converged on iteration " <<steps
-          << " computed residual "<<sqrt(cp/ssq)
-          << " true residual "    <<sqrt(tr/ssq)
-          << " target "           <<Tolerance <<std::endl;
+	SolverTimer.Stop();
 
-        GCRLogLevel<<"PGCR Time elapsed: Total  "<< SolverTimer.Elapsed() <<std::endl;
-        return;
+	Linop.Op(psi,r);
+	axpy(r,-1.0,src,r);
+	RealD tr = norm2(r);
+	GCRLogLevel<<"PGCR: Converged on iteration " <<steps
+		 << " computed residual "<<sqrt(cp/ssq)
+		 << " true residual "    <<sqrt(tr/ssq)
+		 << " target "           <<Tolerance <<std::endl;
+
+	GCRLogLevel<<"PGCR Time elapsed: Total  "<< SolverTimer.Elapsed() <<std::endl;
+	return;
       }
 
     }
@@ -145,30 +145,32 @@ public:
     ////////////////////////////////
     // history for flexible orthog
     ////////////////////////////////
-    std::vector<Field> q(mmax,grid);        // q = Ap
-    std::vector<Field> p(mmax,grid);        // store mmax conjugate momenta
-    std::vector<RealD> qq(mmax);            // qq = (Ap)^2 = <p|A^\dagger A |p> (denom of \alpha)
+    std::vector<Field> q(mmax,grid);
+    std::vector<Field> p(mmax,grid);
+    std::vector<RealD> qq(mmax);
       
     GCRLogLevel<< "PGCR nStep("<<nstep<<")"<<std::endl;
 
     //////////////////////////////////
-    // initial guess x0 is taken as nonzero.
-    // r0=src-A x0 = src
+    // r0 = src - A x0.  ZeroGuess: on the first cycle x0==0 by caller
+    // contract (enforced here), so r0 = src exactly; skip the apply.
+    // Restart cycles (psi!=0) always do the full computation.
     //////////////////////////////////
-    MatTimer.Start();
-    Linop.Op(psi,Az);
-    //    zAz = innerProduct(Az,psi);
-    zAAz= norm2(Az);
-    MatTimer.Stop();
-    
+    if (ZeroGuess && FirstCycle) {
+      psi = Zero();
+      LinalgTimer.Start();
+      r = src;
+      LinalgTimer.Stop();
+    } else {
+      MatTimer.Start();
+      Linop.Op(psi,Az);
+      MatTimer.Stop();
+      LinalgTimer.Start();
+      r=src-Az;
+      LinalgTimer.Stop();
+    }
+    FirstCycle=0;
 
-    LinalgTimer.Start();
-    r=src-Az;
-    LinalgTimer.Stop();
-    GCRLogLevel<< "PGCR true residual r = src - A psi   "<< norm2(r) <<std::endl;
-
-    this->LogBegin();       // initialize polynomial GCR if needed (TODO think about placement of this)
-    
     /////////////////////
     // p = Prec(r)
     /////////////////////
@@ -190,46 +192,33 @@ public:
     p[0]= z;
     q[0]= Az;
     qq[0]= zAAz;
-
-    std::cout << "||init p - src||: " << norm2(p[0] - src) << std::endl;   // for debugging
     
     cp =norm2(r);
     LinalgTimer.Stop();
-
-    std::vector<ComplexD> all_alphas;
-    std::vector<std::vector<ComplexD>> all_betas;
+    GCRLogLevel<< "PGCR true residual "<< sqrt(cp/SSQ)     <<std::endl;
 
     for(int k=0;k<nstep;k++){
 
       steps++;
 
       int kp     = k+1;
-      int peri_k = k %mmax;     // only store mmax vectors; just roll around if needed
+      int peri_k = k %mmax;
       int peri_kp= kp%mmax;
 
-      // std::cout << "peri_kp = " << peri_kp << std::endl;
-
       LinalgTimer.Start();
       rq= innerProduct(q[peri_k],r); // what if rAr not real?
-      a = rq/qq[peri_k];              // compute alpha_j
+      a = rq/qq[peri_k];
 
-      all_alphas.push_back(a);
+      axpy(psi,a,p[peri_k],psi);         
 
-      axpy(psi,a,p[peri_k],psi);      // update psi --> psi + \alpha p
-
-      cp = axpy_norm(r,-a,q[peri_k],r);       // update r --> r - \alpha D p. Note q = Dp
+      cp = axpy_norm(r,-a,q[peri_k],r);
       LinalgTimer.Stop();
 
-      // LogIterationA(k + 1, a);
-
-      GCRLogLevel<< "GCR step["<<steps<<"]  resid " << cp << " target " <<rsq<<std::endl; 
-
-      // moving this to end of loop so that it doesn't exit beforehand
-      // TODO if I want to uncomment this, I have to split the LogIteration again and put LogIterationA() beforehand
-      // if((k==nstep-1)||(cp<rsq)){
-      //   return cp;
-      // }
+      GCRLogLevel<< "PGCR step["<<steps<<"]  resid " << sqrt(cp/SSQ)<<std::endl;
 
+      if((k==nstep-1)||(cp<rsq)){
+	return cp;
+      }
 
       PrecTimer.Start();
       Preconditioner(r,z);// solve Az = r
@@ -246,205 +235,24 @@ public:
       q[peri_kp]=Az;
       p[peri_kp]=z;
 
-      // Field Dsrc (grid);
-      // Linop.Op(src, Dsrc);
-      // std::cout << "||q[peri_kp] - D(src)||: " << norm2(q[peri_kp] - Dsrc) << std::endl;   // for debugging
-
-          // // delete after testing
-          // std::cout << "Testing Dsq on one for GCR: " << std::endl;
-          // Field myField (grid);
-          // myField = 1.0;
-          // Field out1 (grid); Field out2 (grid);
-          // Linop.HermOp(myField, out1);
-          // Linop.Op(myField, out2);
-          // std::cout << "Dsq.Hermop(ones) has norm " << norm2(out1) << std::endl;
-          // std::cout << "Dsq.Op(ones) has norm " << norm2(out2) << std::endl;
-
-      // basically northog = k+1 if mmax is large
       int northog = ((kp)>(mmax-1))?(mmax-1):(kp);  // if more than mmax done, we orthog all mmax history.
-      // std::cout << "northog: " << northog << std::endl;
-      std::vector<ComplexD> betas (northog);
-      // std::cout << "peri_kp: " << peri_kp << std::endl;
-      // we iterate backwards counting down from the current k+1 index (peri_kp) because we 
       for(int back=0;back<northog;back++){
 
 	int peri_back=(k-back)%mmax;   	  GRID_ASSERT((k-back)>=0);
 
-        // b=-real(innerProduct(q[peri_back],Az))/qq[peri_back];
-        b=-(innerProduct(q[peri_back],Az))/qq[peri_back];     // TODO try complex beta
-        p[peri_kp]=p[peri_kp]+b*p[peri_back];
-        q[peri_kp]=q[peri_kp]+b*q[peri_back];
-
-        // LogIterationB(peri_back, b);
-        // betas[back] = b;    // may need to change the indexing if I ever do it with restarts
-        // std::cout << "[DEBUG] pushing beta for back = " << back << ", peri_back = " << peri_back << std::endl;
-
-        betas[peri_back] = b;    // may need to change the indexing if I ever do it with restarts
+	b=-real(innerProduct(q[peri_back],Az))/qq[peri_back];
+	p[peri_kp]=p[peri_kp]+b*p[peri_back];
+	q[peri_kp]=q[peri_kp]+b*q[peri_back];
 
       }
       qq[peri_kp]=norm2(q[peri_kp]); // could use axpy_norm
       LinalgTimer.Stop();
-
-      // log iteration and update GCR polynomial if necessary.
-      all_betas.push_back(betas);
-      LogIteration(k + 1, a, betas);
-
-      // finish if necessary
-      if((k==nstep-1)||(cp<rsq)){
-        std::cout << "All alphas: " << std::endl << all_alphas << std::endl;
-        std::cout << "All betas: " << std::endl << all_betas << std::endl;
-        LogComplete(all_alphas, all_betas);
-        std::cout << "Exiting GCR." << std::endl;
-        return cp;
-      }
-
     }
     GRID_ASSERT(0); // never reached
     return cp;
   }
 };
-
-class PolynomialFile: Serializable {
-  public:
-    GRID_SERIALIZABLE_CLASS_MEMBERS(PolynomialFile, 
-      std::vector<std::vector<std::complex<double>>>, data,
-      std::vector<std::vector<std::complex<double>>>, betas,
-      std::vector<std::complex<double>>,              alphas
-    );
-};
-
-// Optionally record the GCR polynomial. [PO]: TODO
-template <class Field>
-class PGCRPolynomial : public PrecGeneralisedConjugateResidualNonHermitian<Field> {
-public:
-  std::vector<ComplexD> ak;
-  std::vector<std::vector<ComplexD>> bk;
-  // std::vector<ComplexD> poly_p;
-  std::vector<std::vector<ComplexD>> poly_p;
-  std::vector<ComplexD> poly_Ap;        // polynomial in Ap_j (only store it for last p)
-  std::vector<ComplexD> poly_r;
-  std::vector<ComplexD> polynomial;
-
-  PolynomialFile& PF;
-
-public:
-  PGCRPolynomial(RealD tol, Integer maxit,LinearOperatorBase<Field> &_Linop, LinearFunction<Field> &Prec, int _mmax, int _nstep, PolynomialFile& _PF)
-    : PrecGeneralisedConjugateResidualNonHermitian<Field>(tol, maxit, _Linop, Prec, _mmax, _nstep), PF(_PF)
-  {};
-
-  // think this applies the polynomial in A = Linop to a field src. The coeffs are 
-  // stored in the vector `polynomial`.
-  void PolyOp(const Field &src, Field &psi)
-  {
-    Field tmp(src.Grid());
-    Field AtoN(src.Grid());
-    AtoN = src;
-    psi=AtoN*polynomial[0];
-    for(int n=1;n<polynomial.size();n++){
-      tmp = AtoN;
-      this->Linop.Op(tmp,AtoN);               // iterate A^n
-      psi = psi + polynomial[n]*AtoN;       // psi += poly_n A^n src
-    }
-  }
-
-  // [PO TODO] debug this
-  void PGCRsequence(const Field &src, Field &x)
-  {
-    Field Ap(src.Grid());
-    Field r(src.Grid());
-    // Field p(src.Grid());
-    // p=src;
-    std::vector<Field> p;
-    p.push_back(src);
-    r=src;
-    x=Zero();
-    x.Checkerboard()=src.Checkerboard();
-    for(int k=0;k<ak.size();k++){
-      x = x + ak[k]*p[k];
-      this->Linop.Op(p[k], Ap);
-      r = r - ak[k] * Ap;
-      // p[k] = r;
-      p.push_back(r);
-      for (int i = 0; i < k; i++) {     // [PO TODO] check indices
-        p[k+1] += bk[i, k+1] * p[i];
-      }
-      // p = r + bk[k] * p;
-    }
-  }
-
-  void Solve(const Field &src, Field &psi)
-  {
-    psi=Zero();
-    this->operator()(src, psi);
-  }
-
-  virtual void LogBegin(void)
-  {
-    std::cout << "PGCR::LogBegin() "<<std::endl;
-    ak.resize(0);
-    bk.resize(0);
-    polynomial.resize(0);
-    poly_Ap.push_back(0.0);     // start with (0.0); during first iteration should change to (0.0, 1.0)
-    std::vector<ComplexD> p0_tmp;
-    p0_tmp.push_back(1.0);
-    poly_p.push_back(p0_tmp);
-    poly_r.push_back(1.0);
-  };
-
-  // Updates vector psi and r and initializes vector p[k+1]
-  virtual void LogIteration(int k, ComplexD a, std::vector<ComplexD> betas){
-    std::cout << "PGCR::LogIteration(k = " << k << ")" << std::endl;
-    ak.push_back(a);
-    bk.push_back(betas);
-
-    // update Ap by pushing p[k] to the right
-    poly_Ap.push_back(0.0);   // need to pad the end with an element
-    poly_Ap[0] = 0.0;         // technically this should be unnecessary, as the first component is never set
-    for(int i = 0; i < k; i++){
-      poly_Ap[i+1]=poly_p[k-1][i];        // A\vec{p} = (0, \vec{p}) bc A shifts components of p to the right
-    }
-
-    // update psi_{k+1} --> psi_k + a_k p_k
-    polynomial.push_back(0.0);
-    for(int i = 0; i < k; i++) {
-      polynomial[i] += a * poly_p[k-1][i];
-    }
-    {
-      std::vector<std::complex<double>> poly_stdcmplx(polynomial.begin(), polynomial.end());
-      PF.data.push_back(poly_stdcmplx);
-    }
-
-    //  r_{k+1} --> r_k - a_k A p_k
-    //  p_{k+1} --> r_k + \sum_{i=0}^k \beta_{ik} p_i, input betas = (\beta_{ik})_i
-    poly_r.push_back(0.0);        // should be of size k+1 if we start with k = 1
-    std::vector<ComplexD> p_next (k + 1, ComplexD(0.0));     // p_{k+1} = same size as r_{k+1}
-    for(int i = 0; i < k + 1; i++){
-      poly_r[i] = poly_r[i] - a * poly_Ap[i];     // update r_{k+1} --> r_k - \alpha_k A p_k
-      p_next[i] = poly_r[i];                 // init new vector as r_{k+1}
-    }
-
-    // p_{k+1} --> p_{k+1} + \sum_i \beta_{ij} p_i
-    int nbeta = betas.size();
-    std::cout << "Betas: " << betas << std::endl;
-    for (int j = 0; j < nbeta; j++) {
-      for (int i = 0; i < j+1; i++) {
-        p_next[i] += betas[j] * poly_p[j][i];
-      }
-    }
-    poly_p.push_back(p_next);                 // add p_{k+1} to the list of p's
-  }
-
-  virtual void LogComplete(std::vector<ComplexD>& alphas, std::vector<std::vector<ComplexD>>& betas) {
-    /** Logs all alphas and betas to complete the iterations. */
-    std::cout << "PGCR::LogComplete() "<<std::endl;
-    for (int i = 0; i < alphas.size(); i++) {
-      PF.alphas.push_back(std::complex<double>(alphas[i].real(), alphas[i].imag()));
-      std::vector<std::complex<double>> beta_stdcmplx(betas[i].begin(), betas[i].end());
-      PF.betas.push_back(beta_stdcmplx);
-    }
-  };
-
-};
-
 NAMESPACE_END(Grid);
+
+#undef GCRLogLevel
 #endif

Grid/algorithms/iterative/RestartedLanczosBidiagonalization.h

@@ -1,753 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./Grid/algorithms/iterative/RestartedLanczosBidiagonalization.h
-
-Copyright (C) 2015
-
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_RESTARTED_LANCZOS_BIDIAGONALIZATION_H
-#define GRID_RESTARTED_LANCZOS_BIDIAGONALIZATION_H
-
-NAMESPACE_BEGIN(Grid);
-
-/**
- * Implicitly Restarted Lanczos Bidiagonalization (IRLBA)
- *
- * Computes the p largest (or p smallest) singular triplets of a linear
- * operator A using the Golub-Kahan-Lanczos bidiagonalization with implicit
- * restart via thick-restart / QR shifts.
- *
- * Algorithm (Baglama & Reichel, SIAM J. Sci. Comput. 27(1):19-42, 2005):
- *
- *   Outer loop:
- *     1. Extend the p-step (or seed) bidiagonalization to k steps:
- *           A  V_k = U_k B_k
- *           A^dag U_k = V_k B_k^T + beta_{k+1} v_{k+1} e_k^T
- *     2. Compute SVD:  B_k = X Sigma Y^T
- *     3. Check convergence of the p desired singular values via
- *           |beta_{k+1} * y_{k,i}|  <  tol * sigma_i
- *        where y_{k,i} is the last component of the i-th right singular vector.
- *     4. Apply k-p implicit QR shifts to implicitly compress the basis
- *        to p steps (Sorensen-Lehoucq thick restart):
- *           B_p^+ = X_p^T B_k Y_p   (upper bidiagonal, p x p)
- *        and update the lattice vectors:
- *           V_p^+ = V_k Y_p
- *           U_p^+ = U_k X_p
- *        The new residual coupling is
- *           beta_p^+ v_{p+1}^+ = beta_{k+1} v_{k+1} * (e_k^T Y_p)_p
- *                               + B_k(p,p+1) * (orthogonal tail from QR)
- *     5. Go to step 1.
- *
- * Template parameter
- * ------------------
- *   Field : lattice field type (must support Grid algebra operations)
- *
- * Usage
- * -----
- *   RestartedLanczosBidiagonalization<Field> irlba(Linop, grid, p, k, tol, maxIter);
- *   irlba.run(src);
- *   // Results available via getters.
- */
-template <class Field>
-class RestartedLanczosBidiagonalization {
-
-public:
-  LinearOperatorBase<Field> &Linop;
-  GridBase *Grid;
-
-  int    Nk;       // number of desired singular triplets
-  int    Nm;       // Lanczos basis size (Nm > Nk)
-  RealD  Tolerance;
-  int    MaxIter;
-  bool   largest; // if true, target largest singular values; otherwise smallest
-
-  // Converged singular triplets (filled after run())
-  std::vector<RealD>  singularValues;   // sigma_0 >= sigma_1 >= ...
-  std::vector<Field>  leftVectors;      // approximate left singular vectors
-  std::vector<Field>  rightVectors;     // approximate right singular vectors
-
-private:
-  // Working bases (size up to Nm+1)
-  std::vector<Field>  V;    // right Lanczos vectors
-  std::vector<Field>  U;    // left  Lanczos vectors
-  std::vector<RealD>  alpha;
-  std::vector<RealD>  beta;
-
-  // After a thick restart, the column at index restart_col of U^dag A V
-  // has extra non-zero entries (rows 0..restart_col-2) beyond what the
-  // upper bidiagonal captures.  fvec[j] = <U[j] | A V[restart_col]> for
-  // j = 0..restart_col-1.  (fvec[restart_col-1] == beta[restart_col-1].)
-  // reset_col == -1 means no restart has occurred yet (pure bidiagonal).
-  std::vector<RealD>  fvec;
-  int                 restart_col;
-
-public:
-
-  RestartedLanczosBidiagonalization(LinearOperatorBase<Field> &_Linop,
-                                    GridBase *_Grid,
-                                    int _Nk, int _Nm,
-                                    RealD _tol   = 1.0e-8,
-                                    int   _maxIt = 300,
-                                    bool  _largest = true)
-    : Linop(_Linop), Grid(_Grid),
-      Nk(_Nk), Nm(_Nm),
-      Tolerance(_tol), MaxIter(_maxIt),
-      largest(_largest)
-  {
-    assert(Nm > Nk);
-  }
-
-  /**
-   * Run IRLBA starting from src.
-   * On exit, singularValues, leftVectors, rightVectors are filled with
-   * the Nk converged singular triplets.
-   */
-  void run(const Field &src)
-  {
-    assert(norm2(src) > 0.0);
-
-    singularValues.clear();
-    leftVectors.clear();
-    rightVectors.clear();
-
-    // Allocate working bases
-    V.clear(); U.clear();
-    alpha.clear(); beta.clear();
-    fvec.clear(); restart_col = -1;
-    V.reserve(Nm + 1);
-    U.reserve(Nm);
-
-    // Seed: v_0 = src / ||src||
-    Field vtmp(Grid);
-    vtmp = src;
-    RealD nrm = std::sqrt(norm2(vtmp));
-    vtmp = (1.0 / nrm) * vtmp;
-    V.push_back(vtmp);
-
-    int pStart = 0;  // current basis size at start of extension
-    RealD betaRestart = 0.0; // coupling from previous restart
-
-    for (int iter = 0; iter < MaxIter; ++iter) {
-
-      // ----------------------------------------------------------------
-      // Step 1: extend from pStart steps to Nm steps
-      // ----------------------------------------------------------------
-      extendBasis(pStart, Nm, betaRestart);
-//      verify();
-
-      // ----------------------------------------------------------------
-      // Step 2: SVD of the Nm x Nm B matrix.
-      // iter=0 (pStart==0): B is exactly bidiagonal — use buildBidiagonal.
-      // iter>0 (pStart==Nk): after a thick restart, column restart_col of
-      // U^dag A V has extra off-diagonal entries captured by fvec; use
-      // buildFullB so the Ritz values and restart vectors are computed from
-      // the exact projected matrix A V = U B_full.
-      // ----------------------------------------------------------------
-      Eigen::MatrixXd B = (pStart == 0) ? buildBidiagonal(Nm) : buildFullB(Nm);
-      Eigen::JacobiSVD<Eigen::MatrixXd> svd(B,
-          Eigen::ComputeThinU | Eigen::ComputeThinV);
-
-      Eigen::VectorXd sigma = svd.singularValues();  // descending
-      Eigen::MatrixXd X     = svd.matrixU();          // Nm x Nm left SVecs of B
-      Eigen::MatrixXd Y     = svd.matrixV();          // Nm x Nm right SVecs of B
-
-      // If targeting smallest, reorder so desired ones come first
-      Eigen::VectorXi order = sortOrder(sigma);
-
-      // ----------------------------------------------------------------
-      // Step 3: check convergence of the Nk desired singular values
-      // ----------------------------------------------------------------
-      RealD betaK = beta.back();  // beta_{k+1}
-      // In our convention A V = U B (exact), the residual is in the A^dag
-      // direction: A^dag u_j - sigma_j v_j = betaK * X[Nm-1,j] * V[Nm].
-      // Convergence criterion: |betaK * X[Nm-1, idx]| < tol * sigma_idx.
-      int nconv = 0;
-      for (int i = 0; i < Nk; ++i) {
-        int idx = order(i);
-        RealD res = std::abs(betaK * X(Nm - 1, idx));
-        RealD thr = Tolerance * std::max(sigma(idx), 1.0e-14);
-        std::cout << GridLogMessage
-                  << "IRLBA iter " << iter
-                  << "  sigma[" << i << "] = " << sigma(idx)
-                  << "  res = " << res
-                  << "  thr = " << thr << std::endl;
-        if (res < thr) ++nconv;
-        else break;  // residuals not strictly ordered but break is conservative
-      }
-
-      if (nconv >= Nk) {
-        std::cout << GridLogMessage
-                  << "IRLBA converged: " << nconv << " singular values after "
-                  << iter + 1 << " iterations." << std::endl;
-        // Collect converged triplets
-        extractTriplets(Nm, sigma, X, Y, order, Nk);
-        return;
-      }
-
-      // ----------------------------------------------------------------
-      // Step 4: implicit restart — compress to Nk steps
-      // ----------------------------------------------------------------
-      implicitRestart(Nm, Nk, sigma, X, Y, order, betaK, betaRestart);
-//      verify();
-
-      // Lucky breakdown: exact invariant subspace found; convergence is exact.
-      // B_p^+ = diag(alpha[0..Nk-1]); extract directly from restart basis.
-      if (betaRestart < 1.0e-14) {
-        std::cout << GridLogMessage
-                  << "IRLBA: lucky breakdown after restart (betaRestart = 0)."
-                  << " Extracting " << Nk << " exact Ritz triplets." << std::endl;
-        // Re-run SVD on the p-step diagonal B^+ to get sorted Ritz triplets.
-        Eigen::MatrixXd Bp = buildBidiagonal(Nk);
-        Eigen::JacobiSVD<Eigen::MatrixXd> svdp(Bp,
-            Eigen::ComputeThinU | Eigen::ComputeThinV);
-        Eigen::VectorXi ordp = sortOrder(svdp.singularValues());
-        extractTriplets(Nk, svdp.singularValues(), svdp.matrixU(),
-                        svdp.matrixV(), ordp, Nk);
-        return;
-      }
-
-      pStart = Nk;
-    }
-
-    std::cout << GridLogMessage
-              << "IRLBA: did not converge in " << MaxIter
-              << " iterations. Returning best approximations." << std::endl;
-
-    // Return best available approximations
-    Eigen::MatrixXd B = buildFullB((int)alpha.size());
-    Eigen::JacobiSVD<Eigen::MatrixXd> svd(B,
-        Eigen::ComputeThinU | Eigen::ComputeThinV);
-    Eigen::VectorXd sigma = svd.singularValues();
-    Eigen::MatrixXd X     = svd.matrixU();
-    Eigen::MatrixXd Y     = svd.matrixV();
-    Eigen::VectorXi order = sortOrder(sigma);
-    int nout = std::min(Nk, (int)alpha.size());
-    extractTriplets((int)alpha.size(), sigma, X, Y, order, nout);
-  }
-
-  /* Getters */
-  int getNk() const { return (int)singularValues.size(); }
-  const std::vector<RealD>&  getSingularValues() const { return singularValues; }
-  const std::vector<Field>&  getLeftVectors()    const { return leftVectors; }
-  const std::vector<Field>&  getRightVectors()   const { return rightVectors; }
-
-  /**
-   * Print B_k and U^dag A V to verify the bidiagonalization relation
-   *   A V_m = U_m B_m   (exact in our GK convention)
-   * On the first call (pStart=0), max|B - U^dag A V| should be ~machine epsilon.
-   * After a restart and extension, the column p of U^dag A V deviates from B
-   * by O(betaK): this is expected because the thick restart breaks the Krylov
-   * structure at column p, introducing off-diagonal terms proportional to betaK.
-   * These terms vanish as betaK -> 0 (convergence), so the algorithm is correct.
-   */
-  void verify()
-  {
-    int m  = (int)alpha.size();
-    int nU = (int)U.size();
-    int nV = (int)V.size();
-    if (m == 0) { std::cout << GridLogMessage << "IRLBA verify: empty basis" << std::endl; return; }
-
-    // Build reference matrix Bref (nU x nV):
-    //   Columns 0..m-1 : buildFullB(m)  (bidiagonal + fvec column at restart_col)
-    //   Column  m      : residual column, two cases:
-    //     (a) restart_col == m (right after implicitRestart, before extendBasis):
-    //         V[m] = sgn*V_old[Nm], so <U[i]|A|V[m]> = fvec[i] for all i
-    //     (b) otherwise (pure GK or after extendBasis):
-    //         only entry (m-1, m) = beta[m-1]  (GK recurrence residual)
-    Eigen::MatrixXd Bref = Eigen::MatrixXd::Zero(nU, nV);
-    {
-      Eigen::MatrixXd Bfull = buildFullB(m);
-      int cols = std::min(m, nV);
-      Bref.block(0, 0, m, cols) = Bfull.block(0, 0, m, cols);
-    }
-    if (nV > m && m > 0) {
-      if (restart_col == m && (int)fvec.size() == m) {
-        // Case (a): right after implicitRestart
-        for (int i = 0; i < m; ++i) Bref(i, m) = fvec[i];
-      } else if ((int)beta.size() >= m) {
-        // Case (b): standard GK residual column
-        Bref(m - 1, m) = beta[m - 1];
-      }
-    }
-
-    // Compute M[i,j] = <U[i] | A | V[j]>
-    Eigen::MatrixXd M = Eigen::MatrixXd::Zero(nU, nV);
-    Field Avj(Grid);
-    for (int j = 0; j < nV; ++j) {
-      Linop.Op(V[j], Avj);
-      for (int i = 0; i < nU; ++i) {
-        ComplexD ip = innerProduct(U[i], Avj);
-        M(i, j) = ip.real();
-      }
-    }
-
-    // Print Bref
-    std::cout << GridLogMessage
-              << "IRLBA verify: Bref (" << nU << "x" << nV << "):" << std::endl;
-    for (int i = 0; i < nU; ++i) {
-      std::cout << GridLogMessage << "  row " << i << ": ";
-      for (int j = 0; j < nV; ++j) std::cout << Bref(i,j) << " ";
-      std::cout << std::endl;
-    }
-
-    // Print U^dag A V
-    std::cout << GridLogMessage
-              << "IRLBA verify: U^dag A V (" << nU << "x" << nV << "):" << std::endl;
-    for (int i = 0; i < nU; ++i) {
-      std::cout << GridLogMessage << "  row " << i << ": ";
-      for (int j = 0; j < nV; ++j) std::cout << M(i,j) << " ";
-      std::cout << std::endl;
-    }
-
-    // Max deviation over the full nU x nV matrix
-    RealD maxdev = (Bref - M).cwiseAbs().maxCoeff();
-    std::cout << GridLogMessage
-              << "IRLBA verify: max|Bref - U^dag A V| = " << maxdev << std::endl;
-
-    // Beta
-    std::cout << GridLogMessage << "IRLBA verify: beta[0.." << (int)beta.size()-1 << "] = ";
-    for (auto b : beta) std::cout << b << " ";
-    std::cout << std::endl;
-  }
-
-private:
-
-  // ------------------------------------------------------------------
-  // Build the m x m upper-bidiagonal matrix from alpha[0..m-1], beta[0..m-2]
-  // ------------------------------------------------------------------
-  Eigen::MatrixXd buildBidiagonal(int m) const
-  {
-    Eigen::MatrixXd B = Eigen::MatrixXd::Zero(m, m);
-    for (int k = 0; k < m; ++k) {
-      B(k, k) = alpha[k];
-      if (k + 1 < m && k < (int)beta.size())
-        B(k, k + 1) = beta[k];
-    }
-    return B;
-  }
-
-  // ------------------------------------------------------------------
-  // Build the full m x m B matrix, including the non-bidiagonal column
-  // at restart_col that arises after a thick restart.
-  //
-  // After restart, A V[restart_col] has projections onto all U[0..restart_col-1]
-  // (not just U[restart_col-1]).  These are stored in fvec[0..restart_col-1]
-  // and make column restart_col of U^dag A V non-bidiagonal.
-  // ------------------------------------------------------------------
-  Eigen::MatrixXd buildFullB(int m) const
-  {
-    Eigen::MatrixXd B = buildBidiagonal(m);
-    if (restart_col >= 0 && restart_col < m && (int)fvec.size() > 0) {
-      for (int j = 0; j < restart_col && j < (int)fvec.size(); ++j){
-        B(j, restart_col) = fvec[j];
-        std::cout << GridLogDebug << "buildFullB: B  " <<j<<" "<<restart_col<<B(j, restart_col)<<std::endl;
-      }
-    }
-    return B;
-  }
-
-  // ------------------------------------------------------------------
-  // Return a permutation vector that puts the desired Nk singular values
-  // first (largest first if largest==true, smallest first otherwise).
-  // Eigen's JacobiSVD already returns sigma in descending order, so for
-  // largest we just return 0,1,...,m-1; for smallest we reverse.
-  // ------------------------------------------------------------------
-  Eigen::VectorXi sortOrder(const Eigen::VectorXd &sigma) const
-  {
-    int m = (int)sigma.size();
-    Eigen::VectorXi ord(m);
-    if (largest) {
-      for (int i = 0; i < m; ++i) ord(i) = i;
-    } else {
-      for (int i = 0; i < m; ++i) ord(i) = m - 1 - i;
-    }
-    return ord;
-  }
-
-  // ------------------------------------------------------------------
-  // Extend the Lanczos bidiagonalization from pStart to kEnd steps.
-  // On first call pStart==0 (V[0] already set).
-  // On restart calls V[0..pStart], U[0..pStart-1], alpha[0..pStart-1],
-  // beta[0..pStart-1] are already set; betaRestart is the coupling
-  // beta_{pStart} that drives the first new U step.
-  // ------------------------------------------------------------------
-  void extendBasis(int pStart, int kEnd, RealD betaRestart)
-  {
-    // Truncate containers to pStart (Lattice has no default constructor)
-    if ((int)V.size() > pStart + 1) V.erase(V.begin() + pStart + 1, V.end());
-    if ((int)U.size() > pStart)     U.erase(U.begin() + pStart,     U.end());
-    alpha.resize(pStart);
-    beta.resize(pStart);
-
-    Field p(Grid), r(Grid);
-
-    for (int k = pStart; k < kEnd; ++k) {
-
-      // p = A v_k
-      Linop.Op(V[k], p);
-
-      // Remove previous left vector coupling
-      if (k > 0) {
-        p = p - beta[k - 1] * U[k - 1];
-      }
-      // On the first step after a restart, beta[pStart-1] was already set;
-      // but V[pStart] was already constructed including the beta correction,
-      // so no extra subtraction needed here beyond the standard recurrence.
-
-      // Reorthogonalize p against U, then alpha_k = ||p||, u_k = p/alpha_k
-      reorthogonalize(p, U);
-      RealD ak = std::sqrt(norm2(p));
-      if (ak < 1.0e-14) {
-        std::cout << GridLogMessage
-                  << "IRLBA extendBasis: lucky breakdown at step " << k
-                  << " (alpha = " << ak << ")" << std::endl;
-        alpha.push_back(ak);
-        Field zero(Grid); zero = Zero();
-        U.push_back(zero);
-        beta.push_back(0.0);
-        V.push_back(zero);
-        break;
-      }
-      alpha.push_back(ak);
-
-      Field u(Grid);
-      u = (1.0 / ak) * p;
-      U.push_back(u);
-
-      // r = A^dag u_k - alpha_k v_k, reorthogonalize, then beta_{k+1} = ||r||
-      Linop.AdjOp(U[k], r);
-      r = r - ak * V[k];
-      reorthogonalize(r, V);
-
-      RealD bk = std::sqrt(norm2(r));
-      beta.push_back(bk);
-
-      std::cout << GridLogMessage
-                << "IRLBA extend step " << k
-                << "  alpha = " << ak
-                << "  beta  = " << bk << std::endl;
-
-      // Always push v_{k+1} (needed as residual direction for restart)
-      if (bk < 1.0e-14) {
-        std::cout << GridLogMessage
-                  << "IRLBA extendBasis: lucky breakdown (beta = 0) at step "
-                  << k << std::endl;
-        Field zero(Grid); zero = Zero();
-        V.push_back(zero);
-        break;
-      }
-      Field vnext(Grid);
-      vnext = (1.0 / bk) * r;
-      V.push_back(vnext);
-
-      if (k == kEnd - 1) break;  // v_{k+1} pushed above; stop here
-    }
-  }
-
-public:
-  // ------------------------------------------------------------------
-  // Block reorthogonalization helpers.
-  // Declared public because CUDA extended lambdas cannot live inside
-  // private/protected member functions.
-  //
-  // batchInnerProducts: computes c[j] = <basis[j], vec> for all j
-  //   in a single GPU pass (one accelerator_barrier instead of n).
-  //   Queues n pairs of (per-site kernel, reduceKernel) to computeStream
-  //   without intermediate CPU syncs, then syncs once at the end.
-  //
-  // batchUpdate: computes vec -= sum_j c[j]*basis[j] in one GPU kernel.
-  //
-  // reorthogonalize: two-pass Classical Gram-Schmidt (CGS2) using the
-  //   two helpers above.  Each pass costs 2 GPU syncs (1 IP + 1 update)
-  //   instead of 2n syncs per pass in the old sequential MGS.
-  // ------------------------------------------------------------------
-
-  void batchInnerProducts(const Field &vec,
-                          const std::vector<Field> &basis,
-                          std::vector<ComplexD> &c)
-  {
-    int n = (int)basis.size();
-    c.resize(n);
-    if (n == 0) return;
-
-    typedef typename Field::vector_object         vobj;
-    typedef decltype(innerProduct(vobj(), vobj())) inner_t;
-    typedef decltype(basis[0].View(AcceleratorRead)) View;
-
-    GridBase *grid = vec.Grid();
-    uint64_t oSites = grid->oSites();
-    uint64_t nsimd  = grid->Nsimd();
-
-    // all_ip[j * oSites + ss] = per-site inner product of basis[j] and vec at site ss.
-    // Layout: n contiguous blocks of oSites each.
-    deviceVector<inner_t> all_ip((uint64_t)n * oSites);
-    inner_t *all_ip_p = &all_ip[0];
-
-    hostVector<View>   h_basis_v(n);
-    deviceVector<View> d_basis_v(n);
-    for (int j = 0; j < n; ++j) {
-      h_basis_v[j] = basis[j].View(AcceleratorRead);
-      acceleratorPut(d_basis_v[j], h_basis_v[j]);
-    }
-    View *basis_vp = &d_basis_v[0];
-
-    // Queue n per-site kernels to the accelerator stream — no intermediate barriers.
-    autoView(vec_v, vec, AcceleratorRead);
-    for (int j = 0; j < n; ++j) {
-      int      jj      = j;
-      uint64_t oSites_ = oSites;
-      accelerator_for(ss, oSites, nsimd, {
-        auto x = coalescedRead(basis_vp[jj][ss]);
-        auto y = coalescedRead(vec_v[ss]);
-        coalescedWrite(all_ip_p[jj * oSites_ + ss], innerProduct(x, y));
-      });
-    }
-
-    // ONE sync after all n kernels
-    accelerator_barrier();
-
-    // Copy all per-site results to host
-    hostVector<inner_t> all_ip_h((uint64_t)n * oSites);
-    acceleratorCopyFromDevice(all_ip_p, &all_ip_h[0], (uint64_t)n * oSites * sizeof(inner_t));
-
-    // Reduce on host: sum over oSites, then collapse SIMD lanes via Reduce(TensorRemove(...))
-    // TensorRemove strips the iSinglet tensor wrapper to expose the SIMD scalar type.
-    // Reduce sums all nsimd lanes and returns a plain scalar (RealD or ComplexD).
-    std::vector<ComplexD> raw(n);
-    for (int j = 0; j < n; ++j) {
-      inner_t sum = Zero();
-      for (uint64_t ss = 0; ss < oSites; ++ss)
-        sum += all_ip_h[(uint64_t)j * oSites + ss];
-      raw[j] = ComplexD(Reduce(TensorRemove(sum)));
-    }
-    grid->GlobalSumVector(&raw[0], n);
-    for (int j = 0; j < n; ++j) c[j] = raw[j];
-
-    for (int j = 0; j < n; ++j) h_basis_v[j].ViewClose();
-  }
-
-  void batchUpdate(Field &vec,
-                   const std::vector<Field> &basis,
-                   const std::vector<ComplexD> &c)
-  {
-    int n = (int)basis.size();
-    if (n == 0) return;
-
-    typedef typename Field::vector_object vobj;
-    typedef decltype(basis[0].View(AcceleratorRead)) View;
-
-    GridBase *grid = vec.Grid();
-    uint64_t oSites = grid->oSites();
-    uint64_t nsimd  = grid->Nsimd();
-
-    // Split complex coefficients into real/imag double arrays on device.
-    // Using doubles avoids potential ComplexD-device-code compatibility issues.
-    hostVector<double>   h_re(n), h_im(n);
-    deviceVector<double> d_re(n), d_im(n);
-    for (int k = 0; k < n; ++k) {
-      h_re[k] = c[k].real();
-      h_im[k] = c[k].imag();
-    }
-    acceleratorCopyToDevice(&h_re[0], &d_re[0], n * sizeof(double));
-    acceleratorCopyToDevice(&h_im[0], &d_im[0], n * sizeof(double));
-    double *re_p = &d_re[0];
-    double *im_p = &d_im[0];
-
-    // Basis views
-    hostVector<View>   h_basis_v(n);
-    deviceVector<View> d_basis_v(n);
-    for (int k = 0; k < n; ++k) {
-      h_basis_v[k] = basis[k].View(AcceleratorRead);
-      acceleratorPut(d_basis_v[k], h_basis_v[k]);
-    }
-    View *basis_vp = &d_basis_v[0];
-
-    // Single kernel: vec[ss] -= sum_k (re[k] + i*im[k]) * basis[k][ss]
-    autoView(vec_v, vec, AcceleratorWrite);
-    accelerator_for(ss, oSites, nsimd, {
-      auto v = coalescedRead(vec_v[ss]);
-      for (int k = 0; k < n; ++k) {
-        auto b = coalescedRead(basis_vp[k][ss]);
-        v = v - re_p[k] * b - timesI(im_p[k] * b);
-      }
-      coalescedWrite(vec_v[ss], v);
-    });
-
-    for (int k = 0; k < n; ++k) h_basis_v[k].ViewClose();
-  }
-
-  // ------------------------------------------------------------------
-  // Full reorthogonalization using two-pass Classical Gram-Schmidt (CGS2).
-  // Each pass calls batchInnerProducts (1 GPU sync) + batchUpdate (1 sync),
-  // replacing the old 2n GPU syncs per pass from sequential MGS.
-  // ------------------------------------------------------------------
-  void reorthogonalize(Field &vec, const std::vector<Field> &basis)
-  {
-    if (basis.empty()) return;
-    std::vector<ComplexD> c;
-    for (int pass = 0; pass < 2; ++pass) {
-      batchInnerProducts(vec, basis, c);
-      batchUpdate(vec, basis, c);
-    }
-  }
-
-  // ------------------------------------------------------------------
-  // Implicit restart: given the Nm-step bidiagonalization and its SVD,
-  // compress to Nk steps via implicit QR shifts applied to B_k.
-  //
-  // The "shifts" are the Nm - Nk singular values we want to deflate
-  // (those NOT in the desired set).  We apply them as implicit QR steps
-  // to the bidiagonal matrix, then update the lattice bases accordingly.
-  //
-  // After this call:
-  //   V[0..Nk],  U[0..Nk-1],  alpha[0..Nk-1],  beta[0..Nk-1]  are updated.
-  //   betaRestart  ← new beta_Nk coupling for the next extension.
-  // ------------------------------------------------------------------
-  void implicitRestart(int k, int p,
-                       const Eigen::VectorXd &sigma,
-                       const Eigen::MatrixXd &X,
-                       const Eigen::MatrixXd &Y,
-                       const Eigen::VectorXi &order,
-                       RealD betaK,
-                       RealD &betaRestart)
-  {
-    // Thick restart (Baglama & Reichel, Sec. 2.2):
-    //
-    // Given B_k = X Sigma Y^T, define the new p-step basis by:
-    //   V^+_i = V_k * y_{order(i)}      (right sing. vec. of B_k)
-    //   U^+_i = U_k * x_{order(i)}      (left  sing. vec. of B_k)
-    //
-    // Then A V^+_i = A V_k y_{order(i)} = U_k B_k y_{order(i)}
-    //             = sigma_{order(i)} U_k x_{order(i)} = sigma_{order(i)} U^+_i
-    //
-    // So B_p^+ = diag(sigma_{order(0)}, ..., sigma_{order(p-1)}) — DIAGONAL,
-    // all internal betas are zero.
-    //
-    // The residual coupling comes from A^dag U_k = V_k B_k^T + betaK V[k] e_{k-1}^T:
-    //   A^dag U^+_{p-1} - sigma_{order(p-1)} V^+_{p-1}
-    //     = V_k (B_k^T x_{order(p-1)} - sigma_{order(p-1)} y_{order(p-1)})
-    //       + betaK * X(k-1, order(p-1)) * V[k]
-    //     = betaK * X(k-1, order(p-1)) * V[k]   (since B_k^T x_j = sigma_j y_j)
-    //
-    // Therefore: betaRestart = |betaK * X(k-1, order(p-1))|
-    //            V[p] = sign(X(k-1, order(p-1))) * V[k]
-
-    // ---- Build new lattice vectors ----
-    std::vector<Field> Vnew, Unew;
-    Vnew.reserve(p + 1);
-    Unew.reserve(p);
-
-    for (int i = 0; i < p; ++i) {
-      int idx = order(i);
-      Field vi(Grid); vi = Zero();
-      for (int j = 0; j < k; ++j)
-        vi = vi + Y(j, idx) * V[j];
-      Vnew.push_back(vi);
-    }
-
-    for (int i = 0; i < p; ++i) {
-      int idx = order(i);
-      Field ui(Grid); ui = Zero();
-      for (int j = 0; j < k; ++j)
-        ui = ui + X(j, idx) * U[j];
-      Unew.push_back(ui);
-    }
-
-    // New v_{p} (0-indexed: V[p]) = sign * V[k]
-    // From A^dag U_k = V_k B_k^T + betaK V[k] e_{k-1}^T:
-    //   A^dag U^+_j - sigma_j V^+_j = betaK * X(k-1, order(j)) * V[k]
-    // The last Ritz pair (j=p-1) defines betaRestart and the sign of V[p].
-    // All p couplings (j=0..p-1) are stored in fvec so that buildFullB can
-    // reconstruct the exact column p of U^dag A V after the next extension.
-    RealD coeff = betaK * X(k - 1, order(p - 1));
-    betaRestart  = std::abs(coeff);
-    RealD sgn = (coeff >= 0.0) ? 1.0 : -1.0;
-
-    fvec.resize(p);
-    for (int j = 0; j < p; ++j)
-      fvec[j] = betaK * X(k - 1, order(j)) * sgn;
-    // fvec[p-1] == betaRestart by construction
-    restart_col = p;
-
-    Field vp(Grid);
-    if (betaRestart > 1.0e-14) {
-      vp = sgn * V[k];
-    } else {
-      betaRestart = 0.0;
-      vp = Zero();
-    }
-    Vnew.push_back(vp);  // V[p]
-
-    // ---- New alpha, beta ----
-    // B_p^+ is diagonal: alpha^+_i = sigma_{order(i)}, all internal beta = 0
-    std::vector<RealD> alpha_new(p), beta_new(p);
-    for (int i = 0; i < p; ++i) alpha_new[i] = sigma(order(i));
-    for (int i = 0; i < p - 1; ++i) beta_new[i] = 0.0;
-    beta_new[p - 1] = betaRestart;
-
-    // ---- Commit new state ----
-    V = Vnew;
-    U = Unew;
-    alpha = alpha_new;
-    beta  = beta_new;
-
-    std::cout << GridLogMessage
-              << "IRLBA restart: compressed to " << p << " steps,"
-              << "  new beta_p = " << betaRestart << std::endl;
-  }
-
-  // ------------------------------------------------------------------
-  // Extract the desired singular triplets into the public output vectors.
-  // ------------------------------------------------------------------
-  void extractTriplets(int m,
-                       const Eigen::VectorXd &sigma,
-                       const Eigen::MatrixXd &X,
-                       const Eigen::MatrixXd &Y,
-                       const Eigen::VectorXi &order,
-                       int nout)
-  {
-    singularValues.resize(nout);
-    leftVectors.clear();   leftVectors.reserve(nout);
-    rightVectors.clear();  rightVectors.reserve(nout);
-
-    for (int i = 0; i < nout; ++i) {
-      int idx = order(i);
-      singularValues[i] = sigma(idx);
-
-      // Left singular vector of A:  svec_L = U_m * x_i
-      Field svL(Grid); svL = Zero();
-      for (int j = 0; j < m && j < (int)U.size(); ++j)
-        svL = svL + X(j, idx) * U[j];
-      leftVectors.push_back(svL);
-
-      // Right singular vector of A:  svec_R = V_m * y_i
-      Field svR(Grid); svR = Zero();
-      for (int j = 0; j < m && j < (int)V.size(); ++j)
-        svR = svR + Y(j, idx) * V[j];
-      rightVectors.push_back(svR);
-    }
-  }
-};
-
-NAMESPACE_END(Grid);
-#endif

Grid/algorithms/iterative/SimpleLanczos.h

@@ -1,931 +0,0 @@
-    /*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./lib/algorithms/iterative/ImplicitlyRestartedLanczos.h
-
-    Copyright (C) 2015
-
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-#ifndef GRID_LANC_H
-#define GRID_LANC_H
-
-#include <string.h>		//memset
-
-#ifdef USE_LAPACK
-#ifdef USE_MKL
-#include<mkl_lapack.h>
-#else
-void LAPACK_dstegr (char *jobz, char *range, int *n, double *d, double *e,
-		    double *vl, double *vu, int *il, int *iu, double *abstol,
-		    int *m, double *w, double *z, int *ldz, int *isuppz,
-		    double *work, int *lwork, int *iwork, int *liwork,
-		    int *info);
-//#include <lapacke/lapacke.h>
-#endif
-#endif
-
-//#include <Grid/algorithms/densematrix/DenseMatrix.h>
-
-// eliminate temorary vector in calc()
-#define MEM_SAVE
-
-namespace Grid
-{
-
-  struct Bisection
-  {
-
-#if 0
-    static void get_eig2 (int row_num, std::vector < RealD > &ALPHA,
-			  std::vector < RealD > &BETA,
-			  std::vector < RealD > &eig)
-    {
-      int i, j;
-        std::vector < RealD > evec1 (row_num + 3);
-        std::vector < RealD > evec2 (row_num + 3);
-      RealD eps2;
-        ALPHA[1] = 0.;
-        BETHA[1] = 0.;
-      for (i = 0; i < row_num - 1; i++)
-	{
-	  ALPHA[i + 1] = A[i * (row_num + 1)].real ();
-	  BETHA[i + 2] = A[i * (row_num + 1) + 1].real ();
-	}
-      ALPHA[row_num] = A[(row_num - 1) * (row_num + 1)].real ();
-        bisec (ALPHA, BETHA, row_num, 1, row_num, 1e-10, 1e-10, evec1, eps2);
-        bisec (ALPHA, BETHA, row_num, 1, row_num, 1e-16, 1e-16, evec2, eps2);
-
-      // Do we really need to sort here?
-      int begin = 1;
-      int end = row_num;
-      int swapped = 1;
-      while (swapped)
-	{
-	  swapped = 0;
-	  for (i = begin; i < end; i++)
-	    {
-	      if (mag (evec2[i]) > mag (evec2[i + 1]))
-		{
-		  swap (evec2 + i, evec2 + i + 1);
-		  swapped = 1;
-		}
-	    }
-	  end--;
-	  for (i = end - 1; i >= begin; i--)
-	    {
-	      if (mag (evec2[i]) > mag (evec2[i + 1]))
-		{
-		  swap (evec2 + i, evec2 + i + 1);
-		  swapped = 1;
-		}
-	    }
-	  begin++;
-	}
-
-      for (i = 0; i < row_num; i++)
-	{
-	  for (j = 0; j < row_num; j++)
-	    {
-	      if (i == j)
-		H[i * row_num + j] = evec2[i + 1];
-	      else
-		H[i * row_num + j] = 0.;
-	    }
-	}
-    }
-#endif
-
-    static void bisec (std::vector < RealD > &c,
-		       std::vector < RealD > &b,
-		       int n,
-		       int m1,
-		       int m2,
-		       RealD eps1,
-		       RealD relfeh, std::vector < RealD > &x, RealD & eps2)
-    {
-      std::vector < RealD > wu (n + 2);
-
-      RealD h, q, x1, xu, x0, xmin, xmax;
-      int i, a, k;
-
-      b[1] = 0.0;
-      xmin = c[n] - fabs (b[n]);
-      xmax = c[n] + fabs (b[n]);
-      for (i = 1; i < n; i++)
-	{
-	  h = fabs (b[i]) + fabs (b[i + 1]);
-	  if (c[i] + h > xmax)
-	    xmax = c[i] + h;
-	  if (c[i] - h < xmin)
-	    xmin = c[i] - h;
-	}
-      xmax *= 2.;
-
-      eps2 = relfeh * ((xmin + xmax) > 0.0 ? xmax : -xmin);
-      if (eps1 <= 0.0)
-	eps1 = eps2;
-      eps2 = 0.5 * eps1 + 7.0 * (eps2);
-      x0 = xmax;
-      for (i = m1; i <= m2; i++)
-	{
-	  x[i] = xmax;
-	  wu[i] = xmin;
-	}
-
-      for (k = m2; k >= m1; k--)
-	{
-	  xu = xmin;
-	  i = k;
-	  do
-	    {
-	      if (xu < wu[i])
-		{
-		  xu = wu[i];
-		  i = m1 - 1;
-		}
-	      i--;
-	    }
-	  while (i >= m1);
-	  if (x0 > x[k])
-	    x0 = x[k];
-	  while ((x0 - xu) > 2 * relfeh * (fabs (xu) + fabs (x0)) + eps1)
-	    {
-	      x1 = (xu + x0) / 2;
-
-	      a = 0;
-	      q = 1.0;
-	      for (i = 1; i <= n; i++)
-		{
-		  q =
-		    c[i] - x1 -
-		    ((q != 0.0) ? b[i] * b[i] / q : fabs (b[i]) / relfeh);
-		  if (q < 0)
-		    a++;
-		}
-//      printf("x1=%0.14e a=%d\n",x1,a);
-	      if (a < k)
-		{
-		  if (a < m1)
-		    {
-		      xu = x1;
-		      wu[m1] = x1;
-		    }
-		  else
-		    {
-		      xu = x1;
-		      wu[a + 1] = x1;
-		      if (x[a] > x1)
-			x[a] = x1;
-		    }
-		}
-	      else
-		x0 = x1;
-	    }
-	  printf ("x0=%0.14e xu=%0.14e k=%d\n", x0, xu, k);
-	  x[k] = (x0 + xu) / 2;
-	}
-    }
-  };
-
-/////////////////////////////////////////////////////////////
-// Implicitly restarted lanczos
-/////////////////////////////////////////////////////////////
-
-
-  template < class Field > class SimpleLanczos
-  {
-
-    const RealD small = 1.0e-16;
-  public:
-    int lock;
-    int get;
-    int Niter;
-    int converged;
-
-    int Nstop;			// Number of evecs checked for convergence
-    int Nk;			// Number of converged sought
-    int Np;			// Np -- Number of spare vecs in kryloc space
-    int Nm;			// Nm -- total number of vectors
-
-
-    RealD OrthoTime;
-
-    RealD eresid;
-
-//    SortEigen < Field > _sort;
-
-    LinearFunction < Field > &_Linop;
-
-//    OperatorFunction < Field > &_poly;
-
-    /////////////////////////
-    // Constructor
-    /////////////////////////
-    void init (void)
-    {
-    };
-//    void Abort (int ff, std::vector < RealD > &evals, DenseVector < Denstd::vector  < RealD > >&evecs);
-
-    SimpleLanczos (LinearFunction < Field > &Linop,	// op
-//		   OperatorFunction < Field > &poly,	// polynmial
-		   int _Nstop,	// sought vecs
-		   int _Nk,	// sought vecs
-		   int _Nm,	// spare vecs
-		   RealD _eresid,	// resid in lmdue deficit 
-		   int _Niter):	// Max iterations
-     
-      _Linop (Linop),
- //     _poly (poly),
-      Nstop (_Nstop), Nk (_Nk), Nm (_Nm), eresid (_eresid), Niter (_Niter)
-    {
-      Np = Nm - Nk;
-      assert (Np > 0);
-    };
-
-    /////////////////////////
-    // Sanity checked this routine (step) against Saad.
-    /////////////////////////
-    void RitzMatrix (std::vector < Field > &evec, int k)
-    {
-
-      if (1)
-	return;
-
-      GridBase *grid = evec[0].Grid();
-      Field w (grid);
-      std::cout << GridLogMessage << "RitzMatrix " << std::endl;
-      for (int i = 0; i < k; i++)
-	{
-	  _Linop(evec[i], w);
-//      _poly(_Linop,evec[i],w);
-	  std::cout << GridLogMessage << "[" << i << "] ";
-	  for (int j = 0; j < k; j++)
-	    {
-	      ComplexD in = innerProduct (evec[j], w);
-	      if (fabs ((double) i - j) > 1)
-		{
-		  if (abs (in) > 1.0e-9)
-		    {
-		      std::cout << GridLogMessage << "oops" << std::endl;
-		      abort ();
-		    }
-		  else
-		    std::cout << GridLogMessage << " 0 ";
-		}
-	      else
-		{
-		  std::cout << GridLogMessage << " " << in << " ";
-		}
-	    }
-	  std::cout << GridLogMessage << std::endl;
-	}
-    }
-
-    void step (std::vector < RealD > &lmd,
-	       std::vector < RealD > &lme,
-	       Field & last, Field & current, Field & next, uint64_t k)
-    {
-      if (lmd.size () <= k)
-	lmd.resize (k + Nm);
-      if (lme.size () <= k)
-	lme.resize (k + Nm);
-
-
-//      _poly(_Linop,current,next );   // 3. wk:=Avk−βkv_{k−1}
-      _Linop(current, next);	// 3. wk:=Avk−βkv_{k−1}
-      if (k > 0)
-	{
-	  next -= lme[k - 1] * last;
-	}
-//      std::cout<<GridLogMessage << "<last|next>" << innerProduct(last,next) <<std::endl;
-
-      ComplexD zalph = innerProduct (current, next);	// 4. αk:=(wk,vk)
-      RealD alph = real (zalph);
-
-      next = next - alph * current;	// 5. wk:=wk−αkvk
-//      std::cout<<GridLogMessage << "<current|next>" << innerProduct(current,next) <<std::endl;
-
-      RealD beta = normalise (next);	// 6. βk+1 := ∥wk∥2. If βk+1 = 0 then Stop
-      // 7. vk+1 := wk/βk+1
-//       norm=beta;
-
-      int interval = Nm / 100 + 1;
-      if ((k % interval) == 0)
-	std::
-	  cout << GridLogMessage << k << " : alpha = " << zalph << " beta " <<
-	  beta << std::endl;
-      const RealD tiny = 1.0e-20;
-      if (beta < tiny)
-	{
-	  std::cout << GridLogMessage << " beta is tiny " << beta << std::
-	    endl;
-	}
-      lmd[k] = alph;
-      lme[k] = beta;
-
-    }
-
-    void qr_decomp (std::vector < RealD > &lmd,
-		    std::vector  < RealD > &lme,
-		    int Nk,
-		    int Nm,
-		    std::vector  < RealD > &Qt, RealD Dsh, int kmin, int kmax)
-    {
-      int k = kmin - 1;
-      RealD x;
-
-      RealD Fden = 1.0 / hypot (lmd[k] - Dsh, lme[k]);
-      RealD c = (lmd[k] - Dsh) * Fden;
-      RealD s = -lme[k] * Fden;
-
-      RealD tmpa1 = lmd[k];
-      RealD tmpa2 = lmd[k + 1];
-      RealD tmpb = lme[k];
-
-      lmd[k] = c * c * tmpa1 + s * s * tmpa2 - 2.0 * c * s * tmpb;
-      lmd[k + 1] = s * s * tmpa1 + c * c * tmpa2 + 2.0 * c * s * tmpb;
-      lme[k] = c * s * (tmpa1 - tmpa2) + (c * c - s * s) * tmpb;
-      x = -s * lme[k + 1];
-      lme[k + 1] = c * lme[k + 1];
-
-      for (int i = 0; i < Nk; ++i)
-	{
-	  RealD Qtmp1 = Qt[i + Nm * k];
-	  RealD Qtmp2 = Qt[i + Nm * (k + 1)];
-	  Qt[i + Nm * k] = c * Qtmp1 - s * Qtmp2;
-	  Qt[i + Nm * (k + 1)] = s * Qtmp1 + c * Qtmp2;
-	}
-
-      // Givens transformations
-      for (int k = kmin; k < kmax - 1; ++k)
-	{
-
-	  RealD Fden = 1.0 / hypot (x, lme[k - 1]);
-	  RealD c = lme[k - 1] * Fden;
-	  RealD s = -x * Fden;
-
-	  RealD tmpa1 = lmd[k];
-	  RealD tmpa2 = lmd[k + 1];
-	  RealD tmpb = lme[k];
-
-	  lmd[k] = c * c * tmpa1 + s * s * tmpa2 - 2.0 * c * s * tmpb;
-	  lmd[k + 1] = s * s * tmpa1 + c * c * tmpa2 + 2.0 * c * s * tmpb;
-	  lme[k] = c * s * (tmpa1 - tmpa2) + (c * c - s * s) * tmpb;
-	  lme[k - 1] = c * lme[k - 1] - s * x;
-
-	  if (k != kmax - 2)
-	    {
-	      x = -s * lme[k + 1];
-	      lme[k + 1] = c * lme[k + 1];
-	    }
-
-	  for (int i = 0; i < Nk; ++i)
-	    {
-	      RealD Qtmp1 = Qt[i + Nm * k];
-	      RealD Qtmp2 = Qt[i + Nm * (k + 1)];
-	      Qt[i + Nm * k] = c * Qtmp1 - s * Qtmp2;
-	      Qt[i + Nm * (k + 1)] = s * Qtmp1 + c * Qtmp2;
-	    }
-	}
-    }
-
-#if 0
-#ifdef USE_LAPACK
-#ifdef USE_MKL
-#define LAPACK_INT MKL_INT
-#else
-#define LAPACK_INT long long
-#endif
-    void diagonalize_lapack (std::vector  < RealD > &lmd, std::vector  < RealD > &lme, int N1,	// all
-			     int N2,	// get
-			     GridBase * grid)
-    {
-      const int size = Nm;
-      LAPACK_INT NN = N1;
-      double evals_tmp[NN];
-      double DD[NN];
-      double EE[NN];
-      for (int i = 0; i < NN; i++)
-	for (int j = i - 1; j <= i + 1; j++)
-	  if (j < NN && j >= 0)
-	    {
-	      if (i == j)
-		DD[i] = lmd[i];
-	      if (i == j)
-		evals_tmp[i] = lmd[i];
-	      if (j == (i - 1))
-		EE[j] = lme[j];
-	    }
-      LAPACK_INT evals_found;
-      LAPACK_INT lwork =
-	((18 * NN) >
-	 (1 + 4 * NN + NN * NN) ? (18 * NN) : (1 + 4 * NN + NN * NN));
-      LAPACK_INT liwork = 3 + NN * 10;
-      LAPACK_INT iwork[liwork];
-      double work[lwork];
-      LAPACK_INT isuppz[2 * NN];
-      char jobz = 'N';		// calculate evals only
-      char range = 'I';		// calculate il-th to iu-th evals
-      //    char range = 'A'; // calculate all evals
-      char uplo = 'U';		// refer to upper half of original matrix
-      char compz = 'I';		// Compute eigenvectors of tridiagonal matrix
-      int ifail[NN];
-      LAPACK_INT info;
-//  int total = QMP_get_number_of_nodes();
-//  int node = QMP_get_node_number();
-//  GridBase *grid = evec[0]._grid;
-      int total = grid->_Nprocessors;
-      int node = grid->_processor;
-      int interval = (NN / total) + 1;
-      double vl = 0.0, vu = 0.0;
-      LAPACK_INT il = interval * node + 1, iu = interval * (node + 1);
-      if (iu > NN)
-	iu = NN;
-      double tol = 0.0;
-      if (1)
-	{
-	  memset (evals_tmp, 0, sizeof (double) * NN);
-	  if (il <= NN)
-	    {
-	      printf ("total=%d node=%d il=%d iu=%d\n", total, node, il, iu);
-#ifdef USE_MKL
-	      dstegr (&jobz, &range, &NN,
-#else
-	      LAPACK_dstegr (&jobz, &range, &NN,
-#endif
-			     (double *) DD, (double *) EE, &vl, &vu, &il, &iu,	// these four are ignored if second parameteris 'A'
-			     &tol,	// tolerance
-			     &evals_found, evals_tmp, (double *) NULL, &NN,
-			     isuppz, work, &lwork, iwork, &liwork, &info);
-	      for (int i = iu - 1; i >= il - 1; i--)
-		{
-		  printf ("node=%d evals_found=%d evals_tmp[%d] = %g\n", node,
-			  evals_found, i - (il - 1), evals_tmp[i - (il - 1)]);
-		  evals_tmp[i] = evals_tmp[i - (il - 1)];
-		  if (il > 1)
-		    evals_tmp[i - (il - 1)] = 0.;
-		}
-	    }
-	  {
-	    grid->GlobalSumVector (evals_tmp, NN);
-	  }
-	}
-// cheating a bit. It is better to sort instead of just reversing it, but the document of the routine says evals are sorted in increasing order. qr gives evals in decreasing order.
-    }
-#undef LAPACK_INT
-#endif
-
-
-    void diagonalize (std::vector  < RealD > &lmd,
-		      std::vector  < RealD > &lme,
-		      int N2, int N1, GridBase * grid)
-    {
-
-#ifdef USE_LAPACK
-      const int check_lapack = 0;	// just use lapack if 0, check against lapack if 1
-
-      if (!check_lapack)
-	return diagonalize_lapack (lmd, lme, N2, N1, grid);
-
-//      diagonalize_lapack(lmd2,lme2,Nm2,Nm,Qt,grid);
-#endif
-    }
-#endif
-
-    static RealD normalise (Field & v)
-    {
-      RealD nn = norm2 (v);
-      nn = sqrt (nn);
-      v = v * (1.0 / nn);
-      return nn;
-    }
-
-    void orthogonalize (Field & w, std::vector < Field > &evec, int k)
-    {
-      double t0 = -usecond () / 1e6;
-      typedef typename Field::scalar_type MyComplex;
-      MyComplex ip;
-
-      if (0)
-	{
-	  for (int j = 0; j < k; ++j)
-	    {
-	      normalise (evec[j]);
-	      for (int i = 0; i < j; i++)
-		{
-		  ip = innerProduct (evec[i], evec[j]);	// are the evecs normalised? ; this assumes so.
-		  evec[j] = evec[j] - ip * evec[i];
-		}
-	    }
-	}
-
-      for (int j = 0; j < k; ++j)
-	{
-	  ip = innerProduct (evec[j], w);	// are the evecs normalised? ; this assumes so.
-	  w = w - ip * evec[j];
-	}
-      normalise (w);
-      t0 += usecond () / 1e6;
-      OrthoTime += t0;
-    }
-
-    void setUnit_Qt (int Nm, std::vector < RealD > &Qt)
-    {
-      for (int i = 0; i < Qt.size (); ++i)
-	Qt[i] = 0.0;
-      for (int k = 0; k < Nm; ++k)
-	Qt[k + k * Nm] = 1.0;
-    }
-
-
-    void calc (std::vector < RealD > &eval, const Field & src, int &Nconv)
-    {
-
-      GridBase *grid = src.Grid();
-//      assert(grid == src._grid);
-
-      std::
-	cout << GridLogMessage << " -- Nk = " << Nk << " Np = " << Np << std::
-	endl;
-      std::cout << GridLogMessage << " -- Nm = " << Nm << std::endl;
-      std::cout << GridLogMessage << " -- size of eval   = " << eval.
-	size () << std::endl;
-
-//      assert(c.size() && Nm == eval.size());
-
-      std::vector < RealD > lme (Nm);
-      std::vector < RealD > lmd (Nm);
-
-
-      Field current (grid);
-      Field last (grid);
-      Field next (grid);
-
-      Nconv = 0;
-
-      RealD beta_k;
-
-      // Set initial vector
-      // (uniform vector) Why not src??
-      //      evec[0] = 1.0;
-      current = src;
-      std::cout << GridLogMessage << "norm2(src)= " << norm2 (src) << std::
-	endl;
-      normalise (current);
-      std::
-	cout << GridLogMessage << "norm2(evec[0])= " << norm2 (current) <<
-	std::endl;
-
-      // Initial Nk steps
-      OrthoTime = 0.;
-      double t0 = usecond () / 1e6;
-      RealD norm;		// sqrt norm of last vector
-
-      uint64_t iter = 0;
-
-      bool initted = false;
-      std::vector < RealD > low (Nstop * 10);
-      std::vector < RealD > high (Nstop * 10);
-      RealD cont = 0.;
-      while (1) {
-	  cont = 0.;
-	  std::vector < RealD > lme2 (Nm);
-	  std::vector < RealD > lmd2 (Nm);
-	  for (uint64_t k = 0; k < Nm; ++k, iter++) {
-	      step (lmd, lme, last, current, next, iter);
-	      last = current;
-	      current = next;
-	    }
-	  double t1 = usecond () / 1e6;
-	  std::cout << GridLogMessage << "IRL::Initial steps: " << t1 -
-	    t0 << "seconds" << std::endl;
-	  t0 = t1;
-	  std::
-	    cout << GridLogMessage << "IRL::Initial steps:OrthoTime " <<
-	    OrthoTime << "seconds" << std::endl;
-
-	  // getting eigenvalues
-	  lmd2.resize (iter + 2);
-	  lme2.resize (iter + 2);
-	  for (uint64_t k = 0; k < iter; ++k) {
-	      lmd2[k + 1] = lmd[k];
-	      lme2[k + 2] = lme[k];
-	    }
-	  t1 = usecond () / 1e6;
-	  std::cout << GridLogMessage << "IRL:: copy: " << t1 -
-	    t0 << "seconds" << std::endl;
-	  t0 = t1;
-	  {
-	    int total = grid->_Nprocessors;
-	    int node = grid->_processor;
-	    int interval = (Nstop / total) + 1;
-	    int iu = (iter + 1) - (interval * node + 1);
-	    int il = (iter + 1) - (interval * (node + 1));
-	    std::vector < RealD > eval2 (iter + 3);
-	    RealD eps2;
-	    Bisection::bisec (lmd2, lme2, iter, il, iu, 1e-16, 1e-10, eval2,
-			      eps2);
-//        diagonalize(eval2,lme2,iter,Nk,grid);
-	    RealD diff = 0.;
-	    for (int i = il; i <= iu; i++) {
-		if (initted)
-		  diff =
-		    fabs (eval2[i] - high[iu-i]) / (fabs (eval2[i]) +
-						      fabs (high[iu-i]));
-		if (initted && (diff > eresid))
-		  cont = 1.;
-		if (initted)
-		  printf ("eval[%d]=%0.14e %0.14e, %0.14e\n", i, eval2[i],
-			  high[iu-i], diff);
-		high[iu-i] = eval2[i];
-	      }
-	    il = (interval * node + 1);
-	    iu = (interval * (node + 1));
-	    Bisection::bisec (lmd2, lme2, iter, il, iu, 1e-16, 1e-10, eval2,
-			      eps2);
-	    for (int i = il; i <= iu; i++) {
-		if (initted)
-		  diff =
-		    fabs (eval2[i] - low[i]) / (fabs (eval2[i]) +
-						fabs (low[i]));
-		if (initted && (diff > eresid))
-		  cont = 1.;
-		if (initted)
-		  printf ("eval[%d]=%0.14e %0.14e, %0.14e\n", i, eval2[i],
-			  low[i], diff);
-		low[i] = eval2[i];
-	      }
-	    t1 = usecond () / 1e6;
-	    std::cout << GridLogMessage << "IRL:: diagonalize: " << t1 -
-	      t0 << "seconds" << std::endl;
-	    t0 = t1;
-	  }
-
-	  for (uint64_t k = 0; k < Nk; ++k) {
-//          eval[k] = eval2[k];
-	    }
-	  if (initted)
-	    {
-	      grid->GlobalSumVector (&cont, 1);
-	      if (cont < 1.) return;
-	    }
-	  initted = true;
-	}
-
-    }
-
-
-
-
-
-#if 0
-
-/**
-   There is some matrix Q such that for any vector y
-   Q.e_1 = y and Q is unitary.
-**/
-    template < class T >
-      static T orthQ (DenseMatrix < T > &Q, std::vector < T > y)
-    {
-      int N = y.size ();	//Matrix Size
-      Fill (Q, 0.0);
-      T tau;
-      for (int i = 0; i < N; i++)
-	{
-	  Q[i][0] = y[i];
-	}
-      T sig = conj (y[0]) * y[0];
-      T tau0 = fabs (sqrt (sig));
-
-      for (int j = 1; j < N; j++)
-	{
-	  sig += conj (y[j]) * y[j];
-	  tau = abs (sqrt (sig));
-
-	  if (abs (tau0) > 0.0)
-	    {
-
-	      T gam = conj ((y[j] / tau) / tau0);
-	      for (int k = 0; k <= j - 1; k++)
-		{
-		  Q[k][j] = -gam * y[k];
-		}
-	      Q[j][j] = tau0 / tau;
-	    }
-	  else
-	    {
-	      Q[j - 1][j] = 1.0;
-	    }
-	  tau0 = tau;
-	}
-      return tau;
-    }
-
-/**
-	There is some matrix Q such that for any vector y
-	Q.e_k = y and Q is unitary.
-**/
-    template < class T >
-      static T orthU (DenseMatrix < T > &Q, std::vector < T > y)
-    {
-      T tau = orthQ (Q, y);
-      SL (Q);
-      return tau;
-    }
-
-
-/**
-	Wind up with a matrix with the first con rows untouched
-
-say con = 2
-	Q is such that Qdag H Q has {x, x, val, 0, 0, 0, 0, ...} as 1st colum
-	and the matrix is upper hessenberg
-	and with f and Q appropriately modidied with Q is the arnoldi factorization
-
-**/
-
-    template < class T > static void Lock (DenseMatrix < T > &H,	///Hess mtx     
-					   DenseMatrix < T > &Q,	///Lock Transform
-					   T val,	///value to be locked
-					   int con,	///number already locked
-					   RealD small, int dfg, bool herm)
-    {
-      //ForceTridiagonal(H);
-
-      int M = H.dim;
-      DenseVector < T > vec;
-      Resize (vec, M - con);
-
-      DenseMatrix < T > AH;
-      Resize (AH, M - con, M - con);
-      AH = GetSubMtx (H, con, M, con, M);
-
-      DenseMatrix < T > QQ;
-      Resize (QQ, M - con, M - con);
-
-      Unity (Q);
-      Unity (QQ);
-
-      DenseVector < T > evals;
-      Resize (evals, M - con);
-      DenseMatrix < T > evecs;
-      Resize (evecs, M - con, M - con);
-
-      Wilkinson < T > (AH, evals, evecs, small);
-
-      int k = 0;
-      RealD cold = abs (val - evals[k]);
-      for (int i = 1; i < M - con; i++)
-	{
-	  RealD cnew = abs (val - evals[i]);
-	  if (cnew < cold)
-	    {
-	      k = i;
-	      cold = cnew;
-	    }
-	}
-      vec = evecs[k];
-
-      ComplexD tau;
-      orthQ (QQ, vec);
-      //orthQM(QQ,AH,vec);
-
-      AH = Hermitian (QQ) * AH;
-      AH = AH * QQ;
-
-      for (int i = con; i < M; i++)
-	{
-	  for (int j = con; j < M; j++)
-	    {
-	      Q[i][j] = QQ[i - con][j - con];
-	      H[i][j] = AH[i - con][j - con];
-	    }
-	}
-
-      for (int j = M - 1; j > con + 2; j--)
-	{
-
-	  DenseMatrix < T > U;
-	  Resize (U, j - 1 - con, j - 1 - con);
-	  DenseVector < T > z;
-	  Resize (z, j - 1 - con);
-	  T nm = norm (z);
-	  for (int k = con + 0; k < j - 1; k++)
-	    {
-	      z[k - con] = conj (H (j, k + 1));
-	    }
-	  normalise (z);
-
-	  RealD tmp = 0;
-	  for (int i = 0; i < z.size () - 1; i++)
-	    {
-	      tmp = tmp + abs (z[i]);
-	    }
-
-	  if (tmp < small / ((RealD) z.size () - 1.0))
-	    {
-	      continue;
-	    }
-
-	  tau = orthU (U, z);
-
-	  DenseMatrix < T > Hb;
-	  Resize (Hb, j - 1 - con, M);
-
-	  for (int a = 0; a < M; a++)
-	    {
-	      for (int b = 0; b < j - 1 - con; b++)
-		{
-		  T sum = 0;
-		  for (int c = 0; c < j - 1 - con; c++)
-		    {
-		      sum += H[a][con + 1 + c] * U[c][b];
-		    }		//sum += H(a,con+1+c)*U(c,b);}
-		  Hb[b][a] = sum;
-		}
-	    }
-
-	  for (int k = con + 1; k < j; k++)
-	    {
-	      for (int l = 0; l < M; l++)
-		{
-		  H[l][k] = Hb[k - 1 - con][l];
-		}
-	    }			//H(Hb[k-1-con][l] , l,k);}}
-
-	  DenseMatrix < T > Qb;
-	  Resize (Qb, M, M);
-
-	  for (int a = 0; a < M; a++)
-	    {
-	      for (int b = 0; b < j - 1 - con; b++)
-		{
-		  T sum = 0;
-		  for (int c = 0; c < j - 1 - con; c++)
-		    {
-		      sum += Q[a][con + 1 + c] * U[c][b];
-		    }		//sum += Q(a,con+1+c)*U(c,b);}
-		  Qb[b][a] = sum;
-		}
-	    }
-
-	  for (int k = con + 1; k < j; k++)
-	    {
-	      for (int l = 0; l < M; l++)
-		{
-		  Q[l][k] = Qb[k - 1 - con][l];
-		}
-	    }			//Q(Qb[k-1-con][l] , l,k);}}
-
-	  DenseMatrix < T > Hc;
-	  Resize (Hc, M, M);
-
-	  for (int a = 0; a < j - 1 - con; a++)
-	    {
-	      for (int b = 0; b < M; b++)
-		{
-		  T sum = 0;
-		  for (int c = 0; c < j - 1 - con; c++)
-		    {
-		      sum += conj (U[c][a]) * H[con + 1 + c][b];
-		    }		//sum += conj( U(c,a) )*H(con+1+c,b);}
-		  Hc[b][a] = sum;
-		}
-	    }
-
-	  for (int k = 0; k < M; k++)
-	    {
-	      for (int l = con + 1; l < j; l++)
-		{
-		  H[l][k] = Hc[k][l - 1 - con];
-		}
-	    }			//H(Hc[k][l-1-con] , l,k);}}
-
-	}
-    }
-#endif
-
-
-  };
-
-}
-#endif

Grid/algorithms/iterative/SplitGridBlockKrylovSchur.h

@@ -1,124 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./lib/algorithms/iterative/SplitGridBlockKrylovSchur.h
-
-Copyright (C) 2015
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_SPLIT_BLOCK_KRYLOV_SCHUR_H
-#define GRID_SPLIT_BLOCK_KRYLOV_SCHUR_H
-
-NAMESPACE_BEGIN(Grid);
-
-/**
- * Block Krylov-Schur eigensolver with Split-Grid batched operator application.
- *
- * Derives from BlockKrylovSchur<Field> and overrides only the operator
- * application: instead of calling Linop.Op() once per vector, mrhs vectors
- * are packed onto a smaller split grid with Grid_split, the polynomial-
- * filtered operator poly(SLinop) is applied once to the combined field, and
- * the results are unpacked with Grid_unsplit.  All other algorithmic logic
- * (Arnoldi orthogonalisation, Schur restart, convergence test, etc.) is
- * inherited unchanged from the base class.
- *
- * Constructor extras (beyond BlockKrylovSchur)
- * -------------------------------------------
- * SLinop  : split-grid linear operator used inside poly
- * poly    : polynomial filter OperatorFunction applied on the split grid
- * SGrid   : split-grid GridBase (mrhs fields packed here)
- * mrhs    : RHS batched per split-grid call; Nblock must be divisible by mrhs
- *
- * Notes
- * -----
- * - The Rayleigh quotient H is built from poly(A), so its Ritz values are
- *   eigenvalues of poly(A).  Convergence is assessed against the full-grid
- *   Linop (inherited from base), giving true A-eigenvalues.
- * - Grid_ (inherited) is the full grid; SGrid is the split grid.
- */
-template<class Field>
-class SplitGridBlockKrylovSchur : public BlockKrylovSchur<Field> {
-
-  using Base = BlockKrylovSchur<Field>;
-
-  // Bring protected base members into scope
-  using Base::Nblock;
-  using Base::Grid_;
-  using Base::basis;
-
-  // Split-grid extras
-  LinearOperatorBase<Field>& SLinop;
-  OperatorFunction<Field>&   poly;
-  GridBase*                  SGrid;
-  int                        mrhs;
-
-public:
-
-  SplitGridBlockKrylovSchur(LinearOperatorBase<Field>& _Linop,
-                             LinearOperatorBase<Field>& _SLinop,
-                             OperatorFunction<Field>&   _poly,
-                             GridBase*                  _FGrid,
-                             GridBase*                  _SGrid,
-                             int                        _mrhs,
-                             RealD                      _Tolerance,
-                             RitzFilter                 _rf = EvalReSmall)
-    : Base(_Linop, _FGrid, _Tolerance, _rf),
-      SLinop(_SLinop), poly(_poly), SGrid(_SGrid), mrhs(_mrhs)
-  {}
-
-protected:
-
-  // Validate mrhs divisibility before the Arnoldi loop starts
-  void preRun() override
-  {
-    assert(Nblock % mrhs == 0 && "Nblock must be divisible by mrhs");
-  }
-
-  // Apply poly(A) to basis[kBase .. kBase+Nblock-1] via Grid_split / Grid_unsplit,
-  // batching mrhs vectors per split-grid call.
-  void applyBlock(std::vector<Field>& W, int kBase) override
-  {
-    std::vector<Field> in(mrhs, Field(Grid_));
-    Field s_in(SGrid);
-    Field s_out(SGrid);
-
-    int k_start = 0;
-    while (k_start < Nblock) {
-      for (int u = 0; u < mrhs; u++)
-        in[u] = basis[kBase + k_start + u];
-
-      Grid_split(in, s_in);
-      poly(SLinop, s_in, s_out);
-      Grid_unsplit(in, s_out);
-
-      for (int u = 0; u < mrhs; u++)
-        W[k_start + u] = in[u];
-
-      k_start += mrhs;
-    }
-  }
-};
-
-NAMESPACE_END(Grid);
-
-#endif  // GRID_SPLIT_BLOCK_KRYLOV_SCHUR_H

Grid/algorithms/iterative/TrueHarmonicBlockKrylovSchur.h

@@ -1,366 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: ./lib/algorithms/iterative/TrueHarmonicBlockKrylovSchur.h
-
-Copyright (C) 2015
-
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-#ifndef GRID_TRUE_HARMONIC_BLOCKED_KRYLOV_SCHUR_H
-#define GRID_TRUE_HARMONIC_BLOCKED_KRYLOV_SCHUR_H
-
-#include <Grid/algorithms/iterative/HarmonicBlockKrylovSchur.h>
-
-NAMESPACE_BEGIN(Grid);
-
-/**
- * True harmonic-Ritz block Krylov-Schur eigensolver.
- *
- * Difference from the base class
- * ------------------------------
- * HarmonicBlockKrylovSchur is a *shift-targeted* Krylov-Schur: it Schur-
- * decomposes (H - sigma I) and sorts the standard Ritz values by |lambda -
- * sigma|.  For interior eigenvalues of Hermitian indefinite operators the
- * standard Ritz values can be spurious ("ghosts"), so the sort can select
- * junk.  This variant performs genuine harmonic Ritz extraction instead.
- *
- * Harmonic extraction
- * -------------------
- * From the exact block Arnoldi relation
- *
- *   A V = V H + F B^dag                                            (1)
- *
- * harmonic Ritz pairs (theta, g) w.r.t. shift sigma satisfy the Petrov-
- * Galerkin condition (A - sigma)Vg - (theta - sigma)Vg  perp  (A - sigma)V,
- * which reduces (with Hs = H - sigma I) to the small eigenproblem
- *
- *   Hhat g = (theta - sigma) g,   Hhat = Hs + Hs^{-H} B B^dag      (2)
- *
- * Exact thick restart
- * -------------------
- * From (2), H g = theta g - Hs^{-H} B (B^dag g), so the residual of every
- * harmonic pair lies in the Nblock-dimensional column space of the single
- * block
- *
- *   W = F - V Hs^{-H} B :
- *   A (V g) - theta (V g) = W (B^dag g)                            (3)
- *
- * Hence {V g_1 .. g_Nk} + cols(W) admits an EXACT truncated Krylov-Schur
- * relation  A Y = Y H_k + What Bc^dag  with dense H_k, and block Arnoldi
- * resumes cleanly from What.  Krylov-Schur (Stewart, SIMAX 23(3), 2001)
- * does not require H to be triangular after restart -- only that the
- * decomposition holds exactly, which (3) guarantees.
- *
- * This is the block analogue of the GMRES-DR restart (Morgan, SISC 24(1),
- * 2002; block version: Morgan, Appl. Numer. Math. 54, 2005).  The exact
- * composition "block harmonic Krylov-Schur" is a derivation from these
- * ingredients rather than a published algorithm; the relation (1) is
- * checked explicitly at every restart when doVerify=true, and identity (3)
- * was validated to machine precision in a standalone Eigen prototype.
- *
- * Caveats
- * -------
- * - If sigma is (numerically) an eigenvalue of H, Hs^{-H} is singular;
- *   an assert fires -- perturb the shift slightly.
- * - Reported eigenvalues come from the eigensystem of the retained H_k
- *   (the true projected Rayleigh quotient Y^dag A Y), as in the base
- *   class, so convergence tests and reporting are inherited unchanged.
- *
- * Usage: identical to HarmonicBlockKrylovSchur.
- */
-template<class Field>
-class TrueHarmonicBlockKrylovSchur : public HarmonicBlockKrylovSchur<Field> {
-
-  typedef HarmonicBlockKrylovSchur<Field> Base;
-  typedef typename Base::CMat CMat;
-  typedef typename Base::CVec CVec;
-
-  // Members of the dependent base class need explicit scope.
-  using Base::Nblock;
-  using Base::Nm;
-  using Base::Nk;
-  using Base::Nstop;
-  using Base::MaxIter;
-  using Base::Tolerance;
-  using Base::shift;
-  using Base::Linop;
-  using Base::Grid_;
-  using Base::ritzFilter;
-  using Base::basis;
-  using Base::H;
-  using Base::F;
-  using Base::B;
-  using Base::beta_k;
-  using Base::rtol;
-  using Base::evals;
-  using Base::littleEvecs;
-  using Base::ritzEstimates;
-
-  using Base::expandStartBlock;
-  using Base::blockArnoldiIteration;
-  using Base::blockQR;
-  using Base::computeEigensystem;
-  using Base::converged;
-  using Base::approxMaxEval;
-
-public:
-  using Base::evecs;
-  using Base::doEvalCheck;
-  using Base::useParityFlip;
-  using Base::useGamma5;
-  using Base::gamma5Func;
-  using Base::verify;
-
-  TrueHarmonicBlockKrylovSchur(LinearOperatorBase<Field>& _Linop, GridBase* _Grid,
-                               RealD _Tolerance, ComplexD _shift = 0.0,
-                               RitzFilter _rf = EvalNormSmall)
-    : Base(_Linop, _Grid, _Tolerance, _shift, _rf)
-  {}
-
-  //--------------------------------------------------------------------
-  // Main entry point (same signature and outer structure as the base;
-  // the Schur-sort restart is replaced by harmonicRestart()).
-  //--------------------------------------------------------------------
-  virtual void operator()(const std::vector<Field>& v0, int _maxIter, int _Nm, int _Nk,
-                          int _Nstop, int _Nblock = 1, bool doubleOrthog = true,
-                          bool doVerify = false) override
-  {
-    MaxIter = _maxIter;
-    Nm      = _Nm;
-    Nk      = _Nk;
-    Nstop   = _Nstop;
-    Nblock  = _Nblock;
-
-    {
-      int divisor = 1;
-      if (useParityFlip) divisor *= 2;
-      if (useGamma5)     divisor *= 2;
-      assert(Nblock % divisor == 0 && (int)v0.size() >= Nblock / divisor);
-    }
-    assert(Nm % Nblock == 0);
-    assert(Nk % Nblock == 0);
-    assert(Nk < Nm);
-    if (useGamma5) assert(gamma5Func && "useGamma5: gamma5Func must be set");
-
-    int N = Nm;
-
-    RealD approxLambdaMax = approxMaxEval(v0[0]);
-    rtol = Tolerance * approxLambdaMax;
-    std::cout << GridLogMessage
-              << "TrueHarmonicBlockKrylovSchur: approx max eval = " << approxLambdaMax
-              << ", rtol = " << rtol
-              << ", shift = " << shift << std::endl;
-
-    H = CMat::Zero(N, N);
-    B = CMat::Zero(N, Nblock);
-
-    int start = 0;
-    std::vector<Field> startBlock = expandStartBlock(v0);
-
-    for (int iter = 0; iter < MaxIter; iter++) {
-      std::cout << GridLogMessage
-                << "TrueHarmonicBlockKrylovSchur: restart iteration " << iter << std::endl;
-
-      // ---- Block Arnoldi: extend from block 'start' to block Nm/Nblock ----
-      blockArnoldiIteration(startBlock, Nm/Nblock, start, doubleOrthog);
-      std::cout << GridLogMessage << "blockArnoldiIteration done " << std::endl;
-      start = Nk/Nblock;
-
-      if (doVerify) {
-        std::string lbl = "iter " + std::to_string(iter) + " after Arnoldi";
-        verify(lbl);
-      }
-
-      // ---- Harmonic extraction + exact thick restart ----
-      harmonicRestart();
-      std::cout << GridLogMessage << "harmonicRestart done " << std::endl;
-
-      // Restart from the residual block W (exact by identity (3))
-      startBlock = F;
-
-      if (doVerify) {
-        std::string lbl = "iter " + std::to_string(iter) + " after harmonic restart";
-        verify(lbl);
-      }
-
-      // ---- Eigensystem of retained H_k = Y^dag A Y for convergence ----
-      CMat Hk = H(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk));
-      computeEigensystem(Hk, Nk);
-
-      int Nconv = converged(Nk);
-      std::cout << GridLogMessage
-                << "TrueHarmonicBlockKrylovSchur: converged " << Nconv
-                << " / " << Nstop << std::endl;
-
-      if (Nconv >= Nstop || iter == MaxIter - 1) {
-        std::cout << GridLogMessage
-                  << "TrueHarmonicBlockKrylovSchur: done after " << iter
-                  << " restarts, " << Nconv << " converged." << std::endl;
-        std::cout << GridLogMessage << "Eigenvalues: " << evals.transpose() << std::endl;
-
-        if (doEvalCheck) {
-          Field w(Grid_);
-          for (int k = 0; k < (int)evecs.size(); k++) {
-            Linop.Op(evecs[k], w);
-            ComplexD eval_est = toStdCmplx(innerProduct(evecs[k], w));
-            w -= eval_est * evecs[k];
-            RealD res = std::sqrt(norm2(w));
-            std::cout << GridLogMessage << "TrueHarmonicBlockKrylovSchur: evec[" << k << "]"
-                      << "  eval_reported = " << evals[k]
-                      << "  eval_est = " << eval_est
-                      << "  || A v - eval_est * v || = " << res << std::endl;
-          }
-        }
-
-        return;
-      }
-    }
-  }
-
-protected:
-
-  //--------------------------------------------------------------------
-  // Harmonic thick restart
-  //--------------------------------------------------------------------
-  // On entry:  A V = V H + F B^dag exact, V = basis (Nm fields), F Nblock
-  //            residual fields, H/B the Nm-sized small matrices.
-  // On exit:   basis = Y (Nk orthonormal fields spanning the harmonic
-  //            Ritz space), F = What (Nblock orthonormal fields, perp Y),
-  //            H top-left Nk x Nk = Y^dag A Y (dense), B top Nk rows the
-  //            new coupling; relation A Y = Y H_k + F B^dag again exact.
-  void harmonicRestart()
-  {
-    int N = Nm;
-
-    // ---- Small harmonic eigenproblem:  Hhat g = (theta - sigma) g ----
-    CMat Hs = H - shift * CMat::Identity(N, N);
-    Eigen::FullPivLU<CMat> lu(Hs.adjoint());
-    assert(lu.isInvertible() &&
-           "TrueHarmonicBlockKrylovSchur: H - shift*I singular; perturb the shift");
-    CMat M    = lu.solve(B);              // N x Nblock :  Hs^{-H} B
-    CMat Hhat = Hs + M * B.adjoint();     // N x N
-
-    Eigen::ComplexEigenSolver<CMat> es(Hhat);
-
-    // Sort harmonic values (theta - sigma) by the Ritz filter
-    // (EvalNormSmall -> closest to the shift).
-    ComplexComparator cComp(ritzFilter);
-    std::vector<int> idx(N);
-    std::iota(idx.begin(), idx.end(), 0);
-    std::sort(idx.begin(), idx.end(), [&](int a, int b){
-      return cComp(toStdCmplx(es.eigenvalues()(a)), toStdCmplx(es.eigenvalues()(b)));
-    });
-
-    CMat G(N, Nk);
-    CVec theta(Nk);
-    for (int k = 0; k < Nk; k++) {
-      G.col(k) = es.eigenvectors().col(idx[k]);
-      theta(k) = es.eigenvalues()(idx[k]) + toStdCmplx(shift);
-    }
-
-    std::cout << GridLogMessage
-              << "TrueHarmonicBlockKrylovSchur: harmonic Ritz values nearest shift:" << std::endl;
-    for (int k = 0; k < Nk; k++)
-      std::cout << GridLogMessage << "  [" << k << "] " << theta(k) << std::endl;
-
-    // ---- Orthonormalise the little vectors:  G = Gt Rg ----
-    Eigen::HouseholderQR<CMat> qr(G);
-    CMat Gt = qr.householderQ() * CMat::Identity(N, Nk);
-    CMat Rg = qr.matrixQR().topLeftCorner(Nk, Nk).template triangularView<Eigen::Upper>();
-    for (int k = 0; k < Nk; k++)
-      assert(std::abs(Rg(k, k)) > 1e-14 &&
-             "TrueHarmonicBlockKrylovSchur: harmonic vectors linearly dependent");
-    CMat RgInv = Rg.inverse();
-
-    // A (V G) = (V G) diag(theta) + W (B^dag G),  W = F - V M
-    //   =>  A (V Gt) = (V Gt) T + W Ct
-    CMat T  = Rg * theta.asDiagonal() * RgInv;   // Nk x Nk
-    CMat Ct = (B.adjoint() * G) * RgInv;         // Nblock x Nk
-
-    // ---- Large-field work ----
-    // Y = V Gt : new orthonormal basis (Nk fields)
-    std::vector<Field> Y;
-    constructThin(Y, basis, Gt, Nk);
-
-    // W = F - V M : Nblock residual directions containing every harmonic
-    // residual (identity (3) in the class comment)
-    std::vector<Field> W = F;
-    for (int t = 0; t < Nblock; t++)
-      for (int j = 0; j < N; j++)
-        W[t] -= basis[j] * M(j, t);
-
-    // Orthogonalise W against Y (two passes), recording P = Y^dag W
-    CMat P = CMat::Zero(Nk, Nblock);
-    for (int pass = 0; pass < 2; pass++) {
-      for (int j = 0; j < Nk; j++) {
-        for (int t = 0; t < Nblock; t++) {
-          ComplexD c = innerProduct(Y[j], W[t]);
-          P(j, t) += toStdCmplx(c);
-          W[t] -= c * Y[j];
-        }
-      }
-    }
-
-    // Block QR of the projected residual: W <- What, returns Rw
-    CMat Rw = blockQR(W);
-
-    // ---- Reassemble the exact Krylov-Schur relation ----
-    //   A Y = Y (T + P Ct) + What (Rw Ct)
-    CMat Hk = T + P * Ct;                        // Nk x Nk, dense
-    CMat Bc = Rw * Ct;                           // Nblock x Nk
-
-    H = CMat::Zero(N, N);
-    H(Eigen::seqN(0, Nk), Eigen::seqN(0, Nk)) = Hk;
-    B = CMat::Zero(N, Nblock);
-    for (int j = 0; j < Nk; j++)
-      for (int t = 0; t < Nblock; t++)
-        B(j, t) = std::conj(Bc(t, j));
-
-    basis = Y;
-    F     = W;
-    beta_k = Bc.norm();
-
-    std::cout << GridLogMessage
-              << "TrueHarmonicBlockKrylovSchur: beta_k = " << beta_k << std::endl;
-  }
-
-  //--------------------------------------------------------------------
-  // Thin basis combination: out[i] = sum_j U[j] * R(j,i), i < ncol
-  //--------------------------------------------------------------------
-  void constructThin(std::vector<Field>& out, const std::vector<Field>& U,
-                     const CMat& R, int ncol)
-  {
-    out.clear();
-    Field tmp(Grid_);
-    for (int i = 0; i < ncol; i++) {
-      tmp = Zero();
-      for (int j = 0; j < (int)U.size(); j++)
-        tmp += U[j] * R(j, i);
-      out.push_back(tmp);
-    }
-  }
-
-};
-
-NAMESPACE_END(Grid);
-
-#endif  // GRID_TRUE_HARMONIC_BLOCKED_KRYLOV_SCHUR_H

Grid/algorithms/multigrid/Aggregates.h

@@ -66,7 +66,21 @@ public:
   {
   };
   
-  
+  void GlobalOrthonormalise(void)
+  {
+    // Normalise all vectors
+    for(int i=0;i<nbasis; i++){
+      RealD scale = std::pow(norm2(subspace[i]),-0.5); 
+      subspace[i] = subspace[i]*scale;
+    }
+    for(int i=0;i<nbasis; i++){
+      for(int j=0;j<i; j++){
+	basisOrthogonalize(subspace,subspace[i],j);
+      }
+      RealD scale = std::pow(norm2(subspace[i]),-0.5); 
+      subspace[i] = subspace[i]*scale;
+    }
+  }  
   void Orthogonalise(void){
     CoarseScalar InnerProd(CoarseGrid); 
     //    std::cout << GridLogMessage <<" Block Gramm-Schmidt pass 1"<<std::endl;
@@ -110,14 +124,16 @@ public:
       
       hermop.Op(noise,Mn); std::cout<<GridLogMessage << "noise   ["<<b<<"] <n|MdagM|n> "<<norm2(Mn)<<std::endl;
 
-      for(int i=0;i<4;i++){
+      for(int i=0;i<2;i++){
 
 	CG(hermop,noise,subspace[b]);
 
 	noise = subspace[b];
 	scale = std::pow(norm2(noise),-0.5); 
 	noise=noise*scale;
-
+	
+	hermop.Op(noise,Mn); std::cout<<GridLogMessage << "intermediate["<<i<<"] <i|MdagM|i> "<<norm2(Mn)<<std::endl;
+	
       }
 
       hermop.Op(noise,Mn); std::cout<<GridLogMessage << "filtered["<<b<<"] <f|MdagM|f> "<<norm2(Mn)<<std::endl;
@@ -134,7 +150,8 @@ public:
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,10,DiracOp,simple_fine,30,30);
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,10,DiracOp,simple_fine,12,12);
     //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,30,DiracOp,simple_fine,12,12);
-    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.001,30,DiracOp,simple_fine,10,10);
+    //    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.0005,30,DiracOp,simple_fine,20,20);
+    PrecGeneralisedConjugateResidualNonHermitian<FineField> GCR(0.0005,30,DiracOp,simple_fine,10,10);
     FineField noise(FineGrid);
     FineField src(FineGrid);
     FineField guess(FineGrid);
@@ -167,6 +184,8 @@ public:
 	noise = subspace[b];
 	scale = std::pow(norm2(noise),-0.5); 
 	noise=noise*scale;
+	
+	DiracOp.Op(noise,Mn); std::cout<<GridLogMessage << "intermediate["<<i<<"] <f|Op|f> "<<innerProduct(noise,Mn)<<" <f|OpDagOp|f>"<<norm2(Mn)<<std::endl;
 
       }
 
@@ -174,6 +193,7 @@ public:
       subspace[b]   = noise;
 
     }
+    GlobalOrthonormalise();
   }
 
   ////////////////////////////////////////////////////////////////////////////////////////////////

Grid/algorithms/multigrid/GeneralCoarsenedMatrix.h

@@ -31,6 +31,7 @@ Author: Peter Boyle <pboyle@bnl.gov>
 
 #include <Grid/lattice/PaddedCell.h>
 #include <Grid/stencil/GeneralLocalStencil.h>
+#include <Grid/algorithms/deflation/MultiRHSBlockProject.h>
 
 NAMESPACE_BEGIN(Grid);
 
@@ -66,6 +67,10 @@ public:
   std::vector<CoarseMatrix> _Adag;
   std::vector<CoarseVector> MultTemporaries;
 
+  int64_t MultCalls;
+  double  MultFlopsAccum;
+  double  MultUsecAccum;
+
   ///////////////////////
   // Interface
   ///////////////////////
@@ -104,19 +109,20 @@ public:
   }
   */
   
-  GeneralCoarsenedMatrix(NonLocalStencilGeometry &_geom,GridBase *FineGrid, GridCartesian * CoarseGrid)
+  GeneralCoarsenedMatrix(NonLocalStencilGeometry &_geom,GridBase *FineGrid, GridCartesian * CoarseGrid,int _herm=1)
     : geom(_geom),
       _FineGrid(FineGrid),
       _CoarseGrid(CoarseGrid),
-      hermitian(1),
+      hermitian(_herm),
       Cell(_geom.Depth(),_CoarseGrid),
-      Stencil(Cell.grids.back(),geom.shifts)
+      Stencil(Cell.grids.back(),geom.shifts),
+      MultCalls(0), MultFlopsAccum(0.0), MultUsecAccum(0.0)
   {
     {
       int npoint = _geom.npoint;
     }
     _A.resize(geom.npoint,CoarseGrid);
-    //    _Adag.resize(geom.npoint,CoarseGrid);
+    if ( !hermitian ) _Adag.resize(geom.npoint,CoarseGrid);
   }
   void M (const CoarseVector &in, CoarseVector &out)
   {
@@ -124,10 +130,10 @@ public:
   }
   void Mdag (const CoarseVector &in, CoarseVector &out)
   {
-    GRID_ASSERT(hermitian);
-    Mult(_A,in,out);
-    //    if ( hermitian ) M(in,out);
-    //    else Mult(_Adag,in,out);
+    if(hermitian)
+      Mult(_A,in,out);
+    else
+      Mult(_Adag,in,out);
   }
   void Mult (std::vector<CoarseMatrix> &A,const CoarseVector &in, CoarseVector &out)
   {
@@ -227,29 +233,28 @@ public:
     text+=usecond();
     ttot+=usecond();
     
-    std::cout << GridLogPerformance<<"Coarse 1rhs Mult Aviews "<<tviews<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Mult exch "<<texch<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Mult mult "<<tmult<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<" of which mult2  "<<tmult2<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Mult ext  "<<text<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Mult temps "<<ttemps<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Mult copy  "<<tcopy<<" us"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Mult tot  "<<ttot<<" us"<<std::endl;
-    //    std::cout << GridLogPerformance<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Kernel flops "<< flops<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Kernel flop/s "<< flops/tmult<<" mflop/s"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse Kernel bytes/s "<< bytes/tmult<<" MB/s"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse overall flops/s "<< flops/ttot<<" mflop/s"<<std::endl;
-    std::cout << GridLogPerformance<<"Coarse total bytes   "<< bytes/1e6<<" MB"<<std::endl;
+    MultCalls++;
+    MultFlopsAccum += flops;
+    MultUsecAccum  += ttot;
+    std::cout << GridLogPerformance
+	      << "Coarse Mult call " << MultCalls
+	      << "  tot " << ttot << " us"
+	      << "  kernel " << tmult << " us"
+	      << "  kernel " << flops/tmult*1e-3 << " GFlop/s"
+	      << "  overall " << MultFlopsAccum/MultUsecAccum*1e-3 << " GFlop/s (cumul)"
+	      << "  bw " << bytes/tmult*1e-3 << " GB/s"
+	      << std::endl;
 
   };
   
   void PopulateAdag(void)
   {
+#if 0
+    // Serial global peek/poke reference implementation
     for(int64_t bidx=0;bidx<CoarseGrid()->gSites() ;bidx++){
       Coordinate bcoor;
       CoarseGrid()->GlobalIndexToGlobalCoor(bidx,bcoor);
-      
+
       for(int p=0;p<geom.npoint;p++){
 	Coordinate scoor = bcoor;
 	for(int mu=0;mu<bcoor.size();mu++){
@@ -262,6 +267,36 @@ public:
 	pokeSite(adj(link),_Adag[pp],bcoor);
       }
     }
+#else
+    // Parallel: _Adag[pp](x) = adj( _A[p](x + s_pp) ),  pp = Reverse(p), s_pp = -s_p.
+    // The neighbour fetch reuses the same padded-cell + stencil machinery as Mult,
+    // reading one matrix element per coalesced access so no whole site matrix
+    // (230KB at nbasis=60) ever lands on a GPU thread stack (HIP limit 128KB).
+    // Halo sites compute garbage neighbours; Cell.Extract discards them.
+    // Must run on the unpadded _A, i.e. before ExchangeCoarseLinks.
+    const int Nsimd = CComplex::Nsimd();
+    for(int p=0;p<geom.npoint;p++){
+      int pp = geom.Reverse(p);
+      CoarseMatrix Apad = Cell.ExchangePeriodic(_A[p]);
+      CoarseMatrix Dpad(Apad.Grid());
+      int64_t osites = Apad.Grid()->oSites();
+      {
+	autoView( Apad_v   , Apad, AcceleratorRead);
+	autoView( Dpad_v   , Dpad, AcceleratorWriteDiscard);
+	autoView( Stencil_v, Stencil, AcceleratorRead);
+	accelerator_for(sj, osites*nbasis, Nsimd, {
+	    int32_t ss = sj/nbasis;
+	    int32_t j  = sj%nbasis;
+	    auto SE = Stencil_v.GetEntry(pp,ss);
+	    for(int i=0;i<nbasis;i++){
+	      auto z = coalescedReadGeneralPermute(Apad_v[SE->_offset](i,j),SE->_permute,Nd);
+	      coalescedWrite(Dpad_v[ss](j,i),conjugate(z));
+	    }
+	});
+      }
+      _Adag[pp] = Cell.Extract(Dpad);
+    }
+#endif
   }
   /////////////////////////////////////////////////////////////
   // 
@@ -417,10 +452,9 @@ public:
 	int osites=CoarseGrid()->oSites();
 	autoView( A_v  , _A[k], AcceleratorWrite);
 	autoView( FT_v  , FT[k], AcceleratorRead);
-	accelerator_for(sss, osites, 1, {
-	    for(int j=0;j<nbasis;j++){
-	      A_v[sss](i,j) = FT_v[sss](j);
-	    }
+	accelerator_for(sss, osites, nbasis, {
+	    int j = acceleratorSIMTlane(nbasis);
+	    A_v[sss](i,j) = FT_v[sss](j);
         });
       }
       tinv+=usecond();
@@ -428,8 +462,8 @@ public:
 
     // Only needed if nonhermitian
     if ( ! hermitian ) {
-      //      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
-      //      PopulateAdag();
+      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
+      PopulateAdag();
     }
 
     // Need to write something to populate Adag from A
@@ -517,13 +551,9 @@ public:
     // Now compute the matrix elements of linop between the orthonormal
     // set of vectors.
     ///////////////////////////////////////////////////////////////////////
-    FineField phaV(grid); // Phased block basis vector
-    FineField MphaV(grid);// Matrix applied
     std::vector<FineComplexField> phaF(npoint,grid);
     std::vector<CoarseComplexField> pha(npoint,CoarseGrid());
-    
-    CoarseVector coarseInner(CoarseGrid());
-    
+
     typedef typename CComplex::scalar_type SComplex;
     FineComplexField one(grid); one=SComplex(1.0);
     FineComplexField zz(grid); zz = Zero();
@@ -542,37 +572,52 @@ public:
       pha[p]  =exp(pha[p]*ci);
 
       blockZAXPY(phaF[p],pha[p],one,zz);
-      
+
     }
     tphase+=usecond();
-    
-    std::vector<CoarseVector> ComputeProj(npoint,CoarseGrid());
-    std::vector<CoarseVector>          FT(npoint,CoarseGrid());
+
+    // Import basis into BLAS layout once; blockProject then reads it once per
+    // basis vector rather than once per (i,p) as in scalar blockProject.
+    // Process all npoint in a single batch.
+    MultiRHSBlockProject<FineField> Projector;
+    Projector.Allocate(nbasis, grid, CoarseGrid());
+    Projector.ImportBasis(U.subspace);
+
+    std::vector<FineField>    phaV_batch(npoint, grid);
+    std::vector<FineField>    MphaV_batch(npoint, grid);
+    std::vector<CoarseVector> proj_batch(npoint, CoarseGrid());
+    std::vector<CoarseVector> ComputeProj(npoint, CoarseGrid());
+    std::vector<CoarseVector>          FT(npoint, CoarseGrid());
+
+    // Pre-allocate BLAS_F and BLAS_C to avoid repeated hipMalloc/hipFree of
+    // ~5.6 GB per blockProject call, which hangs on ROCm for large allocations.
+    Projector.BLAS_F.resize(Projector.fine_vol * Projector.words * npoint);
+    Projector.BLAS_C.resize(Projector.coarse_vol * nbasis * npoint);
+
     for(int i=0;i<nbasis;i++){// Loop over basis vectors
+      accelerator_barrier(); // ensure prior iteration's async writes are retired
       std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<"/"<<nbasis<< std::endl;
-      for(int p=0;p<npoint;p++){ // Loop over momenta in npoint
-	tphaseBZ-=usecond();
-	phaV = phaF[p]*V.subspace[i];
-	tphaseBZ+=usecond();
 
-	/////////////////////////////////////////////////////////////////////
-	// Multiple phased subspace vector by matrix and project to subspace
-	// Remove local bulk phase to leave relative phases
-	/////////////////////////////////////////////////////////////////////
-	tmat-=usecond();
-	linop.Op(phaV,MphaV);
-	tmat+=usecond();
-	//	std::cout << i << " " <<p << " MphaV "<<norm2(MphaV)<<" "<<norm2(phaV)<<std::endl;
+      tphaseBZ-=usecond();
+      for(int p=0;p<npoint;p++)
+	phaV_batch[p] = phaF[p] * V.subspace[i];
+      tphaseBZ+=usecond();
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" phaseBZ done"<< std::endl;
 
-	tproj-=usecond();
-	blockProject(coarseInner,MphaV,U.subspace);
-	coarseInner = conjugate(pha[p]) * coarseInner;
+      tmat-=usecond();
+      for(int p=0;p<npoint;p++)
+	linop.Op(phaV_batch[p], MphaV_batch[p]);
+      tmat+=usecond();
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" mat done"<< std::endl;
 
-	ComputeProj[p] = coarseInner;
-	tproj+=usecond();
-	//	std::cout << i << " " <<p << " ComputeProj "<<norm2(ComputeProj[p])<<std::endl;
-
-      }
+      // One batched GEMM reads BLAS_V once for all npoint vectors.
+      tproj-=usecond();
+      Projector.blockProject(MphaV_batch, proj_batch);
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" blockProject done"<< std::endl;
+      for(int p=0;p<npoint;p++)
+	ComputeProj[p] = conjugate(pha[p]) * proj_batch[p];
+      tproj+=usecond();
+      std::cout << GridLogMessage<< "CoarsenMatrixColoured vec "<<i<<" proj done"<< std::endl;
 
       tinv-=usecond();
       for(int k=0;k<npoint;k++){
@@ -580,14 +625,13 @@ public:
 	for(int l=0;l<npoint;l++){
 	  FT[k]= FT[k]+ invMkl(l,k)*ComputeProj[l];
 	}
-      
+
 	int osites=CoarseGrid()->oSites();
 	autoView( A_v  , _A[k], AcceleratorWrite);
 	autoView( FT_v  , FT[k], AcceleratorRead);
-	accelerator_for(sss, osites, 1, {
-	    for(int j=0;j<nbasis;j++){
-	      A_v[sss](i,j) = FT_v[sss](j);
-	    }
+	accelerator_for(sss, osites, nbasis, {
+	    int j = acceleratorSIMTlane(nbasis);
+	    A_v[sss](i,j) = FT_v[sss](j);
         });
       }
       tinv+=usecond();
@@ -595,13 +639,13 @@ public:
 
     // Only needed if nonhermitian
     if ( ! hermitian ) {
-      //      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
-      //      PopulateAdag();
+      std::cout << GridLogMessage<<"PopulateAdag  "<<std::endl;
+      PopulateAdag();
     }
 
-    for(int p=0;p<geom.npoint;p++){
-      std::cout << " _A["<<p<<"] "<<norm2(_A[p])<<std::endl;
-    }
+    //    for(int p=0;p<geom.npoint;p++){
+    //      std::cout << " _A["<<p<<"] "<<norm2(_A[p])<<std::endl;
+    //    }
 
     // Need to write something to populate Adag from A
     ExchangeCoarseLinks();
@@ -616,7 +660,7 @@ public:
   void ExchangeCoarseLinks(void){
     for(int p=0;p<geom.npoint;p++){
       _A[p] = Cell.ExchangePeriodic(_A[p]);
-      //      _Adag[p]= Cell.ExchangePeriodic(_Adag[p]);
+      if ( !hermitian ) _Adag[p]= Cell.ExchangePeriodic(_Adag[p]);
     }
   }
   virtual  void Mdiag    (const Field &in, Field &out){ GRID_ASSERT(0);};

Grid/allocator/MemoryManager.cc

@@ -63,12 +63,10 @@ void MemoryManager::PrintBytes(void)
   std::cout << " MemoryManager : "<<(total_device>>20)<<" accelerator Mbytes "<<std::endl;
   std::cout << " MemoryManager : "<<(total_host>>20)  <<" cpu         Mbytes "<<std::endl;
   uint64_t cacheBytes;
-  cacheBytes = CacheBytes[Cpu];
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" cpu cache Mbytes "<<std::endl;
-  cacheBytes = CacheBytes[Acc];
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" acc cache Mbytes "<<std::endl;
-  cacheBytes = CacheBytes[Shared];
-  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" shared cache Mbytes "<<std::endl;
+  cacheBytes = HostCacheBytes();
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" cpu alloc cache Mbytes "<<std::endl;
+  cacheBytes = DeviceCacheBytes();
+  std::cout << " MemoryManager : "<<(cacheBytes>>20) <<" acc alloc cache Mbytes "<<std::endl;
   
 #ifdef GRID_CUDA
   cuda_mem();

Grid/allocator/MemoryManager.h

@@ -113,7 +113,7 @@ private:
   static void *Insert(void *ptr,size_t bytes,AllocationCacheEntry *entries,int ncache,int &victim,uint64_t &cbytes) ;
   static void *Lookup(size_t bytes,AllocationCacheEntry *entries,int ncache,uint64_t &cbytes) ;
 
- public:
+public:  
   static void PrintBytes(void);
   static void Audit(std::string s);
   static void Init(void);
@@ -215,6 +215,7 @@ private:
   static void NotifyDeletion(void * CpuPtr);
   static void Print(void);
   static void PrintAll(void);
+  static void EvictAll(void);
   static void PrintState( void* CpuPtr);
   static int   isOpen   (void* CpuPtr);
   static void  ViewClose(void* CpuPtr,ViewMode mode);

Grid/allocator/MemoryManagerCache.cc

@@ -79,6 +79,25 @@ void MemoryManager::EntryErase(uint64_t CpuPtr)
   auto AccCache = EntryLookup(CpuPtr);
   AccViewTable.erase(CpuPtr);
 }
+/////////////////////////////////////////////////////////////////////////////////
+// LRU membership invariant:
+//
+//    LRU_valid == 1   <=>   AccPtr != NULL  &&  accLock == 0  &&  cpuLock == 0
+//
+// i.e. the LRU queue contains exactly the device-resident, completely unlocked
+// entries -- the evictable set.  Membership is maintained EAGERLY at the lock
+// 0<->1 edges, O(1) via the stored LRU_entry iterator:
+//
+//    AcceleratorViewOpen   lock 0->1 : LRUremove   (gated on LRU_valid)
+//    AcceleratorViewClose  accLock->0: LRUinsert   (AccPtr necessarily exists)
+//    CpuViewOpen           lock 0->1 : LRUremove   (gated on LRU_valid)
+//    CpuViewClose          cpuLock->0: LRUinsert   (iff AccPtr exists)
+//    Evict/AccDiscard                : LRUremove   (frees the device copy)
+//
+// Consequences: victims taken from LRU.back() are evictable by construction;
+// Evict() on a locked entry is an invariant violation (asserted), and the
+// eviction loops (EvictVictims/EvictAll) cannot spin.
+/////////////////////////////////////////////////////////////////////////////////
 void  MemoryManager::LRUinsert(AcceleratorViewEntry &AccCache)
 {
   GRID_ASSERT(AccCache.LRU_valid==0);
@@ -130,21 +149,21 @@ void MemoryManager::Evict(AcceleratorViewEntry &AccCache)
 {
   ///////////////////////////////////////////////////////////////////////////
   // Make CPU consistent, remove from Accelerator, remove from LRU, LEAVE CPU only entry
-  // Cannot be acclocked. If allocated must be in LRU pool.
+  // Cannot be locked. If allocated must be in LRU pool.
   //
-  // Nov 2022... Felix issue: Allocating two CpuPtrs, can have an entry in LRU-q with CPUlock.
-  //                          and require to evict the AccPtr copy. Eviction was a mistake in CpuViewOpen
-  //                          but there is a weakness where CpuLock entries are attempted for erase
-  //                          Take these OUT LRU queue when CPU locked?
-  //                          Cannot take out the table as cpuLock data is important.
+  // (Historical: a Nov 2022 incident (two CpuPtrs; eviction called from
+  //  CpuViewOpen -- since excised) could present a cpuLocked entry here, and
+  //  silent-return guards were added.  The LRU membership invariant (see
+  //  LRUinsert) now excludes ALL locked entries from the queue eagerly at the
+  //  lock edges, so a locked victim is an invariant violation: asserted.)
   ///////////////////////////////////////////////////////////////////////////
   GRID_ASSERT(AccCache.state!=Empty);
   
   mprintf("MemoryManager: Evict CpuPtr %lx AccPtr %lx cpuLock %ld accLock %ld",
 	  (uint64_t)AccCache.CpuPtr,(uint64_t)AccCache.AccPtr,
 	  (uint64_t)AccCache.cpuLock,(uint64_t)AccCache.accLock); 
-  if (AccCache.accLock!=0) return;
-  if (AccCache.cpuLock!=0) return;
+  GRID_ASSERT(AccCache.accLock==0);
+  GRID_ASSERT(AccCache.cpuLock==0);
   if(AccCache.state==AccDirty) {
     Flush(AccCache);
   }
@@ -250,6 +269,19 @@ void  MemoryManager::EvictVictims(uint64_t bytes)
     }
   }
 }
+void  MemoryManager::EvictAll(void)
+{
+  while(LRU.size()>0){
+    if ( DeviceLRUBytes > 0){
+      uint64_t victim = LRU.back(); // From the LRU
+      auto AccCacheIterator = EntryLookup(victim);
+      auto & AccCache = AccCacheIterator->second;
+      Evict(AccCache);
+    } else {
+      return;
+    }
+  }
+}
 uint64_t MemoryManager::AcceleratorViewOpen(uint64_t CpuPtr,size_t bytes,ViewMode mode,ViewAdvise hint)
 {
   ////////////////////////////////////////////////////////////////////////////
@@ -383,6 +415,13 @@ void MemoryManager::CpuViewClose(uint64_t CpuPtr)
   GRID_ASSERT(AccCache.accLock==0);
 
   AccCache.cpuLock--;
+  // Return to LRU queue when fully unlocked -- mirrors AcceleratorViewClose.
+  // Asymmetry vs the Acc side: a device copy need not exist for a host view;
+  // only device-resident entries belong in the (evictable) LRU queue.
+  if( (AccCache.cpuLock==0) && (AccCache.AccPtr!=(uint64_t)NULL) ) {
+    dprintf("CpuViewClose %lx cpuLock decremented to zero, move to LRU queue",(uint64_t)CpuPtr);
+    LRUinsert(AccCache);
+  }
 }
 /*
  *  Action  State   StateNext         Flush    Clone
@@ -449,6 +488,14 @@ uint64_t MemoryManager::CpuViewOpen(uint64_t CpuPtr,size_t bytes,ViewMode mode,V
     GRID_ASSERT(0); // should be unreachable
   }
 
+  GRID_ASSERT(AccCache.cpuLock>0);
+  // If view is opened on host must remove from LRU -- mirrors AcceleratorViewOpen.
+  // LRU_valid==1 here implies this is the 0->1 lock edge of a device-resident entry.
+  if(AccCache.LRU_valid==1){
+    dprintf("CpuViewOpen: entry removed from LRU ");
+    LRUremove(AccCache);
+  }
+
   AccCache.transient= transient? EvictNext : 0;
 
   return AccCache.CpuPtr;

Grid/communicator/Communicator_base.h

@@ -238,6 +238,16 @@ public:
   }
   void AllToAll(int dim  ,void *in,void *out,uint64_t words,uint64_t bytes);
   void AllToAll(void  *in,void *out,uint64_t words         ,uint64_t bytes);
+
+  ////////////////////////////////////////////////////////////////////////////
+  // Variable count all to all. Counts and displacements are in units of
+  // "bytes" sized words and are indexed by rank within this communicator.
+  // For exchanges that are a permutation but do not divide evenly between
+  // ranks; AllToAll above is the uniform count special case.
+  ////////////////////////////////////////////////////////////////////////////
+  void AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
+                 void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
+                 uint64_t bytes);
   
   template<class obj> void Broadcast(int root,obj &data)
   {

Grid/communicator/Communicator_mpi3.cc

@@ -945,5 +945,25 @@ void CartesianCommunicator::AllToAll(void  *in,void *out,uint64_t words,uint64_t
   MPI_Alltoall(in,iwords,object,out,iwords,object,communicator);
   MPI_Type_free(&object);
 }
+void CartesianCommunicator::AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
+                                      void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
+                                      uint64_t bytes)
+{
+  FlightRecorder::StepLog("AllToAllV");
+  GRID_ASSERT(sendcounts.size()==(size_t)_Nprocessors);
+  GRID_ASSERT(senddispls.size()==(size_t)_Nprocessors);
+  GRID_ASSERT(recvcounts.size()==(size_t)_Nprocessors);
+  GRID_ASSERT(recvdispls.size()==(size_t)_Nprocessors);
+  // MPI counts are "int"; the caller sizes the word to keep them in range
+  int ibytes = bytes;
+  GRID_ASSERT(bytes == (uint64_t)ibytes);
+  MPI_Datatype object;
+  MPI_Type_contiguous(ibytes,MPI_BYTE,&object);
+  MPI_Type_commit(&object);
+  int ierr = MPI_Alltoallv(in ,(int *)&sendcounts[0],(int *)&senddispls[0],object,
+                           out,(int *)&recvcounts[0],(int *)&recvdispls[0],object,communicator);
+  GRID_ASSERT(ierr==0);
+  MPI_Type_free(&object);
+}
 
 NAMESPACE_END(Grid);

Grid/communicator/Communicator_none.cc

@@ -113,6 +113,17 @@ void CartesianCommunicator::AllToAll(void  *in,void *out,uint64_t words,uint64_t
 {
   bcopy(in,out,bytes*words);
 }
+void CartesianCommunicator::AllToAllV(void *in ,const std::vector<int> &sendcounts,const std::vector<int> &senddispls,
+                                      void *out,const std::vector<int> &recvcounts,const std::vector<int> &recvdispls,
+                                      uint64_t bytes)
+{
+  // Single rank: the exchange degenerates to a copy of our own segment
+  GRID_ASSERT(sendcounts.size()==1);
+  GRID_ASSERT(recvcounts.size()==1);
+  GRID_ASSERT(sendcounts[0]==recvcounts[0]);
+  bcopy((char *)in +(uint64_t)senddispls[0]*bytes,
+        (char *)out+(uint64_t)recvdispls[0]*bytes,bytes*(uint64_t)sendcounts[0]);
+}
 
 int  CartesianCommunicator::RankWorld(void){return 0;}
 void CartesianCommunicator::Barrier(void){}

Grid/lattice/Lattice_base.h

@@ -289,7 +289,7 @@ public:
   ///////////////////////////////////////////
   // move constructor
   ///////////////////////////////////////////
-  Lattice(Lattice && r){ 
+  Lattice(Lattice && r) noexcept { 
     this->_grid = r.Grid();
     this->_odata      = r._odata;
     this->_odata_size = r._odata_size;
@@ -330,7 +330,7 @@ public:
   ///////////////////////////////////////////
   // Move assignment possible if same type
   ///////////////////////////////////////////
-  inline Lattice<vobj> & operator = (Lattice<vobj> && r){
+  inline Lattice<vobj> & operator = (Lattice<vobj> && r) noexcept {
 
     resize(0); // deletes if appropriate
     this->_grid       = r.Grid();

Grid/lattice/Lattice_reduction_gpu.h

@@ -197,12 +197,15 @@ __global__ void reduceKernel(const vobj *lat, sobj *buffer, Iterator n) {
 /////////////////////////////////////////////////////////////////////////////////////////////////////////
 // Possibly promote to double and sum
 /////////////////////////////////////////////////////////////////////////////////////////////////////////
+
+#undef GRID_REDUCTION_TIMING
+
 template <class vobj>
-inline typename vobj::scalar_objectD sumD_gpu_small(const vobj *lat, Integer osites) 
+inline typename vobj::scalar_objectD sumD_gpu_small(const vobj *lat, Integer osites)
 {
   typedef typename vobj::scalar_objectD sobj;
   typedef decltype(lat) Iterator;
-  
+
   Integer nsimd= vobj::Nsimd();
   Integer size = osites*nsimd;
 
@@ -211,41 +214,188 @@ inline typename vobj::scalar_objectD sumD_gpu_small(const vobj *lat, Integer osi
   GRID_ASSERT(ok);
 
   Integer smemSize = numThreads * sizeof(sobj);
-  // Move out of UVM
-  // Turns out I had messed up the synchronise after move to compute stream
-  // as running this on the default stream fools the synchronise
   deviceVector<sobj> buffer(numBlocks);
   sobj *buffer_v = &buffer[0];
   sobj result;
+
+#ifdef GRID_REDUCTION_TIMING
+  RealD t_kernel = -usecond();
+#endif
   reduceKernel<<< numBlocks, numThreads, smemSize, computeStream >>>(lat, buffer_v, size);
   accelerator_barrier();
+#ifdef GRID_REDUCTION_TIMING
+  t_kernel += usecond();
+  RealD t_d2h = -usecond();
+#endif
   acceleratorCopyFromDevice(buffer_v,&result,sizeof(result));
+#ifdef GRID_REDUCTION_TIMING
+  t_d2h += usecond();
+  std::cout << GridLogDebug << "  sumD_gpu_small"
+            << " sizeof(sobj)=" << sizeof(sobj)
+            << " blocks=" << numBlocks << " threads=" << numThreads
+            << " kernel+barrier=" << t_kernel << " us"
+            << " D2H=" << t_d2h << " us" << std::endl;
+#endif
   return result;
 }
 
+// Fused pack+reduce: reads R words of each vobj at word offset 'base',
+// accumulates directly into iVector<iScalar<scalarD>,R> without staging
+// through an intermediate bundle buffer.  One HBM pass instead of three.
+template <int R, class vobj, class sobj, class Iterator>
+__device__ void packReduceBlocks(
+    const iScalar<typename vobj::vector_type> *idat,
+    sobj *g_odata, Iterator osites, int base, int words)
+{
+  constexpr Iterator nsimd = vobj::Nsimd();
+  Iterator blockSize = blockDim.x;
+
+  extern __shared__ __align__(COALESCE_GRANULARITY) unsigned char shmem_pointer[];
+  sobj *sdata = (sobj *)shmem_pointer;
+
+  Iterator tid      = threadIdx.x;
+  Iterator i        = blockIdx.x * (blockSize * 2) + threadIdx.x;
+  Iterator gridSize = blockSize * 2 * gridDim.x;
+  sobj mySum = Zero();
+
+  while (i < osites * nsimd) {
+    Iterator lane = i % nsimd;
+    Iterator ss   = i / nsimd;
+    sobj tmpD; zeroit(tmpD);
+    for (int k = 0; k < R; k++) {
+      auto w = extractLane(lane, idat[ss * words + base + k]);
+      iScalar<typename vobj::scalar_typeD> wd; wd = w;
+      tmpD._internal[k] = wd;
+    }
+    mySum += tmpD;
+
+    if (i + blockSize < osites * nsimd) {
+      lane = (i + blockSize) % nsimd;
+      ss   = (i + blockSize) / nsimd;
+      sobj tmpD2; zeroit(tmpD2);
+      for (int k = 0; k < R; k++) {
+        auto w = extractLane(lane, idat[ss * words + base + k]);
+        iScalar<typename vobj::scalar_typeD> wd; wd = w;
+        tmpD2._internal[k] = wd;
+      }
+      mySum += tmpD2;
+    }
+    i += gridSize;
+  }
+
+  reduceBlock(sdata, mySum, tid);
+  if (tid == 0) g_odata[blockIdx.x] = sdata[0];
+}
+
+template <int R, class vobj, class sobj, class Iterator>
+__global__ void packReduceKernel(
+    const iScalar<typename vobj::vector_type> *idat,
+    sobj *buffer, Iterator osites, int base, int words)
+{
+  Iterator blockSize = blockDim.x;
+
+  packReduceBlocks<R, vobj, sobj>(idat, buffer, osites, base, words);
+
+  if (gridDim.x > 1) {
+    const Iterator tid = threadIdx.x;
+    __shared__ bool amLast;
+    extern __shared__ __align__(COALESCE_GRANULARITY) unsigned char shmem_pointer[];
+    sobj *smem = (sobj *)shmem_pointer;
+
+    acceleratorFence();
+
+    if (tid == 0) {
+      unsigned int ticket = atomicInc(&retirementCount, gridDim.x);
+      amLast = (ticket == gridDim.x - 1);
+    }
+    acceleratorSynchroniseAll();
+
+    if (amLast) {
+      Iterator i = tid;
+      sobj mySum = Zero();
+      while (i < (Iterator)gridDim.x) {
+        mySum += buffer[i];
+        i += blockSize;
+      }
+      reduceBlock(smem, mySum, tid);
+      if (tid == 0) {
+        buffer[0] = smem[0];
+        retirementCount = 0;
+      }
+    }
+  }
+}
+
+template<int R, class vobj>
+inline void sumD_gpu_reduce_words(const vobj *lat, Integer osites,
+                                   typename vobj::scalar_typeD *ret_p, int base)
+{
+  typedef typename vobj::vector_type  vector;
+  typedef typename vobj::scalar_typeD scalarD;
+  using BundleScalarD = iVector<iScalar<scalarD>, R>;
+
+  constexpr int Nsimd = vobj::Nsimd();
+  const int words = sizeof(vobj) / sizeof(vector);
+  const iScalar<vector> *idat = (const iScalar<vector> *)lat;
+
+  Integer size = (Integer)osites * Nsimd;
+  Integer numThreads, numBlocks;
+  int ok = getNumBlocksAndThreads(size, sizeof(BundleScalarD), numThreads, numBlocks);
+  GRID_ASSERT(ok);
+
+  Integer smemSize = numThreads * sizeof(BundleScalarD);
+  deviceVector<BundleScalarD> buffer(numBlocks);
+  BundleScalarD *buffer_v = &buffer[0];
+  BundleScalarD result;
+
+#ifdef GRID_REDUCTION_TIMING
+  RealD t_kernel = -usecond();
+#endif
+  packReduceKernel<R, vobj, BundleScalarD, Integer>
+    <<<numBlocks, numThreads, smemSize, computeStream>>>
+    (idat, buffer_v, osites, base, words);
+  accelerator_barrier();
+#ifdef GRID_REDUCTION_TIMING
+  t_kernel += usecond();
+  RealD t_d2h = -usecond();
+#endif
+  acceleratorCopyFromDevice(buffer_v, &result, sizeof(result));
+#ifdef GRID_REDUCTION_TIMING
+  t_d2h += usecond();
+  std::cout << GridLogDebug << " sumD_gpu_reduce_words R=" << R
+            << " base=" << base
+            << " kernel=" << t_kernel << " D2H=" << t_d2h << " us" << std::endl;
+#endif
+
+  for (int k = 0; k < R; k++)
+    ret_p[base + k] = TensorRemove(result._internal[k]);
+}
+
 template <class vobj>
 inline typename vobj::scalar_objectD sumD_gpu_large(const vobj *lat, Integer osites)
 {
-  typedef typename vobj::vector_type  vector;
-  typedef typename vobj::scalar_typeD scalarD;
-  typedef typename vobj::scalar_objectD sobj;
-  sobj ret;
+  typedef typename vobj::vector_type    vector;
+  typedef typename vobj::scalar_typeD   scalarD;
+  typedef typename vobj::scalar_objectD sobjD;
+
+  const int words = sizeof(vobj) / sizeof(vector);
+  sobjD ret; zeroit(ret);
   scalarD *ret_p = (scalarD *)&ret;
-  
-  const int words = sizeof(vobj)/sizeof(vector);
 
-  deviceVector<vector> buffer(osites);
-  vector *dat = (vector *)lat;
-  vector *buf = &buffer[0];
-  iScalar<vector> *tbuf =(iScalar<vector> *)  &buffer[0];
-  for(int w=0;w<words;w++) {
+#ifdef GRID_REDUCTION_TIMING
+  RealD t_large = -usecond();
+#endif
+  int w = 0;
+  while (w + 12 <= words) { sumD_gpu_reduce_words<12>(lat, osites, ret_p, w); w += 12; }
+  while (w +  4 <= words) { sumD_gpu_reduce_words< 4>(lat, osites, ret_p, w); w +=  4; }
+  while (w      <  words) { sumD_gpu_reduce_words< 1>(lat, osites, ret_p, w); w +=  1; }
+#ifdef GRID_REDUCTION_TIMING
+  t_large += usecond();
+  std::cout << GridLogDebug << "sumD_gpu_large"
+            << " sizeof(sobjD)=" << sizeof(sobjD)
+            << " words=" << words << " total=" << t_large << " us" << std::endl;
+#endif
 
-    accelerator_for(ss,osites,1,{
-	buf[ss] = dat[ss*words+w];
-      });
-      
-    ret_p[w] = sumD_gpu_small(tbuf,osites);
-  }
   return ret;
 }
 
@@ -288,5 +438,11 @@ inline typename vobj::scalar_object sum_gpu_large(const vobj *lat, Integer osite
   result = sumD_gpu_large(lat,osites);
   return result;
 }
+template<class Word> Word checksum_gpu(Word *vec,uint64_t L)
+{
+  Word w;
+  bzero(&w,sizeof(w));
+  return w;
+}
 
 NAMESPACE_END(Grid);

Grid/lattice/Lattice_reduction_sycl.h

@@ -6,28 +6,27 @@ NAMESPACE_BEGIN(Grid);
 
 
 template <class vobj>
-inline typename vobj::scalar_objectD sumD_gpu_tensor(const vobj *lat, Integer osites) 
+inline typename vobj::scalar_objectD sumD_gpu_tensor(const vobj *lat, Integer osites)
 {
-  typedef typename vobj::scalar_object sobj;
+  typedef typename vobj::scalar_object  sobj;
   typedef typename vobj::scalar_objectD sobjD;
 
-  sobj identity; zeroit(identity);
-  sobj ret; zeroit(ret);
-  Integer nsimd= vobj::Nsimd();
-  { 
-    sycl::buffer<sobj, 1> abuff(&ret, {1});
+  sobjD identity; zeroit(identity);
+  sobjD ret;      zeroit(ret);
+  {
+    sycl::buffer<sobjD, 1> abuff(&ret, {1});
     theGridAccelerator->submit([&](sycl::handler &cgh) {
-      auto Reduction = sycl::reduction(abuff,cgh,identity,std::plus<>());
-      cgh.parallel_for(sycl::range<1>{osites},
-                      Reduction,
-                      [=] (sycl::id<1> item, auto &sum) {
-                        auto osite   = item[0];
-                        sum +=Reduce(lat[osite]);
-                      });
+      auto Reduction = sycl::reduction(abuff, cgh, identity, std::plus<>());
+      cgh.parallel_for(sycl::range<1>{(size_t)osites},
+                       Reduction,
+                       [=](sycl::id<1> item, auto &sum) {
+                         sobj s = Reduce(lat[item[0]]);
+                         sobjD sd; sd = s;
+                         sum += sd;
+                       });
     });
   }
-  sobjD dret; convertType(dret,ret);
-  return dret;
+  return ret;
 }
 
 template <class vobj>

Grid/lattice/Lattice_slicesum_core.h

@@ -1,7 +1,6 @@
 #pragma once
 
 #if defined(GRID_CUDA)
-
 #include <cub/cub.cuh>
 #define gpucub cub
 #define gpuError_t cudaError_t
@@ -57,8 +56,13 @@ inline void sliceSumReduction_cub_small(const vobj *Data,
   //copy offsets to device
   acceleratorCopyToDeviceAsynch(&offsets[0],d_offsets,sizeof(int)*(rd+1),computeStream);
   
+#if defined(__CUDACC__) && (__CUDACC_VER_MAJOR__ >= 13)
+  #define GRID_CUB_SUM_OP ::cuda::std::plus<>{}
+#else
+  #define GRID_CUB_SUM_OP ::gpucub::Sum()
+#endif
   
-  gpuError_t gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p,d_out, rd, d_offsets, d_offsets+1, ::gpucub::Sum(), zero_init, computeStream);
+  gpuError_t gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p,d_out, rd, d_offsets, d_offsets+1, GRID_CUB_SUM_OP, zero_init, computeStream);
   if (gpuErr!=gpuSuccess) {
     std::cout << GridLogError << "Lattice_slicesum_gpu.h: Encountered error during gpucub::DeviceSegmentedReduce::Reduce (setup)! Error: " << gpuErr <<std::endl;
     exit(EXIT_FAILURE);
@@ -82,11 +86,13 @@ inline void sliceSumReduction_cub_small(const vobj *Data,
   });
   
   //issue segmented reductions in computeStream
-  gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p, d_out, rd, d_offsets, d_offsets+1,::gpucub::Sum(), zero_init, computeStream);
+  gpuErr = gpucub::DeviceSegmentedReduce::Reduce(temp_storage_array, temp_storage_bytes, rb_p, d_out, rd, d_offsets, d_offsets+1, GRID_CUB_SUM_OP, zero_init, computeStream);
   if (gpuErr!=gpuSuccess) {
     std::cout << GridLogError << "Lattice_slicesum_gpu.h: Encountered error during gpucub::DeviceSegmentedReduce::Reduce! Error: " << gpuErr <<std::endl;
     exit(EXIT_FAILURE);
   }
+
+#undef GRID_CUB_SUM_OP
   
   acceleratorCopyFromDeviceAsynch(d_out,&lvSum[0],rd*sizeof(vobj),computeStream);
   

Grid/lattice/Lattice_trace.h

@@ -124,6 +124,68 @@ Lattice<iScalar<iScalar<iMatrix<vComplexD, N> > > > Inverse(const Lattice<iScala
   return ret;
 }
 
+template<int N>
+Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > > , N> > Inverse(const Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > >, N> > &Umu)
+{
+  GridBase *grid=Umu.Grid();
+  auto lvol = grid->lSites();
+  Lattice<iMatrix<iScalar<iScalar<iScalar<vComplexD> > >, N > > ret(grid);
+  
+  autoView(Umu_v,Umu,CpuRead);
+  autoView(ret_v,ret,CpuWrite);
+  thread_for(site,lvol,{
+    Eigen::MatrixXcd EigenU = Eigen::MatrixXcd::Zero(N,N);
+    Coordinate lcoor;
+    grid->LocalIndexToLocalCoor(site, lcoor);
+    iMatrix<iScalar<iScalar<iScalar<ComplexD> > >, N > Us;
+    iMatrix<iScalar<iScalar<iScalar<ComplexD> > >, N > Ui;
+    peekLocalSite(Us, Umu_v, lcoor);
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	EigenU(i,j) = Us(i,j)()()();
+      }}
+    Eigen::MatrixXcd EigenUinv = EigenU.inverse();
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	Ui(i,j)()()() = EigenUinv(i,j);
+      }}
+    pokeLocalSite(Ui,ret_v,lcoor);
+  });
+  return ret;
+}
+
+template<int N>
+Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > > , N> > Inverse(const Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > >, N> > &Umu)
+{
+  GridBase *grid=Umu.Grid();
+  auto lvol = grid->lSites();
+  Lattice<iMatrix<iScalar<iScalar<iScalar<iScalar<vComplexD> > > >, N > > ret(grid);
+  
+  autoView(Umu_v,Umu,CpuRead);
+  autoView(ret_v,ret,CpuWrite);
+  thread_for(site,lvol,{
+    Eigen::MatrixXcd EigenU = Eigen::MatrixXcd::Zero(N,N);
+    Coordinate lcoor;
+    grid->LocalIndexToLocalCoor(site, lcoor);
+    iMatrix<iScalar<iScalar<iScalar<iScalar<ComplexD> > > >, N > Us;
+    iMatrix<iScalar<iScalar<iScalar<iScalar<ComplexD> > > >, N > Ui;
+    peekLocalSite(Us, Umu_v, lcoor);
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	EigenU(i,j) = Us(i,j)()()()();
+      }}
+    Eigen::MatrixXcd EigenUinv = EigenU.inverse();
+    for(int i=0;i<N;i++){
+      for(int j=0;j<N;j++){
+	Ui(i,j)()()()() = EigenUinv(i,j);
+      }}
+    pokeLocalSite(Ui,ret_v,lcoor);
+  });
+  return ret;
+}
+
+
+
 
 NAMESPACE_END(Grid);
 #endif

Grid/parallelIO/BinaryIO.cc

@@ -2,3 +2,7 @@
 
 int                    Grid::BinaryIO::latticeWriteMaxRetry = -1;
 Grid::BinaryIO::IoPerf Grid::BinaryIO::lastPerf;
+
+// Target size of a single contiguous file extent under BINARYIO_AGGREGATE.
+// 4MB is around the knee for Lustre; exposed so it can be swept at runtime.
+uint64_t               Grid::BinaryIO::aggregateTargetBytes = 4*1024*1024;

Grid/parallelIO/BinaryIO.h

@@ -39,6 +39,7 @@
 #endif
 
 #include <arpa/inet.h>
+#include <sys/stat.h>
 #include <algorithm>
 
 NAMESPACE_BEGIN(Grid);
@@ -87,6 +88,7 @@ class BinaryIO {
 
   static IoPerf lastPerf;
   static int latticeWriteMaxRetry;
+  static uint64_t aggregateTargetBytes;
 
   /////////////////////////////////////////////////////////////////////////////
   // more byte manipulation helpers
@@ -253,12 +255,392 @@ class BinaryIO {
   // Read or Write distributed lexico array of ANY object to a specific location in file 
   //////////////////////////////////////////////////////////////////////////////////////
 
+  static const int BINARYIO_AGGREGATE     = 0x20;
   static const int BINARYIO_MASTER_APPEND = 0x10;
   static const int BINARYIO_UNORDERED     = 0x08;
   static const int BINARYIO_LEXICOGRAPHIC = 0x04;
   static const int BINARYIO_READ          = 0x02;
   static const int BINARYIO_WRITE         = 0x01;
 
+#ifdef USE_MPI_IO
+  /////////////////////////////////////////////////////////////////////////////
+  // Aggregation: self controlled transposition onto an I/O friendly layout.
+  //
+  // Under BINARYIO_LEXICOGRAPHIC the subarray file view handed to MPI-IO has
+  // contiguous runs of only lLattice[0]*sizeof(fobj) bytes -- a few KB for
+  // typical local volumes. Rather than rely on collective buffering to repair
+  // that, redistribute the payload ourselves so every rank owns a contiguous
+  // range of the global lexicographic site ordering, then issue large plain
+  // contiguous writes.
+  //
+  // "Un-splitting" the nunsplit fastest dimensions means the row of ranks
+  // sharing the remaining process coordinates collectively owns whole global
+  // hyperplanes. All data movement is then confined to that row communicator.
+  // Every rank still owns exactly lSites() sites afterwards, so the exchange is
+  // a pure permutation and needs no divisibility condition on the process grid.
+  /////////////////////////////////////////////////////////////////////////////
+  struct AggregationPlan {
+    int      nunsplit{0};  // number of fastest dimensions un-split
+    int      rowsize{0};   // ranks in the aggregation (row) communicator
+    int      rowrank{0};   // our logical (lexicographic) index within the row
+    uint64_t lsites{0};    // sites per rank -- invariant under the permutation
+    uint64_t chunk{0};     // sites in one globally contiguous run owned by the row
+    std::unique_ptr<CartesianCommunicator> rowcomm;
+    // counts and displacements are indexed by rank within rowcomm
+    std::vector<int>      sendcounts, senddispls, recvcounts, recvdispls;
+    std::vector<uint64_t> scatter;     // recv slot -> slot in the aggregated buffer
+    std::vector<uint64_t> extentGsite; // global lex site index of extent start
+    std::vector<uint64_t> extentLocal; // offset of extent within aggregated buffer
+    std::vector<uint64_t> extentSites; // sites in this extent
+  };
+
+  static inline void BuildAggregationPlan(GridBase *grid,uint64_t fobjSize,AggregationPlan &p)
+  {
+    int ndim           = grid->Dimensions();
+    Coordinate psizes  = grid->ProcessorGrid();
+    Coordinate pcoor   = grid->ThisProcessorCoor();
+    Coordinate gLattice= grid->GlobalDimensions();
+    Coordinate lLattice= grid->LocalDimensions();
+    Coordinate lstart  = grid->LocalStarts();
+
+    uint64_t lsites = grid->lSites();
+    p.lsites = lsites;
+
+    //////////////////////////////////////////////////////////////////////////
+    // Un-splitting dims 0..k-1 gives the row a contiguous run of
+    //     chunk(k) = prod_{d<k} gLattice[d] * lLattice[k]
+    // sites, and each rank writes extents of min(chunk,lsites). Take the
+    // smallest k that reaches the target so we disturb as few dimensions --
+    // and move as little data -- as possible.
+    //////////////////////////////////////////////////////////////////////////
+    int k = ndim-1;
+    for(int trial=1; trial<ndim; trial++){
+      uint64_t chunk = lLattice[trial];
+      for(int d=0; d<trial; d++) chunk *= gLattice[d];
+      if ( std::min(chunk,lsites)*fobjSize >= aggregateTargetBytes ) { k = trial; break; }
+    }
+    p.nunsplit = k;
+
+    //////////////////////////////////////////////////////////////////////////
+    // The box the row collectively owns, expressed in global coordinates.
+    // Restricting the global lexicographic order to this box preserves the
+    // ordering, so the row index below is monotone in the global index.
+    //////////////////////////////////////////////////////////////////////////
+    Coordinate B(ndim), S(ndim);
+    for(int d=0; d<ndim; d++){
+      if ( d<k ) { B[d] = gLattice[d]; S[d] = 0;         }
+      else       { B[d] = lLattice[d]; S[d] = lstart[d]; }
+    }
+
+    uint64_t chunk = lLattice[k];
+    for(int d=0; d<k; d++) chunk *= gLattice[d];
+    p.chunk = chunk;
+
+    //////////////////////////////////////////////////////////////////////////
+    // Row communicator: the ranks sharing the process coordinates of the slow
+    // (still split) dimensions. This is the sub-division the Cartesian
+    // communicator already performs for AllToAll(dim,...), widened from one
+    // dimension to the k fastest.
+    //////////////////////////////////////////////////////////////////////////
+    Coordinate row(ndim,1);
+    for(int d=0; d<k; d++) row[d] = psizes[d];
+    int srank;
+    p.rowcomm.reset(new CartesianCommunicator(row,*grid,srank));
+    p.rowsize = p.rowcomm->ProcessorCount();
+
+    //////////////////////////////////////////////////////////////////////////
+    // Our logical index in the row is the forward lexicographic index of the
+    // un-split process coordinates, so that increasing logical index means
+    // increasing global lexicographic position in the file. The communicator
+    // numbers its own ranks by the reversed (MPI) convention, so build the map
+    // between the two rather than assuming either.
+    //////////////////////////////////////////////////////////////////////////
+    int64_t logical=0, lstride=1;
+    for(int d=0; d<k; d++){ logical += pcoor[d]*lstride; lstride *= psizes[d]; }
+    GRID_ASSERT(lstride == (int64_t)p.rowsize);
+    p.rowrank = (int)logical;
+
+    std::vector<uint64_t> commOf(p.rowsize,0);
+    commOf[p.rowrank] = (uint64_t)p.rowcomm->ThisRank();
+    p.rowcomm->GlobalSumVector(&commOf[0],p.rowsize);
+
+    uint64_t mystart = (uint64_t)p.rowrank * lsites;
+    uint64_t myend   = mystart + lsites;
+
+    Coordinate lcoor(ndim), bcoor(ndim), gcoor(ndim);
+
+    //////////////////////////////////////////////////////////////////////////
+    // Send side. Walking our local sites in local lexicographic order walks
+    // the row index monotonically, so the send buffer is iodata untouched and
+    // we need only the per destination counts.
+    //////////////////////////////////////////////////////////////////////////
+    std::vector<int> sendLogical(p.rowsize,0);
+    for(uint64_t L=0; L<lsites; L++){
+      Lexicographic::CoorFromIndex(lcoor,L,lLattice);
+      for(int d=0; d<ndim; d++) bcoor[d] = (d<k) ? (lstart[d]+lcoor[d]) : lcoor[d];
+      int64_t ri; Lexicographic::IndexFromCoor(bcoor,ri,B);
+      sendLogical[ ri/(int64_t)lsites ]++;
+    }
+    p.sendcounts.assign(p.rowsize,0);
+    p.senddispls.assign(p.rowsize,0);
+    { int64_t disp=0;
+      for(int d=0; d<p.rowsize; d++){          // send buffer is in logical order
+        int c = (int)commOf[d];
+        p.sendcounts[c] = sendLogical[d];
+        p.senddispls[c] = (int)disp;
+        disp += sendLogical[d];
+      }
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // Receive side. For each slot of our aggregated range work out which rank
+    // of the row owns it. Within one source the slots arrive in increasing row
+    // index order, which is the order the source sends them in.
+    //////////////////////////////////////////////////////////////////////////
+    std::vector<int> recvLogical(p.rowsize,0), recvDisplLogical(p.rowsize,0);
+    std::vector<int> source(lsites);
+    for(uint64_t pos=0; pos<lsites; pos++){
+      Lexicographic::CoorFromIndex(bcoor,(int64_t)(mystart+pos),B);
+      int64_t j=0, jstride=1;
+      for(int d=0; d<k; d++){ j += (bcoor[d]/lLattice[d])*jstride; jstride *= psizes[d]; }
+      source[pos] = (int)j;
+      recvLogical[j]++;
+    }
+    p.recvcounts.assign(p.rowsize,0);
+    p.recvdispls.assign(p.rowsize,0);
+    { int64_t disp=0;
+      for(int s=0; s<p.rowsize; s++){         // recv buffer is in logical order
+        int c = (int)commOf[s];
+        recvDisplLogical[s] = (int)disp;
+        p.recvcounts[c] = recvLogical[s];
+        p.recvdispls[c] = (int)disp;
+        disp += recvLogical[s];
+      }
+    }
+    p.scatter.resize(lsites);
+    {
+      std::vector<int> fill(p.rowsize,0);
+      for(uint64_t pos=0; pos<lsites; pos++){
+        int j = source[pos];
+        p.scatter[ recvDisplLogical[j] + fill[j]++ ] = pos;
+      }
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // The two sides are derived independently; make them check each other.
+    //////////////////////////////////////////////////////////////////////////
+    {
+      std::vector<uint64_t> sendc(p.rowsize),recvc(p.rowsize);
+      for(int c=0;c<p.rowsize;c++) sendc[c]=(uint64_t)p.sendcounts[c];
+      p.rowcomm->AllToAll(&sendc[0],&recvc[0],1,sizeof(uint64_t));
+      for(int c=0;c<p.rowsize;c++) GRID_ASSERT((int)recvc[c]==p.recvcounts[c]);
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // Decompose our range into globally contiguous file extents.
+    //////////////////////////////////////////////////////////////////////////
+    for(uint64_t c = mystart/chunk; c <= (myend-1)/chunk; c++){
+      uint64_t lo = std::max(mystart, c*chunk);
+      uint64_t hi = std::min(myend, (c+1)*chunk);
+      Lexicographic::CoorFromIndex(bcoor,(int64_t)(c*chunk),B);
+      for(int d=0;d<ndim;d++) gcoor[d] = (d<k) ? bcoor[d] : bcoor[d]+S[d];
+      int64_t gbase; Lexicographic::IndexFromCoor(gcoor,gbase,gLattice);
+      p.extentGsite.push_back( (uint64_t)gbase + (lo - c*chunk) );
+      p.extentLocal.push_back( lo - mystart );
+      p.extentSites.push_back( hi - lo );
+    }
+  }
+
+  static inline void ReportAggregationPlan(GridBase *grid,const AggregationPlan &p,uint64_t fobjSize,const char *what)
+  {
+    if ( !grid->IsBoss() ) return;
+    std::cout << GridLogMessage << "IOobject: aggregate " << what
+              << " un-splitting " << p.nunsplit << " fastest dimensions, row of "
+              << p.rowsize << " ranks" << std::endl;
+    std::cout << GridLogMessage << "IOobject: aggregate " << p.extentSites.size()
+              << " extent(s)/rank, first " << p.extentSites[0]*fobjSize/1024./1024. << " MB"
+              << " (target " << aggregateTargetBytes/1024./1024. << " MB)" << std::endl;
+    std::cout << GridLogMessage << "IOobject: aggregate buffer overhead "
+              << p.lsites*fobjSize/1024./1024. << " MB/rank" << std::endl;
+  }
+
+  ////////////////////////////////////////////////////////////////////////////
+  // Stage timings.  The interesting quantity is the slowest rank, since every
+  // stage is followed sooner or later by a synchronisation, so reduce with
+  // GlobalMax rather than reporting whatever the boss happened to see.
+  ////////////////////////////////////////////////////////////////////////////
+  static inline void ReportStages(GridBase *grid,const char *what,
+                                  const std::vector<const char *> &names,
+                                  std::vector<RealD> &useconds)
+  {
+    GRID_ASSERT(names.size()==useconds.size());
+    for(uint64_t i=0;i<useconds.size();i++) grid->GlobalMax(useconds[i]);
+    if ( grid->IsBoss() ) {
+      std::cout << GridLogMessage << "IOobject: aggregate " << what << " stages (max over ranks, s):";
+      for(uint64_t i=0;i<names.size();i++)
+        std::cout << " " << names[i] << " " << useconds[i]/1.0e6;
+      std::cout << std::endl;
+    }
+  }
+
+  template<class fobj>
+  static inline void AggregateExchange(GridBase *grid,AggregationPlan &p,std::vector<fobj> &iodata,
+                                       std::vector<fobj> &aggregated,int forward)
+  {
+    uint64_t lsites = p.lsites;
+    GridStopWatch talloc,tperm,tcomm;
+
+    talloc.Start();
+    std::vector<fobj> tmp(lsites);
+    talloc.Stop();
+
+    if ( forward ) { // iodata (local order) -> aggregated (lexicographic order)
+      tcomm.Start();
+      p.rowcomm->AllToAllV(&iodata[0],p.sendcounts,p.senddispls,
+                           &tmp[0],   p.recvcounts,p.recvdispls,sizeof(fobj));
+      tcomm.Stop();
+      tperm.Start();
+      thread_for(s,lsites,{ aggregated[p.scatter[s]] = tmp[s]; });
+      tperm.Stop();
+    } else {         // aggregated -> iodata, the exact mirror
+      tperm.Start();
+      thread_for(s,lsites,{ tmp[s] = aggregated[p.scatter[s]]; });
+      tperm.Stop();
+      tcomm.Start();
+      p.rowcomm->AllToAllV(&tmp[0],   p.recvcounts,p.recvdispls,
+                           &iodata[0],p.sendcounts,p.senddispls,sizeof(fobj));
+      tcomm.Stop();
+    }
+
+    std::vector<RealD> us = { (RealD)talloc.useconds(), (RealD)tperm.useconds(), (RealD)tcomm.useconds() };
+    ReportStages(grid,forward?"exchange (write)":"exchange (read)",
+                 {"alloc","permute","alltoallv"},us);
+  }
+
+  template<class fobj>
+  static inline void AggregateWrite(GridBase *grid,AggregationPlan &p,std::vector<fobj> &aggregated,
+                                    std::string file,uint64_t offset)
+  {
+    //////////////////////////////////////////////////////////////////////////
+    // All ranks write concurrently into a shared file, so the file must exist
+    // before any of them open it for update, but it does NOT have to be the
+    // right length first: the extents tile the record exactly, so writing them
+    // extends a short file to precisely offset+payload.
+    //
+    // Records are created in sequence, so this payload ends the file: the
+    // length must end up precisely offset+payload. Anything beyond is left
+    // over from whatever the file previously held and must not survive -- a
+    // shorter new record written over a longer old one would otherwise leave
+    // a trailing fragment of the previous contents masquerading as data.
+    // That is the only case needing a truncate, so stat first and truncate
+    // afterwards only when the size actually came out wrong. Measured on
+    // Frontier, an unconditional truncate up front cost 0.22 to 5.4 s per
+    // record -- 15 to 25% of a 19 GB write and 100% of a small one -- while
+    // create, open and close together cost a few milliseconds. It is per
+    // record, so multi record files do not amortise it away.
+    //
+    // ::truncate is used because the C++ standard library cannot express this.
+    // std::filebuf has no length operation at all; ios::trunc only truncates to
+    // zero; seeking past the end and writing a byte can grow a file but never
+    // shrink one; and there is no portable way to recover a descriptor from a
+    // stream in order to call ftruncate. C++17 does finally offer
+    // std::filesystem::resize_file, but that would be Grid's first <filesystem>
+    // dependency and needs -lstdc++fs on the older toolchains still in use.
+    //////////////////////////////////////////////////////////////////////////
+    GridStopWatch tcreate,ttrunc,tbar,topen,twrite,tclose,tskew;
+    uint64_t need = offset + (uint64_t)grid->_gsites*sizeof(fobj);
+
+    tcreate.Start();
+    if ( grid->IsBoss() ) {
+      // opening for update needs the file to exist; create one only if not
+      std::fstream probe(file,std::ios::binary|std::ios::out|std::ios::in);
+      if ( !probe.is_open() ) {
+        std::ofstream create(file,std::ios::binary|std::ios::out);
+        create.close();
+      }
+    }
+    tcreate.Stop();
+
+    tbar.Start();
+    grid->Barrier();
+    tbar.Stop();
+
+    std::ofstream fout;
+    fout.exceptions( std::fstream::failbit | std::fstream::badbit );
+    try {
+      topen.Start();
+      fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
+      topen.Stop();
+      twrite.Start();
+      for(uint64_t e=0;e<p.extentSites.size();e++){
+        fout.seekp(offset + p.extentGsite[e]*sizeof(fobj));
+        fout.write((char *)&aggregated[p.extentLocal[e]],p.extentSites[e]*sizeof(fobj));
+      }
+      twrite.Stop();
+      tclose.Start();
+      fout.close();   // flushes the stream buffer; does not force writeback
+      tclose.Stop();
+    } catch (const std::fstream::failure& exc) {
+      std::cout << GridLogError << "Error in aggregate write to " << file << std::endl;
+      std::cout << GridLogError << "Exception description: " << exc.what() << std::endl;
+      GridAbort();
+    }
+
+    ////////////////////////////////////////////////////////////////////////
+    // Timed apart from the truncate that follows it.  seek+write above is the
+    // slowest rank; this barrier is what the fastest rank then waits, so the
+    // pair separates the write cost from the spread across ranks. Folding it
+    // into the truncate makes a millisecond stat look like a second.
+    ////////////////////////////////////////////////////////////////////////
+    tskew.Start();
+    grid->Barrier();                 // every extent must be on its way first
+    tskew.Stop();
+
+    ttrunc.Start();
+    if ( grid->IsBoss() ) {
+      struct stat sb;
+      int ierr = ::stat(file.c_str(),&sb);
+      GRID_ASSERT(ierr==0);
+      if ( (uint64_t)sb.st_size != need ) {   // only when a longer record preceded us
+        ierr = ::truncate(file.c_str(),(off_t)need);
+        GRID_ASSERT(ierr==0);
+      }
+    }
+    grid->Barrier();
+    ttrunc.Stop();
+
+    std::vector<RealD> us = { (RealD)tcreate.useconds(), (RealD)tbar.useconds(),
+                              (RealD)topen.useconds(),   (RealD)twrite.useconds(),
+                              (RealD)tclose.useconds(),  (RealD)tskew.useconds(),
+                              (RealD)ttrunc.useconds() };
+    ReportStages(grid,"write",{"create","barrier","open","seek+write","close","skew","stat+truncate"},us);
+  }
+
+  template<class fobj>
+  static inline void AggregateRead(GridBase *grid,AggregationPlan &p,std::vector<fobj> &aggregated,
+                                   std::string file,uint64_t offset)
+  {
+    GridStopWatch topen,tread,tclose;
+    std::ifstream fin;
+    topen.Start();
+    fin.open(file,std::ios::binary|std::ios::in);
+    topen.Stop();
+    tread.Start();
+    for(uint64_t e=0;e<p.extentSites.size();e++){
+      fin.seekg(offset + p.extentGsite[e]*sizeof(fobj));
+      fin.read((char *)&aggregated[p.extentLocal[e]],p.extentSites[e]*sizeof(fobj));
+      GRID_ASSERT(fin.fail()==0);
+    }
+    tread.Stop();
+    tclose.Start();
+    fin.close();
+    tclose.Stop();
+
+    std::vector<RealD> us = { (RealD)topen.useconds(), (RealD)tread.useconds(), (RealD)tclose.useconds() };
+    ReportStages(grid,"read",{"open","seek+read","close"},us);
+  }
+#endif
+
   template<class word,class fobj>
   static inline void IOobject(word w,
 			      GridBase *grid,
@@ -302,6 +684,18 @@ class BinaryIO {
       lStart[d] = 0;
     }
 
+    //////////////////////////////////////////////////////////////////////////////
+    // Aggregate the lexicographic layout onto contiguous per rank extents
+    // ourselves rather than leaving it to MPI-IO collective buffering
+    //////////////////////////////////////////////////////////////////////////////
+    int aggregate = (control & BINARYIO_AGGREGATE)
+                 && (control & BINARYIO_LEXICOGRAPHIC)
+                 && !(control & BINARYIO_MASTER_APPEND)
+                 && (nrank > 1);
+#ifndef USE_MPI_IO
+    GRID_ASSERT(aggregate==0); // BINARYIO_AGGREGATE requires MPI
+#endif
+
 #ifdef USE_MPI_IO
     std::vector<int> distribs(ndim,MPI_DISTRIBUTE_BLOCK);
     std::vector<int> dargs   (ndim,MPI_DISTRIBUTE_DFLT_DARG);
@@ -329,6 +723,8 @@ class BinaryIO {
     ierr = MPI_Type_contiguous(numword,mpiword,&mpiObject);    GRID_ASSERT(ierr==0);
     ierr = MPI_Type_commit(&mpiObject);
 
+    // The subarray view is what aggregation exists to avoid; do not build it
+    if ( !aggregate ) {
     //////////////////////////////////////////////////////////////////////////////
     // File global array data type
     //////////////////////////////////////////////////////////////////////////////
@@ -340,6 +736,7 @@ class BinaryIO {
     //////////////////////////////////////////////////////////////////////////////
     ierr=MPI_Type_create_subarray(ndim,&lLattice[0],&lLattice[0],&lStart[0],MPI_ORDER_FORTRAN, mpiObject,&localArray);    GRID_ASSERT(ierr==0);
     ierr=MPI_Type_commit(&localArray);    GRID_ASSERT(ierr==0);
+    }
 #endif
 
     //////////////////////////////////////////////////////////////////////////////
@@ -358,7 +755,19 @@ class BinaryIO {
 
       timer.Start();
 
-      if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
+      if ( aggregate ) {
+#ifdef USE_MPI_IO
+        std::cout<< GridLogMessage<<"IOobject: aggregate read I/O "<< file<< std::endl;
+        AggregationPlan plan;
+        BuildAggregationPlan(grid,sizeof(fobj),plan);
+        ReportAggregationPlan(grid,plan,sizeof(fobj),"read");
+        std::vector<fobj> aggregated(lsites);
+        AggregateRead(grid,plan,aggregated,file,offset);
+        AggregateExchange(grid,plan,iodata,aggregated,0);
+#else
+        GRID_ASSERT(0);
+#endif
+      } else if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
 #ifdef USE_MPI_IO
 	std::cout<< GridLogMessage<<"IOobject: MPI read I/O "<< file<< std::endl;
 	ierr=MPI_File_open(grid->communicator,(char *) file.c_str(), MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);    GRID_ASSERT(ierr==0);
@@ -416,7 +825,25 @@ class BinaryIO {
       grid->Barrier();
 
       timer.Start();
-      if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
+      if ( aggregate ) {
+#ifdef USE_MPI_IO
+        std::cout << GridLogMessage <<"IOobject: aggregate write I/O " << file << std::endl;
+        AggregationPlan plan;
+        BuildAggregationPlan(grid,sizeof(fobj),plan);
+        ReportAggregationPlan(grid,plan,sizeof(fobj),"write");
+        std::vector<fobj> aggregated(lsites);
+        AggregateExchange(grid,plan,iodata,aggregated,1);
+        AggregateWrite(grid,plan,aggregated,file,offset);
+        ////////////////////////////////////////////////////////////////////////
+        // Not every rank ends at the end of the payload, so the position can
+        // not be recovered from a file handle. Callers (Lime record chaining)
+        // rely on this being the first byte past the record.
+        ////////////////////////////////////////////////////////////////////////
+        offset = offset + (uint64_t)grid->_gsites*sizeof(fobj);
+#else
+        GRID_ASSERT(0);
+#endif
+      } else if ( (control & BINARYIO_LEXICOGRAPHIC) && (nrank > 1) ) {
 #ifdef USE_MPI_IO
         std::cout << GridLogMessage <<"IOobject: MPI write I/O " << file << std::endl;
         ierr = MPI_File_open(grid->communicator, (char *)file.c_str(), MPI_MODE_RDWR | MPI_MODE_CREATE, MPI_INFO_NULL, &fh);
@@ -461,12 +888,26 @@ class BinaryIO {
         
 	std::ofstream fout; 
 	fout.exceptions ( std::fstream::failbit | std::fstream::badbit );
+
+        ////////////////////////////////////////////////////////////////////
+        // Grid's model is that the boss rank performs the metadata
+        // operations and every other rank only seeks and writes into a file
+        // that already exists. Opening with ios::out on all ranks broke that:
+        // it is O_TRUNC, so a rank opening late truncated the file back to
+        // zero after an earlier rank had written its segment, leaving a hole
+        // in its place. The barriers around this block are outside it and do
+        // not order the opens against the writes. Let the boss create and
+        // empty the file, then everyone opens for update only. Same resulting
+        // length, one metadata operation instead of one per rank, no race.
+        ////////////////////////////////////////////////////////////////////
+        if ( !offset && grid->IsBoss() ) { // offset zero: this record starts the file
+          std::ofstream create(file,std::ios::binary|std::ios::out);
+          create.close();
+        }
+        grid->Barrier();
+
 	try {
-	  if (offset) { // Must already exist and contain data
-	    fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
-	  } else {     // Allow create
-	    fout.open(file,std::ios::binary|std::ios::out);
-	  }
+	  fout.open(file,std::ios::binary|std::ios::out|std::ios::in);
 	} catch (const std::fstream::failure& exc) {
 	  std::cout << GridLogError << "Error in opening the file " << file << " for output" <<std::endl;
 	  std::cout << GridLogError << "Exception description: " << exc.what() << std::endl;
@@ -477,7 +918,7 @@ class BinaryIO {
 	  exit(1);
 #endif
 	}
-	
+
 	if ( control & BINARYIO_MASTER_APPEND )  {
 	  try {
 	    fout.seekp(0,fout.end);
@@ -548,7 +989,7 @@ class BinaryIO {
 				       uint32_t &nersc_csum,
 				       uint32_t &scidac_csuma,
 				       uint32_t &scidac_csumb,
-				       int control=BINARYIO_LEXICOGRAPHIC
+				       int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE
 				       )
   {
     typedef typename vobj::scalar_object sobj;
@@ -587,7 +1028,7 @@ class BinaryIO {
 					  uint32_t &nersc_csum,
 					  uint32_t &scidac_csuma,
 					  uint32_t &scidac_csumb,
-					  int control=BINARYIO_LEXICOGRAPHIC)
+					  int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     typedef typename vobj::scalar_object sobj;
     typedef typename vobj::Realified::scalar_type word;    word w=0;
@@ -674,7 +1115,7 @@ class BinaryIO {
     std::cout << GridLogMessage << "RNG read I/O on file " << file << std::endl;
 
     std::vector<RNGstate> iodata(lsites);
-    IOobject(w,grid,iodata,file,offset,format,BINARYIO_READ|BINARYIO_LEXICOGRAPHIC,
+    IOobject(w,grid,iodata,file,offset,format,BINARYIO_READ|BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE,
 	     nersc_csum,scidac_csuma,scidac_csumb);
 
     timer.Start();
@@ -753,7 +1194,7 @@ class BinaryIO {
     });
     timer.Stop();
 
-    IOobject(w,grid,iodata,file,offset,format,BINARYIO_WRITE|BINARYIO_LEXICOGRAPHIC,
+    IOobject(w,grid,iodata,file,offset,format,BINARYIO_WRITE|BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE,
 	     nersc_csum,scidac_csuma,scidac_csumb);
     iodata.resize(1);
     {

Grid/parallelIO/IldgIO.h

@@ -212,7 +212,7 @@ class GridLimeReader : public BinaryIO {
   // Read a generic lattice field and verify checksum
   ////////////////////////////////////////////
   template<class vobj>
-  void readLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC)
+  void readLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     typedef typename vobj::scalar_object sobj;
     scidacChecksum scidacChecksum_;
@@ -260,8 +260,7 @@ class GridLimeReader : public BinaryIO {
               << " / field= " << n2ck << " / rdiff= " << GRID_FIELD_NORM_CALC(FieldNormMetaData_,n2ck) << std::endl;
 	  GRID_FIELD_NORM_CHECK(FieldNormMetaData_,n2ck);
 	}
-//	assert(scidacChecksumVerify(scidacChecksum_,scidac_csuma,scidac_csumb)==1);
-	scidacChecksumVerify(scidacChecksum_,scidac_csuma,scidac_csumb);
+	assert(scidacChecksumVerify(scidacChecksum_,scidac_csuma,scidac_csumb)==1);
 
 	// find out if next field is a GridFieldNorm
 	return;
@@ -415,7 +414,7 @@ class GridLimeWriter : public BinaryIO
   // in communicator used by the field.Grid()
   ////////////////////////////////////////////////////
   template<class vobj>
-  void writeLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC)
+  void writeLimeLatticeBinaryObject(Lattice<vobj> &field,std::string record_name,int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     ////////////////////////////////////////////////////////////////////
     // NB: FILE and iostream are jointly writing disjoint sequences in the
@@ -520,7 +519,7 @@ class ScidacWriter : public GridLimeWriter {
   template <class vobj, class userRecord>
   void writeScidacFieldRecord(Lattice<vobj> &field,userRecord _userRecord,
                               const unsigned int recordScientificPrec = 0,
-			      int control=BINARYIO_LEXICOGRAPHIC)
+			      int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE)
   {
     GridBase * grid = field.Grid();
 
@@ -562,7 +561,7 @@ class ScidacReader : public GridLimeReader {
   ////////////////////////////////////////////////
   template <class vobj, class userRecord>
   void readScidacFieldRecord(Lattice<vobj> &field,userRecord &_userRecord,
-			     int control=BINARYIO_LEXICOGRAPHIC) 
+			     int control=BINARYIO_LEXICOGRAPHIC|BINARYIO_AGGREGATE) 
   {
     typedef typename vobj::scalar_object sobj;
     GridBase * grid = field.Grid();

Grid/parallelIO/NerscIO.h

@@ -122,7 +122,7 @@ public:
     field.checksum = std::stoul(header["CHECKSUM"],0,16);
     field.ensemble_id      = header["ENSEMBLE_ID"];
     field.ensemble_label   = header["ENSEMBLE_LABEL"];
-//    field.sequence_number  = std::stol(header["SEQUENCE_NUMBER"]);
+    field.sequence_number  = std::stol(header["SEQUENCE_NUMBER"]);
     field.creator          = header["CREATOR"];
     field.creator_hardware = header["CREATOR_HARDWARE"];
     field.creation_date    = header["CREATION_DATE"];

Grid/perfmon/PerfCount.h

@@ -93,8 +93,7 @@ inline uint64_t cyclecount(void){
 }
 #elif defined __x86_64__
 inline uint64_t cyclecount(void){ 
-  uint64_t ret = __rdtsc();
-  return (uint64_t)ret;
+  return (uint64_t)0;
 }
 #else
 

Grid/simd/Simd.h

@@ -113,6 +113,14 @@ accelerator_inline RealD    adj(const RealD  & r){ return r; }
 accelerator_inline ComplexD adj(const ComplexD& r){ return(conjugate(r)); }
 accelerator_inline ComplexF adj(const ComplexF& r ){ return(conjugate(r)); }
 
+#if defined(GRID_CUDA) || defined(GRID_HIP)
+//Provide for convenience
+inline std::complex<double> conjugate(const std::complex<double>& r){ return(conj(r)); }
+inline std::complex<float>  conjugate(const std::complex<float>& r) { return(conj(r)); }
+inline std::complex<double> adj(const std::complex<double>& r)      { return(conj(r)); }
+inline std::complex<float>  adj(const std::complex<float>& r)       { return(conj(r)); }
+#endif
+
 accelerator_inline RealF real(const RealF  & r){ return r; }
 accelerator_inline RealD real(const RealD  & r){ return r; }
 accelerator_inline RealF real(const ComplexF  & r){ return r.real(); }

Grid/threads/Accelerator.cc

@@ -3,7 +3,7 @@
 NAMESPACE_BEGIN(Grid);
 int      world_rank; // Use to control world rank for print guarding
 int      acceleratorAbortOnGpuError=1;
-uint32_t accelerator_threads=2;
+uint32_t accelerator_threads=8;
 uint32_t acceleratorThreads(void)       {return accelerator_threads;};
 void     acceleratorThreads(uint32_t t) {accelerator_threads = t;};
 

Grid/threads/Accelerator.h

@@ -96,7 +96,9 @@ void     acceleratorInit(void);
 
 #ifdef GRID_CUDA
 
+NAMESPACE_END(Grid);
 #include <cuda.h>
+NAMESPACE_BEGIN(Grid);
 
 #ifdef __CUDA_ARCH__
 #define GRID_SIMT
@@ -432,22 +434,20 @@ accelerator_inline int acceleratorSIMTlane(int Nsimd) {
 
 #define accelerator_for2dNB( iter1, num1, iter2, num2, nsimd, ... )	\
   {									\
-    typedef uint64_t Iterator;						\
-    auto lambda = [=] accelerator					\
-      (Iterator iter1,Iterator iter2,Iterator lane ) mutable {		\
-      { __VA_ARGS__;}							\
-    };									\
-    int nt=acceleratorThreads();					\
-    dim3 hip_threads(nsimd, nt, 1);					 \
-    dim3 hip_blocks ((num1+nt-1)/nt,num2,1); \
-    if(hip_threads.x * hip_threads.y * hip_threads.z <= 64){ \
-      hipLaunchKernelGGL(LambdaApply64,hip_blocks,hip_threads,		\
-   	                 0,computeStream,						\
-			 num1,num2,nsimd, lambda);			\
-    } else { \
-      hipLaunchKernelGGL(LambdaApply,hip_blocks,hip_threads,		\
-			 0,computeStream,				\
-			 num1,num2,nsimd, lambda);			\
+    if (num1*num2) { \
+      typedef uint64_t Iterator;						\
+      auto lambda = [=] accelerator					\
+        (Iterator iter1,Iterator iter2,Iterator lane ) mutable {		\
+        { __VA_ARGS__;}							\
+      };									\
+      int nt=acceleratorThreads();					\
+      dim3 hip_threads(nsimd, nt, 1);					 \
+      dim3 hip_blocks ((num1+nt-1)/nt,num2,1); \
+      if(hip_threads.x * hip_threads.y * hip_threads.z <= 64){ \
+        LambdaApply64<<<hip_blocks,hip_threads,0,computeStream>>>(num1,num2,nsimd,lambda);			\
+      } else { \
+        LambdaApply<<<hip_blocks,hip_threads,0,computeStream>>>(num1,num2,nsimd,lambda);			\
+      } \
     } \
   }
 

Grid/threads/ThreadReduction.h

@@ -28,11 +28,6 @@ Author: paboyle <paboyle@ph.ed.ac.uk>
 /*  END LEGAL */
 #pragma once 
 
-#ifndef MIN
-#define MIN(x,y) ((x)>(y)?(y):(x))
-#endif
-
-
 // Introduce a class to gain deterministic bit reproducible reduction.
 // make static; perhaps just a namespace is required.
 NAMESPACE_BEGIN(Grid);

benchmarks/Benchmark_usqcd.cc

@@ -877,7 +877,7 @@ int main (int argc, char ** argv)
   int do_su4=0;
   int do_memory=1;
   int do_comms =1;
-  int do_blas  =0;
+  int do_blas  =1;
   int do_dslash=1;
 
   int sel=4;

configure.ac

@@ -174,7 +174,7 @@ esac
 ############### fermions
 AC_ARG_ENABLE([fermion-instantiations],
      [AS_HELP_STRING([--enable-fermion-instantiations=yes|no],[enable fermion instantiations])],
-     [ac_FERMION_INSTANTIATIONS=${enable_fermion_instantiations}], [ac_FERMION_INSTANTIATIONS=yes])
+     [ac_FERMION_REPS=${enable_fermion_instantiations}], [ac_FERMION_INSTANTIATIONS=yes])
 
 AM_CONDITIONAL(BUILD_FERMION_INSTANTIATIONS, [ test "${ac_FERMION_INSTANTIATIONS}X" == "yesX" ])
 

examples/Example_arnoldi.cc

@@ -1,430 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./tests/Test_padded_cell.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// copied here from Test_general_coarse_pvdagm.cc
-
-#include <cstdlib>
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-
-// Hermitize a DWF operator by squaring it
-template<class Matrix,class Field>
-class SquaredLinearOperator : public LinearOperatorBase<Field> {
-
-  public:
-  Matrix &_Mat;
-
-  public:
-    SquaredLinearOperator(Matrix &Mat): _Mat(Mat) {};
-
-    void OpDiag (const Field &in, Field &out) {    assert(0);  }
-    void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-    void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-    void Op     (const Field &in, Field &out){
-      // std::cout << "Op is overloaded as HermOp" << std::endl;
-      HermOp(in, out);
-    }
-    void AdjOp     (const Field &in, Field &out){
-      HermOp(in, out);
-    }
-    void _Op     (const Field &in, Field &out){
-      // std::cout << "Op: M "<<std::endl;
-      _Mat.M(in, out);
-    }
-    void _AdjOp     (const Field &in, Field &out){
-      // std::cout << "AdjOp: Mdag "<<std::endl;
-      _Mat.Mdag(in, out);
-    }
-    void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-    void HermOp(const Field &in, Field &out){
-      // std::cout << "HermOp: Mdag M Mdag M"<<std::endl;
-      Field tmp(in.Grid());
-      _Op(in,tmp);
-      _AdjOp(tmp,out);
-    }
-};
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    //    _Mat.M(in,tmp);
-    //    _PV.Mdag(tmp,out);
-    //    _PV.M(out,tmp);
-    //    _Mat.Mdag(tmp,out);
-    Op(in,tmp);
-    AdjOp(tmp,out);
-    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix, class Field>
-class ShiftedComplexPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  ComplexD shift;
-public:
-ShiftedComplexPVdagMLinearOperator(ComplexD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-  
-  void resetShift(ComplexD newShift) {
-    shift = newShift;
-  }
-};
-
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_)
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out) 
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    //    auto CoarseGrid = _CoarseOperator.Grid();
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    std::cout<<GridLogMessage << "Calling PreSmoother " <<std::endl;
-
-    //    std::cout<<GridLogMessage << "Calling PreSmoother input residual "<<norm2(in) <<std::endl;
-    double t;
-    // Fine Smoother
-    //    out = in;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-
-    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Update the residual
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);   
-    //    std::cout<<GridLogMessage <<"Residual-1 now " <<norm2(vec1)<<std::endl;
-
-    // Fine to Coarse 
-    t=-usecond();
-    _Aggregates.ProjectToSubspace  (Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Coarse correction
-    t=-usecond();
-    Csol = Zero();
-    _CoarseSolve(Csrc,Csol);
-    //Csol=Zero();
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Coarse to Fine
-    t=-usecond();  
-    //    _CoarseOperator.PromoteFromSubspace(_Aggregates,Csol,vec1);
-    _Aggregates.PromoteFromSubspace(Csol,vec1); 
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Residual
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);  
-    //    std::cout<<GridLogMessage <<"Residual-2 now " <<norm2(vec1)<<std::endl;
-
-    // Fine Smoother
-    t=-usecond();
-    //    vec2=vec1;
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add( out,out,vec2);
-    std::cout<<GridLogMessage << "Done " <<std::endl;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls=16;
-
-//   GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  std::vector<int> lat_size {16, 16, 16, 32};
-  std::cout << "Lattice size: " << lat_size << std::endl;
-  GridCartesian * UGrid = SpaceTimeGrid::makeFourDimGrid(lat_size, 
-								          GridDefaultSimd(Nd,vComplex::Nsimd()),
-								          GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  // Construct a coarsened grid
-  // poare TODO: replace this with the following line?
-  Coordinate clatt = lat_size;
-//   Coordinate clatt = GridDefaultLatt();              // [PO] initial line before I edited it
-  for(int d=0;d<clatt.size();d++){
-    clatt[d] = clatt[d]/2;
-    //    clatt[d] = clatt[d]/4;
-  }
-  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());;
-  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  std::vector<int> cseeds({5,6,7,8});
-  GridParallelRNG          RNG5(FGrid);   RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG          RNG4(UGrid);   RNG4.SeedFixedIntegers(seeds4);
-  GridParallelRNG          CRNG(Coarse5d);CRNG.SeedFixedIntegers(cseeds);
-
-  LatticeFermion    src(FGrid); random(RNG5,src);
-  LatticeFermion result(FGrid); result=Zero();
-  LatticeFermion    ref(FGrid); ref=Zero();
-  LatticeFermion    tmp(FGrid);
-  LatticeFermion    err(FGrid);
-  LatticeGaugeField Umu(UGrid);
-
-  FieldMetaData header;
-//   std::string file("ckpoint_lat.4000");
-  std::string file("/Users/patrickoare/libraries/PETSc-Grid/ckpoint_lat.4000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  RealD mass=0.01;
-  RealD M5=1.8;
-
-  DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
-  DomainWallFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5);
-
-  // const int nbasis = 20;            // size of approximate basis for low-mode space
-  const int nbasis = 3;            // size of approximate basis for low-mode space
-  const int cb = 0 ;
-  LatticeFermion prom(FGrid);
-
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector CoarseVector;
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-
-  typedef PVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef ShiftedComplexPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedComplexPVdagM_t;
-  PVdagM_t PVdagM(Ddwf, Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv);
-  SquaredLinearOperator<DomainWallFermionD, LatticeFermionD> Dsq (Ddwf);
-  NonHermitianLinearOperator<DomainWallFermionD, LatticeFermionD> DLinOp (Ddwf);
-
-  // PowerMethod<LatticeFermion> PM; PM(PVdagM, src);
-  int Nm = 10;
-  int Nk = 6;
-  // int Nm = 6;         // Nm = 6 case is acting really strangely... with Nm = 6 and Nm = 3 it zeros out the Hessenberg and also makes it imaginary?
-  // int Nk = 2;
-  // int Nk = Nm+1;     // if just running once
-  // int maxIter = 5;
-  // int maxIter = 1;
-  int maxIter = 3;
-  // int maxIter = 100;
-  int Nstop = 6;
-
-  Coordinate origin ({0,0,0,0});
-  auto tmpSrc = peekSite(src, origin);
-  std::cout << "[DEBUG] Source at origin = " <<  tmpSrc << std::endl;
-  LatticeFermion src2 = src;
-
-  // Run Lanczos and Arnoldi on a Hermitian matrix
-  // Arnoldi Arn (Dsq, FGrid, 1e-8, false);
-  // Arn(src, 1, Nm, -1);
-  Arnoldi Arn (Dsq, FGrid, 1e-8, EvalNormLarge);      // for comparison to Lanczos
-  // Arn(src, maxIter, Nm, Nk, Nstop);
-
-  // auto tmpSrcDup = peekSite(src, origin);
-  // std::cout << "[DEBUG] Source at origin = " <<  tmpSrcDup << std::endl;
-  // auto tmpSrc2Dup = peekSite(src2, origin);
-  // std::cout << "[DEBUG] Source2 at origin = " <<  tmpSrc2Dup << std::endl;
-
-  Arn(src, maxIter, Nm, Nk, Nstop);
-  std::cout << "Hessenberg mat for symmetric N = " << Nm << std::endl;
-  std::cout << Arn.getHessenbergMat() << std::endl;
-
-  // ImplicitlyRestartedLanczosHermOpTester<LatticeFermionD> SimpleTester (Dsq);
-  // ImplicitlyRestartedLanczos<LatticeFermionD> Lanc (Dsq, Dsq, SimpleTester, Nm, Nm, Nm, 1e-8, Nm);
-  int Nconv;
-  PlainHermOp DsqHermOp (Dsq);
-  // std::vector<RealD> levals (Nm+1); std::vector<LatticeFermionD> levecs (Nm+1, src);
-  // ImplicitlyRestartedLanczos<LatticeFermionD> Lanc (DsqHermOp, DsqHermOp, Nm, Nm, Nm + 1, 1e-8, Nm);
-  std::vector<RealD> levals (Nm+1); std::vector<LatticeFermionD> levecs (Nm, src);
-  ImplicitlyRestartedLanczos<LatticeFermionD> Lanc (DsqHermOp, DsqHermOp, Nstop, Nk, Nm, 1e-8, maxIter);
-  std::cout << GridLogMessage << "Calculating with Lanczos" << std::endl;
-
-  // auto tmpSrc1 = peekSite(src, origin);
-  // std::cout << "[DEBUG] Source at origin = " <<  tmpSrc1 << std::endl;
-  // auto tmpSrc2 = peekSite(src2, origin);
-  // std::cout << "[DEBUG] Source2 at origin = " <<  tmpSrc2 << std::endl;
-  // std::cout << "[DEBUG] Source norm2: " << norm2(src) << std::endl;
-
-  std::cout << "running Lanczos now" << std::endl;
-  Lanc.calc(levals, levecs, src2, Nconv);
-
-  std::cout<<GridLogMessage << "*******************************************" << std::endl;
-  std::cout<<GridLogMessage << "***************** RESULTS *****************" << std::endl;
-  std::cout<<GridLogMessage << "*******************************************" << std::endl;
-
-  std::cout << GridLogMessage << "Arnoldi eigenvalues: " << std::endl << Arn.getEvals() << std::endl;
-  std::cout << GridLogMessage << "Lanczos eigenvalues: " << std::endl << levals << std::endl;
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_gamma5_block_lanczos.cc

@@ -1,188 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Source file: ./examples/Example_gamma5_block_lanczos.cc
-
-    Copyright (C) 2026
-
-    Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-    γ5-Block Lanczos example for the Wilson Dirac operator.
-
-    Reads a gauge configuration from "config" (NERSC format) and Lanczos
-    parameters from "LanParams.xml".  Runs Gamma5BlockLanczos to
-    compute eigenvalues of D_W directly (not H_W = γ5 D_W).
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution
-    directory
-*************************************************************************************/
-/*  END LEGAL */
-
-#include <cstdlib>
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-using namespace std;
-using namespace Grid;
-
-namespace Grid {
-
-struct LanczosParameters : Serializable {
-  GRID_SERIALIZABLE_CLASS_MEMBERS(LanczosParameters,
-    RealD,   mass,
-    Integer, Nstop,
-    Integer, Nk,
-    Integer, Np,
-    Integer, maxIter,
-    Integer, reorthog,
-    Integer, verify,
-    Integer, ReadEvec,
-    RealD,   resid)
-
-  LanczosParameters()
-    : mass(-0.5), Nstop(10), Nk(20), maxIter(100),
-      reorthog(1), verify(0), ReadEvec(0), resid(1e-8)
-  {}
-
-  template<class ReaderClass>
-  LanczosParameters(Reader<ReaderClass>& r) { initialize(r); }
-
-  template<class ReaderClass>
-  void initialize(Reader<ReaderClass>& r) {
-    read(r, "LanczosParameters", *this);
-  }
-};
-
-} // namespace Grid
-
-template<class T>
-void writeField(T& in, std::string const& fname) {
-#ifdef HAVE_LIME
-  std::cout << GridLogMessage << "Writing to: " << fname << std::endl;
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter WR(in.Grid()->IsBoss());
-  WR.open(fname);
-  WR.writeScidacFieldRecord(in, record, 0);
-  WR.close();
-#endif
-}
-
-typedef WilsonFermionD WilsonOp;
-typedef typename WilsonFermionD::FermionField FermionField;
-
-int main(int argc, char** argv)
-{
-  Grid_init(&argc, &argv);
-
-  GridCartesian* UGrid = SpaceTimeGrid::makeFourDimGrid(
-      GridDefaultLatt(),
-      GridDefaultSimd(Nd, vComplex::Nsimd()),
-      GridDefaultMpi());
-  GridRedBlackCartesian* UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-
-  std::vector<int> seeds4({1, 2, 3, 4});
-  GridParallelRNG RNG4(UGrid);
-  RNG4.SeedFixedIntegers(seeds4);
-
-  // Read gauge configuration
-  LatticeGaugeField Umu(UGrid);
-  FieldMetaData header;
-  std::string configFile("config");
-  NerscIO::readConfiguration(Umu, header, configFile);
-  std::cout << GridLogMessage << "Loaded gauge configuration: " << configFile << std::endl;
-
-  // Read Lanczos parameters
-  LanczosParameters Params;
-  {
-    XmlReader rd("LanParams.xml");
-    read(rd, "LanczosParameters", Params);
-  }
-  std::cout << GridLogMessage << Params << std::endl;
-
-  // Build Wilson Dirac operator and wrap in a non-Hermitian linear operator
-  std::vector<Complex> boundary = {1, 1, 1, -1};
-  WilsonOp::ImplParams WilsonParams(boundary);
-  WilsonOp Dwilson(Umu, *UGrid, *UrbGrid, Params.mass, WilsonParams);
-  NonHermitianLinearOperator<WilsonOp, FermionField> DLinOp(Dwilson);
-
-  // γ5 functor: for 4D Wilson fermions γ5 is Gamma(Gamma5)
-  Gamma G5(Gamma::Algebra::Gamma5);
-  auto gamma5 = [&G5](const FermionField& in, FermionField& out) {
-    out = G5 * in;
-  };
-
-  // Starting vectors: two independent random vectors
-  FermionField src(UGrid), src2(UGrid);
-  random(RNG4, src);
-  random(RNG4, src2);
-  std::cout << GridLogMessage << "Using two random starting vectors" << std::endl;
-
-  std::cout << GridLogMessage << std::endl;
-  std::cout << GridLogMessage << "*******************************************" << std::endl;
-  std::cout << GridLogMessage << " Running γ5-Block Lanczos" << std::endl;
-  std::cout << GridLogMessage << "   mass     = " << Params.mass     << std::endl;
-  std::cout << GridLogMessage << "   Nk       = " << Params.Nk       << std::endl;
-  std::cout << GridLogMessage << "   maxIter  = " << Params.maxIter  << std::endl;
-  std::cout << GridLogMessage << "   Nstop    = " << Params.Nstop    << std::endl;
-  std::cout << GridLogMessage << "   resid    = " << Params.resid    << std::endl;
-  std::cout << GridLogMessage << "   reorthog = " << Params.reorthog << std::endl;
-  std::cout << GridLogMessage << "   verify   = " << Params.verify   << std::endl;
-  std::cout << GridLogMessage << "*******************************************" << std::endl;
-  std::cout << GridLogMessage << std::endl;
-
-  Gamma5BlockLanczos<FermionField> G5BL(DLinOp, UGrid, gamma5, Params.resid);
-  G5BL.doEvalCheck = (Params.verify != 0);
-  G5BL.doVerify = (Params.verify != 0);
-//  G5BL(src, src2, Params.maxIter, Params.Nstop, Params.reorthog != 0);
-  G5BL.restart(src, src2, Params.maxIter, Params.Nk+Params.Np, Params.Nk, Params.Nstop, Params.reorthog != 0, EvalNormSmall);
-//  G5BL.implicitRestart(src, src2, Params.maxIter, Params.Nk+Params.Np, Params.Nk, Params.Nstop, Params.reorthog != 0, EvalNormSmall);
-  if (Params.verify ) G5BL.verify("after restart");
-
-  // Summary of results
-  Eigen::VectorXcd        evals     = G5BL.getEvals();
-  std::vector<RealD>      residuals = G5BL.getResiduals();
-  std::vector<FermionField> evecs   = G5BL.getEvecs();
-
-  int Nout = (int)evals.size();
-  std::cout << GridLogMessage << std::endl;
-  std::cout << GridLogMessage << "*******************************************" << std::endl;
-  std::cout << GridLogMessage << " γ5-Block Lanczos: " << Nout << " Ritz pairs" << std::endl;
-  std::cout << GridLogMessage << "*******************************************" << std::endl;
-  for (int i = 0; i < Nout; i++) {
-    std::cout << GridLogMessage
-              << "  [" << std::setw(3) << i << "]"
-              << "  lambda = " << evals(i)
-              << "  |res| = " << residuals[i] << std::endl;
-  }
-
-  // Write the first Nstop eigenvectors (in SCIDAC format when LIME is available)
-  int Nwrite = std::min((int)Params.Nstop, Nout);
-  for (int i = 0; i < Nwrite; i++) {
-    std::string fname = "./g5bl_evec_m" + std::to_string(Params.mass)
-                        + "_" + std::to_string(i);
-    writeField(evecs[i], fname);
-  }
-
-  std::cout << GridLogMessage << std::endl;
-  std::cout << GridLogMessage << "Done" << std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_krylov_coeffs.cc

@@ -1,405 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./tests/Test_padded_cell.cc
-
-    Copyright (C) 2023
-
-Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// Tests code written to read off the Krylov coefficients
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-#include <Grid/algorithms/iterative/ConjugateGradient.h>
-
-using namespace std;
-using namespace Grid;
-
-// Hermitize a DWF operator by squaring it
-template<class Matrix,class Field>
-class SquaredLinearOperator : public LinearOperatorBase<Field> {
-
-  public:
-  Matrix &_Mat;
-
-  public:
-    SquaredLinearOperator(Matrix &Mat): _Mat(Mat) {};
-
-    void OpDiag (const Field &in, Field &out) {    assert(0);  }
-    void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-    void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-    void Op     (const Field &in, Field &out){
-      // std::cout << "Op is overloaded as HermOp" << std::endl;
-      HermOp(in, out);
-    }
-    void AdjOp     (const Field &in, Field &out){
-      HermOp(in, out);
-    }
-    void _Op     (const Field &in, Field &out){
-      // std::cout << "Op: M "<<std::endl;
-      _Mat.M(in, out);
-    }
-    void _AdjOp     (const Field &in, Field &out){
-      // std::cout << "AdjOp: Mdag "<<std::endl;
-      _Mat.Mdag(in, out);
-    }
-    void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-    void HermOp(const Field &in, Field &out){
-      // std::cout << "HermOp: Mdag M Mdag M"<<std::endl;
-      Field tmp(in.Grid());
-      _Op(in,tmp);
-      _AdjOp(tmp,out);
-    }
-};
-
-/**
- * Computes the coefficients in the Krylov expansion for 1/D ~ \sum_{i=0}^N c_i D^i. 
- * 
- * Parameters
- * ----------
- * std::vector<double> &coeffs
- *    Polynomial coeffients to return, with indexing order (c_0, c_1, c_2, ..., c_n). 
- * LinearOperatorBase<FineField> &DiracOp
- *    Dirac operator D. 
- * FineField src
- *    Source field b. 
- * FineField psiStar
- *    Output approximation for D^{-1} b coming from a Krylov method. 
- * int N
- *    Dimension of the polynomial approximation (Krylov space K_{N-1} = {b, Db, D^2 b, ..., D^{N-1} b}).
- */
-void poly_coeffs(std::vector<ComplexD> &coeffs, LinearOperatorBase<LatticeFermion> &DiracOp, LatticeFermion src,
-                  LatticeFermion psiStar, GridCartesian* FGrid, int N, bool use_herm = false)
-{
-  // stdBasis = {b, Db, D^2 b, ..., D^N b}, kryBasis = {k0, k1, ..., kN}
-  std::vector<LatticeFermion> kryBasis;
-  Eigen::VectorXcd psiStarCoeffs (N);
-
-  // Normalize by 1 / ||src||; does not change the polynomial coefficients
-  double srcNorm   = 1 / std::sqrt(norm2(src));
-  kryBasis.push_back(srcNorm * src);                // normalized source
-  psiStar          = srcNorm * psiStar;
-  psiStarCoeffs(0) = innerProduct(kryBasis[0], psiStar);
-
-  // orthonormalize canonical Krylov basis {b, Db, D^2 b, ..., D^{N-1} b} <--> {k_i} and compute components <k_i | psi*>
-  LatticeFermion tmp (FGrid);
-  for (int i = 0; i < N - 1; i++) {               // construct ONB for {b, Db, ..., D^{i+1} b}
-    if (use_herm) {
-      DiracOp.HermOp(kryBasis.back(), tmp);         // tmp \in span{(D^\dag D)^{i+1} b} \oplus span{(D^\dag D)^i b, ..., D^\dag D b, b}
-    } else {
-      DiracOp.Op(kryBasis.back(), tmp);             // tmp \in span{D^{i+1} b} \oplus span{D^i b, ..., Db, b}
-    }
-
-    for (int j = 0; j < i+1; j++) {               // orthogonalize tmp with previous basis vectors
-      ComplexD coeff = innerProduct(kryBasis[j], tmp);      // <k_j | tmp>
-      tmp -= coeff * kryBasis[j];                           // subtract off |k_j><k_j | tmp>; now tmp is perp to |k_j>
-    }
-    double tmpNorm = 1 / std::sqrt(norm2(tmp));
-    kryBasis.push_back(
-      tmpNorm * tmp
-    );                                                      // normalize |k_i> and add to kryBasis
-    psiStarCoeffs(i+1) = innerProduct(kryBasis[i+1], psiStar);  // compute < k_i | psi* >
-  }
-
-  // To verify the basis is ONB
-  // for (int i = 0; i < N; i++) {
-  //   for (int j = 0; j < N; j++) {
-  //     std::cout << "<ki|kj> for (i,j) = (" << i << ", " << j << ") = "  << innerProduct(kryBasis[i], kryBasis[j]) << std::endl;
-  //   }
-  // }
-
-  // Compute change of basis matrix
-  LatticeFermion tmp2 (FGrid);
-  Eigen::MatrixXcd M = Eigen::MatrixXcd::Zero(N, N);
-  tmp = kryBasis[0];       // current Krylov vector; starts with tmp = src (normalized)
-  for (int i = 0; i < N; i++) {
-    for (int j = 0; j < i + 1; j++) {    // fill column with components of kryVec. Only need j <= i to get orthonormal components
-      M(j, i) = innerProduct(kryBasis[j], tmp);
-    }    
-    if (use_herm) {     // tmp --> D^\dag D(tmp)
-      DiracOp.HermOp(tmp, tmp2);
-      tmp = tmp2;
-    } else {      // tmp --> D(tmp). Note that DiracOp.Op(tmp, tmp) will cause a bug
-      DiracOp.Op(tmp, tmp2);
-      tmp = tmp2;
-    }
-  }
-  
-  // Compute M^{-1} @ psiStarCoeffs and copy to coeffs
-  Eigen::VectorXcd res (N);
-  res = M.inverse() * psiStarCoeffs;
-  for (int i = 0; i < N; i++) {
-    coeffs[i] = res(i);
-  }
-
-}
-
-// out file for poly coefficients (should it be complex?)
-// class PolynomialFile: Serializable {
-// public:
-//   GRID_SERIALIZABLE_CLASS_MEMBERS(OutputFile, std::vector< Real >, data);
-// };
-
-std::complex<double> poly_approx(std::complex<double> x, std::vector<std::complex<double>> coeffs) {
-  std::complex<double> px;
-  for (int i = 0; i < coeffs.size(); i++) {
-    px += coeffs[i] * std::pow(x, i);
-  }
-  return px;
-}
-
-/**
- * Returns the approximation psi = \sum_i c_i D^i b resulting from a Krylov solver.
- * 
- * Parameters
- * ----------
- * LatticeFermion &psi
- *    Approximation field, returned psi = \sum_i c_i D^i b.
- * LatticeFermion src
- *    Source b used to generate the Krylov space K_n(D, b).
- * LinearOperatorBase<LatticeFermion> &Linop
- *    Dirac operator used to generate the Krylov space K_n(D, b).
- * std::vector<std::complex<double>> coeffs
- *    Polynomial coefficients returned from the solver. 
- */
-void krylovApprox(LatticeFermion &psi, LatticeFermion src, LinearOperatorBase<LatticeFermion> &Linop, std::vector<ComplexD> coeffs) {
-  psi = Zero();
-  LatticeFermion tmp (psi.Grid());
-  tmp = src;
-  LatticeFermion tmp2 (psi.Grid());
-  for (int i = 0; i < coeffs.size(); i++) {
-      psi = psi + coeffs[i] * tmp;
-      Linop.Op(tmp, tmp2);              // tmp = D*tmp
-      tmp = tmp2;
-  }
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc, &argv);
-  const int Ls = 8;
-  std::vector<int> lat_size {16, 16, 16, 32};
-  std::cout << "Lattice size: " << lat_size << std::endl;
-  GridCartesian * UGrid = SpaceTimeGrid::makeFourDimGrid(lat_size, 
-								          GridDefaultSimd(Nd,vComplex::Nsimd()),
-								          GridDefaultMpi());
-
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  //////////////////////////////////////////////////////////////////////
-  // You can manage seeds however you like.
-  // Recommend SeedUniqueString.
-  //////////////////////////////////////////////////////////////////////
-  // std::vector<int> seeds4({1, 2, 3, 4}); 
-  // GridParallelRNG RNG4(UGrid);
-  // RNG4.SeedFixedIntegers(seeds4);
-
-  // std::vector<int> seeds5({1, 2, 3, 4, 5}); 
-  // GridParallelRNG RNG5(FGrid);
-  // RNG5.SeedFixedIntegers(seeds5);
-
-  // std::string outStrStem = "/Users/patrickoare/Dropbox (MIT)/research/multigrid/grid_out/";
-
-  LatticeGaugeField Umu(UGrid);
-  FieldMetaData header;
-  std::string file("/Users/patrickoare/libraries/PETSc-Grid/ckpoint_lat.4000");
-  NerscIO::readConfiguration(Umu, header, file);
-  
-  RealD mass=0.01;
-  RealD M5=1.8;
-  // RealD M5=1.0;
-  RealD b=1.5;// Scale factor b+c=2, b-c=1
-  RealD c=0.5;
-
-  // load in Dirac operators that we'll use; square it to Hermitize
-  // Dsq just needs to be a Hermitian operator so we can use CG on it
-  DomainWallFermionD Ddwf(Umu, *FGrid, *FrbGrid, *UGrid, *UrbGrid, mass, M5);
-  SquaredLinearOperator<DomainWallFermionD, LatticeFermionD> Dsq (Ddwf);
-  NonHermitianLinearOperator<DomainWallFermionD, LatticeFermionD> DLinOp (Ddwf);
-
-  LatticeFermion src (FGrid); src = 1.0;                              // Source to use 
-  LatticeFermion psiCG (FGrid); psiCG = Zero();                       // Field to solve with for CG
-  LatticeFermion psiGCR (FGrid); psiGCR = Zero();                     // Field to solve with for GCR
-
-  std::cout << GridLogMessage << "*******************************************" << std::endl;
-  std::cout << GridLogMessage << "********** TESTING CG POLY COEFFS *********" << std::endl;
-  std::cout << GridLogMessage << "*******************************************" << std::endl << std::endl;
-
-  double tol = 1.0e-8;
-  int N = 5;           // max iterations (size of Krylov basis)
-
-  // GCR variables
-  int outer_iters = 1;                  // num restarts for GCR
-  TrivialPrecon<LatticeFermionD> prec;  // trivial preconditioner
-
-  ConjugateGradientPolynomial<LatticeFermion> CGP(tol, N, false);
-  CGP(Dsq, src, psiCG);
-
-  // Compute Krylov coeffs directly and compare
-  std::vector<ComplexD> cg_coeffs (N);
-  poly_coeffs(cg_coeffs, Dsq, src, psiCG, FGrid, N, true);
-
-  PolynomialFile PF;
-
-  // Use GCR solver, also get poly coeffs
-  std::vector<ComplexD> gcr_sym_coeffs (N);     // Can try N --> N + 3 to test to see if the last 3 comps are 0
-  PGCRPolynomial<LatticeFermionD> GCRPolySym(tol, outer_iters, Dsq, prec, N+1, N, PF);    // mmax sets the memory, note the last beta doesn't really matter for updating the polynomial
-  GCRPolySym(src, psiGCR);
-  // poly_coeffs(gcr_sym_coeffs, Dsq, src, psi, FGrid, N, true);
-  poly_coeffs(gcr_sym_coeffs, Dsq, src, psiGCR, FGrid, N, true);
-
-  std::cout << GridLogMessage << std::endl << "******** CG POLYNOMIAL COEFFICIENTS *******" << std::endl;
-  std::cout << GridLogMessage << CGP.polynomial << std::endl << std::endl;
-
-  std::cout << GridLogMessage << "****** DIRECT POLYNOMIAL COEFFICIENTS *****" << std::endl;
-  std::cout << GridLogMessage << cg_coeffs << std::endl << std::endl;
-
-  // TODO: try GCR with a Hermitian operator (Dsq)
-  std::cout << GridLogMessage << "****** GCR COEFFICIENTS *****" << std::endl;
-  std::cout << GridLogMessage << GCRPolySym.polynomial << std::endl << std::endl;
-
-  std::cout << GridLogMessage << "****** DIRECT GCR COEFFICIENTS *****" << std::endl;
-  std::cout << GridLogMessage << gcr_sym_coeffs << std::endl << std::endl;
-
-  // test how good the decomposition is
-  std::cout << "Testing fidelity of decomposition by computing ||psi* - sum_i c_i D^i b||^2!" << std::endl;
-  LatticeFermion psiPrime (FGrid);
-
-  // for CG
-  krylovApprox(psiPrime, src, Dsq, cg_coeffs);
-  std::cout << "CG with Dsq, ||psi - psiPrime||^2 = " << norm2(psiCG - psiPrime) << std::endl;
-
-  // for GCR with alpha / beta computation
-  krylovApprox(psiPrime, src, Dsq, GCRPolySym.polynomial);
-  std::cout << "GCR with Dsq, ||psi - psiPrime||^2 = " << norm2(psiGCR - psiPrime) << std::endl;
-
-  // for GCR with alpha / beta computation
-  krylovApprox(psiPrime, src, Dsq, gcr_sym_coeffs);
-  std::cout << "GCR direct with Dsq, ||psi - psiPrime||^2 = " << norm2(psiGCR - psiPrime) << std::endl;
-
-  
-  // std::vector<double> real_cg_diff (N);
-  // for (int i = 0; i < N; i++) { real_cg_diff[i] = std::abs(cg_coeffs[i].real() - CGP.polynomial[i]); }
-  // std::cout << GridLogMessage << "************* COEFF DIFFERENCE ************" << std::endl;
-  // std::cout << GridLogMessage << real_cg_diff << std::endl << std::endl;
-
-  // GCR polynomial reconstruction with Ddwf!
-  std::cout << GridLogMessage << "*******************************************" << std::endl;
-  std::cout << GridLogMessage << "********* TESTING GCR POLY COEFFS *********" << std::endl;
-  std::cout << GridLogMessage << "*******************************************" << std::endl << std::endl;
-
-  // re-init variables
-  src = 1.0;
-  src = (1 / std::sqrt(norm2(src))) * src;
-  psiGCR = Zero(); psiPrime = Zero();
-
-  // test GCR poly
-  PGCRPolynomial<LatticeFermionD> GCRPoly(tol, outer_iters, DLinOp, prec, N+1, N, PF);    // mmax sets the memory, note the last beta doesn't really matter for updating the polynomial
-  GCRPoly(src, psiGCR);
-
-  // Compute Krylov coeffs directly and compare
-  // N = 1;    // compare the N > 1 decomposition with the psi* resulting from N = 1
-  std::vector<ComplexD> gcr_coeffs (N);   // note N --> N + k should just give k coeffs that are 0; this works as intended
-  poly_coeffs(gcr_coeffs, DLinOp, src, psiGCR, FGrid, N, false);
-
-  std::cout << GridLogMessage << "******* GCR POLYNOMIAL COEFFICIENTS *******" << std::endl;
-  std::cout << GridLogMessage << GCRPoly.polynomial << std::endl << std::endl;
-
-  std::cout << GridLogMessage << "****** DIRECT POLYNOMIAL COEFFICIENTS *****" << std::endl;
-  std::cout << GridLogMessage << gcr_coeffs << std::endl << std::endl;
-
-  // test how good the decomposition is
-  std::cout << "Testing fidelity of decomposition by computing ||psi* - sum_i c_i D^i b||^2!" << std::endl;
-
-  // for GCR with alpha / beta computation
-  krylovApprox(psiPrime, src, DLinOp, GCRPoly.polynomial);
-  std::cout << "GCR with Dsq, ||psi - psiPrime||^2 = " << norm2(psiGCR - psiPrime) << std::endl;
-
-  // for GCR with alpha / beta computation
-  krylovApprox(psiPrime, src, DLinOp, gcr_coeffs);
-  std::cout << "GCR direct with Dsq, ||psi - psiPrime||^2 = " << norm2(psiGCR - psiPrime) << std::endl;
-
-  // TESTS TO DO THE N = 2 CASE DIRECTLY
-  /*
-  std::vector<std::complex<double>> alphas {
-    std::complex(0.244300601, 0.00013007545), 
-    std::complex(0.285370971, -0.000160704481)
-  };
-  std::complex<double> beta00 (-0.184661284, -6.52153945e-05);
-  LatticeFermion psi2 (FGrid);
-  LatticeFermion Dsrc (FGrid);
-  DLinOp.Op(src, Dsrc);
-  std::complex<double> c1 = alphas[0] + alphas[1] * (1. + beta00);
-  std::complex<double> c2 = -alphas[0] * alphas[1];
-  psi2 = c1 * src + c2 * Dsrc;
-
-  std::cout << "||b|| = " << norm2(src) << std::endl;
-  std::cout << "||Db|| = " << norm2(Dsrc) << std::endl;
-  // fail; so far this is giving something different than what's being computed in krylovApprox (idk how?)
-
-  std::cout << "c1 and c2 are: " << c1 << " and " << c2 << std::endl;
-  std::cout << "GCRPoly polynomial coeffs are (should equal c1 and c2): " << GCRPoly.polynomial << std::endl;
-  std::cout << "||GCRpsi - psi2||_2^2 = " << norm2(psiGCR - psi2) << std::endl;
-  // pass
-  
-  LatticeFermion src2 (FGrid);
-  src2 = 1.0;
-  src2 = (1 / std::sqrt(norm2(src2))) * src2;
-  std::cout << "||ones - src|| (to verify that src is the same throughout, should be 0) = " << norm2(src2 - src) << std::endl;
-  // pass
-
-  krylovApprox(psiPrime, src, DLinOp, GCRPoly.polynomial);
-  std::cout << "GCR with Dsq, ||psi2 - psiPrime||^2 = " << norm2(psi2 - psiPrime) << std::endl;
-
-  std::vector<ComplexD> psi2_coeffs (N);   // note N --> N + k should just give k coeffs that are 0; this works as intended
-  poly_coeffs(psi2_coeffs, DLinOp, src, psi2, FGrid, N, false);
-  krylovApprox(psiPrime, src, DLinOp, psi2_coeffs);
-  std::cout << "GCR direct with Dsq, ||psi - psiPrime||^2 = " << norm2(psi2 - psiPrime) << std::endl;
-  */
-
-  // std::complex z (10.0, 0.0);     // z = 10
-  // std::cout << GridLogMessage << "************* GCR POLY(z = 10) *************" << std::endl;
-  // std::cout << GridLogMessage << poly_approx(z, GCRPoly.polynomial) << std::endl;
-  // std::cout << GridLogMessage << "************ DIRECT POLY(z = 10) ***********" << std::endl;
-  // std::cout << GridLogMessage << poly_approx(z, gcr_coeffs) << std::endl;
-
-  // std::vector<std::complex<double>> gcr_diff (N);
-  // for (int i = 0; i < N; i++) { gcr_diff[i] = gcr_coeffs[i] - GCRPoly.polynomial[i]; }
-  // std::cout << GridLogMessage << "*********** GCR COEFF DIFFERENCE **********" << std::endl;
-  // std::cout << GridLogMessage << gcr_diff << std::endl << std::endl;
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_krylov_schur.cc

@@ -1,425 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./examples/Example_krylov_schur.cc
-
-    Copyright (C) 2023
-
-    Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-    Author: Patrick Oare <patrickoare@gmail.com>
-    Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-// copied here from Test_general_coarse_pvdagm.cc
-
-#include <cstdlib>
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-using namespace std;
-using namespace Grid;
-
-namespace Grid {
-
-struct LanczosParameters: Serializable {
-  GRID_SERIALIZABLE_CLASS_MEMBERS(LanczosParameters,
-		  		RealD, mass , 
-		  		RealD, mstep , 
-				Integer, Nstop,
-                                Integer, Nk,
-                                Integer, Np,
-                                Integer, ReadEvec,
-                                Integer, maxIter,
-                                Integer, Nblock,
-                                Integer, verify,
-		  		RealD, shift , 
-	  			RealD, resid,
-	  			RealD, ChebyLow,
-	  			RealD, ChebyHigh,
-	  			Integer, ChebyOrder)
-
-  LanczosParameters() {
-    ////////////////////////////// Default values
-      mass = 0;
-    /////////////////////////////////
-  }
-
-  template <class ReaderClass >
-  LanczosParameters(Reader<ReaderClass> & TheReader){
-    initialize(TheReader);
-  }
-
-  template < class ReaderClass > 
-  void initialize(Reader<ReaderClass> &TheReader){
-//    std::cout << GridLogMessage << "Reading HMC\n";
-    read(TheReader, "HMC", *this);
-  }
-
-
-  void print_parameters() const {
-//    std::cout << GridLogMessage << "[HMC parameters] Trajectories            : " << Trajectories << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Start trajectory        : " << StartTrajectory << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Metropolis test (on/off): " << std::boolalpha << MetropolisTest << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Thermalization trajs    : " << NoMetropolisUntil << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Starting type           : " << StartingType << "\n";
-//    MD.print_parameters();
-  }
-  
-};
-
-}
-
-template <class T> void writeFile(T& in, std::string const fname){
-#if 1
-  // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-  std::cout << Grid::GridLogMessage << "Writes to: " << fname << std::endl;
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter WR(in.Grid()->IsBoss());
-  WR.open(fname);
-  WR.writeScidacFieldRecord(in,record,0);
-  WR.close();
-#endif
-  // What is the appropriate way to throw error?
-}
-
-
-typedef WilsonFermionD WilsonOp;
-typedef typename WilsonFermionD::FermionField FermionField;
-
-template<class Matrix,class Field>
-class InvertNonHermitianLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  RealD _stp;
-public:
-  InvertNonHermitianLinearOperator(Matrix &Mat,RealD stp=1e-8): _Mat(Mat),_stp(stp){};
-  // Support for coarsening to a multigrid
-  void OpDiag (const Field &in, Field &out) {
-    assert(0);
-  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {
-    assert(0);
-  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){
-//    _Mat.MdirAll(in,out);
-    assert(0);
-  };
-  void Op     (const Field &in, Field &out){
-    Field tmp(in.Grid());
-    _Mat.Mdag(in,tmp);
-    MdagMLinearOperator<Matrix,Field> HermOp(_Mat);
-    ConjugateGradient<Field> CG(_stp,10000);
-    CG(HermOp,tmp,out);
-//    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    _Mat.Mdag(in,out);
-//    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
-    assert(0);
-  }
-  void HermOp(const Field &in, Field &out){
-    assert(0);
-  }
-};
-
-template<class Field>
-void testSchurFromHess(Arnoldi<Field>& Arn, Field& src, int Nlarge, int Nm, int Nk) {
-
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout << GridLogMessage << "Testing Schur reordering, Nm = " << Nm << ", Nk = " << Nk << std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-
-  std::cout << GridLogMessage << "Running Arnoldi for 1 iteration to get a Hessenberg." << std::endl;
-  Arn(src, 1, Nlarge, Nm, Nlarge);
-  Eigen::MatrixXcd Hess = Arn.getHessenbergMat();
-  std::cout << GridLogMessage << "Hessenberg for use: " << std::endl << Hess << std::endl;
-
-  ComplexSchurDecomposition schur (Hess, true);
-  bool isDecomposed = schur.checkDecomposition();
-  std::cout << "Schur decomp holds? " << isDecomposed << std::endl;
-
-  std::cout << GridLogMessage << "S = " << std::endl << schur.getMatrixS() << std::endl;
-  std::cout << GridLogMessage << "Swapping S(3, 3) with S(4, 4)" << std::endl;
-  schur.swapEvals(3);
-  std::cout << GridLogMessage << "S after swap = " << std::endl << schur.getMatrixS() << std::endl;
-  std::cout << "Schur decomp still holds? " << schur.checkDecomposition() << std::endl;
-
-  // Now move last diagonal element all the way to the front.
-  std::cout << GridLogMessage << "Moving last eval to front. S at start = " << std::endl << schur.getMatrixS() << std::endl;
-  for (int i = 0; i < Nk - 1; i++) {
-    int swapIdx = Nk - 2 - i;
-    schur.swapEvals(swapIdx);
-    std::cout << GridLogMessage << "S after swap of index " << swapIdx << " = " << std::endl << schur.getMatrixS() << std::endl;
-    std::cout << "Schur decomp still holds? " << schur.checkDecomposition() << std::endl;
-  }
-
-  std::cout << GridLogMessage << "Testing Schur reorder" << std::endl;
-  schur.schurReorder(Nk);
-  std::cout << GridLogMessage << "S after reorder = " << std::endl << schur.getMatrixS() << std::endl;
-  std::cout << "Schur decomp still holds? " << schur.checkDecomposition() << std::endl;
-
-}
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  const int Ls=16;
-
-//   GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-//  std::vector<int> lat_size {32, 32, 32, 32};
-//  std::cout << "Lattice size: " << lat_size << std::endl;
-  GridCartesian * UGrid = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), 
-								          GridDefaultSimd(Nd,vComplex::Nsimd()),
-								          GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-
-//  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-//  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-  GridCartesian         * FGrid   = UGrid;
-  GridRedBlackCartesian * FrbGrid = UrbGrid;
-
-  // Construct a coarsened grid
-  // poare TODO: replace this with the following line?
-  Coordinate clatt = GridDefaultLatt();
-//   Coordinate clatt = GridDefaultLatt();              // [PO] initial line before I edited it
-  for(int d=0;d<clatt.size();d++){
-  std::cout << GridLogMessage<< clatt[d] <<std::endl;
-    clatt[d] = clatt[d]/2;
-    //    clatt[d] = clatt[d]/4;
-  }
-  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());;
-  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  std::vector<int> cseeds({5,6,7,8});
-  GridParallelRNG          RNG5(FGrid);   RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG          RNG4(UGrid);   RNG4.SeedFixedIntegers(seeds4);
-  GridParallelRNG          CRNG(Coarse5d);CRNG.SeedFixedIntegers(cseeds);
-
-  LatticeFermion result(FGrid); result=Zero();
-  LatticeFermion    ref(FGrid); ref=Zero();
-  LatticeFermion    tmp(FGrid);
-  LatticeFermion    err(FGrid);
-  LatticeGaugeField Umu(UGrid);
-
-  FieldMetaData header;
-  std::string file("config");
-//  std::string file("Users/patrickoare/libraries/PETSc-Grid/ckpoint_lat.4000");
-  NerscIO::readConfiguration(Umu,header,file);
-
-  LanczosParameters LanParams;
-  {
-    XmlReader  HMCrd("LanParams.xml");
-    read(HMCrd,"LanczosParameters",LanParams);
-  }
-
-  std::cout << GridLogMessage<< LanParams <<std::endl;
-  {
-    XmlWriter HMCwr("LanParams.xml.out");
-    write(HMCwr,"LanczosParameters",LanParams);
-  }
-
-
-  RealD mass=0.01;
-  RealD M5=1.8;
-
-  // PowerMethod<LatticeFermion> PM; PM(PVdagM, src);
-  int Nm = 50;
-  int Nk = 12; 
-  int Np = 38; 
-  // int Nk = Nm+1;     // if just running once
-  int maxIter = 10000;
-  int Nstop = 10;
-  RealD resid = 1.0e-5;
-
-  std::vector<Complex> boundary = {1,1,1,-1};
-  WilsonOp::ImplParams Params(boundary);
-
-//  DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
-//  DomainWallFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5);
-
-  mass=LanParams.mass;
-  std::cout << GridLogIRL<< "mass "<<mass<<std::endl;
-  WilsonOp WilsonOperator(Umu,*UGrid,*UrbGrid,mass,Params);
-
-  // const int nbasis = 20;            // size of approximate basis for low-mode space
-  const int nbasis = 3;            // size of approximate basis for low-mode space
-  const int cb = 0 ;
-  LatticeFermion prom(FGrid);
-
-  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis> LittleDiracOperator;
-  typedef LittleDiracOperator::CoarseVector CoarseVector;
-
-  NextToNearestStencilGeometry5D geom(Coarse5d);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-
-//  typedef PVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> PVdagM_t;
-//  typedef ShiftedPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedPVdagM_t;
-//  typedef ShiftedComplexPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedComplexPVdagM_t;
-//  PVdagM_t PVdagM(Ddwf, Dpv);
-//  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv);
-//  SquaredLinearOperator<DomainWallFermionD, LatticeFermionD> Dsq (Ddwf);
-//  NonHermitianLinearOperator<DomainWallFermionD, LatticeFermionD> DLinOp (Ddwf);
-
-
-  NonHermitianLinearOperator<WilsonOp,FermionField> Dwilson(WilsonOperator); /// <-----
-  InvertNonHermitianLinearOperator<WilsonOp,FermionField> Iwilson(WilsonOperator); /// <-----
-  MdagMLinearOperator<WilsonOp,FermionField> HermOp(WilsonOperator); /// <-----
-  Gamma5HermitianLinearOperator <WilsonOp,LatticeFermion> HermOp2(WilsonOperator); /// <----
-
-  // PowerMethod<LatticeFermion> PM; PM(PVdagM, src);
-  resid=LanParams.resid;
-  Nstop=LanParams.Nstop;
-  Nk=LanParams.Nk;
-  Np=LanParams.Np;
-  maxIter=LanParams.maxIter;
-  Nm = Nk + Np;
-  int Nu=16;
-  std::vector<LatticeFermion> src(Nu,FGrid); 
-  for(int i=0;i<Nu;i++){
-     random(RNG5,src[i]);
-#if 0
-     LatticeFermion src_e(FrbGrid);
-     pickCheckerboard(Even, src_e, src[i]);
-     src_e=Zero();	
-     setCheckerboard(src[i],src_e);
-     std::cout << GridLogMessage << "src["<<i<<"] even sites zeroed " << std::endl;
-#endif
-  }
-
-  if(LanParams.ReadEvec) {
-    std::string evecs_file="evec_in";
-    std::cout << GridLogIRL<< "Reading evecs from "<<evecs_file<<std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacReader RD;
-    RD.open(evecs_file);
-    RD.readScidacFieldRecord(src[0],record);
-    RD.close();
-  }
-
-  Coordinate origin ({0,0,0,0});
-  auto tmpSrc = peekSite(src[0], origin);
-  std::cout << "[DEBUG] Source at origin = " <<  tmpSrc << std::endl;
-  LatticeFermion src2 = src[0];
-
-  // Run KrylovSchur / BlockKrylovSchur / TrueHarmonicBlockKrylovSchur on a Hermitian matrix
-    RealD shift=LanParams.shift;
-
-    // --use-blockKS : off -> plain (non-block) KrylovSchur
-    //                 on  -> block family (BlockKrylovSchur / TrueHarmonicBlockKrylovSchur)
-    // --use-harmonic: off -> no shift/harmonic Ritz extraction
-    //                 on, without --use-blockKS -> plain KrylovSchur with shifted (harmonic) Schur extraction
-    //                 on, with --use-blockKS    -> TrueHarmonicBlockKrylovSchur (true harmonic Ritz)
-    bool useBlockKS  = GridCmdOptionExists(argv, argv+argc, std::string("--use-blockKS"));
-    bool useHarmonic = GridCmdOptionExists(argv, argv+argc, std::string("--use-harmonic"));
-
-    std::vector<LatticeFermion> finalEvecs;
-
-    if (!useBlockKS) {
-      KrylovSchur KrySchur (Dwilson, UGrid, resid,EvalImNormSmall);
-      KrySchur.doEvalCheck=true;
-      if (useHarmonic) {
-        std::cout << GridLogMessage << "Running KrylovSchur (shifted/harmonic Ritz)" << std::endl;
-        KrySchur(src[0], maxIter, Nm, Nk, Nstop,&shift);
-      } else {
-        std::cout << GridLogMessage << "Running KrylovSchur" << std::endl;
-        KrySchur(src[0], maxIter, Nm, Nk, Nstop);
-      }
-      std::cout << GridLogMessage << "KrylovSchur evec.size= " << KrySchur.evecs.size()<< std::endl;
-      finalEvecs = KrySchur.evecs;
-    } else {
-      int Nblock=4;
-      Nblock=LanParams.Nblock;
-      bool if_verify=false;
-      if(LanParams.verify) if_verify=true;
-
-      auto gamma5Lambda = [](const FermionField& v, FermionField& out) {
-        Gamma g5(Gamma::Algebra::Gamma5);
-        out = g5 * v;
-      };
-
-      if (useHarmonic) {
-        std::cout << GridLogMessage << "Running TrueHarmonicBlockKrylovSchur" << std::endl;
-        TrueHarmonicBlockKrylovSchur<FermionField> KrySchur (Dwilson, UGrid, resid,shift,EvalImNormSmall);
-        KrySchur.doEvalCheck=true;
-        KrySchur.gamma5Func = gamma5Lambda;
-        KrySchur(src, maxIter, Nm, Nk, Nstop,Nblock,true,if_verify);
-        std::cout << GridLogMessage << "TrueHarmonicBlockKrylovSchur evec.size= " << KrySchur.evecs.size()<< std::endl;
-        finalEvecs = KrySchur.evecs;
-      } else {
-        std::cout << GridLogMessage << "Running BlockKrylovSchur" << std::endl;
-        BlockKrylovSchur<FermionField> KrySchur (Dwilson, UGrid, resid,EvalImNormSmall);
-        KrySchur.doEvalCheck=true;
-        KrySchur.gamma5Func = gamma5Lambda;
-        KrySchur(src, maxIter, Nm, Nk, Nstop,Nblock,true,if_verify);
-        std::cout << GridLogMessage << "BlockKrylovSchur evec.size= " << KrySchur.evecs.size()<< std::endl;
-        finalEvecs = KrySchur.evecs;
-      }
-    }
-
-  src[0]=finalEvecs[0];
-  std::cout << GridLogMessage << "finalEvecs= "<< finalEvecs.size() <<std::endl;
-  for (int i=1;i<Nk;i++) src[0]+=finalEvecs[i];
-  for (int i=0;i<Nstop;i++)
-  {
-	std::string evfile ("./evec_"+std::to_string(mass)+"_"+std::to_string(i));
-        auto evdensity = localInnerProduct(finalEvecs[i],finalEvecs[i] );
-        writeFile(evdensity,evfile);
-
-  }
-
-  {
-        std::string evfile ("./evec_"+std::to_string(mass)+"_sum");
-//        auto evdensity = localInnerProduct(evec[i],evec[i] );
-        writeFile(src[0],evfile);
-  }
-
-
-  /*
-  std::cout << GridLogMessage << "Running Arnoldi" << std::endl;
-  // Arnoldi Arn (Dsq, FGrid, 1e-8);
-  Arnoldi Arn (DLinOp, FGrid, 1e-8);
-  testSchurFromHess<LatticeFermion>(Arn, src, 10, 6, 4);
-
-  Arnoldi Arn2 (DLinOp, FGrid, 1e-8);
-  testSchurFromHess<LatticeFermion>(Arn2, src, 16, 12, 8);
-  */
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_pvdagm_3level.cc

@@ -0,0 +1,699 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_3level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
+// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
+template<class Field>
+class LuscherGuesser : public LinearFunction<Field> {
+  const std::vector<Field> &psi;
+  Eigen::MatrixXcd          C_inv;
+public:
+  using LinearFunction<Field>::operator();
+  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
+    : psi(psi_), C_inv(Cinv_) {}
+  virtual void operator()(const Field &src, Field &guess) {
+    int N = psi.size();
+    Eigen::VectorXcd b(N);
+    for (int t = 0; t < N; t++)
+      b(t) = TensorRemove(innerProduct(psi[t], src));
+    Eigen::VectorXcd c = C_inv * b;
+    guess = Zero();
+    for (int s = 0; s < N; s++)
+      guess += ComplexD(c(s)) * psi[s];
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_)
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  L2PGCR_baseline.Name("Cbaseline");
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Lüscher deflation guesser for L3PGCR.
+  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
+  //         CoarseCoarseVector space — these cover the zero-momentum
+  //         component of the near-null space of LinOpCC.
+  // Step 2: breed Nextra additional null vectors directly on LinOpCC
+  //         using GCR with random sources — these pick up near-null
+  //         modes at all spatial frequencies not spanned by step 1.
+  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
+  //         full augmented basis and invert directly via Eigen LU.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+
+  {
+    int Nextra = nbasis;  // breed as many extra as we have promoted ones
+    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
+    GridParallelRNG RNG_CC(CoarseCoarse5d);
+    RNG_CC.SeedFixedIntegers({11,13,17,19});
+    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
+    CoarseCoarseVector tmp(CoarseCoarse5d);
+    for (int k = 0; k < Nextra; k++) {
+      CoarseCoarseVector src(CoarseCoarse5d);
+      gaussian(RNG_CC, src);
+      tmp = Zero();
+      nullGCR(src, tmp);
+      psi_cc.push_back(tmp);
+    }
+    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
+              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
+  }
+
+  const int Naug = psi_cc.size();
+  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
+  {
+    CoarseCoarseVector Acc(CoarseCoarse5d);
+    for (int l = 0; l < Naug; l++) {
+      LinOpCC.Op(psi_cc[l], Acc);
+      for (int k = 0; k < Naug; k++)
+        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
+    }
+  }
+  {
+    RealD normCcc      = Ccc.norm();
+    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
+  }
+  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
+  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2 solver: plain GCR, no further coarsening
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3PGCR(1.0e-1,200,LinOpCC,simpleCC,16,16);
+  L3PGCR.Level(3);
+  L3PGCR.Name("CCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.01;
+  int   coarse_smoother_nstep = 12;
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+                       LinOpCoarse,
+                       simpleC,            // no pre-smoother (matches fine-grid setup)
+                       CoarseSmootherGCR,  // post-smoother: 12 GCR steps
+                       LinOpCC,
+                       L3PGCR,
+                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  L2MGsolver.Name("Couter");
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full three-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+                               PVdagM,
+                               simple_fine,
+                               SmootherGCR,
+                               LinOpCoarse,
+                               L2MGsolver,
+                               simpleC);
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, matching CreateSubspaceGCR
+    // (Aggregates.h:196), in case the cached file predates it.  Span-preserving
+    // and globally orthonormal -- NOT the block Orthogonalise() below, which would
+    // defeat the raw-null discipline (runMG promotes the RAW subspace to build L2;
+    // block-GS here -> psi_coarse = e_k).  The raw copy in runMG happens AFTER this.
+    AggregatesGCR.GlobalOrthonormalise();
+    //    AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_3level_madj.cc

@@ -0,0 +1,625 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_3level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+  std::string      _name;
+  
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_,
+		   std::string name = std::string("unnamed"))
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      _name(name),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << _name << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << _name << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << _name <<"PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2 solver: plain GCR, no further coarsening
+  //////////////////////////////////////////////////////////////////////
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+  // L3PGCR is an inner solver inside the L1→2 V-cycle; does not need to converge
+  // to fine-grid precision. Loose tolerance (3e-2) and large restart (64) to allow
+  // the Krylov space to span enough of the near-null spectrum of LinOpCC per cycle.
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3PGCR(1.0e-4,5,LinOpCC,simpleCC,64,64);
+  L3PGCR.Level(3);
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.0;
+  int   coarse_smoother_nstep = 8;
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.0,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.Level(2);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+		      LinOpCoarse,
+		      simpleC,            // no pre-smoother (matches fine-grid setup)
+		      CoarseSmootherGCR,  // post-smoother: 12 GCR steps
+		      LinOpCC,
+		      L3PGCR,
+		      simpleCC,
+		      std::string("LinOpC"));
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full three-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.00,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.Level(1);
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+			      PVdagM,
+			      simple_fine,
+			      SmootherGCR,
+			      LinOpCoarse,
+			      L2MGsolver,
+			      simpleC,
+			      std::string("PVdagM"));
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+
+  RealD madj = 1.0;
+  if ( getenv("MADJ") ) madj=atof(getenv("MADJ"));
+  std::cout << "PV mass set to "<<madj<<std::endl;
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,madj, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.00,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Re-orthogonalise after loading to ensure block-GS condition holds.
+    //    AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded and re-orthogonalised." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_4level.cc

@@ -0,0 +1,798 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_3level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
+// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
+template<class Field>
+class LuscherGuesser : public LinearFunction<Field> {
+  const std::vector<Field> &psi;
+  Eigen::MatrixXcd          C_inv;
+public:
+  using LinearFunction<Field>::operator();
+  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
+    : psi(psi_), C_inv(Cinv_) {}
+  virtual void operator()(const Field &src, Field &guess) {
+    int N = psi.size();
+    Eigen::VectorXcd b(N);
+    for (int t = 0; t < N; t++)
+      b(t) = TensorRemove(innerProduct(psi[t], src));
+    Eigen::VectorXcd c = C_inv * b;
+    guess = Zero();
+    for (int s = 0; s < N; s++)
+      guess += ComplexD(c(s)) * psi[s];
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_)
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  GridCartesian *CoarseCoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  L2PGCR_baseline.Name("Cbaseline");
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Lüscher deflation guesser for L3PGCR.
+  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
+  //         CoarseCoarseVector space — these cover the zero-momentum
+  //         component of the near-null space of LinOpCC.
+  // Step 2: breed Nextra additional null vectors directly on LinOpCC
+  //         using GCR with random sources — these pick up near-null
+  //         modes at all spatial frequencies not spanned by step 1.
+  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
+  //         full augmented basis and invert directly via Eigen LU.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+
+  {
+    int Nextra = nbasis;  // breed as many extra as we have promoted ones
+    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
+    GridParallelRNG RNG_CC(CoarseCoarse5d);
+    RNG_CC.SeedFixedIntegers({11,13,17,19});
+    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
+    CoarseCoarseVector tmp(CoarseCoarse5d);
+    for (int k = 0; k < Nextra; k++) {
+      CoarseCoarseVector src(CoarseCoarse5d);
+      gaussian(RNG_CC, src);
+      tmp = Zero();
+      nullGCR(src, tmp);
+      psi_cc.push_back(tmp);
+    }
+    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
+              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
+  }
+
+  const int Naug = psi_cc.size();
+  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
+  {
+    CoarseCoarseVector Acc(CoarseCoarse5d);
+    for (int l = 0; l < Naug; l++) {
+      LinOpCC.Op(psi_cc[l], Acc);
+      for (int k = 0; k < Naug; k++)
+        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
+    }
+  }
+  {
+    RealD normCcc      = Ccc.norm();
+    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
+  }
+  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
+  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3: coarsen LinOpCC using the RAW promoted psi_cc as aggregation
+  // to build the Level 4 (coarse-coarse-coarse) operator.
+  //   psi_cc[0..nbasis-1] are the coarse-coarse near-null vectors, projected
+  //   from the RAW psi_coarse (themselves projected from the RAW fine null
+  //   vectors) -- the pre-block-GS chain the whole construction depends on.
+  //   CoarsenOperator block-GS orthogonalises AggregatesL3.subspace IN PLACE,
+  //   so assign COPIES of psi_cc and keep psi_cc itself raw.
+  //
+  // Tensor depth deepens once more: innerProduct(CoarseCoarseSiteObj,...) returns
+  // iScalar<vTTComplex>, so CComplex for the L2→L3 level is iScalar<iScalar<vTComplex>>.
+  //////////////////////////////////////////////////////////////////////
+  typedef typename CoarseCoarseVector::vector_object                     CoarseCoarseSiteObj;
+  typedef iScalar<vTTComplex>                                            vTTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseCoarseSiteObj,vTTTComplex,NB>     LittleDiracOperatorL3;
+  typedef typename LittleDiracOperatorL3::CoarseVector                   CoarseCoarseCoarseVector;
+  typedef Aggregation<CoarseCoarseSiteObj,vTTTComplex,NB>               SubspaceL3;
+  typedef MGPreconditioner<CoarseCoarseSiteObj,vTTTComplex,NB>          L2to3MG;
+
+  SubspaceL3 AggregatesL3(CoarseCoarseCoarse5d, CoarseCoarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL3.subspace[k] = psi_cc[k];          // raw promoted; COPY, keeps psi_cc raw
+
+  NextToNearestStencilGeometry5D geom3(CoarseCoarseCoarse5d);
+  LittleDiracOperatorL3 LittleDiracOpL3(geom3, CoarseCoarse5d, CoarseCoarseCoarse5d);
+  LittleDiracOpL3.CoarsenOperator(LinOpCC, AggregatesL3);   // block-GS's AggregatesL3.subspace in place
+
+  NonHermitianLinearOperator<LittleDiracOperatorL3,CoarseCoarseCoarseVector> LinOpCCC(LittleDiracOpL3);
+  TrivialPrecon<CoarseCoarseCoarseVector> simpleCCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 4 bottom solve: GCR on a SHIFTED LinOpCCC.  This is the one level
+  // with no IRS shift, and it is the most non-normal (coarsest) operator, so
+  // the bare bottom GCR wanders in a field of values that wraps the origin and
+  // its iteration count blows out (observed 5..54 iters to hit 0.2).  Solving
+  // (A_ccc + l4_shift) instead slides the FoV off the origin; the correction is
+  // only ever a loose 0.2 approximation anyway, so the detuning is free.
+  // l4_shift defaults to 0.0 => bare LinOpCCC, baseline unchanged until opted in.
+  //////////////////////////////////////////////////////////////////////
+  RealD l4_shift = 0.0;
+  if(getenv("l4_shift")) l4_shift = atof(getenv("l4_shift"));
+  std::cout << GridLogMessage << "PARAM l4_shift = " << l4_shift << std::endl;
+
+  ShiftedLinearOperator<CoarseCoarseCoarseVector> ShiftedLinOpCCC(l4_shift, LinOpCCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector> L4PGCR(1.0e-1,200,ShiftedLinOpCCC,simpleCCC,16,16);
+  L4PGCR.Level(4);
+  L4PGCR.Name("CCCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3 V-cycle: depth-2 SHIFTED smoother on LinOpCC + L4 bottom.
+  // The shift slides the coarse-coarse field of values off the origin so a
+  // 2-step smoother has something to bite on a non-normal operator (IRS idea).
+  //////////////////////////////////////////////////////////////////////
+  RealD cc_smoother_shift = 0.01;
+  int   cc_smoother_nstep = 2;
+  if(getenv("cc_smoother_shift")) cc_smoother_shift = atof(getenv("cc_smoother_shift"));
+  if(getenv("cc_smoother_nstep")) cc_smoother_nstep = atoi(getenv("cc_smoother_nstep"));
+
+  ShiftedLinearOperator<CoarseCoarseVector> ShiftedLinOpCC(cc_smoother_shift, LinOpCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    CoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCC,simpleCC,cc_smoother_nstep,cc_smoother_nstep);
+  CoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
+  CoarseCoarseSmootherGCR.Level(3);
+  CoarseCoarseSmootherGCR.Name("CCsmoother");
+
+  L2to3MG L2to3Precon(AggregatesL3,
+                      LinOpCC,
+                      simpleCC,                  // no pre-smoother
+                      CoarseCoarseSmootherGCR,   // post-smoother: depth-2 shifted GCR
+                      LinOpCCC,
+                      L4PGCR,
+                      simpleCCC);                // trivial guesser at the bottom
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3 (coarse-coarse) solve: GCR preconditioned by the L2→L3 V-cycle.
+  // Replaces the plain L3PGCR of the 3-level build -- the coarse-coarse level
+  // is now smoothed shallowly and recursed rather than solved deeply.
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3MGsolver(1.0e-1,200,LinOpCC,L2to3Precon,16,16);
+  L3MGsolver.Level(3);
+  L3MGsolver.Name("CCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.01;
+  int   coarse_smoother_nstep = 2;   // depth-2 smoother on the coarse level
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+                       LinOpCoarse,
+                       simpleC,            // no pre-smoother (matches fine-grid setup)
+                       CoarseSmootherGCR,  // post-smoother: depth-2 shifted GCR
+                       LinOpCC,
+                       L3MGsolver,         // coarse-coarse solve is now the L2→L3 V-cycle
+                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  L2MGsolver.Name("Couter");
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full three-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Three-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+                               PVdagM,
+                               simple_fine,
+                               SmootherGCR,
+                               LinOpCoarse,
+                               L2MGsolver,
+                               simpleC);
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Three-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // Level 3 coarse-coarse-coarse grid: block clatt2 = {12,12,8,16} -> {6,12,8,8}.
+  // GEOMETRY (mpi 3.6.4.4, Nsimd=4 => SIMD layout {1,1,2,2}, factor 2 on z and t):
+  //   every grid needs z-local and t-local EVEN.  clatt2-local is {4,2,2,4}, so
+  //   z-local=2 is already at its minimum even value and CANNOT be blocked (2->1
+  //   is odd and trips the SIMD assertion); y-local=2 would go to 1 (degenerate).
+  //   Only x and t have room, so block {2,1,1,2}: clatt3 {6,12,8,8}, L4-local
+  //   {2,2,2,2} -- all dims even and >=2.  z stays unblocked by construction.
+  Coordinate clatt3 = clatt2;
+  clatt3[0] /= 2;   // x: 12 -> 6   (x-local 4 -> 2)
+  // clatt3[1] (y) unblocked: y-local 2 -> blocking gives 1 (degenerate)
+  // clatt3[2] (z) unblocked: z-local 2 is SIMD-pinned even, cannot halve
+  clatt3[3] /= 2;   // t: 16 -> 8   (t-local 4 -> 2)
+  std::cout << GridLogMessage << "Level 3 coarse-coarse-coarse lattice: " << clatt3 << std::endl;
+
+  GridCartesian *CoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt3, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, in case the cached file
+    // predates the GlobalOrthonormalise() that CreateSubspaceGCR now applies
+    // (Aggregates.h:196).  It is span-preserving and makes the vectors globally
+    // orthonormal -- it is NOT the block Orthogonalise() below, so it does NOT
+    // cause the psi_coarse->e_k trap.  The RAW subspace copy in runMG happens
+    // AFTER this call, so the raw-null (pre-block-GS) discipline is preserved.
+    AggregatesGCR.GlobalOrthonormalise();
+    // DO NOT block-orthogonalise here: runMG copies subspace[] as the RAW
+    // (pre-block-GS) basis and CoarsenOperator block-GS's it in place later.
+    // Orthogonalising now defeats the raw-null discipline (psi_coarse -> e_k)
+    // and poisons L2/L3/L4.  See project_block_orthogonalise_leak.
+    // AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    CoarseCoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_5level.cc

@@ -0,0 +1,954 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_5level.cc
+
+    Copyright (C) 2023
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+#include <Grid/algorithms/iterative/BiCGSTAB.h>
+
+using namespace std;
+using namespace Grid;
+
+template <class T> void readFile(T& out, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Reading: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    SR.readScidacFieldRecord(out, record);
+    SR.close();
+  #endif
+}
+
+template <class T> void writeFile(T& in, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Writing: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(in.Grid()->IsBoss());
+    SW.open(fname);
+    SW.writeScidacFieldRecord(in, record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+    SW.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SW.writeScidacFieldRecord(subspace[k], record);
+    SW.close();
+  #endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+  #ifdef HAVE_LIME
+    std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+    Grid::emptyUserRecord record;
+    Grid::ScidacReader SR;
+    SR.open(fname);
+    for (int k = 0; k < (int)subspace.size(); k++)
+      SR.readScidacFieldRecord(subspace[k], record);
+    SR.close();
+  #endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  int nApp;
+  int nAppDag;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV), nApp(0), nAppDag(0) {};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    nApp++;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+    nAppDag++;
+  }
+  void clear() { nApp = 0; nAppDag = 0; }
+  void getApplications() {
+    std::cout << GridLogMessage << "# applications of PVdagM: " << nApp << std::endl;
+    std::cout << GridLogMessage << "# applications of PVdagM^dag: " << nAppDag << std::endl;
+    std::cout << GridLogMessage << "# applications total: " << nApp + nAppDag << std::endl;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class MdagPVLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  MdagPVLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+  RealD shift;
+public:
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// Lüscher deflated guesser (arXiv:0706.2298 Sec A.3) for a non-Hermitian solve.
+// C_{st} = <psi[s] | LinOp | psi[t]>;  guess = sum_s c_s psi[s]  where c = C^{-1} psi† src.
+template<class Field>
+class LuscherGuesser : public LinearFunction<Field> {
+  const std::vector<Field> &psi;
+  Eigen::MatrixXcd          C_inv;
+public:
+  using LinearFunction<Field>::operator();
+  LuscherGuesser(const std::vector<Field> &psi_, const Eigen::MatrixXcd &Cinv_)
+    : psi(psi_), C_inv(Cinv_) {}
+  virtual void operator()(const Field &src, Field &guess) {
+    int N = psi.size();
+    Eigen::VectorXcd b(N);
+    for (int t = 0; t < N; t++)
+      b(t) = TensorRemove(innerProduct(psi[t], src));
+    Eigen::VectorXcd c = C_inv * b;
+    guess = Zero();
+    for (int s = 0; s < N; s++)
+      guess += ComplexD(c(s)) * psi[s];
+  }
+};
+
+template<class Fobj,class CComplex,int nbasis>
+class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
+public:
+  using LinearFunction<Lattice<Fobj> >::operator();
+
+  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
+  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
+  typedef LinearOperatorBase<FineField>                            FineOperator;
+  typedef LinearFunction    <FineField>                            FineSmoother;
+  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
+  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
+
+  Aggregates     & _Aggregates;
+  FineOperator   & _FineOperator;
+  FineSmoother   & _PreSmoother;
+  FineSmoother   & _PostSmoother;
+  CoarseOperator & _CoarseOperator;
+  CoarseSolver   & _CoarseSolve;
+  CoarseSolver   & _CoarseGuesser;
+
+  int    level;  void Level(int lv) {level = lv; };
+
+  MGPreconditioner(Aggregates &Agg,
+		   FineOperator &Fine,
+		   FineSmoother &PreSmoother,
+		   FineSmoother &PostSmoother,
+		   CoarseOperator &CoarseOperator_,
+		   CoarseSolver &CoarseSolve_,
+		   CoarseSolver &CoarseGuesser_)
+    : _Aggregates(Agg),
+      _FineOperator(Fine),
+      _PreSmoother(PreSmoother),
+      _PostSmoother(PostSmoother),
+      _CoarseOperator(CoarseOperator_),
+      _CoarseSolve(CoarseSolve_),
+      _CoarseGuesser(CoarseGuesser_),
+      level(1)  {  }
+
+  virtual void operator()(const FineField &in, FineField & out)
+  {
+    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
+    CoarseVector Csrc(CoarseGrid);
+    CoarseVector Csol(CoarseGrid);
+    FineField vec1(in.Grid());
+    FineField vec2(in.Grid());
+
+    double t;
+    out = Zero();
+    t=-usecond();
+    _PreSmoother(in,out);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);
+
+    t=-usecond();
+    _Aggregates.ProjectToSubspace(Csrc,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _CoarseGuesser(Csrc,Csol);
+    _CoarseSolve(Csrc,Csol);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
+
+    t=-usecond();
+    _Aggregates.PromoteFromSubspace(Csol,vec1);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
+
+    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);
+
+    t=-usecond();
+    vec2=Zero();
+    _PostSmoother(vec1,vec2);
+    t+=usecond();
+    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
+
+    add(out,out,vec2);
+  }
+};
+
+// Generic shifted linear operator: wraps any LinearOperatorBase and adds shift*I.
+// Used to condition the coarse-level GCR smoother, analogous to ShiftedPVdagMLinearOperator
+// at the fine level.
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : shift(_shift), _Op(Op) {}
+  void OpDiag   (const Field &in, Field &out)                         { assert(0); }
+  void OpDir    (const Field &in, Field &out, int dir, int disp)      { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out)            { assert(0); }
+  void Op       (const Field &in, Field &out) { _Op.Op(in, out);    out = out + shift * in; }
+  void AdjOp    (const Field &in, Field &out) { _Op.AdjOp(in, out); out = out + shift * in; }
+  void HermOpAndNorm(const Field &in, Field &out, RealD &n1, RealD &n2) { assert(0); }
+  void HermOp   (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<int NB, class PVdagM_t, class ShiftedPVdagM_t, class Subspace, class LittleDiracOperator, class CoarseVector, class TwoLevelMG>
+void runMG(
+  GridCartesian *FGrid,
+  GridCartesian *Coarse5d,
+  GridCartesian *CoarseCoarse5d,
+  GridCartesian *CoarseCoarseCoarse5d,
+  GridCartesian *CoarseCoarseCoarseCoarse5d,
+  NextToNearestStencilGeometry5D geom,
+  PVdagM_t &PVdagM,
+  ShiftedPVdagM_t &ShiftedPVdagM,
+  Subspace &AggregatesPD
+) {
+  std::vector<LatticeFermion> subspace = AggregatesPD.subspace;
+  assert((int)subspace.size() == NB);
+  const int nbasis = NB;
+  const int cb = 0;
+
+  CoarseVector c_src(Coarse5d);
+  CoarseVector c_res(Coarse5d);
+  Complex one(1.0);
+
+  LatticeFermionD f_src(FGrid);
+  LatticeFermionD f_res(FGrid);
+
+  TrivialPrecon<CoarseVector>      simpleC;
+  TrivialPrecon<LatticeFermionD>   simple_fine;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 0→1: coarsen PVdagM, build LinOpCoarse
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom, FGrid, Coarse5d);
+  LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesPD);
+
+  NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+  //////////////////////////////////////////////////////////////////////
+  // Baseline: plain PGCR on LinOpCoarse (reference for comparison)
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: plain PGCR baseline"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2PGCR_baseline(3.0e-2,1100,LinOpCoarse,simpleC,10,10);
+  L2PGCR_baseline.Level(2);
+  L2PGCR_baseline.Name("Cbaseline");
+  c_src = one;
+  c_res = Zero();
+  L2PGCR_baseline(c_src,c_res);
+
+  //////////////////////////////////////////////////////////////////////
+  // psi_coarse: coarse projections of pre-GS fine null vectors.
+  // These are the Level 1 near-null vectors, promoted from Level 0.
+  // Used as the aggregation basis for Level 1→2 coarsening.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseVector> psi_coarse(nbasis, Coarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesPD.ProjectToSubspace(psi_coarse[k], subspace[k]);
+
+  //////////////////////////////////////////////////////////////////////
+  // Diagnostics: W (fine projected matrix) and C (Galerkin check)
+  //////////////////////////////////////////////////////////////////////
+  {
+    Eigen::MatrixXcd W = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    LatticeFermion ftmp(FGrid);
+    for (int j = 0; j < nbasis; j++) {
+      PVdagM.Op(subspace[j], ftmp);
+      for (int i = 0; i < nbasis; i++)
+        W(i,j) = TensorRemove(innerProduct(subspace[i], ftmp));
+    }
+    RealD normW = W.norm();
+    std::cout << GridLogMessage << "Fine projected matrix ||W|| = " << normW << std::endl;
+
+    Eigen::MatrixXcd C = Eigen::MatrixXcd::Zero(nbasis, nbasis);
+    CoarseVector Ac(Coarse5d);
+    for (int l = 0; l < nbasis; l++) {
+      LinOpCoarse.Op(psi_coarse[l], Ac);
+      for (int k = 0; k < nbasis; k++)
+        C(k,l) = TensorRemove(innerProduct(psi_coarse[k], Ac));
+    }
+    RealD normC      = C.norm();
+    RealD normCmCdag = (C - C.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse null matrix ||C||            = " << normC << std::endl;
+    std::cout << GridLogMessage << "Coarse null matrix ||C - C†||/||C|| = " << normCmCdag/normC << std::endl;
+    std::cout << GridLogMessage << "Galerkin check ||C||/||W||          = " << normC/normW << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2: set up aggregation using psi_coarse as subspace.
+  // Block factor 2,2,3,2 (removes odd local sublattice in z given MPI
+  // geometry 3×6×4×4 where z-local at Level 1 is 6).
+  // psi_coarse are assigned directly; CoarsenOperator performs
+  // block-GS orthogonalisation before building LinOpCoarseCoarse.
+  //////////////////////////////////////////////////////////////////////
+  // innerProduct(CoarseSiteObj, CoarseSiteObj) returns iScalar<vTComplex>, so CComplex
+  // for the L1→L2 level must be iScalar<vTComplex>, not vTComplex.
+  typedef typename CoarseVector::vector_object                            CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,NB>            LittleDiracOperatorL2;
+  typedef typename LittleDiracOperatorL2::CoarseVector                   CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,NB>                       SubspaceL2;
+  typedef MGPreconditioner<CoarseSiteObj,vTTComplex,NB>                  L1to2MG;
+
+  SubspaceL2 AggregatesL2(CoarseCoarse5d, Coarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.subspace[k] = psi_coarse[k];
+
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);
+  LittleDiracOperatorL2 LittleDiracOpL2(geom2, Coarse5d, CoarseCoarse5d);
+  LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+
+  NonHermitianLinearOperator<LittleDiracOperatorL2,CoarseCoarseVector> LinOpCC(LittleDiracOpL2);
+
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Lüscher deflation guesser for L3PGCR.
+  // Step 1: project psi_coarse[k] (promoted fine null vectors) to
+  //         CoarseCoarseVector space — these cover the zero-momentum
+  //         component of the near-null space of LinOpCC.
+  // Step 2: breed Nextra additional null vectors directly on LinOpCC
+  //         using GCR with random sources — these pick up near-null
+  //         modes at all spatial frequencies not spanned by step 1.
+  // Step 3: build C_{st} = <psi_cc[s]|LinOpCC|psi_cc[t]> over the
+  //         full augmented basis and invert directly via Eigen LU.
+  //////////////////////////////////////////////////////////////////////
+  std::vector<CoarseCoarseVector> psi_cc(nbasis, CoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+
+  {
+    int Nextra = nbasis;  // breed as many extra as we have promoted ones
+    if ( getenv("CC_NEXTRA") ) Nextra = atoi(getenv("CC_NEXTRA"));
+    GridParallelRNG RNG_CC(CoarseCoarse5d);
+    RNG_CC.SeedFixedIntegers({11,13,17,19});
+    PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+      nullGCR(1e-2, 200, LinOpCC, simpleCC, 32, 32);
+    CoarseCoarseVector tmp(CoarseCoarse5d);
+    for (int k = 0; k < Nextra; k++) {
+      CoarseCoarseVector src(CoarseCoarse5d);
+      gaussian(RNG_CC, src);
+      tmp = Zero();
+      nullGCR(src, tmp);
+      psi_cc.push_back(tmp);
+    }
+    std::cout << GridLogMessage << "LinOpCC deflation basis: " << nbasis
+              << " promoted + " << Nextra << " bred = " << psi_cc.size() << " total" << std::endl;
+  }
+
+  const int Naug = psi_cc.size();
+  Eigen::MatrixXcd Ccc = Eigen::MatrixXcd::Zero(Naug, Naug);
+  {
+    CoarseCoarseVector Acc(CoarseCoarse5d);
+    for (int l = 0; l < Naug; l++) {
+      LinOpCC.Op(psi_cc[l], Acc);
+      for (int k = 0; k < Naug; k++)
+        Ccc(k,l) = TensorRemove(innerProduct(psi_cc[k], Acc));
+    }
+  }
+  {
+    RealD normCcc      = Ccc.norm();
+    RealD normCccmCdag = (Ccc - Ccc.adjoint()).norm();
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc||              = " << normCcc << std::endl;
+    std::cout << GridLogMessage << "Coarse-coarse deflation matrix ||Ccc-Ccc†||/||Ccc|| = " << normCccmCdag/normCcc << std::endl;
+  }
+  Eigen::MatrixXcd Ccc_inv = Ccc.inverse();
+  LuscherGuesser<CoarseCoarseVector> CCDeflGuesser(psi_cc, Ccc_inv);
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3: coarsen LinOpCC using the RAW promoted psi_cc as aggregation
+  // to build the Level 4 (coarse-coarse-coarse) operator.
+  //   psi_cc[0..nbasis-1] are the coarse-coarse near-null vectors, projected
+  //   from the RAW psi_coarse (themselves projected from the RAW fine null
+  //   vectors) -- the pre-block-GS chain the whole construction depends on.
+  //   CoarsenOperator block-GS orthogonalises AggregatesL3.subspace IN PLACE,
+  //   so assign COPIES of psi_cc and keep psi_cc itself raw.
+  //
+  // Tensor depth deepens once more: innerProduct(CoarseCoarseSiteObj,...) returns
+  // iScalar<vTTComplex>, so CComplex for the L2→L3 level is iScalar<iScalar<vTComplex>>.
+  //////////////////////////////////////////////////////////////////////
+  typedef typename CoarseCoarseVector::vector_object                     CoarseCoarseSiteObj;
+  typedef iScalar<vTTComplex>                                            vTTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseCoarseSiteObj,vTTTComplex,NB>     LittleDiracOperatorL3;
+  typedef typename LittleDiracOperatorL3::CoarseVector                   CoarseCoarseCoarseVector;
+  typedef Aggregation<CoarseCoarseSiteObj,vTTTComplex,NB>               SubspaceL3;
+  typedef MGPreconditioner<CoarseCoarseSiteObj,vTTTComplex,NB>          L2to3MG;
+
+  SubspaceL3 AggregatesL3(CoarseCoarseCoarse5d, CoarseCoarse5d, cb);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL3.subspace[k] = psi_cc[k];          // raw promoted; COPY, keeps psi_cc raw
+
+  NextToNearestStencilGeometry5D geom3(CoarseCoarseCoarse5d);
+  LittleDiracOperatorL3 LittleDiracOpL3(geom3, CoarseCoarse5d, CoarseCoarseCoarse5d);
+  LittleDiracOpL3.CoarsenOperator(LinOpCC, AggregatesL3);   // block-GS's AggregatesL3.subspace in place
+
+  NonHermitianLinearOperator<LittleDiracOperatorL3,CoarseCoarseCoarseVector> LinOpCCC(LittleDiracOpL3);
+  TrivialPrecon<CoarseCoarseCoarseVector> simpleCCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3→4: coarsen LinOpCCC to build the Level 5 operator, using a
+  // TRUNCATED basis of only the first NB5 (< nbasis) raw promoted null vectors.
+  //   psi_ccc[k] = raw psi_cc projected through the (block-GS'd) L3 aggregation
+  //   -- the pre-block-GS chain continued one level deeper.  We keep only the
+  //   leading NB5: after the global orthogonalisation of the original fine null
+  //   vectors the early indices retain the most-null content (shared low-mode
+  //   components are peeled in first), so the leading NB5 are the crudely-most-
+  //   null slice.  This is the cheap "first 30" truncation test; a principled
+  //   sigma-ordered rotation of psi_ccc would replace the slice, not the idea.
+  //   NB: a positive result is conservative (sigma-ordering can only help); a
+  //   negative one is inconclusive until the sigma-ordered NB5 is tried.
+  //
+  // Tensor depth deepens once more: CComplex for the L3→L4 level is
+  // iScalar<vTTTComplex>.  NB5 (the coarse dimension) is independent of the
+  // depth -- it just makes the coarsest site vector NB5-dimensional.
+  //////////////////////////////////////////////////////////////////////
+  const int NB5 = 30;   // compile-time: changing it re-instantiates the L4/L5 tensors
+  std::cout << GridLogMessage << "PARAM NB5 (truncated coarsest basis) = " << NB5 << std::endl;
+  assert(NB5 <= nbasis);
+
+  std::vector<CoarseCoarseCoarseVector> psi_ccc(nbasis, CoarseCoarseCoarse5d);
+  for (int k = 0; k < nbasis; k++)
+    AggregatesL3.ProjectToSubspace(psi_ccc[k], psi_cc[k]);   // raw psi_cc -> L4 null vectors
+
+  //////////////////////////////////////////////////////////////////////
+  // Optional sigma-ordering of psi_ccc (SVD_REORDER set): replace the crude
+  // first-NB5 slice with the NB5 genuinely-most-null directions of span(psi_ccc)
+  // under LinOpCCC.  For a NON-NORMAL operator the nullness measure is the
+  // singular value of A restricted to the span -- eig of Q†A†AQ -- NOT the
+  // numerical range Q†AQ (which non-normality contaminates).  Robust route:
+  // whiten by the Gram (drop near-dependent directions), Hermitian-eig the
+  // whitened A†A, rotate.  The printed singular spectrum IS the SVD study: where
+  // it falls off tells you the natural NB5, and the same numbers illuminate why
+  // the earlier singular-subspace deflation re-entered.  Safe here because we
+  // ORDER vectors that then feed a Galerkin projection, not REMOVE a subspace.
+  // Default (unset) leaves psi_ccc in raw order == the "first 30" test.
+  //////////////////////////////////////////////////////////////////////
+  if ( getenv("SVD_REORDER") ) {
+    std::cout << GridLogMessage << "SVD_REORDER: sigma-ordering psi_ccc under LinOpCCC" << std::endl;
+
+    Eigen::MatrixXcd G(nbasis,nbasis);                    // Gram  = Psi^dag Psi
+    for (int i=0;i<nbasis;i++)
+    for (int j=0;j<nbasis;j++)
+      G(i,j) = TensorRemove(innerProduct(psi_ccc[i],psi_ccc[j]));
+
+    std::vector<CoarseCoarseCoarseVector> Apsi(nbasis, CoarseCoarseCoarse5d);
+    for (int j=0;j<nbasis;j++) LinOpCCC.Op(psi_ccc[j], Apsi[j]);
+
+    Eigen::MatrixXcd M(nbasis,nbasis);                    // A^dagA = Psi^dag A^dag A Psi
+    for (int i=0;i<nbasis;i++)
+    for (int j=0;j<nbasis;j++)
+      M(i,j) = TensorRemove(innerProduct(Apsi[i],Apsi[j]));
+
+    // Whiten by the Gram: G = Ug diag(g) Ug^dag; keep g > tol*max; T = Ug diag(1/sqrt g).
+    // Q = Psi T is then orthonormal (Q^dag Q = T^dag G T = I).
+    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> esG(G);
+    Eigen::VectorXd g = esG.eigenvalues();               // ascending, real
+    RealD gmax = g(nbasis-1);
+    RealD gtol = 1.0e-9 * gmax;
+    int keep = 0; for (int i=0;i<nbasis;i++) if (g(i) > gtol) keep++;
+    std::cout << GridLogMessage << "  Gram spectrum: min=" << g(0) << " max=" << gmax
+              << " cond=" << gmax/std::max(g(0),1.0e-300) << " keep=" << keep << "/" << nbasis << std::endl;
+    assert(keep >= NB5);
+
+    Eigen::MatrixXcd T(nbasis, keep);                    // whitening (largest-g first)
+    { int c=0;
+      for (int i=nbasis-1;i>=0;i--) if (g(i) > gtol) { T.col(c) = esG.eigenvectors().col(i)/std::sqrt(g(i)); c++; }
+    }
+
+    Eigen::MatrixXcd Mw = T.adjoint() * M * T;           // whitened A^dagA (keep x keep, Hermitian)
+    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> esM(Mw);
+    Eigen::VectorXd s2 = esM.eigenvalues();              // ascending sigma^2 (most-null first)
+    std::cout << GridLogMessage << "  Singular spectrum sigma_k (most-null first):" << std::endl;
+    for (int k=0;k<keep;k++)
+      std::cout << GridLogMessage << "    sigma[" << k << "] = " << std::sqrt(std::max(s2(k),0.0)) << std::endl;
+
+    Eigen::MatrixXcd R = T * esM.eigenvectors();         // coeffs over Psi, sigma-ordered orthonormal dirs
+    std::vector<CoarseCoarseCoarseVector> phi(keep, CoarseCoarseCoarse5d);
+    for (int k=0;k<keep;k++) {
+      phi[k] = Zero();
+      for (int j=0;j<nbasis;j++)
+        phi[k] = phi[k] + ComplexD(R(j,k)) * psi_ccc[j];
+    }
+    for (int k=0;k<keep;k++) psi_ccc[k] = phi[k];        // psi_ccc[0..NB5-1] now = most-null dirs
+    std::cout << GridLogMessage << "SVD_REORDER: psi_ccc replaced by sigma-ordered directions" << std::endl;
+  }
+
+  typedef typename CoarseCoarseCoarseVector::vector_object              CoarseCoarseCoarseSiteObj;
+  typedef iScalar<vTTTComplex>                                          vTTTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5> LittleDiracOperatorL4;
+  typedef typename LittleDiracOperatorL4::CoarseVector                  CoarseCoarseCoarseCoarseVector;
+  typedef Aggregation<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5>       SubspaceL4;
+  typedef MGPreconditioner<CoarseCoarseCoarseSiteObj,vTTTTComplex,NB5>  L3to4MG;
+
+  SubspaceL4 AggregatesL4(CoarseCoarseCoarseCoarse5d, CoarseCoarseCoarse5d, cb);
+  for (int k = 0; k < NB5; k++)
+    AggregatesL4.subspace[k] = psi_ccc[k];         // FIRST NB5 raw promoted vectors (truncation)
+
+  NextToNearestStencilGeometry5D geom4(CoarseCoarseCoarseCoarse5d);
+  LittleDiracOperatorL4 LittleDiracOpL4(geom4, CoarseCoarseCoarse5d, CoarseCoarseCoarseCoarse5d);
+  LittleDiracOpL4.CoarsenOperator(LinOpCCC, AggregatesL4);   // block-GS's AggregatesL4.subspace in place
+
+  NonHermitianLinearOperator<LittleDiracOperatorL4,CoarseCoarseCoarseCoarseVector> LinOpCCCC(LittleDiracOpL4);
+  TrivialPrecon<CoarseCoarseCoarseCoarseVector> simpleCCCC;
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 5 bottom solve: GCR on a SHIFTED LinOpCCCC (the coarsest, most
+  // non-normal operator).  l5_shift slides its field of values off the origin;
+  // defaults to 0.0 (bare LinOpCCCC) until opted in.  This is the level a dense
+  // direct inverse would eventually replace: rank = NB5 * sites(clatt4).
+  //////////////////////////////////////////////////////////////////////
+  RealD l5_shift = 0.0;
+  if(getenv("l5_shift")) l5_shift = atof(getenv("l5_shift"));
+  std::cout << GridLogMessage << "PARAM l5_shift = " << l5_shift << std::endl;
+
+  ShiftedLinearOperator<CoarseCoarseCoarseCoarseVector> ShiftedLinOpCCCC(l5_shift, LinOpCCCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseCoarseVector> L5PGCR(1.0e-1,200,ShiftedLinOpCCCC,simpleCCCC,16,16);
+  L5PGCR.Level(5);
+  L5PGCR.Name("CCCCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3→4 V-cycle: depth-2 SHIFTED smoother on LinOpCCC + Level 5 bottom.
+  // Level 4 is no longer the bottom -- it is smoothed shallowly and recursed to
+  // Level 5, mirroring how Level 3 recurses to Level 4.
+  //////////////////////////////////////////////////////////////////////
+  RealD ccc_smoother_shift = 0.05;
+  int   ccc_smoother_nstep = 2;
+  if(getenv("ccc_smoother_shift")) ccc_smoother_shift = atof(getenv("ccc_smoother_shift"));
+  if(getenv("ccc_smoother_nstep")) ccc_smoother_nstep = atoi(getenv("ccc_smoother_nstep"));
+
+  ShiftedLinearOperator<CoarseCoarseCoarseVector> ShiftedLinOpCCC(ccc_smoother_shift, LinOpCCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector>
+    CoarseCoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCCC,simpleCCC,ccc_smoother_nstep,ccc_smoother_nstep);
+  CoarseCoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
+  CoarseCoarseCoarseSmootherGCR.Level(4);
+  CoarseCoarseCoarseSmootherGCR.Name("CCCsmoother");
+
+  L3to4MG L3to4Precon(AggregatesL4,
+                      LinOpCCC,
+                      simpleCCC,                     // no pre-smoother
+                      CoarseCoarseCoarseSmootherGCR, // post-smoother: depth-2 shifted GCR
+                      LinOpCCCC,
+                      L5PGCR,
+                      simpleCCCC);                   // trivial guesser at the bottom
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 4 (coarse-coarse-coarse) solve: GCR preconditioned by the L3→L4 V-cycle.
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseCoarseVector> L4MGsolver(1.0e-1,200,LinOpCCC,L3to4Precon,16,16);
+  L4MGsolver.Level(4);
+  L4MGsolver.Name("CCCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 2→3 V-cycle: depth-2 SHIFTED smoother on LinOpCC + Level 4 solve.
+  // The shift slides the coarse-coarse field of values off the origin so a
+  // 2-step smoother has something to bite on a non-normal operator (IRS idea).
+  //////////////////////////////////////////////////////////////////////
+  RealD cc_smoother_shift = 0.01;
+  int   cc_smoother_nstep = 2;
+  if(getenv("cc_smoother_shift")) cc_smoother_shift = atof(getenv("cc_smoother_shift"));
+  if(getenv("cc_smoother_nstep")) cc_smoother_nstep = atoi(getenv("cc_smoother_nstep"));
+
+  ShiftedLinearOperator<CoarseCoarseVector> ShiftedLinOpCC(cc_smoother_shift, LinOpCC);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    CoarseCoarseSmootherGCR(0.01,1,ShiftedLinOpCC,simpleCC,cc_smoother_nstep,cc_smoother_nstep);
+  CoarseCoarseSmootherGCR.SetZeroGuess(1);  // smoother slot: caller zeroes guess
+  CoarseCoarseSmootherGCR.Level(3);
+  CoarseCoarseSmootherGCR.Name("CCsmoother");
+
+  L2to3MG L2to3Precon(AggregatesL3,
+                      LinOpCC,
+                      simpleCC,                  // no pre-smoother
+                      CoarseCoarseSmootherGCR,   // post-smoother: depth-2 shifted GCR
+                      LinOpCCC,
+                      L4MGsolver,                // coarse solve is now the L3→L4 V-cycle
+                      simpleCCC);                // trivial guesser
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 3 (coarse-coarse) solve: GCR preconditioned by the L2→L3 V-cycle.
+  // Replaces the plain L3PGCR of the 3-level build -- the coarse-coarse level
+  // is now smoothed shallowly and recursed rather than solved deeply.
+  //////////////////////////////////////////////////////////////////////
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector> L3MGsolver(1.0e-1,200,LinOpCC,L2to3Precon,16,16);
+  L3MGsolver.Level(3);
+  L3MGsolver.Name("CCouter");
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarse-level GCR smoother for Level 1→2 V-cycle.
+  // Mirrors fine-grid SmootherGCR: shifted operator + fixed step count.
+  // coarse_smoother_shift and coarse_smoother_nstep are the tuning knobs.
+  //////////////////////////////////////////////////////////////////////
+  RealD coarse_smoother_shift = 0.01;
+  int   coarse_smoother_nstep = 2;   // depth-2 smoother on the coarse level
+  if(getenv("coarse_smoother_shift")) coarse_smoother_shift = atof(getenv("coarse_smoother_shift"));
+  if(getenv("coarse_smoother_nstep")) coarse_smoother_nstep = atoi(getenv("coarse_smoother_nstep"));
+     
+  ShiftedLinearOperator<CoarseVector> ShiftedLinOpCoarse(coarse_smoother_shift, LinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> CoarseSmootherGCR(0.01,1,ShiftedLinOpCoarse,simpleC,coarse_smoother_nstep,coarse_smoother_nstep);
+  CoarseSmootherGCR.SetZeroGuess(1);   // smoother slot: caller zeroes guess
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+
+  //////////////////////////////////////////////////////////////////////
+  // Level 1→2 V-cycle preconditioner.
+  //////////////////////////////////////////////////////////////////////
+  L1to2MG L1to2Precon(AggregatesL2,
+                       LinOpCoarse,
+                       simpleC,            // no pre-smoother (matches fine-grid setup)
+                       CoarseSmootherGCR,  // post-smoother: depth-2 shifted GCR
+                       LinOpCC,
+                       L3MGsolver,         // coarse-coarse solve is now the L2→L3 V-cycle
+                       CCDeflGuesser);     // Lüscher guesser: psi_cc C^{-1} psi_cc†
+
+  //////////////////////////////////////////////////////////////////////
+  // Standalone Level 1 two-level solve test.
+  // Compare against plain PGCR baseline above.
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Level 1 solve: two-level MG preconditioned PGCR"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector> L2MGsolver(3.0e-2,200,LinOpCoarse,L1to2Precon,16,16);
+  L2MGsolver.Level(2);
+  L2MGsolver.Name("Couter");
+  c_res = Zero();
+  L2MGsolver(c_src,c_res);
+
+  std::cout << GridLogMessage << "Level 1 two-level test: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+
+  //////////////////////////////////////////////////////////////////////
+  // Full five-level outer solve
+  //////////////////////////////////////////////////////////////////////
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+  std::cout<<GridLogMessage<<" Five-level outer solve"<<std::endl;
+  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> SmootherGCR(0.01,1,ShiftedPVdagM,simple_fine,16,16);
+  SmootherGCR.SetZeroGuess(1);         // pre/post smoother slots zero their guess
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+
+  f_src = one;
+
+  // Pre-smoother: none (TrivialPrecon); post-smoother: shifted PGCR.
+  // Coarse solver: L2MGsolver (PGCR preconditioned by Level 1→2 V-cycle).
+  TwoLevelMG ThreeLevelPrecon(AggregatesPD,
+                               PVdagM,
+                               simple_fine,
+                               SmootherGCR,
+                               LinOpCoarse,
+                               L2MGsolver,
+                               simpleC);
+
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermion> L1PGCR(1.0e-8,1000,PVdagM,ThreeLevelPrecon,16,16);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+
+  f_res = Zero();
+  L1PGCR(f_src,f_res);
+
+  std::cout << GridLogMessage << "Five-level outer solve: PVdagM operator uses:" << std::endl;
+  PVdagM.getApplications();
+  PVdagM.clear();
+}
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  const int Ls   = 24;
+  RealD     M5   = 1.8;
+  RealD     b    = 1.5;
+  RealD     c    = 0.5;
+  RealD     mass = 0.00078;
+  if ( getenv("MASS") ) mass = atof(getenv("MASS"));
+
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Mass: " << mass << ", Ls: " << Ls << ", b=" << b << ", c=" << c << std::endl;
+  std::cout << GridLogMessage << "nbasis: " << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 coarse grid: block 2^4 from fine (48×48×48×96 → 24×24×24×48, Ls=1)
+  Coordinate clatt = lat_size;
+  for (int d = 0; d < 4; d++) clatt[d] /= 2;
+  std::cout << GridLogMessage << "Level 1 coarse lattice: " << clatt << std::endl;
+
+  GridCartesian *Coarse4d  = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d  = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  // Level 2 coarse-coarse grid: block 2,2,3,3 from Level 1 (24×24×24×48 → 12×12×8×16, Ls=1).
+  // MPI geometry 3.6.4.4 (288 ranks): fine local {16,8,12,24}.
+  // Level 1 local {8,4,6,12}; Level 2 local {4,2,2,4}.
+  // z blocked by 3: z-Level1-local=6; 6/3=2 (even), 6/2=3 (odd) → must use 3.
+  // t blocked by 3: t-Level1-local=12; 12/3=4 divisible by Nsimd=4 (gen-simd-width=64).
+  //   t-block=2 gives t2-local=6, 6 mod 4 ≠ 0, fails Grid SIMD assertion. ✓
+  // With {4,2,2,4}: Nsimd=4 goes into x or t (both =4). ✓
+  Coordinate clatt2 = clatt;
+  clatt2[0] /= 2;
+  clatt2[1] /= 2;
+  clatt2[2] /= 3;
+  clatt2[3] /= 3;
+  std::cout << GridLogMessage << "Level 2 coarse-coarse lattice: " << clatt2 << std::endl;
+
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt2, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // Level 3 coarse-coarse-coarse grid: block clatt2 = {12,12,8,16} -> {6,12,8,8}.
+  // GEOMETRY (mpi 3.6.4.4, Nsimd=4 => SIMD layout {1,1,2,2}, factor 2 on z and t):
+  //   every grid needs z-local and t-local EVEN.  clatt2-local is {4,2,2,4}, so
+  //   z-local=2 is already at its minimum even value and CANNOT be blocked (2->1
+  //   is odd and trips the SIMD assertion); y-local=2 would go to 1 (degenerate).
+  //   Only x and t have room, so block {2,1,1,2}: clatt3 {6,12,8,8}, L4-local
+  //   {2,2,2,2} -- all dims even and >=2.  z stays unblocked by construction.
+  Coordinate clatt3 = clatt2;
+  clatt3[0] /= 2;   // x: 12 -> 6   (x-local 4 -> 2)
+  // clatt3[1] (y) unblocked: y-local 2 -> blocking gives 1 (degenerate)
+  // clatt3[2] (z) unblocked: z-local 2 is SIMD-pinned even, cannot halve
+  clatt3[3] /= 2;   // t: 16 -> 8   (t-local 4 -> 2)
+  std::cout << GridLogMessage << "Level 3 coarse-coarse-coarse lattice: " << clatt3 << std::endl;
+
+  GridCartesian *CoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt3, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarse4d);
+
+  // Level 4 coarse^4 grid: block clatt3 = {6,12,8,8} -> {3,6,8,8} via {2,2,1,1}.
+  //   mpi 3.6.4.4 => clatt4-local {1,1,2,2}: z-local=2, t-local=2 stay EVEN (SIMD
+  //   factor 2 pins them), so z,t are unblocked; x,y (SIMD factor 1) halve to
+  //   local 1 -- fully distributed but legal for the halo-depth-1 NextToNearest
+  //   stencil.  1152 sites; with NB5=30 that is the 34,560-rank coarsest operator
+  //   a dense direct inverse would target.
+  Coordinate clatt4 = clatt3;
+  clatt4[0] /= 2;   // x: 6  -> 3   (x-local 2 -> 1)
+  clatt4[1] /= 2;   // y: 12 -> 6   (y-local 2 -> 1)
+  // clatt4[2] (z) unblocked: z-local 2 is SIMD-pinned even
+  // clatt4[3] (t) unblocked: t-local 2 is SIMD-pinned even
+  std::cout << GridLogMessage << "Level 4 coarse^4 lattice: " << clatt4 << std::endl;
+
+  GridCartesian *CoarseCoarseCoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(clatt4, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarseCoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarseCoarseCoarse4d);
+
+  std::vector<int> seeds4({1,2,3,4});
+  std::vector<int> seeds5({5,6,7,8});
+  GridParallelRNG  RNG5(FGrid);  RNG5.SeedFixedIntegers(seeds5);
+  GridParallelRNG  RNG4(UGrid);  RNG4.SeedFixedIntegers(seeds4);
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  RealD b_ = 1.5;
+  RealD c_ = 0.5;
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b_,c_);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b_,c_);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+  typedef MGPreconditioner<vSpinColourVector,vTComplex,nbasis>        TwoLevelMG;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(0.01,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  // Subspace cache: save after generation, reload on subsequent runs to skip expensive setup.
+  // Set SUBSPACE_FILE to override the default path.
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  // Check if subspace file exists (boss rank checks, result broadcast via GlobalSum).
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+    // Insurance: GLOBAL (whole-lattice) orthonormalise, in case the cached file
+    // predates the GlobalOrthonormalise() that CreateSubspaceGCR now applies
+    // (Aggregates.h:196).  It is span-preserving and makes the vectors globally
+    // orthonormal -- it is NOT the block Orthogonalise() below, so it does NOT
+    // cause the psi_coarse->e_k trap.  It also (re)establishes the weak nullness
+    // gradient (shared most-null components peeled into the early indices) that
+    // the "first NB5" truncation relies on.  Idempotent if the file was already
+    // globally orthonormal.  The RAW subspace copy in runMG happens AFTER this
+    // call, so the raw-null (pre-block-GS) discipline is preserved.
+    AggregatesGCR.GlobalOrthonormalise();
+    // DO NOT block-orthogonalise here: runMG copies subspace[] as the RAW
+    // (pre-block-GS) basis and CoarsenOperator block-GS's it in place later.
+    // Orthogonalising now defeats the raw-null discipline (psi_coarse -> e_k)
+    // and poisons L2/L3/L4.  See project_block_orthogonalise_leak.
+    // AggregatesGCR.Orthogonalise();
+    std::cout << GridLogMessage << "Subspace loaded, globally orthonormalised (raw block basis preserved)." << std::endl;
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    std::cout << GridLogMessage << "Subspace generation: PVdagM operator uses:" << std::endl;
+    PVdagM.getApplications();
+    PVdagM.clear();
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+    std::cout << GridLogMessage << "Subspace saved to: " << subspace_file << std::endl;
+  }
+
+  runMG<nbasis,PVdagM_t,ShiftedPVdagM_t,Subspace,LittleDiracOperator,CoarseVector,TwoLevelMG>(
+    FGrid,
+    Coarse5d,
+    CoarseCoarse5d,
+    CoarseCoarseCoarse5d,
+    CoarseCoarseCoarseCoarse5d,
+    geom,
+    PVdagM,
+    ShiftedPVdagM,
+    AggregatesGCR
+  );
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_census.cc

@@ -0,0 +1,717 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_census.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// Spectral census of the coarsened PVdagM operator A_c.
+//
+// Measures the three sets that discriminate between the candidate explanations
+// for slow coarse-grid Krylov convergence:
+//
+//   0. Fine Ritz diagonal of RAW subspace vectors (pre-block-orthog).
+//      NB CoarsenOperator block-orthogonalises subspace[] IN PLACE; all
+//      nulliness/deflation bases must be built from a raw copy.
+//   1. Adjoint correctness check    <y|A x> == <A^dag y|x>      (fail-fast)
+//   2. Raw-vector coarse images vs A_c: RQ (must equal CENSUS 0 by Galerkin),
+//      ||A_c psi_c||/||psi_c||, and representability error
+//   3. sigma_max^2 = lambda_max(A_c^dag A_c)  via power method
+//   4. Low singular values          Chebyshev-filtered IRL on A_c^dag A_c
+//      -> sigma_min census = pseudospectrum of A_c evaluated at the origin
+//   5. Half-plane margin            lambda_min/max of H = (A_c + A_c^dag)/2
+//      -> min Re W(A_c);  positive-real check (Eisenstat-Elman-Schultz bound)
+//
+// Interpretation:
+//   sigma_min ~ min|lambda|, ~nbasis tiny then gap : effectively normal, bipartite
+//   sigma_min ~ min|lambda|, dense low tail        : normal but rank-starved
+//   sigma_min << min|lambda|                       : non-normal near origin
+//   lambda_min(H) < 0                              : half-plane condition violated
+//
+// Requires the dagger code path in GeneralCoarsenedMatrix:
+//   _Adag allocated, PopulateAdag active, _Adag exchanged, hermitian=0.
+//
+// Env vars:
+//   MASS               fermion mass                     (default 0.00078)
+//   SUBSPACE_FILE      subspace cache path
+//   CoarseSolverShift  shift baked into coarsening      (default 0.0: pure Galerkin)
+//   CENSUS_NSTOP       converged low modes wanted       (default 60)
+//   CENSUS_NK          Lanczos Nk                       (default 96)
+//   CENSUS_NM          Lanczos Nm                       (default 192)
+//   CENSUS_TOL         Lanczos residual                 (default 1e-5)
+//   CENSUS_MAXIT       Lanczos max restarts             (default 50)
+//   CHEBY_LO           filter low edge in sigma^2       (default 4.0)
+//   CHEBY_HI           filter high edge; 0 = auto from power method x1.1
+//   CHEBY_ORDER        filter order                     (default 401)
+//                      filter gain at 0 ~ cosh(order*2*sqrt(lo/hi)); with
+//                      hi~2200, lo=4, order=401 => gain ~ 1e14. lo=0.01 at
+//                      order 201 gives gain ~1.4 (stagnation).
+
+#include <Grid/Grid.h>
+#include <Grid/Grid_Eigen_Dense.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+
+using namespace std;
+using namespace Grid;
+
+RealD mass              = 0.00078;
+RealD CoarseSolverShift = 0.0;
+int   CensusNstop       = 60;
+int   CensusNk          = 96;
+int   CensusNm          = 192;
+RealD CensusTol         = 1.0e-5;
+int   CensusMaxIt       = 50;
+RealD ChebyLo           = 4.0;   // sigma^2 cutoff: amplifies sigma < 2. Filter gain ~ cosh(order*2*sqrt(lo/hi))
+RealD ChebyHi           = 0.0;   // 0 => auto: 1.1 * power-method sigma_max^2
+int   ChebyOrder        = 401;
+RealD CGdeflTol         = 1.0e-8; // CENSUS 6 deflated-CG tolerance
+int   CGdeflMaxIt       = 4000;   // CENSUS 6 deflated-CG max iterations
+int   DeflRank          = 0;      // CENSUS 6 deflation rank; 0 => all available per basis
+
+void ParseEnvironment(void)
+{
+  if(getenv("MASS"))              mass              = atof(getenv("MASS"));
+  if(getenv("CoarseSolverShift")) CoarseSolverShift = atof(getenv("CoarseSolverShift"));
+  if(getenv("CENSUS_NSTOP"))      CensusNstop       = atoi(getenv("CENSUS_NSTOP"));
+  if(getenv("CENSUS_NK"))         CensusNk          = atoi(getenv("CENSUS_NK"));
+  if(getenv("CENSUS_NM"))         CensusNm          = atoi(getenv("CENSUS_NM"));
+  if(getenv("CENSUS_TOL"))        CensusTol         = atof(getenv("CENSUS_TOL"));
+  if(getenv("CENSUS_MAXIT"))      CensusMaxIt       = atoi(getenv("CENSUS_MAXIT"));
+  if(getenv("CHEBY_LO"))          ChebyLo           = atof(getenv("CHEBY_LO"));
+  if(getenv("CHEBY_HI"))          ChebyHi           = atof(getenv("CHEBY_HI"));
+  if(getenv("CHEBY_ORDER"))       ChebyOrder        = atoi(getenv("CHEBY_ORDER"));
+  if(getenv("CGDEFL_TOL"))        CGdeflTol         = atof(getenv("CGDEFL_TOL"));
+  if(getenv("CGDEFL_MAXIT"))      CGdeflMaxIt       = atoi(getenv("CGDEFL_MAXIT"));
+  if(getenv("DEFL_RANK"))         DeflRank          = atoi(getenv("DEFL_RANK"));
+
+  std::cout << GridLogMessage << "PARAM: MASS              " << mass              << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverShift " << CoarseSolverShift << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_NSTOP      " << CensusNstop       << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_NK         " << CensusNk          << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_NM         " << CensusNm          << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_TOL        " << CensusTol         << std::endl;
+  std::cout << GridLogMessage << "PARAM: CENSUS_MAXIT      " << CensusMaxIt       << std::endl;
+  std::cout << GridLogMessage << "PARAM: CHEBY_LO          " << ChebyLo           << std::endl;
+  std::cout << GridLogMessage << "PARAM: CHEBY_HI          " << ChebyHi           << std::endl;
+  std::cout << GridLogMessage << "PARAM: CHEBY_ORDER       " << ChebyOrder        << std::endl;
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  std::cout << Grid::GridLogMessage << "Saving subspace (" << subspace.size() << " vectors) to: " << fname << std::endl;
+  Grid::emptyUserRecord record;
+  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+  SW.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++)
+    SW.writeScidacFieldRecord(subspace[k], record);
+  SW.close();
+#endif
+}
+
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  std::cout << Grid::GridLogMessage << "Loading subspace (" << subspace.size() << " vectors) from: " << fname << std::endl;
+  Grid::emptyUserRecord record;
+  Grid::ScidacReader SR;
+  SR.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++)
+    SR.readScidacFieldRecord(subspace[k], record);
+  SR.close();
+#endif
+}
+
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  RealD shift;
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+    out = out + shift * in;
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(tmp,out);
+    _Mat.Mdag(in,tmp);
+    out = out + shift * in;
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+// H = (A + A^dag)/2 : Hermitian part of the coarse operator.
+// lambda_min(H) = min Re W(A) is the half-plane margin; the EES GCR
+// convergence theorem requires it positive.
+template<class Matrix,class Field>
+class HermitianPartOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+public:
+  HermitianPartOperator(Matrix &Mat): _Mat(Mat) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,out);
+    _Mat.Mdag(in,tmp);
+    out = 0.5*(out + tmp);
+  }
+};
+
+// s*I - Op : power method on this gives s - lambda_min(Op) for Hermitian Op.
+template<class Field>
+class ShiftedNegatedOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD s;
+public:
+  ShiftedNegatedOperator(RealD _s, LinearOperatorBase<Field> &Op): _Op(Op), s(_s) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    _Op.HermOp(in,out);
+    out = s*in - out;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+  ParseEnvironment();
+
+  const int Ls=24;
+  RealD M5=1.8;
+  RealD b=1.5;
+  RealD c=0.5;
+  const int nbasis = 60;
+
+  std::cout << GridLogMessage << "Census of coarse PVdagM: mass=" << mass << " Ls=" << Ls << " nbasis=" << nbasis << std::endl;
+
+  std::vector<int> lat_size {48, 48, 48, 96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Blocking: default matches Example_pvdagm.cc; override with e.g. BLOCK=2.2.2.2
+  Coordinate clatt = lat_size;
+  Coordinate Block({4,4,6,4});
+  if ( getenv("BLOCK") ) {
+    GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block);
+    GRID_ASSERT(Block.size()==4);
+  }
+  for(int d=0;d<clatt.size();d++){
+    GRID_ASSERT(lat_size[d] % Block[d] == 0);
+    clatt[d] = lat_size[d]/Block[d];
+  }
+  std::cout << GridLogMessage << "Block " << Block << "  coarse lattice " << clatt << std::endl;
+
+  GridCartesian *Coarse4d =  SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d =  SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+
+  GridParallelRNG RNG5(FGrid);    RNG5.SeedFixedIntegers({5,6,7,8});
+  GridParallelRNG RNG4(UGrid);    RNG4.SeedFixedIntegers({1,2,3,4});
+  GridParallelRNG CRNG(Coarse5d); CRNG.SeedFixedIntegers({5,6,7,8});
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  LittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                           CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>             Subspace;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(CoarseSolverShift,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom(Coarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // Subspace: load from cache or generate
+  //////////////////////////////////////////////////////////////////////
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                              + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+
+  uint64_t file_exists = 0;
+  if ( UGrid->IsBoss() ) {
+    std::ifstream f(subspace_file);
+    file_exists = f.good() ? 1 : 0;
+  }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb = 0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+
+  if ( file_exists ) {
+    std::cout << GridLogMessage << "*** Loading subspace from disk ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Keep the RAW (pre-block-orthogonalisation) near-null vectors.
+  // CoarsenOperator block-orthogonalises subspace[] IN PLACE, after which
+  // subspace[k] is the orthonormal basis phi_k and Project(phi_k) = e_k,
+  // the block-constant unit vector -- NOT a near-null direction.
+  // All nulliness measurements and any deflation basis must use raw[].
+  //////////////////////////////////////////////////////////////////////
+  std::vector<LatticeFermionD> raw(nbasis,FGrid);
+  for(int k=0;k<nbasis;k++) raw[k] = AggregatesGCR.subspace[k];
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 0: fine-grid Ritz diagonal on the loaded/generated raw vectors.
+  // Expect Re <psi|A|psi>/<psi|psi> ~ the nulliness achieved at generation
+  // (~2e-3). O(0.1-10) values mean the cache holds orthogonalised vectors
+  // and must be regenerated.
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 0: fine Ritz diagonal of raw subspace vectors" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  {
+    LatticeFermionD Ap(FGrid);
+    for(int k=0;k<nbasis;k++){
+      PVdagM.Op(raw[k],Ap);
+      RealD    n2psi = norm2(raw[k]);
+      ComplexD rq    = innerProduct(raw[k],Ap)/n2psi;
+      std::cout << GridLogMessage << "CENSUS: raw[" << k << "]  <psi|A|psi>/<psi|psi> = " << rq
+                << "   ||A psi||/||psi|| = " << std::sqrt(norm2(Ap)/n2psi) << std::endl;
+    }
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarsen. hermitian=0 is REQUIRED: enables PopulateAdag so that
+  // Mdag applies A^dag rather than silently aliasing to A.
+  //////////////////////////////////////////////////////////////////////
+  LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d,0);
+  if ( CoarseSolverShift != 0.0 ) {
+    std::cout << GridLogMessage << "Coarsening SHIFTED operator, shift=" << CoarseSolverShift << std::endl;
+    LittleDiracOpPV.CoarsenOperator(ShiftedPVdagM, AggregatesGCR);
+  } else {
+    std::cout << GridLogMessage << "Coarsening pure Galerkin operator (no shift)" << std::endl;
+    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);
+  }
+
+  CoarseVector c_x(Coarse5d);
+  CoarseVector c_y(Coarse5d);
+  CoarseVector c_t1(Coarse5d);
+  CoarseVector c_t2(Coarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 1: adjoint correctness (fail fast)
+  //   <y|A x> == <A^dag y|x>  for random x,y
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 1: adjoint correctness of dagger code path" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  random(CRNG,c_x);
+  random(CRNG,c_y);
+  LittleDiracOpPV.M   (c_x,c_t1);       // A x
+  LittleDiracOpPV.Mdag(c_y,c_t2);       // A^dag y
+  ComplexD ip1 = innerProduct(c_y,c_t1);   // <y|A x>
+  ComplexD ip2 = innerProduct(c_t2,c_x);   // <A^dag y|x>
+  RealD reldiff = abs(ip1-ip2)/abs(ip1);
+  std::cout << GridLogMessage << "CENSUS: <y|Ax>       = " << ip1 << std::endl;
+  std::cout << GridLogMessage << "CENSUS: <Adag y|x>   = " << ip2 << std::endl;
+  std::cout << GridLogMessage << "CENSUS: rel diff     = " << reldiff << "  (expect ~1e-14; FAIL if O(1))" << std::endl;
+  GRID_ASSERT(reldiff < 1.0e-8);
+
+  // Coarse near-null ("global") vectors psi_c[k] = P^dag raw[k], stored for the
+  // Ritz-matrix + deflation study in CENSUS 6 (filled in CENSUS 2's projection
+  // loop below, before raw[]/subspace[] are freed).
+  std::vector<CoarseVector> psi_c(nbasis,Coarse5d);
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 2: nulliness of the RAW vectors' coarse images against A_c.
+  //   psi_c[k] = P^dag raw[k].  Galerkin guarantees the Rayleigh quotient
+  //   equals CENSUS 0's fine value exactly (raw[k] is in span of its own
+  //   chopped pieces) -- agreement is a machine-precision validation of
+  //   the coarsening.  ||A_c psi_c||/||psi_c|| is the sigma-relevant norm.
+  //   The representability column ||raw - P psi_c||/||raw|| must be ~eps.
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 2: raw-vector coarse images against coarse operator" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  {
+    LatticeFermionD back(FGrid);
+    for(int k=0;k<nbasis;k++){
+      AggregatesGCR.ProjectToSubspace(c_x, raw[k]);
+      psi_c[k] = c_x;                       // store coarse near-null vector for CENSUS 6
+      AggregatesGCR.PromoteFromSubspace(c_x, back);
+      back = back - raw[k];
+      RealD represent = std::sqrt(norm2(back)/norm2(raw[k]));
+      LittleDiracOpPV.M(c_x, c_t1);
+      RealD    n2psi = norm2(c_x);
+      RealD    n2Apsi= norm2(c_t1);
+      ComplexD rq    = innerProduct(c_x,c_t1) / n2psi;
+      std::cout << GridLogMessage << "CENSUS: psi_c[" << k << "]  <psi|A|psi>/<psi|psi> = " << rq
+                << "   ||A psi||/||psi|| = " << std::sqrt(n2Apsi/n2psi)
+                << "   represent_err = " << represent << std::endl;
+    }
+  }
+
+  // Fine subspace + raw copy are needed only through CENSUS 2; CENSUS 3-5 are
+  // entirely coarse (LittleDiracOpPV only), and the CENSUS 4 evec save writes the
+  // coarse vectors directly.  Release the ~2*nbasis fine 5D fields (~14 GB/GCD at
+  // 2^4) HERE, before the order-ChebyOrder Lanczos whose padded coarse temporaries
+  // otherwise push host memory over the top on top of _A + _Adag (the AccCache
+  // CpuPtr!=NULL abort seen mid-iteration).
+  // Direct orthonormality check of the fine near-null vectors (GlobalOrthonormalise
+  // in CreateSubspaceGCR).  raw is freed just below, so this runs here, not CENSUS 6.
+  // If this is ~0 but the coarse Gram S (CENSUS 6) is not, the gap is representability,
+  // not orthonormality.
+  {
+    Eigen::MatrixXcd Gfine(nbasis,nbasis);
+    for(int i=0;i<nbasis;i++){
+      for(int j=i;j<nbasis;j++){
+        ComplexD g = innerProduct(raw[i],raw[j]);
+        Gfine(i,j) = std::complex<double>(g.real(),g.imag());
+        Gfine(j,i) = std::conj(Gfine(i,j));
+      }
+    }
+    double GmI = (Gfine - Eigen::MatrixXcd::Identity(nbasis,nbasis)).norm();
+    std::cout << GridLogMessage << "CENSUS 2b: fine Gram ||<raw_i|raw_j> - I||_F = " << GmI
+              << "   (expect ~0 if fine vectors orthonormal)" << std::endl;
+  }
+
+  raw.clear();                    raw.shrink_to_fit();
+  AggregatesGCR.subspace.clear(); AggregatesGCR.subspace.shrink_to_fit();
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 3: sigma_max^2 = lambda_max( A_c^dag A_c ) by power method
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 3: power method for sigma_max" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  MdagMLinearOperator<LittleDiracOperator,CoarseVector> HermOpAdagA(LittleDiracOpPV);
+  random(CRNG,c_x);
+  PowerMethod<CoarseVector> PM;
+  RealD sigmax2 = PM(HermOpAdagA,c_x);
+  std::cout << GridLogMessage << "CENSUS: lambda_max(AdagA) = " << sigmax2
+            << "   sigma_max = " << std::sqrt(sigmax2) << std::endl;
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 4: low singular values via Chebyshev-filtered IRL on A^dag A
+  //   The low end of sigma(A_c) is the pseudospectrum of A_c at z=0.
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 4: Chebyshev-filtered Lanczos, low sigma^2" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  RealD cheby_hi = (ChebyHi > 0.0) ? ChebyHi : 1.1*sigmax2;
+  std::cout << GridLogMessage << "Chebyshev filter [" << ChebyLo << "," << cheby_hi << "] order " << ChebyOrder << std::endl;
+
+  // eval/evec/Nconv hoisted out of the block so CENSUS 6 can deflate with them.
+  std::vector<RealD>        eval(CensusNm);
+  std::vector<CoarseVector> evec(CensusNm,Coarse5d);
+  int Nconv=0;
+  {
+    Chebyshev<CoarseVector>      Cheby(ChebyLo,cheby_hi,ChebyOrder);
+    FunctionHermOp<CoarseVector> OpCheby(Cheby,HermOpAdagA);
+    PlainHermOp<CoarseVector>    Op     (HermOpAdagA);
+
+    ImplicitlyRestartedLanczos<CoarseVector> IRL(OpCheby,Op,CensusNstop,CensusNk,CensusNm,CensusTol,CensusMaxIt);
+
+    random(CRNG,c_x);
+    IRL.calc(eval,evec,c_x,Nconv);
+
+    std::cout << GridLogMessage << "CENSUS: converged " << Nconv << " modes of AdagA" << std::endl;
+    for(int i=0;i<Nconv;i++){
+      std::cout << GridLogMessage << "CENSUS: sigma[" << i << "]^2 = " << eval[i]
+                << "   sigma = " << std::sqrt(std::max(eval[i],0.0)) << std::endl;
+    }
+
+    // Optionally persist the low right-singular-vector basis: this IS the
+    // deflation basis for the coarse solve (ADEF1 / MultiRHSDeflation).
+    // Set CENSUS_EVEC_FILE to enable.
+    if ( getenv("CENSUS_EVEC_FILE") && Nconv>0 ) {
+#ifdef HAVE_LIME
+      std::string evec_file(getenv("CENSUS_EVEC_FILE"));
+      std::string eval_file = evec_file + ".evals.xml";
+      std::cout << GridLogMessage << "CENSUS: saving " << Nconv << " singular vectors to " << evec_file << std::endl;
+      emptyUserRecord record;
+      ScidacWriter WR(evec[0].Grid()->IsBoss());
+      WR.open(evec_file);
+      for(int i=0;i<Nconv;i++) WR.writeScidacFieldRecord(evec[i],record);
+      WR.close();
+      XmlWriter WRx(eval_file);
+      std::vector<RealD> eval_out(eval.begin(),eval.begin()+Nconv); // don't shrink shared eval
+      write(WRx,"evals",eval_out);
+#endif
+    }
+  }
+
+  // NB: evec/eval stay sized CensusNm (Lattice has no default ctor, so
+  // std::vector<CoarseVector>::resize won't instantiate).  They match in size,
+  // which is all DeflatedGuesser asserts; CENSUS 6 only ever indexes [0,Nconv).
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 5: half-plane margin from the Hermitian part
+  //   lambda_min(H) = min Re W(A_c) > 0 <=> positive-real (EES applies)
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 5: Hermitian part H=(A+Adag)/2, half-plane margin" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  HermitianPartOperator<LittleDiracOperator,CoarseVector> HermPart(LittleDiracOpPV);
+
+  random(CRNG,c_x);
+  RealD lamHmax = PM(HermPart,c_x);
+  std::cout << GridLogMessage << "CENSUS: lambda_max(H) = " << lamHmax << std::endl;
+
+  // lambda_min(H): most-negative eigenvalue via Chebyshev-filtered IRL on H.
+  // A shifted power method cannot separate it from the dense low tail (which is
+  // why the earlier -0.006 is suspect); Cheby(lo, hi>=lambda_max) amplifies the
+  // most-negative mode hardest so IRL isolates the true bottom of the spectrum.
+  RealD hpLo    = getenv("HPLANE_CHEBY_LO")    ? atof(getenv("HPLANE_CHEBY_LO"))    : 0.1;
+  RealD hpHi    = getenv("HPLANE_CHEBY_HI")    ? atof(getenv("HPLANE_CHEBY_HI"))    : 1.1*lamHmax;
+  int   hpOrder = getenv("HPLANE_CHEBY_ORDER") ? atoi(getenv("HPLANE_CHEBY_ORDER")) : 61;
+  // Grid's Chebyshev filter MUST be odd order (positive for x < -1, where the low/
+  // negative modes map); an even order flips the sign there and the IRL blows up.
+  if(hpOrder%2==0){ hpOrder++;
+    std::cout<<GridLogMessage<<"HPLANE_CHEBY_ORDER forced odd -> "<<hpOrder<<std::endl; }
+  int   hpNstop = getenv("HPLANE_NSTOP")       ? atoi(getenv("HPLANE_NSTOP"))       : 8;
+  int   hpNk    = getenv("HPLANE_NK")          ? atoi(getenv("HPLANE_NK"))          : 24;
+  int   hpNm    = getenv("HPLANE_NM")          ? atoi(getenv("HPLANE_NM"))          : 48;
+  RealD hpTol   = getenv("HPLANE_TOL")         ? atof(getenv("HPLANE_TOL"))         : 1.0e-4;
+  int   hpMaxIt = getenv("HPLANE_MAXIT")       ? atoi(getenv("HPLANE_MAXIT"))       : 20;
+
+  Chebyshev<CoarseVector>      HCheby(hpLo,hpHi,hpOrder);
+  FunctionHermOp<CoarseVector> HOpCheby(HCheby,HermPart);
+  PlainHermOp<CoarseVector>    HOpPlain(HermPart);
+  ImplicitlyRestartedLanczos<CoarseVector> HIRL(HOpCheby,HOpPlain,hpNstop,hpNk,hpNm,hpTol,hpMaxIt);
+  std::vector<RealD>        heval(hpNm);
+  std::vector<CoarseVector> hevec(hpNm,Coarse5d);
+  int hNconv=0;
+  random(CRNG,c_x);
+  HIRL.calc(heval,hevec,c_x,hNconv);
+  RealD lamHmin = (hNconv>0) ? heval[0] : 9.99e99;
+  for(int kk=0;kk<hNconv;kk++) lamHmin = std::min(lamHmin, heval[kk]);
+  std::cout << GridLogMessage << "CENSUS: IRL H-bottom converged " << hNconv
+            << " eigenvalues; most-negative = " << lamHmin << std::endl;
+  std::cout << GridLogMessage << "CENSUS: lambda_min(H) = " << lamHmin
+            << "  (positive-real / half-plane margin; NEGATIVE => GCR unguaranteed)" << std::endl;
+
+  //////////////////////////////////////////////////////////////////////
+  // CENSUS 6: Ritz matrix of the coarse near-null basis + deflated-CG study
+  //
+  //   C_ij = <psi_c^i | A^dag A | psi_c^j>,   S_ij = <psi_c^i | psi_c^j>.
+  //   psi_c are NOT orthonormal (raw near-null projected to coarse), so the
+  //   Rayleigh-Ritz problem is the GENERALISED Hermitian one  C v = theta S v.
+  //   Its eigenpairs (theta_i, g_i = sum_j V(j,i) psi_c^j) are the best approximate
+  //   eigenpairs of A^dag A available from span{psi_c}; Eigen normalises so that
+  //   V^dag S V = I, hence <g_i|g_j> = delta_ij and the g_i are an orthonormal
+  //   DeflatedGuesser basis.  Compare theta_i to the Lanczos sigma_i^2, then run
+  //   three CG solves on A^dag A: [1] no deflation, [2] Lanczos-eigenvector
+  //   deflated guess, [3] Ritz global-vector deflated guess (g_i treated as pure
+  //   eigenvectors with eigenvalue theta_i).
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS 6: Ritz matrix C_ij = <psi_c^i|AdagA|psi_c^j> + deflated CG" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+
+  std::vector<CoarseVector> Apsi(nbasis,Coarse5d);
+  for(int j=0;j<nbasis;j++) HermOpAdagA.HermOp(psi_c[j],Apsi[j]);   // A^dag A psi_c^j
+
+  Eigen::MatrixXcd Cmat(nbasis,nbasis);
+  Eigen::MatrixXcd Smat(nbasis,nbasis);
+  for(int i=0;i<nbasis;i++){
+    for(int j=0;j<nbasis;j++){
+      ComplexD cij = innerProduct(psi_c[i],Apsi[j]);
+      ComplexD sij = innerProduct(psi_c[i],psi_c[j]);
+      Cmat(i,j) = std::complex<double>(cij.real(),cij.imag());
+      Smat(i,j) = std::complex<double>(sij.real(),sij.imag());
+    }
+  }
+
+  {
+    Eigen::SelfAdjointEigenSolver<Eigen::MatrixXcd> ses(Smat);
+    Eigen::MatrixXcd Id = Eigen::MatrixXcd::Identity(nbasis,nbasis);
+    double SmI = (Smat - Id).norm();   // ||S - I||_F : ~0 iff psi_c orthonormal
+    std::cout << GridLogMessage << "CENSUS 6: Gram S eig range [" << ses.eigenvalues()(0)
+              << ", " << ses.eigenvalues()(nbasis-1)
+              << "]   ||S - I||_F = " << SmI
+              << "   (expect ~0: fine vectors are GlobalOrthonormalise'd => psi_c orthonormal)" << std::endl;
+  }
+  Eigen::GeneralizedSelfAdjointEigenSolver<Eigen::MatrixXcd> ges(Cmat,Smat);
+  Eigen::VectorXd  theta = ges.eigenvalues();    // ascending, real
+  Eigen::MatrixXcd Vr    = ges.eigenvectors();   // columns; V^dag S V = I
+
+  int ncmp = std::min((int)nbasis,Nconv);
+  std::cout << GridLogMessage << "CENSUS 6: Ritz theta vs Lanczos sigma^2 (both ascending):" << std::endl;
+  for(int i=0;i<nbasis;i++){
+    if(i<ncmp)
+      std::cout << GridLogMessage << "CENSUS: theta["<<i<<"] = "<<theta(i)
+                <<"   sqrt = "<<std::sqrt(std::max(theta(i),0.0))
+                <<"   | sigma^2 = "<<eval[i]<<"   theta/sigma^2 = "<<theta(i)/eval[i] << std::endl;
+    else
+      std::cout << GridLogMessage << "CENSUS: theta["<<i<<"] = "<<theta(i)
+                <<"   sqrt = "<<std::sqrt(std::max(theta(i),0.0)) << std::endl;
+  }
+
+  // Ritz global vectors g_i = sum_j V(j,i) psi_c^j  (S-orthonormal), eigenvalue theta_i
+  std::vector<CoarseVector> gvec(nbasis,Coarse5d);
+  std::vector<RealD>        gval(nbasis);
+  for(int i=0;i<nbasis;i++){
+    gvec[i] = Zero();
+    for(int j=0;j<nbasis;j++){
+      ComplexD coeff(Vr(j,i).real(),Vr(j,i).imag());
+      axpy(gvec[i],coeff,psi_c[j],gvec[i]);
+    }
+    gval[i] = theta(i);
+  }
+
+  // How good are the diagonalised global vectors as actual eigenvectors of A^dag A?
+  {
+    CoarseVector Ag(Coarse5d), rr(Coarse5d);
+    int nchk = std::min((int)nbasis,16);
+    for(int i=0;i<nchk;i++){
+      HermOpAdagA.HermOp(gvec[i],Ag);
+      axpy(rr,-gval[i],gvec[i],Ag);          // rr = A^dag A g - theta g
+      RealD rn = std::sqrt(norm2(rr));
+      std::cout << GridLogMessage << "CENSUS 6: Ritz resid ["<<i<<"] ||AdagA g - theta g||/theta = "
+                << rn/std::max(gval[i],1.0e-30) << "   (theta="<<gval[i]<<")" << std::endl;
+    }
+  }
+
+  // --- Three CG solves on A^dag A, common random source ---
+  int rankLanc = (DeflRank>0) ? std::min(DeflRank,Nconv)       : Nconv;
+  int rankRitz = (DeflRank>0) ? std::min(DeflRank,(int)nbasis) : (int)nbasis;
+  std::cout << GridLogMessage << "CENSUS 6: CG tol "<<CGdeflTol<<" maxit "<<CGdeflMaxIt
+            << " ; deflation ranks -- Lanczos "<<rankLanc<<", Ritz "<<rankRitz << std::endl;
+
+  CoarseVector cg_src(Coarse5d);  random(CRNG,cg_src);
+  CoarseVector cg_x  (Coarse5d);
+  ConjugateGradient<CoarseVector> CGdefl(CGdeflTol,CGdeflMaxIt,false);
+
+  cg_x = Zero();
+  CGdefl(HermOpAdagA,cg_src,cg_x);
+  std::cout << GridLogMessage << "CENSUS 6: [1] no deflation          : iters = "
+            << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
+
+  if(rankLanc>0){
+    DeflatedGuesser<CoarseVector> guessL(evec,eval,rankLanc);
+    guessL(cg_src,cg_x);
+    CGdefl(HermOpAdagA,cg_src,cg_x);
+    std::cout << GridLogMessage << "CENSUS 6: [2] Lanczos-evec deflation : iters = "
+              << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
+  }
+
+  {
+    DeflatedGuesser<CoarseVector> guessR(gvec,gval,rankRitz);
+    guessR(cg_src,cg_x);
+    CGdefl(HermOpAdagA,cg_src,cg_x);
+    std::cout << GridLogMessage << "CENSUS 6: [3] Ritz-vector deflation  : iters = "
+              << CGdefl.IterationsToComplete << "   true_resid = " << CGdefl.TrueResidual << std::endl;
+  }
+
+  //////////////////////////////////////////////////////////////////////
+  // Summary
+  //////////////////////////////////////////////////////////////////////
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "CENSUS SUMMARY" << std::endl;
+  std::cout << GridLogMessage << "  sigma_max        = " << std::sqrt(sigmax2) << std::endl;
+  std::cout << GridLogMessage << "  lambda_max(H)    = " << lamHmax  << std::endl;
+  std::cout << GridLogMessage << "  lambda_min(H)    = " << lamHmin  << std::endl;
+  std::cout << GridLogMessage << "  low sigma census : see CENSUS 4 table above" << std::endl;
+  std::cout << GridLogMessage << "  Compare min sigma with |lambda| from Krylov-Schur (Patrick):" << std::endl;
+  std::cout << GridLogMessage << "    sigma_min ~ min|lambda| : effectively normal; deflation rank is the issue" << std::endl;
+  std::cout << GridLogMessage << "    sigma_min << min|lambda|: non-normal; need two-sided/singular-vector deflation" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_halfplane.cc

@@ -0,0 +1,278 @@
+/*
+ * Example_pvdagm_halfplane.cc
+ *
+ * Standalone fine-operator diagnostic: the EES half-plane margin of the
+ * (non-Hermitian) PV-preconditioned Mobius DWF operator
+ *
+ *     A(m_adj) = D_adj^dag D_light      (D_adj plays the Pauli-Villars role)
+ *
+ * as a function of the adjoint mass m_adj, dialled from the light quark mass
+ * up to the Pauli-Villars mass (=1).  No coarse grid, no subspace, no Lanczos
+ * -- pure power-method spectral tests on the fine grid.
+ *
+ * Purpose: A is the LEFT preconditioner for inverting the light operator.
+ * To solve  D_light X = B  we iterate the preconditioned system
+ *     (D_adj^dag D_light) X = D_adj^dag B ,
+ * whose solution X is independent of m_adj -- only the conditioning and the
+ * iterative convergence change.  m_adj = m_light is the usual CGNR (symmetric
+ * normal equations); m_adj = 1 is the Pauli-Villars preconditioned system.
+ * The sweep asks which m_adj keeps the preconditioned operator well-behaved
+ * (positive-real / EES-guaranteed) while buying the wider spectral range.
+ *
+ * For the Hermitian part  H(A) = (A + A^dag)/2  we measure, per m_adj:
+ *
+ *   lambda_max(H)   -- power method on H
+ *   lambda_min(H)   -- power method on (sI - H) => min Re W(A), the half-plane
+ *                      margin.  EES (Eisenstat-Elman-Schultz 1983, Thm 3.3)
+ *                      GUARANTEES GCR convergence with rate
+ *                          [ 1 - lambda_min(H)^2 / sigma_max^2 ]^{1/2}
+ *                      ONLY when lambda_min(H) > 0 (positive-real / A's field
+ *                      of values in the open right half-plane).  A negative
+ *                      value means the guarantee is lost (not that GCR
+ *                      diverges); the magnitude is then the distance-to-
+ *                      positive-realness, i.e. the shift/deflation needed to
+ *                      recover it.
+ *   sigma_max       -- power method on A^dag A (= A.HermOp)
+ *
+ * Endpoints:
+ *   m_adj = m_light  =>  A = M^dag M, Hermitian PD, positive-real by
+ *                        construction, lambda_min(H) = sigma_min^2 > 0 (the
+ *                        squared / CGNR operator).
+ *   m_adj = 1        =>  A = PV^dag M, the standard PVdagM operator.
+ *
+ * Env: MASS, M5, MOBIUS_B, MOBIUS_C, LS, CONFIG,
+ *      MADJ_LIST (comma separated) OR MADJ_MIN / MADJ_MAX / MADJ_N (geometric).
+ *
+ * Caveat: lambda_min(H) via a shifted power method can be soft when it sits
+ * near zero over a dense low spectrum.  The SIGN and the TREND across m_adj
+ * are the robust signal; confirm an individual near-zero value with a proper
+ * shifted Lanczos if it is load-bearing.
+ */
+
+#include <Grid/Grid.h>
+
+using namespace std;
+using namespace Grid;
+
+//////////////////////////////////////////////////////////////////////
+// A = PV^dag M : Op = _PV.Mdag . _Mat.M ,  AdjOp = _Mat.Mdag . _PV.M
+//////////////////////////////////////////////////////////////////////
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat;
+  Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _Mat.M(in,tmp);
+    _PV.Mdag(tmp,out);
+  }
+  void AdjOp     (const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _PV.M(in,tmp);
+    _Mat.Mdag(tmp,out);
+  }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){ // A^dag A
+    Field tmp(in.Grid());
+    Op(in,tmp);
+    AdjOp(tmp,out);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// H = (A + A^dag)/2 for a general non-Hermitian LinearOperator A.
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class HermitianPartLinOp : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_A;
+public:
+  HermitianPartLinOp(LinearOperatorBase<Field> &A): _A(A) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    Field tmp(in.Grid());
+    _A.Op(in,out);      // A in
+    _A.AdjOp(in,tmp);   // A^dag in
+    out = 0.5*(out + tmp);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// s*I - Op : power method on this gives s - lambda_min(Op) for Hermitian Op.
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class ShiftedNegatedOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op;
+  RealD s;
+public:
+  ShiftedNegatedOperator(RealD _s, LinearOperatorBase<Field> &Op): _Op(Op), s(_s) {};
+  void OpDiag (const Field &in, Field &out) {    assert(0);  }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
+  void Op     (const Field &in, Field &out){ HermOp(in,out); }
+  void AdjOp  (const Field &in, Field &out){ HermOp(in,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){
+    HermOp(in,out);
+    ComplexD dot = innerProduct(in,out);
+    n1=real(dot);
+    n2=norm2(out);
+  }
+  void HermOp(const Field &in, Field &out){
+    _Op.HermOp(in,out);
+    out = s*in - out;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  RealD mass = 0.00078;
+  RealD M5   = 1.8;
+  RealD b    = 1.5;
+  RealD c    = 0.5;
+  int   Ls   = 24;
+  std::string config("ckpoint_lat.1000");
+
+  if(getenv("MASS"))     mass = atof(getenv("MASS"));
+  if(getenv("M5"))       M5   = atof(getenv("M5"));
+  if(getenv("MOBIUS_B")) b    = atof(getenv("MOBIUS_B"));
+  if(getenv("MOBIUS_C")) c    = atof(getenv("MOBIUS_C"));
+  if(getenv("LS"))       Ls   = atoi(getenv("LS"));
+  if(getenv("CONFIG"))   config = std::string(getenv("CONFIG"));
+
+  // Adjoint-mass sweep: explicit list, or geometric MADJ_MIN..MADJ_MAX in MADJ_N steps.
+  std::vector<RealD> madj_list;
+  if(getenv("MADJ_LIST")){
+    std::stringstream ss(getenv("MADJ_LIST"));
+    std::string tok;
+    while(std::getline(ss,tok,',')) if(tok.size()) madj_list.push_back(std::stod(tok));
+  } else {
+    int   N  = getenv("MADJ_N")   ? atoi(getenv("MADJ_N"))   : 6;
+    RealD lo = getenv("MADJ_MIN") ? atof(getenv("MADJ_MIN")) : mass;
+    RealD hi = getenv("MADJ_MAX") ? atof(getenv("MADJ_MAX")) : 1.0;
+    GRID_ASSERT(N>=1);
+    for(int i=0;i<N;i++)
+      madj_list.push_back( (N==1) ? lo : lo*std::pow(hi/lo, double(i)/double(N-1)) );
+  }
+
+  // lambda_min(H) is the most-negative eigenvalue; resolved by Chebyshev-filtered
+  // Lanczos on H (a shifted power method cannot separate it from the dense low tail).
+  RealD HalfChebyLo    = getenv("HALF_CHEBY_LO")    ? atof(getenv("HALF_CHEBY_LO"))    : 0.1;
+  RealD HalfChebyHi    = getenv("HALF_CHEBY_HI")    ? atof(getenv("HALF_CHEBY_HI"))    : 0.0; // 0 => auto
+  int   HalfChebyOrder = getenv("HALF_CHEBY_ORDER") ? atoi(getenv("HALF_CHEBY_ORDER")) : 61;
+  // Grid's Chebyshev filter MUST be odd order: only then is the polynomial positive
+  // for x < -1, the region the low/negative modes map to.  An even order flips the
+  // sign there, the filtered operator explodes negative, and the IRL never converges.
+  if(HalfChebyOrder%2==0){ HalfChebyOrder++;
+    std::cout<<GridLogMessage<<"HALF_CHEBY_ORDER forced odd -> "<<HalfChebyOrder<<std::endl; }
+  int   HalfNstop      = getenv("HALF_NSTOP")       ? atoi(getenv("HALF_NSTOP"))       : 8;
+  int   HalfNk         = getenv("HALF_NK")          ? atoi(getenv("HALF_NK"))          : 24;
+  int   HalfNm         = getenv("HALF_NM")          ? atoi(getenv("HALF_NM"))          : 48;
+  RealD HalfTol        = getenv("HALF_TOL")         ? atof(getenv("HALF_TOL"))         : 1.0e-4;
+  int   HalfMaxIt      = getenv("HALF_MAXIT")       ? atoi(getenv("HALF_MAXIT"))       : 20;
+
+  std::vector<int> lat = {48,48,48,96};
+
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
+
+  std::cout << GridLogMessage << "PARAM: MASS(light) " << mass << "  M5 " << M5
+            << "  b " << b << "  c " << c << "  Ls " << Ls << std::endl;
+  std::cout << GridLogMessage << "PARAM: CONFIG " << config << std::endl;
+
+  LatticeGaugeField Umu(UGrid);
+  FieldMetaData header;
+  std::cout << GridLogMessage << "Reading gauge field " << config << std::endl;
+  NerscIO::readConfiguration(Umu,header,config);
+
+  // Fixed light operator (never changes across the sweep).
+  MobiusFermionD Dlight(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid, mass, M5, b, c);
+
+  LatticeFermionD x(FGrid);
+
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "FINE HALF-PLANE SWEEP  A(m_adj) = D_adj^dag D_light" << std::endl;
+  std::cout << GridLogMessage << "  m_adj = " << mass << " => M^dag M (positive-real); m_adj = 1 => PVdagM" << std::endl;
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+
+  for(auto madj : madj_list){
+
+    MobiusFermionD Dadj(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid, madj, M5, b, c);
+
+    PVdagMLinearOperator<MobiusFermionD,LatticeFermionD> A(Dlight,Dadj); // A = Dadj^dag Dlight
+    HermitianPartLinOp<LatticeFermionD>                  H(A);
+
+    PowerMethod<LatticeFermionD> PM;
+
+    random(RNG5,x); RealD lamHmax = PM(H,x);
+
+    // lambda_min(H): most-negative eigenvalue via Chebyshev-filtered IRL on H.
+    // Cheby(lo,hi) amplifies eigenvalues below lo; with hi>=lambda_max(H) the most
+    // negative mode is amplified hardest, so IRL isolates the true bottom of the
+    // (possibly indefinite) spectrum where the shifted power method could not.
+    RealD fhi = (HalfChebyHi>0.0)? HalfChebyHi : 1.1*lamHmax;
+    Chebyshev<LatticeFermionD>      Cheby(HalfChebyLo,fhi,HalfChebyOrder);
+    FunctionHermOp<LatticeFermionD> OpCheby(Cheby,H);
+    PlainHermOp<LatticeFermionD>    OpPlain(H);
+    ImplicitlyRestartedLanczos<LatticeFermionD> IRL(OpCheby,OpPlain,HalfNstop,HalfNk,HalfNm,HalfTol,HalfMaxIt);
+    std::vector<RealD>           heval(HalfNm);
+    std::vector<LatticeFermionD> hevec(HalfNm,FGrid);
+    int hNconv=0;
+    random(RNG5,x);
+    IRL.calc(heval,hevec,x,hNconv);
+    RealD lamHmin = (hNconv>0) ? heval[0] : 9.99e99;
+    for(int kk=0;kk<hNconv;kk++) lamHmin = std::min(lamHmin, heval[kk]);
+    std::cout << GridLogMessage << "  (IRL H-bottom: " << hNconv
+              << " converged, most-negative eval " << lamHmin << ")" << std::endl;
+
+    random(RNG5,x); RealD sigmax2 = PM(A,x);   // A.HermOp = A^dag A
+    RealD sigmax  = std::sqrt(sigmax2);
+
+    bool  posreal = (lamHmin > 0.0);
+    RealD ratefac = posreal ? std::sqrt(1.0 - lamHmin*lamHmin/sigmax2) : 0.0; // EES per-iter
+    RealD iters8  = (posreal && ratefac < 1.0) ? std::log(1.0e-8)/std::log(ratefac) : 0.0;
+
+    std::cout << GridLogMessage << "HALFPLANE: m_adj " << madj
+              << "  lambda_min(H) " << lamHmin
+              << "  lambda_max(H) " << lamHmax
+              << "  sigma_max " << sigmax
+              << "  positive_real " << (posreal ? "YES" : "NO ")
+              << (posreal
+                    ? ("  EES_rate " + std::to_string(ratefac) + "  EES_iters(1e-8) " + std::to_string(iters8))
+                    : ("  margin_below_zero " + std::to_string(-lamHmin) + " (EES guarantee lost)"))
+              << std::endl;
+  }
+
+  std::cout << GridLogMessage << "=================================================" << std::endl;
+  std::cout << GridLogMessage << "Reading: lambda_min(H) > 0 => EES guarantees GCR at the quoted rate." << std::endl;
+  std::cout << GridLogMessage << "         crossing to < 0 as m_adj -> 1 marks loss of positive-realness." << std::endl;
+  std::cout << GridLogMessage << "         (non-normality: eigenvalues may still be right-half-plane.)" << std::endl;
+  std::cout << GridLogMessage << "Done" << std::endl;
+
+  Grid_finalize();
+  return 0;
+}

examples/Example_pvdagm_mrhs_3level.cc

@@ -0,0 +1,613 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./examples/Example_pvdagm_mrhs_3level.cc
+
+    Copyright (C) 2026
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// MultiRHS (valence) THREE-level multigrid for PVdagM.
+//
+// This is exactly the plain three-level algorithm of Example_pvdagm_3level_SVDdefl.cc
+// with L3_DEFL=0 (NO deflation), applied to the enlarged block-diagonal mRHS system:
+// the coarse and coarse-coarse levels run a SINGLE Krylov (one GCR polynomial, inner
+// products summed over rhs) on the packed 6D mrhs fields, so both coarse levels batch
+// through GEMM (MultiGeneralCoarsenedMatrix) -- the valence throughput win at BOTH levels.
+//
+// Level structure (each coarse level is a single-field PGCR on a packed 6D mrhs field):
+//   L1 (fine)         : std::vector<LatticeFermionD>, MrhsPGCRNonHermitian on PVdagM,
+//                       preconditioned by the L1->L2 mrhs V-cycle (MrhsTwoLevelMG).
+//   L2 (coarse)       : 6D mrhs coarse field, PGCR, preconditioned by the L2->L3 mrhs
+//                       V-cycle (MrhsCoarseThreeLevelPrec) -- coarse-coarse correction + coarse smoother.
+//   L3 (coarse-coarse): 6D mrhs coarse-coarse field, PGCR (the innermost solve).
+//
+// RAW-NULL DISCIPLINE (critical -- see project_block_orthogonalise_leak): the L2->L3
+// aggregation MUST be built from RAW fine near-null vectors (pre block-GS).  We take a
+// raw copy of the loaded subspace BEFORE the L1->L2 CoarsenOperator (which block-
+// orthonormalises in place) and project THAT.  Guards print ||<psi|psi> - I||: ~0.23 =
+// content preserved, ~N_coarse = the e_k leak is back.
+//
+// Env: MASS SUBSPACE_FILE NRHS
+//      BLOCK  (dotted, default 2.2.2.2)  BLOCK2 (dotted, default 2.2.3.3)
+//      FineSmootherShift FineSmootherOrder
+//      CoarseSmootherShift CoarseSmootherNstep
+//      CoarseSolverTol CoarseSolverOrder
+//      L3_TOL L3_MAXIT L3_NSTEP
+//      OuterMmax OuterNstep OuterTol
+
+#include <Grid/Grid.h>
+#include <Grid/lattice/PaddedCell.h>
+#include <Grid/stencil/GeneralLocalStencil.h>
+#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
+
+using namespace std;
+using namespace Grid;
+
+RealD FineSmootherShift    = 0.1;
+int   FineSmootherOrder    = 16;
+RealD CoarseSmootherShift  = 0.1;
+int   CoarseSmootherNstep  = 4;
+RealD CoarseSolverTol      = 0.03;
+int   CoarseSolverOrder    = 200;
+RealD L3Tol                = 2.5e-1;
+int   L3MaxIt              = 50;
+int   L3Nstep              = 50;
+RealD OuterTol             = 1.0e-8;
+int   OuterMmax            = 8;
+int   OuterNstep           = 8;
+int   Nrhs                 = 12;
+RealD mass                 = 0.00078;
+
+void ParseEnvironment(void)
+{
+  if(getenv("MASS"))               mass               = atof(getenv("MASS"));
+  if(getenv("FineSmootherShift"))  FineSmootherShift  = atof(getenv("FineSmootherShift"));
+  if(getenv("FineSmootherOrder"))  FineSmootherOrder  = atoi(getenv("FineSmootherOrder"));
+  if(getenv("CoarseSmootherShift"))CoarseSmootherShift= atof(getenv("CoarseSmootherShift"));
+  if(getenv("CoarseSmootherNstep"))CoarseSmootherNstep= atoi(getenv("CoarseSmootherNstep"));
+  if(getenv("CoarseSolverTol"))    CoarseSolverTol    = atof(getenv("CoarseSolverTol"));
+  if(getenv("CoarseSolverOrder"))  CoarseSolverOrder  = atoi(getenv("CoarseSolverOrder"));
+  if(getenv("L3_TOL"))             L3Tol              = atof(getenv("L3_TOL"));
+  if(getenv("L3_MAXIT"))           L3MaxIt            = atoi(getenv("L3_MAXIT"));
+  if(getenv("L3_NSTEP"))           L3Nstep            = atoi(getenv("L3_NSTEP"));
+  if(getenv("OuterTol"))           OuterTol           = atof(getenv("OuterTol"));
+  if(getenv("OuterMmax"))          OuterMmax          = atoi(getenv("OuterMmax"));
+  if(getenv("OuterNstep"))         OuterNstep         = atoi(getenv("OuterNstep"));
+  if(getenv("NRHS"))               Nrhs               = atoi(getenv("NRHS"));
+
+  std::cout << GridLogMessage << "PARAM: MASS               " << mass               << std::endl;
+  std::cout << GridLogMessage << "PARAM: NRHS               " << Nrhs               << std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherShift  " << FineSmootherShift  << std::endl;
+  std::cout << GridLogMessage << "PARAM: FineSmootherOrder  " << FineSmootherOrder  << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSmootherShift" << CoarseSmootherShift<< std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSmootherNstep" << CoarseSmootherNstep<< std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverTol    " << CoarseSolverTol    << std::endl;
+  std::cout << GridLogMessage << "PARAM: CoarseSolverOrder  " << CoarseSolverOrder  << std::endl;
+  std::cout << GridLogMessage << "PARAM: L3_TOL             " << L3Tol              << std::endl;
+  std::cout << GridLogMessage << "PARAM: OuterMmax          " << OuterMmax          << std::endl;
+  std::cout << GridLogMessage << "PARAM: OuterNstep         " << OuterNstep         << std::endl;
+}
+
+template <class Field>
+void saveSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  Grid::emptyUserRecord record;
+  Grid::ScidacWriter SW(subspace[0].Grid()->IsBoss());
+  SW.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++) SW.writeScidacFieldRecord(subspace[k], record);
+  SW.close();
+#endif
+}
+template <class Field>
+void loadSubspace(std::vector<Field> &subspace, std::string const fname){
+#ifdef HAVE_LIME
+  Grid::emptyUserRecord record;
+  Grid::ScidacReader SR;
+  SR.open(fname);
+  for (int k = 0; k < (int)subspace.size(); k++) SR.readScidacFieldRecord(subspace[k], record);
+  SR.close();
+#endif
+}
+
+//////////////////////////////////////////////////////////////////////
+// A = PV^dag M (non-Hermitian), and shifted variant for smoothers.
+//////////////////////////////////////////////////////////////////////
+template<class Matrix,class Field>
+class PVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat; Matrix &_PV;
+public:
+  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV) {};
+  void OpDiag (const Field &in, Field &out) { assert(0); }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
+  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); }
+  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(in,tmp); _Mat.Mdag(tmp,out); }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ HermOp(in,out); ComplexD d=innerProduct(in,out); n1=real(d); n2=norm2(out); }
+  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+template<class Matrix,class Field>
+class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
+  Matrix &_Mat; Matrix &_PV;
+public:
+  RealD shift;
+  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
+  void OpDiag (const Field &in, Field &out) { assert(0); }
+  void OpDir  (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll  (const Field &in, std::vector<Field> &out){ assert(0); };
+  void Op     (const Field &in, Field &out){ Field tmp(in.Grid()); _Mat.M(in,tmp); _PV.Mdag(tmp,out); out = out + shift*in; }
+  void AdjOp  (const Field &in, Field &out){ Field tmp(in.Grid()); _PV.M(tmp,out); _Mat.Mdag(in,tmp); out = out + shift*in; }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp(const Field &in, Field &out){ Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+// Generic shift wrapper (for the coarse-level smoother on the 6D mrhs coarse operator).
+template<class Field>
+class ShiftedLinearOperator : public LinearOperatorBase<Field> {
+  LinearOperatorBase<Field> &_Op; RealD shift;
+public:
+  ShiftedLinearOperator(RealD _shift, LinearOperatorBase<Field> &Op) : _Op(Op), shift(_shift) {}
+  void OpDiag  (const Field &in, Field &out) { assert(0); }
+  void OpDir   (const Field &in, Field &out,int dir,int disp) { assert(0); }
+  void OpDirAll (const Field &in, std::vector<Field> &out) { assert(0); }
+  void Op      (const Field &in, Field &out) { _Op.Op(in,out);    out = out + shift*in; }
+  void AdjOp   (const Field &in, Field &out) { _Op.AdjOp(in,out); out = out + shift*in; }
+  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){ assert(0); }
+  void HermOp  (const Field &in, Field &out) { Field tmp(in.Grid()); Op(in,tmp); AdjOp(tmp,out); }
+};
+
+//////////////////////////////////////////////////////////////////////
+// mrhs interfaces + single-polynomial mrhs PGCR (verbatim from Example_pvdagm_mrhs.cc):
+// reductions summed over rhs -> one alpha/beta per step for the enlarged system.
+//////////////////////////////////////////////////////////////////////
+template<class Field>
+class MrhsLinearFunction {
+public:
+  virtual void operator()(std::vector<Field> &in, std::vector<Field> &out) = 0;
+};
+
+template<class Field>
+class MrhsPGCRNonHermitian {
+public:
+  RealD Tolerance; Integer MaxIterations; int mmax,nstep,steps,level;
+  int ZeroGuess = 0; int FirstCycle = 0;   // caller contract: zero guess => first-cycle r0 = src
+  std::string name = "Level 1";
+  LinearOperatorBase<Field> &Linop;
+  MrhsLinearFunction<Field> &Preconditioner;
+  void Level(int lv){ name = "Level " + std::to_string(lv); level=lv; }
+  void Name(std::string n){ name = n; }
+  void SetZeroGuess(int z){ ZeroGuess=z; }
+  MrhsPGCRNonHermitian(RealD tol,Integer maxit,LinearOperatorBase<Field> &_Linop,MrhsLinearFunction<Field> &Prec,int _mmax,int _nstep)
+    : Tolerance(tol),MaxIterations(maxit),Linop(_Linop),Preconditioner(Prec),mmax(_mmax),nstep(_nstep){ level=1; }
+  static RealD vnorm2(std::vector<Field> &x){ RealD s=0; for(auto &f:x) s+=norm2(f); return s; }
+  static ComplexD vinnerProduct(std::vector<Field> &x,std::vector<Field> &y){ ComplexD s(0); for(int r=0;r<(int)x.size();r++) s+=innerProduct(x[r],y[r]); return s; }
+  static void vaxpy(std::vector<Field> &z,ComplexD a,std::vector<Field> &x,std::vector<Field> &y){ for(int r=0;r<(int)z.size();r++) axpy(z[r],a,x[r],y[r]); }
+  void vOp(std::vector<Field> &in,std::vector<Field> &out){ for(int r=0;r<(int)in.size();r++) Linop.Op(in[r],out[r]); }
+  void operator()(std::vector<Field> &src,std::vector<Field> &psi){
+    RealD cp,ssq,rsq; int nrhs=src.size(); GridBase *grid=src[0].Grid();
+    ssq=vnorm2(src); rsq=Tolerance*Tolerance*ssq;
+    std::vector<Field> r(nrhs,grid);
+    GridStopWatch T; T.Start(); steps=0; FirstCycle=1;
+    for(int k=0;k<MaxIterations;k++){
+      cp=GCRnStep(src,psi,rsq);
+      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR("<<mmax<<","<<nstep<<") "<<steps<<" steps cp = "<<cp<<" target "<<rsq<<std::endl;
+      if(cp<rsq){
+        T.Stop(); vOp(psi,r); for(int rr=0;rr<nrhs;rr++) axpy(r[rr],-1.0,src[rr],r[rr]);
+        RealD tr=vnorm2(r);
+        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR: Converged on iteration "<<steps
+                 <<" computed residual "<<std::sqrt(cp/ssq)<<" true residual "<<std::sqrt(tr/ssq)<<" target "<<Tolerance<<std::endl;
+        std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR Time elapsed: Total "<<T.Elapsed()<<std::endl;
+        for(int rr=0;rr<nrhs;rr++){ RealD rn=std::sqrt(norm2(r[rr])/norm2(src[rr])); std::cout<<GridLogMessage<<"MrhsPGCR per-rhs true residual["<<rr<<"] = "<<rn<<std::endl; }
+        return;
+      }
+    }
+    std::cout<<GridLogMessage<<"MrhsPGCR: did not converge"<<std::endl;
+  }
+  RealD GCRnStep(std::vector<Field> &src,std::vector<Field> &psi,RealD rsq){
+    RealD cp; ComplexD a,b,rq; RealD zAAz; int nrhs=src.size(); GridBase *grid=src[0].Grid();
+    std::vector<Field> r(nrhs,grid),z(nrhs,grid),Az(nrhs,grid);
+    std::vector< std::vector<Field> > q(mmax,std::vector<Field>(nrhs,grid));
+    std::vector< std::vector<Field> > p(mmax,std::vector<Field>(nrhs,grid));
+    std::vector<RealD> qq(mmax);
+    std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR nStep("<<nstep<<")"<<std::endl;
+    if (ZeroGuess && FirstCycle) { for(int rr=0;rr<nrhs;rr++){ psi[rr]=Zero(); r[rr]=src[rr]; } }
+    else                         { vOp(psi,Az); for(int rr=0;rr<nrhs;rr++) r[rr]=src[rr]-Az[rr]; }
+    FirstCycle=0;
+    Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
+    p[0]=z; q[0]=Az; qq[0]=zAAz; cp=vnorm2(r);
+    for(int k=0;k<nstep;k++){
+      steps++; int kp=k+1, peri_k=k%mmax, peri_kp=kp%mmax;
+      rq=vinnerProduct(q[peri_k],r); a=rq/qq[peri_k];
+      vaxpy(psi,a,p[peri_k],psi); vaxpy(r,-a,q[peri_k],r); cp=vnorm2(r);
+      std::cout<<GridLogMessage<<std::string(level,'\t')<<" "<<name<<" MrhsPGCR step["<<steps<<"]  resid "<<cp<<" target "<<rsq<<std::endl;
+      if((k==nstep-1)||(cp<rsq)) return cp;
+      Preconditioner(r,z); vOp(z,Az); zAAz=vnorm2(Az);
+      q[peri_kp]=Az; p[peri_kp]=z;
+      int northog=((kp)>(mmax-1))?(mmax-1):(kp);
+      for(int back=0;back<northog;back++){ int peri_back=(k-back)%mmax; GRID_ASSERT((k-back)>=0);
+        b=-real(vinnerProduct(q[peri_back],Az))/qq[peri_back];
+        vaxpy(p[peri_kp],b,p[peri_back],p[peri_kp]); vaxpy(q[peri_kp],b,q[peri_back],q[peri_kp]); }
+      qq[peri_kp]=vnorm2(q[peri_kp]);
+    }
+    GRID_ASSERT(0); return cp;
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// L2->L3 mrhs V-cycle: a LinearFunction on the 6D mrhs COARSE field.
+// Mirrors Example_pvdagm_mrhs.cc's MrhsTwoLevelMG one level down, and the
+// single-RHS MGPreconditioner of Example_pvdagm_3level_SVDdefl.cc:
+//   out = in (trivial pre)
+//   r = in - A_coarse out
+//   restrict (unpack 6D coarse -> blockProject -> pack 6D coarse-coarse)
+//   ONE coarse-coarse solve (L3, GEMM)
+//   prolong (unpack -> blockPromote -> pack); out += correction
+//   r = in - A_coarse out
+//   coarse smoother (shifted 6D coarse op); out += smooth(r)
+//////////////////////////////////////////////////////////////////////
+template<class CoarseField, class CoarseCoarseField>
+class MrhsCoarseThreeLevelPrec : public LinearFunction<CoarseField> {
+public:
+  LinearOperatorBase<CoarseField>          &_CoarseOp;          // mrhs coarse op (6D)
+  LinearFunction<CoarseField>              &_CoarseSmoother;    // shifted 6D coarse smoother
+  MultiRHSBlockProject<CoarseField>        &_Projector;         // L2->L3 (vector-based)
+  LinearFunction<CoarseCoarseField>        &_CoarseCoarseSolve; // L3 solve (6D cc)
+  GridBase *_Coarse5d, *_CoarseCoarse5d, *_CoarseCoarseMrhs;
+  int _nrhs;
+
+  MrhsCoarseThreeLevelPrec(LinearOperatorBase<CoarseField> &CoarseOp,
+                           LinearFunction<CoarseField> &CoarseSmoother,
+                           MultiRHSBlockProject<CoarseField> &Projector,
+                           LinearFunction<CoarseCoarseField> &CoarseCoarseSolve,
+                           GridBase *Coarse5d, GridBase *CoarseCoarse5d, GridBase *CoarseCoarseMrhs, int nrhs)
+    : _CoarseOp(CoarseOp), _CoarseSmoother(CoarseSmoother), _Projector(Projector),
+      _CoarseCoarseSolve(CoarseCoarseSolve),
+      _Coarse5d(Coarse5d), _CoarseCoarse5d(CoarseCoarse5d), _CoarseCoarseMrhs(CoarseCoarseMrhs), _nrhs(nrhs) {}
+
+  using LinearFunction<CoarseField>::operator();
+  virtual void operator()(const CoarseField &in, CoarseField &out) {
+    int nrhs=_nrhs; double t;
+    CoarseField vec1(in.Grid());
+    CoarseField vec2(in.Grid());
+
+    // trivial pre-smoother
+    out = in;
+
+    // residual (6D coarse)
+    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
+
+    // restrict: unpack 6D coarse -> vector<Coarse> -> blockProject -> vector<CoarseCoarse> -> pack 6D cc
+    std::vector<CoarseField>       csplit(nrhs,_Coarse5d);
+    std::vector<CoarseCoarseField> ccsplit(nrhs,_CoarseCoarse5d);
+    CoarseCoarseField CCsrc(_CoarseCoarseMrhs);
+    CoarseCoarseField CCsol(_CoarseCoarseMrhs);
+
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(csplit[r],vec1,r,0);
+    _Projector.blockProject(csplit,ccsplit);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(ccsplit[r],CCsrc,r,0);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L2->L3 restrict took "<<t/1000.0<<"ms"<<std::endl;
+
+    // L3 solve (6D coarse-coarse, GEMM)
+    t=-usecond();
+    CCsol=Zero();
+    _CoarseCoarseSolve(CCsrc,CCsol);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L3 coarse-coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+
+    // prolong: unpack 6D cc -> blockPromote -> pack 6D coarse; add correction
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(ccsplit[r],CCsol,r,0);
+    _Projector.blockPromote(csplit,ccsplit);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(csplit[r],vec1,r,0);
+    add(out,out,vec1);
+    t+=usecond();
+    std::cout<<GridLogMessage<<"L2->L3 prolong took "<<t/1000.0<<"ms"<<std::endl;
+
+    // residual + coarse smoother (6D coarse)
+    _CoarseOp.Op(out,vec1);  sub(vec1,in,vec1);
+    vec2=Zero();
+    _CoarseSmoother(vec1,vec2);
+    add(out,out,vec2);
+  }
+};
+
+//////////////////////////////////////////////////////////////////////
+// L1->L2 mrhs V-cycle (verbatim from Example_pvdagm_mrhs.cc):
+// per-rhs fine smoother + batched restriction + ONE coarse solve + batched prolong.
+// The coarse solve passed in is now itself three-level (preconditioned by L2->L3).
+//////////////////////////////////////////////////////////////////////
+template<class FineField, class MrhsCoarseVector, class FineSmoother>
+class MrhsTwoLevelMG : public MrhsLinearFunction<FineField> {
+public:
+  typedef MrhsCoarseVector CoarseVector;
+  LinearOperatorBase<FineField>   &_FineOperator;
+  FineSmoother                    &_PostSmoother;
+  MultiRHSBlockProject<FineField> &_Projector;
+  LinearFunction<CoarseVector>    &_CoarseSolve;
+  GridBase *_CoarseGrid, *_CoarseGridMrhs;
+  MrhsTwoLevelMG(LinearOperatorBase<FineField> &FineOp, FineSmoother &Post,
+                 MultiRHSBlockProject<FineField> &Projector, LinearFunction<CoarseVector> &CoarseSolve,
+                 GridBase *CoarseGrid, GridBase *CoarseGridMrhs)
+    : _FineOperator(FineOp),_PostSmoother(Post),_Projector(Projector),_CoarseSolve(CoarseSolve),
+      _CoarseGrid(CoarseGrid),_CoarseGridMrhs(CoarseGridMrhs){}
+  virtual void operator()(std::vector<FineField> &in, std::vector<FineField> &out){
+    int nrhs=in.size(); GridBase *fgrid=in[0].Grid(); double t;
+    std::vector<FineField> vec1(nrhs,fgrid),vec2(nrhs,fgrid);
+    for(int r=0;r<nrhs;r++) out[r]=in[r];
+    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
+    std::vector<CoarseVector> Csrc_split(nrhs,_CoarseGrid), Csol_split(nrhs,_CoarseGrid);
+    CoarseVector CsrcMrhs(_CoarseGridMrhs), CsolMrhs(_CoarseGridMrhs);
+    t=-usecond();
+    _Projector.blockProject(vec1,Csrc_split);
+    for(int r=0;r<nrhs;r++) InsertSliceFast(Csrc_split[r],CsrcMrhs,r,0);
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs project+pack took "<<t/1000.0<<"ms"<<std::endl;
+    t=-usecond(); CsolMrhs=Zero(); _CoarseSolve(CsrcMrhs,CsolMrhs); t+=usecond();
+    std::cout<<GridLogMessage<<"Mrhs coarse solve took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+    t=-usecond();
+    for(int r=0;r<nrhs;r++) ExtractSliceFast(Csol_split[r],CsolMrhs,r,0);
+    _Projector.blockPromote(vec1,Csol_split);
+    for(int r=0;r<nrhs;r++) add(out[r],out[r],vec1[r]);
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs unpack+promote took "<<t/1000.0<<"ms"<<std::endl;
+    for(int r=0;r<nrhs;r++){ _FineOperator.Op(out[r],vec1[r]); sub(vec1[r],in[r],vec1[r]); }
+    t=-usecond();
+    for(int r=0;r<nrhs;r++){ vec2[r]=Zero(); _PostSmoother(vec1[r],vec2[r]); add(out[r],out[r],vec2[r]); }
+    t+=usecond(); std::cout<<GridLogMessage<<"Mrhs post-smooth took "<<t/1000.0<<"ms  ("<<t/1000.0/nrhs<<"ms/rhs)"<<std::endl;
+  }
+};
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+  ParseEnvironment();
+
+  const int Ls=24; RealD M5=1.8, b=1.5, c=0.5;
+  const int nbasis=60; const int nrhs=Nrhs;
+  GRID_ASSERT(nrhs % vComplex::Nsimd() == 0);
+
+  std::vector<int> lat_size {48,48,48,96};
+  GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(lat_size, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
+  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
+  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
+
+  // Level 1 blocking (default 2^4)
+  Coordinate clatt = lat_size;
+  Coordinate Block({2,2,2,2});
+  if ( getenv("BLOCK") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK")),Block); GRID_ASSERT(Block.size()==4); }
+  for(int d=0;d<4;d++){ GRID_ASSERT(lat_size[d]%Block[d]==0); clatt[d]=lat_size[d]/Block[d]; }
+  std::cout << GridLogMessage << "Block  " << Block  << "  coarse lattice       " << clatt << std::endl;
+
+  // Level 2 blocking (default 2,2,3,3)  -- matches Example_pvdagm_3level_SVDdefl
+  Coordinate cclatt = clatt;
+  Coordinate Block2({2,2,3,3});
+  if ( getenv("BLOCK2") ){ GridCmdOptionIntVector(std::string(getenv("BLOCK2")),Block2); GRID_ASSERT(Block2.size()==4); }
+  for(int d=0;d<4;d++){ GRID_ASSERT(clatt[d]%Block2[d]==0); cclatt[d]=clatt[d]/Block2[d]; }
+  std::cout << GridLogMessage << "Block2 " << Block2 << "  coarse-coarse lattice " << cclatt << std::endl;
+
+  GridCartesian *Coarse4d = SpaceTimeGrid::makeFourDimGrid(clatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *Coarse5d = SpaceTimeGrid::makeFiveDimGrid(1,Coarse4d);
+  GridCartesian *CoarseCoarse4d = SpaceTimeGrid::makeFourDimGrid(cclatt, GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
+  GridCartesian *CoarseCoarse5d = SpaceTimeGrid::makeFiveDimGrid(1,CoarseCoarse4d);
+
+  // 6D mrhs grids: rhs is dim 0, SIMD across rhs (pattern: Test_general_coarse_hdcg_phys48.cc)
+  Coordinate mpi=GridDefaultMpi();
+  Coordinate rhMpi ({1,1,mpi[0],mpi[1],mpi[2],mpi[3]});
+  Coordinate rhSimd({vComplex::Nsimd(),1,1,1,1,1});
+  Coordinate rhLatt ({nrhs,1,clatt[0], clatt[1], clatt[2], clatt[3]});
+  Coordinate rhLatt2({nrhs,1,cclatt[0],cclatt[1],cclatt[2],cclatt[3]});
+  GridCartesian *CoarseMrhs       = new GridCartesian(rhLatt, rhSimd,rhMpi);
+  GridCartesian *CoarseCoarseMrhs = new GridCartesian(rhLatt2,rhSimd,rhMpi);
+
+  GridParallelRNG RNG5(FGrid); RNG5.SeedFixedIntegers({5,6,7,8});
+
+  LatticeGaugeField Umu(UGrid);
+  std::cout << GridLogMessage << "Reading gauge field" << std::endl;
+  FieldMetaData header;
+  std::string file("/ccs/home/poare/ckpoint_lat.1000");
+  NerscIO::readConfiguration(Umu,header,file);
+
+  MobiusFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,b,c);
+  MobiusFermionD Dpv (Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0, M5,b,c);
+
+  typedef PVdagMLinearOperator<MobiusFermionD,LatticeFermionD>        PVdagM_t;
+  typedef ShiftedPVdagMLinearOperator<MobiusFermionD,LatticeFermionD> ShiftedPVdagM_t;
+
+  // Level 1 tensor types
+  typedef GeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>       LittleDiracOperator;
+  typedef MultiGeneralCoarsenedMatrix<vSpinColourVector,vTComplex,nbasis>  MrhsLittleDiracOperator;
+  typedef LittleDiracOperator::CoarseVector                                CoarseVector;
+  typedef Aggregation<vSpinColourVector,vTComplex,nbasis>                  Subspace;
+
+  // Level 2 tensor types (coarsening deepens the nest by one iScalar -- see CLAUDE.md)
+  typedef CoarseVector::vector_object                                      CoarseSiteObj;
+  typedef iScalar<vTComplex>                                              vTTComplex;
+  typedef GeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>          LittleDiracOperatorL2;
+  typedef MultiGeneralCoarsenedMatrix<CoarseSiteObj,vTTComplex,nbasis>     MrhsLittleDiracOperatorL2;
+  typedef LittleDiracOperatorL2::CoarseVector                              CoarseCoarseVector;
+  typedef Aggregation<CoarseSiteObj,vTTComplex,nbasis>                     SubspaceL2;
+
+  PVdagM_t        PVdagM(Ddwf,Dpv);
+  ShiftedPVdagM_t ShiftedPVdagM(FineSmootherShift,Ddwf,Dpv);
+
+  NextToNearestStencilGeometry5D geom (Coarse5d);
+  NextToNearestStencilGeometry5D geom2(CoarseCoarse5d);   // 33-point at L2->L3, matching SVDdefl
+
+  //////////////////////////////////////////////////////////////////////
+  // Subspace: load RAW (no Orthogonalise!), or generate.
+  //////////////////////////////////////////////////////////////////////
+  std::string subspace_file = "/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/subspace_nb"
+                            + std::to_string(nbasis) + ".scidac";
+  if ( getenv("SUBSPACE_FILE") ) subspace_file = std::string(getenv("SUBSPACE_FILE"));
+  uint64_t file_exists=0;
+  if ( UGrid->IsBoss() ){ std::ifstream f(subspace_file); file_exists=f.good()?1:0; }
+  UGrid->GlobalSum(file_exists);
+
+  const int cb=0;
+  Subspace AggregatesGCR(Coarse5d,FGrid,cb);
+  if ( file_exists ){
+    std::cout << GridLogMessage << "*** Loading subspace from disk (kept RAW) ***" << std::endl;
+    loadSubspace(AggregatesGCR.subspace, subspace_file);
+  } else {
+    std::cout << GridLogMessage << "*** GCR subspace generation ***" << std::endl;
+    AggregatesGCR.CreateSubspaceGCR(RNG5,PVdagM,nbasis);
+    saveSubspace(AggregatesGCR.subspace, subspace_file);
+  }
+
+  // RAW copy of the fine null vectors BEFORE CoarsenOperator block-orthonormalises in place.
+  std::vector<LatticeFermionD> rawNull(nbasis,FGrid);
+  for(int k=0;k<nbasis;k++) rawNull[k]=AggregatesGCR.subspace[k];
+
+  //////////////////////////////////////////////////////////////////////
+  // Coarsen L1->L2 and L2->L3 with SINGLE-RHS machinery, import into the mrhs
+  // operators via CopyMatrix.  The single-RHS L1->L2 coarse operator must stay
+  // alive to be the "fine" operator for the L2->L3 coarsening, so BOTH single-RHS
+  // ops (and their padded _A) live in one scope and free together.  [MEMORY: this
+  // is the setup peak -- L1->L2 padded _A (~large at 2^4) + L2->L3 padded _A.]
+  //////////////////////////////////////////////////////////////////////
+  MrhsLittleDiracOperator   mrhsLittleDiracOpPV(geom,  CoarseMrhs);
+  MrhsLittleDiracOperatorL2 mrhsLittleDiracOpL2(geom2, CoarseCoarseMrhs);
+  MultiRHSBlockProject<LatticeFermionD> MrhsProjector;
+  MultiRHSBlockProject<CoarseVector>    MrhsProjectorL2;
+  {
+    // --- L1->L2 single-RHS coarse operator (kept alive for the L2->L3 coarsening) ---
+    LittleDiracOperator LittleDiracOpPV(geom,FGrid,Coarse5d);
+    LittleDiracOpPV.CoarsenOperator(PVdagM, AggregatesGCR);   // orthonormalises AggregatesGCR.subspace in place
+    mrhsLittleDiracOpPV.CopyMatrix(LittleDiracOpPV);
+    MrhsProjector.Allocate(nbasis,FGrid,Coarse5d);
+    MrhsProjector.ImportBasis(AggregatesGCR.subspace);       // orthonormalised, matches the coarse op
+    NonHermitianLinearOperator<LittleDiracOperator,CoarseVector> LinOpCoarse(LittleDiracOpPV);
+
+    // --- psi_coarse = P^dag (RAW fine null) -> Galerkin images, NOT e_k ---
+    std::vector<CoarseVector> psi_coarse(nbasis,Coarse5d);
+    for(int k=0;k<nbasis;k++) AggregatesGCR.ProjectToSubspace(psi_coarse[k], rawNull[k]);
+    rawNull.clear(); rawNull.shrink_to_fit();
+    {
+      RealD s2=0.0;
+      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
+        ComplexD sij=TensorRemove(innerProduct(psi_coarse[i],psi_coarse[j]));
+        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
+      }
+      std::cout<<GridLogMessage<<"GUARD: ||<psi_coarse|psi_coarse> - I||_F = "<<std::sqrt(s2)
+               <<"   (~0.23 good; ~sqrt(N_coarse)="<<std::sqrt((double)Coarse5d->gSites())<<" = e_k leak)"<<std::endl;
+    }
+
+    // --- L2->L3 single-RHS coarsening (coarsen the single-RHS LinOpCoarse) ---
+    SubspaceL2 AggregatesL2(CoarseCoarse5d,Coarse5d,cb);
+    for(int k=0;k<nbasis;k++) AggregatesL2.subspace[k]=psi_coarse[k];
+    LittleDiracOperatorL2 LittleDiracOpL2(geom2,Coarse5d,CoarseCoarse5d);
+    LittleDiracOpL2.CoarsenOperator(LinOpCoarse, AggregatesL2);
+    mrhsLittleDiracOpL2.CopyMatrix(LittleDiracOpL2);
+    MrhsProjectorL2.Allocate(nbasis,Coarse5d,CoarseCoarse5d);
+    MrhsProjectorL2.ImportBasis(AggregatesL2.subspace);      // orthonormalised by CoarsenOperator
+
+    // --- guard psi_cc (RAW psi_coarse; AggregatesL2 holds a separate orthonormalised copy) ---
+    {
+      std::vector<CoarseCoarseVector> psi_cc(nbasis,CoarseCoarse5d);
+      for(int k=0;k<nbasis;k++) AggregatesL2.ProjectToSubspace(psi_cc[k], psi_coarse[k]);
+      RealD s2=0.0;
+      for(int i=0;i<nbasis;i++) for(int j=0;j<nbasis;j++){
+        ComplexD sij=TensorRemove(innerProduct(psi_cc[i],psi_cc[j]));
+        ComplexD d=sij-(i==j?ComplexD(1.0):ComplexD(0.0)); s2+=real(d)*real(d)+imag(d)*imag(d);
+      }
+      std::cout<<GridLogMessage<<"GUARD: ||<psi_cc|psi_cc> - I||_F = "<<std::sqrt(s2)
+               <<"   (~0.23 good; ~sqrt(N_cc)="<<std::sqrt((double)CoarseCoarse5d->gSites())<<" = e_k leak)"<<std::endl;
+    }
+  } // both single-RHS ops + padded _A + AggregatesL2 + psi_coarse freed here
+
+  NonHermitianLinearOperator<MrhsLittleDiracOperator,CoarseVector>         mrhsLinOpCoarse(mrhsLittleDiracOpPV);
+  NonHermitianLinearOperator<MrhsLittleDiracOperatorL2,CoarseCoarseVector> mrhsLinOpCC(mrhsLittleDiracOpL2);
+
+  //////////////////////////////////////////////////////////////////////
+  // Solvers, innermost first.
+  //////////////////////////////////////////////////////////////////////
+  TrivialPrecon<CoarseVector>       simpleC;
+  TrivialPrecon<CoarseCoarseVector> simpleCC;
+  TrivialPrecon<LatticeFermionD>    simple_fine;
+
+  // L3 (coarse-coarse) solve: PGCR on the 6D cc operator
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseCoarseVector>
+    L3PGCR(L3Tol,L3MaxIt,mrhsLinOpCC,simpleCC,L3Nstep,L3Nstep);
+  L3PGCR.Level(3);
+  L3PGCR.Name("CCouter");
+  L3PGCR.SetZeroGuess(1);              // caller zeroes CCsol
+
+  // L2 coarse smoother: shifted 6D coarse op, fixed nstep
+  ShiftedLinearOperator<CoarseVector> ShiftedMrhsCoarse(CoarseSmootherShift, mrhsLinOpCoarse);
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
+    CoarseSmootherGCR(0.01,1,ShiftedMrhsCoarse,simpleC,CoarseSmootherNstep,CoarseSmootherNstep);
+  CoarseSmootherGCR.Level(2);
+  CoarseSmootherGCR.Name("Csmoother");
+  CoarseSmootherGCR.SetZeroGuess(1);   // caller zeroes vec2
+
+  // L2->L3 V-cycle preconditioner (operates on 6D coarse field)
+  MrhsCoarseThreeLevelPrec<CoarseVector,CoarseCoarseVector>
+    L2to3Precon(mrhsLinOpCoarse, CoarseSmootherGCR, MrhsProjectorL2, L3PGCR,
+                Coarse5d, CoarseCoarse5d, CoarseCoarseMrhs, nrhs);
+
+  // L2 coarse solve: PGCR on 6D coarse op, preconditioned by the L2->L3 V-cycle
+  PrecGeneralisedConjugateResidualNonHermitian<CoarseVector>
+    L2PGCR(CoarseSolverTol, CoarseSolverOrder/16, mrhsLinOpCoarse, L2to3Precon, 16, 16);
+  L2PGCR.Level(2);
+  L2PGCR.Name("Couter");
+  L2PGCR.SetZeroGuess(1);              // caller zeroes CsolMrhs
+
+  // Fine smoother (per-rhs, looped in the L1->L2 V-cycle)
+  PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD>
+    SmootherGCR(0.0,1,ShiftedPVdagM,simple_fine,FineSmootherOrder,FineSmootherOrder);
+  SmootherGCR.Level(1);
+  SmootherGCR.Name("Fsmoother");
+  SmootherGCR.SetZeroGuess(1);         // caller zeroes vec2[r]
+
+  // L1->L2 V-cycle (fine); its coarse solve is the three-level L2PGCR
+  typedef PrecGeneralisedConjugateResidualNonHermitian<LatticeFermionD> FineSmoother_t;
+  MrhsTwoLevelMG<LatticeFermionD,CoarseVector,FineSmoother_t>
+    ThreeLevelPrecon(PVdagM, SmootherGCR, MrhsProjector, L2PGCR, Coarse5d, CoarseMrhs);
+
+  // Outer mrhs solve
+  MrhsPGCRNonHermitian<LatticeFermionD>
+    L1PGCR(OuterTol,1000,PVdagM,ThreeLevelPrecon,OuterMmax,OuterNstep);
+  L1PGCR.Level(1);
+  L1PGCR.Name("Fouter");
+  L1PGCR.SetZeroGuess(1);              // sol[r]=Zero() at source setup
+
+  //////////////////////////////////////////////////////////////////////
+  // Sources and solve
+  //////////////////////////////////////////////////////////////////////
+  std::vector<LatticeFermionD> src(nrhs,FGrid), sol(nrhs,FGrid);
+  for(int r=0;r<nrhs;r++){ gaussian(RNG5,src[r]); sol[r]=Zero(); }
+
+  std::cout << GridLogMessage << "**********************************************" << std::endl;
+  std::cout << GridLogMessage << " MultiRHS THREE-level solve: " << nrhs << " RHS " << std::endl;
+  std::cout << GridLogMessage << "**********************************************" << std::endl;
+
+  GridStopWatch w; w.Start();
+  L1PGCR(src,sol);
+  w.Stop();
+  std::cout << GridLogMessage << "MultiRHS 3-level solve total " << w.Elapsed()
+            << "  (per RHS: " << w.useconds()/1.0e6/nrhs << " s)" << std::endl;
+
+  { LatticeFermionD Ax(FGrid); RealD worst=0.0;
+    for(int r=0;r<nrhs;r++){ PVdagM.Op(sol[r],Ax); Ax=Ax-src[r];
+      RealD rn=std::sqrt(norm2(Ax)/norm2(src[r]));
+      std::cout << GridLogMessage << "FINAL: rhs["<<r<<"] true residual = " << rn << std::endl;
+      worst=std::max(worst,rn); }
+    std::cout << GridLogMessage << "FINAL: worst-case residual = " << worst << std::endl;
+  }
+
+  std::cout << GridLogMessage << "Done" << std::endl;
+  Grid_finalize();
+  return 0;
+}

examples/Example_spec_kryschur.cc

@@ -1,498 +0,0 @@
-/*************************************************************************************
-
-    Runs the Krylov-Schur algorithm on a (pre-conditioned) domain-wall fermion operator 
-    to determine part of its spectrum. 
-
-    Usage : 
-      $ ./Example_spec_kryschur <Nm> <Nk> <maxiter> <Nstop> <inFile> <outDir> <?rf>
-
-      Nm = Maximum size of approximation subspace.
-      Nk = Size of truncation subspace
-      maxiter = Maximum number of iterations.
-      Nstop   = Stop when Nstop eigenvalues have converged. 
-      inFile  = Gauge configuration to read in.
-      outDir  = Directory to write output to.
-      rf      = (Optional) RitzFilter to sort with. Takes in any string in 
-                  {EvalNormSmall, EvalNormLarge, EvalReSmall, EvalReLarge, EvalImSmall, EvalImLarge}
-    
-    Output:
-      ${outDir}/evals.txt  = Contains all eigenvalues. Each line is formatted as `$idx $eval $ritz`, where:
-                              - $idx is the index of the eigenvalue.
-                              - $eval is the eigenvalue, formated as "(re,im)".
-                              - $ritz is the Ritz estimate of the eigenvalue (deviation from being a true eigenvalue)
-      ${outDir}/evec${idx} = Eigenvector $idx written out in SCIDAC format (if LIME is enabled).
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./tests/Test_padded_cell.cc
-
-    Copyright (C) 2023
-
-    Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-    Author: Patrick Oare <poare@bnl.edu>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-#include <cstdlib>
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-#include <Grid/parallelIO/IldgIOtypes.h>
-#include <Grid/parallelIO/IldgIO.h>
-
-using namespace std;
-using namespace Grid;
-
-namespace Grid {
-
-struct LanczosParameters: Serializable {
-  GRID_SERIALIZABLE_CLASS_MEMBERS(LanczosParameters,
-		  		RealD, mass ,
-		  		RealD, mstep ,
-				Integer, Nstop,
-                                Integer, Nk,
-                                Integer, Np,
-                                Integer, ReadEvec,
-	  			RealD, resid,
-	  			RealD, ChebyLow,
-	  			RealD, ChebyHigh,
-	  			Integer, ChebyOrder)
-
-  LanczosParameters() {
-    /////////////////////////////////
-  }
-
-  template <class ReaderClass >
-  LanczosParameters(Reader<ReaderClass> & TheReader){
-    initialize(TheReader);
-  }
-
-  template < class ReaderClass >
-  void initialize(Reader<ReaderClass> &TheReader){
-    read(TheReader, "HMC", *this);
-  }
-
-
-  void print_parameters() const {
-//    std::cout << GridLogMessage << "[HMC parameters] Trajectories            : " << Trajectories << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Start trajectory        : " << StartTrajectory << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Metropolis test (on/off): " << std::boolalpha << MetropolisTest << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Thermalization trajs    : " << NoMetropolisUntil << "\n";
-//    std::cout << GridLogMessage << "[HMC parameters] Starting type           : " << StartingType << "\n";
-//    MD.print_parameters();
-  }
-
-};
-
-}
-
-template <class T> void writeFile(T& in, std::string const fname){
-#ifdef HAVE_LIME
-  // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-  std::cout << Grid::GridLogMessage << "Writes to: " << fname << std::endl;
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter WR(in.Grid()->IsBoss());
-  WR.open(fname);
-  WR.writeScidacFieldRecord(in,record,0); // Lexico
-  WR.close();
-#endif
-}
-
-/**
- * Writes the eigensystem of a Krylov Schur object to a directory.
- *
- * Parameters
- * ----------
- * std::string path
- *    Directory to write to.
- */
-template <class Field>
-void writeEigensystem(KrylovSchur<Field> KS, std::string outDir) {
-  int Nk = KS.getNk();
-  std::cout << GridLogMessage << "Writing output to directory: " << outDir << std::endl;
-
-  // Write evals
-  std::string evalPath = outDir + "/evals.txt";
-  std::ofstream fEval;
-  fEval.open(evalPath);
-  Eigen::VectorXcd evals = KS.getEvals();
-  std::vector<RealD> ritz  = KS.getRitzEstimates();
-  for (int i = 0; i < Nk; i++) {
-    // write eigenvalues and Ritz estimates
-    fEval << i << " " << evals(i) << " " << ritz[i];
-    if (i < Nk - 1) { fEval << "\n"; }
-  }
-  fEval.close();
-
-  // Write evecs (TODO: very heavy on storage costs! Don't write them all out)
-  // std::vector<Field> evecs = KS.getEvecs();
-  // for (int i = 0; i < Nk; i++) {
-  //   std::string fName = outDir + "/evec" + std::to_string(i);
-  //   writeFile(evecs[i], fName);     // using method from Grid/HMC/ComputeWilsonFlow.cc
-  // }
-}
-// Hermitize a DWF operator by squaring it
-template<class Matrix,class Field>
-class SquaredLinearOperator : public LinearOperatorBase<Field> {
-
-  public:
-  Matrix &_Mat;
-
-  public:
-    SquaredLinearOperator(Matrix &Mat): _Mat(Mat) {};
-
-    void OpDiag (const Field &in, Field &out) {    assert(0);  }
-    void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-    void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-    void Op     (const Field &in, Field &out){
-      // std::cout << "Op is overloaded as HermOp" << std::endl;
-      HermOp(in, out);
-    }
-    void AdjOp     (const Field &in, Field &out){
-      HermOp(in, out);
-    }
-    void _Op     (const Field &in, Field &out){
-      // std::cout << "Op: M "<<std::endl;
-      _Mat.M(in, out);
-    }
-    void _AdjOp     (const Field &in, Field &out){
-      // std::cout << "AdjOp: Mdag "<<std::endl;
-      _Mat.Mdag(in, out);
-    }
-    void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-    void HermOp(const Field &in, Field &out){
-      // std::cout << "HermOp: Mdag M Mdag M"<<std::endl;
-      Field tmp(in.Grid());
-      _Op(in,tmp);
-      _AdjOp(tmp,out);
-    }
-};
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    //    _Mat.M(in,tmp);
-    //    _PV.Mdag(tmp,out);
-    //    _PV.M(out,tmp);
-    //    _Mat.Mdag(tmp,out);
-    Op(in,tmp);
-    AdjOp(tmp,out);
-    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix, class Field>
-class ShiftedComplexPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  ComplexD shift;
-public:
-ShiftedComplexPVdagMLinearOperator(ComplexD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-  
-  void resetShift(ComplexD newShift) {
-    shift = newShift;
-  }
-};
-
-#if 0
-template<class Fobj,class CComplex,int nbasis>
-class MGPreconditioner : public LinearFunction< Lattice<Fobj> > {
-public:
-  using LinearFunction<Lattice<Fobj> >::operator();
-
-  typedef Aggregation<Fobj,CComplex,nbasis> Aggregates;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::FineField    FineField;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseVector CoarseVector;
-  typedef typename Aggregation<Fobj,CComplex,nbasis>::CoarseMatrix CoarseMatrix;
-  typedef LinearOperatorBase<FineField>                            FineOperator;
-  typedef LinearFunction    <FineField>                            FineSmoother;
-  typedef LinearOperatorBase<CoarseVector>                         CoarseOperator;
-  typedef LinearFunction    <CoarseVector>                         CoarseSolver;
-  Aggregates     & _Aggregates;
-  FineOperator   & _FineOperator;
-  FineSmoother   & _PreSmoother;
-  FineSmoother   & _PostSmoother;
-  CoarseOperator & _CoarseOperator;
-  CoarseSolver   & _CoarseSolve;
-
-  int    level;  void Level(int lv) {level = lv; };
-
-  MGPreconditioner(Aggregates &Agg,
-		   FineOperator &Fine,
-		   FineSmoother &PreSmoother,
-		   FineSmoother &PostSmoother,
-		   CoarseOperator &CoarseOperator_,
-		   CoarseSolver &CoarseSolve_)
-    : _Aggregates(Agg),
-      _FineOperator(Fine),
-      _PreSmoother(PreSmoother),
-      _PostSmoother(PostSmoother),
-      _CoarseOperator(CoarseOperator_),
-      _CoarseSolve(CoarseSolve_),
-      level(1)  {  }
-
-  virtual void operator()(const FineField &in, FineField & out) 
-  {
-    GridBase *CoarseGrid = _Aggregates.CoarseGrid;
-    //    auto CoarseGrid = _CoarseOperator.Grid();
-    CoarseVector Csrc(CoarseGrid);
-    CoarseVector Csol(CoarseGrid);
-    FineField vec1(in.Grid());
-    FineField vec2(in.Grid());
-
-    std::cout<<GridLogMessage << "Calling PreSmoother " <<std::endl;
-
-    //    std::cout<<GridLogMessage << "Calling PreSmoother input residual "<<norm2(in) <<std::endl;
-    double t;
-    // Fine Smoother
-    //    out = in;
-    out = Zero();
-    t=-usecond();
-    _PreSmoother(in,out);
-    t+=usecond();
-
-    std::cout<<GridLogMessage << "PreSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Update the residual
-    _FineOperator.Op(out,vec1);  sub(vec1, in ,vec1);   
-    //    std::cout<<GridLogMessage <<"Residual-1 now " <<norm2(vec1)<<std::endl;
-
-    // Fine to Coarse 
-    t=-usecond();
-    _Aggregates.ProjectToSubspace  (Csrc,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Project to coarse took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Coarse correction
-    t=-usecond();
-    Csol = Zero();
-    _CoarseSolve(Csrc,Csol);
-    //Csol=Zero();
-    t+=usecond();
-    std::cout<<GridLogMessage << "Coarse solve took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Coarse to Fine
-    t=-usecond();  
-    //    _CoarseOperator.PromoteFromSubspace(_Aggregates,Csol,vec1);
-    _Aggregates.PromoteFromSubspace(Csol,vec1); 
-    add(out,out,vec1);
-    t+=usecond();
-    std::cout<<GridLogMessage << "Promote to this level took "<< t/1000.0<< "ms" <<std::endl;
-
-    // Residual
-    _FineOperator.Op(out,vec1);  sub(vec1 ,in , vec1);  
-    //    std::cout<<GridLogMessage <<"Residual-2 now " <<norm2(vec1)<<std::endl;
-
-    // Fine Smoother
-    t=-usecond();
-    //    vec2=vec1;
-    vec2=Zero();
-    _PostSmoother(vec1,vec2);
-    t+=usecond();
-    std::cout<<GridLogMessage << "PostSmoother took "<< t/1000.0<< "ms" <<std::endl;
-
-    add( out,out,vec2);
-    std::cout<<GridLogMessage << "Done " <<std::endl;
-  }
-};
-#endif
-
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  // Usage : $ ./Example_spec_kryschur <Nm> <Nk> <maaxiter> <Nstop> <inFile> <outDir>
-  std::string NmStr      = argv[1];
-  std::string NkStr      = argv[2];
-  std::string maxIterStr = argv[3];
-  std::string NstopStr   = argv[4];
-  std::string file       = argv[5];
-  std::string outDir     = argv[6];
-
-  RitzFilter RF;
-  if (argc == 8) {
-    std::string rf       = argv[7];
-    RF = selectRitzFilter(rf);
-  } else {
-    RF = EvalReSmall;
-  }
-  std::cout << "Sorting eigenvalues using " << rfToString(RF) << std::endl;
-
-  //const int Ls=16;
-  const int Ls = 8;
-
-//   GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  //std::vector<int> lat_size {16, 16, 16, 32};
-  std::vector<int> lat_size {8, 8, 8, 8};
-  std::cout << "Lattice size: " << lat_size << std::endl;
-  GridCartesian * UGrid = SpaceTimeGrid::makeFourDimGrid(lat_size, 
-								          GridDefaultSimd(Nd,vComplex::Nsimd()),
-								          GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-
-  GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-
-  std::vector<int> seeds4({1,2,3,4});
-  std::vector<int> seeds5({5,6,7,8});
-  GridParallelRNG          RNG5(FGrid);   RNG5.SeedFixedIntegers(seeds5);
-  GridParallelRNG          RNG4(UGrid);   RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeFermion    src(FGrid); random(RNG5,src);
-  LatticeGaugeField Umu(UGrid);
-
-  FieldMetaData header;
-  NerscIO::readConfiguration(Umu,header,file);
-
-  // RealD mass=0.01;
-  RealD mass=0.001;
-  RealD M5=1.8;
-
-  DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
-  DomainWallFermionD Dpv(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,1.0,M5);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-
-  typedef PVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> PVdagM_t;
-  typedef ShiftedPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedPVdagM_t;
-  typedef ShiftedComplexPVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> ShiftedComplexPVdagM_t;
-  PVdagM_t PVdagM(Ddwf, Dpv);
-  ShiftedPVdagM_t ShiftedPVdagM(0.1,Ddwf,Dpv);
-  SquaredLinearOperator<DomainWallFermionD, LatticeFermionD> Dsq (Ddwf);
-  NonHermitianLinearOperator<DomainWallFermionD, LatticeFermionD> DLinOp (Ddwf);
-
-  int Nm = std::stoi(NmStr);
-  int Nk = std::stoi(NkStr);
-  int maxIter = std::stoi(maxIterStr);
-  int Nstop = std::stoi(NstopStr);
-
-  std::cout << GridLogMessage << "Runnning Krylov Schur. Nm = " << Nm << ", Nk = " << Nk << ", maxIter = " << maxIter 
-                  << ", Nstop = " << Nstop << std::endl;
-  
-  KrylovSchur KrySchur (PVdagM, FGrid, 1e-8, RF);      // use preconditioned PV^\dag D_{dwf}
-  // KrylovSchur KrySchur (DLinOp, FGrid, 1e-8, RF);         // use D_{dwf}
-  KrySchur(src, maxIter, Nm, Nk, Nstop);
-
-  std::cout<<GridLogMessage << "*******************************************" << std::endl;
-  std::cout<<GridLogMessage << "***************** RESULTS *****************" << std::endl;
-  std::cout<<GridLogMessage << "*******************************************" << std::endl;
-
-  std::cout << GridLogMessage << "Krylov Schur eigenvalues: " << std::endl << KrySchur.getEvals() << std::endl;
-
-  writeEigensystem(KrySchur, outDir);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_wilson_evecs.cc

@@ -1,383 +0,0 @@
-/*************************************************************************************
-
-    Script for studying the Wilson eigenvectors resulting from the Krylov-Schur process. 
-
-    Usage : 
-      $ ./Example_spec_kryschur <Nm> <Nk> <maxiter> <Nstop> <inFile> <outDir> <?rf>
-
-      Nm = Maximum size of approximation subspace.
-      Nk = Size of truncation subspace
-      maxiter = Maximum number of iterations.
-      Nstop   = Stop when Nstop eigenvalues have converged. 
-      inFile  = Gauge configuration to read in.
-      outDir  = Directory to write output to.
-      rf      = (Optional) RitzFilter to sort with. Takes in any string in 
-                  {EvalNormSmall, EvalNormLarge, EvalReSmall, EvalReLarge, EvalImSmall, EvalImLarge}
-    
-    Output:
-      ${outDir}/evals.txt  = Contains all eigenvalues. Each line is formatted as `$idx $eval $ritz`, where:
-                              - $idx is the index of the eigenvalue.
-                              - $eval is the eigenvalue, formated as "(re,im)".
-                              - $ritz is the Ritz estimate of the eigenvalue (deviation from being a true eigenvalue)
-      ${outDir}/evec${idx} = Eigenvector $idx written out in SCIDAC format (if LIME is enabled).
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./tests/Test_padded_cell.cc
-
-    Copyright (C) 2023
-
-    Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-    Author: Patrick Oare <poare@bnl.edu>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-#include <cstdlib>
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-#include <Grid/parallelIO/IldgIOtypes.h>
-#include <Grid/parallelIO/IldgIO.h>
-
-using namespace std;
-using namespace Grid;
-
-template <class T> void writeFile(T& in, std::string const fname){  
-  #ifdef HAVE_LIME
-    // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-    std::cout << Grid::GridLogMessage << "Writes to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter WR(in.Grid()->IsBoss());
-    WR.open(fname);
-    WR.writeScidacFieldRecord(in,record,0); // Lexico
-    WR.close();
-  #endif
-}
-
-template <class T> void readFile(T& out, std::string const fname){  
-  #ifdef HAVE_LIME
-    // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-    std::cout << Grid::GridLogMessage << "Reads at: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    // Grid::ScidacReader SR(out.Grid()->IsBoss());
-    Grid::ScidacReader SR;
-    SR.open(fname);
-    SR.readScidacFieldRecord(out, record);
-    SR.close();
-  #endif
-}
-
-/**
- * Writes the eigensystem of a Krylov Schur object to a directory. 
- * 
- * Parameters
- * ----------
- * std::string path
- *    Directory to write to. 
- */
-template <class Field>
-void writeEigensystem(KrylovSchur<Field> KS, std::string outDir) {
-  int Nk = KS.getNk();
-  std::cout << GridLogMessage << "Writing output to directory: " << outDir << std::endl;
-  
-  // Write evals
-  std::string evalPath = outDir + "/evals.txt";
-  std::ofstream fEval;
-  fEval.open(evalPath);
-  Eigen::VectorXcd evals = KS.getEvals();
-  std::vector<RealD> ritz  = KS.getRitzEstimates();
-  for (int i = 0; i < Nk; i++) {
-    // write eigenvalues and Ritz estimates
-    fEval << i << " " << evals(i) << " " << ritz[i];
-    if (i < Nk - 1) { fEval << "\n"; }
-  }
-  fEval.close();
-  
-  // Write evecs
-  int Nevecs = Nk;          // don't write all of them
-  std::vector<Field> evecs = KS.getEvecs();
-  for (int i = 0; i < Nevecs; i++) {
-    std::string fName = outDir + "/evec" + std::to_string(i);
-    writeFile(evecs[i], fName);     // using method from Grid/HMC/ComputeWilsonFlow.cc
-  }
-}
-
-// Hermitize a DWF operator by squaring it
-template<class Matrix,class Field>
-class SquaredLinearOperator : public LinearOperatorBase<Field> {
-
-  public:
-  Matrix &_Mat;
-
-  public:
-    SquaredLinearOperator(Matrix &Mat): _Mat(Mat) {};
-
-    void OpDiag (const Field &in, Field &out) {    assert(0);  }
-    void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-    void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-    void Op     (const Field &in, Field &out){
-      // std::cout << "Op is overloaded as HermOp" << std::endl;
-      HermOp(in, out);
-    }
-    void AdjOp     (const Field &in, Field &out){
-      HermOp(in, out);
-    }
-    void _Op     (const Field &in, Field &out){
-      // std::cout << "Op: M "<<std::endl;
-      _Mat.M(in, out);
-    }
-    void _AdjOp     (const Field &in, Field &out){
-      // std::cout << "AdjOp: Mdag "<<std::endl;
-      _Mat.Mdag(in, out);
-    }
-    void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-    void HermOp(const Field &in, Field &out){
-      // std::cout << "HermOp: Mdag M Mdag M"<<std::endl;
-      Field tmp(in.Grid());
-      _Op(in,tmp);
-      _AdjOp(tmp,out);
-    }
-};
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    //    _Mat.M(in,tmp);
-    //    _PV.Mdag(tmp,out);
-    //    _PV.M(out,tmp);
-    //    _Mat.Mdag(tmp,out);
-    Op(in,tmp);
-    AdjOp(tmp,out);
-    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix, class Field>
-class ShiftedComplexPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  ComplexD shift;
-public:
-ShiftedComplexPVdagMLinearOperator(ComplexD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-  
-  void resetShift(ComplexD newShift) {
-    shift = newShift;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  // Usage : $ ./Example_wilson_evecs ${inFile}
-  std::string file       = argv[1];
-
-  const int Ls=16;
-
-//   GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  //std::vector<int> lat_size {16, 16, 16, 32};
-  std::vector<int> lat_size {32, 32, 32, 32};
-  std::cout << "Lattice size: " << lat_size << std::endl;
-  GridCartesian * UGrid = SpaceTimeGrid::makeFourDimGrid(lat_size, 
-								          GridDefaultSimd(Nd,vComplex::Nsimd()),
-								          GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  // GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  // GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-  GridCartesian * FGrid = UGrid;
-  GridRedBlackCartesian * FrbGrid = UrbGrid;
-
-  std::vector<int> seeds4({1,2,3,4});
-  GridParallelRNG RNG4(UGrid);
-  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeFermion    src(FGrid); random(RNG4, src);
-  LatticeGaugeField Umu(UGrid);
-
-  FieldMetaData header;
-  NerscIO::readConfiguration(Umu, header, file);
-
-  std::cout << GridLogMessage << "Loaded configuration" << std::endl;
-
-  // RealD mass = 0.01;
-  RealD M5 = 1.8;
-
-  // Wilson mass
-  RealD mass = -1.6;
-
-  std::cout << GridLogMessage << "masses specified" << std::endl;
-
-  std::vector<Complex> boundary = {1,1,1,-1};
-  WilsonFermionD::ImplParams Params(boundary);
-
-  // DomainWallFermionD Ddwf(Umu, *FGrid, *FrbGrid, *UGrid, *UrbGrid, mass, M5);
-  // NonHermitianLinearOperator<DomainWallFermionD, LatticeFermionD> DLinOp (Ddwf);
-
-  // WilsonFermionD Dwilson(Umu, *FGrid, *FrbGrid, mass);
-  WilsonFermionD Dwilson(Umu, *UGrid, *UrbGrid, mass, Params);
-  NonHermitianLinearOperator<WilsonFermionD, LatticeFermionD> DLinOp (Dwilson);
-
-  std::cout << GridLogMessage << "Dirac operator defined" << std::endl;
-
-  std::string eigenPath = "/home/poare/lqcd/multigrid/spectra/32cube-rho0.124-tau4/U_smr_3.000000/Nm72_Nk24_8111835.aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov/";
-
-  std::cout << GridLogMessage << "Loading eigenvalues" << std::endl;
-  std::ifstream evalFile(eigenPath + "evals.txt");
-  std::string str;
-  std::vector<ComplexD> evals;
-  while (std::getline(evalFile, str)) {
-      std::cout << GridLogMessage << "Reading line: " << str << std::endl;
-      int i1 = str.find("(") + 1;
-      int i2 = str.find(",") + 1;
-      int i3 = str.find(")");
-      std::cout << "i1,i2,i3 = " << i1 << "," << i2 << "," << i3 << std::endl;
-      std::string reStr = str.substr(i1, i2 - i1);
-      std::string imStr = str.substr(i2, i3 - i2);
-      std::cout << GridLogMessage << "Parsed re = " << reStr << " and im = " << imStr << std::endl;
-      // ComplexD z (std::stof(reStr), std::stof(imStr));
-      ComplexD z (std::stod(reStr), std::stod(imStr));
-      evals.push_back(z);
-  }
-  std::cout << GridLogMessage << "Eigenvalues: " << evals << std::endl;
-
-  int Nevecs = 24;
-  std::vector<LatticeFermion> evecs;
-  LatticeFermion evec (FGrid);
-  for (int i = 0; i < Nevecs; i++) {
-    std::string evecPath = eigenPath + "evec" + std::to_string(i);
-    readFile(evec, evecPath);
-    evecs.push_back(evec);
-  }
-  std::cout << GridLogMessage << "Evecs loaded" << std::endl;
-
-  // Compute < evec | D - \lambda | evec >
-  std::cout << GridLogMessage << "Testing eigenvectors" << std::endl;
-  LatticeFermion Devec (FGrid);
-  ComplexD ritz;
-  for (int i = 0; i < Nevecs; i++) {
-    Devec = Zero();
-    DLinOp.Op(evecs[i], Devec);
-    ritz = std::sqrt(norm2(Devec - evals[i] * evecs[i]));
-    std::cout << GridLogMessage << "i = " << i << ", || (D - lambda) |vi> || = " << ritz << std::endl;
-  }
-  // Eigen::MatrixXcd Dw_evecs;
-  // Dw_evecs = Eigen::MatrixXcd::Zero(Nevecs, Nevecs);
-  // for (int i = 0; i < Nevecs; i++) {
-  //   Linop.Op(evecs[i], Devec);
-  //   for (int j = 0; j < Nevecs; j++) {
-
-  //   }
-  // }
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/Example_wilson_spectrum.cc

@@ -1,374 +0,0 @@
-/*************************************************************************************
-
-    Runs the Krylov-Schur algorithm on a Wilson fermion operator to determine part of its spectrum. 
-
-    TODO rename this file: really is running the topology change jobs on Aurora. 
-
-    Usage : 
-      $ ./Example_spec_kryschur <Nm> <Nk> <maxiter> <Nstop> <inFile> <outDir> <?rf>
-
-      Nm = Maximum size of approximation subspace.
-      Nk = Size of truncation subspace
-      maxiter = Maximum number of iterations.
-      Nstop   = Stop when Nstop eigenvalues have converged. 
-      inFile  = Gauge configuration to read in.
-      outDir  = Directory to write output to.
-      rf      = (Optional) RitzFilter to sort with. Takes in any string in 
-                  {EvalNormSmall, EvalNormLarge, EvalReSmall, EvalReLarge, EvalImSmall, EvalImLarge}
-    
-    Output:
-      ${outDir}/evals.txt  = Contains all eigenvalues. Each line is formatted as `$idx $eval $ritz`, where:
-                              - $idx is the index of the eigenvalue.
-                              - $eval is the eigenvalue, formated as "(re,im)".
-                              - $ritz is the Ritz estimate of the eigenvalue (deviation from being a true eigenvalue)
-      ${outDir}/evec${idx} = Eigenvector $idx written out in SCIDAC format (if LIME is enabled).
-
-    Grid physics library, www.github.com/paboyle/Grid 
-
-    Source file: ./tests/Test_padded_cell.cc
-
-    Copyright (C) 2023
-
-    Author: Peter Boyle <paboyle@ph.ed.ac.uk>
-    Author: Patrick Oare <poare@bnl.edu>
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License along
-    with this program; if not, write to the Free Software Foundation, Inc.,
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-    See the full license in the file "LICENSE" in the top level distribution directory
-    *************************************************************************************/
-    /*  END LEGAL */
-
-#include <cstdlib>
-
-#include <Grid/Grid.h>
-#include <Grid/lattice/PaddedCell.h>
-#include <Grid/stencil/GeneralLocalStencil.h>
-
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidual.h>
-#include <Grid/algorithms/iterative/PrecGeneralisedConjugateResidualNonHermitian.h>
-#include <Grid/algorithms/iterative/BiCGSTAB.h>
-
-#include <Grid/parallelIO/IldgIOtypes.h>
-#include <Grid/parallelIO/IldgIO.h>
-
-using namespace std;
-using namespace Grid;
-
-template <class T> void writeFile(T& in, std::string const fname){  
-  #ifdef HAVE_LIME
-    // Ref: https://github.com/paboyle/Grid/blob/feature/scidac-wp1/tests/debug/Test_general_coarse_hdcg_phys48.cc#L111
-    std::cout << Grid::GridLogMessage << "Writes to: " << fname << std::endl;
-    Grid::emptyUserRecord record;
-    Grid::ScidacWriter WR(in.Grid()->IsBoss());
-    WR.open(fname);
-    WR.writeScidacFieldRecord(in,record,0); // Lexico
-    WR.close();
-  #endif
-}
-
-/**
- * Writes the eigensystem of a Krylov Schur object to a directory. 
- * 
- * Parameters
- * ----------
- * std::string path
- *    Directory to write to. 
- */
-template <class Field>
-void writeEigensystem(KrylovSchur<Field> KS, std::string outDir) {
-  int Nk = KS.getNk();
-  std::cout << GridLogMessage << "Writing output to directory: " << outDir << std::endl;
-  
-  // Write evals
-  std::string evalPath = outDir + "/evals.txt";
-  std::ofstream fEval;
-  fEval.open(evalPath);
-  Eigen::VectorXcd evals = KS.getEvals();
-  std::vector<RealD> ritz  = KS.getRitzEstimates();
-  for (int i = 0; i < Nk; i++) {
-    // write eigenvalues and Ritz estimates
-    fEval << i << " " << evals(i) << " " << ritz[i];
-    if (i < Nk - 1) { fEval << "\n"; }
-  }
-  fEval.close();
-  
-  // Write evecs
-  int Nevecs = Nk;          // don't write all of them
-  std::vector<Field> evecs = KS.getEvecs();
-  for (int i = 0; i < Nevecs; i++) {
-    std::string fName = outDir + "/evec" + std::to_string(i);
-    writeFile(evecs[i], fName);     // using method from Grid/HMC/ComputeWilsonFlow.cc
-  }
-}
-
-// Hermitize a DWF operator by squaring it
-template<class Matrix,class Field>
-class SquaredLinearOperator : public LinearOperatorBase<Field> {
-
-  public:
-  Matrix &_Mat;
-
-  public:
-    SquaredLinearOperator(Matrix &Mat): _Mat(Mat) {};
-
-    void OpDiag (const Field &in, Field &out) {    assert(0);  }
-    void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-    void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-    void Op     (const Field &in, Field &out){
-      // std::cout << "Op is overloaded as HermOp" << std::endl;
-      HermOp(in, out);
-    }
-    void AdjOp     (const Field &in, Field &out){
-      HermOp(in, out);
-    }
-    void _Op     (const Field &in, Field &out){
-      // std::cout << "Op: M "<<std::endl;
-      _Mat.M(in, out);
-    }
-    void _AdjOp     (const Field &in, Field &out){
-      // std::cout << "AdjOp: Mdag "<<std::endl;
-      _Mat.Mdag(in, out);
-    }
-    void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-    void HermOp(const Field &in, Field &out){
-      // std::cout << "HermOp: Mdag M Mdag M"<<std::endl;
-      Field tmp(in.Grid());
-      _Op(in,tmp);
-      _AdjOp(tmp,out);
-    }
-};
-
-template<class Matrix,class Field>
-class PVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-public:
-  PVdagMLinearOperator(Matrix &Mat,Matrix &PV): _Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(in,tmp);
-    _Mat.Mdag(tmp,out);
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    //    _Mat.M(in,tmp);
-    //    _PV.Mdag(tmp,out);
-    //    _PV.M(out,tmp);
-    //    _Mat.Mdag(tmp,out);
-    Op(in,tmp);
-    AdjOp(tmp,out);
-    //    std::cout << "HermOp done "<<norm2(out)<<std::endl;
-  }
-};
-
-template<class Matrix,class Field>
-class ShiftedPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  RealD shift;
-public:
-  ShiftedPVdagMLinearOperator(RealD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-};
-
-template<class Matrix, class Field>
-class ShiftedComplexPVdagMLinearOperator : public LinearOperatorBase<Field> {
-  Matrix &_Mat;
-  Matrix &_PV;
-  ComplexD shift;
-public:
-ShiftedComplexPVdagMLinearOperator(ComplexD _shift,Matrix &Mat,Matrix &PV): shift(_shift),_Mat(Mat),_PV(PV){};
-
-  void OpDiag (const Field &in, Field &out) {    assert(0);  }
-  void OpDir  (const Field &in, Field &out,int dir,int disp) {    assert(0);  }
-  void OpDirAll  (const Field &in, std::vector<Field> &out){    assert(0);  };
-  void Op     (const Field &in, Field &out){
-    std::cout << "Op: PVdag M "<<std::endl;
-    Field tmp(in.Grid());
-    _Mat.M(in,tmp);
-    _PV.Mdag(tmp,out);
-    out = out + shift * in;
-  }
-  void AdjOp     (const Field &in, Field &out){
-    std::cout << "AdjOp: Mdag PV "<<std::endl;
-    Field tmp(in.Grid());
-    _PV.M(tmp,out);
-    _Mat.Mdag(in,tmp);
-    out = out + shift * in;
-  }
-  void HermOpAndNorm(const Field &in, Field &out,RealD &n1,RealD &n2){    assert(0);  }
-  void HermOp(const Field &in, Field &out){
-    std::cout << "HermOp: Mdag PV PVdag M"<<std::endl;
-    Field tmp(in.Grid());
-    Op(in,tmp);
-    AdjOp(tmp,out);
-  }
-  
-  void resetShift(ComplexD newShift) {
-    shift = newShift;
-  }
-};
-
-int main (int argc, char ** argv)
-{
-  Grid_init(&argc,&argv);
-
-  // Usage : $ ./Example_spec_kryschur <Nm> <Nk> <maaxiter> <Nstop> <inFile> <outDir>
-  std::string NmStr      = argv[1];
-  std::string NkStr      = argv[2];
-  std::string maxIterStr = argv[3];
-  std::string NstopStr   = argv[4];
-  std::string file       = argv[5];
-  std::string outDir     = argv[6];
-
-  // RitzFilter RF;
-  // if (argc == 8) {
-  //   std::string rf       = argv[7];
-  //   RF = selectRitzFilter(rf);
-  // } else {
-  //   RF = EvalReSmall;
-  // }
-  // RitzFilter RF;
-  std::string rf       = argv[7];
-  RitzFilter RF        = selectRitzFilter(rf);
-  std::cout << "Sorting eigenvalues using " << rfToString(RF) << std::endl;
-
-  const int Ls=16;
-
-//   GridCartesian         * UGrid   = SpaceTimeGrid::makeFourDimGrid(GridDefaultLatt(), GridDefaultSimd(Nd,vComplex::Nsimd()),GridDefaultMpi());
-  //std::vector<int> lat_size {16, 16, 16, 32};
-  std::vector<int> lat_size {32, 32, 32, 32};
-  std::cout << "Lattice size: " << lat_size << std::endl;
-  GridCartesian * UGrid = SpaceTimeGrid::makeFourDimGrid(lat_size, 
-								          GridDefaultSimd(Nd,vComplex::Nsimd()),
-								          GridDefaultMpi());
-  GridRedBlackCartesian * UrbGrid = SpaceTimeGrid::makeFourDimRedBlackGrid(UGrid);
-  // GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,UGrid);
-  // GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,UGrid);
-  GridCartesian * FGrid = UGrid;
-  GridRedBlackCartesian * FrbGrid = UrbGrid;
-
-  std::vector<int> seeds4({1,2,3,4});
-  GridParallelRNG RNG4(UGrid);
-  RNG4.SeedFixedIntegers(seeds4);
-
-  LatticeFermion    src(FGrid); random(RNG4, src);
-  LatticeGaugeField Umu(UGrid);
-
-  FieldMetaData header;
-  NerscIO::readConfiguration(Umu, header, file);
-
-  std::cout << GridLogMessage << "Loaded configuration" << std::endl;
-
-  // RealD mass = 0.01;
-  RealD M5 = 1.8;
-
-  // Wilson mass
-  RealD mass = -1.6;
-
-  std::cout << GridLogMessage << "masses specified" << std::endl;
-
-  std::vector<Complex> boundary = {1,1,1,-1};
-  WilsonFermionD::ImplParams Params(boundary);
-
-  // DomainWallFermionD Ddwf(Umu, *FGrid, *FrbGrid, *UGrid, *UrbGrid, mass, M5);
-  // NonHermitianLinearOperator<DomainWallFermionD, LatticeFermionD> DLinOp (Ddwf);
-
-  // WilsonFermionD Dwilson(Umu, *FGrid, *FrbGrid, mass);
-  WilsonFermionD Dwilson(Umu, *UGrid, *UrbGrid, mass, Params);
-  NonHermitianLinearOperator<WilsonFermionD, LatticeFermionD> DLinOp (Dwilson);
-
-  std::cout << GridLogMessage << "Dirac operator defined" << std::endl;
-
-  // Define PV^dag D (if we want)
-  // DomainWallFermionD Dpv(Umu, *FGrid, *FrbGrid, *UGrid, *UrbGrid, 1.0, M5);
-  // typedef PVdagMLinearOperator<DomainWallFermionD,LatticeFermionD> PVdagM_t;
-  // PVdagM_t PVdagM(Ddwf, Dpv);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-
-  // SquaredLinearOperator<WilsonFermionD, LatticeFermionD> Dsq (DWilson);
-  // NonHermitianLinearOperator<WilsonFermionD, LatticeFermionD> DLinOp (DWilson);
-
-  int Nm = std::stoi(NmStr);
-  int Nk = std::stoi(NkStr);
-  int maxIter = std::stoi(maxIterStr);
-  int Nstop = std::stoi(NstopStr);
-
-  std::cout << GridLogMessage << "Runnning Krylov Schur. Nm = " << Nm << ", Nk = " << Nk << ", maxIter = " << maxIter 
-                  << ", Nstop = " << Nstop << std::endl;
-  
-  // KrylovSchur KrySchur (PVdagM, FGrid, 1e-8, RF);         // use PV^\dag M
-  KrylovSchur KrySchur (DLinOp, FGrid, 1e-8, RF);         // use Ddwf
-  KrySchur(src, maxIter, Nm, Nk, Nstop);
-
-  std::cout << GridLogMessage << "Checking eigensystem." << std::endl;
-  KrySchur.checkRitzEstimate();
-
-  std::cout<<GridLogMessage << "*******************************************" << std::endl;
-  std::cout<<GridLogMessage << "***************** RESULTS *****************" << std::endl;
-  std::cout<<GridLogMessage << "*******************************************" << std::endl;
-
-  std::cout << GridLogMessage << "Krylov Schur eigenvalues: " << std::endl << KrySchur.getEvals() << std::endl;
-
-  writeEigensystem(KrySchur, outDir);
-
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage<<"*******************************************"<<std::endl;
-  std::cout<<GridLogMessage<<std::endl;
-  std::cout<<GridLogMessage << "Done "<< std::endl;
-
-  Grid_finalize();
-  return 0;
-}

examples/LanParams.xml

@@ -1,22 +0,0 @@
-<?xml version="1.0"?>
-<grid>
-  <LanczosParameters>
-    <mass>0</mass>
-    <mstep>-0.025</mstep>
-    <M5>1.8</M5>
-    <Ls>48</Ls>
-    <Nstop>100</Nstop>
-    <Nk>100</Nk>
-    <Np>100</Np>
-    <ReadEvec>0</ReadEvec>
-    <maxIter>1000</maxIter>
-    <reorthog>1</reorthog>
-    <Nblock>4</Nblock>
-    <verify>0</verify>
-    <shift>1.5</shift>
-    <resid>1e-8</resid>
-    <ChebyLow>1</ChebyLow>
-    <ChebyHigh>100</ChebyHigh>
-    <ChebyOrder>51</ChebyOrder>
-  </LanczosParameters>
-</grid>

examples/fft5d.cc

@@ -1,469 +0,0 @@
-/*************************************************************************************
-
-  fft5d.cc — Fourier analysis of a time series of 4-D lattice scalar fields.
-
-  Designed for force-norm files from FTHMC (one RealD per site, SCIDAC or binary).
-  Assembles a (4+1)-D structure [trajectory][t][z][y][x] using Grid lattice fields
-  and performs three FFT analyses:
-
-    --fft spatial
-        4-D spatial FFT using Grid's FFT class (MPI+GPU parallel via Cshift/cufft).
-        Outputs shell-averaged P(|k|^2) averaged over trajectories, and a
-        per-mode table P(kt,kz,ky,kx).
-
-    --fft traj
-        1-D trajectory-axis FFT at each lattice site using FFTW3 locally at each
-        MPI rank.  Outputs site-averaged P(f_traj) and autocorrelation C(lag).
-        Results are combined with GlobalSumVector.
-
-    --fft all
-        Both of the above, plus a 2-D cross spectrum P(f_traj, |k_spatial|^2):
-        spatial FFT per trajectory, then trajectory FFTW on the momentum-space series.
-
-  Normalisations:
-    Spatial FFT  (volume V): P_spatial(k) = |F(k)|^2 / V^2
-      => (1/V) * sum_k P(k) = site mean-square per trajectory
-    Trajectory FFT (Ntraj):  P_traj(f) = |F(f)|^2 / Ntraj^2
-      => sum_f P(f) = site mean-square over trajectories
-
-  Build: add to examples/Make.inc (see bottom of this file), then make.
-
-  Usage (follows Grid conventions):
-    fft5d --grid Lx.Ly.Lz.Lt [--mpi Px.Py.Pz.Pt] \
-          [--fft spatial|traj|all] [--format scidac|binary] \
-          [--output PREFIX] file1 file2 ...
-
-*************************************************************************************/
-
-#include <Grid/Grid.h>
-// Grid's FFT.h uses cufft on CUDA builds; for the trajectory axis we also need
-// CPU-side FFTW3 (already linked via -lfftw3 in GRID_LIBS).
-#include <fftw3.h>
-
-#include <algorithm>
-#include <cassert>
-#include <cmath>
-#include <fstream>
-#include <iostream>
-#include <string>
-#include <vector>
-
-using namespace Grid;
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Global coordinate of a local site (osite, lane) on this MPI rank
-// ─────────────────────────────────────────────────────────────────────────────
-static void globalCoor(int osite, int lane, GridCartesian* g, Coordinate& gc)
-{
-    Coordinate oc, ic;
-    Lexicographic::CoorFromIndex(oc, osite, g->_rdimensions);
-    Lexicographic::CoorFromIndex(ic, lane,  g->_simd_layout);
-    for (int d = 0; d < Nd; d++)
-        gc[d] = g->_processor_coor[d] * g->_ldimensions[d]
-              + oc[d] * g->_simd_layout[d] + ic[d];
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Shell-average a power LatticeRealD over |k|^2 bins (MPI-aware via GlobalSumVector)
-// ─────────────────────────────────────────────────────────────────────────────
-static void shellAverage(const LatticeRealD& power, GridCartesian* g,
-                         double norm, std::ostream& ofs)
-{
-    const int Nsimd = vRealD::Nsimd();
-    Coordinate fdims = g->_fdimensions;
-    int maxk2 = 0;
-    for (int d = 0; d < Nd; d++) { int h = fdims[d]/2; maxk2 += h*h; }
-
-    std::vector<RealD> psum(maxk2+1, 0.0), cnt(maxk2+1, 0.0);
-    {
-        auto pv = power.View(CpuRead);
-        Coordinate gc(Nd);
-        for (int os = 0; os < (int)g->oSites(); os++) {
-            for (int lane = 0; lane < Nsimd; lane++) {
-                globalCoor(os, lane, g, gc);
-                int k2 = 0;
-                for (int d = 0; d < Nd; d++) {
-                    int kd = std::min(gc[d], fdims[d] - gc[d]);
-                    k2 += kd*kd;
-                }
-                psum[k2] += (RealD)extractLane(lane, pv[os]);
-                cnt [k2] += 1.0;
-            }
-        }
-    }
-    g->GlobalSumVector(psum.data(), (int)psum.size());
-    g->GlobalSumVector(cnt .data(), (int)cnt .size());
-
-    for (int k2 = 0; k2 <= maxk2; k2++)
-        if (cnt[k2] > 0)
-            ofs << k2 << "  " << std::sqrt((double)k2) << "  "
-                << (psum[k2] / cnt[k2]) * norm << "\n";
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Convert LatticeRealD → LatticeComplexD (zero imaginary part)
-// ─────────────────────────────────────────────────────────────────────────────
-static LatticeComplexD toComplex(const LatticeRealD& r)
-{
-    std::vector<RealD>    lr;
-    unvectorizeToLexOrdArray(lr, r);
-    std::vector<ComplexD> lc(lr.size());
-    for (size_t i = 0; i < lr.size(); i++) lc[i] = ComplexD(lr[i], 0.0);
-    LatticeComplexD c(r.Grid());
-    vectorizeFromLexOrdArray(lc, c);
-    return c;
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// File readers
-// ─────────────────────────────────────────────────────────────────────────────
-static LatticeComplexD readScidac(const std::string& fname, GridCartesian* g)
-{
-    LatticeRealD field(g);
-    emptyUserRecord rec;
-    ScidacReader RD;
-    RD.open(fname);
-    RD.readScidacFieldRecord(field, rec);
-    RD.close();
-    return toComplex(field);
-}
-
-static LatticeComplexD readBinary(const std::string& fname, GridCartesian* g)
-{
-    // Raw IEEE doubles in lex order [x][y][z][t] written serially.
-    // Rank 0 reads the file, broadcasts to all ranks via GlobalSumVector.
-    Coordinate fdims = g->_fdimensions;
-    long vol4 = 1; for (int d = 0; d < Nd; d++) vol4 *= fdims[d];
-
-    std::vector<RealD> buf(vol4, 0.0);
-    if (g->IsBoss()) {
-        std::ifstream f(fname, std::ios::binary);
-        if (!f) throw std::runtime_error("Cannot open: " + fname);
-        f.read(reinterpret_cast<char*>(buf.data()), vol4 * sizeof(RealD));
-    }
-    g->GlobalSumVector(buf.data(), (int)vol4);   // broadcast from rank 0
-
-    LatticeRealD field(g);
-    Coordinate ldims = g->_ldimensions, pcoor = g->_processor_coor;
-    for (long ls = 0; ls < g->lSites(); ls++) {
-        Coordinate lc;
-        g->LocalIndexToLocalCoor(ls, lc);
-        long glex = 0, stride = 1;
-        for (int d = 0; d < Nd; d++) {
-            glex += (pcoor[d]*ldims[d] + lc[d]) * stride;
-            stride *= fdims[d];
-        }
-        pokeLocalSite(buf[glex], field, lc);
-    }
-    return toComplex(field);
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Load all files into a trajectory vector
-// ─────────────────────────────────────────────────────────────────────────────
-static std::vector<LatticeComplexD>
-loadFiles(const std::vector<std::string>& files, GridCartesian* g,
-          const std::string& fmt)
-{
-    std::vector<LatticeComplexD> traj;
-    traj.reserve(files.size());
-    for (int n = 0; n < (int)files.size(); n++) {
-        std::cout << GridLogMessage << "[" << n << "] reading " << files[n] << "\n";
-        if (fmt == "scidac") traj.push_back(readScidac(files[n], g));
-        else                 traj.push_back(readBinary (files[n], g));
-    }
-    return traj;
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Spatial FFT analysis
-// ─────────────────────────────────────────────────────────────────────────────
-static void analyzeSpatial(const std::vector<LatticeComplexD>& traj,
-                            GridCartesian* g, const std::string& pfx)
-{
-    int  Ntraj = (int)traj.size();
-    long vol4  = 1; for (int d = 0; d < Nd; d++) vol4 *= g->_fdimensions[d];
-
-    FFT theFFT(g);
-    LatticeRealD pavg(g); pavg = Zero();
-    for (int n = 0; n < Ntraj; n++) {
-        LatticeComplexD fk(g);
-        theFFT.FFT_all_dim(fk, traj[n], FFT::forward);
-        // Evaluate product before applying real() — real() is not defined for
-        // unevaluated LatticeBinaryExpression.
-        LatticeComplexD fk_sq(g); fk_sq = conjugate(fk) * fk;
-        std::vector<ComplexD> lc; unvectorizeToLexOrdArray(lc, fk_sq);
-        std::vector<RealD> lr(lc.size());
-        for (size_t i = 0; i < lc.size(); i++) lr[i] = lc[i].real();
-        LatticeRealD pk(g); vectorizeFromLexOrdArray(lr, pk);
-        pavg += pk;
-    }
-    pavg *= (1.0 / Ntraj);
-
-    // Shell-averaged spectrum
-    {
-        std::ofstream fs(pfx + "_spatial_shell.dat");
-        fs << "# k2  |k|  P_shell_avg   (P = |F|^2 / V^2 / shell_count)\n";
-        shellAverage(pavg, g, 1.0 / ((double)vol4 * vol4), fs);
-        std::cout << GridLogMessage << "Written: " << pfx << "_spatial_shell.dat\n";
-    }
-
-    // Per-mode table (rank 0 only, via peekSite)
-    if (g->IsBoss()) {
-        std::ofstream fm(pfx + "_spatial_modes.dat");
-        fm << "# kt  kz  ky  kx  k2  |k|  P_traj_avg\n";
-        Coordinate fdims = g->_fdimensions, site(Nd);
-        double norm = 1.0 / ((double)vol4 * vol4);
-        for (site[3]=0; site[3]<fdims[3]; site[3]++) { int kt=std::min(site[3],fdims[3]-site[3]);
-        for (site[2]=0; site[2]<fdims[2]; site[2]++) { int kz=std::min(site[2],fdims[2]-site[2]);
-        for (site[1]=0; site[1]<fdims[1]; site[1]++) { int ky=std::min(site[1],fdims[1]-site[1]);
-        for (site[0]=0; site[0]<fdims[0]; site[0]++) { int kx=std::min(site[0],fdims[0]-site[0]);
-            RealD p; peekSite(p, pavg, site);
-            int k2 = kt*kt+kz*kz+ky*ky+kx*kx;
-            fm << kt<<" "<<kz<<" "<<ky<<" "<<kx<<" "<<k2<<" "
-               <<std::sqrt((double)k2)<<" "<<p*norm<<"\n";
-        }}}}
-        std::cout << GridLogMessage << "Written: " << pfx << "_spatial_modes.dat\n";
-    }
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Local FFTW trajectory FFT: each rank FFTs its own local sites independently
-// ─────────────────────────────────────────────────────────────────────────────
-static void trajFFT(const std::vector<LatticeComplexD>& in,
-                    std::vector<LatticeComplexD>& out)
-{
-    int    Ntraj  = (int)in.size();
-    GridBase* g   = in[0].Grid();
-    long   lsites = g->lSites();
-
-    // fftw_complex is double[2] — std::vector<double[2]> is not supported by nvcc.
-    // Use std::vector<double> with 2x size and reinterpret_cast.
-    std::vector<double> ibuf((long)Ntraj * lsites * 2, 0.0);
-    for (int n = 0; n < Ntraj; n++) {
-        std::vector<ComplexD> lc;
-        unvectorizeToLexOrdArray(lc, in[n]);
-        for (long s = 0; s < lsites; s++) {
-            ibuf[((long)n*lsites + s)*2 + 0] = lc[s].real();
-            ibuf[((long)n*lsites + s)*2 + 1] = lc[s].imag();
-        }
-    }
-
-    // lsites transforms of length Ntraj, stride=lsites, dist=1
-    std::vector<double> obuf((long)Ntraj * lsites * 2, 0.0);
-    fftw_complex* iptr = reinterpret_cast<fftw_complex*>(ibuf.data());
-    fftw_complex* optr = reinterpret_cast<fftw_complex*>(obuf.data());
-    int n1[1] = {Ntraj};
-    fftw_plan p = fftw_plan_many_dft(
-        1, n1, (int)lsites,
-        iptr, nullptr, (int)lsites, 1,
-        optr, nullptr, (int)lsites, 1,
-        FFTW_FORWARD, FFTW_ESTIMATE);
-    fftw_execute(p);
-    fftw_destroy_plan(p);
-
-    // vector::assign triggers _M_fill_assign which needs a default constructor;
-    // Lattice has none.  Use explicit emplace_back instead.
-    out.clear(); out.reserve(Ntraj);
-    for (int k = 0; k < Ntraj; k++) out.emplace_back(g);
-    for (int k = 0; k < Ntraj; k++) {
-        std::vector<ComplexD> lc(lsites);
-        for (long s = 0; s < lsites; s++)
-            lc[s] = ComplexD(obuf[((long)k*lsites+s)*2], obuf[((long)k*lsites+s)*2+1]);
-        vectorizeFromLexOrdArray(lc, out[k]);
-    }
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Trajectory-axis FFT analysis
-// ─────────────────────────────────────────────────────────────────────────────
-static void analyzeTraj(const std::vector<LatticeComplexD>& traj,
-                         GridCartesian* g, const std::string& pfx)
-{
-    int  Ntraj = (int)traj.size();
-    int  Nf    = Ntraj / 2 + 1;
-    long vol4  = 1; for (int d = 0; d < Nd; d++) vol4 *= g->_fdimensions[d];
-
-    std::vector<LatticeComplexD> ftraj;
-    trajFFT(traj, ftraj);
-
-    // P(k) = (1/vol4) * sum_sites |F_traj(k)|^2 / Ntraj^2
-    std::vector<RealD> Pavg(Ntraj, 0.0);
-    for (int k = 0; k < Ntraj; k++) {
-        LatticeComplexD tmp(g); tmp = conjugate(ftraj[k]) * ftraj[k];
-        std::vector<ComplexD> lc; unvectorizeToLexOrdArray(lc, tmp);
-        std::vector<RealD> lr(lc.size());
-        for (size_t i = 0; i < lc.size(); i++) lr[i] = lc[i].real();
-        LatticeRealD pk(g); vectorizeFromLexOrdArray(lr, pk);
-        RealD s = 0.0;
-        for (long ls = 0; ls < g->lSites(); ls++) {
-            Coordinate lc; g->LocalIndexToLocalCoor(ls, lc);
-            RealD v; peekLocalSite(v, pk, lc);
-            s += v;
-        }
-        g->GlobalSum(s);
-        Pavg[k] = s / ((double)vol4 * Ntraj * Ntraj);
-    }
-
-    {
-        std::ofstream f(pfx + "_traj_power.dat");
-        f << "# k  freq  P_avg   (sum_k P = site mean-square)\n";
-        for (int k = 0; k < Nf; k++)
-            f << k << "  " << (double)k/Ntraj << "  " << Pavg[k] << "\n";
-        std::cout << GridLogMessage << "Written: " << pfx << "_traj_power.dat\n";
-    }
-
-    // Autocorrelation via IFFT of the power spectrum
-    std::vector<double> Pc_buf(Nf * 2, 0.0);
-    for (int k = 0; k < Nf; k++) { Pc_buf[k*2] = Pavg[k]; Pc_buf[k*2+1] = 0.0; }
-    fftw_complex* Pc = reinterpret_cast<fftw_complex*>(Pc_buf.data());
-    std::vector<double> acorr(Ntraj, 0.0);
-    fftw_plan ip = fftw_plan_dft_c2r(1, &Ntraj, Pc, acorr.data(), FFTW_ESTIMATE);
-    fftw_execute(ip); fftw_destroy_plan(ip);
-    double c0 = acorr[0] / Ntraj;
-    {
-        std::ofstream f(pfx + "_traj_autocorr.dat");
-        f << "# lag  C(lag)  C(lag)/C(0)\n";
-        for (int lag = 0; lag < Ntraj/2; lag++) {
-            double c = acorr[lag] / Ntraj;
-            f << lag << "  " << c << "  " << (c0 > 0 ? c/c0 : 0.0) << "\n";
-        }
-        std::cout << GridLogMessage << "Written: " << pfx << "_traj_autocorr.dat\n";
-    }
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Full 5-D analysis: spatial FFT → trajectory FFT → 2-D cross spectrum
-// ─────────────────────────────────────────────────────────────────────────────
-static void analyzeAll5D(const std::vector<LatticeComplexD>& traj,
-                          GridCartesian* g, const std::string& pfx)
-{
-    int  Ntraj = (int)traj.size();
-    int  Nf    = Ntraj / 2 + 1;
-    long vol4  = 1; for (int d = 0; d < Nd; d++) vol4 *= g->_fdimensions[d];
-    Coordinate fdims = g->_fdimensions;
-
-    // Step 1: spatial FFT for each trajectory
-    FFT theFFT(g);
-    std::vector<LatticeComplexD> sfft(Ntraj, LatticeComplexD(g));
-    for (int n = 0; n < Ntraj; n++)
-        theFFT.FFT_all_dim(sfft[n], traj[n], FFT::forward);
-
-    // Step 2: trajectory FFTW on the momentum-space series
-    std::vector<LatticeComplexD> tfft;
-    trajFFT(sfft, tfft);
-
-    // Step 3: P(f_traj, |k_spatial|^2) shell-averaged
-    int maxk2 = 0;
-    for (int d = 0; d < Nd; d++) { int h = fdims[d]/2; maxk2 += h*h; }
-
-    long nb = (long)Nf * (maxk2+1);
-    std::vector<RealD> p2d(nb, 0.0), cnt2d(nb, 0.0);
-    const int Nsimd = vComplexD::Nsimd();
-    double norm = 1.0 / ((double)Ntraj * Ntraj * vol4 * vol4);
-
-    for (int kf = 0; kf < Nf; kf++) {
-        LatticeComplexD tmp(g); tmp = conjugate(tfft[kf]) * tfft[kf];
-        std::vector<ComplexD> lc2; unvectorizeToLexOrdArray(lc2, tmp);
-        std::vector<RealD> lr2(lc2.size());
-        for (size_t i = 0; i < lc2.size(); i++) lr2[i] = lc2[i].real();
-        LatticeRealD pk(g); vectorizeFromLexOrdArray(lr2, pk);
-        auto pv = pk.View(CpuRead);
-        Coordinate gc(Nd);
-        for (int os = 0; os < (int)g->oSites(); os++) {
-            for (int lane = 0; lane < Nsimd; lane++) {
-                globalCoor(os, lane, g, gc);
-                int k2 = 0;
-                for (int d = 0; d < Nd; d++) {
-                    int kd = std::min(gc[d], fdims[d] - gc[d]);
-                    k2 += kd*kd;
-                }
-                long b = (long)kf*(maxk2+1) + k2;
-                p2d [b] += (RealD)extractLane(lane, pv[os]);
-                cnt2d[b] += 1.0;
-            }
-        }
-    }
-    g->GlobalSumVector(p2d .data(), (int)nb);
-    g->GlobalSumVector(cnt2d.data(), (int)nb);
-
-    if (g->IsBoss()) {
-        std::ofstream f(pfx + "_5d_cross.dat");
-        f << "# k_traj  freq_traj  k2_spatial  |k_spatial|  P_avg\n";
-        for (int kf = 0; kf < Nf; kf++) {
-            double freq = (double)kf / Ntraj;
-            for (int k2 = 0; k2 <= maxk2; k2++) {
-                long b = (long)kf*(maxk2+1) + k2;
-                if (cnt2d[b] > 0)
-                    f << kf << "  " << freq << "  " << k2 << "  "
-                      << std::sqrt((double)k2) << "  "
-                      << (p2d[b]/cnt2d[b]) * norm << "\n";
-            }
-            f << "\n";   // blank line between kf slices for gnuplot pm3d
-        }
-        std::cout << GridLogMessage << "Written: " << pfx << "_5d_cross.dat\n";
-    }
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Main
-// ─────────────────────────────────────────────────────────────────────────────
-static void usage(const char* prog)
-{
-    std::cerr
-        << "Usage: " << prog << " --grid Lx.Ly.Lz.Lt [--mpi Px.Py.Pz.Pt]\n"
-        << "       [--fft spatial|traj|all] [--format scidac|binary]\n"
-        << "       [--output PREFIX] file1 [file2 ...]\n";
-    exit(1);
-}
-
-int main(int argc, char** argv)
-{
-    Grid_init(&argc, &argv);   // consumes --grid, --mpi, etc.
-
-    std::string fftMode = "all";
-    std::string fmt     = "scidac";
-    std::string pfx     = "fft5d";
-    std::vector<std::string> files;
-
-    for (int i = 1; i < argc; i++) {
-        std::string a = argv[i];
-        if      (a == "--fft"    && i+1 < argc) fftMode = argv[++i];
-        else if (a == "--format" && i+1 < argc) fmt     = argv[++i];
-        else if (a == "--output" && i+1 < argc) pfx     = argv[++i];
-        else if (!a.empty() && a[0] != '-')     files.push_back(a);
-        // unknown --flags skipped (may be Grid flags already consumed)
-    }
-
-    if (files.empty()) usage(argv[0]);
-
-    GridCartesian* grid = SpaceTimeGrid::makeFourDimGrid(
-        GridDefaultLatt(),
-        GridDefaultSimd(Nd, vComplexD::Nsimd()),
-        GridDefaultMpi());
-
-    Coordinate latt = GridDefaultLatt();
-    std::cout << GridLogMessage << "Lattice : "
-              << latt[0]<<"x"<<latt[1]<<"x"<<latt[2]<<"x"<<latt[3] << "\n"
-              << GridLogMessage << "Ntraj   : " << files.size() << "\n"
-              << GridLogMessage << "FFT     : " << fftMode << "\n"
-              << GridLogMessage << "Format  : " << fmt << "\n"
-              << GridLogMessage << "Output  : " << pfx << "\n";
-
-    auto traj = loadFiles(files, grid, fmt);
-
-    if (fftMode == "spatial" || fftMode == "all") analyzeSpatial(traj, grid, pfx);
-    if (fftMode == "traj"    || fftMode == "all") analyzeTraj   (traj, grid, pfx);
-    if (fftMode == "all")                         analyzeAll5D  (traj, grid, pfx);
-
-    Grid_finalize();
-    return 0;
-}
-
-/*
-  Add to examples/Make.inc to build:
-
-    bin_PROGRAMS += ... fft5d
-    fft5d_SOURCES = fft5d.cc
-    fft5d_LDADD   = $(top_builddir)/Grid/libGrid.a
-*/

scripts/filelist

@@ -11,7 +11,7 @@ CCFILES=`find . -name '*.cc' -not -path '*/instantiation/*/*' -not -path '*/gamm
 
 ZWILS_FERMION_FILES=` find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/ZWilsonImpl*' `
 WILS_FERMION_FILES=`  find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/WilsonImpl*' `
-STAG_FERMION_FILES=`  find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/Staggered*' `
+STAG_FERMION_FILES=`  find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/StaggeredImpl*' `
 GP_FERMION_FILES=`    find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/Gparity*' `
 ADJ_FERMION_FILES=`   find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/WilsonAdj*' `
 TWOIND_FERMION_FILES=`find . -name '*.cc' -path '*/instantiation/*' -path '*/instantiation/WilsonTwoIndex*'`

skills/communication-overlap.md

@@ -0,0 +1,196 @@
+---
+name: communication-overlap
+description: Design and implement communication/computation overlap pipelines for GPU+MPI codes — per-packet event tracking, host-staging through pinned memory, internode/intranode bandwidth separation, and the 7-phase pipeline pattern that replaces broken accelerator-aware MPI paths.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+---
+
+# Communication/Computation Overlap Pipeline Design
+
+## Why GPU-Direct MPI Is Often Not the Right Default
+
+GPU-direct RDMA (passing GPU buffer pointers directly to MPI) is appealing because it eliminates explicit D2H/H2D copies. In practice on several leadership systems:
+
+- **Bandwidth**: RDMA at 30% of wirespeed has been observed on Pontevecchio/Aurora. The overhead of staging through pinned host memory can be *lower* total latency than slow RDMA.
+- **Correctness**: Device buffer aliasing in `MPI_Sendrecv` (see `mpi-heterogeneous.md`) makes direct GPU-to-GPU transfer unreliable.
+- **Overlap**: Host-staging enables fine-grained overlap — each packet's D2H can be issued as a separate asynchronous event, and the corresponding MPI send can fire as soon as *that packet* arrives in host memory, not after all packets are ready.
+
+The pipeline pattern below was developed to replace broken MPICH accelerator-aware paths. It achieves genuine computation/communication overlap by tracking per-packet GPU events.
+
+## The 7-Phase Pipeline
+
+Given a set of halo exchange operations (each identified by a `packet_index`):
+
+### Phase 0: Prepare data on device
+Pack halo data into contiguous GPU buffers. One buffer per direction/neighbour.
+
+### Phase 1: Post receives + start D2H
+Post all `MPI_Irecv` calls immediately (into pinned host buffers). Simultaneously, start asynchronous D2H copies for all send buffers:
+
+```cpp
+for (auto &pkt : send_packets) {
+    MPI_Irecv(pkt.host_recv_buf, pkt.bytes, MPI_BYTE,
+               pkt.src_rank, pkt.tag, comm, &pkt.recv_req);
+    
+    acceleratorCopyFromDeviceAsync(pkt.device_send_buf,
+                                   pkt.host_send_buf,
+                                   pkt.bytes, &pkt.d2h_event);
+}
+```
+
+The key: `pkt.d2h_event` is a per-packet GPU event (e.g. `cudaEvent_t`, `hipEvent_t`, or SYCL event). We can poll individual packet completion rather than waiting for all.
+
+### Phase 2: Fire sends as D2H completes (packet by packet)
+Poll packet D2H events. As each packet becomes ready in host memory, immediately fire the corresponding `MPI_Isend`. Also start intranode D2D copies at this point — these are deferred until now to avoid competing with the internode D2H on PCIe bandwidth:
+
+```cpp
+bool all_sent = false;
+while (!all_sent) {
+    all_sent = true;
+    for (auto &pkt : send_packets) {
+        if (!pkt.sent && acceleratorEventIsComplete(pkt.d2h_event)) {
+            MPI_Isend(pkt.host_send_buf, pkt.bytes, MPI_BYTE,
+                       pkt.dst_rank, pkt.tag, comm, &pkt.send_req);
+            pkt.sent = true;
+            start_intranode_copy(pkt);  // now safe, D2H is done
+        }
+        if (!pkt.sent) all_sent = false;
+    }
+}
+```
+
+### Phase 3: Poll receives + start H2D as each arrives
+`MPI_Test` individual receive requests. As each completes, immediately start the H2D copy into device-resident halo buffer:
+
+```cpp
+bool all_recvd = false;
+while (!all_recvd) {
+    all_recvd = true;
+    for (auto &pkt : recv_packets) {
+        if (!pkt.h2d_started) {
+            int flag = 0;
+            MPI_Test(&pkt.recv_req, &flag, MPI_STATUS_IGNORE);
+            if (flag) {
+                acceleratorCopyToDeviceAsync(pkt.host_recv_buf,
+                                              pkt.device_recv_buf,
+                                              pkt.bytes, &pkt.h2d_event);
+                pkt.h2d_started = true;
+            }
+        }
+        if (!pkt.h2d_started) all_recvd = false;
+    }
+}
+```
+
+### Phase 4: Wait for all sends
+```cpp
+std::vector<MPI_Request> send_reqs;
+for (auto &pkt : send_packets) send_reqs.push_back(pkt.send_req);
+MPI_Waitall(send_reqs.size(), send_reqs.data(), MPI_STATUSES_IGNORE);
+```
+
+### Phase 5: Wait for all H2D copies
+```cpp
+for (auto &pkt : recv_packets) acceleratorEventWait(pkt.h2d_event);
+```
+
+### Phase 6: Run interior computation
+The interior (non-halo) computation can run from Phase 1 onwards, overlapped with all of the above:
+
+```cpp
+// Launched in Phase 1, runs in parallel with the pipeline
+accelerator_for(ss, interior_sites, ...) { compute_interior(ss); }
+```
+
+Synchronise with interior before using the full field:
+```cpp
+accelerator_barrier();  // interior kernel done
+// Halo H2D is also complete (Phase 5 above)
+// Now safe to use full field
+```
+
+## Per-Packet Event Tracking Data Structure
+
+```cpp
+struct Packet {
+    // Buffers
+    void *device_send_buf;
+    void *host_send_buf;    // pinned
+    void *device_recv_buf;
+    void *host_recv_buf;    // pinned
+    size_t bytes;
+    
+    // MPI
+    int src_rank, dst_rank, tag;
+    MPI_Request send_req, recv_req;
+    
+    // GPU events (one per packet, not one global barrier)
+    AcceleratorEvent d2h_event;
+    AcceleratorEvent h2d_event;
+    
+    // State flags
+    bool sent = false;
+    bool h2d_started = false;
+};
+```
+
+The critical design point: `d2h_event` and `h2d_event` are **per-packet**, not global. This allows the MPI send for packet 0 to fire while packet 1's D2H is still in progress.
+
+## Internode vs Intranode Separation
+
+PCIe (GPU-to-CPU) and NVLink/xGMI (GPU-to-GPU within a node) are separate bandwidth resources. They do not compete with each other, but they *do* compete with each other for transactions if both are active simultaneously.
+
+Strategy: complete all internode D2H copies first (to maximise NIC injection bandwidth), then start intranode D2D copies (which use NVLink/xGMI and do not contend with PCIe for internode traffic):
+
+```cpp
+// In Phase 2: start intranode D2D only after D2H is confirmed complete
+if (pkt.is_intranode && pkt.d2h_done) {
+    // Use peer access (cudaMemcpyPeerAsync / hipMemcpyPeerAsync)
+    // rather than staging through host for intranode
+    cudaMemcpyPeerAsync(pkt.peer_recv_buf, pkt.dst_device,
+                         pkt.device_send_buf, pkt.src_device,
+                         pkt.bytes, computeStream);
+}
+```
+
+Grid reference: `Grid/communicator/Communicator_mpi3.cc` — search for `NVLINK_GET` and `ACCELERATOR_AWARE_MPI` conditional blocks.
+
+## Pinned Memory Allocation
+
+All host staging buffers must be pinned (page-locked) for async D2H/H2D:
+
+```cpp
+// CUDA
+cudaMallocHost(&host_buf, bytes);
+cudaFreeHost(host_buf);
+
+// HIP  
+hipHostMalloc(&host_buf, bytes, hipHostMallocDefault);
+hipHostFree(host_buf);
+
+// SYCL
+host_buf = sycl::malloc_host(bytes, *queue);
+sycl::free(host_buf, *queue);
+```
+
+Pre-allocate at startup. Repeated `cudaMallocHost` in the hot path adds latency from the OS memory manager.
+
+## Checksumming in the Pipeline
+
+Insert checksum computation before D2H (on the GPU-resident data) and verification after H2D (on the received GPU-resident data). See `correctness-verification.md` for the checksum pattern. The salting (`packet_index + 1000 * tag`) detects packet transposition — critical for diagnosing MPI buffer aliasing bugs where two packets' contents are swapped.
+
+## Smoke Test for a New System
+
+Before running physics, validate the pipeline on a synthetic benchmark:
+
+```cpp
+// Send a buffer of known values, receive and check
+// Run at multiple message sizes: 4KB, 64KB, 1MB, 16MB
+// Run at multiple process counts: 2, 8, 64, 512
+// Verify checksums on every packet
+// Measure bandwidth: should be ≥ 80% of FDR/HDR/NDR peak for host-staged
+```
+
+Any bandwidth below 50% of theoretical, or any checksum failure, indicates a problem in the communication stack that must be resolved before production runs.

skills/compiler-validation.md

@@ -0,0 +1,154 @@
+---
+name: compiler-validation
+description: Identify GPU compiler code generation bugs, distinguish them from hardware and runtime bugs, construct minimal reproducers, and validate correctness of generated assembly for performance-critical HPC kernels.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+  - Bash(objdump)
+---
+
+# Compiler Validation for GPU HPC Codes
+
+## Why Compiler Bugs Are Distinct
+
+Compiler bugs have a unique diagnostic signature: they produce *deterministically wrong* results. The same input always produces the same wrong output. This distinguishes them from:
+
+- Hardware bugs: usually stochastic (wrong answer sometimes, correct answer other times)
+- Runtime bugs (premature barrier, buffer aliasing): often stochastic or history-dependent
+- Race conditions: non-deterministic
+
+**The determinism test**: run the same kernel 100 times with the same input. If the wrong answer is always the same wrong answer, suspect the compiler.
+
+## The Minimal Reproducer Protocol
+
+When a kernel produces wrong results, isolate the compiler as quickly as possible:
+
+**Step 1: Eliminate the physics**. Reduce the failing kernel to the smallest possible computation that still exhibits the bug. Replace QCD fields with `double` arrays. Replace lattice operations with scalar arithmetic. The goal is a 20-line CUDA/HIP/SYCL file that any compiler engineer can compile and run.
+
+**Step 2: Binary search over optimisation levels**. Compile at `-O0` (or equivalent). If the answer becomes correct, the bug is in an optimisation pass. Then test `-O1`, `-O2`, `-O3` individually to find which optimisation level introduces the bug.
+
+```bash
+# HIP example
+hipcc -O0 minimal_repro.cc -o test_O0 && ./test_O0   # should be correct
+hipcc -O1 minimal_repro.cc -o test_O1 && ./test_O1   # compare
+hipcc -O2 minimal_repro.cc -o test_O2 && ./test_O2   # compare
+```
+
+**Step 3: Identify the optimisation pass**. For LLVM-based compilers (clang, hipcc, dpcpp, nvcc via ptxas):
+```bash
+# Disable individual optimisation passes:
+hipcc -O2 -mllvm -disable-loop-unrolling minimal_repro.cc -o test
+hipcc -O2 -fno-vectorize minimal_repro.cc -o test
+hipcc -O2 -fno-slp-vectorize minimal_repro.cc -o test
+```
+
+**Step 4: Inspect the generated code**. For CUDA/HIP, use `--generate-line-info` and `cuobjdump` or `roc-obj-extract` to get annotated assembly:
+```bash
+# CUDA
+nvcc -O2 --generate-line-info --keep minimal_repro.cu
+cuobjdump --dump-ptx minimal_repro.o
+# HIP/ROCm
+hipcc -O2 --save-temps minimal_repro.cc
+llvm-objdump -d minimal_repro.o
+# SYCL/DPC++
+icpx -O2 -fsycl -Xclang -ast-dump minimal_repro.cc 2>&1 | grep -A5 "suspicious_expr"
+```
+
+Look for: incorrect register spill/fill sequences, loop trip count miscalculation, vectorisation across iteration boundaries, incorrect address arithmetic.
+
+## Known Compiler Bug Patterns in GPU Code
+
+### Register Pressure / Spill Bugs
+High register usage forces spills to local memory. Some compiler versions generate incorrect spill/fill code — the value is written to local memory but a stale register value is read back instead of the spilled value.
+
+**Signature**: Wrong answer with high-register-count kernels; becomes correct when `--maxrregcount=N` forces lower register count (more spilling) or higher (`--maxrregcount=256`, fewer spills).
+
+**Diagnostic**: Check register usage:
+```bash
+nvcc -O2 --ptxas-options=-v minimal_repro.cu 2>&1 | grep "registers"
+hipcc -O2 --offload-arch=gfx90a --save-temps minimal_repro.cc
+llvm-mc --arch=amdgcn minimal_repro.s 2>&1 | grep "VGPRs"
+```
+
+### Vectorisation Across Loop Boundaries
+The compiler vectorises two successive loop iterations as a SIMD unit when they have a data dependency that the compiler has incorrectly determined does not exist.
+
+**Signature**: Wrong answer that becomes correct when the loop body is extracted to a non-inlined function (disabling auto-vectorisation across iterations).
+
+### Incorrect Constant Propagation
+The compiler evaluates a compile-time expression incorrectly, substituting a wrong constant. Common in template-heavy code where `sizeof(T)` or `alignof(T)` is used in arithmetic that the compiler folds at compile time.
+
+**Signature**: Wrong array index or wrong stride. Inspecting the generated assembly shows a literal constant where you expect a computed value.
+
+## Stress Patterns for Compiler Validation
+
+These patterns exercise the compiler in ways that commonly expose bugs:
+
+```cpp
+// 1. Aliased pointer write followed by immediate read
+// (tests correct handling of write-after-write in register allocation)
+__global__ void alias_stress(double *a, double *b, int n) {
+    int i = blockIdx.x * blockDim.x + threadIdx.x;
+    if (i < n) {
+        a[i] = a[i] * 2.0;
+        b[i] = a[i] + 1.0;  // must read the updated value, not the original
+    }
+}
+
+// 2. Mixed-precision accumulation
+// (tests correct type promotion in FMA sequences)
+__global__ void precision_stress(float *in, double *out, int n) {
+    double acc = 0.0;
+    for (int i = 0; i < n; i++) acc += (double)in[i];
+    *out = acc;
+}
+
+// 3. Large struct in shared memory
+// (tests alignment and offset calculation for non-power-of-2-sized objects)
+struct S { double x[3]; };  // sizeof = 24 bytes, not a power of 2
+__global__ void struct_stress(S *in, S *out, int n) {
+    extern __shared__ S smem[];
+    int tid = threadIdx.x;
+    smem[tid] = in[tid];
+    __syncthreads();
+    out[tid] = smem[(tid + 1) % blockDim.x];
+}
+```
+
+## Separating Compiler from Runtime/Hardware
+
+When results are deterministically wrong:
+
+| Test | Compiler bug | Runtime/hardware bug |
+|---|---|---|
+| Recompile at -O0 | Fixes it | No effect |
+| Run on CPU (host code equivalent) | Fixes it | No effect |
+| Reorder loop iterations | Changes wrong answer | No effect or different pattern |
+| Different compiler version | Fixes or changes wrong answer | No effect |
+| Different GPU of same model | Same wrong answer | Different or no error |
+| Different GPU model | Fixes it (ISA-specific codegen bug) | May or may not fix |
+
+## Reporting to Compiler Teams
+
+A compiler bug report needs:
+1. Minimal reproducer (< 50 lines)
+2. Compiler version (`hipcc --version`, `nvcc --version`, `icpx --version`)
+3. GPU model and driver version
+4. Exact wrong and correct answers (hexfloat for reproducibility)
+5. Which compile flags change the behaviour
+6. Generated assembly for the correct and incorrect variants
+
+File with: LLVM Bugzilla (for hipcc/clang/dpcpp backends), NVIDIA bug portal (nvcc/ptxas), or vendor-specific developer forum. The minimal reproducer is the single most important element — without it, compiler teams cannot prioritise.
+
+## Pragmatic In-Production Workaround
+
+When a compiler bug is confirmed but the fix is not yet available, the lowest-risk workaround is to mark the affected function with reduced optimisation:
+
+```cpp
+#pragma clang optimize off  // clang/hipcc/dpcpp
+void __attribute__((optimize("O0"))) affected_kernel_host_wrapper() { ... }
+// For device code, use per-file compilation flags via CMake/Makefile
+```
+
+Document the workaround with a comment referencing the compiler bug report number so it can be removed when the compiler is updated.

skills/correctness-verification.md

@@ -0,0 +1,169 @@
+---
+name: correctness-verification
+description: Implement application-level correctness verification for HPC codes on unreliable hardware — double-run pattern, deterministic reductions, per-packet checksums, and flight recorder step logging.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+---
+
+# Correctness Verification Infrastructure for HPC Codes
+
+## The Problem
+
+Leadership computing facilities sometimes have hardware or firmware bugs below the level visible to application code. The accelerator runtime can return from `q.wait()` or `cudaDeviceSynchronize()` before work is actually complete, or silently produce wrong answers in DMA transfers. Standard testing does not catch these because they are non-deterministic and often topology-dependent (fail only at specific process counts or on specific node configurations).
+
+The symptoms look like numerical instabilities, random MPI hangs, or wrong physics results — not like crashes. Without deliberate infrastructure, diagnosing root cause takes months.
+
+## The Double-Run Pattern
+
+The most reliable correctness check for non-deterministic hardware bugs is to run every computation twice and compare bit-identical fingerprints.
+
+**Key constraint**: the second run must use a *deterministic* code path. Non-deterministic floating-point ordering (e.g. from MPI_Allreduce with different reduction trees on retry) produces false mismatches. See `mpi-heterogeneous.md` for how to make reductions deterministic.
+
+```cpp
+// Pseudocode: double-run a step and compare CRC fingerprints
+void run_step_verified(State &state) {
+    state.save_checkpoint();
+    
+    uint64_t crc_a = run_step_and_fingerprint(state);
+    state.restore_checkpoint();
+    uint64_t crc_b = run_step_and_fingerprint(state);
+    
+    if (crc_a != crc_b) {
+        report_mismatch("step", crc_a, crc_b);
+        // Policy: abort, retry from checkpoint, or continue with alarm
+    }
+}
+```
+
+**Fingerprinting**: XOR-fold a CRC32 over all floating-point data after each step. XOR is order-independent, so it works across distributed nodes without communication. For field data:
+
+```cpp
+uint64_t fingerprint(const double *data, size_t n) {
+    uint64_t acc = 0;
+    for (size_t i = 0; i < n; i++) {
+        uint64_t bits;
+        memcpy(&bits, &data[i], sizeof(bits));
+        acc ^= crc32(bits);
+    }
+    return acc;
+}
+```
+
+On GPU, compute the XOR reduction on-device (avoids D2H transfer of the full field):
+
+```cpp
+// SYCL
+uint64_t svm_xor(uint64_t *vec, uint64_t L) {
+    uint64_t ret = 0;
+    { sycl::buffer<uint64_t,1> abuff(&ret, {1});
+      theGridAccelerator->submit([&](sycl::handler &cgh) {
+        auto R = sycl::reduction(abuff, cgh, uint64_t(0), std::bit_xor<>());
+        cgh.parallel_for(sycl::range<1>{L}, R,
+          [=](sycl::id<1> i, auto &sum) { sum ^= vec[i]; });
+      }); }
+    theGridAccelerator->wait();
+    return ret;
+}
+```
+
+## Per-Packet Communication Checksums
+
+Silent data corruption in MPI buffers (documented in MPICH with device-resident buffers; see `mpi-heterogeneous.md`) requires per-packet verification, not just end-to-end. The pattern:
+
+1. Before packing a send buffer, compute a GPU-side checksum of the payload.
+2. Append the checksum to the host staging buffer alongside the data.
+3. After receiving and copying to device, recompute the checksum on-device and compare.
+
+Salt each checksum with `packet_index + 1000 * mpi_tag` to detect transposition (packet A landing in packet B's slot):
+
+```cpp
+uint64_t salt = (uint64_t)packet_index + 1000ULL * mpi_tag;
+checksum_send = checksum_gpu(payload_gpu, payload_words) ^ salt;
+// ... transmit payload + checksum_send ...
+checksum_recv = checksum_gpu(payload_gpu_recv, payload_words) ^ salt;
+assert(checksum_recv == checksum_send);
+```
+
+Grid reference: `Grid/communicator/Communicator_mpi3.cc`, `#ifdef GRID_CHECKSUM_COMMS`.
+
+## Flight Recorder: Step-Level Logging
+
+Maintain a monotonic counter that names the current operation. On a hang, this is the only way to know *which* operation the process is stuck in without a debugger.
+
+```cpp
+struct FlightRecorder {
+    std::atomic<uint64_t> step_counter{0};
+    const char *step_name = "init";
+    
+    void step_log(const char *name) {
+        step_name = name;
+        step_counter.fetch_add(1, std::memory_order_relaxed);
+    }
+};
+extern FlightRecorder gRecorder;
+```
+
+In Record mode, also store floating-point norms and communication checksums to vectors. In Verify mode, compare against stored values:
+
+```cpp
+void norm_log(double val) {
+    if (mode == Record) norm_log_vec.push_back(val);
+    if (mode == Verify) {
+        double expected = norm_log_vec[norm_counter];
+        if (val != expected) {  // bit-exact for deterministic paths
+            std::cerr << "MISMATCH at step " << step_counter
+                      << " (" << step_name << "): "
+                      << std::hexfloat << val << " vs " << expected << "\n";
+            print_backtrace();
+        }
+        norm_counter++;
+    }
+}
+```
+
+Grid reference: `Grid/util/FlightRecorder.h`, `Grid/util/FlightRecorder.cc`.
+
+## Signal Handler for Hang Detection
+
+Install a SIGHUP handler that dumps the current flight recorder state. This is async-safe only if the handler writes to a pre-allocated buffer using `write()` (not `printf`):
+
+```cpp
+static char hang_buf[4096];
+
+static void sighup_handler(int) {
+    int n = snprintf(hang_buf, sizeof(hang_buf),
+        "rank=%d step=%llu name=%s\n",
+        mpi_rank,
+        (unsigned long long)gRecorder.step_counter.load(),
+        gRecorder.step_name);
+    write(STDERR_FILENO, hang_buf, n);
+    // Optional: call backtrace_symbols_fd (async-safe on Linux)
+    void *frames[64];
+    int depth = backtrace(frames, 64);
+    backtrace_symbols_fd(frames, depth, STDERR_FILENO);
+}
+
+// In main():
+signal(SIGHUP, sighup_handler);
+```
+
+To diagnose a hang across all ranks: `kill -HUP $(pgrep my_app)` or via job scheduler.
+
+## What to Verify at Each Step
+
+| Data type | Fingerprint method | Frequency |
+|---|---|---|
+| Lattice fields | XOR of CRC32 over float64 words | Every algorithmic step |
+| Communication buffers | GPU XOR reduction, salted | Every MPI operation |
+| Scalar reductions | Bit-exact match of double | Every GlobalSum |
+| Iteration counters | Exact integer match | Every solver iteration |
+
+## When to Abort vs Continue
+
+- **Abort immediately**: communication checksum mismatch (data is corrupt, continuing will silently propagate errors).
+- **Log and continue**: norm mismatch in Verify mode if you need to map out which operations are unreliable.
+- **Retry from checkpoint**: double-run mismatch when the underlying bug is non-deterministic (second retry will usually pass).
+
+Track the mismatch rate over a production run. A rate above ~1/1000 steps indicates a systemic hardware issue that should be escalated to the facility.

skills/gpu-memory-performance.md

@@ -0,0 +1,205 @@
+---
+name: gpu-memory-performance
+description: Diagnose and fix GPU memory bandwidth and occupancy problems in Grid HPC kernels — acceleratorThreads() pitfalls, LambdaApply thread mapping, coalescedRead/Write idiom, when to use accelerator_for vs a hand-rolled __global__ kernel, and fused vs staged HBM access patterns.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+---
+
+# GPU Memory Performance in Grid
+
+## Nsimd on GPU builds
+
+With `GEN_SIMD_WIDTH=64B` (the typical production setting), `Nsimd` is **not 1**:
+
+| Scalar type | `sizeof` | `Nsimd = 64 / sizeof` |
+|---|---|---|
+| `ComplexD` | 16 B | **4** |
+| `ComplexF` | 8 B  | **8** |
+| `RealD`    | 8 B  | **8** |
+| `RealF`    | 4 B  | **16** |
+
+So for `LatticePropagatorD` (scalar type `ComplexD`), `Nsimd=4` and the SIMD lane runs `threadIdx.x ∈ {0,1,2,3}`.  True `Nsimd=1` scalar-GPU builds are the exception, not the rule.
+
+## The acceleratorThreads() Trap
+
+`acceleratorThreads()` is a runtime-settable global (default **8**) that controls the `blockDim.y` of every `accelerator_for` launch.  It is NOT the SIMD width — it is the number of sites processed per block in the y-dimension.
+
+```cpp
+// Grid/threads/Accelerator.cc
+uint32_t accelerator_threads = 8;
+```
+
+With `accelerator_for(ss, osites, nsimd, ...)`, the launch is:
+
+```
+dim3 threads(nsimd, acceleratorThreads(), 1)
+dim3 blocks ((osites + acceleratorThreads() - 1) / acceleratorThreads(), 1, 1)
+```
+
+Total threads per block = `Nsimd × acceleratorThreads()`.  With `GEN_SIMD_WIDTH=64B` and `Nsimd=8` (fp32 / ComplexF):
+
+| `acceleratorThreads()` | threads/block | AMD wavefront | note |
+|---|---|---|---|
+| 2 (original default) | 16 | 25% | sub-wavefront, poor |
+| 4 | 32 | 50% | half wavefront |
+| **8 (current default)** | **64** | **100%** | **one full wavefront — Dslash sweet spot** |
+| 16 | 128 | 200% | two wavefronts/block; register-pressure cliff for heavy kernels |
+| 32 | 256 | 400% | severe register spill for stencil kernels |
+
+**Why 8 and not higher?**  Compute-heavy kernels like the Domain Wall Dslash carry many live registers per thread (spinors + gauge links + projections).  Doubling `acceleratorThreads` from 8→16 doubles the register demand per block, which on AMD GFX90A triggers a hard occupancy cliff: `Benchmark_dwf_fp32` drops from 1.7 TF/s (nt=8) to ~300 GF/s (nt=16).  The sweet spot is one full wavefront per block, which with `Nsimd=8` (fp32) means `nt=8`.
+
+For fp64 work (`Nsimd=4`), `nt=8` gives 32 threads = half a wavefront (AMD pads to 64 with idle lanes).  Kernels that are not register-limited (e.g. simple lattice arithmetic) can benefit from `--accelerator-threads 16` at runtime.  Reduction kernels bypass `acceleratorThreads()` entirely via `getNumBlocksAndThreads`.
+
+**Why was the default ever 2?**  Before the `threadIdx.x`/`threadIdx.y` remap (see LambdaApply section below), the site index lived in `threadIdx.x` — the fast, coalescing dimension.  Increasing `acceleratorThreads` widened the block in the *site* direction, so adjacent threads in a warp hit adjacent sites, each stride `sizeof(vobj)` apart in AoS memory — breaking coalescing.  On early NVIDIA ports with `Nsimd≈8`, `nt=2` gave 16 threads = 50% of a 32-thread warp; NVIDIA recovers this via multiple concurrent blocks per SM, so occupancy was barely tolerable.  AMD has no such multiplier when blocks are already sub-wavefront.  After the remap put the site index in `threadIdx.y` and the SIMD lane in `threadIdx.x`, coalescing became independent of `acceleratorThreads`, removing the constraint.
+
+**Diagnostic**: observed bandwidth << peak, kernel time >> expected from data volume.  Check with `--accelerator-threads 32` at runtime.  A large speedup confirms occupancy starvation.
+
+**Fix options** (in order of preference):
+1. Kernel needs its own thread count — use `getNumBlocksAndThreads` and launch a `__global__` kernel directly (see below).
+2. Temporarily acceptable: set `--accelerator-threads 32` at the application level.  Note this affects every `accelerator_for` site in the binary.
+
+## LambdaApply Thread Mapping
+
+`accelerator_for` and `accelerator_for2d` go through `LambdaApply`:
+
+```cpp
+// HIP/CUDA LambdaApply kernel:
+uint64_t x = threadIdx.y + blockDim.y * blockIdx.x;  // iter1 (site index)
+uint64_t y = threadIdx.z + blockDim.z * blockIdx.y;  // iter2
+uint64_t z = threadIdx.x;                             // lane (SIMD lane)
+Lambda(x, y, z);
+```
+
+`threadIdx.x` is the **fast** (lane) dimension — consecutive thread IDs within a warp/wavefront correspond to consecutive lane values on the **same** site, not consecutive sites.
+
+With `GEN_SIMD_WIDTH=64B` and `Nsimd=4` (PropagatorD), a 64-thread AMD wavefront contains 64/4 = 16 sites, each processed by 4 lanes.  Adjacent threads within the wavefront read different lanes of the same site — this is a broadcast pattern (hardware handles this efficiently), not a stride.  The stride between consecutive *sites* (`sizeof(vobj)`) only appears between groups of `Nsimd` threads, spaced `Nsimd` apart in threadIdx.x — not between adjacent threads within a warp.
+
+## coalescedRead / coalescedWrite
+
+These are Grid's canonical way to read/write one SIMD lane from a vector type inside a `GRID_SIMT` kernel:
+
+```cpp
+// accelerator_for(ss, osites, Nsimd, {
+//   lane = acceleratorSIMTlane(Nsimd) = threadIdx.x  ∈ {0..Nsimd-1}
+auto scalar_val = coalescedRead(field[ss]);       // extractLane(lane, field[ss])
+coalescedWrite(field[ss], scalar_val);             // insertLane(lane, field[ss], scalar_val)
+```
+
+For `vobj` aggregate types, `coalescedRead` calls `extractLane(lane, vobj)` which recurses through the tensor hierarchy and returns `vobj::scalar_object`.
+
+For `vsimd` (raw SIMD vector) types, it casts to `scalar_type*` and indexes with `lane`.
+
+## Coalescing the Iteration Structure
+
+For an AoS input array where each site is `words` 16-byte elements, adjacent threads reading the same site's consecutive words achieve coalesced access:
+
+```cpp
+// Good: k varies across threads in a block → consecutive 16-byte reads
+accelerator_for2d(k, R, ss, osites, Nsimd, {
+    coalescedWrite(out[ss]._internal[k],
+                   coalescedRead(idat[ss * words + base + k]));
+});
+// dim3(Nsimd, nt, 1): threadIdx.y = k (consecutive words, coalesced)
+//                      threadIdx.x = lane (SIMD sub-lane, coalesced for Nsimd>1)
+```
+
+```cpp
+// Bad: each thread reads all R words of its site serially
+accelerator_for(ss, osites, 1, {
+    Bundle b;
+    for (int k = 0; k < R; k++)
+        b._internal[k] = idat[ss * words + base + k];  // serial, not coalesced across threads
+    out[ss] = b;                                         // bulk struct write
+});
+```
+
+The bad pattern also accumulates a large struct in registers (192 bytes for R=12), increasing register pressure and reducing occupancy further.
+
+## When to Use a __global__ Kernel Instead of accelerator_for
+
+`accelerator_for` is correct for site-parallel work where `acceleratorThreads()` is tuned appropriately.  Use a direct `__global__` kernel when:
+
+- The kernel requires a **specific thread count** for correctness or performance (reductions, shared-memory algorithms).
+- The optimal thread count depends on `sizeof(sobj)` and `sharedMemPerBlock`, not on a runtime global.
+- You need the retirement-count pattern for cross-block final reduction.
+
+Pattern: use `getNumBlocksAndThreads` to pick `numThreads` and `numBlocks`:
+
+```cpp
+Integer numThreads, numBlocks;
+int ok = getNumBlocksAndThreads(n, sizeof(sobj), numThreads, numBlocks);
+// starts at warpSize (32/64), doubles while 2*threads*sizeof(sobj) < sharedMemPerBlock
+// gives 64–256 threads/block → correct occupancy independent of acceleratorThreads()
+Integer smemSize = numThreads * sizeof(sobj);
+myKernel<<<numBlocks, numThreads, smemSize, computeStream>>>(args...);
+```
+
+Grid's `reduceKernel` uses this pattern and achieves ~400 GB/s on MI250X.
+
+## Fused vs Staged HBM Access
+
+A staged pack+reduce reads the data **three times**:
+
+```
+pack kernel:   reads vobj array (N bytes), writes bundle buffer (N bytes)
+reduce kernel: reads bundle buffer (N bytes), writes tiny result buffer
+```
+
+Total HBM: 3N bytes for N bytes of useful input.
+
+A fused kernel reads the data **once**:
+
+```
+packReduceKernel: reads R words of vobj array (N bytes), reduces in-place
+```
+
+Total HBM: N bytes.  Register pressure increases (R words held per thread) but the 3× HBM saving dominates for large objects.
+
+The fused pattern in Grid's `sumD_gpu_reduce_words<R>`:
+
+```cpp
+template <int R, class vobj, class sobj, class Iterator>
+__device__ void packReduceBlocks(
+    const iScalar<typename vobj::vector_type> *idat,
+    sobj *g_odata, Iterator osites, int base, int words)
+{
+  // sobj = iVector<iScalar<scalarD>, R>  (R double-precision scalars per site)
+  constexpr Iterator nsimd = vobj::Nsimd();
+  ...
+  while (i < osites * nsimd) {
+    Iterator lane = i % nsimd;
+    Iterator ss   = i / nsimd;
+    sobj tmpD; zeroit(tmpD);
+    for (int k = 0; k < R; k++) {
+      auto w = extractLane(lane, idat[ss * words + base + k]);
+      iScalar<typename vobj::scalar_typeD> wd; wd = w;   // float→double promotion
+      tmpD._internal[k] = wd;
+    }
+    mySum += tmpD;
+    ...
+  }
+  reduceBlock(sdata, mySum, tid);
+}
+```
+
+Launched with `getNumBlocksAndThreads` → 128 threads/block for R=12 (`BundleScalarD`=192 B, sharedMem=64 KB) → correct occupancy without depending on `acceleratorThreads()`.
+
+## Observed Numbers on MI250X (32^4 LatticePropagatorD, Nsimd=4, GEN_SIMD_WIDTH=64B)
+
+| Configuration | pack µs/group | reduce µs/group | total µs | GB/s |
+|---|---|---|---|---|
+| acceleratorThreads=2 (8 threads/block), staged | 10,080 | 470 | 126,909 | 50 |
+| acceleratorThreads=16 (64 threads/block), staged | 342 | 310 | 8,251 | 297 |
+| acceleratorThreads=16, fused (128 threads/block via getNumBlocksAndThreads) | — | 349 | 4,584 | 546 |
+
+The fused kernel at 349 µs/group reads 201 MB at 576 GB/s — 36% of MI250X HBM peak.  The remaining gap from peak is the in-kernel serial loop over R=12 words and the 12 serial kernel launches.
+
+## Quick Checklist When a Kernel Is Slow
+
+1. **Check Nsimd**: `GEN_SIMD_WIDTH=64B` → Nsimd=4 (ComplexD), 8 (ComplexF).  Total threads/block = `Nsimd × acceleratorThreads()`.  With old default nt=2 and Nsimd=4: 8 threads = 12.5% of AMD wavefront.
+2. Check threads per block: for `accelerator_for` kernels use `--accelerator-threads 32` and measure; a large speedup confirms occupancy starvation.
+3. Check for bulk struct accumulation in registers (`Bundle b; for(...) b._internal[k] = ...;`).  Replace with per-element writes via `coalescedWrite`.
+4. Check for staged HBM access (pack → buffer → reduce).  Count the passes; fuse if ≥ 2 passes over the same data.
+5. For reduction kernels, always use `getNumBlocksAndThreads` rather than `accelerator_for` so thread count is independent of `acceleratorThreads()`.

skills/gpu-runtime-correctness.md

@@ -0,0 +1,101 @@
+---
+name: gpu-runtime-correctness
+description: Detect and work around GPU runtime correctness failures — premature completion signalling, infinite poll hangs, stale completion flags, and the double-wait diagnostic pattern. Covers CUDA, HIP/ROCm, and SYCL/Level Zero runtimes.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+---
+
+# GPU Runtime Correctness
+
+## The Completion Signalling Problem
+
+GPU runtimes expose a synchronisation primitive — `cudaDeviceSynchronize()`, `hipDeviceSynchronize()`, `q.wait()` — that is supposed to block until all previously submitted GPU work is complete. On several production systems, this guarantee has been violated in two distinct ways:
+
+### Failure Mode A: Premature Return
+The wait returns before the GPU work is done. The subsequent CPU code reads stale data from the output buffer. This is the most dangerous failure because it looks like a numerical instability, not a crash. Results are wrong but the program exits normally.
+
+**Identifying Premature Return**: Insert a second, independent wait immediately after the first. If a second `q.wait()` "fixes" incorrect results that appeared with a single `q.wait()`, the first wait was returning prematurely.
+
+```cpp
+// Diagnostic version — if this stabilises results, you have premature return
+accelerator_barrier();  // first wait
+accelerator_barrier();  // second wait (diagnostic)
+```
+
+Production fix: submit a trivially cheap no-op kernel after the real work and wait for it. The no-op kernel cannot complete until all previous commands in the queue are done (command queue ordering guarantee), so waiting for the no-op is a stronger barrier than waiting for the queue itself:
+
+```cpp
+// Lightweight fence kernel
+template<class T>
+__global__ void noop_kernel(T *p) { if (threadIdx.x == 0) (void)(*p); }
+
+void strong_barrier(T *device_ptr) {
+    noop_kernel<<<1, 1, 0, computeStream>>>(device_ptr);
+    cudaStreamSynchronize(computeStream);  // wait for the no-op
+}
+```
+
+### Failure Mode B: Infinite Poll
+The wait enters a polling loop that never terminates. The process consumes 100% CPU in a runtime library. The GPU has either stopped signalling progress entirely, or the completion flag is in a memory region that has become incoherent.
+
+This is distinct from Failure Mode A: with premature return the CPU proceeds; with infinite poll the CPU is stuck.
+
+**Identifying Infinite Poll**: `top` shows the MPI rank at 100% CPU. `perf top -p PID` or `strace -p PID` shows the process burning cycles inside the GPU runtime library (e.g. `libze_intel_gpu.so`, `libamdhip64.so`).
+
+**Documented instances**: 
+- Intel Level Zero on Pontevecchio (Aurora): both premature return *and* infinite poll have been observed as independent bugs on the same system.
+- The two failure modes can co-exist and have overlapping symptoms at the application level.
+
+## Completion Signalling Architecture
+
+Understanding why these bugs happen requires knowing how completion signalling works:
+
+```
+GPU command processor
+  → signals completion by writing to a host-visible memory address
+  → CPU runtime polls that address (or uses OS event notification via ioctl)
+```
+
+A premature return means the memory write happened before the actual work completed (e.g. the signal is on a different command stream that has not been serialised with the work stream). An infinite poll means the memory write never happens (hardware or driver bug preventing the signal from being written).
+
+**Implication**: `accelerator_barrier()` is not an unconditional correctness guarantee on all production systems. Application-level verification (double-run, checksums) is necessary as a second line of defence.
+
+## The Double-Wait Pattern in Practice
+
+The double-wait is a pragmatic workaround when premature return is suspected but not yet confirmed. It adds latency but does not change correctness if the barrier is working properly, so it is safe to enable in production:
+
+```cpp
+#ifdef WORKAROUND_PREMATURE_BARRIER
+#define accelerator_barrier() do { \
+    real_accelerator_barrier();    \
+    real_accelerator_barrier();    \
+} while(0)
+#endif
+```
+
+Monitor whether this changes observed behaviour. If double-wait eliminates wrong answers, you have confirmed premature return. If it does not help but inserting a no-op kernel does, the issue is with the wait primitive specifically, not with the underlying completion signal.
+
+## SYCL/Level Zero Specifics
+
+Level Zero (the backend for Intel GPU runtimes) separates command submission from synchronisation. A `q.wait()` should wait for all previously submitted command lists to retire. Documented bugs include:
+
+- `q.wait()` returning before the associated fence in Level Zero has been signalled.
+- `q.wait()` entering an `ioctl(i915, I915_GEM_WAIT)` call that never returns (kernel driver bug, not runtime bug).
+
+The latter requires a node reboot and cannot be worked around in application code. Detect it by checking process state (`D` in `ps aux`) and the kernel function via `/proc/PID/wchan`.
+
+## Stream Ordering and Compute Streams
+
+All GPU work must be submitted to the *same* stream/queue if you rely on in-order execution guarantees. Mixing default stream and non-default streams invalidates ordering assumptions on some backends.
+
+Grid uses `computeStream` (CUDA/HIP) or `theGridAccelerator` (SYCL) consistently throughout. If mixing Grid with third-party GPU code, ensure the third-party code is directed to the same stream, or insert explicit inter-stream barriers.
+
+## Checklist for New GPU Code
+
+1. Every kernel launch is followed by an `accelerator_barrier()` before reading device-side output on the host.
+2. All device-to-host copies use an explicit stream synchronisation after the copy, not before.
+3. If results are non-deterministic across runs, insert a second barrier and observe whether reproducibility improves.
+4. For correctness-critical operations (reductions that will be compared against reference values), add the double-run checksum test from `correctness-verification.md`.
+5. If the process hangs at 100% CPU in a runtime library function, this is a driver/runtime bug — there is no application-level fix beyond scheduling a node reboot.

skills/hang-diagnosis.md

@@ -0,0 +1,102 @@
+---
+name: hang-diagnosis
+description: Diagnose and isolate process hangs on HPC systems — distinguishing kernel-level ioctl hangs, infinite poll loops, collective deadlocks, and GPU completion signalling failures using async-safe signal handlers and flight recorder step counters.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+  - Bash(strace)
+  - Bash(gdb)
+---
+
+# Hang Diagnosis on HPC Systems
+
+## Taxonomy of Hangs
+
+Not all hangs are the same. Misidentifying the type leads to wrong mitigation. The four distinct classes encountered on production leadership systems:
+
+### 1. Kernel-level ioctl hang (never returns)
+The process is in `D` (uninterruptible sleep) state. `strace` shows it blocked in an `ioctl` syscall. The GPU device driver has entered an unrecoverable state.
+
+**Diagnosis**: `ps aux | grep D` — the process shows `D` state. `cat /proc/PID/wchan` shows `i915_gem_wait_for_error` or similar.
+
+**Resolution**: Only a driver reload or node reboot recovers it. Log the node identifier and request replacement from the facility scheduler.
+
+### 2. Infinite poll loop (`q.wait()` or `cudaDeviceSynchronize()` never returns)
+The process is in `R` (running) state, consuming 100% CPU. A polling loop inside the runtime is checking a completion flag that never becomes true, either because the hardware never sets it or because the flag is in a memory region not visible to the polling thread.
+
+**Diagnosis**: `top` shows the rank at 100% CPU. `strace -p PID` shows repeated `futex` or `read` syscalls with zero-length results, or no syscalls at all (pure spinloop). `perf top -p PID` shows the process burning cycles in a single tight loop in a runtime library (e.g., `ze_intel_gpu.so`).
+
+**Resolution**: The double-wait workaround — submit a trivially cheap kernel after the operation under test to act as a fence, then wait for the trivial kernel. See `gpu-runtime-correctness.md`.
+
+### 3. Collective deadlock
+One or more ranks are blocked in an MPI call, usually `MPI_Allreduce` or `MPI_Barrier`, while others are not. Root cause: a topology-dependent bug in the MPI library's collective algorithm where some ranks' contributions never arrive.
+
+**Diagnosis**: Flight recorder step logs show some ranks at step N (inside the collective) while others are at step N+1 or stuck at step N with different `step_name` strings. The hung ranks will show `D` or `S` state in `ps`.
+
+**Resolution**: Replace `MPI_Allreduce` with a deterministic point-to-point tree reduction. See `mpi-heterogeneous.md`.
+
+### 4. Premature return from wait (silent wrong answer, not a hang)
+The runtime returns from `q.wait()` before the GPU work is complete. The next operation reads stale data. This is not a hang — it manifests as a wrong answer or non-deterministic floating-point results. It is listed here because it is the most confusing failure mode: the code appears to run correctly and completes normally.
+
+**Diagnosis**: Double-run with checksum (see `correctness-verification.md`). Insert a second `q.wait()` after the first and observe if results become reproducible. If inserting the second wait "fixes" wrong answers, the first wait was returning prematurely.
+
+## Flight Recorder for Hang Localization
+
+The most important diagnostic tool is knowing *which operation* a process is in when it hangs. Maintain a named step counter:
+
+```cpp
+// Call at the start of every major operation
+FlightRecorder::StepLog("MPI_Allreduce::norm");
+// ... do the operation ...
+FlightRecorder::StepLog("MPI_Allreduce::done");
+```
+
+On SIGHUP, dump rank, step counter value, and step name to stderr in an async-safe manner:
+
+```cpp
+static void sighup_handler(int) {
+    char buf[256];
+    int n = snprintf(buf, sizeof(buf), "rank %d: step %llu '%s'\n",
+        comm_rank,
+        (unsigned long long)step_counter,
+        step_name);
+    write(2, buf, n);
+    // backtrace_symbols_fd is async-safe on Linux glibc
+    void *frames[32];
+    backtrace_symbols_fd(frames, backtrace(frames, 32), 2);
+}
+signal(SIGHUP, sighup_handler);
+```
+
+Broadcast SIGHUP to all ranks from outside the job:
+```bash
+# In a separate shell while the job is hung
+squeue --job $JOBID -o "%i %N" | awk '{print $2}' | \
+  xargs -I{} ssh {} "pkill -SIGHUP -f my_application"
+```
+
+The step names from all ranks will reveal which collective operation has diverged.
+
+## Distinguishing Driver Hang from MPI Hang
+
+| Symptom | Driver hang | MPI hang |
+|---|---|---|
+| Process state | `D` (ioctl) or `R` (spinloop) | `S` (blocked in syscall) |
+| `strace` | blocked `ioctl` or tight loop | blocked `recvmsg` / `read` |
+| Scope | single rank / single node | subset of ranks, pattern-dependent |
+| Recovery | reboot node | cancel job |
+| Flight recorder | step name is a GPU operation | step name is a collective |
+
+## Reducing Diagnostic Time
+
+1. **Name every collective operation** in the flight recorder before calling it.
+2. **Separate GPU work from MPI work** in the code so the step name unambiguously identifies which subsystem is hung.
+3. **Log node identifiers** alongside step names so flaky nodes can be identified and blacklisted.
+4. **Request flight recorder dumps from all ranks simultaneously** (SIGHUP broadcast) rather than attaching a debugger — attaching `gdb` to one rank of a hung MPI job usually deadlocks the debugger too.
+
+## What Not to Do
+
+- Do not `kill -9` a hung rank immediately — get the flight recorder dump first, otherwise diagnostic information is lost.
+- Do not assume the first rank that prints an error is the faulty one — collective hangs are frequently caused by the *last* rank to arrive at the barrier.
+- Do not use `MPI_Abort` in the hang handler — it may itself hang on some implementations. Use `_exit(1)` to force termination.

skills/mpi-heterogeneous.md

@@ -0,0 +1,182 @@
+---
+name: mpi-heterogeneous
+description: Diagnose and work around MPI correctness bugs on heterogeneous (CPU+GPU) systems — device buffer aliasing in MPI_Sendrecv, AARCH64 PLT corruption from libfabric, topology-dependent allreduce hangs, mixed-ABI HIP runtime from wrong GTL library (Frontier/ROCm), and deterministic point-to-point reduction trees as a replacement for MPI_Allreduce.
+user-invocable: true
+allowed-tools:
+  - Read
+  - Bash(grep -r)
+---
+
+# MPI Correctness on Heterogeneous HPC Systems
+
+## The Core Problem
+
+MPI libraries were designed for CPU-resident buffers. When GPU-resident buffers are passed directly (GPU-aware MPI / GPU direct RDMA), several correctness assumptions break:
+
+- **Buffer aliasing**: The MPI library may internally alias send/receive buffer addresses for `MPI_Sendrecv` in ways that are safe for CPU memory but wrong for GPU memory with different cache coherency rules.
+- **RDMA bandwidth**: GPU direct RDMA on some fabrics operates at a fraction of peak wirespeed (documented at ~30% on Pontevecchio/Aurora), making host-staging mandatory for performance even when correctness is not an issue.
+- **Collective tree topology**: `MPI_Allreduce` implementations may select reduction trees based on process count or communicator topology that expose rank-ordering bugs, causing hangs on some configurations but not others.
+
+## Bug Class 1: Device Buffer Aliasing in MPI_Sendrecv
+
+**Symptom**: `MPI_Sendrecv` with GPU-resident send and receive buffers produces wrong results. The received data matches neither the expected payload nor a host-staged copy. The failure is *deterministic* for a given problem size and process count, but *history-dependent* — earlier sends affect which alias is selected.
+
+**Root cause**: The MPI library internally reuses GPU buffer addresses for temporary staging without proper device memory ordering. When the same physical GPU memory pages appear in both the send and receive paths, writes from one path corrupt the other.
+
+**Diagnosis**: 
+1. Enable per-packet checksumming (see `correctness-verification.md`). If the checksum on the received packet does not match the sent checksum, the data was corrupted in transit.
+2. Replace `MPI_Sendrecv` with separate `MPI_Isend` + `MPI_Irecv` + `MPI_Waitall`. If this fixes the problem, the bug is in the `MPI_Sendrecv` implementation's internal buffer handling.
+3. Stage through host memory (`cudaMemcpy`/`hipMemcpy` to a host buffer, then `MPI_Sendrecv` on host buffers, then copy back). If this fixes the problem, confirms GPU-specific aliasing.
+
+**Reported as**: MPICH issue #7302. Affects MPICH on Intel Pontevecchio (Aurora) with device-resident buffers.
+
+**Workaround**: Do not use `MPI_Sendrecv` with GPU buffers. Use asynchronous send/receive pairs or host-staging. See `communication-overlap.md` for the full pipeline pattern.
+
+## Bug Class 2: PLT Corruption on AARCH64 (libfabric)
+
+**Symptom**: Application crashes or hangs on first `MPI_Comm_dup` call on AARCH64 systems (e.g. NVIDIA Grace/H200). Backtrace shows a bad instruction in the PLT (Procedure Linkage Table) for `MPI_Comm_dup` — specifically a `br x15` instruction that should instead be a proper trampoline.
+
+**Root cause**: `libfabric`'s memory registration cache monitor patches PLT entries at runtime to intercept memory allocation calls. Its AARCH64 trampoline generation writes an incorrect instruction sequence, leaving `br x15` (branch to whatever happens to be in x15) in the PLT entry. The next call through that PLT entry executes garbage.
+
+**Diagnosis**:
+```bash
+# Check if the PLT entry is corrupted
+objdump -d /proc/PID/exe | grep -A5 "MPI_Comm_dup@plt"
+# Look for "br x15" — this should be a proper stub, not a register branch
+```
+
+Or check the disassembly of the live process:
+```bash
+gdb -p PID -batch -ex "disassemble 'MPI_Comm_dup@plt'"
+```
+
+**Workaround**:
+```bash
+export FI_MR_CACHE_MONITOR=disabled
+```
+This prevents libfabric from patching PLT entries. It may reduce MR cache performance but restores correctness.
+
+**Reported as**: libfabric issue #11451. Affects systems using AARCH64 + libfabric OFI provider (Cray Slingshot, AWS EFA) with memory registration cache enabled.
+
+## Bug Class 3: Topology-Dependent Allreduce Hangs
+
+**Symptom**: `MPI_Allreduce` hangs indefinitely on some node configurations but completes correctly on others. The failure correlates with process count (e.g. fails at 512 ranks, works at 256) or network topology (fails when crossing specific router boundaries).
+
+**Root cause**: The MPI library's collective selection algorithm picks a reduction tree implementation that assumes symmetric participation from all ranks. A bug in one rank's contribution path (e.g. a GPU-side buffer not yet flushed when MPI reads it, due to premature barrier — see `gpu-runtime-correctness.md`) causes that rank to send wrong or incomplete data, and the tree-reduction protocol deadlocks waiting for data that never arrives correctly.
+
+**Diagnosis**: Flight recorder step logging (see `hang-diagnosis.md`). SIGHUP broadcast to all ranks. Ranks that are hung will show step name `MPI_Allreduce::...`; ranks that completed will show the next step. The hung ranks are the *recipients* of the stale data, not necessarily the *cause*.
+
+**Workaround — deterministic P2P reduction tree**:
+
+Replace `MPI_Allreduce` with an explicit point-to-point binary tree reduction. This is slower for large communicators but:
+1. Is immune to topology-dependent collective bugs.
+2. Is deterministic in floating-point ordering (the tree is fixed, not chosen at runtime).
+3. Makes the hang location explicit — each P2P operation is a named step in the flight recorder.
+
+```cpp
+// Binary tree reduction: rank 0 collects, then broadcasts
+void GlobalSumP2P(double *data, int count, MPI_Comm comm) {
+    int rank, size;
+    MPI_Comm_rank(comm, &rank); MPI_Comm_size(comm, &size);
+    
+    // Reduce phase: even ranks receive from odd neighbours
+    for (int stride = 1; stride < size; stride *= 2) {
+        if (rank % (2*stride) == 0) {
+            int partner = rank + stride;
+            if (partner < size) {
+                std::vector<double> tmp(count);
+                MPI_Recv(tmp.data(), count, MPI_DOUBLE, partner, 0, comm, MPI_STATUS_IGNORE);
+                for (int i = 0; i < count; i++) data[i] += tmp[i];
+            }
+        } else if (rank % stride == 0) {
+            int partner = rank - stride;
+            MPI_Send(data, count, MPI_DOUBLE, partner, 0, comm);
+            break;
+        }
+    }
+    // Broadcast phase
+    for (int stride = /* highest power of 2 <= size */; stride >= 1; stride /= 2) {
+        if (rank % (2*stride) == 0) {
+            int partner = rank + stride;
+            if (partner < size)
+                MPI_Send(data, count, MPI_DOUBLE, partner, 0, comm);
+        } else if (rank % stride == 0) {
+            int partner = rank - stride;
+            MPI_Recv(data, count, MPI_DOUBLE, partner, 0, comm, MPI_STATUS_IGNORE);
+        }
+    }
+}
+```
+
+Grid reference: `USE_GRID_REDUCTION` macro in `Grid/communicator/Communicator_mpi3.cc`.
+
+## Bug Class 4: Mixed HIP ABI from Wrong GTL Library (Frontier / ROCm)
+
+**Symptom**: `HIPFFT_PARSE_ERROR` (error code 12) returned by `hipfftPlanMany` / `hipfftMakePlanMany` / `hipfftPlan1d` for FFT sizes G < 32, but G ≥ 32 succeeds. The failure only occurs with an empty rocFFT kernel cache (`~/.cache/rocfft`); a warm cache may mask it. Host-side operations and GPU kernels that do not invoke rocFFT JIT work correctly.
+
+**Root cause — mixed HIP ABI**: rocFFT uses JIT compilation (via `libamd_comgr`) for small transforms (G < 32); for G ≥ 32 it uses pre-compiled device code bundled in the library, so the JIT path is never exercised. When two HIP runtime versions are loaded in the same process — e.g. `libamdhip64.so.7` (ROCm 7) and `libamdhip64.so.6` (ROCm 6) — the rocFFT JIT cannot complete successfully.
+
+The hidden source of the old library is the Cray MPI GPU Transport Layer. On Frontier, `cray-mpich`'s `libmpi_gtl_hsa.so` may be compiled against `libamdhip64.so.6` (ROCm 6 ABI) even when the loaded ROCm module is 7.0.2. Because `LD_LIBRARY_PATH` picks up the GTL directory before the ROCm 7 library directory, `libamdhip64.so.6` is pulled in first, and both ABI versions end up resident in the process.
+
+**Diagnosis**:
+```bash
+# Check which libamdhip64 versions are actually linked into your binary at runtime
+ldd --verbose ./your_binary 2>&1 | grep amdhip
+# Bad output — two different .so versions:
+#   libamdhip64.so.6 => /opt/rocm-6.4.2/lib/libamdhip64.so.6
+#   libamdhip64.so.7 => /opt/rocm-7.0.2/lib/libamdhip64.so.7
+# Good output — only one:
+#   libamdhip64.so.7 => /opt/rocm-7.0.2/lib/libamdhip64.so.7
+```
+
+If two versions appear, the problem is the GTL/LD_LIBRARY_PATH ordering.
+
+**Fix — correct module stack and LD_LIBRARY_PATH ordering (Frontier)**:
+```bash
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.0.2
+# Prepend CRAY_LD_LIBRARY_PATH so the ROCm-7-aware GTL is found first
+export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
+# Ensure ROCm 7 LLVM libs (needed by libamd_comgr JIT) are on the path
+export LD_LIBRARY_PATH=/opt/rocm-7.0.2/lib/llvm/lib/:$LD_LIBRARY_PATH
+```
+
+The critical step is prepending `CRAY_LD_LIBRARY_PATH`: this ensures the GTL library built against the ROCm 7 ABI is resolved before any older version that may appear further down `LD_LIBRARY_PATH`. Without this step, a stale symlink or directory ordering can silently load the wrong `libmpi_gtl_hsa.so`.
+
+**Reproducer**: `tests/debug/Test_hipfft_repro.cc` — standalone hipFFT test (no Grid headers) that sweeps G and howmany values matching realistic Grid lattice geometries. Compile with:
+```bash
+hipcc -o Test_hipfft_repro Test_hipfft_repro.cc -lhipfft
+rm -rf ~/.cache/rocfft   # empty cache required to trigger JIT path
+./Test_hipfft_repro
+```
+
+**Reference**: `systems/WorkArounds.txt`, Frontier section — GPU mapping, XPMEM, and `FI_MR_CACHE_MONITOR=disabled` settings for Frontier are documented there.
+
+**Systems affected**: Frontier (ORNL, MI250X). Likely applies to any Cray PE system where the loaded `cray-mpich` GTL was compiled against an older ROCm ABI than the runtime ROCm module. LumiG (CSC, MI250X) uses the same Cray PE and may exhibit the same issue.
+
+## Compile-Time Guard Structure
+
+Recommended macro structure to switch between the workaround paths:
+
+```cpp
+// In configure / CMake, expose as options:
+// ACCELERATOR_AWARE_MPI  — use GPU direct (fast, potentially broken)
+// GRID_CHECKSUM_COMMS    — per-packet checksums (overhead: ~5%)
+// USE_GRID_REDUCTION     — P2P tree instead of MPI_Allreduce (slower, deterministic)
+// FI_MR_CACHE_MONITOR    — libfabric PLT workaround (env var, not compile-time)
+```
+
+On a known-good system, enable `ACCELERATOR_AWARE_MPI` and disable the others. On a system with known bugs, disable `ACCELERATOR_AWARE_MPI` and enable `GRID_CHECKSUM_COMMS` + `USE_GRID_REDUCTION` as needed.
+
+## Escalation Checklist
+
+Before concluding a bug is in your code:
+
+1. [ ] Can you reproduce with a minimal reproducer (two MPI ranks, no physics code)?
+2. [ ] Does the failure rate correlate with buffer size, process count, or network route?
+3. [ ] Does staging through host memory eliminate the failure?
+4. [ ] Is the failure deterministic for a given input (same answer, always wrong) or stochastic?
+5. [ ] Does the failure appear on a different MPI implementation (e.g. OpenMPI vs MPICH)?
+
+Deterministic wrong answers that reproduce with minimal reproducers and disappear with host-staging are strong evidence of an MPI library bug. File with the MPI library issue tracker with the minimal reproducer.

systems/Booster/config-command

@@ -8,7 +8,6 @@ LIME=/p/home/jusers/boyle2/juwels/gm2dwf/boyle/
     --disable-gparity \
     --disable-fermion-reps \
     --with-lime=$LIME \
-    --enable-accelerator-cshift \
     --disable-unified \
     CXX=nvcc \
     LDFLAGS="-cudart shared " \

systems/Frontier/aggregate_io_scaling.slurm

@@ -0,0 +1,104 @@
+#!/bin/bash -l
+
+# Node count scan.  Everything that could confound the comparison is held
+# fixed and only the number of Lustre clients changes:
+#
+#   local volume   8.8.32.128        = 262144 sites x 576 B = 151 MB / rank
+#   aggregation    k=2, row of 16, 8 extents of 18 MB        (identical at
+#                                                             every node count)
+#   old path       contiguous run = lLattice[0]*576 = 4.6 KB (identical too)
+#
+# So the ratio of contiguous extent between the two layout preserving paths is
+# a constant 4000x across the whole scan, and any change in their relative
+# bandwidth is a property of the client count alone.  That is the variable the
+# 4 node runs could not move.
+#
+# Trim the tail of the run list if you want a smaller allocation; the
+# geometries are independent.
+
+#SBATCH --job-name=ioScale
+#SBATCH --nodes=128
+#SBATCH --ntasks-per-node=8
+#SBATCH --cpus-per-task=7
+#SBATCH --gpus-per-node=8
+#SBATCH --time=02:00:00
+#SBATCH --account=phy157_dwf
+#SBATCH --gpu-bind=none
+#SBATCH --exclusive
+#SBATCH --mem=0
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3 7 6 5 4)
+export NUMA_MAP=(3 3 1 1 2 2 0 0)
+export GPU=\${GPU_MAP[\$SLURM_LOCALID]}
+export NUMA=\${NUMA_MAP[\$SLURM_LOCALID]}
+unset ROCR_VISIBLE_DEVICES
+export HIP_VISIBLE_DEVICES=\$GPU
+
+rank=\$SLURM_PROCID
+CMD="numactl -m \$NUMA -N \$NUMA  \$*"
+exec \$CMD
+EOF
+
+chmod +x ./select_gpu
+SELECT=$SLURM_SUBMIT_DIR/select_gpu
+
+root=$HOME/ParallelIO/systems/Frontier/
+source ${root}/sourceme-rocm7.2.sh
+
+export OMP_NUM_THREADS=7
+export MPICH_GPU_SUPPORT_ENABLED=1
+
+WORK=/lustre/orion/phy157/proj-shared/phy157_dwf/paboyle/aggscale.$SLURM_JOB_ID
+mkdir -p $WORK
+cd       $WORK
+
+# The largest rung writes three files of 154 GB, so budget ~700 GB for the
+# whole scan and check the quota before submitting.
+#
+# Do NOT stripe by default.  Measured on Frontier at 128 nodes / 1024 ranks,
+# 154.6 GB record, best of 3 (MB/s):
+#
+#                     default layout    lfs setstripe -c -1 -S 8M
+#   raw                    97673                81728
+#   MPI-IO lexicographic   31675                77534
+#   aggregate             189535                86063
+#
+# The two layout preserving paths want opposite things.  Wide striping is what
+# rescues MPI-IO at scale, 32 -> 78 GB/s, which is presumably what careful
+# striping has been buying in production.  It costs the aggregate path a factor
+# of 2.2, and costs raw as well.  With the default layout MPI-IO plateaus at
+# ~30 GB/s from 64 nodes upward -- the 20-40 GB/s seen in production -- while
+# the aggregate path keeps scaling.  Uncomment only to reproduce that.
+#lfs setstripe -c -1 -S 8M $WORK
+lfs getstripe -d $WORK      # record the inherited layout; needed to read the numbers
+
+BIN=$SLURM_SUBMIT_DIR/tests/IO/Test_aggregate_io
+ARGS="--aggregate-target 4194304 --io-no-correctness --io-reps 3"
+
+run () {   # run <nodes> <grid> <mpi> <comment>
+  local nodes=$1 nranks=$(( $1 * 8 ))
+  echo
+  echo "==================================================================="
+  echo "=== SCAN nodes=$nodes ranks=$nranks  grid=$2  mpi=$3   $4"
+  echo "==================================================================="
+  srun -N$nodes -n$nranks --ntasks-per-node=8 $SELECT $BIN --grid $2 --mpi $3 \
+       --accelerator-threads 8 --shm 2048 --device-mem 32000 $ARGS
+  echo "=== exit $?"
+}
+
+#            nodes  global lattice      decomposition   record
+run    4     32.32.64.128    4.4.2.1   "4.8 GB"
+run    8     32.32.64.256    4.4.2.2   "9.7 GB"
+run   16     32.32.64.512    4.4.2.4   "19.3 GB"
+run   32     32.32.128.512   4.4.4.4   "38.6 GB"
+run   64     32.32.128.1024  4.4.4.8   "77.3 GB"
+run  128     32.32.256.1024  4.4.8.8   "154.6 GB"
+
+echo
+echo "=== done.  Files left in $WORK"
+echo "=== For cold read numbers, submit a SECOND job on a fresh allocation"
+echo "=== with WORK pointed at this directory and --io-read-only added to"
+echo "=== ARGS.  Reading back what this job wrote only measures page cache."
+ls -l $WORK

systems/Frontier/config-command

@@ -1,4 +1,5 @@
 CLIME=`spack find --paths c-lime@2-3-9 | grep c-lime| cut -c 15-`
+
 ../../configure --enable-comms=mpi-auto \
 --with-lime=$CLIME \
 --enable-unified=no \
@@ -9,12 +10,13 @@ CLIME=`spack find --paths c-lime@2-3-9 | grep c-lime| cut -c 15-`
 --disable-gparity \
 --disable-fermion-reps \
 --enable-simd=GPU \
---with-gmp=$OLCF_GMP_ROOT \
---with-mpfr=/opt/cray/pe/gcc/mpfr/3.1.4/ \
+--with-openssl=$OPENSSL \
+--with-gmp=$GMP \
+--with-mpfr=$MPFR \
 --disable-fermion-reps \
 CXX=hipcc MPICXX=mpicxx \
-CXXFLAGS="-fPIC -I${ROCM_PATH}/include/ -I${MPICH_DIR}/include -L/lib64 " \
- LDFLAGS="-L/lib64 -L${ROCM_PATH}/lib -L${MPICH_DIR}/lib -lmpi -L${CRAY_MPICH_ROOTDIR}/gtl/lib -lmpi_gtl_hsa -lhipblas -lrocblas -lhipfft"
+CXXFLAGS="-fPIC -I${ROCM_PATH}/include/ -I${MPICH_DIR}/include " \
+ LDFLAGS="-L${ROCM_PATH}/lib -L${MPICH_DIR}/lib -lmpi -lmpi_gtl_hsa -lhipblas -lrocsolver -lrocblas -lhipfft -lamdhip64"
 
 
 

systems/Frontier/sourceme-rocm6.4.sh

@@ -0,0 +1,15 @@
+
+echo spack
+. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
+
+export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
+export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
+export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
+export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
+
+module load cce/20.0.0
+module load cpe/25.09
+module load rocm/6.4.0
+export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/rocm-6.4.0/lib/llvm/lib/:$LD_LIBRARY_PATH
+module load emacs

systems/Frontier/sourceme-rocm7.0.sh

@@ -0,0 +1,13 @@
+
+echo spack
+. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
+
+
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.0.2
+
+export LD_LIBRARY_PATH=/opt/rocm-7.0.2/lib/llvm/lib/:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CRAY_LD_LIBRARY_PATH
+
+module load emacs

systems/Frontier/sourceme-rocm7.2.sh

@@ -0,0 +1,15 @@
+
+echo spack
+. /autofs/nccs-svm1_home1/paboyle/spack/share/spack/setup-env.sh
+
+export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
+export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
+export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
+export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
+
+module load cce/21.0.0
+module load cpe/26.03
+module load rocm/7.2.0
+export LD_LIBRARY_PATH=/opt/rocm-7.2.0/lib/llvm/lib/:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CRAY_LD_LIBRARY_PATH
+module load emacs

systems/Frontier/sourceme.sh

@@ -1,25 +0,0 @@
-
-echo spack
-. /autofs/nccs-svm1_home1/paboyle/Crusher/Grid/spack/share/spack/setup-env.sh
-
-module load cce/15.0.1
-module load rocm/5.3.0
-module load cray-fftw
-module load craype-accel-amd-gfx90a
-
-#Ugly hacks to get down level software working on current system
-export LD_LIBRARY_PATH=/opt/cray/libfabric/1.20.1/lib64/:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=/opt/gcc/mpfr/3.1.4/lib:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=`pwd`/:$LD_LIBRARY_PATH
-ln -s /opt/rocm-6.0.0/lib/libamdhip64.so.6 .
-
-#echo spack load c-lime
-#spack load c-lime
-#module load emacs 
-##module load PrgEnv-gnu
-##module load cray-mpich
-##module load cray-fftw
-##module load craype-accel-amd-gfx90a
-##export LD_LIBRARY_PATH=/opt/gcc/mpfr/3.1.4/lib:$LD_LIBRARY_PATH
-#Hack for lib
-##export LD_LIBRARY_PATH=`pwd`/:$LD_LIBRARY_PATH

systems/Perlmutter/config-command

@@ -1,17 +1,16 @@
 DIR=`pwd`
-PREFIX=$DIR/../Prequisites/install/
+
 ../../configure \
     --enable-comms=mpi \
     --enable-simd=GPU \
     --enable-shm=nvlink \
     --enable-gen-simd-width=64 \
+    --with-gmp=$GMP \
+    --with-mpfr=$MPFR \
     --enable-accelerator=cuda \
-    --enable-setdevice \
-    --disable-accelerator-cshift \
-    --with-gmp=$PREFIX \
     --disable-fermion-reps \
     --disable-unified \
     --disable-gparity \
     CXX=nvcc \
     LDFLAGS="-cudart shared " \
-    CXXFLAGS="-ccbin CC -gencode arch=compute_80,code=sm_80 -std=c++14 -cudart shared"
+    CXXFLAGS="-ccbin CC -gencode arch=compute_80,code=sm_80 -std=c++17 -cudart shared"

systems/Perlmutter/dwf384.slurm

@@ -0,0 +1,39 @@
+#!/bin/bash
+##SBATCH -A m5294_g
+#SBATCH -A mp13_g
+#m3886_g
+#SBATCH -C gpu
+#SBATCH -q premium
+#SBATCH -t 00:20
+#SBATCH -c 32
+#SBATCH -N 384
+#SBATCH -n 1536
+#SBATCH --ntasks-per-node=4
+#SBATCH --gpus-per-task=1
+#SBATCH --exclusive
+#SBATCH --gpu-bind=none
+
+export SLURM_CPU_BIND="cores"
+export MPICH_GPU_SUPPORT_ENABLED=1
+export MPICH_RDMA_ENABLED_CUDA=1
+export MPICH_GPU_IPC_ENABLED=1
+export MPICH_GPU_EAGER_REGISTER_HOST_MEM=0
+export MPICH_GPU_NO_ASYNC_MEMCPY=0
+#export MPICH_SMP_SINGLE_COPY_MODE=CMA
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3)
+export NUMA_MAP=( 0 1 2 3 )
+export GPU=\$SLURM_LOCALID
+export NUMA=\$SLURM_LOCALID
+export CUDA_VISIBLE_DEVICES=\$GPU
+exec numactl -m \$NUMA -N \$NUMA \$*
+EOF
+
+chmod +x ./select_gpu
+
+export VOL=128.128.128.288
+OPT="--comms-overlap --shm-mpi 0"
+srun ./select_gpu ./benchmarks/Benchmark_dwf_fp32 --mpi 4.8.4.12 --grid $VOL --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT
+

systems/Perlmutter/dwf4.slurm

@@ -1,5 +1,6 @@
 #!/bin/bash
-#SBATCH -A m3886_g
+#SBATCH -A m5294_g
+#m3886_g
 #SBATCH -C gpu
 #SBATCH -q debug
 #SBATCH -t 0:20:00
@@ -19,9 +20,21 @@ export MPICH_GPU_EAGER_REGISTER_HOST_MEM=0
 export MPICH_GPU_NO_ASYNC_MEMCPY=0
 #export MPICH_SMP_SINGLE_COPY_MODE=CMA
 
-OPT="--comms-sequential --shm-mpi 1"
-VOL=64.64.64.64
-srun ./benchmarks/Benchmark_dwf_fp32 --mpi 2.2.1.1 --grid $VOL --accelerator-threads 8 --shm 2048 $OPT
-#srun ./benchmarks/Benchmark_dwf_fp32 --mpi 2.1.1.4 --grid $VOL --accelerator-threads 8 --shm 2048 $OPT
-#srun ./benchmarks/Benchmark_dwf_fp32 --mpi 1.1.1.8 --grid $VOL --accelerator-threads 8 --shm 2048 $OPT
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3)
+export NUMA_MAP=( 0 1 2 3 )
+export GPU=\$SLURM_LOCALID
+export NUMA=\$SLURM_LOCALID
+export CUDA_VISIBLE_DEVICES=\$GPU
+exec numactl -m \$NUMA -N \$NUMA \$*
+EOF
+
+chmod +x ./select_gpu
+
+OPT="--comms-sequential --shm-mpi 0"
+VOL=64.64.32.32
+srun ./select_gpu ./benchmarks/Benchmark_dwf_fp32 --mpi 2.2.1.1 --grid $VOL --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT
+OPT="--comms-overlap --shm-mpi 0"
+srun ./select_gpu ./benchmarks/Benchmark_dwf_fp32 --mpi 2.2.1.1 --grid $VOL --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT
 

systems/Perlmutter/sourceme.sh

@@ -1,4 +1,6 @@
-
 export CRAY_ACCEL_TARGET=nvidia80
+source /global/homes/p/pboyle/spack/share/spack/setup-env.sh
+export MPFR=`spack find --paths mpfr | grep mpfr | cut -c 13-`
+export GMP=`spack find --paths gmp   | grep gmp  | cut -c 12-`
 
-module load PrgEnv-gnu cpe-cuda cudatoolkit/11.4
+module load PrgEnv-gnu cpe-cuda cudatoolkit/12.0

systems/Perlmutter/tests/cg32.slurm

@@ -0,0 +1,44 @@
+#!/bin/bash
+##SBATCH -A m5294_g
+#SBATCH -A mp13_g
+#m3886_g
+#SBATCH -C gpu
+#SBATCH -q premium
+#SBATCH -t 00:10
+#SBATCH -c 32
+#SBATCH -N 32
+#SBATCH -n 128
+#SBATCH --ntasks-per-node=4
+#SBATCH --gpus-per-task=1
+#SBATCH --exclusive
+#SBATCH --gpu-bind=none
+
+export SLURM_CPU_BIND="cores"
+export MPICH_GPU_SUPPORT_ENABLED=1
+export MPICH_RDMA_ENABLED_CUDA=1
+export MPICH_GPU_IPC_ENABLED=1
+export MPICH_GPU_EAGER_REGISTER_HOST_MEM=0
+export MPICH_GPU_NO_ASYNC_MEMCPY=0
+#export MPICH_SMP_SINGLE_COPY_MODE=CMA
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3)
+export NUMA_MAP=( 0 1 2 3 )
+export GPU=\$SLURM_LOCALID
+export NUMA=\$SLURM_LOCALID
+export CUDA_VISIBLE_DEVICES=\$GPU
+exec numactl -m \$NUMA -N \$NUMA \$*
+EOF
+
+chmod +x ./select_gpu
+
+OPT="--comms-overlap --shm-mpi 0"
+#
+# Local volume WAS 32.16.32.24
+#
+# 384 nodes
+#srun ./select_gpu ./Test_dwf_mixedcg_prec --seconds 300 --grid 128.128.128.288 --mpi 4.8.4.12 --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT > job.log
+# 32 nodes, same volume per node
+srun ./select_gpu ./Test_dwf_mixedcg_prec --seconds 300 --grid 64.32.64.96 --mpi 2.2.2.4 --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT > job.log
+

systems/Perlmutter/tests/cg384.slurm

@@ -0,0 +1,38 @@
+#!/bin/bash
+##SBATCH -A m5294_g
+#SBATCH -A mp13_g
+#m3886_g
+#SBATCH -C gpu
+#SBATCH -q premium
+#SBATCH -t 00:10
+#SBATCH -c 32
+#SBATCH -N 384
+#SBATCH -n 1536
+#SBATCH --ntasks-per-node=4
+#SBATCH --gpus-per-task=1
+#SBATCH --exclusive
+#SBATCH --gpu-bind=none
+
+export SLURM_CPU_BIND="cores"
+export MPICH_GPU_SUPPORT_ENABLED=1
+export MPICH_RDMA_ENABLED_CUDA=1
+export MPICH_GPU_IPC_ENABLED=1
+export MPICH_GPU_EAGER_REGISTER_HOST_MEM=0
+export MPICH_GPU_NO_ASYNC_MEMCPY=0
+#export MPICH_SMP_SINGLE_COPY_MODE=CMA
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3)
+export NUMA_MAP=( 0 1 2 3 )
+export GPU=\$SLURM_LOCALID
+export NUMA=\$SLURM_LOCALID
+export CUDA_VISIBLE_DEVICES=\$GPU
+exec numactl -m \$NUMA -N \$NUMA \$*
+EOF
+
+chmod +x ./select_gpu
+
+OPT="--comms-overlap --shm-mpi 0"
+srun ./select_gpu ./Test_dwf_mixedcg_prec --seconds 300 --grid 128.128.128.288 --mpi 4.8.4.12 --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT > job.log
+

systems/Perlmutter/usqcd4.slurm

@@ -0,0 +1,39 @@
+#!/bin/bash
+#SBATCH -A m5294_g
+#m3886_g
+#SBATCH -C gpu
+#SBATCH -q debug
+#SBATCH -t 0:30:00
+#SBATCH -c 32
+#SBATCH -N 4
+#SBATCH -n 16
+#SBATCH --ntasks-per-node=4
+#SBATCH --gpus-per-task=1
+#SBATCH --exclusive
+#SBATCH --gpu-bind=none
+
+export SLURM_CPU_BIND="cores"
+export MPICH_GPU_SUPPORT_ENABLED=1
+export MPICH_RDMA_ENABLED_CUDA=1
+export MPICH_GPU_IPC_ENABLED=1
+export MPICH_GPU_EAGER_REGISTER_HOST_MEM=0
+export MPICH_GPU_NO_ASYNC_MEMCPY=0
+#export MPICH_SMP_SINGLE_COPY_MODE=CMA
+
+cat << EOF > select_gpu
+#!/bin/bash
+export GPU_MAP=(0 1 2 3)
+export NUMA_MAP=( 0 1 2 3 )
+export GPU=\$SLURM_LOCALID
+export NUMA=\$SLURM_LOCALID
+export CUDA_VISIBLE_DEVICES=\$GPU
+exec numactl -m \$NUMA -N \$NUMA \$*
+EOF
+
+chmod +x ./select_gpu
+
+OPT="--comms-sequential --shm-mpi 0"
+VOL=64.64.32.32
+srun ./select_gpu ./benchmarks/Benchmark_usqcd --mpi 2.2.2.2 --grid $VOL --device-mem 16000 --accelerator-threads 8 --shm 2048 $OPT > usqcd.log
+
+

systems/mac-arm/config-command-mpi

@@ -3,7 +3,10 @@
 CXX=mpicxx ../../configure \
 	   --enable-simd=GEN \
 	   --enable-comms=mpi-auto \
-	   --enable-Sp=yes \
+	   --enable-Sp=no \
+	   --disable-fermion-reps \
+	   --disable-gparity \
+	   --with-fftw=$FFTW \
 	   --enable-unified=yes \
 	   --prefix /Users/peterboyle/QCD/vtk/Grid/install \
 	   --with-lime=$CLIME \

systems/mac-arm/sourceme.sh

@@ -0,0 +1,11 @@
+source /Users/peterboyle/QCD//Spack/spack//share/spack/setup-env.sh
+
+export FFTW=`spack find --paths fftw    | grep ^fftw   | awk '{print $2}' `
+#export HDF5=`spack find --paths hdf5+cxx   | grep ^hdf5   | awk '{print $2}' `
+export CLIME=`spack find --paths c-lime | grep ^c-lime | awk '{print $2}' `
+export MPFR=`spack find --paths mpfr    | grep ^mpfr  | awk '{print $2}' `
+export OPENSSL=`spack find --paths openssl | grep openssl | awk  '{print $2}' `
+export GMP=`spack find --paths gmp      | grep ^gmp | awk '{print $2}' `
+
+export LD_LIBRARY_PATH=$MPFR/lib:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$GMP/lib:$LD_LIBRARY_PATH

tests/IO/Test_aggregate_io.cc

@@ -0,0 +1,458 @@
+    /*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/IO/Test_aggregate_io.cc
+
+    Copyright (C) 2015
+
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+
+// Correctness and performance test for BINARYIO_AGGREGATE.
+//
+// Correctness, per aggregateTargetBytes:
+//   1. write via the MPI-IO lexicographic path      -> ref.bin
+//   2. write via the aggregate path                 -> agg.bin
+//   3. the two files must be byte identical         <- proves the layout matches
+//   4. read agg.bin back through the aggregate path <- proves the mirror inverts
+//   5. write and read the non-lexicographic path    -> raw.bin.  Its layout is
+//      different by construction (each rank owns one contiguous segment in
+//      rank order) so it cannot be compared byte for byte, but the NERSC and
+//      SciDAC checksums are computed from the global site index and are
+//      therefore layout independent: they must match the other two paths.
+//   6. a record written over a longer pre-existing file must leave the file at
+//      exactly offset+payload, with no trailing fragment of the old contents
+//
+// Performance: three paths, both directions, timed with the client page cache
+// dropped before every read so that a read back reports filesystem bandwidth
+// rather than memory bandwidth.  The non-lexicographic path is the zero
+// overhead reference: no transposition, no layout independence, one disjoint
+// contiguous segment per rank, which is the arrangement that reaches full
+// filesystem bandwidth on a leadership machine.  It is the upper bound the
+// other two are trying to approach.
+//
+// Options:
+//   --aggregate-target <bytes>   sweep this one target only (default: sweep
+//                                1, 1024, 64K, 4M)
+//   --io-reps <n>                repetitions in the performance section
+//                                (default 3; 0 disables it)
+//   --io-no-correctness          skip the correctness section, which reads the
+//                                whole file on one rank and is not affordable
+//                                at very large volume
+//   --io-read-only               time reads only, of files left in place by an
+//                                earlier job.  Reading back what this job just
+//                                wrote measures the client page cache; a fresh
+//                                allocation pointed at the same directory is
+//                                the only way to get a cold read without root.
+//
+// The exchange is only meaningfully exercised when the fast dimensions are
+// split across ranks; --mpi 1.1.X.Y leaves the rows of size one and the test
+// then passes vacuously.  Non-uniform AllToAllV counts additionally need an
+// odd process factor in a fast dimension and a small local volume.
+
+#include <Grid/Grid.h>
+#include <fcntl.h>
+
+using namespace Grid;
+
+/////////////////////////////////////////////////////////////////////////////
+// Compare in chunks.  Slurping both files into memory is fine for a few MB
+// and fatal for the multi-GB records this test is meant to reach.
+/////////////////////////////////////////////////////////////////////////////
+static bool FilesIdentical(std::string a,std::string b)
+{
+  std::ifstream fa(a,std::ios::binary), fb(b,std::ios::binary);
+  if ( !fa.good() || !fb.good() ) {
+    std::cout<<GridLogMessage<<"  could not open "<<a<<" and/or "<<b<<std::endl;
+    return false;
+  }
+  fa.seekg(0,std::ios::end); fb.seekg(0,std::ios::end);
+  uint64_t sa = (uint64_t)fa.tellg(), sb = (uint64_t)fb.tellg();
+  if ( sa != sb ) {
+    std::cout<<GridLogMessage<<"  size mismatch "<<sa<<" vs "<<sb<<std::endl;
+    return false;
+  }
+  fa.seekg(0,std::ios::beg); fb.seekg(0,std::ios::beg);
+
+  const uint64_t chunk = 8*1024*1024;
+  std::vector<char> va(chunk), vb(chunk);
+  uint64_t done=0;
+  while ( done < sa ) {
+    uint64_t n = std::min(chunk,sa-done);
+    fa.read(&va[0],n);
+    fb.read(&vb[0],n);
+    for(uint64_t i=0;i<n;i++){
+      if ( va[i]!=vb[i] ) {
+        std::cout<<GridLogMessage<<"  first differing byte at "<<done+i<<" of "<<sa<<std::endl;
+        return false;
+      }
+    }
+    done += n;
+  }
+  return true;
+}
+
+/////////////////////////////////////////////////////////////////////////////
+// Reading back a file we have just written measures the client page cache,
+// not the filesystem: the earlier runs of this test reported 8 GB/s on reads
+// and ~1 GB/s on writes for the same data.  POSIX_FADV_DONTNEED asks the
+// kernel to drop the cached pages for the file.  It is advisory and every
+// rank must do it, since each client caches independently, so treat this as
+// best effort rather than a guarantee of a cold read.
+/////////////////////////////////////////////////////////////////////////////
+static void DropCache(GridBase *grid,std::string file)
+{
+  grid->Barrier();
+  int fd = ::open(file.c_str(),O_RDONLY);
+  if ( fd >= 0 ) {
+#ifdef POSIX_FADV_DONTNEED
+    ::posix_fadvise(fd,0,0,POSIX_FADV_DONTNEED);
+#endif
+    ::close(fd);
+  }
+  grid->Barrier();
+}
+
+static uint64_t OptionU64(int argc,char **argv,const char *opt,uint64_t def)
+{
+  if ( GridCmdOptionExists(argv,argv+argc,opt) ) {
+    std::string arg = GridCmdOptionPayload(argv,argv+argc,opt);
+    return (uint64_t)std::stoull(arg);
+  }
+  return def;
+}
+
+int main(int argc,char **argv)
+{
+  Grid_init(&argc,&argv);
+
+  Coordinate latt = GridDefaultLatt();
+  Coordinate simd = GridDefaultSimd(Nd,vComplexD::Nsimd());
+  Coordinate mpi  = GridDefaultMpi();
+  GridCartesian grid(latt,simd,mpi);
+
+  typedef vLorentzColourMatrixD vobj;
+  typedef LorentzColourMatrixD  sobj;
+
+  GridParallelRNG pRNG(&grid);
+  pRNG.SeedFixedIntegers(std::vector<int>({1,2,3,4}));
+  LatticeGaugeFieldD Umu(&grid);
+  random(pRNG,Umu);
+
+  BinarySimpleMunger<sobj,sobj> munge;
+  const std::string format("IEEE64BIG");
+
+  const int lex = BinaryIO::BINARYIO_LEXICOGRAPHIC;
+  const int agg = BinaryIO::BINARYIO_LEXICOGRAPHIC|BinaryIO::BINARYIO_AGGREGATE;
+  const int raw = 0;   // no BINARYIO_LEXICOGRAPHIC: contiguous segment per rank
+
+  uint64_t payload = (uint64_t)grid._gsites*sizeof(sobj);
+
+  std::vector<uint64_t> targets = {1, 1024, 64*1024, 4*1024*1024};
+  if ( GridCmdOptionExists(argv,argv+argc,"--aggregate-target") ) {
+    targets.clear();
+    targets.push_back(OptionU64(argc,argv,"--aggregate-target",4*1024*1024));
+  }
+  uint64_t reps        = OptionU64(argc,argv,"--io-reps",3);
+  bool     correctness = !GridCmdOptionExists(argv,argv+argc,"--io-no-correctness");
+  // Read only: time reads of files left by an earlier job.  The only way to
+  // get a cold client cache without root is to read on an allocation that did
+  // not write the data, so run one job to write and a second, pointed at the
+  // same directory, with this flag.
+  bool     readonly    = GridCmdOptionExists(argv,argv+argc,"--io-read-only");
+  if ( readonly ) correctness = false;
+
+  std::cout<<GridLogMessage<<"Record payload "<<payload<<" bytes = "
+           <<payload/1024./1024.<<" MB, "
+           <<payload/(RealD)grid.ProcessorCount()/1024./1024.<<" MB/rank"<<std::endl;
+
+  int failures=0;
+
+  //////////////////////////////////////////////////////////////////////////
+  // Correctness
+  //////////////////////////////////////////////////////////////////////////
+  if ( correctness ) for(auto target : targets){
+
+    std::cout<<GridLogMessage<<"=== correctness, aggregateTargetBytes = "<<target<<" ==="<<std::endl;
+
+    uint32_t n1,a1,b1, n2,a2,b2, n3,a3,b3;
+    uint64_t off;
+
+    // Start from a clean slate.  The aggregate path sets the file length to
+    // exactly offset+payload; the MPI-IO path (MPI_MODE_CREATE) leaves any
+    // pre-existing tail in place.  Comparing stale files would therefore
+    // report a size mismatch that says nothing about the payload.
+    if ( grid.IsBoss() ) { ::unlink("ref.bin"); ::unlink("agg.bin"); ::unlink("raw.bin"); }
+    grid.Barrier();
+
+    off=0;
+    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"ref.bin",munge,off,format,n1,a1,b1,lex);
+
+    BinaryIO::aggregateTargetBytes = target;
+    off=0;
+    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"agg.bin",munge,off,format,n2,a2,b2,agg);
+
+    grid.Barrier();
+
+    if ( grid.IsBoss() ) {
+      if ( !FilesIdentical("ref.bin","agg.bin") ) {
+        std::cout<<GridLogError<<"  FAIL: aggregate file differs from lexicographic file"<<std::endl;
+        failures++;
+      } else {
+        std::cout<<GridLogMessage<<"  files byte identical"<<std::endl;
+      }
+    }
+    if ( (n1!=n2)||(a1!=a2)||(b1!=b2) ) {
+      std::cout<<GridLogError<<"  FAIL: checksum mismatch between paths"<<std::endl;
+      failures++;
+    }
+    // writeLatticeObject takes offset by value, so the out-parameter that
+    // IOobject sets never reaches us here and cannot be checked directly.
+    // The observable equivalent is the file length: both paths must leave the
+    // record ending at exactly offset+payload.
+    if ( grid.IsBoss() ) {
+      for(auto f : {std::string("ref.bin"),std::string("agg.bin")}){
+        std::ifstream fs(f,std::ios::binary|std::ios::ate);
+        uint64_t sz = (uint64_t)fs.tellg();
+        if ( sz != payload ) {
+          std::cout<<GridLogError<<"  FAIL: "<<f<<" is "<<sz<<" bytes, expected "<<payload<<std::endl;
+          failures++;
+        }
+      }
+    }
+
+    LatticeGaugeFieldD Uchk(&grid);
+    DropCache(&grid,"agg.bin");
+    off=0;
+    BinaryIO::readLatticeObject<vobj,sobj>(Uchk,"agg.bin",munge,off,format,n3,a3,b3,agg);
+    if ( (n3!=n1)||(a3!=a1)||(b3!=b1) ) {
+      std::cout<<GridLogError<<"  FAIL: read back checksum mismatch"<<std::endl;
+      failures++;
+    }
+    Uchk = Uchk - Umu;
+    RealD residual = norm2(Uchk);
+    std::cout<<GridLogMessage<<"  read back residual "<<residual<<std::endl;
+    if ( residual != 0.0 ) {
+      std::cout<<GridLogError<<"  FAIL: read back does not reproduce the field"<<std::endl;
+      failures++;
+    }
+
+    ////////////////////////////////////////////////////////////////////////
+    // Non-lexicographic.  Different file layout by construction, so compare
+    // by checksum and by round trip rather than by bytes.
+    ////////////////////////////////////////////////////////////////////////
+    uint32_t n4,a4,b4, n5,a5,b5;
+    off=0;
+    BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"raw.bin",munge,off,format,n4,a4,b4,raw);
+    grid.Barrier();
+    if ( (n4!=n1)||(a4!=a1)||(b4!=b1) ) {
+      std::cout<<GridLogError<<"  FAIL: raw path checksum differs; the NERSC and"
+               <<" SciDAC checksums are layout independent and must agree"<<std::endl;
+      failures++;
+    }
+    DropCache(&grid,"raw.bin");
+    off=0;
+    BinaryIO::readLatticeObject<vobj,sobj>(Uchk,"raw.bin",munge,off,format,n5,a5,b5,raw);
+    if ( (n5!=n1)||(a5!=a1)||(b5!=b1) ) {
+      std::cout<<GridLogError<<"  FAIL: raw read back checksum mismatch"<<std::endl;
+      failures++;
+    }
+    Uchk = Uchk - Umu;
+    RealD rawresidual = norm2(Uchk);
+    std::cout<<GridLogMessage<<"  raw read back residual "<<rawresidual<<std::endl;
+    if ( rawresidual != 0.0 ) {
+      std::cout<<GridLogError<<"  FAIL: raw read back does not reproduce the field"<<std::endl;
+      failures++;
+    }
+  }
+
+  //////////////////////////////////////////////////////////////////////////
+  // Truncation.  offset!=0 is the case that matters: it is what ILDG and
+  // NERSC use, and it is the branch that used to only ever grow the file.
+  // The oversized starting file is made by extending a sparse one rather
+  // than writing padding from a single rank, which does not scale.
+  //////////////////////////////////////////////////////////////////////////
+  if ( correctness ) {
+    BinaryIO::aggregateTargetBytes = 4*1024*1024;
+    for(uint64_t testOffset : {(uint64_t)0, (uint64_t)1024}){
+
+      uint64_t expect = testOffset + payload;
+
+      if ( grid.IsBoss() ) {
+        { std::ofstream create("trunc.bin",std::ios::binary|std::ios::out); create.close(); }
+        int ierr = ::truncate("trunc.bin",(off_t)(expect+65536));
+        GRID_ASSERT(ierr==0);
+      }
+      grid.Barrier();
+
+      uint32_t n,a,b;
+      uint64_t off = testOffset;
+      BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"trunc.bin",munge,off,format,n,a,b,agg);
+      grid.Barrier();
+
+      if ( grid.IsBoss() ) {
+        std::ifstream f("trunc.bin",std::ios::binary|std::ios::ate);
+        uint64_t sz = (uint64_t)f.tellg();
+        f.close();
+        if ( sz != expect ) {
+          std::cout<<GridLogError<<"  FAIL: offset "<<testOffset<<" left file "<<sz
+                   <<" bytes, expected "<<expect<<std::endl;
+          failures++;
+        } else {
+          std::cout<<GridLogMessage<<"  truncation ok at offset "<<testOffset
+                   <<": file is exactly "<<sz<<" bytes"<<std::endl;
+        }
+      }
+    }
+  }
+
+  //////////////////////////////////////////////////////////////////////////
+  // Performance.  Four numbers per repetition: write and read, old path and
+  // new.  Reads are preceded by a cache drop; writes are not, so a write
+  // number is "time to hand the data to the client cache and close", the
+  // same convention for both paths.
+  //////////////////////////////////////////////////////////////////////////
+  if ( reps ) {
+
+    LatticeGaugeFieldD Uio(&grid);
+    uint32_t n,a,b;
+
+    // The NERSC and SciDAC checksums are computed from the global site index,
+    // so all three layouts must produce the same values.  This costs nothing
+    // and is the only correctness check available at a volume where the byte
+    // for byte comparison (single rank, whole file) is unaffordable.
+    uint32_t cn[6],ca[6],cb[6];
+    auto agreeing = [&](const char *what,int lo,int hi){
+      for(int i=lo+1;i<=hi;i++){
+        if ( (cn[i]!=cn[lo])||(ca[i]!=ca[lo])||(cb[i]!=cb[lo]) ) {
+          std::cout<<GridLogError<<"  FAIL: "<<what<<" checksums disagree between paths"<<std::endl;
+          return false;
+        }
+      }
+      return true;
+    };
+
+    for(auto target : targets){
+
+      BinaryIO::aggregateTargetBytes = target;
+      std::cout<<GridLogMessage<<"=== performance, aggregateTargetBytes = "<<target<<" ==="<<std::endl;
+
+      std::vector<RealD> wref,wagg,wraw,rref,ragg,rraw;
+
+      for(uint64_t rep=0;rep<reps;rep++){
+
+        uint64_t off;
+
+        // Unlink only before the first repetition.  Lustre metadata cost is
+        // per file, not per byte, so rep 0 reports "create the file and write
+        // it" and the later reps report the steady state of overwriting an
+        // existing file -- which is what a multi record file does for every
+        // record after the first, and what production actually looks like.
+        if ( (rep==0) && !readonly && grid.IsBoss() ) {
+          ::unlink("ref.bin"); ::unlink("agg.bin"); ::unlink("raw.bin");
+        }
+        grid.Barrier();
+
+        if ( !readonly ) {
+
+        off=0;
+        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"ref.bin",munge,off,format,n,a,b,lex);
+        wref.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[0]=n; ca[0]=a; cb[0]=b;
+
+        off=0;
+        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"agg.bin",munge,off,format,n,a,b,agg);
+        wagg.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[1]=n; ca[1]=a; cb[1]=b;
+
+        off=0;
+        BinaryIO::writeLatticeObject<vobj,sobj>(Umu,"raw.bin",munge,off,format,n,a,b,raw);
+        wraw.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[2]=n; ca[2]=a; cb[2]=b;
+        if ( !agreeing("write",0,2) ) failures++;
+
+        }  // !readonly
+
+        DropCache(&grid,"ref.bin");
+        off=0;
+        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"ref.bin",munge,off,format,n,a,b,lex);
+        rref.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[3]=n; ca[3]=a; cb[3]=b;
+
+        DropCache(&grid,"agg.bin");
+        off=0;
+        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"agg.bin",munge,off,format,n,a,b,agg);
+        ragg.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[4]=n; ca[4]=a; cb[4]=b;
+
+        DropCache(&grid,"raw.bin");
+        off=0;
+        BinaryIO::readLatticeObject<vobj,sobj>(Uio,"raw.bin",munge,off,format,n,a,b,raw);
+        rraw.push_back(BinaryIO::lastPerf.mbytesPerSecond);
+        cn[5]=n; ca[5]=a; cb[5]=b;
+        if ( !agreeing("read back",readonly?3:0,5) ) failures++;
+      }
+
+      if ( grid.IsBoss() ) {
+        auto report = [&](const char *name,std::vector<RealD> &v){
+          if ( v.empty() ) return;
+          RealD best=0, sum=0;
+          for(auto x : v){ if(x>best) best=x; sum+=x; }
+          // First sample includes file creation, later ones do not; quote both
+          // rather than a mean that mixes the two.
+          std::cout<<GridLogMessage<<"  PERF target="<<target<<" "<<name
+                   <<" best "<<best<<" MB/s, mean "<<sum/v.size()
+                   <<" MB/s, first(cold create) "<<v[0]<<" MB/s, samples";
+          for(auto x : v) std::cout<<" "<<x;
+          std::cout<<std::endl;
+        };
+        report("write raw       ",wraw);   // zero overhead reference
+        report("write MPI-IO    ",wref);
+        report("write aggregate ",wagg);
+        report("read  raw       ",rraw);
+        report("read  MPI-IO    ",rref);
+        report("read  aggregate ",ragg);
+
+        // Fraction of the zero overhead reference that each layout preserving
+        // path achieves.  This is the number the whole exercise is about.
+        auto best = [](std::vector<RealD> &v){ RealD m=0; for(auto x:v) if(x>m) m=x; return m; };
+        if ( !wraw.empty() && best(wraw) > 0 ) {
+          std::cout<<GridLogMessage<<"  PERF target="<<target
+                   <<" write fraction of raw: MPI-IO "<<best(wref)/best(wraw)
+                   <<"  aggregate "<<best(wagg)/best(wraw)<<std::endl;
+        }
+        if ( !rraw.empty() && best(rraw) > 0 ) {
+          std::cout<<GridLogMessage<<"  PERF target="<<target
+                   <<" read  fraction of raw: MPI-IO "<<best(rref)/best(rraw)
+                   <<"  aggregate "<<best(ragg)/best(rraw)<<std::endl;
+        }
+      }
+    }
+  }
+
+  if ( grid.IsBoss() ) {
+    if ( failures ) std::cout<<GridLogError  <<failures<<" FAILURE(S)"<<std::endl;
+    else            std::cout<<GridLogMessage<<"ALL AGGREGATE IO TESTS PASSED"<<std::endl;
+  }
+
+  Grid_finalize();
+  return failures!=0;
+}

tests/Test_fft_memory.cc

@@ -0,0 +1,261 @@
+/*************************************************************************************
+
+ Test_fft_memory.cc
+
+ Memory growth test for PlannedFFT on a spin-colour matrix (propagator) field.
+
+ The test creates a single PlannedFFT object (which allocates FFTW plans once),
+ then repeatedly applies FFT_all_dim to the same propagator 400 times.
+
+ If PlannedFFT is working correctly the RSS should remain flat after the first
+ iteration — no new plans, no new deviceVector allocations beyond the per-call
+ pencil buffer which is freed at the end of each FFT_dim_execute call.
+
+ Build exactly like any other Grid test, e.g.:
+   make Test_fft_memory
+ or compile manually:
+   $(CXX) $(CXXFLAGS) Test_fft_memory.cc -o Test_fft_memory $(LDFLAGS)
+
+*************************************************************************************/
+
+#include <Grid/Grid.h>
+using namespace Grid;
+
+// --------------------------------------------------------------------------
+// Helper: read RSS (resident set size) in kB from /proc/self/status.
+// Returns 0 on platforms where /proc is unavailable.
+// --------------------------------------------------------------------------
+static long getCPURSSKb()
+{
+  long rss = 0;
+  FILE *fp = fopen("/proc/self/status", "r");
+  if (!fp) return -1;
+  char line[256];
+  while (fgets(line, sizeof(line), fp)) {
+    if (strncmp(line, "VmRSS:", 6) == 0) {
+      sscanf(line + 6, "%ld", &rss);
+      break;
+    }
+  }
+  fclose(fp);
+  return rss;
+}
+
+static long getGPUUsedMb()
+{
+#if defined(GRID_CUDA)
+  size_t free_bytes  = 0;
+  size_t total_bytes = 0;
+  cudaError_t err = cudaMemGetInfo(&free_bytes, &total_bytes);
+  if (err != cudaSuccess) return -1;
+  return (long)((total_bytes - free_bytes) / (1024 * 1024));
+ 
+#elif defined(GRID_HIP)
+  size_t free_bytes  = 0;
+  size_t total_bytes = 0;
+  hipError_t err = hipMemGetInfo(&free_bytes, &total_bytes);
+  if (err != hipSuccess) return -1;
+  return (long)((total_bytes - free_bytes) / (1024 * 1024));
+ 
+#else
+  return -1;   // CPU-only build: no GPU to query
+#endif
+}
+
+// ============================================================
+//  Convenience struct — one snapshot of both sides
+// ============================================================
+struct MemSnapshot {
+  long cpu_rss_kb;   // host RSS in kB  (-1 if unavailable)
+  long gpu_used_mb;  // device used in MB (-1 if no GPU)
+};
+ 
+static MemSnapshot takeSnapshot()
+{
+  MemSnapshot s;
+  s.cpu_rss_kb  = getCPURSSKb();
+  s.gpu_used_mb = getGPUUsedMb();
+  return s;
+}
+ 
+// ============================================================
+//  Pretty-print one row of the monitoring table
+// ============================================================
+static void printRow(int iter,
+                     const MemSnapshot &now,
+                     const MemSnapshot &prev)
+{
+  long cpu_delta = (now.cpu_rss_kb  >= 0 && prev.cpu_rss_kb  >= 0)
+                 ? now.cpu_rss_kb  - prev.cpu_rss_kb  : 0;
+  long gpu_delta = (now.gpu_used_mb >= 0 && prev.gpu_used_mb >= 0)
+                 ? now.gpu_used_mb - prev.gpu_used_mb : 0;
+ 
+  // Sign prefix so deltas are unambiguous
+  auto sign = [](long v) -> const char* { return v >= 0 ? "+" : ""; };
+ 
+  std::cout << GridLogMessage
+            << std::setw(6)  << iter
+            << "  CPU: " << std::setw(10) << now.cpu_rss_kb  << " kB"
+            << "  (" << sign(cpu_delta) << std::setw(7) << cpu_delta << " kB)"
+            << "  GPU: " << std::setw(7)  << now.gpu_used_mb << " MB"
+            << "  (" << sign(gpu_delta) << std::setw(5) << gpu_delta << " MB)"
+            << "\n";
+}
+
+// ============================================================
+
+int main(int argc, char **argv)
+{
+  Grid_init(&argc, &argv);
+
+  int threads = GridThread::GetThreads();
+  std::cout << GridLogMessage
+            << "Grid is setup to use " << threads << " threads" << std::endl;
+
+  // ------------------------------------------------------------------
+  // Grid setup — use whatever lattice/mpi/simd was passed on the CLI,
+  // e.g.  --grid 8.8.8.8 --mpi 1.1.1.1
+  // ------------------------------------------------------------------
+  Coordinate latt_size   = GridDefaultLatt();
+  Coordinate simd_layout = GridDefaultSimd(Nd, vComplexD::Nsimd());
+  Coordinate mpi_layout  = GridDefaultMpi();
+
+  GridCartesian GRID(latt_size, simd_layout, mpi_layout);
+
+  int vol = 1;
+  for (int d = 0; d < (int)latt_size.size(); d++) vol *= latt_size[d];
+
+  std::cout << GridLogMessage << "Lattice : ";
+  for (int d = 0; d < Nd; d++) std::cout << latt_size[d] << " ";
+  std::cout << std::endl;
+
+  // ------------------------------------------------------------------
+  // Propagator field: SpinColourMatrix = 12x12 complex, i.e.
+  // LatticePropagatorD (= Lattice<iSpinColourMatrix<vComplexD>>).
+  // This is the standard QCD quark propagator type.
+  // ------------------------------------------------------------------
+  LatticePropagatorD prop(&GRID);
+  
+  // ------------------------------------------------------------------
+  // Fill the propagator with a momentum-space plane wave,
+  // following the pattern from Test_fft.cc.
+  // We set each spin-colour component (a,b) to exp(i * sum_mu p_mu x_mu)
+  // with a fixed momentum p = (1,2,1,2).
+  // ------------------------------------------------------------------
+  Coordinate pvec({1, 2, 1, 2});
+
+  LatticeComplexD phase(&GRID);
+  LatticeComplexD coor(&GRID);
+  ComplexD ci(0.0, 1.0);
+
+  phase = Zero();
+  for (int mu = 0; mu < Nd; mu++) {
+    RealD TwoPiL = M_PI * 2.0 / latt_size[mu];
+    LatticeCoordinate(coor, mu);
+    phase = phase + (TwoPiL * pvec[mu]) * coor;
+  }
+  phase = exp(phase * ci);   // e^{i p.x}
+
+  // Broadcast the phase into every spin-colour matrix entry
+  prop = Zero();
+  prop = prop + phase;
+
+  std::cout << GridLogMessage
+            << "Propagator norm2 = " << norm2(prop) << std::endl;
+
+  // ------------------------------------------------------------------
+  // Baseline snapshot BEFORE PlannedFFT construction
+  // ------------------------------------------------------------------
+  MemSnapshot snap_before_plan = takeSnapshot();
+  std::cout << GridLogMessage
+            << "[mem] Before PlannedFFT construction"
+            << "  CPU: " << snap_before_plan.cpu_rss_kb  << " kB"
+            << "  GPU: " << snap_before_plan.gpu_used_mb << " MB"
+            << std::endl;
+ 
+  // ------------------------------------------------------------------
+  // Create the PlannedFFT — plans are allocated here ONCE for all
+  // dimensions and stored inside the object.
+  // ------------------------------------------------------------------
+  PlannedFFT<iSpinColourMatrix<vComplexD>> plannedFFT(&GRID);
+
+  // ------------------------------------------------------------------
+  // Snapshot AFTER plan construction — this is the true baseline
+  // for the loop, because cufftPlanMany itself grabs device memory.
+  // ------------------------------------------------------------------
+  MemSnapshot snap_after_plan = takeSnapshot();
+  std::cout << GridLogMessage
+            << "[mem] After  PlannedFFT construction"
+            << "  CPU: " << snap_after_plan.cpu_rss_kb  << " kB"
+            << "  GPU: " << snap_after_plan.gpu_used_mb << " MB"
+            << "  (plan overhead:"
+            << "  CPU +" << snap_after_plan.cpu_rss_kb - snap_before_plan.cpu_rss_kb << " kB"
+            << "  GPU +" << snap_after_plan.gpu_used_mb - snap_before_plan.gpu_used_mb << " MB)"
+            << std::endl;
+
+  MemoryManager::Print();
+  // ------------------------------------------------------------------
+  // 400-iteration loop.
+  // Each iteration computes the full 4d forward FFT of `prop`.
+  // We deliberately do NOT cache the result — we always start from
+  // the same `prop` so the FFT is recomputed identically each time.
+  // The point is to watch memory, not correctness.
+  // ------------------------------------------------------------------
+  const int Niter = 40;
+  const int Niter2 = 32;
+
+  // Print header for the memory table
+  std::cout << GridLogMessage
+              << "\n"
+              << std::setw(6)  << "iter"
+              << "  CPU: " << std::setw(10) << "RSS[kB]"
+              << "  (  delta  )"
+              << "  GPU: " << std::setw(7)  << "used[MB]"
+              << "  (delta)"
+              << "\n";
+ 
+  MemSnapshot snap_prev = snap_after_plan;
+
+  for (int i = 0; i < Niter; i++) {
+    std::vector<LatticePropagatorD> G;
+
+    for (int j = 0; j < Niter2; j++) {
+      LatticePropagatorD prop_fft(&GRID);
+
+      // Full 4d forward FFT using the pre-built plans
+      plannedFFT.FFT_all_dim(prop_fft, prop, FFT::forward);
+
+      G.push_back(prop_fft);
+    }
+ 
+    // cudaMemGetInfo reflects the state *after* any pooled frees have
+    // been committed, so this is accurate without an explicit sync —
+    // FFT_dim_execute already calls accelerator_barrier() internally.
+    MemSnapshot snap_now = takeSnapshot();
+    printRow(i, snap_now, snap_prev);
+    MemoryManager::Print();
+    snap_prev = snap_now;
+  }
+
+  // ------------------------------------------------------------------
+  // Summary
+  // ------------------------------------------------------------------
+  MemSnapshot snap_final = takeSnapshot();
+ 
+  long cpu_growth = snap_final.cpu_rss_kb  - snap_after_plan.cpu_rss_kb;
+  long gpu_growth = snap_final.gpu_used_mb - snap_after_plan.gpu_used_mb;
+ 
+  std::cout << GridLogMessage
+            << "\n==== Memory summary (baseline = after plan construction) ====\n"
+            << "  CPU RSS growth over " << Niter << " FFTs : "
+            << cpu_growth << " kB"
+            << (cpu_growth == 0 ? "  OK" : "  *** GROWING ***") << "\n"
+            << "  GPU used growth over " << Niter << " FFTs : "
+            << gpu_growth << " MB"
+            << (gpu_growth == 0 ? "  OK" : "  *** GROWING ***") << "\n"
+            << "  Note: first-call watermark from pool fill is expected and benign.\n"
+            << "        A leak shows as continuous growth beyond iter ~2-3.\n";
+ 
+  Grid_finalize();
+  return 0;
+}

tests/core/Test_planned_fft.cc

@@ -0,0 +1,321 @@
+/*************************************************************************************
+
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/core/Test_planned_fft.cc
+
+    Copyright (C) 2015
+
+Author: Azusa Yamaguchi <ayamaguc@staffmail.ed.ac.uk>
+Author: Peter Boyle <paboyle@ph.ed.ac.uk>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+*************************************************************************************/
+/*  END LEGAL */
+#include <Grid/Grid.h>
+
+using namespace Grid;
+
+int main (int argc, char ** argv)
+{
+  Grid_init(&argc,&argv);
+
+  int threads = GridThread::GetThreads();
+  std::cout<<GridLogMessage << "Grid is setup to use "<<threads<<" threads"<<std::endl;
+
+  Coordinate latt_size   = GridDefaultLatt();
+  Coordinate simd_layout = GridDefaultSimd(Nd,vComplexD::Nsimd());
+  Coordinate mpi_layout  = GridDefaultMpi();
+
+  int vol = 1;
+  for(int d=0;d<latt_size.size();d++) vol *= latt_size[d];
+
+  GridCartesian         GRID(latt_size,simd_layout,mpi_layout);
+  GridRedBlackCartesian RBGRID(&GRID);
+
+  LatticeComplexD     one(&GRID);
+  LatticeComplexD      zz(&GRID);
+  LatticeComplexD       C(&GRID);
+  LatticeComplexD  Ctilde(&GRID);
+  LatticeComplexD  Cref  (&GRID);
+  LatticeComplexD  Csav  (&GRID);
+  LatticeComplexD    coor(&GRID);
+
+  LatticeSpinMatrixD    S(&GRID);
+  LatticeSpinMatrixD    Stilde(&GRID);
+
+  Coordinate p({1,3,2,3});
+
+  one = ComplexD(1.0,0.0);
+  zz  = ComplexD(0.0,0.0);
+  ComplexD ci(0.0,1.0);
+
+  std::cout<<"*************************************************"<<std::endl;
+  std::cout<<"Testing Fourier form of known plane wave         "<<std::endl;
+  std::cout<<"*************************************************"<<std::endl;
+  C=Zero();
+  for(int mu=0;mu<4;mu++){
+    RealD TwoPiL =  M_PI * 2.0/ latt_size[mu];
+    LatticeCoordinate(coor,mu);
+    C = C + (TwoPiL * p[mu]) * coor;
+  }
+  C = exp(C*ci);
+  Csav = C;
+  S=Zero();
+  S = S+C;
+
+  // PlannedFFT is templated on the lattice element type (vector_object), not the Lattice<> itself.
+  PlannedFFT<LatticeComplexD::vector_object>    theFFT(&GRID);
+  PlannedFFT<LatticeSpinMatrixD::vector_object> theFFT_spin(&GRID);
+
+  Ctilde=C;
+  std::cout<<" Benchmarking PlannedFFT of LatticeComplex  "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,0,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,1,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,2,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+  theFFT.FFT_dim(Ctilde,Ctilde,3,FFTbase::forward); std::cout << theFFT.MFlops()<<" Mflops "<<std::endl;
+
+  TComplexD cVol;
+  cVol()()() = vol;
+
+  Cref=Zero();
+  pokeSite(cVol,Cref,p);
+
+  Cref=Cref-Ctilde;
+  std::cout << "diff scalar "<<norm2(Cref) << std::endl;
+
+  C=Csav;
+  theFFT.FFT_all_dim(Ctilde,C,FFTbase::forward);
+  theFFT.FFT_all_dim(Cref,Ctilde,FFTbase::backward);
+
+  std::cout << norm2(C) << " " << norm2(Ctilde) << " " << norm2(Cref)<< " vol " << vol<< std::endl;
+
+  Cref= Cref - C;
+  std::cout << " invertible check " << norm2(Cref)<<std::endl;
+
+  Stilde=S;
+  std::cout<<" Benchmarking PlannedFFT of LatticeSpinMatrix  "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,0,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,1,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,2,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+  theFFT_spin.FFT_dim(Stilde,Stilde,3,FFTbase::forward); std::cout << theFFT_spin.MFlops()<<" mflops "<<std::endl;
+
+  SpinMatrixD Sp;
+  Sp = Zero(); Sp = Sp+cVol;
+
+  S=Zero();
+  pokeSite(Sp,S,p);
+
+  S= S-Stilde;
+  std::cout << "diff FT[SpinMat] "<<norm2(S) << std::endl;
+
+  std::vector<int> seeds({1,2,3,4});
+  GridSerialRNG          sRNG;  sRNG.SeedFixedIntegers(seeds);
+  GridParallelRNG          pRNG(&GRID);
+  pRNG.SeedFixedIntegers(seeds);
+
+  LatticeGaugeFieldD Umu(&GRID);
+  SU<Nc>::ColdConfiguration(pRNG,Umu);
+
+  ////////////////////////////////////////////////////
+  // Wilson test
+  ////////////////////////////////////////////////////
+  {
+    LatticeFermionD    src(&GRID); gaussian(pRNG,src);
+    LatticeFermionD    tmp(&GRID);
+    LatticeFermionD    ref(&GRID);
+
+    RealD mass=0.01;
+    WilsonFermionD Dw(Umu,GRID,RBGRID,mass);
+
+    Dw.M(src,tmp);
+
+    std::cout << "Dw src = " <<norm2(src)<<std::endl;
+    std::cout << "Dw tmp = " <<norm2(tmp)<<std::endl;
+
+    Dw.FreePropagator(tmp,ref,mass);
+
+    std::cout << "Dw ref = " <<norm2(ref)<<std::endl;
+
+    ref = ref - src;
+    std::cout << "Dw ref-src = " <<norm2(ref)<<std::endl;
+  }
+
+  ////////////////////////////////////////////////////
+  // Dwf matrix — verify Fourier representation using PlannedFFT<LatticeFermionD>
+  ////////////////////////////////////////////////////
+  {
+    std::cout<<"****************************************"<<std::endl;
+    std::cout<<"Testing Fourier representation of Ddwf"<<std::endl;
+    std::cout<<"****************************************"<<std::endl;
+
+    const int Ls=16;
+    const int sdir=0;
+    RealD mass=0.01;
+    RealD M5  =1.0;
+    Gamma G5(Gamma::Algebra::Gamma5);
+
+    GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,&GRID);
+    GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,&GRID);
+
+    DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,GRID,RBGRID,mass,M5);
+
+    GridParallelRNG          RNG5(FGrid);      RNG5.SeedFixedIntegers(seeds);
+    LatticeFermionD    src5(FGrid); gaussian(RNG5,src5);
+    LatticeFermionD    src5_p(FGrid);
+    LatticeFermionD    result5(FGrid);
+    LatticeFermionD    ref5(FGrid);
+    LatticeFermionD    tmp5(FGrid);
+
+    Ddwf.M(src5,tmp5);
+    ref5 = tmp5;
+
+    PlannedFFT<LatticeFermionD::vector_object> theFFT5(FGrid);
+
+    theFFT5.FFT_dim(result5,tmp5,1,FFTbase::forward); tmp5 = result5;
+    std::cout<<"Fourier xformed Ddwf 1 "<<norm2(result5)<<std::endl;
+    theFFT5.FFT_dim(result5,tmp5,2,FFTbase::forward); tmp5 = result5;
+    std::cout<<"Fourier xformed Ddwf 2 "<<norm2(result5)<<std::endl;
+    theFFT5.FFT_dim(result5,tmp5,3,FFTbase::forward); tmp5 = result5;
+    std::cout<<"Fourier xformed Ddwf 3 "<<norm2(result5)<<std::endl;
+    theFFT5.FFT_dim(result5,tmp5,4,FFTbase::forward);
+    std::cout<<"Fourier xformed Ddwf 4 "<<norm2(result5)<<std::endl;
+    result5 = result5*ComplexD(::sqrt(1.0/vol),0.0);
+
+    std::cout<<"Fourier xformed Ddwf "<<norm2(result5)<<std::endl;
+
+    tmp5 = src5;
+    theFFT5.FFT_dim(src5_p,tmp5,1,FFTbase::forward); tmp5 = src5_p;
+    theFFT5.FFT_dim(src5_p,tmp5,2,FFTbase::forward); tmp5 = src5_p;
+    theFFT5.FFT_dim(src5_p,tmp5,3,FFTbase::forward); tmp5 = src5_p;
+    theFFT5.FFT_dim(src5_p,tmp5,4,FFTbase::forward); src5_p = src5_p*ComplexD(::sqrt(1.0/vol),0.0);
+
+    std::cout<<"Fourier xformed src5"<< norm2(src5)<<" -> "<<norm2(src5_p)<<std::endl;
+
+    Gamma::Algebra Gmu [] = {
+      Gamma::Algebra::GammaX,
+      Gamma::Algebra::GammaY,
+      Gamma::Algebra::GammaZ,
+      Gamma::Algebra::GammaT,
+      Gamma::Algebra::Gamma5
+    };
+    LatticeFermionD    Kinetic(FGrid); Kinetic = Zero();
+    LatticeComplexD    kmu(FGrid);
+    LatticeInteger     scoor(FGrid);
+    LatticeComplexD    sk (FGrid); sk = Zero();
+    LatticeComplexD    sk2(FGrid); sk2= Zero();
+    LatticeComplexD    W(FGrid); W= Zero();
+    LatticeComplexD    one5(FGrid); one5 =ComplexD(1.0,0.0);
+
+    for(int mu=0;mu<Nd;mu++) {
+      LatticeCoordinate(kmu,mu+1);
+      RealD TwoPiL =  M_PI * 2.0/ latt_size[mu];
+      kmu = TwoPiL * kmu;
+      sk2 = sk2 + 2.0*sin(kmu*0.5)*sin(kmu*0.5);
+      sk  = sk  +     sin(kmu)    *sin(kmu);
+      Kinetic = Kinetic + sin(kmu)*ci*(Gamma(Gmu[mu])*src5_p);
+    }
+    std::cout << " src5    "<<norm2(src5_p)<<std::endl;
+    std::cout << " Kinetic "<<norm2(Kinetic)<<std::endl;
+
+    W = one5 - M5 + sk2;
+    std::cout << " W "<<norm2(W)<<std::endl;
+    Kinetic = Kinetic + W * src5_p;
+    std::cout << " Kinetic "<<norm2(Kinetic)<<std::endl;
+
+    LatticeCoordinate(scoor,sdir);
+
+    tmp5 = Cshift(src5_p,sdir,+1);
+    tmp5 = (tmp5 - G5*tmp5)*0.5;
+    tmp5 = where(scoor==Integer(Ls-1),mass*tmp5,-tmp5);
+    Kinetic = Kinetic + tmp5;
+
+    tmp5 = Cshift(src5_p,sdir,-1);
+    tmp5 = (tmp5 + G5*tmp5)*0.5;
+    tmp5 = where(scoor==Integer(0),mass*tmp5,-tmp5);
+    Kinetic = Kinetic + tmp5;
+
+    std::cout<<"Momentum space Ddwf  "<< norm2(Kinetic)<<std::endl;
+    std::cout<<"Stencil Ddwf         "<< norm2(result5)<<std::endl;
+
+    result5 = result5 - Kinetic;
+    std::cout<<"diff "<< norm2(result5)<<std::endl;
+    GRID_ASSERT(norm2(result5)<1.0e-4);
+  }
+
+  ////////////////////////////////////////////////////
+  // Dwf prop
+  ////////////////////////////////////////////////////
+  {
+    std::cout<<"****************************************"<<std::endl;
+    std::cout << "Testing Ddwf Ht Mom space 4d propagator \n";
+    std::cout<<"****************************************"<<std::endl;
+
+    LatticeFermionD    src(&GRID); gaussian(pRNG,src);
+    LatticeFermionD    tmp(&GRID);
+    LatticeFermionD    ref(&GRID);
+    LatticeFermionD    diff(&GRID);
+
+    Coordinate point(4,0);
+    src=Zero();
+    SpinColourVectorD ferm; gaussian(sRNG,ferm);
+    pokeSite(ferm,src,point);
+
+    const int Ls=32;
+    GridCartesian         * FGrid   = SpaceTimeGrid::makeFiveDimGrid(Ls,&GRID);
+    GridRedBlackCartesian * FrbGrid = SpaceTimeGrid::makeFiveDimRedBlackGrid(Ls,&GRID);
+
+    RealD mass=0.01;
+    RealD M5  =0.8;
+    DomainWallFermionD Ddwf(Umu,*FGrid,*FrbGrid,GRID,RBGRID,mass,M5);
+
+    std::cout << " Solving by FFT and Feynman rules" <<std::endl;
+    bool fiveD = false;
+    Ddwf.FreePropagator(src,ref,mass,fiveD);
+
+    Gamma G5(Gamma::Algebra::Gamma5);
+
+    LatticeFermionD    src5(FGrid); src5=Zero();
+    LatticeFermionD    tmp5(FGrid);
+    LatticeFermionD    result5(FGrid); result5=Zero();
+    LatticeFermionD    result4(&GRID);
+    const int sdir=0;
+
+    tmp =   (src + G5*src)*0.5;      InsertSlice(tmp,src5,   0,sdir);
+    tmp =   (src - G5*src)*0.5;      InsertSlice(tmp,src5,Ls-1,sdir);
+
+    std::cout << " Solving by Conjugate Gradient (CGNE)" <<std::endl;
+    Ddwf.Mdag(src5,tmp5);
+    src5=tmp5;
+    MdagMLinearOperator<DomainWallFermionD,LatticeFermionD> HermOp(Ddwf);
+    ConjugateGradient<LatticeFermionD> CG(1.0e-8,10000);
+    CG(HermOp,src5,result5);
+
+    ExtractSlice(tmp,result5,0   ,sdir);   result4 =         (tmp-G5*tmp)*0.5;
+    ExtractSlice(tmp,result5,Ls-1,sdir);   result4 = result4+(tmp+G5*tmp)*0.5;
+
+    std::cout << " Taking difference" <<std::endl;
+    std::cout << "Ddwf result4 "<<norm2(result4)<<std::endl;
+    std::cout << "Ddwf ref     "<<norm2(ref)<<std::endl;
+
+    diff = ref - result4;
+    std::cout << "result - ref     "<<norm2(diff)<<std::endl;
+    GRID_ASSERT(norm2(diff)<1.0e-4);
+  }
+
+  Grid_finalize();
+}

tests/core/Test_plaquette_stats.cc

@@ -1,271 +0,0 @@
-/*************************************************************************************
-
-Grid physics library, www.github.com/paboyle/Grid
-
-Source file: tests/core/Test_plaquette_stats.cc
-
-Copyright (C) 2015
-
-Author: Chulwoo Jung <chulwoo@bnl.gov>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along
-with this program; if not, write to the Free Software Foundation, Inc.,
-51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-See the full license in the file "LICENSE" in the top level distribution directory
-*************************************************************************************/
-/*  END LEGAL */
-
-/**
- * Test_plaquette_stats
- *
- * Measure every plaquette Re Tr[U_mu(x) U_nu(x+mu) U_mu†(x+nu) U_nu†(x)] / Nc
- * across all sites and all (mu,nu) planes and report max, min, and average.
- *
- * Usage:
- *   ./Test_plaquette_stats [Grid options] [--file <nersc_config>] [--hot]
- *                          [--threshold <val>]
- *
- *   --file <path>       Read gauge field from NERSC-format file
- *   --hot               Use random (hot) SU(3) start (default: cold/unit start)
- *   --threshold <val>   Print coordinates of every plaquette with Re Tr/Nc < val
- *
- * Grid size defaults to 8^3 x 16; override with --grid (e.g. --grid 4.4.4.8).
- */
-
-#include <Grid/Grid.h>
-#include <limits>
-
-using namespace std;
-using namespace Grid;
-
-template<class T>
-static void writeFile(T& in, const std::string& fname)
-{
-#ifdef HAVE_LIME
-  std::cout << GridLogMessage << "Writing: " << fname << std::endl;
-  Grid::emptyUserRecord record;
-  Grid::ScidacWriter WR(in.Grid()->IsBoss());
-  WR.open(fname);
-  WR.writeScidacFieldRecord(in, record, 0);
-  WR.close();
-#endif
-}
-
-// Return the plane label string for output
-static std::string planeName(int mu, int nu)
-{
-  const char dirs[] = "xyzt";
-  std::string s;
-  s += dirs[mu];
-  s += dirs[nu];
-  return s;
-}
-
-int main(int argc, char** argv)
-{
-  Grid_init(&argc, &argv);
-
-  // ---- Lattice geometry ----
-  Coordinate latt_size   = GridDefaultLatt();
-  if (latt_size.size() == 0) {
-    latt_size = Coordinate(std::vector<int>{8, 8, 8, 16});
-  }
-  Coordinate simd_layout = GridDefaultSimd(Nd, vComplex::Nsimd());
-  Coordinate mpi_layout  = GridDefaultMpi();
-
-  GridCartesian grid(latt_size, simd_layout, mpi_layout);
-
-  std::cout << GridLogMessage << "Lattice: ";
-  for (int d = 0; d < Nd; d++)
-    std::cout << latt_size[d] << (d < Nd-1 ? "x" : "\n");
-
-  // ---- Gauge field ----
-  LatticeGaugeField Umu(&grid);
-
-  // Check for --file argument
-  std::string config_file = "";
-  for (int i = 1; i < argc - 1; i++) {
-    if (std::string(argv[i]) == "--file") {
-      config_file = argv[i+1];
-      break;
-    }
-  }
-
-  bool doHot = false;
-  for (int i = 1; i < argc; i++) {
-    if (std::string(argv[i]) == "--hot") { doHot = true; break; }
-  }
-
-  double threshold = std::numeric_limits<double>::quiet_NaN();
-  bool   doThresh  = false;
-  for (int i = 1; i < argc - 1; i++) {
-    if (std::string(argv[i]) == "--threshold") {
-      threshold = std::stod(argv[i+1]);
-      doThresh  = true;
-      break;
-    }
-  }
-
-  if (!config_file.empty()) {
-    std::cout << GridLogMessage << "Reading gauge field from " << config_file << std::endl;
-    FieldMetaData header;
-    NerscIO::readConfiguration(Umu, header, config_file);
-  } else {
-    std::vector<int> seeds({1, 2, 3, 4});
-    GridParallelRNG pRNG(&grid);
-    pRNG.SeedFixedIntegers(seeds);
-    if (doHot) {
-      std::cout << GridLogMessage << "Generating hot (random SU(3)) start" << std::endl;
-      SU<Nc>::HotConfiguration(pRNG, Umu);
-    } else {
-      std::cout << GridLogMessage << "Using cold (unit) gauge start" << std::endl;
-      SU<Nc>::ColdConfiguration(pRNG, Umu);
-    }
-  }
-
-  // ---- Extract link matrices ----
-  std::vector<LatticeColourMatrix> U(Nd, &grid);
-  for (int mu = 0; mu < Nd; mu++)
-    U[mu] = PeekIndex<LorentzIndex>(Umu, mu);
-
-  // ---- Per-plane plaquette statistics ----
-  //
-  // For each (mu, nu) plane (mu > nu) compute
-  //   P_munu(x) = Re Tr[plaquette] / Nc
-  // then report max, min, mean over all sites.
-  //
-  // WilsonLoops::traceDirPlaquette gives Tr[U_mu U_nu U_mu† U_nu†] (complex).
-
-  double vol = grid.gSites();
-
-  // Accumulate site-average plaquette (sum over planes / Nplanes / Nc)
-  LatticeComplex plaq_all(&grid);
-  plaq_all = Zero();
-  int Nplanes = 0;
-
-  // Per-site running sum over planes (Re Tr/Nc, normalised per plane)
-  LatticeRealD plaq_sum(&grid);
-  plaq_sum = Zero();
-
-  std::cout << GridLogMessage
-            << "======== Per-plane plaquette statistics ========" << std::endl;
-  std::cout << GridLogMessage
-            << std::setw(6)  << "plane"
-            << std::setw(20) << "max"
-            << std::setw(20) << "min"
-            << std::setw(20) << "average"
-            << std::endl;
-
-  for (int mu = 1; mu < Nd; mu++) {
-    for (int nu = 0; nu < mu; nu++) {
-
-      // Per-site trace of plaquette in (mu,nu) plane
-      LatticeComplex sitePlaq(&grid);
-      ColourWilsonLoops::traceDirPlaquette(sitePlaq, U, mu, nu);
-
-      plaq_all = plaq_all + sitePlaq;
-      Nplanes++;
-
-      // --- global average via sum() ---
-      TComplex Tsum  = sum(sitePlaq);
-      ComplexD csum  = TensorRemove(Tsum);
-      RealD    avg   = csum.real() / vol / Nc;
-
-      // --- global max and min via unvectorize + GlobalMax/GlobalMin ---
-      std::vector<TComplex> sv;
-      unvectorizeToLexOrdArray(sv, sitePlaq);
-
-      RealD local_max = -1e38, local_min = 1e38;
-      for (auto& tc : sv) {
-        RealD r = TensorRemove(tc).real() / Nc;
-        if (r > local_max) local_max = r;
-        if (r < local_min) local_min = r;
-      }
-
-      // Reduce across MPI ranks (no GlobalMin; negate to use GlobalMax)
-      grid.GlobalMax(local_max);
-      local_min = -local_min;
-      grid.GlobalMax(local_min);
-      local_min = -local_min;
-
-      std::cout << GridLogMessage
-                << std::setw(6)  << planeName(mu, nu)
-                << std::setw(20) << std::setprecision(10) << local_max
-                << std::setw(20) << std::setprecision(10) << local_min
-                << std::setw(20) << std::setprecision(10) << avg
-                << std::endl;
-
-      if (doThresh) {
-        std::vector<RealD> vr(sv.size(), 0.0);
-        for (int s = 0; s < (int)sv.size(); s++) {
-          RealD val = TensorRemove(sv[s]).real() / Nc;
-          if (val < threshold) {
-            Coordinate lc(Nd), gc(Nd);
-            Lexicographic::CoorFromIndex(lc, s, grid._ldimensions);
-            for (int d = 0; d < Nd; d++)
-              gc[d] = grid._processor_coor[d] * grid._ldimensions[d] + lc[d];
-            std::cerr << "BELOW_THRESHOLD  plane=" << planeName(mu, nu)
-//                      << "  global=(" << grid._processor_coor[0] << "," << grid._processor_coor[1] << "," << grid._processor_coor[2] << "," << grid._processor_coor[3] << ")"
-//                      << "  dim=(" << grid._ldimensions[0] << "," << grid._ldimensions[1] << "," << grid._ldimensions[2] << "," << grid._ldimensions[3] << ")"
-                      << "  site=(" << gc[0] << "," << gc[1] << "," << gc[2] << "," << gc[3] << ")"
-                      << "  P=" << std::setprecision(10) << val << std::endl;
-            vr[s] = val;
-          }
-        }
-        LatticeRealD plaqRe(&grid);
-        vectorizeFromLexOrdArray(vr, plaqRe);
-        plaq_sum = plaq_sum + plaqRe;
-      }
-    }
-  }
-
-  // Write per-site sum of below-threshold plaquette values over all planes
-  if (doThresh)
-    writeFile(plaq_sum, "./plaq_sum");
-
-  // ---- Overall (averaged over all planes) statistics ----
-  // plaq_all = sum of Tr[...] over all 6 (mu,nu) planes
-  // Normalise to Re Tr / Nc per plane
-  plaq_all = plaq_all * (1.0 / Nc / Nplanes);
-
-  TComplex Tsum_all = sum(plaq_all);
-  RealD    avg_all  = TensorRemove(Tsum_all).real() / vol;
-
-  std::vector<TComplex> sv_all;
-  unvectorizeToLexOrdArray(sv_all, plaq_all);
-
-  RealD max_all = -1e38, min_all = 1e38;
-  for (auto& tc : sv_all) {
-    RealD r = TensorRemove(tc).real();
-    if (r > max_all) max_all = r;
-    if (r < min_all) min_all = r;
-  }
-  grid.GlobalMax(max_all);
-  min_all = -min_all;
-  grid.GlobalMax(min_all);
-  min_all = -min_all;
-
-  // Cross-check with built-in avgPlaquette
-  RealD avg_builtin = ColourWilsonLoops::avgPlaquette(Umu);
-
-  std::cout << GridLogMessage
-            << "======== Overall plaquette statistics (all planes) ========" << std::endl;
-  std::cout << GridLogMessage << "  max     = " << max_all  << std::endl;
-  std::cout << GridLogMessage << "  min     = " << min_all  << std::endl;
-  std::cout << GridLogMessage << "  average = " << avg_all  << std::endl;
-  std::cout << GridLogMessage << "  avgPlaquette (builtin check) = " << avg_builtin << std::endl;
-
-  Grid_finalize();
-  return 0;
-}

tests/debug/Test_hipfft_bug_fail.cc

@@ -0,0 +1,76 @@
+/*
+ * Isolating the hipfft HIPFFT_PARSE_ERROR on ROCm 7 / hipFFT 1.0.20.
+ *
+ * Tests three orderings with an empty rocFFT cache to find which GPU
+ * operation before plan creation triggers the failure:
+ *   A) hipMalloc only            — hypothesis: passes (no async GPU work)
+ *   B) hipMalloc + hipMemset     — hypothesis: fails  (async GPU work in flight)
+ *   C) hipMalloc + hipMemset     — hypothesis: passes (work completed before plan)
+ *      + hipDeviceSynchronize
+ *
+ * Compile:
+ *   hipcc -o Test_hipfft_bug_fail Test_hipfft_bug_fail.cc -lhipfft
+ *
+ * Run with empty cache:
+ *   rm -rf ~/.cache/
+ *   ./Test_hipfft_bug_fail
+ */
+
+#include <cstdio>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+static const char *res(hipfftResult rv) {
+  return rv == HIPFFT_SUCCESS ? "SUCCESS" : "PARSE_ERROR";
+}
+
+static hipfftResult makePlan(int G, int howmany) {
+  int n[] = {G};
+  hipfftHandle p;
+  size_t workSize = 0;
+  hipfftCreate(&p);
+  hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                       nullptr, 1, G, nullptr, 1, G,
+                                       HIPFFT_Z2Z, howmany, &workSize);
+  hipfftDestroy(p);
+  return rv;
+}
+
+int main(void) {
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, 0);
+  printf("Device: %s\n", prop.name);
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  for (int G : {4, 8, 16, 32}) {
+    int howmany = 512;
+    long nelems = (long)G * howmany;
+    hipfftDoubleComplex *buf = nullptr;
+    hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+
+    // Tests ordered so each runs before a prior success can populate the cache.
+
+    // B first: hipMalloc + hipMemset (async GPU work in flight)
+    // If this fails, A (no hipMemset) will pass, confirming hipMemset is the trigger.
+    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+    hipfftResult rvB = makePlan(G, howmany);
+    printf("G=%-4d  B) hipMalloc + hipMemset       : %s\n", G, res(rvB));
+
+    // C: hipMalloc + hipMemset + sync — does syncing before plan creation fix it?
+    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+    hipDeviceSynchronize();
+    hipfftResult rvC = makePlan(G, howmany);
+    printf("G=%-4d  C) hipMalloc + hipMemset + sync: %s\n", G, res(rvC));
+
+    // A last: hipMalloc only, no async GPU work — should always pass
+    hipfftResult rvA = makePlan(G, howmany);
+    printf("G=%-4d  A) hipMalloc only             : %s\n\n", G, res(rvA));
+
+    hipFree(buf);
+  }
+
+  return 0;
+}

tests/debug/Test_hipfft_bug_pass.cc

@@ -0,0 +1,61 @@
+/*
+ * Minimal program demonstrating the workaround for the hipfft ROCm 7 bug.
+ *
+ * Workaround: create the hipfft plan BEFORE any hipMalloc.  Plan creation
+ * for G < 32 then succeeds even with an empty rocFFT cache.
+ *
+ * Compile:
+ *   hipcc -o Test_hipfft_bug_pass Test_hipfft_bug_pass.cc -lhipfft
+ *
+ * Run:
+ *   rm -rf ~/.cache/rocfft
+ *   ./Test_hipfft_bug_pass
+ *
+ * Expected: all G values succeed.
+ * Compare with Test_hipfft_bug_fail.cc which uses the opposite ordering.
+ */
+
+#include <cstdio>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+int main(void) {
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, 0);
+  printf("Device: %s\n", prop.name);
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  for (int G : {8, 16, 32}) {
+    int howmany = 512;
+    int n[] = {G};
+    long nelems = (long)G * howmany;
+
+    // Plan created BEFORE hipMalloc — succeeds for all G
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftCreate(&p);
+    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                         nullptr, 1, G, nullptr, 1, G,
+                                         HIPFFT_Z2Z, howmany, &workSize);
+    printf("G=%-4d  plan-then-hipMalloc: %d (%s)\n",
+           G, (int)rv, rv == HIPFFT_SUCCESS ? "HIPFFT_SUCCESS" : "HIPFFT_PARSE_ERROR");
+
+    if (rv == HIPFFT_SUCCESS) {
+      hipfftDoubleComplex *buf = nullptr;
+      hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+      hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("G=%-4d  execFwd:             %d (%s)\n",
+             G, (int)rv, rv == HIPFFT_SUCCESS ? "HIPFFT_SUCCESS" : "FAILED");
+      hipFree(buf);
+    }
+
+    hipfftDestroy(p);
+  }
+
+  return 0;
+}

tests/debug/Test_hipfft_minimal.cc

@@ -0,0 +1,142 @@
+/*
+ * Minimal reproducer for hipfftMakePlanMany / hipfftPlanMany failures.
+ *
+ * Compile on Frontier (no Grid headers needed):
+ *   hipcc -o Test_hipfft_minimal Test_hipfft_minimal.cc -lhipfft
+ *
+ * Run:
+ *   ./Test_hipfft_minimal
+ */
+
+#include <cstdio>
+#include <cstdlib>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+static const char *hipfftResultString(hipfftResult r) {
+  switch (r) {
+  case HIPFFT_SUCCESS:                  return "HIPFFT_SUCCESS";
+  case HIPFFT_INVALID_PLAN:             return "HIPFFT_INVALID_PLAN";
+  case HIPFFT_ALLOC_FAILED:             return "HIPFFT_ALLOC_FAILED";
+  case HIPFFT_INVALID_TYPE:             return "HIPFFT_INVALID_TYPE";
+  case HIPFFT_INVALID_VALUE:            return "HIPFFT_INVALID_VALUE";
+  case HIPFFT_INTERNAL_ERROR:           return "HIPFFT_INTERNAL_ERROR";
+  case HIPFFT_EXEC_FAILED:              return "HIPFFT_EXEC_FAILED";
+  case HIPFFT_SETUP_FAILED:             return "HIPFFT_SETUP_FAILED";
+  case HIPFFT_INVALID_SIZE:             return "HIPFFT_INVALID_SIZE";
+  case HIPFFT_UNALIGNED_DATA:           return "HIPFFT_UNALIGNED_DATA";
+  case HIPFFT_INCOMPLETE_PARAMETER_LIST:return "HIPFFT_INCOMPLETE_PARAMETER_LIST";
+  case HIPFFT_INVALID_DEVICE:           return "HIPFFT_INVALID_DEVICE";
+  case HIPFFT_PARSE_ERROR:              return "HIPFFT_PARSE_ERROR";
+  case HIPFFT_NO_WORKSPACE:             return "HIPFFT_NO_WORKSPACE";
+  case HIPFFT_NOT_IMPLEMENTED:          return "HIPFFT_NOT_IMPLEMENTED";
+  case HIPFFT_NOT_SUPPORTED:            return "HIPFFT_NOT_SUPPORTED";
+  default:                              return "UNKNOWN";
+  }
+}
+
+// Plan creation + execution for (G, howmany).
+// Tests two orderings to isolate whether a prior hipMalloc poisons hipfft
+// plan creation for small G on ROCm 7:
+//   A) plan BEFORE hipMalloc  — hypothesis: succeeds
+//   B) hipMalloc BEFORE plan  — hypothesis: fails for G < 32
+static void tryPlanAndExec(int G, long howmany) {
+  int n[] = {G};
+  long nelems = (long)G * howmany;
+
+  printf("--- G=%-4d  howmany=%-10ld  total_elems=%-12ld ---\n",
+         G, howmany, nelems);
+
+  // --- A: create plan first, allocate buffer afterwards ---
+  {
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftCreate(&p);
+    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                         nullptr, 1, G, nullptr, 1, G,
+                                         HIPFFT_Z2Z, (int)howmany, &workSize);
+    printf("  plan-first  create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      hipfftDoubleComplex *buf = nullptr;
+      hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+      hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  plan-first  execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipFree(buf);
+    }
+    hipfftDestroy(p);
+  }
+
+  // --- B: hipMalloc first, create plan afterwards ---
+  {
+    hipfftDoubleComplex *buf = nullptr;
+    hipMalloc(&buf, nelems * sizeof(hipfftDoubleComplex));
+    hipMemset(buf, 0, nelems * sizeof(hipfftDoubleComplex));
+
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftCreate(&p);
+    hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                         nullptr, 1, G, nullptr, 1, G,
+                                         HIPFFT_Z2Z, (int)howmany, &workSize);
+    printf("  malloc-first create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, buf, buf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  malloc-first execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+    }
+    hipfftDestroy(p);
+    hipFree(buf);
+  }
+
+  printf("\n");
+}
+
+int main(void) {
+  // Print HIP device info
+  int device = 0;
+  hipGetDevice(&device);
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, device);
+  printf("Device %d: %s  warpSize=%d\n\n", device, prop.name, prop.warpSize);
+
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  // Original sweep with small howmany (these passed first time)
+  printf("=== Small howmany (original sweep) ===\n\n");
+  for (int G : {4, 8, 12, 16, 24, 32, 48, 64})
+    tryPlanAndExec(G, 512);
+
+  // Grid-realistic howmany values derived from actual lattice geometries.
+  // howmany = Ncomp * product(ldimensions[d] for d != dim)
+  // For LatticeComplexD: Ncomp=1.
+  printf("=== Grid-realistic parameters ===\n\n");
+
+  // --grid 16.16.16.16  4D FFT  (KNOWN TO FAIL in Grid)
+  // Each dim: G=16, Nperp=16^3=4096
+  tryPlanAndExec(16, 4096);
+
+  // --grid 32.32.32.32  4D FFT  (KNOWN TO SUCCEED in Grid)
+  // Each dim: G=32, Nperp=32^3=32768
+  tryPlanAndExec(32, 32768);
+
+  // --grid 32.32.32.32 Ls=8  5D DWF FFT  (KNOWN TO FAIL on dim 0 in Grid)
+  // dim 0: G=8,  Nperp=32^4=1048576
+  tryPlanAndExec(8, 1048576);
+  // dim 1-4: G=32, Nperp=8*32^3=262144
+  tryPlanAndExec(32, 262144);
+
+  // Extra intermediate cases to bracket the failure
+  tryPlanAndExec(16, 1024);
+  tryPlanAndExec(16, 2048);
+  tryPlanAndExec(16, 8192);
+  tryPlanAndExec(8,  4096);
+  tryPlanAndExec(8,  65536);
+  tryPlanAndExec(8,  262144);
+
+  return 0;
+}

tests/debug/Test_hipfft_repro.cc

@@ -0,0 +1,168 @@
+/*
+ * Reproducer for HIPFFT_PARSE_ERROR (error 12) from hipfftMakePlanMany on
+ * ROCm 7 / hipFFT 1.0.20 (Frontier, MI210 login and MI250X compute nodes).
+ *
+ * Observed failure: G < 32 returns HIPFFT_PARSE_ERROR from all three plan
+ * creation APIs (hipfftPlanMany, hipfftMakePlanMany, hipfftPlan1d) when a
+ * device buffer is allocated and zeroed with hipMalloc+hipMemset before the
+ * plan creation call.  G >= 32 succeeds.
+ *
+ * Contrast with Test_hipfft_minimal.cc (plan-first ordering) which passes
+ * for all G even with an empty rocFFT cache.
+ *
+ * Compile on Frontier (no Grid headers needed):
+ *   hipcc -o Test_hipfft_repro Test_hipfft_repro.cc -lhipfft
+ *
+ * Run with empty cache to reproduce the failure:
+ *   rm -rf ~/.cache/rocfft
+ *   ./Test_hipfft_repro
+ */
+
+#include <cstdio>
+#include <cstdlib>
+#include <hipfft/hipfft.h>
+#include <hip/hip_runtime.h>
+
+static const char *hipfftResultString(hipfftResult r) {
+  switch (r) {
+  case HIPFFT_SUCCESS:                  return "HIPFFT_SUCCESS";
+  case HIPFFT_INVALID_PLAN:             return "HIPFFT_INVALID_PLAN";
+  case HIPFFT_ALLOC_FAILED:             return "HIPFFT_ALLOC_FAILED";
+  case HIPFFT_INVALID_TYPE:             return "HIPFFT_INVALID_TYPE";
+  case HIPFFT_INVALID_VALUE:            return "HIPFFT_INVALID_VALUE";
+  case HIPFFT_INTERNAL_ERROR:           return "HIPFFT_INTERNAL_ERROR";
+  case HIPFFT_EXEC_FAILED:              return "HIPFFT_EXEC_FAILED";
+  case HIPFFT_SETUP_FAILED:             return "HIPFFT_SETUP_FAILED";
+  case HIPFFT_INVALID_SIZE:             return "HIPFFT_INVALID_SIZE";
+  case HIPFFT_UNALIGNED_DATA:           return "HIPFFT_UNALIGNED_DATA";
+  case HIPFFT_INCOMPLETE_PARAMETER_LIST:return "HIPFFT_INCOMPLETE_PARAMETER_LIST";
+  case HIPFFT_INVALID_DEVICE:           return "HIPFFT_INVALID_DEVICE";
+  case HIPFFT_PARSE_ERROR:              return "HIPFFT_PARSE_ERROR";
+  case HIPFFT_NO_WORKSPACE:             return "HIPFFT_NO_WORKSPACE";
+  case HIPFFT_NOT_IMPLEMENTED:          return "HIPFFT_NOT_IMPLEMENTED";
+  case HIPFFT_NOT_SUPPORTED:            return "HIPFFT_NOT_SUPPORTED";
+  default:                              return "UNKNOWN";
+  }
+}
+
+// Plan creation + execution for (G, howmany) using hipfftCreate+hipfftMakePlanMany.
+// This is the path Grid's FFT.h now uses.
+static void tryPlanAndExec(int G, long howmany) {
+  int n[] = {G};
+  long nelems = (long)G * howmany;
+
+  printf("--- G=%-4d  howmany=%-10ld  total_elems=%-12ld ---\n",
+         G, howmany, nelems);
+
+  // Allocate device buffer (hipfftDoubleComplex = 16 bytes each)
+  hipfftDoubleComplex *dbuf = nullptr;
+  hipError_t herr = hipMalloc(&dbuf, nelems * sizeof(hipfftDoubleComplex));
+  if (herr != hipSuccess) {
+    printf("  hipMalloc failed (%d) for %ld elems — skipping\n\n", (int)herr, nelems);
+    return;
+  }
+  hipMemset(dbuf, 0, nelems * sizeof(hipfftDoubleComplex));
+
+  // 1. hipfftPlanMany (one-step, nullptr embed) — current Grid path
+  {
+    hipfftHandle p;
+    hipfftResult rv = hipfftPlanMany(&p, 1, n,
+                                     nullptr, 1, G,
+                                     nullptr, 1, G,
+                                     HIPFFT_Z2Z, (int)howmany);
+    printf("  hipfftPlanMany   create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  hipfftPlanMany   execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipfftDestroy(p);
+    }
+  }
+
+  // 2. hipfftCreate + hipfftMakePlanMany (two-step) — also current Grid path
+  {
+    hipfftHandle p;
+    size_t workSize = 0;
+    hipfftResult rc = hipfftCreate(&p);
+    if (rc == HIPFFT_SUCCESS) {
+      hipfftResult rv = hipfftMakePlanMany(p, 1, n,
+                                           nullptr, 1, G,
+                                           nullptr, 1, G,
+                                           HIPFFT_Z2Z, (int)howmany, &workSize);
+      printf("  hipfftMakePlanMany      : %d (%s)  workSize=%zu\n",
+             (int)rv, hipfftResultString(rv), workSize);
+      if (rv == HIPFFT_SUCCESS) {
+        rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+        hipDeviceSynchronize();
+        printf("  hipfftMakePlanMany exec : %d (%s)\n", (int)rv, hipfftResultString(rv));
+      }
+      hipfftDestroy(p);
+    } else {
+      printf("  hipfftCreate            : %d (%s)\n", (int)rc, hipfftResultString(rc));
+    }
+  }
+
+  // 3. hipfftPlan1d (simplest API, batch = howmany)
+  {
+    hipfftHandle p;
+    hipfftResult rv = hipfftPlan1d(&p, G, HIPFFT_Z2Z, (int)howmany);
+    printf("  hipfftPlan1d     create : %d (%s)\n", (int)rv, hipfftResultString(rv));
+    if (rv == HIPFFT_SUCCESS) {
+      rv = hipfftExecZ2Z(p, dbuf, dbuf, HIPFFT_FORWARD);
+      hipDeviceSynchronize();
+      printf("  hipfftPlan1d     execFwd: %d (%s)\n", (int)rv, hipfftResultString(rv));
+      hipfftDestroy(p);
+    }
+  }
+
+  hipFree(dbuf);
+  printf("\n");
+}
+
+int main(void) {
+  // Print HIP device info
+  int device = 0;
+  hipGetDevice(&device);
+  hipDeviceProp_t prop;
+  hipGetDeviceProperties(&prop, device);
+  printf("Device %d: %s  warpSize=%d\n\n", device, prop.name, prop.warpSize);
+
+#ifdef hipfftVersionMinor
+  printf("hipFFT version: %d.%d.%d\n\n",
+         hipfftVersionMajor, hipfftVersionMinor, hipfftVersionPatch);
+#endif
+
+  // Original sweep with small howmany (these passed first time)
+  printf("=== Small howmany (original sweep) ===\n\n");
+  for (int G : {4, 8, 12, 16, 24, 32, 48, 64})
+    tryPlanAndExec(G, 512);
+
+  // Grid-realistic howmany values derived from actual lattice geometries.
+  // howmany = Ncomp * product(ldimensions[d] for d != dim)
+  // For LatticeComplexD: Ncomp=1.
+  printf("=== Grid-realistic parameters ===\n\n");
+
+  // --grid 16.16.16.16  4D FFT  (KNOWN TO FAIL in Grid)
+  // Each dim: G=16, Nperp=16^3=4096
+  tryPlanAndExec(16, 4096);
+
+  // --grid 32.32.32.32  4D FFT  (KNOWN TO SUCCEED in Grid)
+  // Each dim: G=32, Nperp=32^3=32768
+  tryPlanAndExec(32, 32768);
+
+  // --grid 32.32.32.32 Ls=8  5D DWF FFT  (KNOWN TO FAIL on dim 0 in Grid)
+  // dim 0: G=8,  Nperp=32^4=1048576
+  tryPlanAndExec(8, 1048576);
+  // dim 1-4: G=32, Nperp=8*32^3=262144
+  tryPlanAndExec(32, 262144);
+
+  // Extra intermediate cases to bracket the failure
+  tryPlanAndExec(16, 1024);
+  tryPlanAndExec(16, 2048);
+  tryPlanAndExec(16, 8192);
+  tryPlanAndExec(8,  4096);
+  tryPlanAndExec(8,  65536);
+  tryPlanAndExec(8,  262144);
+
+  return 0;
+}

tests/debug/Test_reduction.cc

@@ -0,0 +1,167 @@
+/*************************************************************************************
+    Grid physics library, www.github.com/paboyle/Grid
+
+    Source file: ./tests/debug/Test_reduction.cc
+
+    Copyright (C) 2024
+
+Author: Peter Boyle <pboyle@bnl.gov>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+    See the full license in the file "LICENSE" in the top level distribution directory
+    *************************************************************************************/
+    /*  END LEGAL */
+#include <Grid/Grid.h>
+
+using namespace std;
+using namespace Grid;
+
+static int passed = 0;
+static int failed = 0;
+
+static void check(bool ok, const std::string &msg)
+{
+  if (ok) {
+    std::cout << GridLogMessage << "PASS  " << msg << std::endl;
+    passed++;
+  } else {
+    std::cout << GridLogMessage << "FAIL  " << msg << std::endl;
+    failed++;
+  }
+}
+
+// Squared magnitude of a Grid scalar tensor aggregate: innerProduct(a,a).
+// For iScalar:      real(conj(a)*a)
+// For iMatrix<T,N>: sum_{i,j} real(conj(a_ij)*a_ij)  (Frobenius)
+// Named squaredSum to make clear the squaring is applied to the aggregate
+// (the sum), not to individual site values before summing.
+template<class T>
+RealD squaredSum(const T &a)
+{
+  return (RealD)real(TensorRemove(innerProduct(a, a)));
+}
+
+template<class Field>
+void testReduction(GridCartesian *grid, GridParallelRNG &rng,
+                   const std::string &name, int Ncomp)
+{
+  typedef typename Field::vector_object vobj;
+  typedef typename vobj::scalar_object  sobj;
+  typedef typename vobj::scalar_type    scalar_type;
+
+  const Integer V      = grid->_gsites;
+  const Integer osites = grid->oSites();
+
+  // Detect single vs double precision by comparing fundamental scalar sizes.
+  const bool isFloat = (sizeof(scalar_type) < sizeof(ComplexD));
+
+  std::cout << GridLogMessage << "=== " << name << " ===" << std::endl;
+
+  Field field(grid);
+
+  //--------------------------------------------------------------------
+  // a) Timing: Niter timed calls reporting us/call and GB/s.
+  //--------------------------------------------------------------------
+#if defined(GRID_CUDA) || defined(GRID_HIP) || defined(GRID_SYCL)
+  {
+    const int Nwarm = 5;
+    const int Niter = 100;
+
+    gaussian(rng, field);
+
+    {
+      autoView(v, field, AcceleratorRead);
+      for (int i = 0; i < Nwarm; i++) sum_gpu(&v[0], osites);
+    }
+
+    RealD t_new;
+    {
+      autoView(v, field, AcceleratorRead);
+      t_new = -usecond();
+      for (int i = 0; i < Niter; i++) sum_gpu(&v[0], osites);
+      t_new += usecond();
+    }
+
+    RealD bytes = (RealD)osites * sizeof(vobj);
+    RealD GBs   = bytes / (t_new / Niter) * 1e-3;
+
+    std::cout << GridLogMessage << name << " timing (" << Niter << " calls):" << std::endl;
+    std::cout << GridLogMessage
+              << "  sum_gpu " << t_new/Niter << " us   " << GBs << " GB/s" << std::endl;
+  }
+#endif
+
+  //--------------------------------------------------------------------
+  // b) Constant field via field = 1.0.
+  //
+  //    Grid's iMatrix::operator=(scalar) sets only the diagonal, so:
+  //      LatticeComplex       -> scalar 1.0        (Ncomp = 1  nonzero per site)
+  //      LatticeColourMatrix  -> Nc x Nc identity  (Ncomp = Nc nonzero per site)
+  //      LatticePropagator    -> (Ns*Nc)^2 identity (Ncomp = Ns*Nc nonzero per site)
+  //
+  //    After GlobalSum: sum_result has Ncomp diagonal entries each equal to V,
+  //    all off-diagonal entries zero.  Grid's recursive innerProduct computes
+  //    the Frobenius inner product (sum of |element|^2 over all indices), giving
+  //
+  //        innerProduct(sum_result, sum_result) = Ncomp * V^2
+  //--------------------------------------------------------------------
+  {
+    field = 1.0;
+    sobj sum_result = sum(field);   // uses new GPU path + GlobalSum
+
+    RealD got      = squaredSum(sum_result);
+    RealD expected = (RealD)Ncomp * (RealD)V * (RealD)V;
+    RealD reldiff  = std::abs(got - expected) / expected;
+
+    std::cout << GridLogMessage
+              << name << " const: got " << got
+              << "  expected " << expected
+              << "  reldiff "  << reldiff << std::endl;
+    check(reldiff < 1e-8, name + " const: innerProduct(sum,sum) = Ncomp*V^2");
+  }
+}
+
+int main(int argc, char **argv)
+{
+  Grid_init(&argc, &argv);
+
+  Coordinate latt = GridDefaultLatt();
+  Coordinate mpi  = GridDefaultMpi();
+
+  GridCartesian *UGrid   = SpaceTimeGrid::makeFourDimGrid(latt, GridDefaultSimd(Nd, vComplexD::Nsimd()), mpi);
+  GridCartesian *UGrid_f = SpaceTimeGrid::makeFourDimGrid(latt, GridDefaultSimd(Nd, vComplexF::Nsimd()), mpi);
+
+  GridParallelRNG rng(UGrid);
+  rng.SeedFixedIntegers({1, 2, 3, 4});
+  GridParallelRNG rng_f(UGrid_f);
+  rng_f.SeedFixedIntegers({1, 2, 3, 4});
+
+  std::cout << GridLogMessage << "Lattice : " << latt << std::endl;
+  std::cout << GridLogMessage << "Volume  : " << UGrid->_gsites << std::endl;
+
+  testReduction<LatticeComplexF>      (UGrid_f, rng_f, "LatticeComplexF",      1      );
+  testReduction<LatticeComplexD>      (UGrid,   rng,   "LatticeComplexD",      1      );
+  testReduction<LatticeColourMatrixF> (UGrid_f, rng_f, "LatticeColourMatrixF", Nc     );
+  testReduction<LatticeColourMatrixD> (UGrid,   rng,   "LatticeColourMatrixD", Nc     );
+  testReduction<LatticePropagatorF>   (UGrid_f, rng_f, "LatticePropagatorF",   Ns*Nc  );
+  testReduction<LatticePropagatorD>   (UGrid,   rng,   "LatticePropagatorD",   Ns*Nc  );
+
+  std::cout << GridLogMessage << "==============================" << std::endl;
+  std::cout << GridLogMessage << passed << " PASSED   " << failed << " FAILED" << std::endl;
+
+  Grid_finalize();
+  return (failed > 0) ? EXIT_FAILURE : EXIT_SUCCESS;
+}

tests/lanczos/LanParams.xml

@@ -1,18 +1,14 @@
 <?xml version="1.0"?>
 <grid>
   <LanczosParameters>
-    <mass>0</mass>
-    <mstep>-0.025</mstep>
+    <mass>0.00107</mass>
     <M5>1.8</M5>
     <Ls>48</Ls>
-    <Nstop>5</Nstop>
-    <Nk>5</Nk>
-    <Np>5</Np>
-    <ReadEvec>0</ReadEvec>
-    <maxIter>10000</maxIter>
-    <resid>1e-10</resid>
-    <ChebyLow>1</ChebyLow>
-    <ChebyHigh>100</ChebyHigh>
-    <ChebyOrder>51</ChebyOrder>
+    <Nstop>10</Nstop>
+    <Nk>15</Nk>
+    <Np>85</Np>
+    <ChebyLow>0.003</ChebyLow>
+    <ChebyHigh>60</ChebyHigh>
+    <ChebyOrder>201</ChebyOrder>
   </LanczosParameters>
 </grid>

tests/lanczos/Test_blocked_krylov_schur.cc

@@ -1,99 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Test for BlockedKrylovSchur: verifies the KS decomposition A V = V H + F B^dag
-    by explicit operator applies, before and after each restart.
-
-    Uses DumbOperator (diagonal, real, Hermitian) from Test_synthetic_lanczos.
-
-    Tests Nblock=1 (scalar, regression) and Nblock=2,3 (exercises the B^H fix).
-
-*************************************************************************************/
-#include <Grid/Grid.h>
-
-using namespace std;
-using namespace Grid;
-
-// Diagonal real Hermitian operator (eigenvalues = scale lattice sites)
-template<class Field>
-class DumbOperator : public LinearOperatorBase<Field> {
-public:
-  LatticeComplex scale;
-
-  DumbOperator(GridBase* grid) : scale(grid) {
-    GridParallelRNG pRNG(grid);
-    std::vector<int> seeds({5,6,7,8});
-    pRNG.SeedFixedIntegers(seeds);
-    random(pRNG, scale);
-    scale = exp(-Grid::real(scale) * 3.0);
-  }
-
-  void OpDirAll(const Field& in, std::vector<Field>& out) {}
-  void OpDiag(const Field& in, Field& out) {}
-  void OpDir(const Field& in, Field& out, int dir, int disp) {}
-
-  void Op(const Field& in, Field& out) { out = scale * in; }
-  void AdjOp(const Field& in, Field& out) { out = scale * in; }
-  void HermOp(const Field& in, Field& out) { out = scale * in; }
-  void HermOpAndNorm(const Field& in, Field& out, double& n1, double& n2) {
-    out = scale * in;
-    ComplexD d = innerProduct(in, out);  n1 = real(d);
-    d = innerProduct(out, out);          n2 = real(d);
-  }
-};
-
-int main(int argc, char** argv)
-{
-  Grid_init(&argc, &argv);
-
-  GridCartesian* grid = SpaceTimeGrid::makeFourDimGrid(
-    GridDefaultLatt(),
-    GridDefaultSimd(Nd, vComplex::Nsimd()),
-    GridDefaultMpi());
-
-  GridParallelRNG RNG(grid);
-  RNG.SeedFixedIntegers({1,2,3,4});
-
-  typedef LatticeComplex Field;
-
-  DumbOperator<Field> op(grid);
-
-  int nFail = 0;
-
-  //--------------------------------------------------------------------
-  // Helper lambda: run BKS with doVerify and check it doesn't crash
-  //--------------------------------------------------------------------
-  auto runTest = [&](const std::string& label, int Nblock, int Nm, int Nk,
-                     int maxIter, int Nstop) {
-    std::cout << GridLogMessage << "===== " << label << " =====" << std::endl;
-
-    BlockKrylovSchur<Field> bks(op, grid, 1e-6, EvalReSmall);
-
-    std::vector<Field> v0(Nblock, Field(grid));
-    for (int t = 0; t < Nblock; t++) random(RNG, v0[t]);
-
-    bks(v0, maxIter, Nm, Nk, Nstop, Nblock,
-        /*doubleOrthog=*/true, /*doVerify=*/true);
-
-    std::cout << GridLogMessage << label << " done." << std::endl;
-  };
-
-  // Test 1: Nblock=1 — scalar case, regression (Nm,Nk now total vectors)
-  runTest("Nblock=1  Nm=10 Nk=5  maxIter=3", 1, 10, 5, 3, 5);
-
-  // Test 2: Nblock=2 — exercises the B^H fix for off-diagonal elements
-  runTest("Nblock=2  Nm=16 Nk=8  maxIter=3", 2, 16, 8, 3, 4);
-
-  // Test 3: Nblock=3 — further stress-test the B^H fix
-  runTest("Nblock=3  Nm=27 Nk=9  maxIter=3", 3, 27, 9, 3, 3);
-
-  // Test 4: Nblock=2, larger cycle — more restarts
-  runTest("Nblock=2  Nm=24 Nk=12  maxIter=5", 2, 24, 12, 5, 6);
-
-  if (nFail == 0)
-    std::cout << GridLogMessage << "All BlockedKrylovSchur tests completed." << std::endl;
-
-  Grid_finalize();
-  return nFail;
-}

tests/lanczos/Test_dwf_G5R5.cc

@@ -32,13 +32,9 @@ directory
 using namespace std;
 using namespace Grid;
 
-#if 0
+//typedef WilsonFermionD FermionOp;
 typedef DomainWallFermionD FermionOp;
 typedef typename DomainWallFermionD::FermionField FermionField;
-#else
-typedef MobiusFermionD FermionOp;
-typedef typename MobiusFermionD::FermionField FermionField;
-#endif
 
 template <class T> void writeFile(T& in, std::string const fname){  
 #ifdef HAVE_LIME
@@ -183,14 +179,12 @@ int main(int argc, char** argv) {
   Np=LanParams.Np;
 
   int Nm = Nk + Np;
+  int MaxIt = 100;
+  RealD resid = 1.0e-4;
 
-  int MaxIt = 10000;
-  RealD resid = 1.0e-5;
-  RealD mob_b=1.5;
 
 //while ( mass > - 5.0){
-//  FermionOp Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
-  FermionOp Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5,mob_b,mob_b-1.);
+  FermionOp Ddwf(Umu,*FGrid,*FrbGrid,*UGrid,*UrbGrid,mass,M5);
   MdagMLinearOperator<FermionOp,FermionField> HermOp(Ddwf); /// <-----
 //  Gamma5HermitianLinearOperator <FermionOp,LatticeFermion> HermOp2(WilsonOperator); /// <-----
   Gamma5R5HermitianLinearOperator<FermionOp, LatticeFermion> G5R5Herm(Ddwf);

tests/lanczos/Test_gamma5_block_lanczos.cc

@@ -1,89 +0,0 @@
-/*************************************************************************************
-
-    Grid physics library, www.github.com/paboyle/Grid
-
-    Test for Gamma5BlockLanczos on a simple diagonal operator.
-
-    The operator is D = diag(scale_i) where scale is complex random.
-    γ5 is taken to be the identity (scalar field has no spin structure),
-    so the γ5-inner product reduces to the standard Euclidean inner product
-    and the algorithm should find the eigenvalues of D directly.
-
-    For a genuine Wilson Dirac test, pass the actual γ5 functor.
-
-*************************************************************************************/
-#include <Grid/Grid.h>
-
-using namespace std;
-using namespace Grid;
-
-// Diagonal complex operator: out = scale * in
-template<class Field>
-class DumbOperator : public LinearOperatorBase<Field> {
-public:
-  LatticeComplex scale;
-  DumbOperator(GridBase* grid) : scale(grid) {
-    GridParallelRNG pRNG(grid);
-    pRNG.SeedFixedIntegers({5,6,7,8});
-    random(pRNG, scale);
-    scale = exp(-Grid::real(scale) * 3.0);
-  }
-  void OpDirAll(const Field& in, std::vector<Field>& out) {}
-  void OpDiag(const Field& in, Field& out) {}
-  void OpDir(const Field& in, Field& out, int dir, int disp) {}
-  void Op(const Field& in, Field& out) { out = scale * in; }
-  void AdjOp(const Field& in, Field& out) { out = scale * in; }
-  void HermOp(const Field& in, Field& out) { out = scale * in; }
-  void HermOpAndNorm(const Field& in, Field& out, double& n1, double& n2) {
-    out = scale * in;
-    ComplexD d = innerProduct(in, out); n1 = real(d);
-    d = innerProduct(out, out);         n2 = real(d);
-  }
-};
-
-int main(int argc, char** argv)
-{
-  Grid_init(&argc, &argv);
-
-  GridCartesian* grid = SpaceTimeGrid::makeFourDimGrid(
-    GridDefaultLatt(),
-    GridDefaultSimd(Nd, vComplex::Nsimd()),
-    GridDefaultMpi());
-
-  GridParallelRNG RNG(grid);
-  RNG.SeedFixedIntegers({1,2,3,4});
-
-  typedef LatticeComplex Field;
-  DumbOperator<Field> op(grid);
-
-  // For LatticeComplex (scalar field) γ5 = identity
-  auto gamma5 = [](const Field& in, Field& out){ out = in; };
-
-  Field v0(grid);
-  random(RNG, v0);
-
-  const int maxSteps = 20;
-  const int Nstop    = 4;
-  const RealD tol    = 1e-6;
-
-  std::cout << GridLogMessage
-            << "\n========================================" << std::endl;
-  std::cout << GridLogMessage
-            << " Gamma5BlockLanczos  (maxSteps=" << maxSteps
-            << " Nstop=" << Nstop << ")" << std::endl;
-  std::cout << GridLogMessage
-            << "========================================\n" << std::endl;
-
-  Gamma5BlockLanczos<Field> g5bl(op, grid, gamma5, tol);
-  g5bl.doEvalCheck = true;
-  g5bl(v0, maxSteps, Nstop, /*reorthog=*/true);
-
-  auto evals = g5bl.getEvals();
-  std::cout << GridLogMessage
-            << "Gamma5BlockLanczos eigenvalues (" << evals.size() << "):" << std::endl;
-  for (int k = 0; k < (int)evals.size(); k++)
-    std::cout << GridLogMessage << "  [" << k << "] " << evals(k) << std::endl;
-
-  Grid_finalize();
-  return 0;
-}