portelli/Grid
1
0
Fork 0
mirror of https://github.com/paboyle/Grid.git synced 2026-09-04 16:59:36 +01:00
Code Releases Activity

Improved FFT -- 1.5-2x when there are 2-6 ranks in a given axis of the cartesian communicator.

Browse Source 
Barrel shift -> all to all (x2) and distributed FFT work fully load balanced without redundant work.
There is little more I can do now on FFT. Comms dominated and running distributed work dividing bandwidth optimal RingAllToAll

/ccs/home/paboyle/ParallelIO/systems/Frontier/tests/core/Test_fft_prop --mpi 3.6.4.4 --grid 48.48.48.96 --accelerator-threads 8 --shm 4096 --shm-mpi 1 --device-mem
32000 --log Error,Warning,Message,Performance

*************************************************
 Benchmarking FFT of LatticeFermionD on plane wave
*************************************************
Grid : Performance : 0.501524 s :  FFT took     0.001311 s (transpose P=3)
Grid : Performance : 0.501531 s :  FFT pack     5.9e-05 s
Grid : Performance : 0.501533 s :  FFT alltoall 0.000828 s
Grid : Performance : 0.501534 s :  FFT reorder  0.000204 s
Grid : Performance : 0.501535 s :  FFT kernels  1e-05 s
Grid : Performance : 0.501536 s :  FFT unpack   5e-05 s
Grid : Performance : 0.509992 s :  FFT took     0.001829 s (transpose P=6)
Grid : Performance : 0.510000 s :  FFT pack     6e-05 s
Grid : Performance : 0.510002 s :  FFT alltoall 0.001436 s
Grid : Performance : 0.510003 s :  FFT reorder  0.000202 s
Grid : Performance : 0.510005 s :  FFT kernels  9e-06 s
Grid : Performance : 0.510006 s :  FFT unpack   5.3e-05 s
Grid : Performance : 0.517690 s :  FFT took     0.001599 s (transpose P=4)
Grid : Performance : 0.517698 s :  FFT pack     6e-05 s
Grid : Performance : 0.517700 s :  FFT alltoall 0.001258 s
Grid : Performance : 0.517701 s :  FFT reorder  0.0002 s
Grid : Performance : 0.517702 s :  FFT kernels  9e-06 s
Grid : Performance : 0.517703 s :  FFT unpack   4.9e-05 s
Grid : Performance : 0.524858 s :  FFT took     0.001561 s (transpose P=4)
Grid : Performance : 0.524865 s :  FFT pack     5.8e-05 s
Grid : Performance : 0.524867 s :  FFT alltoall 0.001213 s
Grid : Performance : 0.524868 s :  FFT reorder  0.000209 s
Grid : Performance : 0.524869 s :  FFT kernels  8e-06 s
Grid : Performance : 0.524870 s :  FFT unpack   4.9e-05 s
*************************************************
 FFT of [48 48 48 96] LatticeFermionD took 0.030916 s
*************************************************
This commit is contained in:
Peter Boyle
2026-09-03 17:33:32 -04:00
parent e767694b82
commit 357ede3664
5 changed files with 295 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
systems/Frontier/nrhs_fault_36.job
+8 -3
View File
@@ -69,6 +69,7 @@ module load libfabric
BIN=$root/examples/Example_pvdagm_v2_3level_DenseCoarseMatrix
OPTS1="--accelerator-threads 8 --shm 4096 --shm-mpi 1 --device-mem 32000"
OVERLAP="--comms-overlap" # set empty for a no-overlap job
vol=48.48.48.96
MPI_GEOM=3.6.4.4
@@ -99,7 +100,7 @@ run_cell () {
name=$1; shift
echo "----- $name : $* -----"
export GRID_STDOUT_ROOT=$RUNDIR/fault36_$name
env "$@" srun -N36 -n288 --kill-on-bad-exit=1 ./select_gpu $BIN --mpi ${MPI_GEOM} --grid $vol $OPTS1 --comms-overlap \
env "$@" srun -N36 -n288 --kill-on-bad-exit=1 ./select_gpu $BIN --mpi ${MPI_GEOM} --grid $vol $OPTS1 $OVERLAP \
--debug-stdout --log Error,Warning,Message,Memory > log.fault36.$name 2>&1
echo " exit $?"; sleep 30
f=$(grep -l "Memory access fault" $GRID_STDOUT_ROOT/*/Grid.stderr.* 2>/dev/null | head -1)
@@ -117,6 +118,8 @@ run_cell () {
#run_cell A_serialised AMD_SERIALIZE_KERNEL=3 AMD_SERIALIZE_COPY=3 AMD_LOG_LEVEL=3 # 5371703-era: OOM found; done
#run_cell B_plain AMD_LOG_LEVEL=1
# NOTE (2026-08-29): FineSloppyComms is NOT set in this job; the example defaults it to 1, so every
# cell below ran with SLOPPY comms ON (confirmed by the PARAM line). Only 5371826/5372414 set it to 0.
# Ladder for the NO_TRANSLATION / hang fault (both at NRHS=12):
# 5371703 sloppy ON, no kdreg2 -> NO_TRANSLATION (Dhop halo) at outer 36
# 5371826 sloppy OFF, no kdreg2 -> 12-RHS converged, HANG in single-RHS solve
@@ -146,5 +149,7 @@ run_cell L_cache4 FI_MR_CACHE_MAX_COUNT=4 # PB: room for Pa
# at the first exchange (as the reproducer predicts for the cache-off path); G-M all solved
# Nrhs 6 + Nrhs 1 (NOTE: NRHS=6 above, the three failures in the ledger were at NRHS=12).
# Missing control: the new PaddedCell, NO env knobs, at NRHS=12, twice (intermittent failure).
run_cell N_control NRHS=12
run_cell N_control2 NRHS=12
run_cell N_control NRHS=12 # 5374670: PASSED
run_cell N_control2 NRHS=12 # 5374670: NO_TRANSLATION, Dhop Waitall(16), 3 ranks; last logged event on the faulting rank = eviction 49 of the 62-eviction restart burst after outer step 48 (170 MB hipFree via the ring cache), the failing exchange followed it
# Overlap on/off is a WHOLE-JOB A/B: set OVERLAP empty above and submit again. Measured
# 2026-08-29/30: it does not change the failure rate.