Commit 4979cdd3 authored by GIORGIO AMATI's avatar GIORGIO AMATI

fix

parent 4eb929f0
......@@ -3,30 +3,30 @@ SPECFEM3D_GLOBE benchmark:
DIRECTORY STRUCTURE:
----------
---------------------------------------
.
├── benchmark: directory to execute SPECFEM3D_GLOBE benchmark.
| ├── bin
| ├── CPU: here you'll find all config/build file for CPU
├── benchmark: directory to execute SPECFEM3D_GLOBE benchmark.
| ├── bi here you'll find all exe files
| ├── CPU here you'll find all config/build file for CPU
| ├── CPU_environment.sh (file)
| ├── DATA: here you'll find data file for the test
| ├── GPU: here you'll find all config/build file for GPU
| ├── DATA here you'll find data file for the test
| ├── GPU here you'll find all config/build file for GPU
| ├── GPU_environment.sh (file)
| ├── logs: here you'll find log file (configure, compilation, ..)\
| └── OUTPUT_FILES: here you'll find output files
| ├── logs here you'll find log file (configure, compilation, ..)\
| └── OUTPUT_FILES here you'll find output files
|
├── README (this file)
├── reference: directory results obtained using 64 Marconi A2 KNL nodes (CPU)
| ├── big : here you'll find output files for the big test case
| └── small : here you'll find output files for the small test case
├── reference_Davide: directory results obtained using (old) Davide system (Pwr8+P100 GPU)
└── reference_Marconi: directory results obtained using Marconi100 system (Pwr9+V100 GPU)
├── big : here you'll find output files for the big test case
└── small : here you'll find output files for the small test case
├── README this file
├── reference directory results obtained using 64 Marconi A2 KNL nodes (CPU)
| ├── big here you'll find output files for the big test case
| └── small here you'll find output files for the small test case
├── reference_Davide directory results obtained using (old) Davide system (Pwr8+P100 GPU)
└── reference_Marconi directory results obtained using Marconi100 system (Pwr9+V100 GPU)
├── big here you'll find output files for the big test case
└── small here you'll find output files for the small test case
Note: For GPU reference see reference_Marconi directory
----------
---------------------------------------
TO RUN THE BENCHMARK:
......@@ -37,7 +37,8 @@ git clone https://github.com/geodynamics/specfem3d_globe.git git
1) Go in the "benchmark" directory
2) Configure & Select the environment: two different environment are available
2) Select & Configure the environment: two different environment are available
Compiler and libraries defined works fine in CINECA environment. To modify to your needs
* ./CPU_environment.sh to setup CPU environment
- using intel/intelmpi rel. 2018
......@@ -52,68 +53,68 @@ git clone https://github.com/geodynamics/specfem3d_globe.git git
CPU/00-setting.big (16 node, 64 GPU)
CPU/00-setting.small (1 node, 1 GPU)
3) Configure and adapt your environment (compilers et al.) to your needs the file
- 00-settings,
(choose between ".big" or ".small" version in CPU/GPU directory)
3) Choose your setting
./00-settings (choose between ".big" or ".small" version in CPU/GPU directory)
4) Configure the run (according to the size and the decomposition) defined in 00-setting file
./1-configure,
./01-configure,
5) Build the exe files.
./02-build,
6) Submit the job to the right queueing system. The script submit "04-submit" file.
./03-run
in 04-submit file, using SLURM queing system (customized for Marconi100 environment), a two step procedure is performed
in 04-submit file, using SLURM queing system (customized for Marconi100 environment), a two step procedure is performed
1) Building the data (xmeshfem3D.x, only CPU). It populates MPI_DATABASE directory. For the big test case ask for about 1 TB.
2) Start the efective run (xspecfem3D, GPU or CPU based)
----------
---------------------------------------
TEST CASE
For CPU small dataset choose CPU/00-settings.small: it is defined as
* nex_xi=nex_eta=128,
* nproc_xi=nproc_eta=4, (i.e. mpi decomposion along the two dimensions)
* nodes=1, (i.e. number of nodes involved)
* nex_xi=nex_eta=128
* nproc_xi=nproc_eta=4 (i.e. mpi decomposion along the two dimensions)
* nodes=1 (i.e. number of nodes involved)
* ranks=16 (i.e. total number of mpi tasks)
Reference values are obtained using Marconi SKL
For GPU small dataset choose GPU/00-settings.small: it is defined as
(one task for eacu GPU)
* nex_xi=nex_eta=128,
* nproc_xi=nproc_eta=1, (i.e. mpi decomposion along the two dimensions)
* nodes=1, (i.e. number of nodes involved)
(one task for each GPU)
* nex_xi=nex_eta=128
* nproc_xi=nproc_eta=1 (i.e. mpi decomposion along the two dimensions)
* nodes=1 (i.e. number of nodes involved)
* ranks=1 (i.e. total number of mpi tasks)
Reference values are obtained using Marconi KNL (for 2minutes simulation instead of 10)
Reference values are obtained using Marconi100
For CPU big dataset choose in CPU/00-settings.large: it is defined as
* nex_xi=nex_eta=512,
* nproc_xi=nproc_eta=64, (i.e. mpi decomposion along the two dimensions)
* nodes=64, (i.e. number of nodes involved)
* nex_xi=nex_eta=512
* nproc_xi=nproc_eta=64 (i.e. mpi decomposion along the two dimensions)
* nodes=64 (i.e. number of nodes involved)
* ranks=4096 (i.e. total number of mpi tasks)
Reference values are obtained using Marconi100
Reference values are obtained using Marconi KNL (for 2minutes simulation instead of 10)
For CPU big dataset choose in CPU/00-settings.large: it is defined as
* nex_xi=nex_eta=512,
* nproc_xi=nproc_eta=8, (i.e. mpi decomposion along the two dimensions)
* nodes=16, (i.e. number of nodes involved)
* nex_xi=nex_eta=512
* nproc_xi=nproc_eta=8 (i.e. mpi decomposion along the two dimensions)
* nodes=16 (i.e. number of nodes involved)
* ranks=64 (i.e. total number of mpi tasks)
(Marconi100 has 4 GPU for each node)
Reference values are obtained using Marconi100
(Marconi100 has 4 GPU for each node)
---------------------------------------
COMMENTS
1) If you modify the decomposition (e.g. the number of tasks) you have
* to reconfigure (01-configure)
* to recompile the code (02-build)
1) Every time you modify the configuration (e.g. the number of tasks) you have
* to re-configure (01-configure)
* to re-compile the code (02-build)
2) If you want to increase the duration of the simulation you have change the in DATA/Par_file the
2) If you want to increase/decrease the duration of the simulation you have change the in DATA/Par_file the
"RECORD_LENGTH_IN_MINUTES", set by default to 10 minutes
3) Data building time could be huge compared with solver time, the Solver time is the only important figure. The data build time is an interesting figure for understanding the workflow impact
3) Data building time could be huge compared with solver time
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment