btrzx1 (emil, 2020)
General remarks
The cluster btrzx1 went into operation in August 2020. Besides two login nodes, it consists of 372 nodes which are connected by an InfiniBand network and a new Panasas file system. Unlike the previous clusters, btrzx1 uses Slurm as resource manager instead of PBS/Torque. Moreover, the ITS file server (e.g., the ITS home directory) is not mounted on the cluster for performance reasons but every users has a separate home directory which lies on the Panasas file system.
Acknowledging emil / Publications
As with other DFG-funded projects, results must be made available to the general public in an appropriate manner. The publications must contain a reference to the DFG funding (so-called “Funding Acknowledgement”) in the language of the publication, stating the project number.
Whenever the emil has been used to produce results used in a publication or posters, we kindly request citing the service in the acknowledgements:
Calculations were performed using the emil-cluster of the Bayreuth Centre for High Performance Computing (https://www.bzhpc.uni-bayreuth.de), funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 422127126.
Whereby the funding acknowledgement is mandatory.
ogin
The login nodes of btrzx1 (emil) are accessible with ssh via emil.rz.uni-bayreuth.de only from university networks. If you are outside the university, a VPN connection is required.
Compute nodes (state October 2023)
- 357 nodes (Type-A; normal and dev partition)
- 2x AMD Epyc 7302 16c CPU @ 3.00GHz, 32 cores total
- 2x 64GB RAM
- 500 GB HDD
- 12 nodes (Type-B; WAP partition)
- 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
- 16x 16GB RAM
- 256 GB nvme
- 2 nodes (Type-C; 1 WAP , 1 GPU Partition )
- 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
- 16x 32GB RAM
- 500 GB nvme
- 1x NVIDIA Tesla V100 16GB
- 1 node (Type-D; GPU partition)
- 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
- 16x 32GB RAM
- 500 GB nvme
- 2x NVIDIA Tesla V100 16GB
Queues / Partitions
- normal
Priority: multifiactor, most weight on the group's financial share in the cluster and consumed ressources
Wall time limit: 24 hours
Restrictions: cpu nodes only
- WAP
Priority: multifiactor, most weight on the group's financial share in the partition (WAP Mathe) and consumed ressources
Wall time limit: 24 hours
Restrictions: no
- GPU
Priority: multifiactor, most weight on the group's financial share in the cluster and consumed ressources
Wall time limit: 24 hours
Restrictions: single gpu nodes only
- dev
Priority: multifactor, most weight on consumed ressources
Wall time limit: 30 Minutes (default), 1 hour (max)
Restrictions: max 4 nodes per job
Network
- Infiniband (56 Gbit/s)
- 2-level Fat Tree (Blocking factor 2)
User file space (network and local)
- Panasas network file system
- /home: 100GB per User
- /scratch: no soft-quota
- no snapshots, no backup
- wiped down to 40% usage on 60% usage by earliest mtime
- Commissioning & Extension
- August 2020
Resource Manager & Scheduler
- Slurm
Operating system
- CentOS 7.9
Node topology (likwid-topology -g)
- 2x AMD Epyc 7302Einklappen
-
-------------------------------------------------------------------------------- CPU name: AMD EPYC 7302 16-Core Processor CPU type: AMD K17 (Zen2) architecture CPU stepping: 0 ******************************************************************************** Hardware Thread Topology ******************************************************************************** Sockets: 2 Cores per socket: 16 Threads per core: 1 -------------------------------------------------------------------------------- HWThread Thread Core Socket Available 0 0 0 0 * 1 0 1 0 * 2 0 2 0 * 3 0 3 0 * 4 0 4 0 * 5 0 5 0 * 6 0 6 0 * 7 0 7 0 * 8 0 8 0 * 9 0 9 0 * 10 0 10 0 * 11 0 11 0 * 12 0 12 0 * 13 0 13 0 * 14 0 14 0 * 15 0 15 0 * 16 0 16 1 * 17 0 17 1 * 18 0 18 1 * 19 0 19 1 * 20 0 20 1 * 21 0 21 1 * 22 0 22 1 * 23 0 23 1 * 24 0 24 1 * 25 0 25 1 * 26 0 26 1 * 27 0 27 1 * 28 0 28 1 * 29 0 29 1 * 30 0 30 1 * 31 0 31 1 * -------------------------------------------------------------------------------- Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ) Socket 1: ( 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ) -------------------------------------------------------------------------------- ******************************************************************************** Cache Topology ******************************************************************************** Level: 1 Size: 32 kB Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) -------------------------------------------------------------------------------- Level: 2 Size: 512 kB Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) -------------------------------------------------------------------------------- Level: 3 Size: 16 MB Cache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) ( 24 25 ) ( 26 27 ) ( 28 29 ) ( 30 31 ) -------------------------------------------------------------------------------- ******************************************************************************** NUMA Topology ******************************************************************************** NUMA domains: 2 -------------------------------------------------------------------------------- Domain: 0 Processors: ( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ) Distances: 10 32 Free memory: 61130.3 MB Total memory: 65403.8 MB -------------------------------------------------------------------------------- Domain: 1 Processors: ( 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ) Distances: 32 10 Free memory: 63574.7 MB Total memory: 65534 MB -------------------------------------------------------------------------------- ******************************************************************************** Graphical Topology ******************************************************************************** Socket 0: +-----------------------------------------------------------------------------------------------------------------+ | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | | 0 | | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | | 10 | | 11 | | 12 | | 13 | | 14 | | 15 | | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ | | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | | +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ | +-----------------------------------------------------------------------------------------------------------------+ Socket 1: +-----------------------------------------------------------------------------------------------------------------+ | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | | 16 | | 17 | | 18 | | 19 | | 20 | | 21 | | 22 | | 23 | | 24 | | 25 | | 26 | | 27 | | 28 | | 29 | | 30 | | 31 | | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB | | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | | +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ | | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | 16 MB | | | +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ | +-----------------------------------------------------------------------------------------------------------------+
MPI benchmarks (OSU)
- MPI BandwidthEinklappen
-
P2P MPI Bandwidth between processes on the same socket (by core), the same node but different sockets (by socket), and different nodes (by node).
- MPI LatencyEinklappen
-
Single-point measurements of the point-2-point MPI latency (limit: size 0) between processes on the same socket (by core), the same node but different sockets (by socket), and different nodes (by node).
Node-level performance (likwid-bench)
- Parallel data transfer rate (close)Einklappen
-
Parallel data transfer rate (Processes are close on a single socket).
- Parallel data transfer rate (scattered)Einklappen
-
Parallel data transfer rate (Processes are scattered accross the node).
- Vector triadEinklappen
-
Performance of the vector triad on a single core, a single socket (16 processes), and a node (32 processes).