btrzx1 (emil, 2020)

General remarks

The cluster btrzx1 went into operation in August 2020. Besides two login nodes, it consists of 372 nodes which are connected by an InfiniBand network and a new Panasas file system. Unlike the previous clusters, btrzx1 uses Slurm as resource manager instead of PBS/Torque. Moreover, the ITS file server (e.g., the ITS home directory) is not mounted on the cluster for performance reasons but every users has a separate home directory which lies on the Panasas file system.

Login

The login nodes of btrzx1 (emil) are accessible with ssh via emil.rz.uni-bayreuth.de only from university networks. If you are outside the university, a VPN connection is required.

Compute nodes (state October 2023)

357 nodes (Type-A; normal and dev partition)
- 2x AMD Epyc 7302 16c CPU @ 3.00GHz, 32 cores total
- 2x 64GB RAM
- 500 GB HDD
12 nodes (Type-B; WAP partition)
- 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
- 16x 16GB RAM
- 256 GB nvme

2 nodes (Type-C; 1 WAP , 1 GPU Partition )
- 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
- 16x 32GB RAM
- 500 GB nvme
- 1x NVIDIA Tesla V100 16GB
1 node (Type-D; GPU partition)
- 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
- 16x 32GB RAM
- 500 GB nvme
- 2x NVIDIA Tesla V100 16GB

Queues / Partitions

default
Priority: multifiactor, most weight on the group's financial share in the cluster and consumed ressources
Wall time limit: 24 hours
Restrictions: cpu nodes only

WAP
Priority: multifiactor, most weight on the group's financial share in the partition (WAP Mathe) and consumed ressources
Wall time limit: 24 hours
Restrictions: no

gpu
Priority: multifiactor, most weight on the group's financial share in the cluster and consumed ressources
Wall time limit: 24 hours
Restrictions: single gpu nodes only

dev
Priority: multifactor, most weight on consumed ressources
Wall time limit: 30 Minutes (default), 1 hour (max)
Restrictions: max 4 nodes per job

Network

Infiniband (56 Gbit/s)
2-level Fat Tree (Blocking factor 2)

User file space (network and local)

Panasas network file system
- /home: 100GB per User
- /scratch: no soft-quota
  - no snapshots, no backup
  - wiped down to 40% usage on 60% usage by earliest mtime
Local disk (/tmp):
- Type-A: ~412GB
- Type-B: ~70GB
- Type-C/D: ~182GB

Commissioning & Extension

August 2020

Resource Manager & Scheduler

Slurm

Operating system

CentOS 7.9

Node topology (likwid-topology -g)

2x AMD Epyc 7302 Einklappen

--------------------------------------------------------------------------------
CPU name: AMD EPYC 7302 16-Core Processor 
CPU type: AMD K17 (Zen2) architecture
CPU stepping: 0
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets: 2
Cores per socket: 16
Threads per core: 1
--------------------------------------------------------------------------------
HWThread Thread Core Socket Available
0 0 0 0 *
1 0 1 0 *
2 0 2 0 *
3 0 3 0 *
4 0 4 0 *
5 0 5 0 *
6 0 6 0 *
7 0 7 0 *
8 0 8 0 *
9 0 9 0 *
10 0 10 0 *
11 0 11 0 *
12 0 12 0 *
13 0 13 0 *
14 0 14 0 *
15 0 15 0 *
16 0 16 1 *
17 0 17 1 *
18 0 18 1 *
19 0 19 1 *
20 0 20 1 *
21 0 21 1 *
22 0 22 1 *
23 0 23 1 *
24 0 24 1 *
25 0 25 1 *
26 0 26 1 *
27 0 27 1 *
28 0 28 1 *
29 0 29 1 *
30 0 30 1 *
31 0 31 1 *
--------------------------------------------------------------------------------
Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
Socket 1: ( 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 )
--------------------------------------------------------------------------------
********************************************************************************
Cache Topology
********************************************************************************
Level: 1
Size: 32 kB
Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 )
--------------------------------------------------------------------------------
Level: 2
Size: 512 kB
Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 )
--------------------------------------------------------------------------------
Level: 3
Size: 16 MB
Cache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) ( 24 25 ) ( 26 27 ) ( 28 29 ) ( 30 31 )
--------------------------------------------------------------------------------
********************************************************************************
NUMA Topology
********************************************************************************
NUMA domains: 2
--------------------------------------------------------------------------------
Domain: 0
Processors: ( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
Distances: 10 32
Free memory: 61130.3 MB
Total memory: 65403.8 MB
--------------------------------------------------------------------------------
Domain: 1
Processors: ( 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 )
Distances: 32 10
Free memory: 63574.7 MB
Total memory: 65534 MB
--------------------------------------------------------------------------------


********************************************************************************
Graphical Topology
********************************************************************************
Socket 0:

+-----------------------------------------------------------------------------------------------------------------+
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |  0 | |  1 | |  2 | |  3 | |  4 | |  5 | |  6 | |  7 | |  8 | |  9 | | 10 | | 11 | | 12 | | 13 | | 14 | | 15 | |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
| |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
+-----------------------------------------------------------------------------------------------------------------+
Socket 1:

+-----------------------------------------------------------------------------------------------------------------+
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| | 16 | | 17 | | 18 | | 19 | | 20 | | 21 | | 22 | | 23 | | 24 | | 25 | | 26 | | 27 | | 28 | | 29 | | 30 | | 31 | |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
| |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
+-----------------------------------------------------------------------------------------------------------------+

MPI benchmarks (OSU)

MPI Bandwidth Einklappen: P2P MPI Bandwidth between processes on the same socket (by core), the same node but different sockets (by socket), and different nodes (by node).
MPI Latency Einklappen: Single-point measurements of the point-2-point MPI latency (limit: size 0) between processes on the same socket (by core), the same node but different sockets (by socket), and different nodes (by node).

Node-level performance (likwid-bench)

Parallel data transfer rate (close)Einklappen: Parallel data transfer rate (Processes are close on a single socket).
Parallel data transfer rate (scattered)Einklappen: Parallel data transfer rate (Processes are scattered accross the node).
Vector triad Einklappen: Performance of the vector triad on a single core, a single socket (16 processes), and a node (32 processes).