btrzx3 (2013,deprecated)
The cluster btrzx3 has reached end of life, please use one of btrzx[1,2,4].
General remarks
The cluster btrzx3 went into operation in March 2013. Besides two login nodes, it features two different node types with the properties p2 and p4 which are connected by a QDR InfiniBand network. p2 and p4 nodes are equipped with two or four 12-core AMD Opteron CPUs, respectively. A special particularity of the nodes in btrzx3, which can be relevant considering performance, is the number of memory interfaces per CPU. In contrast to the other clusters in which each CPU has a single memory interface, a single 12-core AMD CPU divides its cores into two sets of 6 cores with a separate memory interface each. The latter can become especially relevant for running OpenMP programs (see NUMA). Moreover, two adjacent physical cores share all of their caches as well as their FPU. Therefore, it can make sense to use every second core only when running parallel jobs, e.g., by dropping every second entry in the host file. (Note, however, that all cores must still be requested through ppn=24 or ppn=48, respectively, to get complete nodes.)
Login nodes
- btrzx3-1.rz.uni-bayreuth.de
- btrzx3-2.rz.uni-bayreuth.de
Compute nodes (state June 2018)
- 417 nodes (p2)
2x AMD Opteron 6348 @ 2.80GHz (Abu Dhabi)
(in total 24 physical cores, each 2 share a FPU)
of those: 278 with 4x 16GB RAM (p2r64), 135 with 4x 32GB RAM (p2r128), and 4 with 4x 64GB RAM (p2r256) - 9 nodes (p4)
4x AMD Opteron 6348 @ 2.80GHz (Abu Dhabi)
(in total 48 physical cores, each 2 share a FPU)
of those: 2 with 8x 16GB RAM (p4r128) and 7 with 8x 32GB RAM (p4r256)
Queues
- default
maximum walltime limit: 50 hours
Restrictions: no - long
maximum walltime limit: 200 hours
Restrictions: can only occupy half of the cluster
Network
- Quadruple Data Rate InfiniBand
- 2-level Fat Tree (Blocking factor 2)
User file space (network and local)
- ITS home directories (/home)
- Panasas file system (/panasas/data & /panasas/scratch)
- Local disk (/tmp, 333 GByte)
Commissioning & Extension
- Commissioning: März 2013
Resource Manager & Scheduler
- PBS Torque
- Maui
Operating system
- CentOS
Node topology (likwid-topology -g)
- p2: 2x AMD Opteron 6348Einklappen
-
-------------------------------------------------------------------------------- CPU name: AMD Opteron(tm) Processor 6348 CPU type: AMD Interlagos processor CPU stepping: 0 ******************************************************************************** Hardware Thread Topology ******************************************************************************** Sockets: 2 Cores per socket: 12 Threads per core: 1 -------------------------------------------------------------------------------- HWThread Thread Core Socket Available 0 0 0 0 * 1 0 1 0 * 2 0 2 0 * 3 0 3 0 * 4 0 4 0 * 5 0 5 0 * 6 0 6 0 * 7 0 7 0 * 8 0 8 0 * 9 0 9 0 * 10 0 10 0 * 11 0 11 0 * 12 0 12 1 * 13 0 13 1 * 14 0 14 1 * 15 0 15 1 * 16 0 16 1 * 17 0 17 1 * 18 0 18 1 * 19 0 19 1 * 20 0 20 1 * 21 0 21 1 * 22 0 22 1 * 23 0 23 1 * -------------------------------------------------------------------------------- Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 ) Socket 1: ( 12 13 14 15 16 17 18 19 20 21 22 23 ) -------------------------------------------------------------------------------- ******************************************************************************** Cache Topology ******************************************************************************** Level: 1 Size: 16 kB Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) -------------------------------------------------------------------------------- Level: 2 Size: 2 MB Cache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) -------------------------------------------------------------------------------- Level: 3 Size: 6 MB Cache groups: ( 0 1 2 3 4 5 ) ( 6 7 8 9 10 11 ) ( 12 13 14 15 16 17 ) ( 18 19 20 21 22 23 ) -------------------------------------------------------------------------------- ******************************************************************************** NUMA Topology ******************************************************************************** NUMA domains: 4 -------------------------------------------------------------------------------- Domain: 0 Processors: ( 0 1 2 3 4 5 ) Distances: 10 16 22 16 Free memory: 15403.3 MB Total memory: 16383.1 MB -------------------------------------------------------------------------------- Domain: 1 Processors: ( 6 7 8 9 10 11 ) Distances: 16 10 16 22 Free memory: 15874.5 MB Total memory: 16384 MB -------------------------------------------------------------------------------- Domain: 3 Processors: ( 12 13 14 15 16 17 ) Distances: 22 16 10 16 Free memory: 16009.3 MB Total memory: 16368 MB -------------------------------------------------------------------------------- Domain: 2 Processors: ( 18 19 20 21 22 23 ) Distances: 16 22 16 10 Free memory: 15942.7 MB Total memory: 16384 MB -------------------------------------------------------------------------------- ******************************************************************************** Graphical Topology ******************************************************************************** Socket 0: +-------------------------------------------------------------------------------------------------------------------------+ | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 0 | | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | | 10 | | 11 | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | +---------------------------------------------------------+ +---------------------------------------------------------+ | | | 6 MB | | 6 MB | | | +---------------------------------------------------------+ +---------------------------------------------------------+ | +-------------------------------------------------------------------------------------------------------------------------+ Socket 1: +-------------------------------------------------------------------------------------------------------------------------+ | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 12 | | 13 | | 14 | | 15 | | 16 | | 17 | | 18 | | 19 | | 20 | | 21 | | 22 | | 23 | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | +---------------------------------------------------------+ +---------------------------------------------------------+ | | | 6 MB | | 6 MB | | | +---------------------------------------------------------+ +---------------------------------------------------------+ | +-------------------------------------------------------------------------------------------------------------------------+
- p4: 4x AMD Opteron 6348Einklappen
-
-------------------------------------------------------------------------------- CPU name: AMD Opteron(tm) Processor 6348 CPU type: AMD Interlagos processor CPU stepping: 0 ******************************************************************************** Hardware Thread Topology ******************************************************************************** Sockets: 4 Cores per socket: 12 Threads per core: 1 -------------------------------------------------------------------------------- HWThread Thread Core Socket Available 0 0 0 0 * 1 0 1 0 * 2 0 2 0 * 3 0 3 0 * 4 0 4 0 * 5 0 5 0 * 6 0 6 0 * 7 0 7 0 * 8 0 8 0 * 9 0 9 0 * 10 0 10 0 * 11 0 11 0 * 12 0 12 1 * 13 0 13 1 * 14 0 14 1 * 15 0 15 1 * 16 0 16 1 * 17 0 17 1 * 18 0 18 1 * 19 0 19 1 * 20 0 20 1 * 21 0 21 1 * 22 0 22 1 * 23 0 23 1 * 24 0 24 2 * 25 0 25 2 * 26 0 26 2 * 27 0 27 2 * 28 0 28 2 * 29 0 29 2 * 30 0 30 2 * 31 0 31 2 * 32 0 32 2 * 33 0 33 2 * 34 0 34 2 * 35 0 35 2 * 36 0 36 3 * 37 0 37 3 * 38 0 38 3 * 39 0 39 3 * 40 0 40 3 * 41 0 41 3 * 42 0 42 3 * 43 0 43 3 * 44 0 44 3 * 45 0 45 3 * 46 0 46 3 * 47 0 47 3 * -------------------------------------------------------------------------------- Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 ) Socket 1: ( 12 13 14 15 16 17 18 19 20 21 22 23 ) Socket 2: ( 24 25 26 27 28 29 30 31 32 33 34 35 ) Socket 3: ( 36 37 38 39 40 41 42 43 44 45 46 47 ) -------------------------------------------------------------------------------- ******************************************************************************** Cache Topology ******************************************************************************** Level: 1 Size: 16 kB Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 ) -------------------------------------------------------------------------------- Level: 2 Size: 2 MB Cache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) ( 24 25 ) ( 26 27 ) ( 28 29 ) ( 30 31 ) ( 32 33 ) ( 34 35 ) ( 36 37 ) ( 38 39 ) ( 40 41 ) ( 42 43 ) ( 44 45 ) ( 46 47 ) -------------------------------------------------------------------------------- Level: 3 Size: 6 MB Cache groups: ( 0 1 2 3 4 5 ) ( 6 7 8 9 10 11 ) ( 12 13 14 15 16 17 ) ( 18 19 20 21 22 23 ) ( 24 25 26 27 28 29 ) ( 30 31 32 33 34 35 ) ( 36 37 38 39 40 41 ) ( 42 43 44 45 46 47 ) -------------------------------------------------------------------------------- ******************************************************************************** NUMA Topology ******************************************************************************** NUMA domains: 8 -------------------------------------------------------------------------------- Domain: 0 Processors: ( 0 1 2 3 4 5 ) Distances: 10 16 16 22 16 22 16 22 Free memory: 20688.4 MB Total memory: 32765.9 MB -------------------------------------------------------------------------------- Domain: 1 Processors: ( 6 7 8 9 10 11 ) Distances: 16 10 22 16 22 16 22 16 Free memory: 25987.3 MB Total memory: 32768 MB -------------------------------------------------------------------------------- Domain: 2 Processors: ( 12 13 14 15 16 17 ) Distances: 16 22 10 16 16 22 16 22 Free memory: 30863.1 MB Total memory: 32768 MB -------------------------------------------------------------------------------- Domain: 3 Processors: ( 18 19 20 21 22 23 ) Distances: 22 16 16 10 22 16 22 16 Free memory: 31376.6 MB Total memory: 32768 MB -------------------------------------------------------------------------------- Domain: 4 Processors: ( 24 25 26 27 28 29 ) Distances: 16 22 16 22 10 16 16 22 Free memory: 30528.3 MB Total memory: 32768 MB -------------------------------------------------------------------------------- Domain: 5 Processors: ( 30 31 32 33 34 35 ) Distances: 22 16 22 16 16 10 22 16 Free memory: 30876.5 MB Total memory: 32768 MB -------------------------------------------------------------------------------- Domain: 6 Processors: ( 36 37 38 39 40 41 ) Distances: 16 22 16 22 16 22 10 16 Free memory: 31137.9 MB Total memory: 32768 MB -------------------------------------------------------------------------------- Domain: 7 Processors: ( 42 43 44 45 46 47 ) Distances: 22 16 22 16 22 16 16 10 Free memory: 31351.5 MB Total memory: 32752 MB -------------------------------------------------------------------------------- ******************************************************************************** Graphical Topology ******************************************************************************** Socket 0: +-------------------------------------------------------------------------------------------------------------------------+ | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 0 | | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | | 10 | | 11 | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | +---------------------------------------------------------+ +---------------------------------------------------------+ | | | 6 MB | | 6 MB | | | +---------------------------------------------------------+ +---------------------------------------------------------+ | +-------------------------------------------------------------------------------------------------------------------------+ Socket 1: +-------------------------------------------------------------------------------------------------------------------------+ | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 12 | | 13 | | 14 | | 15 | | 16 | | 17 | | 18 | | 19 | | 20 | | 21 | | 22 | | 23 | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | +---------------------------------------------------------+ +---------------------------------------------------------+ | | | 6 MB | | 6 MB | | | +---------------------------------------------------------+ +---------------------------------------------------------+ | +-------------------------------------------------------------------------------------------------------------------------+ Socket 2: +-------------------------------------------------------------------------------------------------------------------------+ | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 24 | | 25 | | 26 | | 27 | | 28 | | 29 | | 30 | | 31 | | 32 | | 33 | | 34 | | 35 | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | +---------------------------------------------------------+ +---------------------------------------------------------+ | | | 6 MB | | 6 MB | | | +---------------------------------------------------------+ +---------------------------------------------------------+ | +-------------------------------------------------------------------------------------------------------------------------+ Socket 3: +-------------------------------------------------------------------------------------------------------------------------+ | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 36 | | 37 | | 38 | | 39 | | 40 | | 41 | | 42 | | 43 | | 44 | | 45 | | 46 | | 47 | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | 2 MB | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ | | +---------------------------------------------------------+ +---------------------------------------------------------+ | | | 6 MB | | 6 MB | | | +---------------------------------------------------------+ +---------------------------------------------------------+ | +-------------------------------------------------------------------------------------------------------------------------+
Node-level performance (likwid-bench)
- Parallel data transfer rate - p2Einklappen
-
Parallel data transfer rate on btrzx3 p2 nodes. In the 2-socket data set, both sockets are balanced.
- Vector triad - p2Einklappen
-
Performance of the vector triad on btrzx3 p2 nodes.
MPI benchmarks (OSU)
- MPI BandwidthEinklappen
-
P2P MPI Bandwidth between processes on the same socket (by core), the same node but different sockets (by socket), and different nodes (by node) using ofi or shm:tmi fabric interfaces.
- MPI LatencyEinklappen
-
Single-point measurements of the point-2-point MPI latency (limit: size 0) between processes on the same socket (by core), the same node but different sockets (by socket), and different nodes in the same or different racks (by node).
Statistics
- 2017 - Job size [p2]Einklappen
-
Spent computation time by job size and runtime.
- 2018 - Job size [p2]Einklappen
-
Spent computation time by job size and runtime.
Current workload
Link to the workload page