Druckansicht der Internetadresse:

BZHPC - Bayreuth Centre for High Performance Computing

Seite drucken

btrzx1 (emil, 2020)

General remarks

The cluster btrzx1 went into operation in August 2020. Besides two login nodes, it consists of 372 nodes which are connected by an InfiniBand network and a new Panasas file system. Unlike the previous clusters, btrzx1 uses Slurm as resource manager instead of PBS/Torque. Moreover, the ITS file server (e.g., the ITS home directory) is not mounted on the cluster for performance reasons but every users has a separate home directory which lies on the Panasas file system.

Acknowledging emil / Publications

As with other DFG-funded projects, results must be made available to the general public in an appropriate manner. The publications must contain a reference to the DFG funding (so-called “Funding Acknowledgement”) in the language of the publication, stating the project number.

Whenever the emil has been used to produce results used in a publication or posters, we kindly request citing the service in the acknowledgements:

Calculations were performed using the emil-cluster of the Bayreuth Centre for High Performance Computing (https://www.bzhpc.uni-bayreuth.de), funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 422127126.

Whereby the funding acknowledgement is mandatory.

ogin

The login nodes of btrzx1 (emil) are accessible with ssh via emil.rz.uni-bayreuth.de only from university networks. If you are outside the university, a VPN connection is required.

Compute nodes (state October 2023)

  • 357 nodes (Type-A; normal and dev partition) 
    • 2x AMD Epyc 7302 16c CPU @ 3.00GHz, 32 cores total 
    • 2x 64GB RAM
    • 500 GB HDD 
  • 12 nodes (Type-B; WAP partition)
    • 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
    • 16x 16GB RAM
    • 256 GB nvme
  • 2 nodes (Type-C; 1 WAP , 1 GPU Partition )
    • 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
    • 16x 32GB RAM
    • 500 GB nvme
    • 1x NVIDIA Tesla V100 16GB
  • 1 node (Type-D; GPU partition)
    • 2x AMD Epyc 7352 24c CPU @ 2.30GHz, 48 cores total
    • 16x 32GB RAM
    • 500 GB nvme
    • 2x NVIDIA Tesla V100 16GB

Queues / Partitions

  • normal
    Priority: multifiactor, most weight on the group's financial share in the cluster and consumed ressources
    Wall time limit: 24 hours
    Restrictions: cpu nodes only
  • WAP
    Priority: multifiactor, most weight on the group's financial share in the partition (WAP Mathe) and consumed ressources
    Wall time limit: 24 hours
    Restrictions: no
  • GPU
    Priority: multifiactor, most weight on the group's financial share in the cluster and consumed ressources 
    Wall time limit: 24 hours
    Restrictions: single gpu nodes only
  • dev
    Priority: multifactor,  most weight on consumed ressources
    Wall time limit: 30 Minutes (default), 1 hour (max)
    Restrictions: max 4 nodes per job

Network

  • Infiniband (56 Gbit/s)
  • 2-level Fat Tree (Blocking factor 2)

User file space (network and local)

  • Panasas network file system
    • /home: 100GB per User
    • /scratch: no soft-quota
      • no snapshots, no backup
      • wiped down to 40% usage on 60% usage by earliest mtime 
  • Commissioning & Extension
  • August 2020

Resource Manager & Scheduler

  • Slurm

Operating system

  • ​CentOS 7.9

Node topology (likwid-topology -g)

2x AMD Epyc 7302Einklappen
--------------------------------------------------------------------------------
CPU name: AMD EPYC 7302 16-Core Processor 
CPU type: AMD K17 (Zen2) architecture
CPU stepping: 0
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets: 2
Cores per socket: 16
Threads per core: 1
--------------------------------------------------------------------------------
HWThread Thread Core Socket Available
0 0 0 0 *
1 0 1 0 *
2 0 2 0 *
3 0 3 0 *
4 0 4 0 *
5 0 5 0 *
6 0 6 0 *
7 0 7 0 *
8 0 8 0 *
9 0 9 0 *
10 0 10 0 *
11 0 11 0 *
12 0 12 0 *
13 0 13 0 *
14 0 14 0 *
15 0 15 0 *
16 0 16 1 *
17 0 17 1 *
18 0 18 1 *
19 0 19 1 *
20 0 20 1 *
21 0 21 1 *
22 0 22 1 *
23 0 23 1 *
24 0 24 1 *
25 0 25 1 *
26 0 26 1 *
27 0 27 1 *
28 0 28 1 *
29 0 29 1 *
30 0 30 1 *
31 0 31 1 *
--------------------------------------------------------------------------------
Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
Socket 1: ( 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 )
--------------------------------------------------------------------------------
********************************************************************************
Cache Topology
********************************************************************************
Level: 1
Size: 32 kB
Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 )
--------------------------------------------------------------------------------
Level: 2
Size: 512 kB
Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 )
--------------------------------------------------------------------------------
Level: 3
Size: 16 MB
Cache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) ( 24 25 ) ( 26 27 ) ( 28 29 ) ( 30 31 )
--------------------------------------------------------------------------------
********************************************************************************
NUMA Topology
********************************************************************************
NUMA domains: 2
--------------------------------------------------------------------------------
Domain: 0
Processors: ( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 )
Distances: 10 32
Free memory: 61130.3 MB
Total memory: 65403.8 MB
--------------------------------------------------------------------------------
Domain: 1
Processors: ( 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 )
Distances: 32 10
Free memory: 63574.7 MB
Total memory: 65534 MB
--------------------------------------------------------------------------------


********************************************************************************
Graphical Topology
********************************************************************************
Socket 0:

+-----------------------------------------------------------------------------------------------------------------+
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |  0 | |  1 | |  2 | |  3 | |  4 | |  5 | |  6 | |  7 | |  8 | |  9 | | 10 | | 11 | | 12 | | 13 | | 14 | | 15 | |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
| |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
+-----------------------------------------------------------------------------------------------------------------+
Socket 1:

+-----------------------------------------------------------------------------------------------------------------+
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| | 16 | | 17 | | 18 | | 19 | | 20 | | 21 | | 22 | | 23 | | 24 | | 25 | | 26 | | 27 | | 28 | | 29 | | 30 | | 31 | |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |32kB| |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |512kB |
| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
| |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |   16 MB   | |
| +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ +-----------+ |
+-----------------------------------------------------------------------------------------------------------------+

MPI benchmarks (OSU)

MPI BandwidthEinklappen
MPI LatencyEinklappen

Node-level performance (likwid-bench)

Parallel data transfer rate (close)Einklappen
Parallel data transfer rate (scattered)Einklappen
Vector triadEinklappen

Verantwortlich für die Redaktion: Webmaster BZHPC

Facebook Twitter Youtube-Kanal Instagram UBT-A Kontakt