Druckansicht der Internetadresse:

Forschungszentrum für wissenschaftliches Rechnen an der Universität Bayreuth

Seite drucken

btrzx4 (2018)

General remarks

The cluster btrzx4 is an extension of btrzx2 which went into operation in April 2018 and was purchased within the project HiPerSim4all of the European Social Fund with the aim to strengthen small and medium business on a regional scale. Besides two login nodes, it features four different node types with the properties compute20, ssd, phi, and cuda which are connected by an Intel Omni-Path network. While the phi nodes are equipped with a single Intel Xeon Phi CPU, the compute20, ssd, and cuda nodes all contain two 10-core Intel Xeon CPUs, eventually with a 1TB SSD (ssd) or a Nvidia Tesla GPU (cuda).

Login nodes

  • btrzx4-1.rz.uni-bayreuth.de
  • btrzx4-2.rz.uni-bayreuth.de

Compute nodes (state June 2018)

  • 79 nodes (compute20)
    2x Intel Xeon E5-2630 v4 @ 2.2GHz (Broadwell) with 2x 64GB RAM
    (in total 20 physical cores, hyperthreading disabled)
  • 5 nodes (ssd)
    2x Intel Xeon E5-2630 v4 @ 2.2GHz (Broadwell) with 2x 64GB RAM
    (in total 20 physical nodes, hyperthreading disabled) and
    1 TB SSD
  • 2 nodes (phi)
    1x Intel Xeon Phi 7210 @ 1.3GHz (Knights Landing) with 1x 256GB RAM
    (in total 64 physcal cores, 4 threads per core)
  • 5 nodes (cuda)
    2x Intel Xeon E5-2630 v4 @ 2.2GHz (Broadwell) with 2x 128GB RAM
    (in total 20 physical cores, hyperthreading disabled) and
    1x Nvidia Tesla P100 (12GB)

Queues

  • default
    Wall time limit: 100 hours
    Restrictions: no

Network

  • Intel Omni-Path
  • 2-level Fat Tree (Blocking factor 2)

User file space (network and local)

  • ITS home directories (/home)
  • Panasas file system (/data & /scratch)
  • Local disk (/tmp, 840 GByte)

Commissioning & Extension

  • Commissioning: Apr. 2018

Resource Manager & Scheduler

  • PBS Torque
  • Maui

Operating system

  • CentOS

Node topology (likwid-topology -g)

compute20, ssd, & cuda: 2x Intel Xeon E5-2630 v4Einklappen
--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
CPU type:	Intel Xeon Broadwell EN/EP/EX processor
CPU stepping:	1
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets:		2
Cores per socket:	10
Threads per core:	1
--------------------------------------------------------------------------------
HWThread	Thread		Core		Socket		Available
0		0		0		0		*
1		0		1		0		*
2		0		2		0		*
3		0		3		0		*
4		0		4		0		*
5		0		5		0		*
6		0		6		0		*
7		0		7		0		*
8		0		8		0		*
9		0		9		0		*
10		0		10		1		*
11		0		11		1		*
12		0		12		1		*
13		0		13		1		*
14		0		14		1		*
15		0		15		1		*
16		0		16		1		*
17		0		17		1		*
18		0		18		1		*
19		0		19		1		*
--------------------------------------------------------------------------------
Socket 0:		( 0 1 2 3 4 5 6 7 8 9 )
Socket 1:		( 10 11 12 13 14 15 16 17 18 19 )
--------------------------------------------------------------------------------
********************************************************************************
Cache Topology
********************************************************************************
Level:			1
Size:			32 kB
Cache groups:		( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 )
--------------------------------------------------------------------------------
Level:			2
Size:			256 kB
Cache groups:		( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 )
--------------------------------------------------------------------------------
Level:			3
Size:			25 MB
Cache groups:		( 0 1 2 3 4 5 6 7 8 9 ) ( 10 11 12 13 14 15 16 17 18 19 )
--------------------------------------------------------------------------------
********************************************************************************
NUMA Topology
********************************************************************************
NUMA domains:		2
--------------------------------------------------------------------------------
Domain:			0
Processors:		( 0 1 2 3 4 5 6 7 8 9 )
Distances:		10 21
Free memory:		59312.3 MB
Total memory:		65441 MB
--------------------------------------------------------------------------------
Domain:			1
Processors:		( 10 11 12 13 14 15 16 17 18 19 )
Distances:		21 10
Free memory:		62872.7 MB
Total memory:		65536 MB
--------------------------------------------------------------------------------


********************************************************************************
Graphical Topology
********************************************************************************
Socket 0:

+---------------------------------------------------------------------------------------------------------------+
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| |    0   | |    1   | |    2   | |    3   | |    4   | |    5   | |    6   | |    7   | |    8   | |    9   | |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| +-----------------------------------------------------------------------------------------------------------+ |
| |                                                   25 MB                                                   | |
| +-----------------------------------------------------------------------------------------------------------+ |
+---------------------------------------------------------------------------------------------------------------+

Socket 1:

+---------------------------------------------------------------------------------------------------------------+
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| |   10   | |   11   | |   12   | |   13   | |   14   | |   15   | |   16   | |   17   | |   18   | |   19   | |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |  32 kB | |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | | 256 kB | |
| +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ +--------+ |
| +-----------------------------------------------------------------------------------------------------------+ |
| |                                                   25 MB                                                   | |
| +-----------------------------------------------------------------------------------------------------------+ |
+---------------------------------------------------------------------------------------------------------------+

phi: 1x Intel Xeon Phi 7210Einklappen
--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz
CPU type:	Intel Xeon Phi (Knights Landing) (Co)Processor
CPU stepping:	1
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets:		1
Cores per socket:	64
Threads per core:	1
--------------------------------------------------------------------------------
HWThread	Thread		Core		Socket		Available
0		0		0		0		*
1		0		1		0		*
2		0		2		0		*
3		0		3		0		*
4		0		4		0		*
5		0		5		0		*
6		0		6		0		*
7		0		7		0		*
8		0		8		0		*
9		0		9		0		*
10		0		10		0		*
11		0		11		0		*
12		0		12		0		*
13		0		13		0		*
14		0		14		0		*
15		0		15		0		*
16		0		16		0		*
17		0		17		0		*
18		0		18		0		*
19		0		19		0		*
20		0		20		0		*
21		0		21		0		*
22		0		22		0		*
23		0		23		0		*
24		0		24		0		*
25		0		25		0		*
26		0		26		0		*
27		0		27		0		*
28		0		28		0		*
29		0		29		0		*
30		0		30		0		*
31		0		31		0		*
32		0		32		0		*
33		0		33		0		*
34		0		34		0		*
35		0		35		0		*
36		0		36		0		*
37		0		37		0		*
38		0		38		0		*
39		0		39		0		*
40		0		40		0		*
41		0		41		0		*
42		0		42		0		*
43		0		43		0		*
44		0		44		0		*
45		0		45		0		*
46		0		46		0		*
47		0		47		0		*
48		0		48		0		*
49		0		49		0		*
50		0		50		0		*
51		0		51		0		*
52		0		52		0		*
53		0		53		0		*
54		0		54		0		*
55		0		55		0		*
56		0		56		0		*
57		0		57		0		*
58		0		58		0		*
59		0		59		0		*
60		0		60		0		*
61		0		61		0		*
62		0		62		0		*
63		0		63		0		*
--------------------------------------------------------------------------------
Socket 0:		( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 )
--------------------------------------------------------------------------------
********************************************************************************
Cache Topology
********************************************************************************
Level:			1
Size:			32 kB
Cache groups:		( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 ) ( 48 ) ( 49 ) ( 50 ) ( 51 ) ( 52 ) ( 53 ) ( 54 ) ( 55 ) ( 56 ) ( 57 ) ( 58 ) ( 59 ) ( 60 ) ( 61 ) ( 62 ) ( 63 )
--------------------------------------------------------------------------------
Level:			2
Size:			1 MB
Cache groups:		( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) ( 24 25 ) ( 26 27 ) ( 28 29 ) ( 30 31 ) ( 32 33 ) ( 34 35 ) ( 36 37 ) ( 38 39 ) ( 40 41 ) ( 42 43 ) ( 44 45 ) ( 46 47 ) ( 48 49 ) ( 50 51 ) ( 52 53 ) ( 54 55 ) ( 56 57 ) ( 58 59 ) ( 60 61 ) ( 62 63 )
--------------------------------------------------------------------------------
********************************************************************************
NUMA Topology
********************************************************************************
NUMA domains:		1
--------------------------------------------------------------------------------
Domain:			0
Processors:		( 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 )
Distances:		10
Free memory:		183806 MB
Total memory:		193180 MB
--------------------------------------------------------------------------------


********************************************************************************
Graphical Topology
********************************************************************************
Socket 0:

+----------------------------------------- ... -----------------------------------------+
| +-------+ +-------+ +-------+ +-------+  ...  +-------+ +-------+ +-------+ +-------+ |
| |   0   | |   1   | |   2   | |   3   |  ...  |   60  | |   61  | |   62  | |   63  | |
| +-------+ +-------+ +-------+ +-------+  ...  +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+  ...  +-------+ +-------+ +-------+ +-------+ |
| | 32 kB | | 32 kB | | 32 kB | | 32 kB |  ...  | 32 kB | | 32 kB | | 32 kB | | 32 kB | |
| +-------+ +-------+ +-------+ +-------+  ...  +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+  ...  +-----------------+ +-----------------+ |
| |       1 MB      | |       1 MB      |  ...  |       1 MB      | |       1 MB      | |
| +-----------------+ +-----------------+  ...  +-----------------+ +-----------------+ |
+----------------------------------------- ... -----------------------------------------+

Node-level performance (likwid-bench)

Parallel data transfer rate - compute20Einklappen
Vector triad - compute20Einklappen
Parallel data transfer rate - phiEinklappen
Vector triad - phiEinklappen

MPI benchmarks (OSU)

MPI BandwidthEinklappen
MPI LatencyEinklappen

Statistics

2018 - Job size [compute20]Einklappen

Verantwortlich für die Redaktion: Dr.rer.nat. Ingo Schelter

Facebook Twitter Youtube-Kanal Instagram Blog Kontakt