Druckansicht der Internetadresse:

Forschungszentrum für wissenschaftliches Rechnen an der Universität Bayreuth

Seite drucken

btrzx3 (2013)

General remarks

The cluster btrzx3 went into operation in March 2013. Besides two login nodes, it features two different node types with the properties p2 and p4 which are connected by a QDR InfiniBand network. p2 and p4 nodes are equipped with two or four 12-core AMD Opteron CPUs, respectively. A special particularity of the nodes in btrzx3, which can be relevant considering performance, is the number of memory interfaces per CPU. In contrast to the other clusters in which each CPU has a single memory interface, a single 12-core AMD CPU divides its cores into two sets of 6 cores with a separate memory interface each. The latter can become especially relevant for running OpenMP programs (see NUMA). Moreover, two adjacent physical cores share all of their caches as well as their FPU. Therefore, it can make sense to use every second core only when running parallel jobs, e.g., by dropping every second entry in the host file. (Note, however, that all cores must still be requested through ppn=24 or ppn=48, respectively, to get complete nodes.)

Login nodes

  • btrzx3-1.rz.uni-bayreuth.de
  • btrzx3-2.rz.uni-bayreuth.de

Compute nodes (state June 2018)

  • 417 nodes (p2)
    2x AMD Opteron 6348 @ 2.80GHz (Abu Dhabi)
    (in total 24 physical cores, each 2 share a FPU)
    of those: 278 with 4x 16GB RAM (p2r64), 135 with 4x 32GB RAM (p2r128), and 4 with 4x 64GB RAM (p2r256)
  • 9 nodes (p4)
    4x AMD Opteron 6348 @ 2.80GHz (Abu Dhabi)
    (in total 48 physical cores, each 2 share a FPU)
    of those: 2 with 8x 16GB RAM (p4r128) and 7 with 8x 32GB RAM (p4r256)

Queues

  • default
    maximum walltime limit: 50 hours
    Restrictions: no
  • long
    maximum walltime limit: 200 hours
    Restrictions: can only occupy half of the cluster

Network

  • Quadruple Data Rate InfiniBand
  • 2-level Fat Tree (Blocking factor 2)

User file space (network and local)

  • ITS home directories (/home)
  • Panasas file system (/panasas/data & /panasas/scratch)
  • Local disk (/tmp, 333 GByte)

Commissioning & Extension

  • Commissioning: März 2013

Resource Manager & Scheduler

  • PBS Torque
  • Maui

Operating system

  • CentOS

Node topology (likwid-topology -g)

p2: 2x AMD Opteron 6348Einklappen
--------------------------------------------------------------------------------
CPU name:	AMD Opteron(tm) Processor 6348                 
CPU type:	AMD Interlagos processor
CPU stepping:	0
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets:		2
Cores per socket:	12
Threads per core:	1
--------------------------------------------------------------------------------
HWThread	Thread		Core		Socket		Available
0		0		0		0		*
1		0		1		0		*
2		0		2		0		*
3		0		3		0		*
4		0		4		0		*
5		0		5		0		*
6		0		6		0		*
7		0		7		0		*
8		0		8		0		*
9		0		9		0		*
10		0		10		0		*
11		0		11		0		*
12		0		12		1		*
13		0		13		1		*
14		0		14		1		*
15		0		15		1		*
16		0		16		1		*
17		0		17		1		*
18		0		18		1		*
19		0		19		1		*
20		0		20		1		*
21		0		21		1		*
22		0		22		1		*
23		0		23		1		*
--------------------------------------------------------------------------------
Socket 0:		( 0 1 2 3 4 5 6 7 8 9 10 11 )
Socket 1:		( 12 13 14 15 16 17 18 19 20 21 22 23 )
--------------------------------------------------------------------------------
********************************************************************************
Cache Topology
********************************************************************************
Level:			1
Size:			16 kB
Cache groups:		( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 )
--------------------------------------------------------------------------------
Level:			2
Size:			2 MB
Cache groups:		( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 )
--------------------------------------------------------------------------------
Level:			3
Size:			6 MB
Cache groups:		( 0 1 2 3 4 5 ) ( 6 7 8 9 10 11 ) ( 12 13 14 15 16 17 ) ( 18 19 20 21 22 23 )
--------------------------------------------------------------------------------
********************************************************************************
NUMA Topology
********************************************************************************
NUMA domains:		4
--------------------------------------------------------------------------------
Domain:			0
Processors:		( 0 1 2 3 4 5 )
Distances:		10 16 22 16
Free memory:		15403.3 MB
Total memory:		16383.1 MB
--------------------------------------------------------------------------------
Domain:			1
Processors:		( 6 7 8 9 10 11 )
Distances:		16 10 16 22
Free memory:		15874.5 MB
Total memory:		16384 MB
--------------------------------------------------------------------------------
Domain:			3
Processors:		( 12 13 14 15 16 17 )
Distances:		22 16 10 16
Free memory:		16009.3 MB
Total memory:		16368 MB
--------------------------------------------------------------------------------
Domain:			2
Processors:		( 18 19 20 21 22 23 )
Distances:		16 22 16 10
Free memory:		15942.7 MB
Total memory:		16384 MB
--------------------------------------------------------------------------------


********************************************************************************
Graphical Topology
********************************************************************************
Socket 0:

+-------------------------------------------------------------------------------------------------------------------------+
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| |   0   | |   1   | |   2   | |   3   | |   4   | |   5   | |   6   | |   7   | |   8   | |   9   | |   10  | |   11  | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
| |                           6 MB                          | |                           6 MB                          | |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
+-------------------------------------------------------------------------------------------------------------------------+

Socket 1:

+-------------------------------------------------------------------------------------------------------------------------+
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| |   12  | |   13  | |   14  | |   15  | |   16  | |   17  | |   18  | |   19  | |   20  | |   21  | |   22  | |   23  | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
| |                           6 MB                          | |                           6 MB                          | |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
+-------------------------------------------------------------------------------------------------------------------------+

p4: 4x AMD Opteron 6348Einklappen
--------------------------------------------------------------------------------
CPU name:	AMD Opteron(tm) Processor 6348                 
CPU type:	AMD Interlagos processor
CPU stepping:	0
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets:		4
Cores per socket:	12
Threads per core:	1
--------------------------------------------------------------------------------
HWThread	Thread		Core		Socket		Available
0		0		0		0		*
1		0		1		0		*
2		0		2		0		*
3		0		3		0		*
4		0		4		0		*
5		0		5		0		*
6		0		6		0		*
7		0		7		0		*
8		0		8		0		*
9		0		9		0		*
10		0		10		0		*
11		0		11		0		*
12		0		12		1		*
13		0		13		1		*
14		0		14		1		*
15		0		15		1		*
16		0		16		1		*
17		0		17		1		*
18		0		18		1		*
19		0		19		1		*
20		0		20		1		*
21		0		21		1		*
22		0		22		1		*
23		0		23		1		*
24		0		24		2		*
25		0		25		2		*
26		0		26		2		*
27		0		27		2		*
28		0		28		2		*
29		0		29		2		*
30		0		30		2		*
31		0		31		2		*
32		0		32		2		*
33		0		33		2		*
34		0		34		2		*
35		0		35		2		*
36		0		36		3		*
37		0		37		3		*
38		0		38		3		*
39		0		39		3		*
40		0		40		3		*
41		0		41		3		*
42		0		42		3		*
43		0		43		3		*
44		0		44		3		*
45		0		45		3		*
46		0		46		3		*
47		0		47		3		*
--------------------------------------------------------------------------------
Socket 0:		( 0 1 2 3 4 5 6 7 8 9 10 11 )
Socket 1:		( 12 13 14 15 16 17 18 19 20 21 22 23 )
Socket 2:		( 24 25 26 27 28 29 30 31 32 33 34 35 )
Socket 3:		( 36 37 38 39 40 41 42 43 44 45 46 47 )
--------------------------------------------------------------------------------
********************************************************************************
Cache Topology
********************************************************************************
Level:			1
Size:			16 kB
Cache groups:		( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 )
--------------------------------------------------------------------------------
Level:			2
Size:			2 MB
Cache groups:		( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 ) ( 16 17 ) ( 18 19 ) ( 20 21 ) ( 22 23 ) ( 24 25 ) ( 26 27 ) ( 28 29 ) ( 30 31 ) ( 32 33 ) ( 34 35 ) ( 36 37 ) ( 38 39 ) ( 40 41 ) ( 42 43 ) ( 44 45 ) ( 46 47 )
--------------------------------------------------------------------------------
Level:			3
Size:			6 MB
Cache groups:		( 0 1 2 3 4 5 ) ( 6 7 8 9 10 11 ) ( 12 13 14 15 16 17 ) ( 18 19 20 21 22 23 ) ( 24 25 26 27 28 29 ) ( 30 31 32 33 34 35 ) ( 36 37 38 39 40 41 ) ( 42 43 44 45 46 47 )
--------------------------------------------------------------------------------
********************************************************************************
NUMA Topology
********************************************************************************
NUMA domains:		8
--------------------------------------------------------------------------------
Domain:			0
Processors:		( 0 1 2 3 4 5 )
Distances:		10 16 16 22 16 22 16 22
Free memory:		20688.4 MB
Total memory:		32765.9 MB
--------------------------------------------------------------------------------
Domain:			1
Processors:		( 6 7 8 9 10 11 )
Distances:		16 10 22 16 22 16 22 16
Free memory:		25987.3 MB
Total memory:		32768 MB
--------------------------------------------------------------------------------
Domain:			2
Processors:		( 12 13 14 15 16 17 )
Distances:		16 22 10 16 16 22 16 22
Free memory:		30863.1 MB
Total memory:		32768 MB
--------------------------------------------------------------------------------
Domain:			3
Processors:		( 18 19 20 21 22 23 )
Distances:		22 16 16 10 22 16 22 16
Free memory:		31376.6 MB
Total memory:		32768 MB
--------------------------------------------------------------------------------
Domain:			4
Processors:		( 24 25 26 27 28 29 )
Distances:		16 22 16 22 10 16 16 22
Free memory:		30528.3 MB
Total memory:		32768 MB
--------------------------------------------------------------------------------
Domain:			5
Processors:		( 30 31 32 33 34 35 )
Distances:		22 16 22 16 16 10 22 16
Free memory:		30876.5 MB
Total memory:		32768 MB
--------------------------------------------------------------------------------
Domain:			6
Processors:		( 36 37 38 39 40 41 )
Distances:		16 22 16 22 16 22 10 16
Free memory:		31137.9 MB
Total memory:		32768 MB
--------------------------------------------------------------------------------
Domain:			7
Processors:		( 42 43 44 45 46 47 )
Distances:		22 16 22 16 22 16 16 10
Free memory:		31351.5 MB
Total memory:		32752 MB
--------------------------------------------------------------------------------


********************************************************************************
Graphical Topology
********************************************************************************
Socket 0:

+-------------------------------------------------------------------------------------------------------------------------+
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| |   0   | |   1   | |   2   | |   3   | |   4   | |   5   | |   6   | |   7   | |   8   | |   9   | |   10  | |   11  | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
| |                           6 MB                          | |                           6 MB                          | |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
+-------------------------------------------------------------------------------------------------------------------------+

Socket 1:

+-------------------------------------------------------------------------------------------------------------------------+
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| |   12  | |   13  | |   14  | |   15  | |   16  | |   17  | |   18  | |   19  | |   20  | |   21  | |   22  | |   23  | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
| |                           6 MB                          | |                           6 MB                          | |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
+-------------------------------------------------------------------------------------------------------------------------+

Socket 2:

+-------------------------------------------------------------------------------------------------------------------------+
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| |   24  | |   25  | |   26  | |   27  | |   28  | |   29  | |   30  | |   31  | |   32  | |   33  | |   34  | |   35  | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
| |                           6 MB                          | |                           6 MB                          | |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
+-------------------------------------------------------------------------------------------------------------------------+

Socket 3:

+-------------------------------------------------------------------------------------------------------------------------+
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| |   36  | |   37  | |   38  | |   39  | |   40  | |   41  | |   42  | |   43  | |   44  | |   45  | |   46  | |   47  | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | | 16 kB | |
| +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |       2 MB      | |
| +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ +-----------------+ |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
| |                           6 MB                          | |                           6 MB                          | |
| +---------------------------------------------------------+ +---------------------------------------------------------+ |
+-------------------------------------------------------------------------------------------------------------------------+

Node-level performance (likwid-bench)

Parallel data transfer rate - p2Einklappen
Vector triad - p2Einklappen

MPI benchmarks (OSU)

MPI BandwidthEinklappen
MPI LatencyEinklappen

Statistics

2017 - Job size [p2]Einklappen
2018 - Job size [p2]Einklappen

Verantwortlich für die Redaktion: Dr.rer.nat. Ingo Schelter

Facebook Twitter Youtube-Kanal Instagram Blog Kontakt