Discovery & IOC Phase Network Architecture

1. Discovery & IOC Phase Network Architecture

Discovery & IOC
Phase Network
Architecture

2. Environments

Discovery Phase => Phase-1
IOC Phase => Phase-2

3.

Data Center Module-15 Overview
Module 15 in Data Center
Location: Astana
Tenant: AI Fund of Republic of Kazakhstan
42
1
2
COM TOR - CE6865[10]
41
COM BMC IP - CE5855 [ 5 ]
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
9
22
10
21
9
20
10
19
9
18
10
17
9
16
10
15
9
14
10
13
9
12
10
11
9
10
10
9
9
1
8
2288H V5 APP-30
2288H V5 APP-29
2288H V5 APP-28
2288H V5 APP-27
2288H V5 APP-26
2288H V5 APP-25
2288H V5 APP-24
2288H V5 APP-23
0
7
9
6
10
2288H V5 APP-22
5
5
5
5
5
5
5
5
5
5
9
4
2288H V5 HDFS-8
10
3
2
9
1
10
2288H V5 HDFS-7
5
5
1
2
COM TOR - CE6865[9]
42
42
41
41
41
40
40
40
39
39
39
38
38
38
42
42
41
41
40
40
39
39
38
38
37
37
37
37
37
36
36
36
36
35
35
34
34
35
34
33
33
32
STO TOR - CE6865[6]
STO TOR - CE6865[5]
42
42
41
41
40
40
39
39
39
38
38
38
37
37
37
37
36
36
36
36
36
35
35
35
35
35
35
34
34
34
34
34
34
33
33
33
33
33
33
33
32
32
32
32
32
32
32
32
31
31
31
31
31
31
31
31
31
30
30
30
30
30
30
30
30
30
29
29
29
29
29
29
29
29
29
28
28
28
28
28
28
28
28
28
27
27
27
27
27
27
27
27
27
26
26
26
26
26
26
26
26
26
25
25
25
25
25
25
25
25
25
24
24
24
24
24
24
24
24
24
23
23
23
23
23
23
23
23
23
22
22
22
22
22
22
22
22
22
21
21
21
21
21
21
21
21
21
20
20
20
20
20
20
20
20
20
19
19
19
19
19
19
19
18
18
18
18
8
18
18
17
17
17
17
7
17
17
16
16
16
16
8
16
16
15
15
15
15
7
15
15
14
14
14
14
8
14
14
13
13
13
13
13
13
13
13
12
12
12
12
12
12
12
12
11
11
11
11
10
10
10
10
9
9
9
9
9
8
8
8
8
8
7
7
7
7
6
6
5
5
5
4
4
4
7
3
3
3
8
2
2
2
7
1
1
1
8
9
10
9
10
9
10
9
10
9
10
9
10
9
10
9
10
9
10
9
10
2288H V5 APP-21
19
2288H V5 APP-20
2288H V5 APP-19
2288H V5 APP-18
2288H V5 APP-17
2288H V5 APP-16
2288H V5 APP-15
2288H V5 APP-14
2288H V5 HDFS-6
2288H V5 HDFS-5
Cooling system
D15R06
5
5
5
5
5
5
5
5
5
5
Aggregate Switch - CE8850 [ 2 ]
41
COM BMC IP - CE5855 [ 4 ]
40
39
1
2
7
18
8
17
7
16
8
15
7
14
8
13
7
12
8
11
7
10
8
9
7
8
8
7
7
6
8
2288H V5 APP-13
2288H V5 APP-12
2288H V5 APP-11
2288H V5 APP-10
2288H V5 APP-09
2288H V5 APP-08
2288H V5 APP-07
2288H V5 HDFS-4
2288H V5 HDFS-3
D15R05
FIREWALL - USG6600E [ 2 ]
38
COM TOR - CE6865[8]
4
4
4
4
1
2
COM TOR - CE6865[7]
18
17
7
16
8
15
7
14
8
2288H V5 HDFS-2
2288H V5 HDFS-1
4
10
OBS 5280V2-18
6
4
6
OBS 5280V2-17
STO BMC IP - CE5855 [ 3]
7
2288H V5 APP-06
2288H V5 APP-05
2288H V5 APP-04
19
4
4
4
5
18
8
17
7
16
8
15
7
14
8
3
5
2288H V5 APP-02
2288H V5 APP-01
4
4
4
11
11
10
10
9
9
9
8
8
8
7
7
10
OBS 5280V2-12
6
OBS 5280V2-11
6
6
6
6
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
2
2
1
1
6
3
5
3
OBS 5280V2-16
6
3
1
D15R03
D15R04
6
3
5
6
3
2
3
5
3
OBS 5280V2-13
6
6
3
5
7
OBS 5280V2-14
2288H V5 APP-03
5
11
OBS 5280V2-15
6
7
5
2
4
3
5
7
4
4
5
11
4
4
42
1
2
40
FIREWALL - USG6600E [ 3 ]
42
1
2
41
42
OBS 5280V2-10
6
2
3
1
Cooling system
2
1
D15R02
1
D15R01
UPS System
Power System
Doors
D15R13
D15R12
1
2
Cooling system
COM TOR - CE6865[0]
D15R11
D15R10
Optical Patch Panel
42
42
42
42
41
41
41
41
40
40
40
40
39
39
39
39
38
38
38
38
BMC IP Switch - CE5855 [ 1 ]
37
37
37
37
1
2
D15R09
COM TOR - CE6865[2]
D15R08
42
41
41
41
Optical Patch Panel
40
40
40
40
FIREWALL - USG6000E [ 1 ]
39
39
39
39
38
38
38
38
37
37
37
COM TOR - CE6865[1]
42
Cooling system
42
41
1
2
42
Aggregate Switch - CE8850 [ 1 ]
1
2
D15R07
STO TOR - CE6865[3]
42
1
2
Fire extinguishing system
STO TOR - CE6865[4]
42
41
41
40
40
39
39
38
38
37
37
37
STO BMC IP - CE5855 [ 2 ]
36
36
36
36
36
36
36
36
36
36
35
35
35
35
35
35
35
35
35
35
34
34
34
34
34
34
34
34
34
34
33
33
33
33
33
33
33
33
33
33
32
32
32
32
32
32
32
32
32
32
31
31
31
31
31
31
31
31
31
31
30
30
30
30
30
30
30
30
30
30
29
29
29
29
29
29
29
29
29
29
28
28
28
28
28
28
28
28
28
28
27
27
27
27
27
27
27
27
27
27
26
26
26
26
26
26
26
26
26
26
25
25
25
25
25
25
25
25
25
25
24
24
24
24
24
24
24
24
24
24
23
23
23
23
23
23
23
23
23
23
22
22
22
22
22
22
22
22
22
22
21
21
3
21
21
21
3
21
21
21
21
21
20
20
2
20
20
20
2
20
20
20
20
20
19
19
19
19
19
19
19
19
19
18
18
3
18
18
3
18
18
18
18
18
17
17
2
17
17
2
17
17
17
17
17
16
16
3
16
16
3
16
16
16
16
15
15
2
15
15
2
15
15
15
15
14
14
14
14
14
14
14
14
13
13
3
13
13
3
13
13
13
13
12
12
2
12
12
2
12
12
11
11
3
11
11
3
11
11
10
10
2
10
10
2
10
10
9
9
9
9
9
9
8
8
3
8
8
3
8
8
7
7
2
7
7
2
7
7
6
6
3
6
6
3
6
6
5
5
2
5
5
2
5
5
4
4
4
4
4
4
4
4
3
3
3
3
3
3
3
3
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
S3-4
S3-3
1
1
3
S3-2
1
3
S3-1
ES-5
ES-4
ES-3
1
1
1
1
These two racks have only one
Unprotected Power Line with C13
sockets
CDH-6
1
19
2
ES-2
ES-1
1
1
18
3
17
2
16
3
15
2
CDH-5
1
CDH-4
1
14
1
CDH-10
CDH-9
1
1
13
3
12
2
11
3
10
2
CDH-3
1
CDH-7
1
1
3
11
1
CDH-2
1
9
CDH-8
12
10
OBS 5280V2-03
4
2
9
8
3
7
2
6
3
5
2
8
CDH-1
1
Jump Station
7
6
1
3
OBS 5280V2-02
4
2
5
3
OBS 5280V2-01
4
2
12
3
15
14
13
11
OBS 5280V2-06
4
16
2
10
1st 10GE port connected to TOR Switch #
2nd 10GE port connected to TOR Switch #
Device Function
3
4
2288H V5
2
4
8
3
2
6
4
4
2288H V5 HDFS
698.68 W
OBS Taishan 200
882.98 W
470 W
CE5855 BMC Switch
103 W
CE8850 Aggregate
Switch
965 W
Phase-1 Devices Count
2288H V5_8160_Computing - 20
CE5855-48T4S2Q-EI_BMC_Mgmt - 01
CE6865-48S8CQ-EI_Computing - 02
USG6655E(Firewall) - 01
Phase-2 Devices Count
2288H V5_8160_Computing - 30
2288H V5_8160_HDFS - 08
FusionStorage OBS - 18
CE5855-48T4S2Q-EI_BMC_Mgmt - 02
CE6865-48S8CQ-EI_Computing - 04
CE5855-48T4S2Q-EI_Storage -02
CE6865-48S8CQ-EI_Storage - 04
USG6655E(Firewall) - 02
CE8850-64CQ-EI - 02
USG6655E Firewall
7
2
2
2
6
5
3
OBS 5280V2-04
4
4
3
3
OBS 5280V2-07
4
1
2
2
1
Power Consumption Device Wise
661.08 W
8
OBS 5280V2-08
5
3
10
9
7
OBS 5280V2-05
4
3
iBMC port connected to BMC Switch #
2288H V5 Compute
2
9
Device Number
CE6865 TOR Switch
11
OBS 5280V2-09
Device Model
COM/STO TOR - CE6865 [ 3 ]
12
3
156 W

4.

Network Topology-Discovery(Phase1) & IOC(Phase2)
Phase2
Phase1
1. Connect the Phase1 TOR SW to Spine
2. Migrate the Phase1 FW to Phase2 FW
100GE
40GE
25GE
10GE
40GE
GE
Spine CE8850
100GE
100GE
100GE
CE6865
CE5855
CE5855
CE6865
2*10GE
CE5855
CE6865
CPU
CPU
GE
Bond
Sharing
2*10GE
30*2288H V5 Computing
2*10GE
8*2288H V5 HDFS
2*2*25GE
Bond
Sharing
18* FusionStorage OBS Node

5.

Network Topology-Phase1
Phase1
100GE
40GE
25GE
After Phase-2 Implementation
This firewall will be repurposed
10GE
GE
40GE
CE6865
CE5855
100GE
2*10GE

6.

Phase-1 Present Architecture

7.

Phase-1 Present Architecture Problems
Servers are connected to single TOR switch.
TOR switch does not have redundancy.
Network Throughput (All internal & external traffic going via firewall device).
Between servers to TOR Switches , TOR to TOR and TOR to firewall device network speed is
10GE.

8.

Phase-1 Expected Architecture
42
41
42
VLAN 220
41
40
40
39
39
38
38
37
37
36
36
35
35
34
34
34
33
33
33
32
32
32
31
31
31
30
30
30
29
29
29
28
28
28
27
27
27
26
26
26
25
25
25
24
24
24
23
23
23
22
22
22
21
21
20
20
19
19
21
20
CDH-6/192.168.40.17
192.168.30.17
19
18
17
16
15
18
CDH-5/192.168.40.16
192.168.30.16
CDH-4/192.168.40.15
192.168.30.15
CDH-3/192.168.40.14
192.168.30.14
14
13
12
11
10
CDH-2/192.168.40.13
192.168.30.13
9
8
7
CDH-1/192.168.40.12
6
5
Jump
Station/192.168.40.11
4
192.168.40.x
3
VLAN 221
2
192.168.30.12
192.168.30.11
B
M
C
I
P
S
V
L
A
N
2
2
0
17
16
15
41
39
TOR SWITCH-1/192.168.20.x
38
37
36
TOR SWITCH-2/192.168.20.x
ES-2/192.168.50.12
ES-1/192.168.50.11
35
18
192.168.30.23
192.168.30.22
B
M
C
192.168.30.21
I
P
S
14
13
12
11
10
CDH-10/192.168.40.21
CDH-9/192.168.40.20
192.168.30.20
9
8
7
6
5
40
VLAN 111
CDH-8/192.168.40.19
CDH-7/192.168.40.18
4
192.168.30.19
192.168.30.18
V
L
A
N
2
2
0
17
16
15
S3-4/192.168.50.19
192.168.30.30
S3-3/192.168.50.18
192.168.30.29
S3-2/192.168.50.17
192.168.30.28
14
13
12
11
10
S3-1/192.168.50.16
ES-5/192.168.50.15
192.168.30.27
8
7
6
5
ES-4/192.168.50.14
ES-3/192.168.50.13
B
M
C
I
P
S
V
L
A
N
192.168.30.26
2
2
0
9
4
192.168.50.x.
3
VLAN 222
3
192.168.40.x
VLAN
2 221
192.168.50.x.
1 222
VLAN
1
42
BMC SWITCH/192.168.30.x
192.168.30.25
192.168.30.24
100GE
40GE
10GE
2
1
1GE
Rack-10
Rack-11
Rack Diagram
Rack-12

9.

Phase-1 Architecture After Changes
All servers should be connected to 2 different TOR switches.
2 TOR switches connect to each other.
VLAN should be span across all TOR Switches.
TOR Switches will connect IOC phase Aggregate Switches.
After connectivity Between TOR Switches and TOR to Aggregate Switch network bandwidth is
100GE.
All internal traffic going between TOR Switches and external traffic going via Aggregate and
firewall devices.
After all configuration firewall device will be remove from Discovery Phase.

10.

Phase-1 Internal VLANS
VLAN-111 => Network Devices(TOR Switches , Firewall)
VLAN-220 => BMC Management network
VLAN-221 => CDH, Docker Swarm (192.168.40.x)
VLAN-222 => Elasticsearch , S3 Storage (192.168.50.x)

11.

Phase-1 External VLANS
VLAN-111 => Network Devices(TOR Switches , Firewall)
VLAN-220 => BMC Management network
VLAN-221 => CDH, Docker Swarm (192.168.40.x)
VLAN-222 => Elasticsearch , S3 Storage (192.168.50.x)

12.

Network Topology-Phase2
100GE
40GE
25GE
40GE
10GE
GE
Spine CE8850
100GE
100GE
100GE
CE5855
CE5855
CE6865
CE6865
CPU
CPU
GE
Bond
Sharing
2*10GE
2*10GE
2*2*25GE
Bond
Sharing
18* FusionStorage OBS Node
30*2288H V5 Computing
8*2288H V5 HDFS

13.

Phase-2 Internal VLAN Requirements
Phase-1 same VLANS need to configure across all the Switches
S3 service separate VLAN (Based on Huawei Implementation Team)
S3 servers BMC Management.
Elasticsearch Service
Oracle Database
CDH and Docker Swarm

14.

Phase-2 External VLAN Requirements
Phase-1 same VLANS need to configure across all the Switches.
Servers BMC Management
S3 Devices BMC Management.
Elasticsearch Service
Oracle Database
CDH and Docker Swarm

15.

Devices Checklist
After Phase-1&2 implementation we need confirmation about spare devices working
condition.