VERIZON CLOUD COMPUTE AND THE AMD SEAMICRO 15000 Paul Curtis Chief Architect Verizon Cloud Compute APU13 Collaboration • Unbelievable amount of support from Seamicro 2 Verizon cloud development goals • Very few different hardware components • Consistent predictable performance • Secure • High performance • Highly available • No modification to customer applications • No special purpose hardware 3 Verizon Cloud Differentiation • Value for Performance – User defined availability and performance – User defined resources • Reserved Performance – Network, Storage and Compute • Workload Simplicity – Seamless integration with other deployments – Single point of control • Security ‒ Market leading security capabilities ‒ Embedded into every aspect of platform • Continuum of Services ‒ Bridging private, public and hybrid clouds ‒ Allow the blending with colocation, managed services, networking 4 SM15000 SYSTEM 10 Rack Units, draws 3-3.5 KW ! Compute – Up to 512 Opteron, Xeon or Atom cores in 10 RU – 2,048 cores in a rack – Up to 64GB DRAM/socket = 4 terabytes/system ! Networking – 10 Gbps half duplex bandwidth to each CPU socket – 16 x 10GbE Line Rate uplinks to the network ! Storage – Up to 1,408 disks: HDD or SSD – Up to 128 Terabytes of internal SSD storage – Up to 5.3 Petabytes of storage ! Fabric – 1.28 Tbps Freedom Supercompute Fabric ! Software – Off the shelf OS, Hypervisors 5 Hardware architecture • There are only three hardware component types. This simplifies maintenance – Arista 7508 a 384 port x 10Ge non-blocking L2 switch – AMD Seamicro SM15000 – SSDs • Network connections 6 Hardware diagram Juniper MX960 for external connectivity Arista 7508 4x10Gb links from arista to each chassis Up to 90 AMD Seamicro SM15000s 7 Verizon’s use of seamicro chassis • 160 GB of external bandwidth (network and storage) • 54 Server cards for customer loads • 2 Server cards for Verizon orchestration • 8 Server cards for storage services • ~1000000 IOPs • 96 T usable SSD storage 8 Combine Hardware and Verizon software to get • A flat layer 2 ethernet switch – ~12000 port 1 Gb/sec – ~1500000 vlans – 8.5M mac address table entries – 11.5M traffic flows. – Software configurable • A storage array – 90M IOPs – 8.6 PB of SSD storage • Scalable router firewall 1Gb- 400Gb/sec • Scalable load balancers 1Gb-400Gb/sec • Configurable IO performance 9 Network Packet flow Hypervisor presents nic of specified speed to VM. Back pressure applied by hypervisor VM Nic Queue Hypervisor text Nic Queue NPU VM Nic Queue Hypervisor Hypervisor fairly mixes flows from different VMs Limited to max Nic speed text Nic Queue NPU Shaped to max speed of receiving nic Queue Policer Queue Layer 2 Switching Shaped to max speed of receiving Nic Random packet drop back pressure form destinatio n queue Layer 2 Switching 10 G NIC 10 G NIC Queue Arista Switch Prioritized queue. 10 Networking Layer 2 • Hypervisor – Shapes egress traffic • NPU – Provides true layer 2 ethernet switching – Polices ingress flows – Shapes egress flows • Arista 7508 – Lots of bandwidth • Remote congestion control – Switch learns speeds of remote flows – Switch performs remote drop if destination is congested • Hardware based security – Each customer network is on its own vlan • Software configurable 11 Data I/O Stack VM HV Paravirt (xvdb) o Storage VLAN AoE Initiator Ethernet NPU Arista S S N NPU o Ethernet AoE Target AIO ZFS Block AoE Initiator Ethernet Ethernet AoE Target Block Device(s) SCARD 12 Storage • Hypervisor – Shapes disk traffic (IOPs and Bandwidth) – Participates in disk replication • AoE – Storage over layer 2 ethernet – Allows storage targets to be any where in world – Shared volumes • Replication • NPU – Shapes read and write bandwidth • Storage Service – Snap shots – Raid • Storage card – AoE target 13 Networking Layer 3+ Layer 3 and above network services just work since they are all based on layer 2 networking. • • • • • Soft routers Load balancers Public IP (No Nat) Tunnels Wan optimizers 14 Inter-data center features • Single user interface • Networks can span multiple data centers • Replicated disks can span multiple data centers • Taking advantage of being part of a network company 15 Availability • No single point of failure for network traffic – “Bonded nics” – “Bonded NPUs” – Fabric reroutes itself – Multiple paths through arista switches • No single point of failure for replicated storage – Raid 1 on SSDs – Multiple storage servers – Option to have replicated volumes span data centers 16 Security • Physical security • DDOS • Network security – Customer traffic on independent VLANs – Untrusted entities (Hypervisors) firewalled from rest of system • Storage security – Each volume on a separate vlan – Storage vlans firewalled (only AoE traffic, no target to target traffic) • Management software – Audit logs – Security alerts 17 Possible Applications • Move a current three tier app with your choice of soft router/firewall/load balancer into the cloud • Bridge a network from your data center to one in the cloud • Move XEN and VMWare VMs into the cloud without modification • Write a clustered app using shared storage • Configure an applications performance so that you know it won’t fall over when it is 3:00 in the afternoon and the cloud gets busy • Write and test a new L3 protocol • Voice • Storage arrays • Network devices 18