Version 0.8 2007 Features PEX 8616 General Features o 16-lane, 4-port PCIe Gen2 switch - Integrated 5.0 GT/s SerDes o 19 x 19mm2, 324-pin FCBGA package o Typical Power: < 3.0 Watts PEX 8616 Key Features o Standards Compliant - PCI Express Base Specification, r2.0 (backwards compatible w/ PCIe r1.0a/1.1) - PCI Power Management Spec, r1.2 - Microsoft Vista Compliant - Supports Access Control Services - Dynamic link-width control - Dynamic SerDes speed control o High Performance - Non-blocking switch fabric - Full line rate on all ports - Packet Cut-Thru with 150ns max packet latency (x4 to x4) - 2KB Max Payload Size - Read Pacing (bandwidth throttling) - Dual-Cast o Flexible Configuration - Ports configurable as x1, x2, x4, x8 - Registers configurable with strapping pins, EEPROM, I2C, or host software - Lane and polarity reversal - Compatible with PCIe 1.0a PM o Dual-Host & Fail-Over Support - Configurable Non-Transparent port - Moveable upstream port - Crosslink port capability o Quality of Service (QoS) - Eight traffic classes per port - Weighted round-robin source port arbitration o Reliability, Availability, Serviceability - 2 Hot Plug Ports with native HP Signals - All ports hot plug capable thru I2C (Hot Plug Controller on every port) - ECRC and Poison bit support - Data Path parity - Memory (RAM) Error Correction - INTA# and FATAL_ERR# signals - Advanced Error Reporting - Port Status bits and GPIO available - Per port error diagnostics - Performance Monitoring • Per port payload & header counters - JTAG AC/DC boundary scan PEX 8616 PCIe Gen2, 5.0GT/s 16-lane, 4-port Switch The ExpressLaneTM PEX 8616 device offers PCI Express switching capability enabling users to add scalable high bandwidth, non-blocking interconnection to a wide variety of applications including workstations, storage systems, communications platforms, embedded systems, and intelligent I/O modules. The PEX 8616 is well suited for fan-out, aggregation, and peer-to-peer applications. High Performance & Low Packet Latency The PEX 8616 architecture supports packet cut-thru with a maximum latency of 150ns (x4 to x4). This, combined with large packet memory and non-blocking internal switch architecture, provides full line rate on all ports for performance-hungry applications such as servers and switch fabrics. The low latency enables applications to achieve high throughput and performance. In addition to low latency, the device supports a max payload size of 2048 bytes, enabling the user to achieve even higher throughput. Data Integrity The PEX 8616 provides end-to-end CRC (ECRC) protection and Poison bit support to enable designs that require end-to-end data integrity. PLX also supports data path parity and memory (RAM) error correction as packets pass through the switch. Flexible Register & Port Configuration The PEX 8616’s 4 ports can be configured to lane widths of x1, x2, x4, or x8. Flexible buffer allocation, along with the device's flexible packet flow control, maximizes throughput for applications where more traffic flows in the downstream, rather than upstream, direction. Any port can be designated as the upstream port, which can be changed dynamically. The PEX 8616 also provides several ways to x8 x4 configure its registers. The device can be configured through PEX 8616 strapping pins, I2C PEX 8616 interface, host software, or an optional serial EEPROM. This allows for x4 x4 x4 x4 x4 easy debug during the development phase, x8 x4 performance monitoring NT during the operation phase, and driver or PEX 8616 PEX 8616 software upgrade. Figure 1 shows some of the PEX 8616’s common port x8 x4 x2 x2 configurations. Figure 1. Common Port Configurations Preliminary - PLX Confidential Dual-Host & Failover Support The PEX 8616 product supports a Non-Transparent (NT) Port, which enables the implementation of multihost systems and intelligent I/O modules in storage, communications, and blade server applications. The NT port allows systems to isolate host memory domains by presenting the processor subsystem as an endpoint rather than another memory system. Base address registers are used to translate addresses; doorbell registers are used to send interrupts between the address domains; and Intelligent I/O Adaptor scratchpad registers Non-Transparent Port (accessible by I/O CPU both CPUs) Secondary Secondary Primary I/O Primary allow interHost Host Host Host processor CPU CPU NT communication Blade Blade PEX 8616 (see Figure 2). In a two-port configuration (as in I/O Figure 1), the PEX NT 8616 can serve as an I/O PEX 8632 NT buffer, isolating two host domains via I/O two x8 links. Figure 2. Non-Transparent Port Dual Cast The PEX 8616 supports Dual Cast, a feature which allows for the copying of data (e.g. packets) from one ingress port to two egress ports allowing for higher performance in dual-graphics, storage, security, and redundant applications. Read Pacing The Read Pacing feature allows users to throttle the amount of read requests being made by downstream devices. When a downstream device requests several long reads back-to-back, the Root Complex gets tied up in serving this downstream port. If this port has a narrow link and is therefore slow in receiving these read packets from the Root Complex, then other downstream ports may become starved – thus, impacting performance. The Read Pacing feature enhances performances by allowing for the adequate servicing of all downstream devices. Hot Plug for High Availability Hot plug capability allows users to replace hardware modules and perform maintenance without powering down the system. The PEX 8616 hot plug capability feature makes it suitable for High Availability (HA) applications. Two downstream ports include a Standard Hot Plug Controller. If the PEX 8616 is used in an application where one or more of its downstream ports connect to PCI Express slots, each port’s Hot Plug Controller can be used to manage the hot-plug event of its associated slot. Every port on the PEX 8616 is equipped with a hot-plug control/status register to support hot-plug capability through external logic via the I2C interface. SerDes Power and Signal Management The PEX 8616 supports software control of the SerDes outputs to allow optimization of power and signal strength in a system. The PLX SerDes implementation supports four levels of power – off, low, typical, and high. The SerDes block also supports loop-back modes and advanced reporting of error conditions, which enables efficient management of the entire system. Interoperability The PEX 8616 is designed to be fully compliant with the PCI Express Base Specification r2.0, and is backwards compatible to PCI Express Base Specification r1.1 and r1.0a. Additionally, it supports auto-negotiation, lane reversal, and polarity reversal. Furthermore, the PEX 8616 is designed for Microsoft Vista compliance. All PLX switches undergo thorough interoperability testing in PLX’s Interoperability Lab and compliance testing at the PCI-SIG plug-fest. Applications & Usage Models Suitable for host-centric as well as peer-to-peer traffic patterns, the PEX 8616 can be configured for a broad range of form factors and applications. Host Centric Fan-out The PEX 8616, with its symmetric or asymmetric lane configuration capability, allows user-specific tuning to a variety of host-centric applications. Figure 3 shows a typical workstation design where the root complex provides a PCI Express link that needs to be expanded to a larger number of smaller ports for a variety of I/O functions. In this example, the PEX 8616 has a 4-lane upstream port and three downstream ports using x4 links. The PEX 8616 can also be used to create PCIe Gen1 (2.5 Gbps) ports. The PEX 8616 is backwards compatible with PCIe Gen1 devices. Therefore, the PEX 8616 enables a Gen 2 native Chip Set to fan-out to Gen 1 endpoints. In Figure 3, the PCIe slots connected to the PEX 8616’s downstream ports can be populated with either PCIe Gen1 or PCIe Gen 2 devices. Conversely, the PEX 8616 can also be used to create Gen 2 ports on a Gen 1 native Chip Set in the same fashion. Preliminary - PLX Confidential CPU CPU CPU CPU Quad-Port FC HBA FC x4 FC Cont. FC Memory Chipset FC Endpoint FC Cont. x4 x16 PEX 8616 x4 x8 FC x16 PEX 8616 Endpoint x4 x4 Figure 5. Quad-Port Fibre-Channel HBA x4 PCIe Gen1 or PCIe Gen2 slots Figure 3. Fan-in/out Usage Network Interface Cards The PEX 8616 can also be utilized in communications applications such as Network Interface Cards (NICs). NICs, like the one shown in Figure 4, can utilize the PEX 8616 for its fan-out capabilities. In the example below, the PEX 8616 is being used on a Dual-port 10Gigabit Ethernet (GE) NIC card. The PEX 8616 utilizes a x8 link to connect to the host and two x4 downstream links to fan-out to the 10GE ports. The peer-to-peer communication feature of the PEX 8616 allows the endpoints to communicate with each other without any intervention or management by the host. Failover Storage Systems The PEX 8616’s Dual Cast feature proves to be very useful in storage systems. In the example shown in Figure 6, the Dual Cast feature enables the PEX 8616 to copy data coming from the host to two downstream ports (see yellow traffic patterns) in one transaction as opposed to having to execute two separate transactions to send data to the redundant chassis. By offloading the task of backing up data onto the secondary system, processor and system performance is enhanced. The PEX 8616’s NT port is used to isolate the backup system from the primary system. CPU CPU CPU CPU CPU CPU CPU CPU Memory Chipset Dual-Port NIC 10 GE MAC/PHY Chipset x4 x4 10 GE x4 MAC/PHY x4 NT PEX 8616 x4 x4 x8 PEX 8616 Host Bus Adapters The PEX 8616 is well suited for Host Bus Adapter cards (HBAs) as well. Figure 5 shows a Quad-Port Fibre Channel HBAs utilizing the PEX 8616 to allow connectivity to two Fibre Channel (FC) controllers. The PEX 8616 uses a x8 link on the upstream port and two x4 downstream links to fan-out to the FC Controllers Each FC controller supports two FC ports. x4 NT x4 Figure 4. 10GE NIC Fan-Out Memory x4 PEX 8616 PEX 8616 Backup System FC Control x4 FC Control 8 Disk Chassis x4 NT PEX 8616 x4 FC Control x4 FC Control 8 Disk Chassis Figure 6. Dual Cast in Storage Systems Preliminary - PLX Confidential Software Usage Model Development Tools From a system model viewpoint, each PCI Express port is a virtual PCI to PCI bridge device and has its own set of PCI Express configuration registers. It is through the upstream port that the BIOS or host can configure the other ports using standard PCI enumeration. The virtual PCI to PCI bridges within the PEX 8616 are compliant to the PCI and PCI Express system models. The Configuration Space Registers (CSRs) in a virtual primary/secondary PCI to PCI bridge are accessible by type 0 configuration cycles through the virtual primary bus interface (matching bus number, device number, and function number). PLX offers hardware and software tools to enable rapid customer design activity. These tools consist of a hardware module (PEX 8616 RDK), hardware documentation (available at www.plxtech.com), and a Software Development Kit (also available at www.plxtech.com). Interrupt Sources/Events The PEX 8616 switch supports the INTx interrupt message type (compatible with PCI 2.3 Interrupt signals) or Message Signaled Interrupts (MSI) when enabled. Interrupts/messages are generated by PEX 8616 for hot plug events, doorbell interrupts, baseline error reporting, and advanced error reporting. ExpressLane PEX 8616 RDK The PEX 8616 RDK is a hardware module containing the PEX 8616 which plugs right into your system. The PEX 8616 RDK can be used to test and validate customer software, or used as an evaluation vehicle for PEX 8616 features and benefits. The PEX 8616 RDK provides everything that a user needs to get their hardware and software development started. For more information, please refer to the PEX 8616 RDK Product Brief. Software Development Kit (SDK) PLX’s Software Development Kit is available for download at www.plxtech.com/sdk. The software development kit includes drivers, source code, and GUI interfaces to aid in configuring and debugging the PEX 8616. For more information, please refer to the PEX 8616 RDK Product Brief. Product Ordering Information PLX Technology, Inc. 870 Maude Ave. Sunnyvale, CA 94085 USA Tel: 1-800-759-3735 Tel: 1-408-774-9060 Fax: 1-408-774-2169 Email: [email protected] Web Site: www.plxtech.com Part Number Description PEX8616-AA50BC G PEX8616-AA RDK 16-Lane, 4-Port PCI Express Switch, Pb-Free (19x19mm2) PEX 8616 Rapid Development Kit Please visit the PLX Web site at http://www.plxtech.com or contact PLX sales at 408-774-9060 for sampling. © 2007 PLX Technology, Inc. All rights reserved. PLX and the PLX logo are registered trademarks of PLX Technology, Inc. ExpressLane is a trademark of PLX Technology, Inc., which may be registered in some jurisdiction. All other product names that appear in this material are for identification purposes only and are acknowledged to be trademarks or registered trademarks of their respective companies. Information supplied by PLX is believed to be accurate and reliable, but PLX Technology, Inc. assumes no responsibility for any errors that may appear in this material. PLX Technology, Inc. reserves the right, without notice, to make changes in product design or specification. PEX8616-SIL-PB-P1-0.8 09/07 Preliminary - PLX Confidential