Version 1.2 2007 Features PEX 8512 General Features o 12-lane PCI Express switch - Integrated SerDes o Up to five ports o 23mm x 23mm, 376-ball PBGA package o Typical Power: 2.0 Watts PEX 8512 Key Features o Standards Compliant - PCI Express Base Specification, r1.1 - Standard SHPC Specification, r1.1 (hot-plug) o High Performance - Non-blocking internal architecture - Full line rate on all ports - Cut-Thru latency: 150ns o Non-Transparent Bridging - Configurable Non-Transparent port for Multi-Host or Intelligent I/O Support o Flexible Configuration - Up to Five configurable ports (x1, x2 and x4) - Configurable with strapping pins, EEPROM, I2C, or Host software - Lane and polarity reversal o PCI Express Power Management - Link power management states: L0, L0s, L1, L2/L3 Ready, L2 and L3 - Device states: D0 and D3hot - Vaux, WAKE#, Beacon support o Spread Spectrum Clock - Dual clock domain o Quality of Service (QoS) - Two Virtual Channels per port - Eight Traffic Classes per port - Fixed and Round-Robin Virtual Channel Port Arbitration o Reliability, Availability, Serviceability - Three Standard Hot-Plug Controllers - Upstream port as hot-plug client - Transaction Layer end-to-end CRC - Poison bit - Advanced Error Reporting - Lane Status bits and GPO available - Per port error diagnostics • Bad DLLPs • Bad TLPs • CRC errors - JTAG boundary scan - Fatal Error out-of-band signal option PEX 8512 Flexible & Versatile PCI Express® Switch Multi-purpose, Feature Rich ExpressLane™ PCI Express Switch The ExpressLane PEX 8512 device offers PCI Express switching capability enabling users to add scalable high bandwidth, non-blocking interconnection to a wide variety of applications including servers, storage systems, communications platforms, blade servers, and embedded-control products. The PEX 8512 is well suited for fan-out, aggregation, peer-topeer, and intelligent I/O module applications. Highly Flexible Port Configurations The PEX 8512 offers 12 lanes which can be distributed into three ports or five ports. In the case of three ports, the ports are configured for symmetric (each port having the same lane width). The PEX 8512 features a flexible central packet memory that allocates a memory buffer for each port as required by the application or endpoint. This buffer allocation along with the device's flexible packet flow control minimizes bottlenecks when the upstream and aggregated downstream bandwidths do not match. Any of the ports can be designated as the upstream port, which can be changed dynamically. End-to-end Packet Integrity The PEX 8512 provides end-to-end CRC protection (ECRC) and Poison bit support to enable designs that require guaranteed error-free packets. These features are optional in the PCI Express specification, but PLX provides them across its entire ExpressLane switch product line. Non-Transparent “Bridging” in a PCI Express Switch The PEX 8512 supports full non-transparent bridging (NTB) functionality to allow implementation of multi-host systems and intelligent I/O modules in applications such as communications, storage, and blade servers. To ensure quick product migration, the non-transparency features are implemented in the same fashion as in standard PCI applications. Non-transparent bridges allow systems to isolate memory domains by presenting the processor subsystem as an endpoint, rather than another memory system. Base address registers are used to translate addresses; doorbell registers are used to send interrupts between the address domains; and scratchpad registers are accessible from both address domains to allow inter-processor communication. Two Virtual Channels The ExpressLane PEX 8512 switch supports two Virtual Channels (VCs) and eight Traffic Classes (TCs). The mapping of Traffic Classes to port-specific Virtual Channels allows for different mappings for different ports. In addition, the devices offer user-selectable Virtual Channel arbitration algorithms to enable users to fine tune the Quality of Service (QoS) required for a specific application. Low Power with Granular SerDes Control The PEX 8512 provides low power capability that is fully compliant with the PCI Express power management specification. In addition, the SerDes physical links can be turned off when unused for even lower power. strength in a system. The PLX SerDes implementation supports four levels of power – off, low, typical, and high. The SerDes block also supports loop-back modes and advanced reporting of error conditions, which enables efficient debug and management of the entire system. Flexible Port Width Configuration The lane width for each port can be individually configured through auto-negotiation, hardware strapping, upstream software configuration, or through an optional EEPROM. The PEX 8512 supports two port configurations as shown in Figure 1. Depending on the application, the PEX 8512 can be configured to have Three ports each with x4 link widths or Five ports one port with x4 link width and four ports with x2 link widths. x4 x4 PEX 8512 PEX 8512 x4 x4 x2 x2 x2 x2 Flexible Virtual Channel Arbitration The ExpressLane PEX 8512 switch supports hardware fixed and Round Robin arbitration schemes for two virtual channels on each port. This allows for the fine tuning of Quality of Service for efficient use of packet buffers and system bandwidth. Applications Suitable for host-centric as well as intelligent I/O applications ExpressLane PEX 8512 can be configured for a wide variety of form factors and applications. Host-Centric Fan-out The ExpressLane PEX 8512 switch, with its symmetric lane configuration capability, allows user specific tuning to a variety of host-centric as well as peer-to-peer applications. x4 PEX 8512 x2 x2 x4 Figure 1. Port Configurations Hot-Plug for High Availability Hot plug capability allows users to replace hardware modules and perform maintenance without powering down the system. The PEX 8512 hot-plug capability and Advanced Error Reporting features makes it suitable for High Availability (HA) applications. Three downstream ports include a Standard Hot-Plug Controller. If the PEX 8512 is used in an application where one or more of its downstream ports connect to PCI Express slots, the port’s Hot-Plug Controller can be used to manage the hot-plug event of its associated slot. Furthermore, its upstream port is a hot-plug client, allowing it to be used on hot-pluggable adapter cards, backplanes, and fabric modules. Fully Compliant Power Management For applications that require power management, the PEX 8512 device supports both link (L0, L0s, L1, L2/L3 Ready, L2 and L3) and device (D0 and D3hot) power management states, in compliance with the PCI Express power management specification. SerDes Power and Signal Management The ExpressLane PEX 8512 supports software control of the SerDes outputs to allow optimization of power and signal Figure 2. Fan-in/out Usage Figure 2 shows a typical server-based design, where the root complex provides a PCI Express link that needs to be fanned into a larger number of ports for a variety of I/O functions, each with different bandwidth requirements. In this example, the PEX 8512 would typically have a 4-lane upstream port, and two downstream ports. The downstream ports are configured with x4 link widths. Adapter Card Aggregation The number and variety of PCI Express native-mode devices is growing quickly. As these devices become mainstream, it will be necessary to create multifunction and multi-port adapter cards with PCI Express capability. The PEX 8512 can be used to create an adapter or mezzanine card that aggregates the PCI Express devices into a single port that can be plugged into a backplane or motherboard. Figure 3 shows the PEX 8512 in this application. x4 Quad GE PEX 8512 x4 Quad GE x4 Figure 3. Aggregation Adapter Card The redundancy of the host and the fabric can be achieved through many possible configurations using NTB function of PEX 8512. In the configuration shown below the host 1 controls the switch 1 and associated I/Os and the host 2 controls the switch 2 and associated I/Os. The hosts and switches are isolated using NTB functionality of PEX 8512 on the host boards. If one of the hosts fails the surviving host can remove the failing host from the configuration while controlling both the switches and all I/Os. Similarly, if one of the switches fails the host associated with that switch can send control messages to its I/Os through the surviving switch using NTB function. The adapter card in Figure 3 can be transparent, in which case the PCI Express I/O devices are just standard I/O endpoints such as Ethernet or Fibre Channel. Or the PEX 8512 can provide a non-transparent port to the system (via the card’s edge connector). In this case, one of the PCI Express devices can be a CPU or other “intelligent” device with onchip processing capability – thus needing address domain isolation from the rest of the system. This approach is commonly used in RAID controllers. Intelligent Adapter Card The PEX 8512 supports the non-transparency feature. Figure 4 illustrates a host system using an intelligent adapter card. Intelligent I/O Adaptor Non-Transparent Port x2 Figure 5. Dual Host/Fabric Mode I/O CPU x2 Primary Primary Host Host I/O x4 CPU Blade NT I/O x2 PEX 8512 I/O I/O x2 Expansion Application The PEX 8512 can also be utilized to expand the number of PCI Express devices in a given system. Figure 6 shows how the PEX 8512 can be used to double the number of ports available from the Root Complex. Each port in the PEX 8512 is configured with a x4 link width. The Peer-to-peer communication feature of the PEX 8512 allows the PCI Express I/O devices to communicate with each other without any centralized control. PEX 8533 CPU I/O Figure 4. Intelligent Adapter Usage In this figure, the CPU on the adapter card is isolated from the host CPU. The PEX 8512 non-transparent port allows the two CPUs to be isolated but communicate with each other through various registers that are designed in the PEX 8512 for that purpose. The host CPU can dynamically re-assign both the upstream port and the non-transparent port of PEX 8512 allowing the system to be reconfigured. Root Complex x4 x4 PEX 8512 PEX 8512 x4 x2 Dual GE x2 Dual GE x2 Dual GE x2 x4 Dual GE FC Dual Host/Fabric Model The ExpressLane PEX 8512 supports applications requiring dual host, host failover, and load-sharing applications through the non-transparency feature. Figure 5 illustrates a dual host system with dual switch fabric in dual-star configuration. PCI Express Slot Figure 6. Expansion Application Software Usage Model Interrupt Sources/Events From a system model viewpoint, each PCI Express port is a virtual PCI to PCI bridge device and has its own set of PCI Express configuration registers. It is through the upstream port that the BIOS or host can configure the other ports using standard PCI enumeration. The virtual PCI-to-PCI bridges within the PEX 8512 are compliant to the PCI and PCI Express system models. The Configuration Space Registers (CSRs) in a virtual primary/secondary PCI-to-PCI bridge are accessible by type 0 configuration cycles through the virtual primary bus interface (matching bus number, device number, and function number). The PEX 8512 supports the INTx interrupt message type (compatible with PCI 2.3 Interrupt signals) or Message Signaled Interrupts (MSI) when enabled. Interrupts/messages are generated by PEX 8512 for hot-plug events, doorbell interrupts, baseline error reporting, and advanced error reporting. Development Tools RDK PLX offers hardware and software tools to enable rapid customer design activity. These tools consist of a PEX 8512 Rapid Development Kit (RDK), hardware documentation, and a Software Development Kit (SDK). The RDK hardware module includes the PEX 8512 with one x4 (card-edge slot) port and two additional x4 ports (see Figure 7). The RDK is available with x4 card edge connector and adapters for x1 edge connectors are available to plug the RDK into smaller slots. The PEX 8512RDK hardware module can be installed in a motherboard and be used as a riser card. The PEX 8512RDK can be used to test and validate customer software. Additionally, it can be used as an evaluation vehicle for PEX 8512 features and benefits. PORT 1 - X4 Lane Status LEDs Manual Reset PORT 2 – X4 Hot Plug Ckt. Non-Iso DIP switch for cfg. Con vert er 12V-to-1V JTAG Header I2C PEX 8512 8512 EEPROM RefClk Reset Ckt. (Upstream Port) PERST# x4 SDK The SDK tool set includes: - Linux & Windows drivers - C/C++ Source code, Objects, libraries - User’s Guides & Application examples Figure 7. PEX 8512RDK Windows/Linux Apps User Space PEX API Kernel Space Hardware PCIe Driver NT PnP Driver PCI Express Interface Product Ordering Information PLX Technology, Inc. 870 W. Maude Ave. Sunnyvale, CA 94085 USA Tel: 1-800-759-3735 Tel: 1-408-774-9060 Fax: 1-408-774-2169 Email: [email protected] Web: www.plxtech.com Part Number Description PEX8512-AC25BI PEX8512-AC25BI G PEX 8512RDK 12 Lane, 5 Port PCIe Switch, 376-ball PBGA 23x23mm pkg 12 Lane, 5 Port PCIe Switch, 376-ball PBGA 23x23mm pkg, Pb-free PEX 8512 Rapid Development Kit with x4 Connector Breakout Board w/ x16 edge connector for additional fan-out to two slots (x4, x2, x2). Breakout Board-422 Please contact PLX sales at 408-774-9060 for sampling. © 2007 PLX Technology, Inc. All rights reserved. PLX and the PLX logo are registered trademarks of PLX Technology, Inc. ExpressLane is a trademark of PLX Technology, Inc., which may be registered in some jurisdiction. All other product names that appear in this material are for identification purposes only and are acknowledged to be trademarks or registered trademarks of their respective companies. Information supplied by PLX is believed to be accurate and reliable, but PLX Technology, Inc. assumes no responsibility for any errors that may appear in this material. PLX Technology, Inc. reserves the right, without notice, to make changes in product design or specification. PEX8512-SIL-PB- 1.2 05/07