Version 1.5 2007 PEX 8548 Features PEX 8548 General Features o 48-lane PCI Express switch - Integrated SerDes o Up to nine configurable ports (x1, x2, x4, x8, x16) o 37.5mm x 37.5mm, 736-ball PBGA package o Typical Power: 4.9 Watts PEX 8548 Key Features o Standard Compliant - PCI Express Base Specification, r1.1 o High Performance - Non-blocking switch fabric - Full line rate on all ports - Packet Cut-Thru with 110ns max packet latency (x16 to x16) o Flexible Configuration - Nine highly flexible & configurable ports (x1, x2, x4, x8, or x16) - Configurable with strapping pins, EEPROM, I2C, or Host software - Lane and polarity reversal o PCI Express Power Management - Link power management states: L0, L0s, L1, L2/L3 Ready, and L3 - Device states: D0 and D3hot o Quality of Service (QoS) - One Virtual Channel per port - Eight Traffic Classes per port - Weighted Round-Robin Ingress Port Arbitration o Reliability, Availability, Serviceability - 3 Standard Hot-Plug Controllers - Upstream port as hot-plug client - Transaction Layer end-to-end CRC - Poison bit - INTA# interrupt signal - Fatal Error (FATAL_ERR#) signal (legacy SERR equivalent) - PCIe baseline error reporting - Advanced Error Reporting - Port Status bits and GPO available - Per port error diagnostics • Bad DLLPs • Bad TLPs • CRC errors - JTAG boundary scan High-Performance 48-lane, 9-port PCIe Switch Multi-purpose, High Performance ExpressLane™ Switch The ExpressLane PEX 8548 device offers PCI Express switching capability enabling users to add scalable high bandwidth, non-blocking interconnection to a wide variety of applications including servers, storage systems, communications platforms, blade servers, and embedded-control products. The PEX 8548 is well suited for fan-out, aggregation, dualgraphics, peer-to-peer, and fabric backplane applications. Highly Flexible Port Configurations The PEX 8548 offers highly configurable ports. There are a maximum of 9 ports that can be configured to any legal width from x1 to x16, in any combination to support your specific bandwidth needs. The ports can be configured for symmetric (each port having the same lane width and traffic load) or asymmetric (ports having different lane widths) traffic. In the event of asymmetric traffic, the PEX 8548 features a flexible central packet memory that allocates a memory buffer for each port as required by the application or endpoint. This buffer allocation along with the device's flexible packet flow control minimizes bottlenecks when the upstream and aggregated downstream bandwidths do not match (are asymmetric). Any of the ports can be designated as the upstream port, which can be changed dynamically. High Performance The PEX 8548 architecture supports packet cut-thru with a max latency of 110ns (x16 to x16). This, combined with large packet memory (1024 byte maximum payload size) and non-blocking internal switch architecture, provide full line rate on all ports for performance-hungry applications such as storage servers or storage switch fabrics. End-to-end Packet Integrity The PEX 8548 provides end-to-end CRC protection (ECRC) and Poison bit support to enable designs that require end-to-end data integrity. These features are optional in the PCI Express specification, but PLX provides them across its entire ExpressLane switch product line. Configuration Flexibility The PEX 8548 provides several ways to configure its operations. The device can be configured through strapping pins, I2C interface, CPU configuration cycles, or an optional serial EEPROM. This allows for easy debug during the development phase, performance monitoring during the operation phase, and driver or software upgrade. Interoperability The PEX 8548 is designed to be fully compliant with the PCI Express Base Specification r1.1. Additionally, it supports auto-negotiation, lane reversal, and polarity reversal. The PEX 8548 also undergoes thorough interoperability testing in PLX’s Interoperability Lab. Low Power with Granular SerDes Control The PEX 8548 provides low power capability that is fully compliant with the PCI Express power management specification. In addition, the SerDes physical links can be turned off when unused for even lower power. Flexible Port Width Configuration The width of each port can be individually configured through auto-negotiation, hardware strapping, host software configuration, I2C interface, or through an optional EEPROM. The PEX 8548 supports a large number of port configurations (see Figure 1). For example, if you are using the PEX 8548 in a fan-out application, you may configure the upstream port as a x16 and the downstream as eight x4 ports; three x8 ports & two x4 ports; or various combinations, as long as you don’t run out of lanes (48) or ports (9). For a dual-graphics application, the device can be configured as three x16 ports. x16 x16 PEX 8548 PEX 8548 advanced error reporting features make it suitable for High Availability (HA) applications. Three downstream ports include a Standard Hot Plug Controller. If the PEX 8548 is used in an application where one or more of its downstream ports connect to PCI Express slots, each port’s Hot Plug Controller can be used to manage the hot-plug event of its associated slot. Furthermore, its upstream port is a hot-plug client, allowing it to be used on hot-pluggable adapter cards, backplanes, and fabric modules. Fully Compliant Power Management For applications that require power management, the PEX 8548 device supports both link (L0, L0s, L1, L2/L3 Ready, and L3) and device (D0 and D3hot) power management states, in compliance with the PCI Express power management specification. SerDes Power and Signal Management The PEX 8548 supports software control of the SerDes outputs to allow optimization of power and signal strength in a system. The PLX SerDes implementation supports four levels of power – off, low, typical, and high. The SerDes block also supports loop-back modes and advanced reporting of error conditions, which enables efficient debug and management of the entire system. Applications x8 x8 x4 x4 x4 x4 x8 x8 x8 x4 x4 x16 x8 PEX 8548 all x8 PEX 8548 x16 Suitable for host-centric as well as peer-to-peer traffic patterns, the PEX 8548 can be configured for a wide variety of form factors and applications. Host Centric Fan-out The PEX 8548, with its symmetric or asymmetric lane configuration capability, allows user-specific tuning to a variety of host-centric applications. x16 CPU Figure 1. Common Port Configurations Low Packet Latency The PEX 8548 supports packet cut-thru with a maximum packet latency of 110ns between symmetric x16 ingress and egress ports. The low latency enables applications to achieve high throughput and performance. In addition to low latency, the device supports a packet payload size of up to 1024 bytes, enabling the user to achieve even higher throughput. Hot Plug for High Availability Hot plug capability allows users to replace hardware modules and perform maintenance without powering down the system. The PEX 8548 hot plug capability and Memory Chip Set x8 or x16 PEX 8548 End Point x8 x4 x8 Figure 2. Fan-in/out Usage Figure 2 shows a typical server-based design where the root complex provides a PCI Express link that needs to be expanded to a larger number of smaller ports for a variety of I/O functions. In this example, the PEX 8548 would have a 16-lane upstream port, and as many as six x8 or x4 downstream ports. The PEX 8548 will also auto-negotiate down to x2 and x1 lane widths to support endpoints requiring smaller links. CPU Root Complex x16 Peer-to-Peer & Backplane Usage The PEX 8548 is also suitable for peer-to-peer applications such as switch fabrics and backplanes. Figure 3 represents a backplane where the device provides peer-to-peer data exchange for a large number of line cards where the CPU/Host plays the management role. I/O I/O I/O I/O x8 x8 PEX 8548 x16 GPU x16 GPU Figure 4. Dual Graphics Fan-out Power PC x8 x8 x8 PEX 8548 Blade Server Switch Fabric The PEX 8548 can be cascaded to create an even greater number of ports. For example, in a Blade Server application, two PEX 8548 switches can be combined to support up to sixteen blades (Figure 5). The switches can be linked to available PCI Express ports. The PEX 8508 switch can isolate each CPU blade using its NonTransparent Bridging function. x8 2 I/O Blades CPU Blade x8 I/O Figure 3. Peer-to-Peer/Backplane Usage Graphics Fan-out Switch The PEX 8548 offers three port configurations which are optimal for High resolution 3D graphics applications. Dual-graphics, high-resolution gaming, high resolution scientific use, and image processing can benefit from its performance features. Figure 4 illustrates usage of the device in a dual graphics gaming application where two GPUs drive a single monitor for the ultimate gaming experience. The upstream x16 port links to the Root Complex and the two downstream ports connect to the graphics modules. The peer-to-peer support of the PEX 8548 allows the two GPU modules to communicate with each other for maximum performance. CPU CPU 14 CPU Blades Root Complex PEX 8508 NT PEX 8548 PEX 8548 Figure 5. Blade Server The number of ports and port widths can be configured as needed. This example assumes a x4 link to each CPU blade and a x8 link to the I/O blades. The link between the two PEX 8548 devices is x8 lanes wide. Software Usage Model From a system model viewpoint, each PCI Express port is a virtual PCI to PCI bridge device and has its own set of PCI Express configuration registers. It is through the upstream port that the BIOS or host can configure the other ports using standard PCI enumeration. The virtual PCI to PCI bridges within the PEX 8548 are compliant to the PCI and PCI Express system models. The Configuration Space Registers (CSRs) in a virtual primary/secondary PCI to PCI bridge are accessible by type 0 configuration cycles through the virtual primary Development Tools PLX offers hardware and software tools to enable rapid customer design activity. These tools consist of a hardware module (PEX 8548 RDK), hardware documentation, and a Software Development Kit (SDK). PORT 8 – X16 Port 8 Status LED Port 12 Status LED Port 0 Status LED PORT 12 – X16 Manual Reset HD Power Connector DIP Switch for Config EEPROM JTAG PEX 8548 I2C RefClk PERST# Port 0 – x16 (Upstream Port) Figure 6. PEX 8548 RDK bus interface (matching bus, device, and function number). Interrupt Sources/Events The PEX 8548 switch supports the INTx interrupt message type (compatible with PCI 2.3 Interrupt signals) or Message Signaled Interrupts (MSI) when enabled. The PEX 8548 generates interrupts/messages for hot plug events, doorbell interrupts, baseline error reporting, and advanced error reporting. ExpressLane PEX 8548 RDK The PEX 8548 RDK hardware module includes the PEX 8548 with one x16 upstream port and two x16 downstream ports (see Figure 6). The RDK comes with two breakout boards which allow you to split each x16 downstream slot into three (x8, x4, x4) or two (x8, x8) separate ports. Using the breakout boards, up to 6 downstream ports can be created. The PEX 8548 RDK board can be installed on a motherboard or used as a riser card. The PEX 8548 RDK can be used to test and validate customer software. Additionally, it can be used as an evaluation vehicle for PEX 8548 features and benefits. The PEX 8548 RDK provides everything that a user needs to get their hardware and software development started. SDK The SDK tool set includes: - Linux & Windows drivers - C/C++ Source code, Objects, libraries - User’s Guides & Application examples Product Ordering Information PLX Technology, Inc. 870 Maude Ave. Sunnyvale, CA 94085 USA Tel: 1-800-759-3735 Tel: 1-408-774-9060 Fax: 1-408-774-2169 Email: [email protected] Web Site: www.plxtech.com Part Number Description PEX 8548-AA25BI PEX 8548-AA25BI G PEX 8548-AA RDK 48-Lane, 9-Port PCI Express Switch 48-Lane, 9-Port PCI Express Switch, Pb-Free PEX 8548 Rapid Development Kit w/ x16 Edge Connector Breakout Board w/ x16 Edge Connector for additional fan-out to two slots (x8, x8) Breakout Board w/ x16 Edge Connector for additional fan-out to three slots (x8, x4, x4) Breakout Board-88 Breakout Board-844 Please visit the PLX Web site at http://www.plxtech.com or contact PLX sales at 408-774-9060 for sampling. © 2007 PLX Technology, Inc. All rights reserved. PLX and the PLX logo are registered trademarks of PLX Technology, Inc. ExpressLane is a trademark of PLX Technology, Inc., which may be registered in some jurisdiction. All other product names that appear in this material are for identification purposes only and are acknowledged to be trademarks or registered trademarks of their respective companies. Information supplied by PLX is believed to be accurate and reliable, but PLX Technology, Inc. assumes no responsibility for any errors that may appear in this material. PLX Technology, Inc. reserves the right, without notice, to make changes in product design or specification. PEX8548-SIL-PB-P1-1.5 01/07