PLX PEX8664

PEX 8664, PCI Express Gen 2 Switch, 64 Lanes, 16 Ports
Features
ƒ
PEX 8664 General Features
o 64-lane, 16-port PCIe Gen2 switch
- Integrated 5.0 GT/s SerDes
o 35 x 35mm2, 1156-ball FCBGA package
o Typical Power: 7.9 Watts
ƒ
The ExpressLaneTM PEX 8664 device offers Multi-Host PCI Express
switching capability enabling users to connect multiple hosts to their
respective endpoints via scalable, high bandwidth, non-blocking
interconnection to a wide variety of applications including servers,
storage systems, and communications platforms. The PEX 8664 is
well suited for fan-out, aggregation, and peer-to-peer applications.
PEX 8664 Key Features
o Standards Compliant
- PCI Express Base Specification, r2.0
(backwards compatible w/ PCIe r1.0a/1.1)
- PCI Power Management Spec, r1.2
- Microsoft Vista Compliant
- Supports Access Control Services
- Dynamic link-width control
- Dynamic SerDes speed control
o High Performance
♦ performancePAK
9 Read Pacing (bandwidth throttling)
9 Multicast
9 Dynamic Buffer/FC Credit Pool
- Non-blocking switch fabric
- Full line rate on all ports
- Packet Cut-Thru with 176ns max packet
latency (x16 to x16)
- 2KB Max Payload Size
o Flexible Configuration
- Ports configurable as x1, x2, x4, x8, x16
- Registers configurable with strapping
pins, EEPROM, I2C, or host software
- Lane and polarity reversal
- Compatible with PCIe 1.0a PM
o Multi-Host & Fail-Over Support
- Configurable Non-Transparent (NT) port
- Failover with NT port
- Up to Five upstream/Host ports with 1+1
or N+1 failover to other upstream ports
o Quality of Service (QoS)
- Eight traffic classes per port
- Weighted round-robin source
port arbitration
o Reliability, Availability, Serviceability
♦ visionPAK
9 Per Port Performance Monitoring
ƒ Per port payload & header counters
9 SerDes Eye Capture
9 Error Injection and Loopback
- 4 Hot Plug Ports with native HP Signals
- All ports hot plug capable thru I2C
(Hot Plug Controller on every port)
- ECRC and Poison bit support
- Data Path parity
- Memory (RAM) Error Correction
- INTA# and FATAL_ERR# signals
- Advanced Error Reporting
- Port Status bits and GPIO available
- Per port error diagnostics
Multi-Host Architecture
The PEX 8664 employs an enhanced version of PLX’s field tested PEX 8648
PCIe switch architecture, which allows users to configure the device in
legacy single-host mode or multi-host mode with up to five host ports
capable of 1+1 (one active & one backup) or N+1 (N active & one backup)
host failover. This powerful architectural enhancement enables users to build
PCIe based systems to support high-availability, failover, redundant and
clustered systems.
High Performance & Low Packet Latency
The PEX 8664 architecture supports packet cut-thru with a maximum
latency of 176ns (x16 to x16). This, combined with large packet memory,
flexible common buffer/FC credit pool and non-blocking internal switch
architecture, provides full line rate on all ports for performance-hungry
applications such as servers and switch fabrics. The low latency enables
applications to achieve high throughput and performance. In addition to low
latency, the device supports a packet payload size of up to 2048 bytes,
enabling the user to achieve even higher throughput.
Data Integrity
The PEX 8664 provides end-to-end CRC (ECRC) protection and Poison bit
support to enable designs that require end-to-end data integrity. PLX also
supports data path parity and memory (RAM) error correction circuitry
throughout the internal data paths as packets pass through the switch.
Flexible Configuration
The PEX 8664’s 16 ports can be
configured to lane widths of x1, x2, x4,
x8, or x16. Flexible buffer allocation,
along with the device's flexible packet
flow control, maximizes throughput
for applications where more traffic
flows in the downstream, rather than
upstream, direction. Any port can be
designated as the upstream port, which
can be changed dynamically. Figure 1
shows some of the PEX 8664’s
common port configurations in legacy
Single-Host mode.
x4
PEX 8664
15 x4
x8
PEX 8664
2 x8 10 x4
x8
PEX 8664
5 x8
4 x4
x16
PEX 8664
6 x8
Figure 1. Common Port Configurations
- JTAG AC/DC boundary scan
© PLX Technology, www.plxtech.com
Page 1 of 1
5/14/2009, Version 1.1
PEX 8664, PCI Express Gen 2 Switch, 64 Lanes, 16 Ports
PEX 8664 allows the hosts to communicate their status
to each other via special door-bell registers. In failover
mode, if a host fails, the host designated for failover will
disable the upstream port attached to the failing host and
program the downstream ports of that host to its own
domain. Figure 4a shows a two host system in MultiHost mode with two virtual switches inside the device
and Figure 4b shows Host 1 disabled after failure and
Host 2 having taken over all of Host 1’s end-points.
The PEX 8664 can also be configured in Multi-Host
mode where users can choose up to five ports as
host/upstream ports and assign a desired number of
downstream ports to each host. In Multi-Host mode, a
virtual switch is created for each host port and its
associated downstream ports inside the device. The
traffic between the ports of a virtual switch is completely
isolated from the traffic in other virtual switches. Figure
2 illustrates some configurations of the PEX 8664 in
Multi-Host mode where each ellipse represents a virtual
switch inside the device.
x8
x8
Host 1
Host 2
Host 1
PEX 8664
The PEX 8664 also
provides several ways to
PEX 8664
PEX 8664
configure its registers. The
device can be configured
x8x8x4x4 x8x8x4x4
3 x8 3 x4 3 x4
4 x4s
5 x4s
through strapping pins,
I2C interface, host
software, or an optional
PEX 8664
PEX 8664
serial EEPROM. This
allows for easy debug
12 x4s
11 x4s
during the development
Figure 2. Common Multi-Host Configurations
phase, performance
monitoring during the operation phase, and driver or
software upgrade.
Dual-Host & Failover Support
In Single-Host mode, the
Primary
Secondary
Primary Host
Host
Secondary Host
Host
PEX 8664 supports a NonCPU
CPU
Transparent (NT) Port,
which enables the
Root
implementation of dualhost
Complex
systems for redundancy and
host
failover capability. The NT
port
NT
allows systems to isolate
PEX 8664
Non-Transparent
host memory domains by
Port
presenting the
End
End
End
processor subsystem
as an
Point
Point
Point
endpoint rather than
Figure 3. Non-Transparent Port
another memory
system. Base address registers are used to translate
addresses; doorbell registers are used to send interrupts
between the address domains; and scratchpad registers
(accessible by both CPUs) allow inter-processor
communication (Figure 3).
Multi-Host & Failover Support
In Multi-Host mode, PEX 8664 can be configured with
up to five upstream host ports, each with its own
dedicated downstream ports. The device can be
configured for 1+1 redundancy or N+1 redundancy. The
© PLX Technology, www.plxtech.com
Host 2
x8 x4 x4
End
Point
End
Point
End
Point
PEX 8664
End
Point
Figure 4a. Multi-Host
End
Point
End
Point
End
Point
End
Point
Figure 4b. Multi-Host Fail-Over
Hot Plug for High Availability
Hot plug capability allows users to replace hardware
modules and perform maintenance without powering
down the system. The PEX 8664 hot plug capability
feature makes it suitable for High Availability (HA)
applications. Four downstream ports include a Standard
Hot Plug Controller. If the PEX 8664 is used in an
application where one or more of its downstream ports
connect to PCI Express slots, each port’s Hot Plug
Controller can be used to manage the hot-plug event of
its associated slot. Every port on the PEX 8664 is
equipped with a hot-plug control/status register to
support hot-plug capability through external logic via the
I2C interface.
SerDes Power and Signal Management
The PEX 8664 supports software control of the SerDes
outputs to allow optimization of power and signal
strength in a system. The PLX SerDes implementation
supports four levels of power – off, low, typical, and
high. The SerDes block also supports loop-back modes
and advanced reporting of error conditions, which
enables efficient management of the entire system.
Interoperability
The PEX 8664 is designed to be fully compliant with the
PCI Express Base Specification r2.0, and is backwards
compatible to PCI Express Base Specification r1.1 and
r1.0a. Additionally, it supports auto-negotiation, lane
reversal, and polarity reversal. Furthermore, the PEX
8664 is tested for Microsoft Vista compliance. All PLX
switches undergo thorough interoperability testing in
PLX’s Interoperability Lab and compliance testing at
the PCI-SIG plug-fest.
Page 2 of 2
5/14/2009, Version 1.1
PEX 8664, PCI Express Gen 2 Switch, 64 Lanes, 16 Ports
performancePAKTM
Exclusive to PLX, performancePAK is a suite of unique
and innovative performance features which allows
PLX’s Gen 2 switches to be the highest performing Gen
2 switches in the market today. The performancePAK
features consists of the Read Pacing, Multicast, and
Dynamic Buffer Pool.
Read Pacing
The Read Pacing feature allows users to throttle the
amount of read requests being made by downstream
devices. When a downstream device requests several
long reads back-to-back, the Root Complex gets tied up
in serving that downstream port. If that port has a narrow
link and is therefore slow in receiving these read packets
from the Root Complex, then other downstream ports
may become starved – thus, impacting performance. The
Read Pacing feature enhances performances by allowing
for the adequate servicing of all downstream devices.
Performance Monitoring
The PEX 8664’s real time performance monitoring
allows users to literally “see” ingress and egress
performance on each port as traffic passes through the
switch using PLX’s Software Development Kit (SDK).
The monitoring is completely passive and therefore has
no affect on overall system performance. Internal
counters provide extensive granularity down to traffic &
packet type and even allows for the filtering of traffic
(i.e. count only Memory Writes).
SerDes Eye Capture
Users can evaluate their system’s signal integrity at the
physical layer using the PEX 8664’s SerDes Eye
Capture feature. Using PLX’s SDK, users can view the
receiver eye of any lane on the switch. Users can then
modify SerDes settings and see the impact on the
receiver eye. Figure 5 shows a screenshot of the SerDes
Eye Capture feature in the SDK.
Multicast
The Multicast feature enables the copying of data
(packets) from one ingress port to multiple (up to 15)
egress ports in one transaction allowing for higher
performance in dual-graphics, storage, security, and
redundant applications, among others. Multicast relieves
the CPU from having to conduct multiple redundant
transactions, resulting in higher system performance.
Dynamic Buffer Pool
The PEX 8664 employs a dynamic buffer pool for Flow
Control (FC) management. As opposed to a static buffer
scheme which assigns fixed, static buffers to each port,
PLX’s dynamic buffer allocation scheme utilizes a
common pool of FC Credits which are shared by other
ports. This shared buffer pool is fully programmable by
the user, so FC credits can be allocated among the ports
as needed. Not only does this prevent wasted buffers and
inappropriate buffer assignments, any unallocated
buffers remain in the common buffer pool and can then
be used for faster FC credit updates.
visionPAKTM
Another PLX exclusive, visionPAK is a debug
diagnostics suite of integrated hardware and software
instruments that users can use to help bring their systems
to market faster. visionPAK features consist of
Performance Monitoring, SerDes Eye Capture, Error
Injection, SerDes Loopback, and more.
© PLX Technology, www.plxtech.com
Figure 5. SerDes Eye Capture
Error Injection & SerDes Loopback
Using the PEX 8664’s Error Injection feature, users can
inject malformed packets and/or fatal errors into their
system and evaluate a system’s ability to detect and
recover from such errors. The PEX 8664 also supports
Internal Tx, External Tx, Recovered Clock, and
Recovered Data Loopback modes.
Applications
Suitable for host-centric as well as peer-to-peer traffic
patterns, the PEX 8664 can be configured for a wide
variety of form factors and applications.
Host Centric Fan-out
The PEX 8664, with its symmetric or asymmetric lane
configuration capability, allows user-specific tuning to a
variety of host-centric applications. Figure 6 shows a
Page 3 of 3
5/14/2009, Version 1.1
PEX 8664, PCI Express Gen 2 Switch, 64 Lanes, 16 Ports
server design where, in a quad or multi processor
system, users can assign endpoints/slots to CPU cores to
distribute the system load. The packets directed to
different CPU cores will go to different (user assigned)
PEX 8664 upstream ports, allowing better queuing and
load balancing capability for higher performance.
CPU
CPU
CPU
x8
x4
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Memory
Chipset
Chipset
x8
Chipset
x8
x16
x8
PEX 8664
x8s
Figure 6. Host Centric Dual Upstream
Embedded or Communications Systems
The PEX 8664’s 64 lanes can come in handy for
embedded or communications applications requiring
heavy processing and/or connectivity to multiple
endpoints. Figure 7a shows an embedded system where
the PEX 8664 is being used to fan-out to eight endpoints
using x8 and x16 links. Figure 7b shows a
communications system where the PEX 8664 is using
x16 downstream links to fan out to three CPUs which
have been configured as endpoints. These CPUs will run
as endpoints, conducting different processing tasks while
the host CPU (connected to the PEX 8664 via a x16
upstream link) manages them.
CPU
Chip
Set
Chip
Set
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
I/O
x8
x4
x8
x8
PEX 8616
PEX 8616
PEX 8612
x4
x4
x4
x4
PEX 8612
x4
x4
FC
FC
x4
x4
FC
FC
x16s
CPU
Endpoint
I/O
I/O
x8
PEX 8664
PEX 8664
x8s
I/O
x4
x16
PEX 8664
I/O
x4 & x8
N+1 Fail-Over in Storage Systems
The PEX 8664’s Multi-Host feature can also be used to
develop storage array clusters where each host manages
a set of storage devices independent of others. Users can
designate one of the hosts as the failover-host for all the
other hosts while actively managing its own endpoints.
The failover-host will communicate with other hosts for
status/heartbeat information and execute a failover event
if/when it gets triggered (see Figure 9).
x4
x16
PEX 8664
Figure 8. Host Fail-Over
x4
CPU
Endpoint
PCIe Gen1 or PCIe Gen2 slots
Endpoint
Endpoint
PEX 8664
Endpoint
x4s
x8s
Endpoint
Endpoint
Endpoint
CPU
Endpoint
CPU
information through doorbell registers or I2C interface.
The devices can be programmed to trigger fail-over if
the heartbeat information is not provided. In the event of
a failure, the surviving device will reset the endpoints
connected to the failing CPU and enumerate them in its
own domain without impacting the operation of
endpoints already in its domain.
I/O
Figure 7a. Embedded System
CPU
Endpoint
FC
CPU
Endpoint
Figure 7b. Comms System
FC
8 Disk Chassis
8 Disk Chassis
8 Disk Chassis
8 Disk Chassis
Figure 9. N+1 Failover
Host Failover
The PEX 8664 can also be utilized in applications where
host failover is required. In the application shown in
Figure 8, two hosts may be active simultaneously and
controlling their own domains while exchange status
© PLX Technology, www.plxtech.com
FC
FC
Page 4 of 4
5/14/2009, Version 1.1
PEX 8664, PCI Express Gen 2 Switch, 64 Lanes, 16 Ports
Software Usage Model
From a system model viewpoint, each PCI Express port
is a virtual PCI to PCI bridge device and has its own set
of PCI Express configuration registers. It is through the
upstream port that the BIOS or host can configure the
other ports using standard PCI enumeration. The virtual
PCI to PCI bridges within the PEX 8664 are compliant
to the PCI and PCI Express system models. The
Configuration Space Registers (CSRs) in a virtual
primary/secondary PCI to PCI bridge are accessible by
type 0 configuration cycles through the virtual primary
bus interface (matching bus number, device number, and
function number).
Interrupt Sources/Events
The PEX 8664 switch supports the INTx interrupt
message type (compatible with PCI 2.3 Interrupt signals)
or Message Signaled Interrupts (MSI) when enabled.
Interrupts/messages are generated by PEX 8664 for hot
plug events, doorbell interrupts, baseline error reporting,
and advanced error reporting.
PEX 8664 Rapid Development Kit (RDK)
The PEX 8664 RDK (see Figure 10) is a hardware
module containing the PEX 8664 which plugs right into
your system. The PEX 8664 RDK can be used to test
and validate customer software, or used as an evaluation
vehicle for PEX 8664 features and benefits. The PEX
8664 RDK provides everything that a user needs to get
their hardware and software development started.
Software Development Kit (SDK)
PLX’s Software Development Kit is available for
download at www.plxtech.com/sdk. The software
development kit includes drivers, source code, and GUI
interfaces to aid in configuring and debugging the PEX
8664. For more information, please refer to the PEX
8664 RDK Product Brief.
Both performancePAK and visionPAK are supported by
PLX’s RDK and SDK, the industry’s most advanced
hardware- and software-development kits.
Product Ordering Information
Part Number
PEX8664-AA50BC F
PEX8664-16U8D BB
RDK
Figure 10. PEX8664-16U8D BB RDK
Description
64-Lane, 16-Port PCI Express Switch,
Pb-Free (35x35mm2)
PEX 8664 Rapid Development Kit with
x16 Upstream and Six x8 Downstream
PLX Technology, Inc. All rights reserved. PLX, the PLX logo, ExpressLane,
Read Pacing and Dual Cast are trademarks of PLX Technology, Inc. All other
product names that appear in this material are for identification purposes only
and are acknowledged to be trademarks or registered trademarks of their
respective companies. Information supplied by PLX is believed to be accurate
and reliable, but PLX assumes no responsibility for any errors that may appear
in this material. PLX reserves the right, without notice, to make changes in
product design or specification.
Development Tools
PLX offers hardware and software tools to enable rapid
customer design activity. These tools consist of a
hardware module (PEX 8664 RDK), hardware
documentation (available at www.plxtech.com), and a
Software Development Kit (also available at
www.plxtech.com).
© PLX Technology, www.plxtech.com
Visit www.plxtech.com for more information.
Page 5 of 5
5/14/2009, Version 1.1