Intel® I/O Expansion Modules for Intel® Platforms based on Intel® Xeon® Processor E54600/2600/2400 Product Families Hardware Specification Intel order number: G30021-004 Revision 1.4 April 2015 Intel® Server Boards and Systems Revision History Intel® I/O Expansion Modules for Intel® Platforms HWS Revision History Date March, 2011 Revision Number 0.5 August, 2011 0.9 Modifications Initial Release. Added the following sections: ® Section 2.4 – Intel I350 Gb Ethernet Controller Section 3.4 – Intel X540 10Gb Ethernet Controller ® Section 4.4 – Intel 82599 10Gb Ethernet Controller Section 5.4 - ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI Section 6 – FDR InfiniBand* ConnectX*-3 I/O Module (AXX1FDRIOIOM) ® August, 2011 0.91 Added figures for port identification. January, 2012 1.0 Deleted Section 5 - QDR InfiniBand* ConnectX*-3 I/O Module, and added new section for Dual Port FDR InfiniBand* ConnectX*-3 I/O Module. November, 2012 1.1 Added AXX10GBTWLHW Updated Table 1 December, 2014 1.2 Added AXX10GBTWLHW2 March, 2015 1.3 Updated Table 1 April, 2015 1.4 Added AXX10GBTWLHW3 and AXX10GBTWLIOM3 Updated Table 1 Disclaimers ® Information in this document is provided in connection with Intel products. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted by this document. Except as provided in Intel's ® ® Terms and Conditions of Sale for such products, Intel assumes no liability whatsoever, and Intel disclaims any ® express or implied warranty, relating to sale and/or use of Intel products including liability or warranties relating to fitness for a particular purpose, merchantability, or infringement of any patent, copyright or other intellectual property ® ® right. Intel products are not intended for use in medical, lifesaving, or life sustaining applications. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or ® "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. This document contains information on products in the design phase of development. Do not finalize a design with this information. Revised information will be published when the product is available. Verify with your local sales office that you have the latest datasheet before finalizing a design. ® The Intel I/O Expansion Modules may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel Corporation server baseboards contain a number of high-density VLSI and power delivery components that ® need adequate airflow to cool. Intel ’s own chassis are designed and tested to meet the intended thermal requirements of these components when the fully integrated system is used together. It is the responsibility of the ® system integrator that chooses not to use Intel developed server building blocks to consult vendor datasheets and operating parameters to determine the amount of air flow required for their specific application and environmental conditions. Intel Corporation cannot be held responsible if components fail or the server board does not operate correctly when used outside any of their published operating or non-operating limits. Intel and Xeon are trademarks or registered trademarks of Intel Corporation. *Other brands and names may be claimed as the property of others. Copyright © Intel Corporation 2015. Revision 1.4 ii Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Table of Contents Table of Contents 1. Introduction ........................................................................................................................1 2. Quad Port Intel® I350 GbE I/O Module (AXX4P1GBPWLIOM) ..........................................4 2.1 Feature Set ............................................................................................................4 2.2 Functional Block Diagram ......................................................................................5 2.3 Mechanical Dimensions .........................................................................................6 2.4 Intel® I350 Gb Ethernet Controller ..........................................................................7 ® 3. Dual Port Intel X540 10GbE I/O Module (AXX10GBTWLIOM and AXX10GBTWLHW)... 9 3.1 Feature Set ............................................................................................................9 3.2 Functional Block Diagram ....................................................................................10 3.3 Mechanical Dimensions .......................................................................................11 3.4 Intel® X540 10Gb Ethernet Controller ...................................................................13 ® 4. Dual Port Intel X540 10GbE I/O Module (AXX10GBTWLHW2)...................................... 15 4.1 Feature Set ..........................................................................................................15 4.2 Functional Block Diagram ....................................................................................16 4.3 Mechanical Dimensions .......................................................................................17 4.4 Intel X540 10Gb Ethernet Controller ...................................................................19 ® 5. Dual Port Intel® X540 10GbE I/O Module (AXX10GBTWLIOM3 and AXX10GBTWLHW3)21 5.1 Feature Set ..........................................................................................................21 5.2 Functional Block Diagram ....................................................................................22 5.3 Mechanical Dimensions .......................................................................................23 5.4 Intel® X540 10Gb Ethernet Controller ...................................................................25 6. Dual Port Intel® 82599 10GbE I/O Module (AXX10GBNIAIOM) ....................................... 26 6.1 Feature Set ..........................................................................................................26 6.2 Functional Block Diagram ....................................................................................27 6.3 Mechanical Dimensions .......................................................................................28 6.4 Intel® 82599 10Gb Ethernet Controller .................................................................30 6.5 LAN LED Functionality .........................................................................................31 7. Single Port FDR InfiniBand* ConnectX*-3 I/O Module (AXX1FDRIBIOM) ..................... 32 7.1 Feature Set ..........................................................................................................32 7.2 Functional Block Diagram ....................................................................................33 7.3 Mechanical Dimensions .......................................................................................34 7.4 ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI ....................................... 35 iii Revision 1.4 Intel order number: G30021-004 Table of Contents Intel® I/O Expansion Modules for Intel® Platforms HWS 7.4.1 Overview ..............................................................................................................35 7.4.2 Key Features ........................................................................................................36 7.5 Network LED Functionality ...................................................................................36 8. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module (AXX2FDRIBIOM) ........................ 37 8.1 Feature Set ..........................................................................................................37 8.2 Functional Block Diagram ....................................................................................38 8.3 Mechanical Dimensions .......................................................................................39 8.4 ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI ....................................... 40 8.4.1 Overview ..............................................................................................................40 8.4.2 Key Features ........................................................................................................41 8.5 Network LED Functionality ...................................................................................41 Revision 1.4 iv Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS List of Figures List of Figures Figure 1. Quad Port Intel® I350 GbE I/O Module .........................................................................4 Figure 2. Quad Port Intel® I350 GbE I/O Module Block Diagram .................................................5 ® Figure 3. Quad Port Intel I350 GbE I/O Port Identification .........................................................5 Figure 4. Quad Port Intel® I350 GbE I/O Module Dimensions; Top and Side Views ....................6 Figure 5. Quad Port Intel® I350 GbE I/O Module Dimensions; Bottom View ...............................7 Figure 6. Dual Port Intel® X540 10GbE I/O Module .....................................................................9 Figure 7. Dual Port Intel® X540 10GbE I/O Module Block Diagram ........................................... 10 Figure 8. Dual Port Intel® X540 10GbE I/O Module Port Identification ...................................... 10 Figure 9. Dual Port Intel® X540 10GbE I/O Module Dimensions; Top and side View ................ 11 Figure 10. Dual Port Intel® X540 10GbE I/O Module Dimensions; Bottom View........................ 12 Figure 11. Dual Port Intel® X540 10GbE I/O Module .................................................................15 ® Figure 12. Dual Port Intel X540 10GbE I/O Module Block Diagram ......................................... 16 Figure 13. Dual Port Intel® X540 10GbE I/O Module Port Identification .................................... 16 Figure 14. Dual Port Intel® X540 10GbE I/O Module Dimensions; Top and side View .............. 17 ® Figure 15. Dual Port Intel X540 10GbE I/O Module Dimensions; Bottom View........................ 18 Figure 16. Dual Port Intel® 82599 10GbE I/O Module ...............................................................26 ® Figure 17. Dual Port Intel 82599 10GbE I/O Module Block Diagram ....................................... 27 ® Figure 18. Dual Port Intel 82599 10GbE I/O Module Port Identification ................................... 27 ® Figure 19. Dual Port Intel 82599 10GbE I/O Module Dimensions; Top and side View ............. 28 ® Figure 20. Dual Port Intel 82599 10GbE I/O Module Dimensions; Bottom View ...................... 29 ® Figure 21. Dual Port Intel 82599 10GbE I/O Module; Rear View ............................................. 31 Figure 22. Single Port FDR InfiniBand* ConnectX*-3 I/O Module ............................................. 32 Figure 23. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Block Diagram ..................... 33 Figure 24. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Port Identification ................. 33 Figure 25. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Top and side View ...................................................................................................................................34 Figure 26. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Bottom View .... 35 Figure 27. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module ................................................ 37 Figure 28. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Block Diagram ........................ 38 Figure 29. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Port Identification .................... 38 Figure 30. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Top and side View39 Figure 31. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Bottom View ....... 40 v Revision 1.4 Intel order number: G30021-004 Table of Contents Intel® I/O Expansion Modules for Intel® Platforms HWS vi Revision 1.4 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS List of Tables List of Tables Table 1. I/O Module Support Matrix ............................................................................................1 Table 2. 80-pin I/O Module Connector Pin-Out ...........................................................................2 Table 3. Dual Port Intel® 82599 10GbE I/O Module LED Functionality ...................................... 31 Table 4. Single Port FDR InfiniBand* ConnectX*-3 I/O Module LED Functionality .................... 36 Table 5. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module LED Functionality....................... 41 vii Revision 1.4 Intel order number: G30021-004 List of Tables Intel® I/O Expansion Modules for Intel® Platforms HWS <This page is intentionally left blank.> Revision 1.4 viii Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 1. Introduction Introduction The Intel® Server Boards support a variety of Intel® I/O Expansion Module options using x4 or x8 PCI Express* Mezzanine connectors on the server board. Each mezzanine connector is a 2 x 40-pin, surface mount, and 0.8 mm pitch header. Intel® I/O Expansion Modules for Intel® Platforms based on Intel® Xeon® processor E54600/2600/2400 product families includes: Quad Port Intel® I350 GbE I/O Module Dual Port Intel® X540 10GbE I/O Module Dual Port Intel® 82599 10GbE I/O Module Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dual Ports FDR InfiniBand* ConnectX*-3 I/O Module Intel® I/O Expansion Modules for Intel® Platforms based on Intel® Xeon® processor E54600/2600/2400 product families are designed to fit Intel® Server Boards based on Intel® Xeon® processor E5-4600/2600/2400 product families. The table below shows the support matrix for the Intel® I/O Expansion Modules. S1600JP S2400BB S2600GZ/GL S2600IP S2600JF S4600LH2/LT2 S2600WT S2600KP S2600TP S12003RP × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × × Dual Port Intel 82599 10GbE I/O Module - AXX10GBNIAIOM Single Port FDR InfiniBand* ConnectX*-3 I/O Module AXX1FDRIBIOM Dual Port FDR InfiniBand* ConnectX*3 I/O Module – AXX2FDRIBIOM Intel® I/O Expansion Modules S2600WP S1400SP Table 1. I/O Module Support Matrix ® Quad Port Intel I350 GbE I/O Module - AXX4P1GBPWLIOM ® Dual Port Intel X540 10GbE I/O Module - AXX10GBTWLIOM ® Dual Port Intel X540 10GbE I/O Module - AXX10GBTWLHW ® Dual Port Intel X540 10GbE I/O Module - AXX10GBTWLHW2 ® Dual Port Intel X540 10GbE I/O Module - AXX10GBTWLHW3 ® Dual Port Intel X540 10GbE I/O Module - AXX10GBTWLIOM3 ® Note: 1 Revision 1.4 Intel order number: G30021-004 Introduction Intel® I/O Expansion Modules for Intel® Platforms HWS 1. Intel® I/O Expansion Modules for Intel® Platforms based on Intel® Xeon® processor E54600/2600/2400 product families are not supported on previous generation Intel® Server Boards. 2. For the latest support information on each Intel® Server Platform, please go to http://serverconfigurator.intel.com/sct_app.aspx. The following table details the pin-out of the I/O module connector: Table 2. 80-pin I/O Module Connector Pin-Out Pin 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Signal 3.3V 3.3V 3.3V 3.3V RSVD GND RSVD+ RSVDGND RSVD RSVD GND SMB CLK SMB DAT GND PCIe Gen3 Tn [7]] PCIe Gen3 Tp [7] GND PCIe Gen3 Tn [6] PCIe Gen3 Tp [6] GND PCIe Gen3 Tn [5] PCIe Gen3 Tp [5] GND PCIe Gen3 Tn [4] PCIe Gen3 Tp [4] GND PCIe Gen3 Tn [3] PCIe Gen3 Tp [3] GND PCIe Gen3 Tn [2] PCIe Gen3 Tp [2] GND PCIe Gen3 Tn [1] Signal 12V 12V 12V 12V FRU/TEMP ADDR [A0] 5VSB FM_IO_MODULE_EN 3.3VSTBY LED_GLOBAL ACT# FM_IOM_PRESENT_N WAKE# PERST# GND rIOM REFCLK+ [0] rIOM REFCLK- [0] GND PCIe Gen3 Rn [7] PCIe Gen3 Rp [7] GND PCIe Gen3 Rn [6] PCIe Gen3 Rp [6] GND PCIe Gen3 Rn [5] PCIe Gen3 Rp [5] GND PCIe Gen3 Rn [4] PCIe Gen3 Rp [4] GND PCIe Gen3 Rn [3] PCIe Gen3 Rp [3] GND PCIe Gen3 Rn [2] PCIe Gen3 Rp [2] GND Pin 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 Revision 1.4 2 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Pin 69 71 73 75 77 79 Signal PCIe Gen3 Tp [1] GND PCIe Gen3 Tn [0] PCIe Gen3 Tp [0] GND RSVD Introduction Signal PCIe Gen3 Rn [1] PCIe Gen3 Rp [1] GND PCIe Gen3 Rn [0] PCIe Gen3 Rp [0] GND Pin 70 72 74 76 78 80 Note: There are several pins reserved for future use. 3 Revision 1.4 Intel order number: G30021-004 Quad Port Intel® I350 GbE I/O Module 2. Intel® I/O Expansion Modules for Intel® Platforms HWS Quad Port Intel® I350 GbE I/O Module (AXX4P1GBPWLIOM) The Quad Port Intel® I350 GbE I/O Module provides four additional 10/100/1000Mbit external connections. This section provides a high-level description of the implementation of this I/O module. ® Figure 1. Quad Port Intel I350 GbE I/O Module 2.1 Feature Set ® The Quad Port Intel I350 GbE I/O Module supports the following feature set: Intel® I350 Controller 17x17mm chip Quad 1Gb RJ45 Ethernet external connections Speed and Link/Act LEDs for each RJ45 connection Onboard 1.0V VR off of 12V and 5V_SB Rails One WOL port operating in 1 GB mode is supported (Port 2) 1.8V_SB linear VR off 3.3V_SB 256kb LAN EEPROM and 1Mb SPI Flash (SPI Flash unstuffed as baseboard BIOS will provide OpROM for PXE and iSCSI support) Three loose screws to mount IOM directly to sheet metal chassis. It's recommended to use same screws as used to install the baseboard 256Byte FRU EEPROM and TMP75 temp sensor Revision 1.4 4 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Quad Port Intel® I350 GbE I/O Module 2.2 Functional Block Diagram NIC Port 1 2 x 40-pin connector NIC Port 2 PCIe Gen2 x4 ® Intel I350 NIC Port 3 NIC Port 4 SPI Flash （empty） EEPROM Figure 2. Quad Port Intel® I350 GbE I/O Module Block Diagram Figure 3. Quad Port Intel® I350 GbE I/O Port Identification 5 Revision 1.4 Intel order number: G30021-004 Quad Port Intel® I350 GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 2.3 Mechanical Dimensions Figure 4. Quad Port Intel® I350 GbE I/O Module Dimensions; Top and Side Views Revision 1.4 6 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Quad Port Intel® I350 GbE I/O Module Figure 5. Quad Port Intel® I350 GbE I/O Module Dimensions; Bottom View 2.4 Intel® I350 Gb Ethernet Controller The Intel® Ethernet Controller I350 is a single, compact, low power component that supports quad port and dual port gigabit Ethernet designs. The device offers four fully-integrated gigabit Ethernet media access control (MAC), physical layer (PHY) ports and four SGMII/SerDes ports that can be connected to an external PHY. The I350 supports PCI Express* (PCIe v2.1 (2.5GT/s and 5GT/s)). The device enables two-port or four-port 1000BASE-T implementations using integrated PHYs. It can be used for server system configurations such as rack mounted or pedestal servers, in an add-on NIC or LAN on Motherboard (LOM) design. Another possible system configuration is for blade servers. Here, the I350 can support up to four SerDes ports as LOM or mezzanine card. It can also be used in embedded applications such as switch add-on cards and network appliances. 7 Revision 1.4 Intel order number: G30021-004 Quad Port Intel® I350 GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 1. External Interfaces provided PCIe v2.1 (2.5GT/s and 5GT/s) x4/x2/x1 MDI (Copper) standard IEEE 802.3 Ethernet interface for 1000BASE-T, 100BASE-TX, and 10BASE-T applications (802.3, 802.3u, and 802.3ab) Serializer-Deserializer (SERDES) to support 1000BASE-SX/LX (optical fiber IEEE802.3) Serializer-Deserializer (SERDES) to support 1000BASE-KX(802.3ap) and 1000BASEBX (PICMIG 3.1) for Gigabit backplane applications SGMII (Serial-GMII Specification) interface for SFP (SFP MSA INF-8074i)/external PHY connections NC-SI (DMTF NC-SI) or SMBus for Manageability connection to BMC IEEE 1149.6 JTAG 2. Performance Enhancements PCIe v2.1 TLP Process Hints (TPH) UDP, TCP and IP Checksum offload UDP and TCP Transmit Segmentation Offload (TSO) SCTP receive and transmit checksum offload 3. Virtualization ready Next Generation VMDq support (8 VMs) Support of up to 8 VMs per port (1 queue allocated to each VM) PCI-SIG I/O SR-IOV support (Direct assignment) Queues per port: 8 TX and 8 RX queues 4. Power saving features Advanced Configuration and Power Interface (ACPI) power management states and wake-up capability Advanced Power Management (APM) wake-up functionality Low power link-disconnect state PCIe v2.1 LTR DMA Coalescing for improved system power management EEE (IEEE802.3az) for reduced power consumption during low link utilization periods 5. IEEE802.1AS - Timing and Synchronization IEEE 1588 Precision Time Protocol support Per-packet timestamp 6. Total Cost of Ownership (TCO) IPMI BMC pass-thru; multi-drop NC-SI Internal BMC to OS and OS to BMC traffic support 7. Additional product details 17x17 (256 Balls) or 25x25 (576 Balls) PBGA package Estimated power: 2.8W (max) in dual port mode and 4.2W (max) in quad port mode Memories have Parity or ECC protection See Intel® Ethernet Controller I350 Datasheet for more details. Revision 1.4 8 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 3. Dual Port Intel® X540 10GbE I/O Module Dual Port Intel® X540 10GbE I/O Module (AXX10GBTWLIOM and AXX10GBTWLHW) ® Figure 6. Dual Port Intel X540 10GbE I/O Module 3.1 Feature Set Following is the feature set: Intel® X540 Controller 25x25mm chip Two 10Gb RJ45 Ethernet external connections Support PXE and iSCSI WOL support of one 1GbE port Speed and Link/Act LEDs for each 10Gb RJ45 connection 16M bit SPI Flash 256Byte FRU EEPROM and TMP75 temp sensor Stainless steel EMI shield AXX10GBTWLHW is the same with AXX10GBTWLIOM except for the heat sink to fit Intel® Server Board S2600JF, S2600WP, and Intel® Server Chassis H2000 Family. 9 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 3.2 Functional Block Diagram NIC Port 1 2 x 40-pin connector PCIe Gen2 x8 Intel® X540 NIC Port 2 SPI Flash ® Figure 7. Dual Port Intel X540 10GbE I/O Module Block Diagram ® Figure 8. Dual Port Intel X540 10GbE I/O Module Port Identification Revision 1.4 10 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® X540 10GbE I/O Module 3.3 Mechanical Dimensions ® Figure 9. Dual Port Intel X540 10GbE I/O Module Dimensions; Top and side View 11 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS ® Figure 10. Dual Port Intel X540 10GbE I/O Module Dimensions; Bottom View Revision 1.4 12 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® X540 10GbE I/O Module 3.4 Intel® X540 10Gb Ethernet Controller 1. General Serial flash interface Device disable capability Package size - 25 mm x 25 mm 2. Networking 10 GbE/1 GbE/100 Mb/s copper PHYs integrated on-chip Support for jumbo frames of up to 15.5 KB Flow control support: send/receive pause frames and receive FIFO thresholds Statistics for management and RMON 802.1q VLAN support TCP segmentation offload: up to 256 KB IPv6 support for IP/TCP and IP/UDP receive checksum offload Fragmented UDP checksum offload for packet reassembly Message Signaled Interrupts (MSI) Message Signaled Interrupts (MSI-X) Interrupt throttling control to limit maximum interrupt rate and improve CPU usage Receive packet split header Multiple Receive Queues (RSS) 8x8 and 16x4 32 transmit queues Receive header replication Dynamic interrupt moderation DCA support TCP timer interrupts No snoop Relaxed ordering Support for 16 Virtual Machines Device Queues (VMDq) per port 3. Host Interface PCIe base specification 2.1 (2.5GT/s or 5GT/s) Bus width — x1, x2, x4, x8 64-bit address support for systems using more than 4 GB of physical memory 4. MAC Functions Descriptor ring management hardware for transmit and receive ACPI register set and power down functionality supportingD0 and D3 states A mechanism for delaying/reducing transmit interrupts Software-controlled global reset bit (resets everything except the configuration registers) Four Software-Definable Pins (SDP) per port Wake up IPv6 wake-up filters Configurable flexible filter (through NVM) LAN function disable capability Programmable memory transmit buffers (160 KB/port) Default configuration by NVM for all LEDs for pre-driver functionality 5. Manageability Eight VLAN L2 filters 16 Flex L3 port filters 13 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS Four Flexible TCO filters Four L3 address filters (IPv4) Advanced pass through-compatible management packet transmit/receive support SMBus interface to an external Manageability Controller (MC) NC-SI interface to an external MC Four L3 address filters (IPv6) Four L2 address filters See Intel® Ethernet Controller 10G X540 Datasheet for more details. Revision 1.4 14 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 4. Dual Port Intel® X540 10GbE I/O Module Dual Port Intel® X540 10GbE I/O Module (AXX10GBTWLHW2) Figure 11. Dual Port Intel® X540 10GbE I/O Module 4.1 Feature Set Following is the feature set: Intel® X540 Controller 25x25mm chip Two 10Gb RJ45 Ethernet external connections Support PXE and iSCSI Speed and Link/Act LEDs for each 10Gb RJ45 connection 16M bit SPI Flash 256Byte FRU EEPROM and TMP75 temp sensor Stainless steel EMI shield 15 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 4.2 Functional Block Diagram NIC Port 1 2 x 40-pin connector PCIe Gen2 x8 Intel® X540 NIC Port 2 SPI Flash Figure 12. Dual Port Intel® X540 10GbE I/O Module Block Diagram Figure 13. Dual Port Intel® X540 10GbE I/O Module Port Identification Revision 1.4 16 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® X540 10GbE I/O Module 4.3 Mechanical Dimensions Figure 14. Dual Port Intel® X540 10GbE I/O Module Dimensions; Top and side View 17 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS Figure 15. Dual Port Intel® X540 10GbE I/O Module Dimensions; Bottom View Revision 1.4 18 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® X540 10GbE I/O Module 4.4 Intel® X540 10Gb Ethernet Controller 1. General Serial flash interface Device disable capability Package size - 25 mm x 25 mm 2. Networking 10 GbE/1 GbE/100 Mb/s copper PHYs integrated on-chip Support for jumbo frames of up to 15.5 KB Flow control support: send/receive pause frames and receive FIFO thresholds Statistics for management and RMON 802.1q VLAN support TCP segmentation offload: up to 256 KB IPv6 support for IP/TCP and IP/UDP receive checksum offload Fragmented UDP checksum offload for packet reassembly Message Signaled Interrupts (MSI) Message Signaled Interrupts (MSI-X) Interrupt throttling control to limit maximum interrupt rate and improve CPU usage Receive packet split header Multiple Receive Queues (RSS) 8x8 and 16x4 32 transmit queues Receive header replication Dynamic interrupt moderation DCA support TCP timer interrupts No snoop Relaxed ordering Support for 16 Virtual Machines Device Queues (VMDq) per port 3. Host Interface PCIe base specification 2.1 (2.5GT/s or 5GT/s) Bus width — x1, x2, x4, x8 64-bit address support for systems using more than 4 GB of physical memory 4. MAC Functions Descriptor ring management hardware for transmit and receive ACPI register set and power down functionality supportingD0 and D3 states A mechanism for delaying/reducing transmit interrupts Software-controlled global reset bit (resets everything except the configuration registers) Four Software-Definable Pins (SDP) per port Wake up IPv6 wake-up filters Configurable flexible filter (through NVM) LAN function disable capability Programmable memory transmit buffers (160 KB/port) Default configuration by NVM for all LEDs for pre-driver functionality 5. Manageability Eight VLAN L2 filters 16 Flex L3 port filters 19 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS Four Flexible TCO filters Four L3 address filters (IPv4) Advanced pass through-compatible management packet transmit/receive support SMBus interface to an external Manageability Controller (MC) NC-SI interface to an external MC Four L3 address filters (IPv6) Four L2 address filters See Intel® Ethernet Controller 10G X540 Datasheet for more details. Revision 1.4 20 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 5. Dual Port Intel® 82599 10GbE I/O Module Dual Port Intel® X540 10GbE I/O Module (AXX10GBTWLIOM3 and AXX10GBTWLHW3) ® Figure 16. Dual Port Intel X540 10GbE I/O Module 5.1 Feature Set Following is the feature set: Intel® X540 Controller 25x25mm chip Two 10Gb RJ45 Ethernet external connections Support PXE and iSCSI Speed and Link/Act LEDs for each 10Gb RJ45 connection 16M bit SPI Flash 256Byte FRU EEPROM and TMP75 temp sensor Stainless steel EMI shield AXX10GBTWLHW3 is the same with AXX10GBTWLIOM3 except for the heat sink to fit Intel® Server Board S2600KP, S2600TP, and Intel® Server Chassis H2000G Family. 21 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® 82599 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 5.2 Functional Block Diagram NIC Port 1 2 x 40-pin connector PCIe Gen2 x8 Intel® X540 NIC Port 2 SPI Flash ® Figure 17. Dual Port Intel X540 10GbE I/O Module Block Diagram ® Figure 18. Dual Port Intel X540 10GbE I/O Module Port Identification Revision 1.4 22 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® 82599 10GbE I/O Module 5.3 Mechanical Dimensions ® Figure 19. Dual Port Intel X540 10GbE I/O Module Dimensions; Top and side View 23 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® 82599 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS ® Figure 20. Dual Port Intel X540 10GbE I/O Module Dimensions; Bottom View Revision 1.4 24 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® 82599 10GbE I/O Module 5.4 Intel® X540 10Gb Ethernet Controller 6. General Serial flash interface Device disable capability Package size - 25 mm x 25 mm 7. Networking 10 GbE/1 GbE/100 Mb/s copper PHYs integrated on-chip Support for jumbo frames of up to 15.5 KB Flow control support: send/receive pause frames and receive FIFO thresholds Statistics for management and RMON 802.1q VLAN support TCP segmentation offload: up to 256 KB IPv6 support for IP/TCP and IP/UDP receive checksum offload Fragmented UDP checksum offload for packet reassembly Message Signaled Interrupts (MSI) Message Signaled Interrupts (MSI-X) Interrupt throttling control to limit maximum interrupt rate and improve CPU usage Receive packet split header Multiple Receive Queues (RSS) 8x8 and 16x4 32 transmit queues Receive header replication Dynamic interrupt moderation DCA support TCP timer interrupts No snoop Relaxed ordering Support for 16 Virtual Machines Device Queues (VMDq) per port 8. Host Interface PCIe base specification 2.1 (2.5GT/s or 5GT/s) Bus width — x1, x2, x4, x8 64-bit address support for systems using more than 4 GB of physical memory 9. MAC Functions Descriptor ring management hardware for transmit and receive ACPI register set and power down functionality supportingD0 and D3 states A mechanism for delaying/reducing transmit interrupts Software-controlled global reset bit (resets everything except the configuration registers) Four Software-Definable Pins (SDP) per port Wake up IPv6 wake-up filters Configurable flexible filter (through NVM) LAN function disable capability Programmable memory transmit buffers (160 KB/port) Default configuration by NVM for all LEDs for pre-driver functionality 10. Manageability Eight VLAN L2 filters 16 Flex L3 port filters 25 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® 82599 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS Four Flexible TCO filters Four L3 address filters (IPv4) Advanced pass through-compatible management packet transmit/receive support SMBus interface to an external Manageability Controller (MC) NC-SI interface to an external MC Four L3 address filters (IPv6) Four L2 address filters See Intel® Ethernet Controller 10G X540 Datasheet for more details. 6. Dual Port Intel® 82599 10GbE I/O Module (AXX10GBNIAIOM) Figure 21. Dual Port Intel® 82599 10GbE I/O Module 6.1 Feature Set Following is the feature set: Intel® 82599 10GbE Controller 25x25mm chip Two 10Gb SFP+ Ethernet external connections Speed and Link/Act LEDs for each 10Gb SFP+ connection Support PXE, iSCSI, and FCoE functionalities 256k bit LAN EEPROM and 4Mbit SPI Flash Revision 1.4 26 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® 82599 10GbE I/O Module 256 Byte FRU EEPROM and TMP75 temp sensor The following SFP+ modules are supported: - Intel® Ethernet SFP+ SR Optics E10GSFPSR - Intel® Ethernet SFP+ LR Optics E10GSFPLR 6.2 Functional Block Diagram SFP+ Connector 1 2 x 40-pin connector PCIe Gen2 x8 Intel® 82599 SFP+ Connector 2 SPI Flash EEPROM Figure 22. Dual Port Intel® 82599 10GbE I/O Module Block Diagram Figure 23. Dual Port Intel® 82599 10GbE I/O Module Port Identification 27 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® 82599 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 6.3 Mechanical Dimensions Figure 24. Dual Port Intel® 82599 10GbE I/O Module Dimensions; Top and side View Revision 1.4 28 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® 82599 10GbE I/O Module Figure 25. Dual Port Intel® 82599 10GbE I/O Module Dimensions; Bottom View 29 Revision 1.4 Intel order number: G30021-004 Dual Port Intel® 82599 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS 6.4 Intel® 82599 10Gb Ethernet Controller 1. General Serial Flash Interface 4-wire SPI EEPROM Interface Protected EEPROM space for private configuration Device disable capability Package Size - 25 mm x 25 mm 2. Networking Complies with the 10 Gb/s and 1 Gb/s Ethernet/802.3ap (KX/KX4/KR) specification Complies with the 10 Gb/s Ethernet/802.3ae (XAUI) specification Complies with the 1000BASE-BX specification Complies with the IEEE 802.3x 100BASE-TX specification Support for jumbo frames of up to 15.5 KB Auto negotiation Clause 73 for supported mode CX4 per 802.3ak Flow control support: send/receive pause frames and receive FIFO thresholds Statistics for management and RMON 802.1q VLAN support TCP segmentation offload: up to 256 KB IPv6 support for IP/TCP and IP/UDP receive checksum offload Fragmented UDP checksum offload for packet reassembly Message Signaled Interrupts (MSI) Message Signaled Interrupts (MSI-X) Interrupt throttling control to limit maximum interrupt rate and improve CPU usage Receive packet split header Multiple receive queues (Flow Director) 16 x 8 and 32 x 4 128 transmit queues Receive header replication Dynamic interrupt moderation DCA support TCP timer interrupts NO snoop Relaxed ordering Support for 64 virtual machines per port (64 VMs x 2 queues) Support for Data Center Bridging (DCB) (802.1Qaz, 802.1Qbb, 802.1p) 3. Host Interface PCIe Base Specification 2.0 (2.5GT/s) or (5GT/s) Bus width — x1, x2, x4, x8 64-bit address support for systems using more than 4 GB of physical memory 4. MAC Functions Descriptor ring management hardware for transmit and receive ACPI register set and power down functionality supportingD0 and D3 states A mechanism for delaying/reducing transmit interrupts Software-controlled global reset bit (resets everything except the configuration registers) Eight Software-Definable Pins (SDP) per port Four of the SDP pins can be configured as general-purpose interrupts Revision 1.4 30 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel® 82599 10GbE I/O Module Wake up Ipv6 wake-up filters Configurable flexible filter (through EEPROM) LAN function disable capability Programmable memory transmit buffers (160 KB/port) Default configuration by EEPROM for all LEDs for pre-driver functionality Support for SR-IOV 5. Manageability Eight VLAN L2 filters 16 flex L3 port filters Four Flexible TCO filters Four L3 address filters (IPv4) Advanced pass through-compatible management packet transmit/receive support SMBus interface to an external manageability controller NC-SI interface to an external manageability controller Four L3 address filters (IPv6) Four L2 address filters See Intel® 82599 10 GbE Controller Datasheet for more details. 6.5 LAN LED Functionality Each of the two SFP+ Ethernet ports on the Dual Port Intel® 82599 10GbE I/O Module will have both a speed LED and a link/activity LED. Table 3. Dual Port Intel® 82599 10GbE I/O Module LED Functionality LED Color LED State Off NIC State Not used Green/Amber (Right) Amber 1 Gbps Green (Left) Green 10 Gbps On Active Connection Blinking Transmit / Receive activity Figure 26. Dual Port Intel® 82599 10GbE I/O Module; Rear View 31 Revision 1.4 Intel order number: G30021-004 Single Port FDR InfiniBand* ConnectX-3* I/O Module 7. Intel® I/O Expansion Modules for Intel® Platforms HWS Single Port FDR InfiniBand* ConnectX*-3 I/O Module (AXX1FDRIBIOM) Figure 27. Single Port FDR InfiniBand* ConnectX*-3 I/O Module 7.1 Feature Set Following is the feature set: Mellanox* Connect X-3* FDR/10GbE silicon Single QSFP+ external connector, standard style QSFP+ cage (not PCI) with EMI spring fingers and with custom heat sink and retention clip for IOM low profile form factor SMB bus isolation and power control to the QSFP+ external connector Link (green) and Activity (amber) LEDs for the QSFP+ connection 16Mbit SPI Flash 256Byte FRU EEPROM and TMP75 temp sensor (stuffed) Stainless steel EMI shield See Mellanox* Connect X-3* Datasheet for additional silicon and software level features supported. Revision 1.4 32 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 7.2 Single Port FDR InfiniBand* ConnectX-3* I/O Module Functional Block Diagram 1.2V VR 2 x 40-pin connector PCIe Gen3 x8 1.8V VR 1.0V VR InfiniBand* Controller ConnectX-3 QSFP+ Port SPI Flash 16M bits Figure 28. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Block Diagram Figure 29. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Port Identification 33 Revision 1.4 Intel order number: G30021-004 Single Port FDR InfiniBand* ConnectX-3* I/O Module 7.3 Intel® I/O Expansion Modules for Intel® Platforms HWS Mechanical Dimensions Figure 30. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Top and side View Revision 1.4 34 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Single Port FDR InfiniBand* ConnectX-3* I/O Module Figure 31. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Bottom View 7.4 ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI 7.4.1 Overview ConnectX-3 adapter devices with Virtual Protocol Interconnect (VPI) supporting InfiniBand* and Ethernet connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 Blade Server and Landed-on-Motherboard designs used in Enterprise Data Centers, High-Performance Computing, and Embedded environments. Clustered data bases, parallel processing, transactional services and high-performance embedded I/O applications will achieve significant performance improvements resulting in 35 Revision 1.4 Intel order number: G30021-004 Single Port FDR InfiniBand* ConnectX-3* I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS reduced completion time and lower cost per operation. ConnectX-3 with VPI also simplifies system development by serving multiple fabrics with one hardware design. 7.4.2 Key Features 1μs MPI ping latency Up to 56Gb/s InfiniBand* or 40 Gigabit Ethernet per port PCI Express 3.0 (up to 8GT/s) CPU offload of transport operations Application offload GPU communication acceleration Precision Clock Synchronization End-to-end QoS and congestion control Hardware-based I/O virtualization Dynamic power management Fibre Channel encapsulation (FCoIB or FCoE) Ethernet encapsulation (EoIB) 17mm X 17mm RoHS-R6 7.5 Network LED Functionality The QSFP+ port on the Single Port FDR InfiniBand* ConnectX*-3 I/O Module will have a green LED and amber LED. The following is a mapping of LED color, function, and physical location with respect to the QSFP+ connector. Table 4. Single Port FDR InfiniBand* ConnectX*-3 I/O Module LED Functionality LED Color Amber (Right) Green (Left) LED State NIC State Off No Activity Blinking Transmit / Receive Activity Off No Active Link Connection On Active Link Connection Revision 1.4 36 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 8. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dual Port FDR InfiniBand* ConnectX*-3 I/O Module (AXX2FDRIBIOM) Figure 32. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module 8.1 Feature Set Following is the feature set: Mellanox* Connect X-3* FDR/10GbE silicon Dual QSFP+ external connectors, standard style QSFP+ cage (not PCI) with EMI spring fingers and with custom heat sink and retention clip for IOM low profile form factor SMB bus isolation and power control to the QSFP+ external connector Link (green) and Activity (amber) LEDs for the QSFP+ connection 16Mbit SPI Flash 256Byte FRU EEPROM and TMP75 temp sensor (stuffed) Stainless steel EMI shield See Mellanox* Connect X-3* Datasheet for additional silicon and software level features supported. 37 Revision 1.4 Intel order number: G30021-004 Dual Port FDR InfiniBand* ConnectX*-3 I/O Module 8.2 Intel® I/O Expansion Modules for Intel® Platforms HWS Functional Block Diagram 1.2V VR 2 x 40-pin connector PCIe Gen3 x8 1.8V VR 1.0V VR InfiniBand* Controller ConnectX-3 QSFP+ Port 1 QSFP+ Port 2 SPI Flash 16M bits Figure 33. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Block Diagram Figure 34. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Port Identification Revision 1.4 38 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS 8.3 Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Mechanical Dimensions Figure 35. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Top and side View 39 Revision 1.4 Intel order number: G30021-004 Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS Figure 36. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Bottom View 8.4 ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI 8.4.1 Overview ConnectX-3 adapter devices with Virtual Protocol Interconnect (VPI)supporting InfiniBand* and Ethernet connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 Blade Server and Landed-on-Motherboard designs used in Enterprise Data Centers, High-Performance Computing, and Embedded environments. Clustered data bases, parallel processing, transactional services and high-performance embedded I/O applications will achieve significant performance improvements resulting in Revision 1.4 40 Intel order number: G30021-004 Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port FDR InfiniBand* ConnectX*-3 I/O Module reduced completion time and lower cost per operation. ConnectX-3 with VPI also simplifies system development by serving multiple fabrics with one hardware design. 8.4.2 Key Features 1μs MPI ping latency Up to 56Gb/s InfiniBand* or 40 Gigabit Ethernet per port PCI Express 3.0 (up to 8GT/s) CPU offload of transport operations Application offload GPU communication acceleration Precision Clock Synchronization End-to-end QoS and congestion control Hardware-based I/O virtualization Dynamic power management Fibre Channel encapsulation (FCoIB or FCoE) Ethernet encapsulation (EoIB) 17mm X 17mm RoHS-R6 Note: Due to PCIe3 x8 bandwidth limitation, both ports cannot run simultaneously at full speed. 8.5 Network LED Functionality The QSFP+ port on the Dual Port FDR InfiniBand* ConnectX*-3 I/O Module will have a green LED and an amber LED. Below is a mapping of LED color, function, and physical location with respect to the QSFP+ connector. Table 5. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module LED Functionality LED Color Amber (Right) Green (Left) LED State NIC State Off No Activity Blinking Transmit / Receive Activity Off No Active Link Connection On Active Link Connection 41 Revision 1.4 Intel order number: G30021-004