Invited talk at SEMICON 2010, December 1, Tokyo CMOS Integrated Silicon Nanophotonics: Enabling Technology for Exascale Computational Systems William Green Solomon Assefa Alexander Rylyakov Clint Schow Folkert Horst Yurii Vlasov Dr. Yurii A. Vlasov, Manager, Silicon Integrated Nanophotonics IBM TJ Watson Research Center, Yorktown Heights, NY © 2010 IBM Corporation Optical Interconnects in HPCS ¾ IBM Petascale & Exascale Computing http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Building Blue Waters Target :10PF Blue Waters will be the most powerful computer in the world for scientific research (tentatively scheduled for Summer of 2011) Blue Waters Blue Waters Building Block 32 IH server nodes 32 TB memory 256 TF (peak) 4 Storage systems 10 Tape drive connections IH Server Node 8 MCM’s (256 cores) 1 TB memory 8 TF (peak) Fully water cooled Multi-chip Module Power7 Chip 8 cores, 32 threads L-cache (32 MB) Up to 256 GF (peak) 45 nm technology 4 Power7 chips 128 GB memory 512 GB/s memory bandwidth 1 TF (peak) Router 1,128 GB/s bandwidth Approximately 10 PF/s peak More than 300,000 cores (homogeneous) More than 1 PetaByte memory More than 10 Petabyte disk storage More than 0.5 Exabyte archival storage More than 1 PF/s sustained on scientific applications Courtesy of Thomas Dunning http://www.research.ibm.com/photonics ~1 PF sustained >300,000 cores >1 PB of memory >10 PB of disk storage ~500 PB of archival storage >100 Gbps connectivity http://www.ncsa.illinois.edu/BlueWaters SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Optical Interconnect: 1.1 TB/s HUB; 1,000,000 links 192 GB/s Host Connection 336 GB/s to 7 other local nodes in the same drawer 240 GB/s to local-remote nodes in the same supernode (4 drawers) 320 GB/s to remote nodes 40 GB/s to general purpose I/O Avago microPODTM [M. Fields, Avago, OFC 2010, paper OTuP1] [A. Benner, IBM, OFC 2010, paper OTuH1] http://www.research.ibm.com/photonics http://www.ncsa.illinois.edu/BlueWaters/ SEMICON, December 1, 2010 Tokyo 4 © 2010 IBM Corporation Silicon Integrated Nanophotonics ¾ An Introduction http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Off-Chip and On-Chip Nanophotonics Interconnects Switches CMOS Driver Detector WDM WDM Modulator CMOS Amplifier Electricalto-Optical N parallel channels Optical-toElectrical N parallel channels CMOS Deserializer CMOS Serializer Message 1Tbps aggregate BW Message 1Tbps aggregate BW Core N Core 1 Goal: Integrate Ultra-dense Nanophotonics Circuits with CMOS chip http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Penetration of optics into HPCS rack back plane on-board on-chip WW volume in 2009 Blue Waters Roadrunner S C HP MareNostrum Single HPC machine will contain a similar number of optical channels as currently exist today in all parallel optical links worldwide Courtesy of M. Taubenblatt http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation CMOS front-end monolithic Nanophotonics integration M1 FET SOI Optical waveguide BOX Si Æ Nanophotonics sharing Si layer with FET body Advantages: λ1 λ1…λ6 λ2 9 Deeply scaled Nanophotonics 9 Most dense integration with CMOS 9 Ultra-low power optical interconnects 9 Same mask set, standard processing 9 Same design environment (e.g. Cadence) 9 Same EDA tools and design flow 9 Possible in-line system-level testing http://www.research.ibm.com/photonics λ3 λ4 λ5 λ6 6-channel WDM SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation IBM Silicon Nanophotonics – Scientific Impact (2003-2010) 2005 Slow Light 2006 Si Modulator 2007 Optical Buffer Journal papers: Conferences: Citation index: Patents: 2008 Si Switch >50 >150 >2,100 >30 2009 APD Detector 2010 Amplifier 2010 Ge Receiver (including 5 Nature, 6 Invited) (including 76 Invited/Plenary) World-class scientific work laid solid foundation for novel technology development http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Silicon Integrated Nanophotonics ¾ SNIPER project at IBM Research (Silicon Nanoscale-Integrated Photonic and Electronic tRansceiver) http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation IBM SNIPER Technology (2008) http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation IBM SNIPER Technology (2008) Goal: Nanophotonics devices integrated into CMOS FEOL process (with minimal change to CMOS flow and minimal additional masks) DETECTOR FEOL CMOS FLOW MODULATOR Shallow Trench Isolation Well implants/Activation Gate formation Source/Drain Activation FIBER COUPLER Silicidation 9Most of the mask levels and processing modules are shared 9Minimal additional photonics modules added Cu back-end wiring Over 30 base patents Photonics as a new feature in standard CMOS http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation SNIPER project at IBM Research ¾ CMOS Transceiver components (after integration of all Nanophotonics modules) http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation CMOS performance: Digital circuits CMOS Ring Oscillator in 130nm CMOS GR Design Performance • 65-stage RO • 10-stage divider • 4-stage amplifier Die photo 9Digital CMOS Circuitry integrated with Photonics modules 9130nm design rules 9Ring oscillator with 12ps delay per stage http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation CMOS performance: Analog circuits Receiver amplifier Design • DC-coupled common gate TIA Performance • 7-stage limiting amplifier LA • open-drain output buffer. Die photo TIA LA 5Gbps open eye 9Analog CMOS Circuitry integrated with Photonics modules 9130nm design rules 9Core area 170x40µm (TIA) and 160x50µm (LA) 928mW power consumption at 5Gbps (TIA and LA) [Assefa et al, JSTQE, September 2010] http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation CMOS performance: Analog circuits Transmitter modulator driver Design • pre-driver Performance • differential output buffer Die photo 5Gbps open eye 9Analog CMOS Circuitry integrated with Photonics modules 9130nm design rules 9Core area 170x60µm 936mW power consumption at 5Gbps http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation SNIPER project at IBM Research ¾ Nanophotonics Transceiver components http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Nanophotonics components: Waveguides Performance Design 9Formed in a standard STI process 9No additional masks required 9Ultra-high optical confinement (mode x-section 0.1µm2) Die photo 9 Waveguides integrated with CMOS Circuitry 9 193nm lithography with 65nm design rules 9 Bend loss 0.02dB/turn with µm-size bends 9 <3dB/cm loss 9 20µm pitch between waveguides 9Sharp bends as small as 2µm radius http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Nanophotonics components: Edge fiber coupler Packaging approach Design fiber Si substrate 8Tbps per 1mm of chip edge Die photo 9 Coupler arrays integrated with CMOS Circuitry 9 20um pitch 9 Coupling loss <1dB [S.Assefa et al, OE, April 2010] [B.Lee et al, OFC 2010, PDP] http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Nanophotonics components: WDM multiplexer Cascaded Mach-Zehnder WDM filter Design Performance • 3-stage MZI • 8 outputs • 3.2nm spacing • 2nm flat-top Die photo 9 WDM integrated with CMOS Circuitry 9 8 channels; 3.2nm spacing 9 2nm flat-top response; 1dB in-band ripple 9 Better than -10dB cross-talk 9 Temperature tolerant within ±10°C 9Footprint 360 x 170 µm 965nm design rules provide control of critical dimensions 9No active tuning required http://www.research.ibm.com/photonics [F.Horst et al, OFC 2010, Invited talk] [F.Horst et al, PTL, 21 (2009)] SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Nanophotonics components: Detectors Design Die photo Performance • Ge-first prior to SD anneal • Standard CMOS metallization • MSM junction to Ge-on-Insulator 40Gbps open eye at 2V bias 9Ge detector integrated with CMOS Circuitry 9Avalanche gain 10dB with only 1.5V bias 940GHz bandwidth with CMOS bias voltages [S.Assefa et al, Nature, March 2010] [S.Assefa et al, OE, April 2010] http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Nanophotonics components: Modulators Design Performance 500 nm Cu Cu Si3N4 W NiSi Si p i n Die photo Over 20GHz modulation bandwidth 9 Modulator integrated with CMOS Circuitry 9 Library of forward and reverse biased devices 9 Over 10Gbps modulation bandwidth [W.Green et al, OE, 2007] [J. Van Campenhout et al, OE, December 2010] http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Nanophotonics components: Switches Design Performance IN OUT OUT • 5-stage ring resonator design • Central ring with PIN diode • 2x2 electro-optical switch Die photo 5ns switching time 9Optical switch integrated with CMOS Circuitry 95ns switching time [Y.Vlasov et al, Nature Phot., April 2008] [J. Van Campenhout et al, OE, April 2009] [J. Van Campenhout et al, CLEO 2010] http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation SNIPER project at IBM Research ¾ Design and Layout (Nanophotonics and CMOS together) http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Co-design and Co-layout of CMOS and Nanophotonics components Ch 2 Ch 1 Mod 1 Ch 4 Ch 3 Mod 2 Mod 3 Ch 6 Ch 5 Mod 4 WDM-Demux Transmitter • Library of Nanophotonics components • Photonics-enabled design rules • Photonics-enabled DRC Mod 5 Mod 6 Det 1 Ch 1 Det 2 Ch 2 Det 3 Ch 3 Det 4 Ch 4 Det 5 Ch 5 Det 6 Ch 6 Receiver WDM-Mux Layout snapshot 9Allows to layout complex CMOS and Nanophotonics circuits http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation SNIPER project at IBM Research ¾ Integration density http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Integrated CMOS and Nanophotonics modules Transmitter: 6 WDM channels 96 channel WDM Transmitter 9Total area 3700µmx350µm 9Area limited by pad frame and decoupling capacitors 90.21mm2 per channel 1 2 3 5 4 6 Receiver: 6 WDM channels 96 channel WDM Receiver 9Total area 3700µmx500µm 9Area limited by pad frame and decoupling capacitors 90.31mm2 per channel 1 2 3 http://www.research.ibm.com/photonics 4 5 6 SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Comparison of IBM CMOS Nanophotonics with existing Si Photonics Si CMOS Photonics Others (Announced) Si CMOS Nano-Photonics IBM (Announcing now) 9 130nm design rules for CMOS 9 130nm design rules for CMOS 9 130nm design rules for Micro-Photonics 9 65nm design rules for Nano-Photonics 9 CMOS FEOL integrated (Ge-last after activation) 9 CMOS FEOL integrated (Ge-first prior to activation) 9 Large Litho variations - active tuning is required 9 Small Litho variations - active tuning not required 9 6mm2 per transceiver channel 9 0.5mm2 per transceiver channel IBM CMOS Nanophotonics technology: ¾ 10x higher integration density ¾ The only amenable for Terabit/s-class single-chip CMOS transceivers ¾ 50channels x 20Gbps = 1Tbps transceiver occupies only 5x5mm2 of a CMOS die (can be smaller than 2x2mm2 without pad frame and capacitors) http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Conclusions ¾ IBM has developed technology for monolithic integration of Nanophotonics WDM transceivers into the standard CMOS front-end ¾ IBM technology shows 10X improvement in integration density ¾ Enables low power Tbps-class CMOS transceivers for future Exascale systems ¾ Pursue integration into deeply scaled IBM CMOS processes http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation Acknowledgements Contributions of many of our colleagues throughout various organizations at IBM Research are gratefully acknowledged and especially Fengnian Xia, Leathen Shi, Jeffrey Sleight, Young-Hee Kim, Chris Jahnes and the staff at IBM MRL. http://www.research.ibm.com/photonics SEMICON, December 1, 2010 Tokyo © 2010 IBM Corporation