Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge November 2014 Reference Design RD1204 Introduction Sony Corporation is a market leading CMOS image sensor provider. Sony has several high resolution image sensors with a proprietary serial Sub-LVDS interface. However, most off-the-shelf Image Signal Processors or Application Processors use industry standard interfaces, such as CMOS parallel or CSI-2. To solve this interface mismatch between the Sony image sensor and the ISP, Lattice Semiconductor created RD1130, Sub-LVDS Serial to CMOS Parallel Sensor Bridge Reference Design, which converted Sony Serial Sub-LVDS image sensors to a CMOS parallel interface bus. As image sensor resolutions and frame rates increase, a new challenge has developed: Most Image Signal Processors with CMOS parallel interfaces are running out of bandwidth as they are only capable of operation up to 150 MHz. As a result of this performance limitation, more Image Signal Processors are now adopting MIPI CSI-2 interfaces in order to receive video at higher bandwidths. This same CSI-2 interface is also what is used by Application Processors that have an embedded ISP. The Lattice sub-LVDS to MIPI CSI-2 Sensor Bridge reference design allows users to achieve higher imaging bandwidths by converting the Sony SubLVDS interface to MIPI CSI-2 bus to allow data transfers up to 3.2 Gbps. Figure 1. Sony Sub-LVDS to CSI-2 Interface Bridge Sub-LVDS MIPI CSI-2 4, 8 or 10 Data Lanes Sony Serial Sub-LVDS Image Sensor 4 Data Lanes Image Signal Processor MachXO3L 1 Clock Lane 1 Clock Lane XVS XHS Key Features • Supports four, eight or ten data lanes from Sony image sensor in 10-bit or 12-bit pixel widths • Recognizes major Sony IMX image sensor sync codes and can operate with IMX136, IMX178, IMX236, IMX226, IMX172, and others (Implemented in Appro camera DM388IPNC-IMX172) • Can generate XVS & XHS for IMX178 image sensors in slave mode • Interfaces to MIPI CSI-2 Receiving Devices with four data lanes up to 3.2 Gbps total bandwidth • Fits in an XO3L-4300 in small 3.8 mm x 3. 8 mm, 81-ball WLCS package* or other MachXO2 or MachXO3 packages * The 4- or 8-lane subLVDS interface fits in the XO3L-4300 81-ball WLCSP package. The 10-lane subLVDS fits into the 121 fcsBGA. © 2014 Lattice Semiconductor Corp. All Lattice trademarks, registered trademarks, patents, and disclaimers are as listed at www.latticesemi.com/legal. All other brand or product names are trademarks or registered trademarks of their respective holders. The specifications and information herein are subject to change without notice. www.latticesemi.com 1 RD1204_1.0 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Functional Description The Lattice Sub-LVDS to MIPI CSI-2 Interface Bridge Reference Design converts serialized, source synchronous Sub-LVDS data from a Sony image sensor to MIPI CSI-2. The interface bridge is capable of the following combinations of Sub-LVDS inputs and MIPI CSI-2 outputs. One should ensure that the maximum bandwidth of the image sensor drive mode does not exceed 3.2 Gbps for MachXO2 or MachXO3 devices. For example, a 4-lane sub-LVDS mode at 576 Mbps for each lane results in 2.3 Gbps. This configuration can be supported because it is less than 3.2 Gbps. Table 1. Modes of Operation Sony Sub-LVDS Input MIPI CSI-2 Output 4-lane, 10-bit RAW 4-lane, 10-bit RAW (RAW10) 4-lane, 12-bit RAW 4-lane, 12-bit RAW (RAW12) 8-lane, 10-bit RAW 4-lane, 10-bit RAW (RAW10) 8-lane, 12-bit RAW 4-lane, 12-bit RAW (RAW12) 10-lane, 10-bit RAW 4-lane, 10-bit RAW (RAW10) 10-lane, 12-bit RAW 4-lane, 12-bit RAW (RAW12) The Sony Sub-LVDS input consists of one sub-LVDS differential clock lane and up to 10 sub-LVDS differential data lanes. The Sub-LVDS signaling for the Sony image sensors is defined as a Vcm = 0.9 V and a Vdiff = 150 mV. The signals from each data and clock lane are received by the LVDS25 receiver in the MachXO3L. The MachXO3L LVDS25 receiver can accept a low differential swing and a range of common mode input voltages. The output from the MachXO3L is a MIPI D-PHY compatible interface supporting HS (High Speed) and LP (Low Power) modes using the external IO resistor network described later in this document. For HS mode, the external resistor network allows emulated IO support for SLVS200. For LP mode, the external resistor network supports bidirectional LVCMOS IO at 1.2 V. The Sub-LVDS data received at the input of the MachXO3L is deserialized in the FPGA using iDDRx4 gearbox primitives in deser.v. This converts each double data rate lane to a single data rate 8-bit bus at slower operating speeds within the FPGA. The data from each lane is then delivered to the sony_serial2parallel module (sony_serial2parallel.v), which converts the deserialized data from the Sony image sensor to 10-bit or 12-bit pixels. This module is a modified version of the RD1130, Sub-LVDS Serial to CMOS Parallel Sensor Bridge Reference Design. The output of this module is a large data bus of pixels with a size dependent on the number of Sony data lanes being used. A large data bus with multiple pixels is used to keep the internal speeds in the FPGA low. The fv and lv output indicates when a frame or line is active respectively. The dvalid_out output indicates when the large pixel data bus has valid pixels available. The multi-pixel bus is then routed to the csi2_tx module (csi2_tx.v), which is a modified version of the Lattice Parallel to RD1183, Parallel to MIPI CSI-2 TX Bridge. This module accepts the large multiple-pixel bus and serializes it into HS (High-Speed) data packets following the MIPI CSI-2 specification. Additionally, the design includes the IMX_Framer module (imx_framer.v). Some of the Sony image sensors require a method for controlling XVS (vertical) and XHS (horizontal) indication signals. For example, the Sony IMX178 can operate in a master mode where it controls XVS and XHS framing information internally or a slave mode where XVS and XHS are controlled via external source. Another example is the Sony IMX172, which only supports slave mode and requires XVS and XVS to be driven by external control source. If external XVS and XHS control is required by the image sensor used, the IMX_Framer module (imx_framer.v) provides this control from within the FPGA. 2 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Figure 2. Top-Level Block Diagram sublvds_to_csi2.v rstn deser.v sclk serial2parallel.v csi2_tx.v CLK fv CH[lane_width-1:0] lv dvalid_out q[lane_width*8-1:0 DCK LPCLK[1:0] D3 LP3[1:0] D2[1:0] LP2[1:0] D1 LP1[1:0] D0 LP0[1:0] imx_framer.v XVS XHS V TOTAL[11:0] H TOTAL[11:0] V H BLANK[11:0] Table 2. Top Level Module Parameters Parameter Options Description bus_width 10, 12 Pixel bus width lane_width 4, 8, 10 Number of serial data lanes on Sony image sensor mode “master” Slave mode instantiates IMX_Framer module, which drive XVS and XHS control signals “slave” Master mode sets XVS and XHS to high impedance as so these signals are not controlled by the FPGA Any 12-bit value Number of INCK clock cycles XVS will be high for when mode = “slave” HTOTAL Any 12-bit value Number of INCK clock cycles XHS will be high for when mode = “slave” V_H_BLANK Any 12-bit value Number of INCK clock cycles XVS and XHS will be high for when mode = “slave” VTOTAL 3 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Table 3. Top Level Design Port List IO Port Direction Description rstn Input Resets all design modules within top level CLK Input Sony serial input clock CH[lane_width-1:0] DCK D[3:0] Input Sony serial data lanes Output MIPI CSI-2 serial HS clock lane Output MIPI CSI-2 serial HS data lanes LPCLK[1:0] Bidirectional MIPI CSI-2 LP control signals for clock lane LP3[1:0] Bidirectional MIPI CSI-2 LP control signals for data lane 3 LP2[1:0] Bidirectional MIPI CSI-2 LP control signals for data lane 2 LP1[1:0] Bidirectional MIPI CSI-2 LP control signals for data lane 1 LP0[1:0] Bidirectional MIPI CSI-2 LP control signals for data lane 0 INCK Input Input clock for XVS and XHS. This clock is shared as the input clock to the Sony image sensor XVS Input Sony slave readout vertical control signal XHS Input Sony slave readout horizontal control signal As described earlier, serial2parallel.v converts 8-bit data from the gearbox to pixel data on each data lane. There are two modules within serial2parallel.v which perform this conversion. The word aligner module receives the 8-bit deserialized data and converts it to 10-bit or 12-bit pixel data. The output of the module is a multi-pixel bus and a dvalid control signal. The dvalid control signal goes active high on clock cycles that have valid pixel data. The parser module looks at the recognition codes from the beginning and the end of each data packet to determine if they are part of an active video line or not. The fv goes high at the beginning of an active video frame and low at the end of the frame. Similarly, the lv goes active high or low at the beginning or end of an active video line respectively. The csi2_tx.v module contains several modules to perform the pixel to MIPI CSI-2 conversion. The first module is a PLL. The PLL is used to generate the serialized clock outputs. There are two serial clock outputs with a 90 degrees phase shift between them. This is used to center align the MIPI serial data with the clock in accordance with the specification. The next module is the byte_packetizer module. This module’s function is to convert pixel data into HS byte packets. The module creates a frame start or frame end MIPI HS short packet when the fv goes low or high respectively. The module additionally creates HS long packets based on the lv going high and pixel data being seen as valid based on dvalid. Additionally, the byte_packetizer handles appending the MIPI packet header and footer and calculating the appropriate, included checksums. LP_HS_DELAY_CNTRL provides all data control for the MIPI output data. This module allows the user to set delay controls for when LP and HS modes start and end on the clock and data lanes as these delays changed based on the speed of data transfer. Delay controls include the number of clocks difference between when the clock lane goes from LP to HS mode and when the data lanes go from LP to HS mode. All parameters are based on the MIPI byte clock which operates at a ¼ the speed of the MIPI bit clock output. 4 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Table 4. Low Power to High Speed Timing Control Parameters Parameter Description LP01CLK_dly Amount of time the clock lane retains the LP01 state at the beginning of a LP to HS transition LP00CLK_dly Amount of time the clock lane retains the LP00 state at the beginning of an LP to HS transition HS00CLK_dly Amount of time the clock lane retains the HS0 state at the beginning and end of an LP to HS transition HSXXCLK_dly Amount of time before the clock lane starts/ends transmitting the clock at the beginning and end of an LP to HS transition CLK2DATA_dly Amount of time that clock toggles before the data lanes start transitioning to HS mode LP01DATA_dly Amount of time the clock lane retains the LP01 state at the beginning of a LP to HS transition LP00DATA_dly Amount of time the clock lane retains the LP00 state at the beginning of an LP to HS transition HS00DATA_dly Amount of time the clock lane retains the HS0 state at the beginning and end of an LP to HS transition HSXXDATA_dly Amount of time before the clock lane starts/ends transmitting the clock at the beginning and end of an LP to HS transition LPHS_clk2data_dly Number of clocks to delay between the MIPI data lanes transitioning from LP to HS mode LPHS_startofdata_dly Number of clocks to delay the MIPI data from LP to HS mode transition HSLP_data2clk_dly Number of clocks to delay the HS to LP mode transition between the MIPI data lanes and MIPI clock lane HSLP_endofdata_dly Number of clocks to delay the MIPI data from the HS to LP mode transition sizeofstartcntr Size for the start timer counter. Number of bits to count LPHS_clk2data_diy+LPHS_startofdata_dly sizeofendcntr Size for the start timer counter. Number of bots to count LPHS_clk2data_diy+LPHS_startofdata_dly 5 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Figure 3. LP to HS Delay Control Parameters Adjustment Waveform Clock Lane P Channel Clock Lane N Channel Data Lane P Channel dly dly AT A_ dly LP 11 D K_ CL K2 CL HS 00 DA TA _ dly dly AT A_ 11 D 00 DA TA _ LP HS 00 01 LP LP CL K_ dly H S CLK 0 0 _d HS CL ly XX K_ d CL C LK l y K2 _ DA dly LP TA 01 _ D A dly TA _d ly LP 00 D HS ATA 00 _ D A dl y TA HS _d 00 ly DA TA _d ly Data Lane N Channel Packaged Design The Sony Sub-LVDS to MIPI CSI-2 Bridge Reference Design is available for MachXO3L and MachXO2 devices. The design is programmable for any of the described modes by adjustment of top level design parameters as previously explained. The packaged design contains a Lattice Diamond® project within the *\project\* directory. Verilog RTL source code is contained within the *\source\* directory. A bitstream for each mode is contained in *\bitstream\* matching the pin assignment described in the “Recommended Device Package and Pin Assignment” section of this document. The testbench and simulation projects are recommended to be accessed through the Simulation Wizard script within Lattice Diamond. This can be done by following the instructions in the Functional Simulation section of this document. 6 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Figure 4. Reference Design Directory Functional Simulation Functional simulation can be performed for any supported mode through the Lattice Diamond and Aldec ActiveHDL coder. With the project open in Diamond, double click on any of the Simulation Wizard scripts. Figure 5. Simulation Wizard Scripts 7 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge The Simulation Wizard window should open. Click Skip to End and then Finish to open Active-HDL. If a waveform viewer does not automatically open, one can be added by clicking File > New > Waveform. Once the Waveform viewer is open, click Design > Compile All to compile the design. Then click Simulation > Initialize Simulation. If prompted to choose the top level simulation file, choose the Verilog test bench (subLVDS_to_CSI2_tb_*bit_*lane) for the design. Next, add the signals to monitor to the waveform viewer from the design Hierarchy (typically the top level IO signals under subLVDS_to_CSI2_inst). Once this is done, click Run to run the simulation. Please note that the simulation will need to run for approximately 500 us before an output can be seen from the CSI-2 interface. This is because the first frame seen by the design is used to determine the line length. Once the line length is determined by the first frame, subsequent frames will be delivered out of the CSI-2 interface using this observed line length. Figure 6. Simulation of MIPI Short Packet and Long Packet Output Figure 7. Simulation of Full Frame Output Note that the IMX_Framer module operates independently of the subLVDS to MIPI test benches to simplify the simulations. In other words, the simulation of the subLVDS to MIPI interface will operate independently and at different operating speeds than XVS and XHS. There is a separate simulation wizard called “framer_tb.spf” which can be used to simulate the IMX framer standalone if desired. 8 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge External Resistor Network Implementation for MIPI D-PHY TX As described in the Lattice RD1182, MIPI D-PHY Reference IP, an external resistor network is needed to accommodate the LP and HS mode transitioning on the same signal pairs as well as the lower 200 mV common mode voltage during HS clock and data transfers. The resistor network needed for MIPI TX implementations supporting both LP and HS modes is provided below. Figure 8. Unidirectional Transmit HS Mode and Bidirectional LP Mode Interface Implementation Lattice FPGA MIPI DPHY RX Device 50 ohm DPHY TX Module LVCMOS12 iDDRx4 LVCMOS12 LVCMOS12 CLOCK_P 330 ohm CLOCK_N 50 ohm 50 ohm 330 ohm DATA0_P 330 ohm DATA0_N LVDS25E LVCMOS12 IO Controller 330 ohm LVDS25E 50 ohm 50 ohm LVCMOS12 330 ohm DATA3_P 330 ohm DATA3_N LVDS25E LVCMOS12 50 ohm 9 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Hardware Block Diagram for Sony Sensors Operating in Slave Mode Most image sensors require an outside clocking source. This is typically from an oscillator on the same PCB as the image sensor, but could be from another source such as the Image Processor as well. When operating the image sensor in slave mode it’s important to note that XVS and XHS signals should be driven at the same relative time and clock frequency as the sensor’s input clock. The easiest way to achieve this is by sharing the input clock between the image sensor and FPGA. Figure 9. INCK Clock Configuration for Sensor Operating in “Slave Mode” Sony Image Sensor (Slave Mode) Oscillator Lattice FPGA INCK Device Pinout and Bank Voltage Requirements Choosing a proper pinout to interface with another Sony and MIPI CSI-2 device is essential to meet functional and timing requirements. The following are rules for choosing a proper pinout on MachXO3L devices for Sony receiver input: • The input clock (CLK) an edge clock on bank 2 • The input clock and data signals (CLK and CH[3:0]) should only use A/B IO pairs on bank 2 • The VCCIO voltage for bank 2 should be 2.5 V • When in doubt, run the pinout through Lattice Diamond software and check for errors The following are rules for choosing a proper pinout on MachXO3L devices for CSI-2 transmitter output: • The output clock (DCK) an edge clock on bank 0 • The output clock and data signals (DCK and D[3:0]) should only use A/B IO pairs on bank 0 • The VCCIO voltage for banks 0 should be 2.5 V • LP signals (LPCLK, LP0, LP1, LP2, LP3) can use any other bank • The VCCIO voltage for the bank containing LP signals (LPCLK, LP0, LP1, LP2, LP3) should be 1.2 V • When in doubt, run the pinout through Lattice Diamond software can check for errors With the rules mentioned above a recommend pinout is provided for the most common packages chosen for this IP. For the MachXO2 the cs132bga is the most common package. The pinouts chosen below are pin compatible with MachXO2-1200, MachXO2-2000 and MachXO2-4000 devices. 10 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Table 5. Recommended Device Package and Pin Assignment MachXO2 2000/ 4000 cs132bga Package MachXO3L 4300 WLCS81 Package MachXO3L 4300 FC121 Package rstn L3 A5 H10 INCK K1 B5 J11 XVS M2 C1 H9 XHS M1 B1 J10 CLK_p N6 J6 L7 CLK_n P6 H6 L6 CH[0]_p M11 H9 K10 CH[0]_n P12 J8 J8 Signal Bank CH[1]_p P8 H8 K9 CH[1]_n M8 G7 L10 CH[2]_p P2 H7 K8 CH[2]_n N2 J7 L9 CH[3]_p N3 H5 L8 CH[3]_n P4 J5 H7 CH[4]_p M4 G4 J7 CH[4]_n N4 H4 K7 CH[5]_p M10 G3 K6 CH[5]_n P11 G2 K5 CH[6]_p M9 H3 L5 CH[6]_n N10 J3 J5 CH[7]_p P3 H2 K4 CH[7]_n M3 J2 L3 CH[8]_p P9 N/A H5 CH[8]_n N9 N/A K3 CH[9]_p N12 N/A K2 CH[9]_n P13 N/A L2 DCK_p A7 A6 B7 DCK_n B7 B6 B6 D0_p B5 D7 B11 D0_n C6 D6 A10 D1_p A2 D5 B9 D1_n B3 C6 C8 D2_p A10 A8 B8 D2_n C11 B8 A8 D3_p C12 E3 D5 D3_n A12 D3 A5 LPCLK[1] E12 F8 C9 LPCLK[0] E14 F7 C11 LP0[1] E13 F6 C10 LP0[0] F12 G9 D10 LP1[1] F13 G8 E9 11 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge MachXO2 2000/ 4000 cs132bga Package Signal Bank MachXO3L 4300 WLCS81 Package MachXO3L 4300 FC121 Package LP1[0] F14 F5 D9 LP2[1] G12 F4 E8 LP2[0] G14 B9 D11 LP3[1] G13 C8 E10 LP3[0] H12 D8 E11 Design Demonstration This design was tested using 4-lane sub-LVDS input and 4-lane DSI-Tx with the Appro IP Camera DM388IPNCIMX172. The device used is XO2 4000HE-6 csBGA132 and Sony IMX172 and IMX178. The image below shows the setup of the test performed. Figure 10. Appro IP Camera with IMX172 and MachXO2 with SubLVDS to CSI-2 Bridge Resource Utilization Table 6. MachXO2/MachXO3L Resource Utilization1 Sony Sub-LVDS Input 4-lane, 10-bit RAW MIPI CSI-2 Output Register LUT EBR PLL Clock Divider 4-lane, 10-bit RAW (RAW10) 877 1089 0 1 2 4-lane, 12-bit RAW 4-lane, 12-bit RAW (RAW12) 865 1369 0 1 2 8-lane, 10-bit RAW 4-lane, 10-bit RAW (RAW10) 1346 1658 5 1 2 8-lane, 12-bit RAW 4-lane, 12-bit RAW (RAW12) 1322 1944 6 1 2 10-lane, 10-bit RAW 4-lane, 10-bit RAW (RAW10) 1801 3233 6 1 2 10-lane, 12-bit RAW 4-lane, 12-bit RAW (RAW12) 1704 2558 7 1 2 1. Tested only with Synplify Pro 12 Sony Sub-LVDS to MIPI CSI-2 Sensor Bridge Internal Design Clocks Maximum Operating Frequencies by Configuration The maximum operating frequencies were obtained by post P&R timing analysis. They do not correlate to clocking ratios (obtained from PLL clock equations) used for proper design operation. Design Considerations Lattice recommends that users of this reference design use the information in the I/O Timing Report to determine the appropriate setup and clock to out values needed to/from the FPGA. Related Documents In addition to using this reference design document to understand the USB3 Video Bridge design solution, you can also refer to other documents that may contain more detailed information that is beyond the scope of this document • MIPI Alliance Specification for Camera Serial Interface 2 (CSI-2) V1.01 • MIPI Alliance Specification for D-PHY V1.1 • Sony IMX136, IMX144, IMX172, IMX178, IMX236 data sheets Technical Support Assistance e-mail: [email protected] Internet: www.latticesemi.com Revision History Date Version November 2014 1.0 Change Summary Initial release. 13