IONOS-IP-Cores Preliminary Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary OVERVIEW Helion's experience in image processing enables designers of video surveillance, automotive, and medical imaging systems to use pre-engineered high quality IP cores directly on their camera systems. The IONOS Imaging IP Cores can be used for pre- and post-processing, image sensor control, FPGA algorithms and DSP algorithms. Helion offers a comprehensive selection of video pipelines, ranging from basic to advanced monochrome and color pipelines, all the way through high resolution, advanced High Dynamic Range Imaging (HDRI) color pipelines. These cores also support Lattice FPGA devices, and are all compatible and simply connected with the Wishbone bus (direct connect without Wishbone on request). FEATURES • Supports image sensors up to 12MP resolution - Offers seamless upgrade path, protects investment - Direct sensor interfaces and setup • 1080p60 streaming data path through FPGA - Supports full HD at 60 frames per second - no external frame buffer required - Offers quality at lower system cost - Extremely Low latency • IP supports up to 192 dB (32Bit) scene dynamic range - Maximum details under difficult lighting conditionsin both dark and light areas in a single image - Exceeds the 150dB automotive manufacturerspecified requirements - HDRI Tonemapping available • Wishbone compatible IP and Mico32 support - Easy to setup and use - doubleclick interconnect solution - platform library and structure header files • Comprehensive IP Suite - End to end ISP solutions Helion is a member of to the ispLeverCORE™ group of independent IP providers who have teamed up with Lattice to bring customers the highest quality, reusable IP cores optimized for Lattice's unique line of FPGA devices. Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary IONOS-IP-CORE OVERVIEW DPC Defect Pixel Correction CSC Color Space Conversion BLC Adaptive Black Level Correction AWB Automatic White Balance CFI Color Filter Interpolation HDRI High Dynamic Range Imaging CCM Color Correction Matrix STAT Statistics OVL Overlay Υ Gamma Correction IONOS-IP-Cores are optimized for Lattice device architectures, resulting in fast, small cores that uses efficiently the latest Lattice architectures. Example Configuration of IONOS Image Processing Pipeline Image Sensor DPC BLC CFI CCM Υ Defect Pixel Correction Adaptive Black Level Correction Color Filter Interpolation Color Gamma Correction Correction WISHBONE Lattice Mico 32 Microprocessor The LatticeMico32™ is a highly configurable 32-bit Hardware architecture “soft” microprocessor core for Lattice FPGA devices Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary DPC Defect Pixel Correction Description: Dead or hot pixels are corrected with the defective pixel correction routine. This routine corrects the defective pixel with interpolated values based on neighbor pixels of the same color channel. Goal: Typical correction methods include detection of cold or hot pixels using either median or averaging estimation on immediate pixel neighborhood. Bayer-Pixel Array defect pixel - hot defect pixel - cold IP-Features - selectable bitwidth (1..32Bit) for input pixel data, HDR/WDR ready defect pixel - hot DPC - image resolution more than 10MP possible, HD ready (720p/1080p) - RAW monochrome (3x3) or RAW Bayer color (5x5) filter array - Median or mean value filter version available - correct delay for sync signals - implied synchronization - reset-input to allow local or global reset - dynamic setup with wishbone-interface - extrapolation or skipping of picture border pixels - switch-off of complete module (bypass function) - ready to use platform library and structure header files for Mico32 Pixel Clock Reset Frame-Sync Frame-Sync Line-Sync 12-Bit-RAW-Bayer Data 1920x1080 5x5 Median DPC Line-Sync Example Resource Utilization Lattice Diamond 1.0 INPUT-DATA OUTPUT-DATA RAW monochrome RAW monochrome or RAW bayer color or RAW bayer color XP2 -7C ECP2M -7C ECP3 -8C 1562 1562 1395 DSP-Sites 0 0 0 EBR-Blocks 8 8 8 135 158 140 Slices WISHBONE Max.Pixelclock in MHz after map trace Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary BLC counts Automatic Black Level Correction red blue green1 green2 Description: Each color channel has a time dependet offset. Color processing requires linear signal behaviours. Therefore all signals must be without any offset. CMOS image sensors have a so called dark rows output to measure this avarage offset for each color channel. Goal: Subtraction of the color channel-specific, line-depending base noise, to achieve an optimal black level result. values red offset blue offset IP-Features green1 offset green2 offset - selectable bitwidth (1..32Bit) for input pixel data, HDR ready - image resolution more than 10MP possible, HD ready (720p/1080p) - Sync signal synchronization to image without black rows/columns - reset - each channel processed separately - Dynamic Setup via wishbone-interface - adjustable integration time for black offset identification - selectable position of dark rows/columns Pixel Clock Reset Frame-Sync Frame-Sync Line-Sync 12-Bit-RAW-Bayer BLC Line-Sync Example Resource Utilization Lattice Diamond 1.0 INPUT-DATA OUTPUT-DATA RAW bayer color RGB bayer color or RAW monochrome or RAW monochrome XP2 -7C ECP2M -7C ECP3 -8C 1070 1074 1071 DSP-Sites 0 0 0 EBR-Blocks 0 0 0 210 239 270 Slices WISHBONE Max.Pixelclock in MHz after map trace Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary CFI Color Filter Array Interpolation Bayer-Pixel Array Sensor Array Description: Since each pixel has a filter with one of three colors (R/G/B), two-thirds of the color data is missing from each. To obtain a full-color image, various demosaicing algorithms can be used to interpolate a set of complete red, green, and blue values for each pixel. Goal: Full color information for each pixel of the sensor. Different versions are available. IP-Features - selectable bitwidth (1..32Bit) for input pixel data, HDR ready - image resolution more than 10MP possible, HD ready (720p/1080p) - Arbitrary picture size - Reset input - Sync signal synchronization - 3x3 bilinear, 5x5 high-quality or 3x3 smart debayer with edge dection - Dynamic setup with Mico32 - Border extrapolation - Definition of first phase - ready to use platform library and structure header files for Mico32 Pixel Clock Reset Frame-Sync Frame-Sync Line-Sync 12-Bit-RAW-Bayer Data 1920x1080 3x3 smart debayer CFI Line-Sync Example Resource Utilization Lattice Diamond 1.0 INPUT-DATA OUTPUT-DATA RAW bayer color RGB interpolated XP2 -7C ECP2M -7C ECP3 -8C 1028 1028 1028 DSP-Sites 0 0 0 EBR-Blocks 4 4 4 135 158 140 Slices WISHBONE Max.Pixelclock in MHz after map trace Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary CCM Color Correction Matrix Description: In order to provide high quality images, acquisition is most important in respect to all further processing steps. A whole variety of current image sensors possess incorrect color rendition due to so called crosscolor effects. This effects leads to wrong color images (e.g. green with too much blue). Goal: Modification of each single color channel by a 3x3 correction matrix. red-out =rr*red+rg*green+rb*blue green-out =gr*red+gg*green+gb*blue blue-out =br*red+bg*green+bb*blue D65 CCM IP-Features - selectable bitwidth (1..32Bit) for each color channel data, HDR ready - image resolution more than 10MP possible, HD ready (720p/1080p) - selectable bitwidth (1..32 Bit) for coefficients /selectable fixcommabits - Full color RGB in- and output - Auto saturation detection - Dynamic setup with wishbone interface - each matrix coefficient is a fixcomma number - switch off complete module (bypass function) - ready to use platform library and structure header files for Mico32 Example Resource Utilization Lattice Diamond 1.0 Pixel Clock Reset Frame-Sync Frame-Sync Line-Sync CCM Line-Sync 3x12Bit RGB data 16 Bit coefficient INPUT-DATA OUTPUT-DATA RGB RGB 12-Bit-RAW-Bayer XP2 -7C ECP2M -7C ECP3 -8C 1113 1113 1071 DSP-Sites 18 18 18 EBR-Blocks 0 0 0 278 274 319 Slices WISHBONE Max.Pixelclock in MHz after map trace Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com D65 Corrected Image IONOS-IP-Cores Preliminary GAMMA Gamma Correction Description: Pixels are iluminated in a linear way. To provide pixel data to common video systems a conversion to a non-linear value encoding may be needed. Goal: Conversion of a RGB-input signal from linear to non-linear value or otherwise under usage of arbitrary gamma correction factors. IP-Features Υ=1.0 - selectable bitwidth (1..32Bit) for each color channel data, HDR ready - image resolution more than 10MP possible, HD ready (720p/1080p) - selectable output datawidth (for RGB) - automatic interpolation between look up table entries (for use of smaller tables) - for all colors, one look up table which is mapped into the wishbone adress space - implied sync signal synchronisation - Reset-input for global/local reset - Separate correction for each single color channel - Dynamic Setup via wishbone-interface - flexible modification of correction parameters - ready-to-use platform library and structure header files for Mico32 - Gamma Compression or Gamma expansion possible to switch during processing Pixel Clock Reset Frame-Sync Frame-Sync Line-Sync Line-Sync Gamma INPUT-DATA Gamma Correction Υ=0.5 Example Resource Utilization Lattice Diamond 1.0 12-Bit-RAW-Bayer OUTPUT-DATA RGB RGB 3x(1..32 Bit) 3x(1..32 Bit) XP2 -7C ECP2M -7C ECP3 -8C Slices 53 53 68 DSP-Sites 0 0 0 EBR-Blocks 6 6 6 218 272 243 WISHBONE Max.Pixelclock in MHz after map trace Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary CSC Color Space Conversion and Chroma up/down sampling RGB Description: Many processes need other input format then RGB. Especially data formats with decreased size, achieved by color downsampling are commonly used (e.g. encoders). Data stream with RGB values for each pixel. Goal: Conversion of picture stream data into different data formats. pixel 1 IP-Features - selectable bitwidth (1..32Bit) for input pixel data, HDR ready - image resolution more than 10MP possible, HD ready (720p/1080p) - Sync signal synchronization - planar or interleaved data formats - selectable output formats: - RGB->YUV420P, YCrCb422 - Color space calculation and color up/down sampling - dynamic setup with wishbone-interface - extrapolation or skipping of picture border pixels - switch-off of complete module (bypass function) - ready to use platform library and structure header files for Mico32 - Output data sorting - YCrCb output format adjustment pixel 2 Frame-Sync Line-Sync R G G G B B B Y Cr WISHBONE YUV (4:4:4) YCrCb (4:4:4) YCrCb (4:2:2) YCrCb (4:1:1)/(4:2:0) YUV420P with external RAM Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com G B Y Y Y Y ... Cr Cr Cb Cb C o l o r information for each 2nd pixel due to chroma subsampling XP2 -7C ECP2M -7C ECP3 -8C 541 542 537 DSP-Sites 9 9 9 EBR-Blocks 3 3 3 115 130 128 Slices <-> RGB (4:4:4) <-> <-> <-> <-> Max.Pixelclock in MHz after map trace Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Full color information for each pixel Example Resource Utilization Lattice Diamond 1.0 OUTPUT-DATA <-> YUV (4:4:4) <-> YCrCb (4:4:4) <-> YCrCb (4:2:2) <-> YCrCb (4:1:1)/(4:2:0) <-> YUV420P with external RAM ... RGB (4:4:4) ->YUV420p (4:1:1) INPUT-DATA RGB (4:4:4) R R Y CSC Line-Sync RGB 4:4:4 YUV/YCrCb 4:2:2 Pixel Clock Frame-Sync pixel 3 ... R Cb Reset YUV IONOS-IP-Cores Preliminary HDRI High Dynamic Range Imaging High Dynamic Range Imaging HDRI Description: Fully configurable comprehensive High Dynamic Range (HDR) pipeline, works with industry standard HDR sensors, and delivers outstanding HDR performance in one of the industry‘s most innovative FPGA based HDR implementations. Goal: Extremely wide 120dB-HDR-IP ensures that no detail in dark areas is lost even when an intruder shines a flashlight directly into the camera lens, while HDR, working in close conjunction with a fast-auto-exposure rapidly adjusts exposure in changing light conditions to offer a system dynamic range of 170dB. IP-Features - selectable bitwidth (1..32 Bit) for each color channel - especially for HDR input data - selectable output bitwidth - image resolution more than 10MP possible, HD ready (720p/1080p) - embedded statistics engine - every 2nd frame - complete new setup of all embedded look Image Processing via BLENDFEST-HDRI Color-Pipeline up table - auto adaptive global and local tonemapping - no external memory needed, for real-time view Pixel Clock Reset Frame-Sync Frame-Sync Line-Sync HDRI Line-Sync INPUT-DATA 120dB HDR 20-Bit-RAW-Bayer 1920x1080 OUTPUT-DATA HDR - RGB Example Resource Utilization Lattice Diamond 1.0 ECP2M -7C ECP3 -8C Slices 8069 9172 DSP-Sites 120 118 EBR-Blocks 68 68 Max.Pixelclock in MHz after map trace 187 132 WISHBONE tonemapped RGB Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary IONOS- IP-CORE OVERVIEW IP-Block-Function Sub-Function Shortcut Options Color Correction CCM Color Correction Matrix Color Interpolation CFI-3B Color Interpolation, 3x3 Bilinear Color Interpolation CFI-3S Color Interpolation, 3x3 Smart Debayer Color Interpolation CFI-5HQ Color Interpolation, 5x5 HQ Debayer Defect-pixel Correction DPC-1M Defect Correction (1D) only monochrome Defect-pixel Correction DPC-2C Defect Correction (2D) color Defect-pixel Correction DPC-2M Defect Correction (2D) monchrome Gamma GC Gamma Correction Green-Channel Balancing GCB Defect Correction Green Channel Test Pattern TG Test Pattern Generator CSF-HDR Color Separation and Fusion for HDR Operations CSC-RGB/YCrCb Color Space Conversion RGB (4:4:4) ->YCrCb (4:2:2) CSC-RGB/YUV Color Space Conversion RGB (4:4:4) ->YUV420P (4:2:0) CSC-YCrCb/RGB Color Space Conversion YCrCb (4:2:2) -> RGB (4:4:4) AE-AFL Antiflicker AE-DRE Dynamic Range Enhancement AE-LDR Base Package Fast Response for LDR AE-HDR Base Package Fast Response for HDR AE-SIA Base Package Slow iterative Approach Auto White Balancing AWB Base Package Linearization (HDR-Mode) HDR-LIN (APTINAMT9M023/M033) Data Linearization for HDR operations HDR-LIN (APTINAMT9V022/023/032/033) Data Linearization for HDR operations COLOR PIPELINE COLOR SPACE CONVERSION & MANAGEMENT SENSOR SETUP & CONTROL Auto Exposure Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary IONOS- IP-CORE OVERVIEW IP-Block-Function Sub-Function Shortcut Options Noise Reduction NR Adaptive Noise Reduction, Temporal Noise Removal (+ext.RAM) Distortion Correction DC Distortion Correction (+ext.RAM) Dithering DI-1W Error Diffusion Dithering 1-Way / Color Reduction Dithering DI-2W Error Diffusion Dithering 2-Way / Color Reduction Aperture Correction AC Filter Kernel with 3x3 user matrix, e.g. Aperture Correction Lens Shading Correction LSC-1C Lens Shading Correction (Luminance) / 1 Channel Lens Shading Correction LSC-3C Lens Shading Correction (RGB) / 3 Channel Rotation ROT Image Rotation (+ext.RAM) Soft-Scaler X-Axis SOSC-X Soft Scaler X-Axis / without ext.RAM Soft-Scaler Y-Axis SOSC-Y Soft Scaler Y-Axis / without ext.RAM HDR/WDR BF-DRI-C-12 BLENDFEST-Dynamic Range Increase-Color, 12bit/ Color Channel HDR/WDR BF-DRI-M-12 BLENDFEST-Dynamic Range Increase-Monochrome, 12bit/Input Channel HDR/WDR BF-DRI-C-24 BLENDFEST-Dynamic Range Increase-Color, 24bit/Color Channel HDR/WDR BF-DRI-M-24 BLENDFEST-Dynamic Range Increase-Monochrome, 24bit/Input Channel Gamma Tone Mapping GTM-EDR1 GTM, HDR Adaptive Algorithm for Efficient Dynamic Range (EDR) Gamma Tone Mapping GTM-EDR2 GTM, LDR Adaptive Algorithm for Efficient Dynamic Range (EDR) Gamma Tone Mapping GTM-24 GTM, HDR up to 24 bit input / Static Gamma Luminance Gamma Tone Mapping GTM-10 GTM, LDR up to 10 bit input / Static Gamma Luminance IMAGE ENHANCEMENT SCALING PICTURE IMPROVEMENT & TONEMAPPING: WIDE DYNAMIC RANGE Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores Preliminary IONOS- IP-CORE OVERVIEW IP-Block-Function Sub-Function Shortcut Options APIX INT-AP-RX APIX-RX Interface and ROM Emulation APIX INT-AP-TX APIX-TX Interface and ROM Emulation Frame Rate Converter INT-FRC 30fps->60fps (+ext.SRAM) i²C Interface INT-i-WC/FC Fast Frame wise Configurator / every frame configuration I²C Interface INT-i-1024/S Slave with 1024 Byte Register / slave for FPGA side configuration Sensor Interface SI Image Sensor Setup: Aptina MT9V022/V023/V024/V032/V033/V034 MT9T031/P031/E001/N001/J001/J003 MT9M023/M033 Output Interface OI-DVI Output Interface for DVI Transmitter Output Interface OI-TFT Output Interface for TFT-Panel Output Interface OI-1120 Output Interface for BT1120 Output Interface OI-656 Output Interface for BT656 Output Interface OI-DSP Output Interface for DSP Wishbone WB-MI I²C like master interface Wishbone WB-SI I²C like slave interface (256 Registers) 8-bit CPU CPU-8 8-bit CPU with Wishbone mapping 32-bit soft-CPU CPU-s32 MICO32 based, with Helion features INTERFACING MICROCONTROLLER AND PERIPHERALS Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com IONOS-IP-Cores IONOS- IP-CORE OVERVIEW IP-Block-Function Sub-Function Shortcut Options Memory Interface MEM-INT-D1 Memory Interface DDR1 SDRAM Interface Controller Memory Interface MEM-INT-D2 Memory Interface DDR2 SDRAM Interface Controller Memory Interface MEM-INT-QD Memory Interface QDR DDR SDRAM Interface Controller Memory Interface MEM-INT Memory Interface SDR SDRAM Interface Controller Wishbone WB-HS-EBR Statistic Engine / Wishbone Histogram Statistics 128 tiles EBR based Wishbone WB-HS-LUT Statistic Engine / Wishbone Histogram Statistics 128 tiles LUT based Wishbone WB-MB/DM Wishbone Memory Block / Dual Port Memory Wishbone WB-SQ-IN Wishbone Slave Quad in port / 4x32 bit Input Port Wishbone WB-SQ-OUT Wishbone Slave Quad out port / 4x32 bit Output Port Character Map OSD-CM OSD Character Map (OSD-CM) 2048 Characters (256x8 128x16 64x32 32x64 16x 128 8x256) 16x16 Pixel per Character up to 4 OSD-CG (64 up to 256 different Characters) user defined resolution and position Character Generator OSD-GEN OSD Character/Symbol Generator (OSD-CG) 16x16 Pixel, 16 Colors(24 bit/ Color) 64 Characters/Symbols user defined Overlay OVL Overlay with 1 graphical object selectable color, transparency and dimension / Bitmap Parameter Inserting PI Data insertion into image stream (selctable) PERIPHERALS OSD / OVERLAY EMBEDDED DATA Copyright 2010 Helion GmbH 10-EN-IO-V0.06 Helion GmbH im Tec-Tower Bismarckstraße 142 47057 Duisburg Germany Fon Fax +49 203 306 1240 +49 203 306 1250 [email protected] www.helionvision.com