BRIDGING
THE
GAP
Impact and Correction of Camera Noise for Computational Microscopy including Precision Localization Nanoscopy
Keith Bennett, Ph.D., Stephanie Fullerton, Ph.D.,
Eiji Toda, Hiroyuki Kawai, Teruo Takahashi
ADAPTED FROM PRESENTATION GIVEN AT METHODS AND APPLICATIONS OF FLUORESCENCE,
GENOA, ITALY
SEPTEMBER 10, 2013
System Division
Cameras are NOT perfect!
Why is a camera manufacturer proclaiming
that cameras are not perfect?
Because NO camera is perfect
&
Because understanding why matters to your science
WHAT IS THE GAP?
The difference between the performance of an actual camera and a theoretically perfect camera
{
Perfect Camera
The GAP Actual Camera
THE AMOUNT OF THE GAP DEPENDS ON:
1. Sensor technology
2. Camera specifications
3. Input photon level
{
{
{
CCD
EMCCD
sCMOS
Quantum Efficiency
Camera Noise
Ultra low light
Low Light
Intermediate
High
{
Read noise
Excess Noise
Photo‐response non‐
uniformity (PRNU)
UNDERSTANDING WHY THERE IS A GAP ENABLES:
• Appropriate camera selection
• Optimized camera usage
• Optimized experimental design • More reliable data analysis
Better
Results
THE REAL CAMERAS
CCD
• Well established technology
• All electron to digital conversion done in one chain
• Limited speed
• Moderate read noise
• Very low dark current
• High QE
• Best pixel response uniformity
THE REAL CAMERAS
EMCCD
• Back‐thinned for increased QE
• High voltage gain register on sensor to achieve on‐chip amplification
• All electron to digital conversion through one chain(either for EM or no EM)
• Read noise is low due to gain
• Stochastic EM amplification adds excess noise and long tail
THE REAL CAMERAS
CMOS
• Newest technology
• Every pixel and column has own amplifier
• Very low mean rms read noise
• Pixel dependent read noise • Fastest speeds and largest field of view
• FPGA processing achieves excellent response uniformity (low PRNU)
SEEING THE
PREDICTED
GAP
Single Pixel Noise & SNR
Fixed Pattern Noise &
Image SNR
(from specs)
THE PERFECT CAMERA
100% QE 0 e‐
read noise
{ Every photon is converted into one electron
{ Every electron is digitized exactly as expected every time
0% fixed {
pattern noise
Every pixel and amplifier perform identically and predictably
In a perfect camera, the
SNR of a single pixel is limited only
by the physics of photon statistics…
i.e. shot noise.
Signal to Noise Ratio (SNR)
Perfect Camera Signal to Noise Ratio
100
10
1
0.1
1
0
10
100
1000
Signal (Photons)
10000
100000
EFFECT OF QE ON THE GAP
Signal to Noise Ratio
Signal to Noise Ratio (SNR)
100
Perfect
QE 70%
QE 50%
10
1
0.1
1
0
10
100
Signal (Photons)
1000
10000
A reduction in
QE reduces
SNR at all light
levels
RELATIVE SNR (rSNR) PLOTS CLEARLY SHOW THE GAP
1
Relative SNR (rSNR)
0.9
0.8
0.7
0.6
Perfect
0.5
QE 70%
0.4
QE 50%
0.3
0.2
rSNR is the SNR for a
camera plotted relative to
the perfect camera
rSNR shows differences
among cameras over full
range of signal level
0.1
0
0.1
1
10
100
1000
10000
Signal (Photons)
{
}
All SNR graphs in this talk will be presented as rSNR
THE SIMPLE SIGNAL TO NOISE RATIO (SNR)
QE: Quantum Efficiency
S: Input Signal Photon Number (photon/pixel)
F: Noise Factor (= 1 for CCD/sCMOS and √2 for EM‐CCD)
Nr: Readout Noise
M: EM Gain (=1 for CCD / CMOS)
Ib: Background
“Changing the Game”
EMCCDS: EXCESS NOISE IS THE REASON FOR THE GAP
SNR for CCD / CMOS
QE P
SNR
QE P
SNR for EM‐CCD
SNR
QE P
QE: Quantum Efficiency,
P:
Input Signal Photon Number,
M: EM Gain
Fn: Noise Factor
(assumes dark current and read noise are negligible)
M QE P
QE P
Fn 2
Fn M QE P
QEeff P
QEeff
QE QE
2
Fn
2
THE REAL CAMERAS
CCD
Sensor Type
EMCCD
CMOS
Charged Coupled Device
Interline Electron Multiplying CCD
Back‐thinned Complimentary Metal Oxide Sensor
Camera Name
ORCA‐R2 ImagEM x2
ORCA Flash4.0 V2
Pixel Number
1024 x 1344
512 x 512
2048 x 2048
6.45 µm x 6.45 µm
16 µm x 16 µm
6.5 µm x 6.5 µm
58 %
90 %
72 % 18 fps / 8 fps
70 fps
100 fps / 30 fps
Relative read noise (Nr/M), single‐frame rms
10 e‐ / 6 e‐
< 0.2 e‐ (M = 200)
1.9 e‐ / 1.3 e‐
Noise Factor (Fn)
1
√2 @ M>10
1
Pixel Size
QE ( @650 nm)
Frame Rate
MIND THE GAP: PREDICTED PIXEL rSNR PERFORMANCE FOR THE MOST
COMMON CAMERAS
2. {
The SNR of an EMCCD above 1 electron/pixel is comparable to a camera with QEeff =QE/2 due to excess noise from EM gain.
1
Relative SNR (rSNR)
1.{
A camera with the highest SNR at the lowest light level may not be the best at higher light levels
0.9
0.8
1.
0.7
0.6
2.
Perfect
0.5
sCMOS Flash4.0 (100 fps)
0.4
ORCA-R2 (CCD)
0.3
ImagEM X2 (EMCCD)
0.2
sCMOS Flash4.0 (30 fps)
ImagEM X2 (BT CCD mode)
0.1
0
0.1
1
10
100
1000
Signal (Photon, no background)
10000
= 650 nm
BEYOND THE SWEET SPOT: THE GAP EXPANDS AT HIGH LIGHT IF PRNU IS NOT CORRECTED
Single Frame rSNR
1.0
• PRNU reduces SNR at high light
0.9
Relative SNR
0.8
0.7
• Cannot be subtracted from image
0.6
0.5
• “Raw” PRNU varies by sensor
0.4
0.3
0.2
0.1
0.0
0.1
1
10
100
1000
10000
Signal (photons)
Model:
•
•
•
•
QE: 70%
Noise Factor (Fn): 1
Read Noise 3
photons rms
PRNU (s): 1%
{
Image
SNR
All SNR curves will be rSNR @ =650 nm
• Can be corrected in camera to
varying degrees
MEASURING
THE
REAL GAP
An in‐depth look at noise in CCD, EMCCD and CMOS cameras
ORCA‐R2 INTERLINE CCD: PREDICTABLE AND ROBUST
{
PRNU is insignificant
Bright Image:
shot noise limited
{
Read noise histogram has single Gaussian distribution Mean intensity: 17,300 e: 130.5ePRNU: not measurable
1.
2.
Read Noise
(Nr/QE)
10000
Count
1000
100
10
1
‐78 ‐60 ‐43 ‐25
‐8
10
27
Dark reading (ph)
45
62
80
=650 nm
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
‐
Shot Noise
& QE
58% QE Limit
10
100
1,000
signal (photon)
10,000
EMCCD: SOME SURPRISING RESULTS
{
1.
• Cannot be removed during
manufacturing
• Must be calibrated by users for their
specific spectrum.
• Individual pixel map required for
correction
850 nm
550 nm
Thickness variations from backthinning process causes spectrallydependent PRNU
Mean: 30157
s: 369.5 (1.2%)
Mean: 30508
s: 432 (1.4%)
Calculated Single Frame rSNR
1.00
0.90
0.80
0.70
rSNR
{
2.
The Gap for EMCCD in CCD mode becomes very wide due to PRNU
0.60
0.50
0.40
eQE: 95% (gain off)
Nr/eQE = 8.4 photons
PRNU: 1.4%
0.30
0.20
0.10
0.00
10
100
1000
Photons
10000
100000
COMPLEX BEHAVIOR: A CLOSER LOOK AT EMCCD SNR WITH HIGH
AND LOW GAIN
-
1
90% QE Limit
0.9
Excess Noise
0.8
0.7
Relative SNR
Excess noise (eQE)
PRNU
Saturation
High read noise
(34 e- @ M=5, 70 fps)
- Gain hard to measure
Saturation
Complex Behavior
0.6
0.5
0.4
0.3
Series1
Gain = 5
Gain = 400
Series2
0.2
0.1
0.01
0.1
0
1
10
100
Input photon number (photon)
1000
10000
EMCCD GAIN CAUSES UNEVEN PROBABILITY DISTRIBUTIONS
In simulated probability distribution functions for EMCCD, the output at high gain is not Poisson due to the electron multiplication process!
2 Photon Average Input
Gain = 200
10 Photon Average Input
Gain = 200
0.00025
0.0007
Probability [a.u.]
Probability [a.u.]
0.0008
Long tail
0.0006
0.0005
0.0004
0.0003
0.0002
0.0002
Long tail
0.00015
0.0001
0.00005
0.0001
0
0
‐5
0
5
Photon equivalent
10
0
5
10
15
20
Photon equivalent
25
30
ORCA‐FLASH4.0 V2 (SCMOS): A VERY COMFORTABLE SWEET SPOT
The “Sweet Spot”
Shot Noise
& QE
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
rSNR
70% QE Limit
1
10
100
1,000
Signal (Photons @650 nm)
10,000
NOT ALL CAMERAS ARE CREATED EQUAL: FLASH4.0 SWEET SPOT BROUGHT TO
YOU BY HAMAMATSU CAMERA ENGINEERS
Bright Image
{
Corrected Data {
Raw Data PRNU: ~2%
PRNU: ~0.5%
Bright Image
Signal amplified and digitized in column‐parallel ADC.
FPGA provides offset and gain correction to the raw digitized signal.
SCMOS: PIXEL‐DEPENDENT READ NOISE
1,000,000
Rms read noise matches
single frame rSNR.
Number of pixels 0.85 e‐Median
100,000
Single Frame Read
Noise (measured)
1.51 e‐ rms
10,000
1,000
100
(30 s/ row)
10
0
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
Temporal read noise (e‐, rms)
10000
1000
Does not fit a Gaussian
distribution, i.e. is not completely
100
10
modeled by a single “read noise.”
1
‐19.7
‐17.8
‐15.8
‐13.9
‐11.9
‐10.0
‐8.0
‐6.1
‐4.1
‐2.2
‐0.2
1.7
3.7
5.6
7.6
9.5
11.5
13.4
15.4
17.3
19.3
Number of pixels
Single Frame Dark Histogram
1
photon equivalent @ 650 nm
BRIDGING
THE GAP
Using knowledge of camera noise to get highest camera performance provides improved Precision Localization Results
sample contrast
WHAT IS
MOST
IMPORTANT?
frame rate
resolution
accuracy
background
TWO PHASES OF PRECISION LOCALIZATION MICROSCOPY
1. Collect Image Data
2. Reconstruct Superresolution Image
Prepare Sample Minimize Background
Optimize Optical System
Consider Camera Induced Noise
Calibrate Camera Implement Noise Corrections
Apply Statistical Algorithms
STANDARD PRACTICE IS NOT THE BEST PRACTICE: USING EMCCD WITH GAIN
YIELDS LEAST ACCURATE RESULTS
CCD QE: 100%, read noise = 1.8 ph, no background; No fixed pattern noise.
Adapted from: J. Chao et al (Ober Lab), Nat. Meth10, 2013) doi:10.1038/nmeth.2396
http://www.wardoberlab.com/
COMPENSATING READ NOISE VARIATION
Courtesy Prof. Joerg Bewersdorf, Yale University
Incorporating pixel‐specific read noise into the Maximum Likelihood Probability Model eliminates and narrows the asymmetric distribution of localized molecules caused by higher read noise pixels.
Courtesy F. Huang, Bewersdorf Lab
SELECTING AND USING CAMERAS: CASE STUDIES
{
{
{
Results
{
Accurate measurement of the distance between two fluorophores of different colors. distance ~0.77 nm using a dichroic beamsplitter to direct each color of light to separate halves of the CCD camera.
{
{
Camera
Correction
Measured PRNU maps for each color. Improved localization relative accuracy by ~2– 4 nm.
Details
Speed: 5 – 50 s / measurement
Light: ~4,000 – 10,000 ph/ mol/frame
~105 ph / mol before bleaching
Imaging: Simultaneous 2 color
Camera: Back‐thinned EM‐
CCD, gain off
Nature (2010)| doi:10.1038/nature09163
Cholera toxin B subunit Results
{
{
{
Camera
Correction
Details
scale bar: 1 m
Localization Microscopy with Minimal Bleaching. Plasma membrane dynamics for > 60 s (594 frames). 40% better 1 m
1 m
localization precision than “conventional” EMCCD localization Implemented detailed statistical EM noise model into maximum likelihood reconstruction probability model.
Speed: ~60s / reconstructed image
Light: ~100 photons /molecule frame
Mag: 630X
Camera: EM‐CCD, Gain ~1000
Courtesy of J. Chao et al (Ober Lab) Adapted from Nat Meth (2013) doi:10.1038/nmeth.2396
Results
{
{
{
32 reconstructed frames / sec (6.6 x 6.6 m2) field of view; fixed and living cells showed cellular dynamics not visible in reconstructions using longer data collection times.
Camera
Correction
Implemented pixel‐specific read noise into probability model for MLEM.
Details
Speed: 0.03 s/ reconstructed image
Light: ~3,000 photons /mol/ frame
Mag: 60X, Camera: sCMOS, 3200 fps
Courtesy of F. Huang,(Bewersdorf Lab)
Localization Precision “conventional” EMCCD vs. sCMOS
Courtesy of F. Huang. Bewersdorf Lab, Yale Adapted from F. Huang et al., Nature Methods 10(7): 653‐658 (2013) MINIMIZING THE GAP: MATCHING THE CAMERA TO YOUR
NEEDS
Light Required
Higher Accuracy
Better Resolution
Lower Sample Contrast
(BT)‐CCD
Scientific CMOS
“Conventional” Localization
EMCCD (Gain‐On)
EMCCD “Conventional” (Gain ON) Localization
UAIM
Speed / Field of View
Faster (or more pixels)
HAVE YOU DONE A GAP ANALYSIS?
1. How much light do I have?
{
{
{
The relative performance of CCD, back‐thinned CCD, EMCCD and sCMOS cameras is light‐level dependent.
2. Do I know my camera’s strengths and weaknesses?
No camera is perfect; proper use is required for the best results and to avoid errors
3. What is the goal of my experiment?
The most appropriate choice of camera depends upon the specific super resolution / localization experiment
Acknowledgements
Prof. Zhen‐li Huang, Huanzhong University of Science and Technology
F. Long et al, OPTICS EXPRESS 17741 (2012) Prof. Joerg Bewersdorf, Yale University
F. Huang et al., Nature Methods 10(7): 653‐658 (2013) See a movie of 32 fps dynamics in Prof. Raimund Ober, Texas Southwestern University
J. Chao et al, Nat Meth (2013) doi:10.1038/nmeth.2396
Hamamatsu
Hiroyuki Kawai: camera measurements
Teruo Takahashi: simulations
Stephanie Fullerton: presentation preparation
Keith Bennett
[email protected]
the supplementary materials of the article by Huang et al.