1 / 43

ISP Introduction

ISP Introduction. 2007-10-26. Purpose. The objective of this presentation is to describe the image processing algorithms of Digital Camera. Most of the algorithms along the data path will be described.

flahertya
Download Presentation

ISP Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ISP Introduction 2007-10-26

  2. Purpose • The objective of this presentation is to describe the image processing algorithms of Digital Camera. Most of the algorithms along the data path will be described. • The intended audiences include algorithm designers, software programmers, IC engineers, application engineers, and project managers as well.

  3. ISP functional block diagram

  4. Black level with G1/G2 filter • This function allows user to adjust the black level so that blacks appear as true blacks. • Formula: • R’ = R + BL_R_OFFSET • G1’ = G1 + BL_G1_OFFSET + α*R • G2’ = G2 + BL_G2_OFFSET + β*B • B’ = B + BL_B_OFFSET • where BL_R_OFFSET, BL_G1_OFFSET, BL_G2_OFFSET, BL_B_OFFSET, α, β are configurable parameters.

  5. DPD/DPC and Noise Reduction • The image from sensor may have noise and dead pixels. This module can do noise cancellation and DPC at the same time. diff40 = abs(p4-p0); diff41= abs(p4-p1); diff42 = abs(p4-p2); diff43 = abs(p4-p3); diff45 = abs(p4-p5); diff46 = abs(p4-p6); diff47 = abs(p4-p7); diff48 = abs(p4-p8); if((diff40>dpc_th) && (diff41>dpc_th) && (diff42>dpc_th) && (diff43>dpc_th) && (diff44>dpc_th) && (diff45>dpc_th) && (diff46>dpc_th) && (diff47>dpc_th)) Is_dead = 1; else is_dead = 0;

  6. If it is a dead pixel, do dead pixel cancellation (DPC) DV = abs(2*p4-p1-p7); DH = abs(2*p4-p3-p5); DDL = abs(2*p4-p0-p8); DDR = abs(2*p4-p2-p6); if((DV<=DH)&&(DV<=DDL)&&(DV<=DDR)) new = (pix1+pix7+1)/2; else if((DH<DV)&&(DH<=DDL)&&(DH<=DDR)) new = (pix3+pix5+1)/2; else if((DDL<DH)&&(DDL<DV)&&(DDL<=DDR)) new = (pix0+pix8+1)/2; else new = (pix2+pix6+1)/2;

  7. If current pixel is not dead pixel, do noise cleaning DV2 = abs(2*p4-p1-p7); DH2 = abs(2*p4-p3-p5); DDL2 = abs(2*p4-p0-p8); DDR2 = abs(2*p4-p2-p6); if ((DV2<=DH2)&&(DV2<=DDL2)&&(DV2<=DDR2)) avg = (p1+p4+p7) / 3; var = abs(p1-avg) + abs(p4-avg) + abs(p7-avg); else if((DH2<DV2)&&(DH2<=DDL2)&&(DH2<=DDR2)) avg = (p3+p4+p5) / 3; var = abs(p3-avg) + abs(p4-avg) + abs(p5-avg); else if((DDL2<DV2)&&(DDL2<DH2)&&(DDL2<=DDR2)) avg = (p0+p4+p8) / 3; var = abs(p0-avg) + abs(p4-avg) + abs(p8-avg); else avg = (p2+p4+p6) / 3; var = abs(p2-avg) + abs(p4-avg) + abs(p6-avg); if ( it is noise ) output = avg ; //? else output = p4 ;

  8. Digital Gain There are two types of digital gain: global digital gain and R/G/B digital gain. The digital gain control is implemented on the input RGB Bayer pattern stream. All the pixel data (R, G and B) are multiplied by a digital global gain. The higher the global gain is, the brighter the image is. The global gain is used to adjust image brightness in Auto Exposure (AE) function. Different from digital global gain, digital RGB gains affect RGB pixel separately. Different color channel uses different gain value, so there are three gain values for R, G and B pixels, respectively. Digital RGB gains are used in Auto White Balance (AWB) function. Implementation: R’ = R * R_gain * Global_gain; G’ = G * G_gain * Global_gain; B’ = B * B_gain * Global_gain;

  9. Lens Shading Correction • This module is used to compensate for the shading effect, which causes the image intensity to get darker on the outer range of the image. We will apply a different gain value for each different (x, y) position in the image. The algorithm works on Bayer image, and works separately on R/G/B channel.

  10. CFA Interpolation In Bayer format, each pixel has only one of R/G/B values. By CFA interpolation, each pixel will have complete R/G/B values. Interpolation of G at R/B location if (DH < DV) else Interpolation of G at B location can be done on the corresponding formula of B values.

  11. CFA Interpolation Interpolation of G at R/B location Interpolation of G at B location can be done on the corresponding formula of B values.

  12. CFA Interpolation Interpolation of R at B location Interpolation of B at R location can be done on the corresponding formula of B values.

  13. Color Correction • Color correction is done using a color matrix and it compensates color deviation due to color filtering and sensing circuits.

  14. RGB Gamma Correction • The transfer function of most CRT displays produces an intensity that is proportional to some power (referred to as gamma) of the signal amplitude. As a result, high-intensity ranges are expanded and low-intensity ranges are compressed.

  15. RGB Gamma Correction

  16. RGB to YCbCr • The conversion uses this formula: • Formula 3: RGB range: 0-255; YCbCr range: 0-255 • Floating-point: • Fixed-point:

  17. Edge Extraction and Sharpness Edge extraction is done on Y channel, and will be used for image sharpening. To calculate one pixel, a 3x5 filter is used.

  18. Edge Extraction and Sharpness The edge map (EM) is further modified through a lookup table (EMLUT). EMLUT is constructed as following: The modified edge map is then added back to the Y channel. That is, Y’ = Y + EMLUT(x).

  19. Edge Extraction and Sharpness When –x1 < x ≤ x1, we think x is the noise, not the edge. There are two methods to calculate the Y value in this range: one is Y = Y – m2x, the other is Y = Avg. DDH = abs(2*Ym,n – Ym,n-1 – Ym, n+1) ; DDV = abs(2*Ym,n – Ym-1,n – Ym+1,n) ; DDL = abs(2*Ym,n – Ym-1,n-1 – Ym+1,n+1) ; DDR = abs(2*Ym,n – Ym-1,n+1 – Ym+1,n-1) ; Find the minimum of then Avg takes the value of that direction.

  20. Edge Extraction and Sharpness To avoid over-shoot on edges, The Y’ is further modified as the following: Find the minimum of the 15 Y’s, denoted by Ymin, and the maximum of the 15 Y’s, denoted by Ymax. if ( Y’ >= Ymax ) Y_out = Ym,n + ( (Y’ – Ymax) >> clip_bits ) else if ( Y’ < Ymin ) Y_out = Ym,n + ( (Y’ – Ymin) >> clip_bits )

  21. False Color Suppression False color suppression is done on U, V channel.Y is not changed, U, V may be changed by false color suppression. Implementation: 1. get YUV422 data; 2. get edge on Y channel; 3. edge = abs(edge); 4. get temp by clipping the edge into [edge_min, edge_max]; Where edge_min and edge_max are user-programmable registers.

  22. u_gain v_gain 1 0 edge_min edge_max edge False Color Suppression • calculate u_gain and v_gain: • u_gain = K_edge*(edge_max - temp); • v_gain = K_edge*(edge_max - temp); • where K_edge=65536/(edge_max - edge_min) ; • 6. do false color suppression on U, V channel when edge is larger than edge_min: • temp_u=(u_gain*(temp_u-128))/65536+128; • temp_v=(v_gain*(temp_v-128))/65536+128; • if edge is less than or equal to edge_min, just output u, v directly. • 7. clip u, v into [0, 255] and output.

  23. BacklightingDetection and Compensation 1. Detection The shape of histogram is very special in backlight condition. Two peaks lie on the two sides of the histogram: one on the low light side, and the other on the high light side. According to this fact, we may detect the backlight condition. To detect Backlighting, we need histogram information , the mean brightness of the whole image (maybe weighted), iWinYmean and the mean brightness of 16 windows WinYmean[0] ~ WinYmean[15];

  24. BacklightingDetection and Compensation 1. Detection When the following adaptive condition is satisfied, it is regarded as in backlight condition: if(int_Sum_White_Rate*(iThresHold+int_Sum_Black_Rate)*20>=(BackLight_White_Thres*iThresHold*iThresHold)&& iWinYmeanMin<BackLight_WinMin_Thres) m_iBackLight = TRUE;

  25. BacklightingDetection and Compensation 2. Compensation When the backlight is detected, and the following adaptive condition is satisfied, it is regarded as needn’t add ET time or sensor gain or digital gain or all: if(int_Sum_Black_Rate*(iThresHold+int_Sum_White_Rate)*32<(Compensate_Black_Thres*iThresHold*iThresHold)|| iWinYmean>Compensate_WinMean_Thres|| int_Sum_White_Rate>((iThresHold*Compensate_White_Thres)>>4)|| iWinYmeanMin>Compensate_WinMinMean_Thres) m_iBkLNeedAddET = FALSE;

  26. Brightness Input: Y, U, V in the range of [0, 255], as source image BRIGHTNESS, in the range of [-128, 127], as brightness adjustment Output: Y, U, V in the range of [0, 255] Implementation: define CLIP255(x) ((x>255)?255:((x<0)?0:x)) Y’ = CLIP255(Y + BRIGHTNESS); U’ = U; V’ = V; r

  27. Contrast Input: Y, U, V in the range of [0, 255], as source image Output: Y’, U’, V’, in the range of [0, 255] Parameters: CONTRAST, in the range of [1, 63] Y_OFFSET, in the range of [-128, 127] Implementation: define CLIP255(x) ((x>255)?255:((x<0)?0:x)) Y’ = CLIP255(((Y*CONTRAST)>>5) + Y_OFFSET); U’ = CLIP255((((U-128)*CONTRAST)>>5)+128); V’ = CLIP255((((V-128)*CONTRAST)>>5)+128);

  28. Hue Input: YUV422 data, Y, U and V are in the range of [0, 255]. Register Hue_sin and Hue_cos are in the range of [-256, 256] Hue angle, within the range of [-45, 45] for case 2. Output: YUV422 data, Y, U and V are in the range of [0, 255]. Implementation: If U and V fall out of [0, 255], clipping operation is applied as following: Clipping(x) = (x<0) ? 0 : ( (x>255)?255:x )

  29. Saturation Input: YUV422 data, Y, U and V are in the range of [0, 255]. Output: YUV422 data, Y, U and V are in the range of [0, 255]. Parameter: Saturation ratio K, within the range of [0, 2]. Formula: If U and Vfalls out of [0, 255], clipping operation is applied as following: Clipping(x) = (x<0) ? 0 : ( (x>255)?255:x )

  30. Brightness, Contrast, Hue and Saturation Control

  31. Auto Expose The luminance of the image changes when the environment varies. In order to keep the luminance in the given range, we must adjust exposure time (ET) of the sensor. If the ET is small, changing ET by one unit (one flickering period) can make the brightness of image change much. In order to make the brightness of image change smoothly, digital global gain is combined with ET adjustment

  32. Auto Expose An image is divided into 16 windows with different weights. Ymean of the frame and Ymean of the 16 windows are calculated as described below.:

  33. Auto White Balance When the white paper moves from one light source to another light source, an image sensor sees different colors under different conditions. Similarly, when a digital camera is moved from outdoors (sunlight) to indoor fluorescent (or incandescent) light conditions, the color in the image shifts. To correct for light source color-temperature changes, the balance among red, green and blue has to be shifted. This “white balancing” is performed by the algorithm below.

  34. Auto White Balance The number (Counter) of the pixels which satisfy the following conditions and their sums (Rsum, Gsum and Bsum)) of red, green and blue are calculated along with image data stream. YbotReg<Y<YtopReg Abs(I)<ItopReg Abs(Q)<QtopReg To deal with the issue of big region of uniform color, we will add the following judgment if (Counter>=TH_GRAY_COUNT) //there is enough valid gray pixels To update RGain, BGain to adjust AWB else //there is not enough gray pixels Keep the Rgain, BGain of the last frame.

  35. Auto White Balance The average value (AwbRmean, AwbGmean and AwbBmean) of the tricolor is calculated by the following formulas: AwbRmean=Rsum/Counter AwbGmean=Gsum/Counter AwbBmean=Bsum/Counter RDiff = abs( AwbRmean-AwbGmean ) BDiff = abs( AwbBmean-AwbGmean ) Based on these differences, Rgain and Bgain will be updated. AWB range control register (AWB threshold1, AWB threshold2, AWB gain step) will control this operation.

  36. Auto White Balance 0 AWB threshold1 AWB threshold2 Rstep = 0 Rstep = 1 Rstep = AwbStep RgainReg – Rstep ( AwbRmean>AwbGmean) RgainNew = RgainReg + Rstep ( AwbRmean<AwbGmean) BgainReg – Bstep ( AwbBmean>AwbGmean) BgainNew = BgainReg + Bstep ( AwbBmean<AwbGmean)

  37. Advance Auto White Balance Data path Stage 1: The task is to adjust the R/G/B balance to an “acceptable” degree. Stage 2: The task is to further fine-tune the R/G/B balance to make it better.

  38. Advance Auto White Balance 1. Stage 1 of the AAWB • For each frame, IC will collect the statistics of one light source, i.e., the number of pixels falling into the region of that light source model, denoted by p_data[i] • p_light_weight[i] = p_data[i]>>COUNTER_SHIFT; • b) After every 5 frames, IC will send an interrupt to firmware, saying 5 statistics are ready. With these 5 new statistics, firmware will perform the following smoothing operations, to reduce the unpleasing effect caused by erroneous statistics: • for( i=0; i<SOURCE_LIGHT_NUMBER; i++) • weighted[i]=(cur_weight*p_light_weight[i] • +(8-cur_weight)*short_pre_weights[i])/8; • for(i=0; i<SOURCE_LIGHT_NUMBER; i++) • short_pre_weights[i]=weighted[i];

  39. Advance Auto White Balance 1. Stage 1 of the AAWB • c) Firmware will apply the smoothed weights on the pre-defined pre_gr_gain and pre_gb_gain to get the weighted gain values: gr_gain and gb_gain. As a by-product, we can also know the light source corresponding to the maximum weight. • d) Firmware will combine the weighted gains into color matrix, and write the new color matrix values into IC registers. 6 multiplications will be involved.

  40. Advance Auto White Balance 2. Stage 2 of the AAWB a) Calculate Rmean, Gmean, Bmean of the output from stage 1, satisfying both of the following conditions: rr<=TH_RGB && gg<=TH_RGB && bb<=TH_RGB Y>=YbotReg && Y<=YtopReg && abs(I)<ItopReg && abs(Q)<QtopReg At the same time, count the number of pixels satisfying both of the conditions. b) Calculate gain_gr, gain_gb: gain_gr=64*Gmean/Rmean, gain_gb=64*Gmean/Bmean. c) If the count in step a) exceeds a threshold TH_GRAY_COUNT, it means there are enough gray pixels in the image, thus gain_gr and gain_gb are reliable statistics. In this case, we will further smooth them with previous values: pre_gr_gain = (pre_gr_gain*3 + gain_gr + 2)/4; pre_gb_gain = (pre_gb_gain*3 + gain_gb + 2)/4;

  41. Gray model under 4 Light sources 545 540 535 A 530 D65 525 CWF 520 515 510 505 Indoor 500 510 520 530 540 550 560 570 580 590 600 610 Advance Auto White Balance 3. How to get the Statistics a) Get the luminance Y from RGB: Y = ( 77*r + 150*g + 29*b ) / 256; For those pixels satisfying (Y≥YbotReg) && (Y≤YtopReg) : b) Transform RGB color space to log(g/b), log(r/b) space with an LUT: horizontal axis is log(g/b), and vertical axis is log(r/b); x = log(g) – log(b); y = log(r) – log(b); c) Rotate x_y space by 45 degree. y’=y-x; x’=y+x;

  42. Other ISP Models Anti Flicker Motion Detection Temporal Noise Reduction Dither or Halftone Auto Focus …………

  43. Questionsand Thanks

More Related