framework for digital camera in linux n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Framework for digital camera in Linux PowerPoint Presentation
Download Presentation
Framework for digital camera in Linux

Loading in 2 Seconds...

play fullscreen
1 / 58

Framework for digital camera in Linux - PowerPoint PPT Presentation


  • 310 Views
  • Uploaded on

Framework for digital camera in Linux. Dongsoo Kim ( dongsoo45.kim@samsung.com ) Heungjun Kim( riverful.kim@samsung.com ) Samsung Electronics. Contents. Introducing mobile camera devices SoC camera subsystem Mobile camera module devices Framework for digital camera

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Framework for digital camera in Linux' - onan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
framework for digital camera in linux

Framework for digital camerain Linux

Dongsoo Kim (dongsoo45.kim@samsung.com)

Heungjun Kim(riverful.kim@samsung.com)Samsung Electronics

contents
Contents
  • Introducing mobile camera devices
    • SoC camera subsystem
    • Mobile camera module devices
  • Framework for digital camera
    • Video4Linux2 and camera device
    • New APIs for high-end mobile camera devices
    • Dual camera framework for 3G handset
  • Porting issues in SoC camera subsystem
terms
Terms
  • AP : Application Processor
  • Camera interface : interface device which is syncing image data from external camera device and gives some commands to them
  • Camera module : external camera device packaged with lens, sensor and ISP on it
  • ISP : Image Signal Processor. Also supports lots of additional functions
  • V4L2 : Video for Linux 2
slide4

This document contains some animations.

Slideshow mode is recommended

what makes camera a camera
What makes camera “a camera”?
  • Strobe support
    • Various program mode : slow sync, red-eye reduction and on….
  • Exposure control
    • Aperture (Iris)
    • Shutter
    • ISO
  • Lens control
    • Focus
    • Zoom
slide7

History of camera phone

First Camera Phone in 2000 with 3.5MP

1.3MP in 2003

5MP in2004

3MP in2004

7MP in 2005

2MP in 2004

10MP in 2006

12MP in 2009

mobile camera module devices
Mobile camera module devices
  • Old and low-end mobile camera modules
mobile camera module devices1
Mobile camera module devices
  • Brand new and high-end mobile camera modules
soc camera subsystem
SoC camera subsystem
  • What is SoC camera subsystem?

ARM SoC solution

Peripherals

External

Camera module

Camera interface

Peripheral3 (USB)

ARM

Core

Peripheral2 (MMC)

Peripheral1(Display)

soc camera subsystem1
SoC camera subsystem

Request for data

Image data in expected format

Camera interface

Command

interface

Resolution handling

Data format handling

(YUV/RAW/JPEG…)

Buffer

handling

I2C

RAW DATA

Camera module

Exposure

control

Lens

control

White balance control

Effect control

Face

detect

Zoom

control

Strobe

control

.

.

.

soc camera subsystem2
SoC camera subsystem
  • Various ways of preview (Live view) in SoC camera subsystems

User Application

ARM

core

COPY!

M

e

m

o

r

y

Frame buffer

DATA

Camera

Interface

DATA

Camera

Module

OK

Cancel

soc camera subsystem3
SoC camera subsystem
  • Various ways of preview (Live view) in SoC camera subsystems

User Application

ARM

core

M

e

m

o

r

y

COPY!

Save

as file

DATA

D

M

A

Camera

Interface

DATA

Camera

Module

Frame buffer

Through DMA

OK

Cancel

soc camera subsystem4
SoC camera subsystem
  • How to capture still shots in SoC camera subsystem
    • Your camera module has a dedicated JPEG encoder on it

ARM

core

User Application

Camera module

DATA

COPY!

RAW

DATA

PROCESS

JPEG

PROCESS

M

e

m

o

r

y

DATA

Save as JPEG file

DATA

Camera

Interface

MEMORY

soc camera subsystem5
SoC camera subsystem
  • How to capture still shots in SoC camera subsystem
    • No JPEG encoder in your camera module

User Application

ARM

core

COPY!

RAW

DATA

M

e

m

o

r

y

DATA

Codec S/W

Camera

Interface

DATA

Camera

Module

Save as

JPEG file

soc camera subsystem6
SoC camera subsystem
  • How to take motion pictures in SoC camera subsystem

COPY!

Frame buffer

ARM

core

User Application

M

e

m

o

r

y

OK

Cancel

DATA

Codec S/W

Save as…

Camera

Interface

DATA

Camera

Module

COPY!

what we ve got in linux to control camera
What we’ve got in Linux to control camera
  • In Kernel
  • V4L2 (AKA Video For Linux 2)
    • The second version of the Video For Linux API
    • Kernel interface for analog radio and video capture and output drivers
  • In user space
  • Multimedia middleware
  • Gstreamer
    • Pipeline based multimedia framework written in the C programming language with the type system based on GObject
  • OpenMAX
    • Cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images
mobile camera module devices2
Mobile camera module devices

Is V4L2 enough for mobile digital camera modules?

No way

video4linux2 and camera device
Video4Linux2 and camera device
  • Camera subsystem in Linux kernel aspect

videobuf

videodev

V4L2

V4l2-dev

V4l2-ioctl

V4l2-int-device

(old)

V4l2-subdev

(new)

videobuf-core

videobuf-sg-dma

V4l2-device

Camera interface

V4L2 driver

Command

interface

Resolution handling

Data format handling

(YUV/RAW/JPEG…)

Buffer

handling

Camera module

V4L2 driver

Exposure

control

Lens

control

White balance control

Effect control

Face

detect

Zoom

control

Strobe

control

.

.

.

abstract view of v4l2 working with video capture device
Abstract view of V4L2 working with video capture device
  • How does it work with V4L2?

Application or Middleware

Open device

Copy memory to userspace

VIDIOC_QUERYCAP

VIDIOC_S_INPUT

VIDIOC_REQBUF

VIDIOC_QBUF

VIDIOC_QBUF

VIDIOC_DQBUF

VIDIOC_QUERYBUF

VIDIOC_STREAMON

new v4l2 apis for exposure control
New V4L2 APIs for exposure control
  • Need for detailed control in exposure
    • We’ve got camera devices with mechanical shutter
    • Iris could be handled also
    • ISO controlled output is somehow useful

Why not using enhanced exposure control like a regular digital camera?

new v4l2 apis for exposure control1
New V4L2 APIs for exposure control
  • What we’ve got in mainline V4L2 right now?

#define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)

enum  v4l2_exposure_auto_type {

        V4L2_EXPOSURE_AUTO = 0,

        V4L2_EXPOSURE_MANUAL = 1,

        V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,

        V4L2_EXPOSURE_APERTURE_PRIORITY = 3

};

#define V4L2_CID_EXPOSURE_ABSOLUTE  (V4L2_CID_CAMERA_CLASS_BASE+2)

#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)

  • No way to control iris
  • No way to control exposure metering method
new v4l2 apis for exposure control2
New V4L2 APIs for exposure control
  • Additional exposure control

enum v4l2_exposure_auto_type {

V4L2_EXPOSURE_AUTO = 0,

V4L2_EXPOSURE_MANUAL = 1,

V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3,

+ /* Additional features for digital camera */

+ V4L2_EXPOSURE_ISO_PRIORITY =4,

};

#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)

#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)

. . .

+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)

new v4l2 apis for exposure control3
New V4L2 APIs for exposure control
  • Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products

+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)

+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)

+#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21)

Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU

{

    .id = V4L2_CID_CAM_APERTURE,

    .type = V4L2_CTRL_TYPE_MENU,

    .name = "Aperture",

    .minimum = 0,

    .maximum = 4,

    .step = 1,

    .default_value = 0,

},

static const char *camera_iris_stages[] = {

/* This module supports 5 Iris stages 

* on it's own but F number depends 

* how you package Lens module

*/

"2.8“, "4“, "5.6“, "8“, "11“, NULL

};

new v4l2 apis for exposure control4
New V4L2 APIs for exposure control
  • New exposure metering control

+/* Exposure Methods */

+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)

+enum v4l2_photometry_mode {

+ V4L2_PHOTOMETRY_MULTISEG = 0,

+ V4L2_PHOTOMETRY_CWA = 1,

+ V4L2_PHOTOMETRY_SPOT = 2,

+ V4L2_PHOTOMETRY_AFSPOT = 3,

+};

new v4l2 apis for lens control
New V4L2 APIs for lens control
  • Need for enhanced Focus mode API
    • Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam
    • High-end camera modules support for various focus mode presets like
      • AF Macro
      • AF-S / AF-C

For compatibility

Focus control API should be defined

new v4l2 apis for lens control1
New V4L2 APIs for lens control
  • Focus Mode & Enumeration values

#define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10)

#define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11)

+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)

+/* Focus Methods */

+enum v4l2_focus_mode {

+ V4L2_FOCUS_AUTO = 0,

+ V4L2_FOCUS_MANUAL = 1,

+ V4L2_FOCUS_MACRO = 2,

+ V4L2_FOCUS_CONTINUOUS = 3,

+};

what is object detection
What is object detection?
  • Object detection

Gee!

It recognized my face!

Hello everyone

what should we consider
What should we consider?
  • What do we set?
    • Detect mode
    • Type of object
    • How many objects to detect
    • Triggered action
  • What do we get?
    • How many objects are detected
    • Detected object’s type
    • Detected object’s coordinates
new v4l2 apis for object recognition
New V4L2 APIs for object recognition
  • New API for object recognition
    • New capability

#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */

#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */

+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000

  • New IOCTL

+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)

+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

new v4l2 apis for object recognition1
New V4L2 APIs for object recognition
  • New API for object detection

+/* Object detection and triggered actions */

+enum v4l2_recog_mode {

+ V4L2_RECOGNITION_MODE_OFF = 0,

+ V4L2_RECOGNITION_MODE_ON = 1,

+ V4L2_RECOGNITION_MODE_LOCK = 2,

+};

+

+enum v4l2_recog_action {

+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */

+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */

+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */

+};

+

+enum v4l2_recog_pattern {

+ V4L2_RECOG_PATTERN_FACE = 0; /* Face */

+ V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */

+ V4L2_RECOG_PATTERN_CHAR = 2; /* Character */

+}

new v4l2 apis for object recognition2
New V4L2 APIs for object recognition
  • New API for object detection

+struct v4l2_recog_rect {

+ enum v4l2_recog_pattern p; /* What pattern detected */

+ struct v4l2_rect o; /* detected area */

+ __u32 reserved[4];

+}

+

+struct v4l2_recognition_data {

+ __u32 detect_cnt; /* detected object counter */

+ struct v4l2_recog_rect obj; /* detected area */

+ __u32 reserved[4];

+};

+

+structv4l2_recognition {

+ enum v4l2_recog_mode mode;

+ enum v4l2_recog_pattern pattern; /* What pattern to detect */

+ __u32 obj_num; /* How many object to detect */

+ __u32 detect_idx; /* select detected object */

+ struct v4l2_recog_data data; /* read only */

+ enum v4l2_recognition_action action;

+ __u32 reserved[4];

+};

what should we consider1
What should we consider?
  • What do we set?
    • Detect mode
    • How many objects to detect
    • Triggered action
  • What do we get?
    • How many objects are detected
    • Detected object’s coordinates
new v4l2 apis for object recognition3
New V4L2 APIs for object recognition
  • New API for object recognition
    • New capability

#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */

#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */

+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000

  • New IOCTL

+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)

+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

new v4l2 apis for object recognition4
New V4L2 APIs for object recognition
  • New API for object detection

+/* Object detection and triggered actions */

+enum v4l2_recognition_mode {

+ V4L2_RECOGNITION_MODE_OFF = 0,

+ V4L2_RECOGNITION_MODE_ON = 1,

+ V4L2_RECOGNITION_MODE_LOCK = 2,

+};

+

+enum v4l2_recognition_action {

+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */

+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */

+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */

+};

+

+struct v4l2_recognition_data {

+ __u8 detect_cnt; /* detected object counter */

+ struct v4l2_rect o; /* detected area */

+};

+

+structv4l2_recognition {

+ enum v4l2_recognition_mode mode;

+ __u8 obj_num; /* How many object to detect */

+ __u8 detect_idx; /* select detected object */

+ struct v4l2_recognition_data data; /* read only */

+ enum v4l2_recognition_action action;

+};

dual camera for 3g handset
Dual camera for 3G handset
  • What is dual camera?
  • Dual camera applications
    • Video telephony
    • Camera applications
      • Taking self portrait shots

Front side camera

: for special purpose

Flip side camera

: for still shots

dual camera for 3g handset1
Dual camera for 3G handset
  • H/W restrictions in Application Processor’s camera peripheral interface

Mega

camera

VGA

camera

Data

path

why should we make single device node
Why should we make single device node?
  • H/W restrictions in Application Processor’s camera peripheral interface

“SINGLE” camera interface

on Application Processor !

Only one camera could be handled at a time

Only one camera could be using data pins at a time

Each camera is different in

Working MCLK

Working resolution (SYNC)

Working PCLK

what do we have in v4l2
What do we have in V4L2?
  • We already have API for dual camera
    • Enumerate video inputs
    •  Query or select the current video input

VIDIOC_ENUMINPUT

VIDIOC_G_INPUT

VIDIOC_S_INPUT

what do we need to consider for dual camera
What do we need to consider for dual camera?
  • What do we need to consider?
  • Let user space application get noticed about camera input devices
    • Don’t forget VIDIOC_ENUMINPUT
    • Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES
  • Context of each camera module
    • Capability : Supported color spaces, resolutions, additional functions…
    • Latest working status : resolution, color space, effect and so on
  • Context of camera interface
    • Working information of the camera module getting switched : Clock information and so on
example of switching camera
Example of switching camera

Megacamera

VGAcamera

  • How to switch active camera

Mega

Enable

VGA

Enable

  • Condition 1
  • How to enable camera module:
    • Give power source
    • Enable expecting camera
    • Give MCLK

Mega

Reset

VGA

Reset

CAM

POWER

MCLK

DATA PINS

CPU

  • Condition 2
  • Shared things
    • Camera power enable pin
    • Camera data pins
    • MCLK
  • Condition 3
  • Dedicated things
    • Camera enable pins
    • Camera reset pins
abstract work flow of switching camera
Abstract work flow of switching camera

Cam

power off

Camera (A) enable pin off

Camera (B) Enable pin on

Cam

power on

Switch to camera (B)

STREAMON

STREAMOFF

Camera (A)

initialize

Start preview

Stop preview

G_INPUT/

S_INPUT

Camera (B) initialize

Start preview

Camera interface reset & setting

Stop preview

Stop DMA

Reset camera interface’s current setting

Make a proper MCLK for camera (B)

Setup resolution, color-space, DMA

i2c issue
I2C issue
  • I2C protocol has start and stop condition
  • Some camera modules generate unexpected noise until their power up sequence
    • Makes I2C bus in bus busy state
i2c issue1
I2C issue
  • How to prevent this malfunctioning issue

Use a “noise free on power up” camera module

OR

Use a level shifter to block and ignore noise from camera module

OR

Change functionality of I2C pins while turning on camera module

resolution issue
Resolution issue
  • All about between camera interface and camera module
    • Expected resolution from camera interface
    • Served resolution from camera module
    • Symptoms depend on camera interface
  • Expecting resolution is bigger than served resolution
    • A “select time out” error in user space application or
    • Preview image is floating in some direction
  • Serving resolution is bigger than expected resolution
    • A “select time out” error in user space application or
    • Preview angle looks like to be zoomed in
color space issue
Color space issue
  • Color order mismatched between camera interface and camera module
lens control issue
Lens control issue
  • Preview freezing while checking lens response
    • Using while() loop while it checks response, preview will freeze until focusing job gets finished
    • If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing.
lens control issue1
Lens control issue
  • Solving preview freezing while checking lens status

struct platform_isp {

const struct platform_data *pdata;

struct v4l2_int_device *v4l2_int_device;

struct i2c_client *i2c_client;

int reschk;

. . . . . . . . .

}

static int ioctl_s_ctrl(struct v4l2_int_device *s,

struct v4l2_control *vc)

{

struct platform_isp *isp = s->priv;

struct i2c_client *client = isp->i2c_client;

switch (vc->id) {

case V4L2_CID_FOCUS_AUTO:

isp->vfm = vc->value;

if (vc->value == V4L2_CID_FOCUS_MACRO) {

err = i2c_command_for_isp( client, focus_macro );

isp->reschk = 1;

}

break;

}

}

If isp->reschk is Set,

The kernel thread is active.

lens control issues
Lens control issues
  • Solving preview freezing while checking lens status

int pseudo_reschk_thread (void *data)

{

int err = 0;

………

do {

switch (isp->reschk) {

case 1:

err = i2c_command_for_isp(client, focus_check);

if (err < 0) {

isp->reschk = 1; /* i2c command function failed. */

} else {

isp->reschk = 0; /* if succeed, reschk ended. */

}

break;

……

}

} while(1);

return 0;

}

Platform Data Vairable forisp, or

static variable.

This function return when the ISP is

matched the focus(+ return) ortimeout(- return).

This is able to re-check to the next check time.

jpeg data synchronization issues
JPEG data synchronization issues
  • JPEG has no synchronization protocol when it gets transferred
  • All about camera interface and camera module’s data synchronization
    • When camera interface supports JPEG sync
      • Follow user manual and make camera module to follow that protocol
    • When camera interface does not support JPEG sync
      • Make synchronization protocol: make JPEG data to be synchronized with VSYNC