slide1
Download
Skip this Video
Download Presentation
触控与多点触控交互技术

Loading in 2 Seconds...

play fullscreen
1 / 120

触控与多点触控交互技术 - PowerPoint PPT Presentation


  • 230 Views
  • Uploaded on

触控与多点触控交互技术. 王锋 昆明理工大学 E-mail: [email protected] Phone: 13700600260. 提纲. 自然人机交互 (NUI) 触控技术的历史 触控研究的重点与目的 我们的工作. 人机交互 – UI. 用户界面 (UI) 技术是人机交互研究的热点问题之一。 一种新的人机交互设备的出现,都会引发用户界面技术的一次重要变化 用户界面在人机系统中负责计算机的输入和输出,产生必要的反馈,并直接影响最终用户对系统的使用。. What is HCI?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' 触控与多点触控交互技术' - emile


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

触控与多点触控交互技术

王锋

昆明理工大学

E-mail: [email protected]

Phone: 13700600260

slide2
提纲
  • 自然人机交互(NUI)
  • 触控技术的历史
  • 触控研究的重点与目的
  • 我们的工作
slide3
人机交互 – UI
  • 用户界面(UI)技术是人机交互研究的热点问题之一。
  • 一种新的人机交互设备的出现,都会引发用户界面技术的一次重要变化
  • 用户界面在人机系统中负责计算机的输入和输出,产生必要的反馈,并直接影响最终用户对系统的使用。
what is hci
What is HCI?
  • User:Am I error again? Computer for People? Or People for Computer???
slide6

董士海先生等曾提到过,用户界面:

    • 批处理 (BATCH PROCESSING)
    • 命令行 (CLI)
    • 图形用户 (GUI)
    • 自然用户界面(NUI)
slide7

当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着“用户自由”的方向发展。当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着“用户自由”的方向发展。

  • 用户逐渐成为交互关系中的主体,以个性化和场景自动感知的新的交互模式得到重视。

[1] J. Canny, "The Future of Human-Computer Interaction," Queue, vol. 4, pp. 24-32, 2006.

[2] B. Myers, "A brief history of human-computer interaction technology," interactions, vol. 5, pp. 44-54, 1998.

slide8

Shneiderman指出,新的计算技术将更为关注人能做什么,但是人的天性和需要并不会因为计算机的发明而改变。Shneiderman指出,新的计算技术将更为关注人能做什么,但是人的天性和需要并不会因为计算机的发明而改变。

  • 利用人与日常世界打交道时所形成的自然交互技能来获得计算机提供的服务更符合人的认知特点。
why nui
Why NUI?
  • GUI(MacOS、Windows操作系统)
  • GUI以桌面为隐喻,采用WIMP范式,但是频繁的菜单选择、按钮操作和键盘命令输入不符合人们的自然交互习惯,操作离散、界面布局复杂,认知负担重,对于非熟练的大众用户更是效率低下,WIMP界面已经成为制约用户生产力提高的瓶颈。
slide10

UI大师Alan Coope曾在著作《The Essentials of Interaction Design》中,在用户心理层次深刻地揭示了流状态:“深深的完全沉思状态”,经常产生“轻微的欢娱”,能够忘记时间的流逝,这就是自然用户界面的真实写照。

slide11

自然人机交互的主旨。

    • 软件交互应该促进和加强流状态,而不是打断或者干扰流状态。这就要求在用户的交互过程中,大部分交互行为都应在一种顺畅的状态下提供给用户。
    • 普适计算之父Mark Weiser, 不可见的交互技术。所谓的不可见,是说工具不会干扰人们的认知,人们的注意焦点是任务,而不是工具。
slide12

通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • NUI研究的关键要点: 在于如何充分利用人自身的自然输入属性,尽可能避免对使用者的大量训练。
  • N. Cross, "Natural intelligence in design," Design Studies, vol. 20, pp. 25-39, 1999.
  • C. Nass and B. Reeves, "Social and natural interfaces: theory and design," in CHI \'97 extended abstracts on Human factors in computing systems: looking to the future, Atlanta, Georgia, 1997, pp. 192-193.
input device vision goals 1945 2015
Input Device - Vision/Goals (1945-2015)

ImmediateIntermediateLong-term

  • Combined speech recognition, character recognition
  • Pen editing
  • Heuristic programming
  • Ubiquitous computing
  • Time sharing
  • Electronic I/O
  • Interactive, real- time system
  • Large scale information storage and retrieval
  • Natural language understanding
  • Speech recognition of arbitrary users
  • Natural user interface
  • Internet of things (C2C, H2C, H2H, H2C2H)
input devices overview
Input Devices (overview)

Sensor Devices

1. Spatial Position/Orientation Sensors

• 2DOF (Mouse)

• 3DOF (Microscribe, FreeD Joystick)

• 6DOF (Polhemus Fastrack)

2. Directional Force Sensors

• 5 DOF (Spacemouse)

• 2 DOF (Joystick)

3. Gesture Recognition

• Data Gloves

4. Eye Tracking

5. Speech Recognition Systems

the first mouse 1964
The First Mouse (1964)

Knee control

Douglas Engelbart

Years before personal computers and desktop information processing became commonplace or even practicable, Douglas Engelbart had invented a number of interactive, user-friendly information access systems that we take for granted today: the computer mouse, windows, shared-screen teleconferencing, hypermedia, groupware, and more.

input devices 1
Input Devices (1)

Directional Force Sensors

SpaceMaster

input devices 2
Input Devices (2)

Gesture Recognition

Dextrous Hand Master, Exos

SUPERGLOVE, Nissho

Cyberglove , 5th Dimension

input devices 3
Input Devices (3)

Spatial Position/Orientation Sensors

Polhemus InsideTrack

(Magnetic Tracking)

MicroScribe

(Mechanical Tracking)

FreeD Joystick

(UltraSonic Tracking)

input devices 4
Input Devices (4)

Visual Haptic Workbench

The Visual Haptic Workbench consists of five hardware components.

The dominant hand of the user experiences haptic feedback from the PHANToM, and the subdominant hand navigates through a menu interface via Pinch glove contact gestures. Head tracking is done with a Polhemus Fastrak receiver mounted on a pair of Stereographics CrystalEyes LCD shutter glasses. The subdominant hand can also be tracked with a separate receiver to facilitate more complex interaction paradigms. The audio subsystem gives the user additional reinforcement cues to clarify

[see also http://haptic.mech.nwu.edu/intro/gallery/]

historical overview 1945 1995
Historical Overview (1945-1995)

[source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]

slide21
提纲
  • NUI的发展与基本理解
  • 触控技术的历史
  • 触控研究的重点与目的
  • 触控技术的前景
touch multi touch
Touch / Multi-touch
  • 触控技术是单点触控与多点触控技术的总称
  • 自然人机交互中的重要组成部分
    • 在操作时不用借助其它媒介可直接触摸
    • Natural affordances
    • Offer a more compelling method to interact with a system than a mouse or other types of pointing devices
slide23
根基
  • 得益于Nakatani天才的软机器(Soft Machine, 1983)的想法
    • 通过手对屏幕上显示的各类操作组件(如按钮等)的操作,实现一种可变的、动态的交互。
    • L. H. Nakatani and J. A. Rohrlich, "Soft machines: A philosophy of user-computer interface design," in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, Boston, Massachusetts, United States, 1983, pp. 19-23.
history of multi touch
History of Multi-touch
  • 1982 - first multi-touch system called Flexible Machine interface developed in University of Toronto.
  • 1983 - Bell labs and Murray Hill published the first paper discussing touch-screen based interfaces.
video place
Video Place
  • 1983 - Video Place /Video Desk (Myron Krueger) A vision based system that tracked the hands and enabled multiple fingers,hands, and people to interact using a rich set of gestures
  • Myron’s work had a staggeringly rich repertoire of gestures, muti-finger, multi-hand and multi-person interaction.
history of multi touch1
History of Multi-touch
  • 1984 Multi-touch Screen (Bell labs, Murray Hill NJ) - integrated with a CRT on an interactive graphics terminal, could manipulate graphical objects with fingers with excellent response time.
  • Myron’s work had a staggeringly rich repertoire of gestures, muti-finger, multi-hand and multi-person interaction.
  • Microsoft began research in this area..
multitouch tablet 1985
Multitouch Tablet 1985
  • Input Research Group, University of Toronto
  • Touch tablet capable of sensing an arbitrary number of simultaneous touch inputs, reporting both location and degree of touch for each
sensor frame carnegie mellon university
Sensor Frame (Carnegie Mellon University)
  • The device used optical sensors in the corners of the frame to detect fingers.
  • At the time that this was done, miniature cameras were essentially unavailable.  Hence, the device used DRAM IC\'s with glass (as opposed to opaque) covers for imaging.
  • It could sense up to three fingers at a time fairly reliably (but due to optical technique used, there was potential for misreadings due to shadows.
1986 bi manual input
1986 Bi Manual Input
  • 1986 Bi Manual Input (University of Toronto) - able to position/scale task and selection/navigate task
apple desktop bus
Apple Desktop Bus
  • 1986 Apple Desktop Bus (ADB) and the trackball scroller Init(Apple Computer/University of Toronto)‏
  • E.g. The macintosh II and macintosh SE
digital desk 1991
Digital Desk 1991
  • (Pierre Wellner,  Rank Xerox EuroPARC, Cambridge) - supported multi-finger and pinching motions (leads to moderm product e.g. Iphone)‏
flip keyboard 1992
Flip Keyboard - 1992
  • Bill Buxton, Xerox PARC
  • A multi-touch pad integrated into the bottom of a keyboard.  You flip the keyboard to gain access to the multi-touch pad for rich gestural control of applications.
  • Graphics on multi-touch surface defining controls for various virtual devices.
history of multi touch2
History of Multi-touch
  • 1992:  Simon (IBM & Bell South) - A single-touch device relied on a touch-screen driven “soft machine”
  • 1992:  Wacom (Japan) - tablet that had multi-device/multi-point sensing capability
starfire inconceivable 5
Starfire - inconceivable [5]
  • 1992:  Starfire (Bruce Tognazinni , SUN Microsystems) - Bruce Tognazinni produced an future envisionment film, Starfire, that included a number of multi-hand, multi-finger interactions, including pinching, etc.
bimanual 1994 2002
Bimanual 1994-2002
  • Alias|Wavefront Toronto
  • Developed a number of innovative techniques for multi-point / multi-handed input for rich manipulation of graphics and other visually represented objects
graspable tangible interfaces 1995
Graspable/Tangible Interfaces 1995
  • Input Research Group, University of Toronto
  • Demonstrated concept and later implementation of sensing the identity, location and even rotation of multiple physical devices on a digital desk-top display and using them to control graphical objects.
active desk 1995 97
Active Desk 1995/97
  • Input Research Group / Ontario Telepresence Project, University of Toronto
  • Simultaneous bimanual and multi-finger interaction on large interactive display surface
t3 wavefront 1997
T3 – Wavefront 1997
  • A bimanual tablet-based system
  • Utilized a number of techniques that work equally well on multi-touch devices
haptic lens 1997
Haptic Lens 1997
  • By Mike Sinclair, Georgia Tech / Microsoft Research
  • A multi-touch sensor that had the feel of clay, in that it deformed the harder you pushed, and resumed it basic form when released. A novel and very interesting approach to this class of device.
fingerworks 1998
Fingerworks 1998
  • Inventor: Newark, Delaware
  • Produced a line of multi-touch products including the iGesture Pad. They supported a fairly rich library of multi-point / multi-finger gestures.
portfolio wall 1999
Portfolio Wall 1999
  • Alias|Wavefront,Toronto On, Canada
  • A digital cork-board on which images could be presented as a group or individually.
  • Its interface was entirely based on finger touch gestures that went well beyond typical touch screen interfaces.
diamond touch 2001 7
Diamond Touch 2001 [7]
  • Mitsubishi Research Labs,Cambridge MA)
  • Capable of distinguishing which person’s fingers/hands are which, as well as location and pressure
smartskin sony 2002
SmartSkin – Sony 2002
  • An architecture for making interactive surfaces that are sensitive to human hand and finger gestures.
  • This sensor recognizes multiple hand positions and their shapes by using capacitive sensing and a mesh-shaped antenna.
  • In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems.
jeff han
Jeff Han
  • FTIR multi-touch.
  • Very elegent implementation of a number of techniques and applications on a table format rear projection surface.
historical overview 1945 19951
Historical Overview (1945-1995)

[source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]

slide53
提纲
  • 自然用户交互
  • 触控技术的历史
  • 触控研究的重点与目的
  • 触控技术的前景
slide54
研究的主要方向
  • NUI的发展与基本理解
  • 触控技术的历史
  • 触控研究的重点与目的
    • 新的多点触控支撑技术的研究
    • 触控技术中的手势与界面设计研究
    • 不同领域的应用研究
  • 我们的工作
implementations
Resistive

Surface Acoustic Wave

Strain gauge

Optical Imaging

Dispersive Signal Technology

Acoustic Pulse

Frustrated Total Internal Reflection (FTIR)

Diffused Illumination (DI)

Capacitive

Shadow Capture

多点触控的实现-Implementations
frustrated total internal reflection
Frustrated Total Internal Reflection
  • Light emitting diodes produce light waves that travel through an acrylic touch screen. Under normal conditions, the light waves stay within the acrylic pane; however, when an object presses on the acrylic the light is scattered downward.
diffused illumination
Diffused Illumination
  • An LED light source is aimed at the screen. When objects touch the acrylic surface, the light reflects back and is picked up by multiple infrared cameras.
  • DI has the added capability for object detection.
  • Most often used in high end systems.
capacitive multi touch
Capacitive Multi-Touch
  • Many layers of protective, bonding, driving, sensing, substrate, and display surfaces
  • Two methods of picking up signal, Mutual capacitance or self capacitance
  • Sensor\'s \'normal\' capacitance field is altered by another capacitance field from the human finger, electronic circuits measure the resultant distortion
shadow capture
Shadow Capture
  • Under normal conditions, light continually travels through the acrylic pane and reaches the camera; however, when an object presses on the acrylic a shadow is cast downward.
architectural overview
Architectural Overview
  • A camera is used to pick up the light reflected when a contact is made with the acrylic screen. The camera is attached to a computer running software that reacts to each touch.
  • A rear projector is used to display the output from the computer. The screen serves two general purposes: to display the image from the projector, and also to absorb all of the light from the projector so that the user on the other side of the acrylic is not blinded.
software
Software
  • Any multi-touch interface requires computer software to manipulate the detected touches received by the camera. Additional programs are then needed to respond to each input and produce a desired output.
  • Touchlib is an open source library for creating multi-touch interaction surfaces. It handles tracking blobs of infrared light and sends programs multi-touch events, such as \'finger down\', \'finger moved\', and \'finger released\'. It interfaces with most major types of webcams and video capture devices.
  • Flash open sound control (FLOSC) is also used to connect Touchlib, which sends out OSC messages.
  • Programs created for multi-touch are generally written in a form of C or flash.
  • http://www.nuigroup.com
slide62
研究的主要方向
  • NUI的发展与基本理解
  • 触控技术的历史
  • 触控研究的重点与目的
    • 新的多点触控支撑技术的研究;
    • 触控技术中的手势与界面设计研究。
    • 不同领域的应用研究
  • 我们的工作
slide63

手势研究现状分析

    • Wu等提出了一系列的设计原则,用以构建轻松的多手的手势。
    • M. Wu and R. Balakrishnan, "Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays," in Proceedings of the 16th annual ACM symposium on User interface software and technology, Vancouver, Canada, 2003, pp. 193-202.
slide64

Wobbrock等 通过对用户自定义手势的研究,深入讨论了人的行为对手势的影响。

  • 用户并不在意手势设计中的手指的数量,但在操作时,更愿意只用单手,手势并受交互范例的显著影响。
  • J. O. Wobbrock, M. R. Morris, and A. D. Wilson, "User-defined gestures for surface computing," in Proceedings of the 27th international conference on Human factors in computing systems Boston, MA, USA: ACM, 2009.
slide65

Epps等进行了类似的研究,

    • 在操作中,用户只对一小部分手势存在高度认可
    • 大量常用的手势还是使用惯用手的食指,然后是张开手或者平摊开手。
    • 一个比较有趣的发现是,人们总会在系统给出反馈的时候习惯性的经常抬起手。
  • J. Epps, S. Lichman, and M. Wu, "A study of hand shape use in tabletop gesture interaction," in CHI \'06 extended abstracts on Human factors in computing systems Montr\&\#233;al, Qu\&\#233;bec, Canada: ACM, 2006, pp. 748-753.
slide66
小结
  • 从整体来看,在触控技术下还缺少对手势的系统研究,多指、双手、多人多手的手势研究还基本是一片空白。同时,现有的手势设计未能充分考虑到多点触控技术,总体研究缺少系统的理论性研究。
slide67
界面技术的改进与发展
  • 一系列研究已经证明,触控技术当前仍然回避不了三个主要的问题:
    • 直接点击精度低,即所谓的肥手指(Fat Finger)问题;
    • 手的遮挡给连续自然操作带来了困难;
    • 长时间操作造成肢体疲劳
    • 效率
slide68

1) 直接点击的改进: Potter等在1988年开始研究人手的精度选择能力, 提出两种点击策略“Land-On”和“First-Contact”,对手指点击过程中如何获得有效的目标选择能力进行了研究;

  • 我们在前期研究中,对五个手指的点击精度进行了实测,结果表明在95%的置信概率情况下,目标直径必须大于1.2厘米。
  • 为了克服人手指无法点击小目标的问题,Kabbash和Buxton提出了面光标(Area Cursor)技术,Worden等在此基础上又进行了改进。
slide69

光标偏移法:Potter等提出的“Take-Off”技术最早实现了光标偏移法。其基本思想是手指位置不是光标位置,实际光标出现在手指位置的上方。在“Take-Off”的基础上,Vogel和Baudisch提出了“Shift”技术。通过该技术,手指按下后,一个弹出的区域(CallOut)显示出被手遮挡的位置,然后对光标位置进行调整。光标偏移法:Potter等提出的“Take-Off”技术最早实现了光标偏移法。其基本思想是手指位置不是光标位置,实际光标出现在手指位置的上方。在“Take-Off”的基础上,Vogel和Baudisch提出了“Shift”技术。通过该技术,手指按下后,一个弹出的区域(CallOut)显示出被手遮挡的位置,然后对光标位置进行调整。

slide71

放大目标或者调整控制-显示比:放大比较小的目标,是提高选择成功率的一种可行办法,放大目标或者调整控制-显示比:放大比较小的目标,是提高选择成功率的一种可行办法,

    • Olwal和Feiner采用不同的鱼眼放大方式来提高选择精度。通过调整控制-显示比(Control-Display ratio)来控制光标大小是另外一种较为可行的提高选择成功率的方法。
    • Benko等提出的双指扩展(Dual Finger Stretch)和双指X菜单(Dual Finger X-Menu)技术,就是通过改变控制-显示比,让光标移动速度变慢,步长变小,从而实现对小目标的精确选择。
slide73

小软件装置(Widget): 通过一个屏幕上的小模块或者小软件系统来辅助目标选择,也是一种较为常见的办法。Albinsson和Zhai提出了一系列辅助软件小程序,选择精度可达到了1个象素。

multi touch
Multi-touch的行业应用
  • CHI’10 Astrophysics。
slide75
提纲
  • 自然人机交互(NUI)
  • 触控技术的历史
  • 触控研究的重点与目的
  • 我们的工作
drawbacks of current system
Drawbacks of Current System
  • Mainly based on point information.
    • Primarily use the center coordinates of the human finger’s contact region.
    • Most interactions mainly rely on touch positions or variations in touch movements.
    • Relatively few research demonstrations have used auxiliary information other than touch position,
      • the shape
      • size of the contact region
same as pen
Same as Pen.

Pressure, Tilt, Azimuth and Twist

widen the input bandwidth

Pressure

(0~1023)

Tilt

(22˚~ 90˚)

Azimuth

(0˚~ 360˚)

Twist

(0˚~ 360˚)

properties of human hand
The number of degrees of freedom (DOF) :

23

The number of finger input properties typically used in current touch sensing interaction techniques :

the (x, y) coordinates

Properties of human hand
four aspects of finger properties
Four aspects of finger properties
  • Position property
  • Motion property
  • Physical property
  • Event property
finger properties in our research
Finger properties in our research
  • Orientation
  • Contact area
  • Contact shape
four tasks in our research
Four tasks in our research

Vertical

tapping

Rocking

Orientation

Rotation

Oblique

tapping

apparatus of experiment
Apparatus of experiment
  • FTIR based widget
  • Color printed A4 sheet of white paper as operational interface
  • Camera: Philips SPC900NC
  • Resolution: 640*480
  • scale of the system in the x and y axes: 0.4 mm/pixel
  • 30 frames in one second
participants
Participants
  • Number of the participants: 12 (8 male, 4 female)
  • Age : 26-37
  • All right-handed
  • a little experience of using touch devices
  • The average physical width (W) and length (L) of finger tips (unit: mm)
number of trials
Number of trials
  • 12 participants

* 5 fingers (Thumb, Index, Middle, Ring, little)

*4 tasks (vertical tapping, oblique tapping, finger rocking, finger orientation rotation)

* 6 repetitions

=1440 trials

results
Results
  • 1. Three states in a touch
  • 2. Tapping Precision
  • 3. Finger Touch Area Shape
  • 4. Finger Touch Area size
  • 5. Finger Touch Area Orientation
three states in a touch
Three states in a touch
  • Three states in a touch
    • Land on state
    • Stable state
    • Lift up state
  • How to determine the stable state
    • The width of the contact area is greater than a predetermined threshold
    • The length is greater than the width
implications for design three states in a touch
Implications for design: three states in a touch
  • The first touch coordinate cannot be treated as the final touch position
  • Use the coordinates derived from the Stable state rather than from the Land On or Lift Up states

Empirical Evaluation for Finger Input Properties In Multi-touch Interaction. Feng Wang & Xiangshi Ren

tapping precision scatter diagrams and normal distributions diagrams
Tapping precision: Scatter diagrams and normal distributions diagrams
  • (a): Scatter diagrams for the index finger in the vertical touch
  • (b): Scatter diagrams for the index finger in the oblique touch
  • (c): The distance from stable position to the target for the index finger in the vertical touch
  • (d): The distance from stable position to the target for the index finger in the oblique touch

(a)

(b)

(c)

(d)

tapping precision the tapping deviation data for the five fingers
Tapping precision: the tapping deviation data for the five fingers

ULCI: upper level of 95% confidence interval (unit: pixels, scale = 0.4 mm/pixel)

shape of finger touch area
Shape of Finger Touch Area
  • The shape of the contact area can be approximately represented by the equation of a rectangle or an ellipse
  • Three parameters: width (minor axis), length (long axis), slant angle
the size and orientation of finger touch area

(unit: VA and OA = pixels2, Range of orientation = degrees)

The Size and Orientation of Finger Touch Area
  • The OA is at least 5.5 times the VA
  • The range of orientation is more than 100 degrees
orientation
Orientation
  • One potentially accessible piece
    • Orientation is a natural cue as it provides the direction a user is pointing in and is used
  • Orientation vector
    • Direction
    • An angle from a point of reference
  • Ambiguous (not in MS Surface, DI)
our goal
Our Goal
  • Present a novel and robust algorithm that accurately and unambiguously detects the orientation vector by considering the dynamics in finger contact.
  • Based on contactinformation only.
finger orientation detection algorithm
FINGER ORIENTATION DETECTION ALGORITHM
  • Detects the directed orientation vector of the user’s finger, based on the shape of the finger contact
  • Two types of finger touch on interactive surfaces
    • vertical touch
    • oblique touch
fitting contact shape
Fitting Contact Shape
  • Algorithm
    • Fitting Contact Shape
      • Elliptical Equation Fitting -> Length, Width
    • Identifying Oblique Touch
      • Area (>=120 mm2)
      • Aspect (>=1.2)
    • Disambiguating Finger Direction
disambiguating finger direction
Disambiguating Finger Direction
  • The human finger has soft and deformable tissues.
  • Distortion of the finger muscle is inevitable upon contact
  • It is apparent that the center point of the finger contact moves inward, towards the user’s palm.
slide97

Variation of the contact center between two consecutive frames t-1 (blue) and t (red).

  • Frame t-1 is the last frame of non-oblique touch state and frame t is the first frame of oblique touch state.
continual orientation tracking
Continual Orientation Tracking
  • In every subsequent frame t+1, the directed finger orientation in the previous frame Φ(t) is used as the cue to disambiguate the current undirected finger orientation θ(t+1)
performance evaluation
PERFORMANCE EVALUATION
  • Goal
    • Evaluation to assess the performance of our finger orientation detection algorithm stability and precision in determining orientation of static and dynamic fingers.
  • Task
    • Four tasks, each examining a different aspect of the algorithm.
    • No any visual feedback concerning the orientation of the finger as detected by the algorithm.
    • As a result, participants had to completely rely on their subjective perception of the finger orientation.
slide100

Apparatus

    • Direct-touch tabletop surface based on FTIR.
    • A camera with a resolution of 640×480 pixels and at a capture rate of 30 fps.
  • Participants
    • Eight volunteers
slide101

Task 1: Static Orientation Stability.

  • Task 2: Static Orientation Precision (165°, 150°, 135°, and 120° )
  • Task 3: Dynamic Orientation Precision.
  • Task 4 : Involuntary Position Variation in Rotation.
results1
Results
  • Algorithm is very stable.
  • The average variation range during each finger dwelling period is 0.59° (std. dev.= 0.15°), in practice we can ignore finger orientation changes that are less than 1°.
  • Our algorithm can provide a static precision within approximately ±5°. Across the complete 360°orientation range, this gives 36 usable orientation levels.
  • The average dynamic orientation error (absolute value) is 14.35° (std. dev. = 9.53°).
  • The average position variation during finger rotation (approximate 2.00mm )
slide103

For all tasks, the disambiguation algorithm generated 13 errors in total.

  • This resulted in a success rate of 96.7% (384 out of 397 trials), indicating good performance of the algorithm.
finger orientation design impact
Finger Orientation Design Impact
  • Interactions Using Finger Orientation
    • Enhancing Target Acquisition
    • Orientation-Sensitive Widgets
  • Inferences from Orientation Information
    • Estimating Occlusion Region
    • Inferring User Position
    • Relationship between Multiple Fingers
    • Enabling Orientation-Invariant Input
interactions using finger orientation
Interactions Using Finger Orientation
  • Enhancing Target Acquisition
    • Directed Bubble Cursor
      • applying different multiplying weights to the target distances based on the target’s relative azimuth angle compared to the finger center and orientation.
interactions using finger orientation1
Interactions Using Finger Orientation
  • Enhancing Target Acquisition
    • Aim and grab
interactions using finger orientation2
Interactions Using Finger Orientation
  • Orientation-Sensitive Widgets
inferences from finger orientation
Inferences From Finger Orientation
  • Estimating Occlusion Region
    • occlusion region : a circular sector opposite to the finger orientation Φ, with the vertex at the center of the finger tip (x, y)
    • the central angle at approximately δ = 60°
inferences from finger orientation1
Inferences From Finger Orientation
  • Inferring User Position
    • The user sits along either one of the two long sides of the tabletop.
    • Knowing the orientation of the finger touch, we can infer that the operating user is sitting at the side opposite to the finger orientation.
inferences from finger orientation2
Inferences From Finger Orientation
  • Relationship between Multiple Fingers
    • Exploit this information to infer the relationship between multiple fingers on the surface.
    • This location is usually to the opposite side of the directions pointed by all finger, and within a reasonable distance from the position of the fingertips
inferences from finger orientation3
Inferences From Finger Orientation
  • we calculate the intersection point I of the two straight lines aligned with their positions and orientations.
inferences from finger orientation4
Inferences From Finger Orientation
  • Enabling Orientation-Invariant Input
    • The orientation of the input gesture can be normalized by a compensated rotation determined by the average finger orientation while performing the gesture.
slide113

Multi-finger mouse emulation

    • Matejka et al. presented SDMouse
    • By considering the orientation of the index finger, we can unambiguously associate fingers to buttons located in a reachable location regardless of the user’s position
algorithm limitations
Algorithm Limitations
  • Assumes an oblique touch
  • The orientation disambiguation step relies on the finger center displacement during the finger landing process and assumes that this displacement is caused solely by the deformation of the finger.
  • All fingers except thumb
technology compatibility
Technology Compatibility
  • Variety of other sensing technologies
    • Capacity-based sensing
    • Embedded optical sensor arrays
    • FTIR-based devices
slide116
结论
  • 1、没有针对触控技术的设计范式与专用用户界面。当前的触控技术依赖传统WIMP范式,没有针对触控技术提出有针对性的用户界面的设计与实现,触控技术采用何种范式可以最大程度地发挥出触控技术的优势仍急待研究,用户对触控技术的使用特别缺乏认知理论和方法的指导;
slide117

没有系统的交互范式研究。现有用户界面中主要依赖手势,交互中用户使用单或双手进行触摸,通过点击、移动等手势,手势基本依靠于触击所形成的坐标信息以及坐标信息的变化,缺乏一个形式化的描述模型。手势的设计全部取决于坐标,输入属性单一,人手还存在大量其它属性,如面积,形状,指向、压力等没有系统的交互范式研究。现有用户界面中主要依赖手势,交互中用户使用单或双手进行触摸,通过点击、移动等手势,手势基本依靠于触击所形成的坐标信息以及坐标信息的变化,缺乏一个形式化的描述模型。手势的设计全部取决于坐标,输入属性单一,人手还存在大量其它属性,如面积,形状,指向、压力等

slide119

4、缺少有效的的评价办法。手势的设计缺少理论指导,无有效的评价办法,基本靠设计者主观设计;而在手势设计时,由于条件增多,单手单指,单手多指、多手多指等一系列情况的出现,设计的手势是否能满足自然、流畅操作的要求,在常规任务下手势的可用性等方面目前存在较多的未知因素。4、缺少有效的的评价办法。手势的设计缺少理论指导,无有效的评价办法,基本靠设计者主观设计;而在手势设计时,由于条件增多,单手单指,单手多指、多手多指等一系列情况的出现,设计的手势是否能满足自然、流畅操作的要求,在常规任务下手势的可用性等方面目前存在较多的未知因素。

slide120

Thanks for your attention.

  • 欢迎到昆明!!
  • 昆明理工大学 云南省计算机技术应用重点实验室, 13700600260,
  • QQ: 1946899
  • Skype: cnwangfeng
  • E-mail: [email protected]
ad