1 / 120

??????????? - PowerPoint PPT Presentation

  • Uploaded on

触控与多点触控交互技术. 王锋 昆明理工大学 E-mail: wangfeng@acm.org Phone: 13700600260. 提纲. 自然人机交互 (NUI) 触控技术的历史 触控研究的重点与目的 我们的工作. 人机交互 – UI. 用户界面 (UI) 技术是人机交互研究的热点问题之一。 一种新的人机交互设备的出现,都会引发用户界面技术的一次重要变化 用户界面在人机系统中负责计算机的输入和输出,产生必要的反馈,并直接影响最终用户对系统的使用。. What is HCI?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '???????????' - emile

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript




E-mail: wangfeng@acm.org

Phone: 13700600260


  • 自然人机交互(NUI)

  • 触控技术的历史

  • 触控研究的重点与目的

  • 我们的工作

人机交互 – UI

  • 用户界面(UI)技术是人机交互研究的热点问题之一。

  • 一种新的人机交互设备的出现,都会引发用户界面技术的一次重要变化

  • 用户界面在人机系统中负责计算机的输入和输出,产生必要的反馈,并直接影响最终用户对系统的使用。

What is hci
What is HCI?

  • User:Am I error again? Computer for People? Or People for Computer???


  • 董士海先生等曾提到过,用户界面:


    • 命令行 (CLI)

    • 图形用户 (GUI)

    • 自然用户界面(NUI)


  • 当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着“用户自由”的方向发展。

  • 用户逐渐成为交互关系中的主体,以个性化和场景自动感知的新的交互模式得到重视。

    [1] J. Canny, "The Future of Human-Computer Interaction," Queue, vol. 4, pp. 24-32, 2006.

    [2] B. Myers, "A brief history of human-computer interaction technology," interactions, vol. 5, pp. 44-54, 1998.


  • Shneiderman当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着指出,新的计算技术将更为关注人能做什么,但是人的天性和需要并不会因为计算机的发明而改变。

  • 利用人与日常世界打交道时所形成的自然交互技能来获得计算机提供的服务更符合人的认知特点。

Why nui
Why NUI? 当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着

  • GUI(MacOS、Windows操作系统)

  • GUI以桌面为隐喻,采用WIMP范式,但是频繁的菜单选择、按钮操作和键盘命令输入不符合人们的自然交互习惯,操作离散、界面布局复杂,认知负担重,对于非熟练的大众用户更是效率低下,WIMP界面已经成为制约用户生产力提高的瓶颈。


  • UI当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着大师Alan Coope曾在著作《The Essentials of Interaction Design》中,在用户心理层次深刻地揭示了流状态:“深深的完全沉思状态”,经常产生“轻微的欢娱”,能够忘记时间的流逝,这就是自然用户界面的真实写照。


  • 自然人机交互的主旨。当前的人机交互研究正向更自然、更高效、更智能的方向,特别是朝着

    • 软件交互应该促进和加强流状态,而不是打断或者干扰流状态。这就要求在用户的交互过程中,大部分交互行为都应在一种顺畅的状态下提供给用户。

    • 普适计算之父Mark Weiser, 不可见的交互技术。所谓的不可见,是说工具不会干扰人们的认知,人们的注意焦点是任务,而不是工具。


Input device vision goals 1945 2015
Input Device - Vision/Goals (1945-2015)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。


  • Combined speech recognition, character recognition

  • Pen editing

  • Heuristic programming

  • Ubiquitous computing

  • Time sharing

  • Electronic I/O

  • Interactive, real- time system

  • Large scale information storage and retrieval

  • Natural language understanding

  • Speech recognition of arbitrary users

  • Natural user interface

  • Internet of things (C2C, H2C, H2H, H2C2H)

Input devices overview
Input Devices (overview)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Sensor Devices

1. Spatial Position/Orientation Sensors

• 2DOF (Mouse)

• 3DOF (Microscribe, FreeD Joystick)

• 6DOF (Polhemus Fastrack)

2. Directional Force Sensors

• 5 DOF (Spacemouse)

• 2 DOF (Joystick)

3. Gesture Recognition

• Data Gloves

4. Eye Tracking

5. Speech Recognition Systems

The first mouse 1964
The First Mouse (1964)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Knee control

Douglas Engelbart

Years before personal computers and desktop information processing became commonplace or even practicable, Douglas Engelbart had invented a number of interactive, user-friendly information access systems that we take for granted today: the computer mouse, windows, shared-screen teleconferencing, hypermedia, groupware, and more.

Input devices 1
Input Devices (1)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Directional Force Sensors


Input devices 2
Input Devices (2)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Gesture Recognition

Dextrous Hand Master, Exos


Cyberglove , 5th Dimension

Input devices 3
Input Devices (3)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Spatial Position/Orientation Sensors

Polhemus InsideTrack

(Magnetic Tracking)


(Mechanical Tracking)

FreeD Joystick

(UltraSonic Tracking)

Input devices 4
Input Devices (4)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Visual Haptic Workbench

The Visual Haptic Workbench consists of five hardware components.

The dominant hand of the user experiences haptic feedback from the PHANToM, and the subdominant hand navigates through a menu interface via Pinch glove contact gestures. Head tracking is done with a Polhemus Fastrak receiver mounted on a pair of Stereographics CrystalEyes LCD shutter glasses. The subdominant hand can also be tracked with a separate receiver to facilitate more complex interaction paradigms. The audio subsystem gives the user additional reinforcement cues to clarify

[see also http://haptic.mech.nwu.edu/intro/gallery/]

Historical overview 1945 1995
Historical Overview (1945-1995)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

[source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]


  • NUI的发展与基本理解

  • 触控技术的历史

  • 触控研究的重点与目的

  • 触控技术的前景

Touch multi touch
Touch / Multi-touch通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 触控技术是单点触控与多点触控技术的总称

  • 自然人机交互中的重要组成部分

    • 在操作时不用借助其它媒介可直接触摸

    • Natural affordances

    • Offer a more compelling method to interact with a system than a mouse or other types of pointing devices


  • 得益于Nakatani天才的软机器(Soft Machine, 1983)的想法

    • 通过手对屏幕上显示的各类操作组件(如按钮等)的操作,实现一种可变的、动态的交互。

    • L. H. Nakatani and J. A. Rohrlich, "Soft machines: A philosophy of user-computer interface design," in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, Boston, Massachusetts, United States, 1983, pp. 19-23.

History of multi touch
History of Multi-touch通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 1982 - first multi-touch system called Flexible Machine interface developed in University of Toronto.

  • 1983 - Bell labs and Murray Hill published the first paper discussing touch-screen based interfaces.

Video place
Video Place通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 1983 - Video Place /Video Desk (Myron Krueger) A vision based system that tracked the hands and enabled multiple fingers,hands, and people to interact using a rich set of gestures

  • Myron’s work had a staggeringly rich repertoire of gestures, muti-finger, multi-hand and multi-person interaction.

History of multi touch1
History of Multi-touch通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 1984 Multi-touch Screen (Bell labs, Murray Hill NJ) - integrated with a CRT on an interactive graphics terminal, could manipulate graphical objects with fingers with excellent response time.

  • Myron’s work had a staggeringly rich repertoire of gestures, muti-finger, multi-hand and multi-person interaction.

  • Microsoft began research in this area..

Multitouch tablet 1985
Multitouch Tablet 1985通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Input Research Group, University of Toronto

  • Touch tablet capable of sensing an arbitrary number of simultaneous touch inputs, reporting both location and degree of touch for each

Sensor frame carnegie mellon university
Sensor Frame 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。(Carnegie Mellon University)

  • The device used optical sensors in the corners of the frame to detect fingers.

  • At the time that this was done, miniature cameras were essentially unavailable.  Hence, the device used DRAM IC's with glass (as opposed to opaque) covers for imaging.

  • It could sense up to three fingers at a time fairly reliably (but due to optical technique used, there was potential for misreadings due to shadows.

1986 bi manual input
1986 Bi Manual Input 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 1986 Bi Manual Input (University of Toronto) - able to position/scale task and selection/navigate task

Apple desktop bus
Apple Desktop Bus通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 1986 Apple Desktop Bus (ADB) and the trackball scroller Init(Apple Computer/University of Toronto)‏

  • E.g. The macintosh II and macintosh SE

Digital desk 1991
Digital Desk 1991通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • (Pierre Wellner,  Rank Xerox EuroPARC, Cambridge) - supported multi-finger and pinching motions (leads to moderm product e.g. Iphone)‏

Flip keyboard 1992
Flip Keyboard - 1992通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Bill Buxton, Xerox PARC

  • A multi-touch pad integrated into the bottom of a keyboard.  You flip the keyboard to gain access to the multi-touch pad for rich gestural control of applications.

  • Graphics on multi-touch surface defining controls for various virtual devices.

History of multi touch2
History of Multi-touch通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 1992:  Simon (IBM & Bell South) - A single-touch device relied on a touch-screen driven “soft machine”

  • 1992:  Wacom (Japan) - tablet that had multi-device/multi-point sensing capability

Starfire inconceivable 5
Starfire通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。 - inconceivable [5]

  • 1992:  Starfire (Bruce Tognazinni , SUN Microsystems) - Bruce Tognazinni produced an future envisionment film, Starfire, that included a number of multi-hand, multi-finger interactions, including pinching, etc.

Starfile video
Starfile Video通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Bimanual 1994 2002
Bimanual 1994-2002 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Alias|Wavefront Toronto

  • Developed a number of innovative techniques for multi-point / multi-handed input for rich manipulation of graphics and other visually represented objects

Graspable tangible interfaces 1995
Graspable/Tangible Interfaces 1995 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Input Research Group, University of Toronto

  • Demonstrated concept and later implementation of sensing the identity, location and even rotation of multiple physical devices on a digital desk-top display and using them to control graphical objects.

Active desk 1995 97
Active Desk 1995/97通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Input Research Group / Ontario Telepresence Project, University of Toronto

  • Simultaneous bimanual and multi-finger interaction on large interactive display surface

T3 wavefront 1997
T3 – Wavefront 1997通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • A bimanual tablet-based system

  • Utilized a number of techniques that work equally well on multi-touch devices

Haptic lens 1997
Haptic Lens 1997通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • By Mike Sinclair, Georgia Tech / Microsoft Research

  • A multi-touch sensor that had the feel of clay, in that it deformed the harder you pushed, and resumed it basic form when released. A novel and very interesting approach to this class of device.

Fingerworks 1998
Fingerworks 1998通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Inventor: Newark, Delaware

  • Produced a line of multi-touch products including the iGesture Pad. They supported a fairly rich library of multi-point / multi-finger gestures.

Portfolio wall 1999
Portfolio Wall 1999通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Alias|Wavefront,Toronto On, Canada

  • A digital cork-board on which images could be presented as a group or individually.

  • Its interface was entirely based on finger touch gestures that went well beyond typical touch screen interfaces.

Diamond touch 2001 7
Diamond Touch 2001 [7] 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Mitsubishi Research Labs,Cambridge MA)

  • Capable of distinguishing which person’s fingers/hands are which, as well as location and pressure

Smartskin sony 2002
SmartSkin 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。– Sony 2002

  • An architecture for making interactive surfaces that are sensitive to human hand and finger gestures.

  • This sensor recognizes multiple hand positions and their shapes by using capacitive sensing and a mesh-shaped antenna.

  • In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems.

Jeff han
Jeff Han通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • FTIR multi-touch.

  • Very elegent implementation of a number of techniques and applications on a table format rear projection surface.

Video of han
Video of Han通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

2007 2010 iphone4
2007-2010 Iphone4通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Iphone macbook air
iPhone通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。& MacBook Air

Microsoft surface
Microsoft Surface通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。


Computer vision based
Computer Vision Based 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。



Historical overview 1945 19951
Historical Overview (1945-1995)通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

[source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]


  • 自然用户交互

  • 触控技术的历史

  • 触控研究的重点与目的

  • 触控技术的前景


  • NUI的发展与基本理解

  • 触控技术的历史

  • 触控研究的重点与目的

    • 新的多点触控支撑技术的研究

    • 触控技术中的手势与界面设计研究

    • 不同领域的应用研究

  • 我们的工作



Surface Acoustic Wave

Strain gauge

Optical Imaging

Dispersive Signal Technology

Acoustic Pulse

Frustrated Total Internal Reflection (FTIR)

Diffused Illumination (DI)


Shadow Capture


Frustrated total internal reflection
Frustrated Total Internal Reflection通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Light emitting diodes produce light waves that travel through an acrylic touch screen. Under normal conditions, the light waves stay within the acrylic pane; however, when an object presses on the acrylic the light is scattered downward.

Diffused illumination
Diffused Illumination通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • An LED light source is aimed at the screen. When objects touch the acrylic surface, the light reflects back and is picked up by multiple infrared cameras.

  • DI has the added capability for object detection.

  • Most often used in high end systems.

Capacitive multi touch
Capacitive Multi-Touch通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Many layers of protective, bonding, driving, sensing, substrate, and display surfaces

  • Two methods of picking up signal, Mutual capacitance or self capacitance

  • Sensor's 'normal' capacitance field is altered by another capacitance field from the human finger, electronic circuits measure the resultant distortion

Shadow capture
Shadow Capture通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Under normal conditions, light continually travels through the acrylic pane and reaches the camera; however, when an object presses on the acrylic a shadow is cast downward.

Architectural overview
Architectural Overview通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • A camera is used to pick up the light reflected when a contact is made with the acrylic screen. The camera is attached to a computer running software that reacts to each touch.

  • A rear projector is used to display the output from the computer. The screen serves two general purposes: to display the image from the projector, and also to absorb all of the light from the projector so that the user on the other side of the acrylic is not blinded.


  • Any multi-touch interface requires computer software to manipulate the detected touches received by the camera. Additional programs are then needed to respond to each input and produce a desired output.

  • Touchlib is an open source library for creating multi-touch interaction surfaces. It handles tracking blobs of infrared light and sends programs multi-touch events, such as 'finger down', 'finger moved', and 'finger released'. It interfaces with most major types of webcams and video capture devices.

  • Flash open sound control (FLOSC) is also used to connect Touchlib, which sends out OSC messages.

  • Programs created for multi-touch are generally written in a form of C or flash.

  • http://www.nuigroup.com


  • NUI的发展与基本理解

  • 触控技术的历史

  • 触控研究的重点与目的

    • 新的多点触控支撑技术的研究;

    • 触控技术中的手势与界面设计研究。

    • 不同领域的应用研究

  • 我们的工作


  • 手势研究现状分析通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

    • Wu等提出了一系列的设计原则,用以构建轻松的多手的手势。

    • M. Wu and R. Balakrishnan, "Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays," in Proceedings of the 16th annual ACM symposium on User interface software and technology, Vancouver, Canada, 2003, pp. 193-202.


  • Wobbrock通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。等 通过对用户自定义手势的研究,深入讨论了人的行为对手势的影响。

  • 用户并不在意手势设计中的手指的数量,但在操作时,更愿意只用单手,手势并受交互范例的显著影响。

  • J. O. Wobbrock, M. R. Morris, and A. D. Wilson, "User-defined gestures for surface computing," in Proceedings of the 27th international conference on Human factors in computing systems Boston, MA, USA: ACM, 2009.


  • Epps通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。等进行了类似的研究,

    • 在操作中,用户只对一小部分手势存在高度认可

    • 大量常用的手势还是使用惯用手的食指,然后是张开手或者平摊开手。

    • 一个比较有趣的发现是,人们总会在系统给出反馈的时候习惯性的经常抬起手。

  • J. Epps, S. Lichman, and M. Wu, "A study of hand shape use in tabletop gesture interaction," in CHI '06 extended abstracts on Human factors in computing systems Montr\&\#233;al, Qu\&\#233;bec, Canada: ACM, 2006, pp. 748-753.


  • 从整体来看,在触控技术下还缺少对手势的系统研究,多指、双手、多人多手的手势研究还基本是一片空白。同时,现有的手势设计未能充分考虑到多点触控技术,总体研究缺少系统的理论性研究。


  • 一系列研究已经证明,触控技术当前仍然回避不了三个主要的问题:

    • 直接点击精度低,即所谓的肥手指(Fat Finger)问题;

    • 手的遮挡给连续自然操作带来了困难;

    • 长时间操作造成肢体疲劳

    • 效率


  • 1) 通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。直接点击的改进: Potter等在1988年开始研究人手的精度选择能力, 提出两种点击策略“Land-On”和“First-Contact”,对手指点击过程中如何获得有效的目标选择能力进行了研究;

  • 我们在前期研究中,对五个手指的点击精度进行了实测,结果表明在95%的置信概率情况下,目标直径必须大于1.2厘米。

  • 为了克服人手指无法点击小目标的问题,Kabbash和Buxton提出了面光标(Area Cursor)技术,Worden等在此基础上又进行了改进。


  • 光标偏移法通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。:Potter等提出的“Take-Off”技术最早实现了光标偏移法。其基本思想是手指位置不是光标位置,实际光标出现在手指位置的上方。在“Take-Off”的基础上,Vogel和Baudisch提出了“Shift”技术。通过该技术,手指按下后,一个弹出的区域(CallOut)显示出被手遮挡的位置,然后对光标位置进行调整。


  • 放大目标或者调整控制通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。-显示比:放大比较小的目标,是提高选择成功率的一种可行办法,

    • Olwal和Feiner采用不同的鱼眼放大方式来提高选择精度。通过调整控制-显示比(Control-Display ratio)来控制光标大小是另外一种较为可行的提高选择成功率的方法。

    • Benko等提出的双指扩展(Dual Finger Stretch)和双指X菜单(Dual Finger X-Menu)技术,就是通过改变控制-显示比,让光标移动速度变慢,步长变小,从而实现对小目标的精确选择。


  • 小软件装置通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。(Widget): 通过一个屏幕上的小模块或者小软件系统来辅助目标选择,也是一种较为常见的办法。Albinsson和Zhai提出了一系列辅助软件小程序,选择精度可达到了1个象素。

Multi touch

  • CHI’10 Astrophysics。


  • 自然人机交互(NUI)

  • 触控技术的历史

  • 触控研究的重点与目的

  • 我们的工作

Drawbacks of current system
Drawbacks of Current System通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Mainly based on point information.

    • Primarily use the center coordinates of the human finger’s contact region.

    • Most interactions mainly rely on touch positions or variations in touch movements.

    • Relatively few research demonstrations have used auxiliary information other than touch position,

      • the shape

      • size of the contact region

Same as pen
Same as Pen.通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

Pressure, Tilt, Azimuth and Twist

widen the input bandwidth




(22˚~ 90˚)


(0˚~ 360˚)


(0˚~ 360˚)

Properties of human hand

The number of degrees of freedom (DOF) :通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。


The number of finger input properties typically used in current touch sensing interaction techniques :

the (x, y) coordinates

Properties of human hand

Four aspects of finger properties
Four aspects of finger properties通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Position property

  • Motion property

  • Physical property

  • Event property

Finger properties in our research
Finger properties in our research通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Orientation

  • Contact area

  • Contact shape

Four tasks in our research
Four tasks in our research通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。








Apparatus of experiment
Apparatus of experiment通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • FTIR based widget

  • Color printed A4 sheet of white paper as operational interface

  • Camera: Philips SPC900NC

  • Resolution: 640*480

  • scale of the system in the x and y axes: 0.4 mm/pixel

  • 30 frames in one second


  • Number of the participants: 12 (8 male, 4 female)

  • Age : 26-37

  • All right-handed

  • a little experience of using touch devices

  • The average physical width (W) and length (L) of finger tips (unit: mm)

Number of trials
Number of trials通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • 12 participants

    * 5 fingers (Thumb, Index, Middle, Ring, little)

    *4 tasks (vertical tapping, oblique tapping, finger rocking, finger orientation rotation)

    * 6 repetitions

    =1440 trials


  • 1. Three states in a touch

  • 2. Tapping Precision

  • 3. Finger Touch Area Shape

  • 4. Finger Touch Area size

  • 5. Finger Touch Area Orientation

Three states in a touch
Three states in a touch通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。

  • Three states in a touch

    • Land on state

    • Stable state

    • Lift up state

  • How to determine the stable state

    • The width of the contact area is greater than a predetermined threshold

    • The length is greater than the width

Implications for design three states in a touch
Implications for design:通过自然人机交互,用户可以利用语音、动作等自然表达,产生与计算机设备的无缝交互,无疑可以明显降低交互困难,提高交互效率。 three states in a touch

  • The first touch coordinate cannot be treated as the final touch position

  • Use the coordinates derived from the Stable state rather than from the Land On or Lift Up states

Empirical Evaluation for Finger Input Properties In Multi-touch Interaction. Feng Wang & Xiangshi Ren

Tapping precision scatter diagrams and normal distributions diagrams
Tapping precision: Scatter diagrams and normal distributions diagrams

  • (a): Scatter diagrams for the index finger in the vertical touch

  • (b): Scatter diagrams for the index finger in the oblique touch

  • (c): The distance from stable position to the target for the index finger in the vertical touch

  • (d): The distance from stable position to the target for the index finger in the oblique touch





Tapping precision the tapping deviation data for the five fingers
Tapping precision: the tapping deviation data for the five fingers

ULCI: upper level of 95% confidence interval (unit: pixels, scale = 0.4 mm/pixel)

Shape of finger touch area
Shape of Finger Touch Area fingers

  • The shape of the contact area can be approximately represented by the equation of a rectangle or an ellipse

  • Three parameters: width (minor axis), length (long axis), slant angle

The size and orientation of finger touch area

(unit: VA and OA = pixels fingers2, Range of orientation = degrees)

The Size and Orientation of Finger Touch Area

  • The OA is at least 5.5 times the VA

  • The range of orientation is more than 100 degrees

Orientation fingers

  • One potentially accessible piece

    • Orientation is a natural cue as it provides the direction a user is pointing in and is used

  • Orientation vector

    • Direction

    • An angle from a point of reference

  • Ambiguous (not in MS Surface, DI)

Our goal
Our Goal fingers

  • Present a novel and robust algorithm that accurately and unambiguously detects the orientation vector by considering the dynamics in finger contact.

  • Based on contactinformation only.

Finger orientation detection algorithm

  • Detects the directed orientation vector of the user’s finger, based on the shape of the finger contact

  • Two types of finger touch on interactive surfaces

    • vertical touch

    • oblique touch

Fitting contact shape
Fitting Contact Shape fingers

  • Algorithm

    • Fitting Contact Shape

      • Elliptical Equation Fitting -> Length, Width

    • Identifying Oblique Touch

      • Area (>=120 mm2)

      • Aspect (>=1.2)

    • Disambiguating Finger Direction

Disambiguating finger direction
Disambiguating Finger Direction fingers

  • The human finger has soft and deformable tissues.

  • Distortion of the finger muscle is inevitable upon contact

  • It is apparent that the center point of the finger contact moves inward, towards the user’s palm.


Continual orientation tracking
Continual Orientation Tracking frames

  • In every subsequent frame t+1, the directed finger orientation in the previous frame Φ(t) is used as the cue to disambiguate the current undirected finger orientation θ(t+1)

Performance evaluation

  • Goal

    • Evaluation to assess the performance of our finger orientation detection algorithm stability and precision in determining orientation of static and dynamic fingers.

  • Task

    • Four tasks, each examining a different aspect of the algorithm.

    • No any visual feedback concerning the orientation of the finger as detected by the algorithm.

    • As a result, participants had to completely rely on their subjective perception of the finger orientation.


  • Apparatus frames

    • Direct-touch tabletop surface based on FTIR.

    • A camera with a resolution of 640×480 pixels and at a capture rate of 30 fps.

  • Participants

    • Eight volunteers


  • Task 1: Static Orientation Stability. frames

  • Task 2: Static Orientation Precision (165°, 150°, 135°, and 120° )

  • Task 3: Dynamic Orientation Precision.

  • Task 4 : Involuntary Position Variation in Rotation.

Results frames

  • Algorithm is very stable.

  • The average variation range during each finger dwelling period is 0.59° (std. dev.= 0.15°), in practice we can ignore finger orientation changes that are less than 1°.

  • Our algorithm can provide a static precision within approximately ±5°. Across the complete 360°orientation range, this gives 36 usable orientation levels.

  • The average dynamic orientation error (absolute value) is 14.35° (std. dev. = 9.53°).

  • The average position variation during finger rotation (approximate 2.00mm )


Finger orientation design impact
Finger Orientation Design Impact errors in total.

  • Interactions Using Finger Orientation

    • Enhancing Target Acquisition

    • Orientation-Sensitive Widgets

  • Inferences from Orientation Information

    • Estimating Occlusion Region

    • Inferring User Position

    • Relationship between Multiple Fingers

    • Enabling Orientation-Invariant Input

Interactions using finger orientation
Interactions Using Finger Orientation errors in total.

  • Enhancing Target Acquisition

    • Directed Bubble Cursor

      • applying different multiplying weights to the target distances based on the target’s relative azimuth angle compared to the finger center and orientation.

Interactions using finger orientation1
Interactions Using Finger Orientation errors in total.

  • Enhancing Target Acquisition

    • Aim and grab

Interactions using finger orientation2
Interactions Using Finger Orientation errors in total.

  • Orientation-Sensitive Widgets

Inferences from finger orientation
Inferences From Finger Orientation errors in total.

  • Estimating Occlusion Region

    • occlusion region : a circular sector opposite to the finger orientation Φ, with the vertex at the center of the finger tip (x, y)

    • the central angle at approximately δ = 60°

Inferences from finger orientation1
Inferences From Finger Orientation errors in total.

  • Inferring User Position

    • The user sits along either one of the two long sides of the tabletop.

    • Knowing the orientation of the finger touch, we can infer that the operating user is sitting at the side opposite to the finger orientation.

Inferences from finger orientation2
Inferences From Finger Orientation errors in total.

  • Relationship between Multiple Fingers

    • Exploit this information to infer the relationship between multiple fingers on the surface.

    • This location is usually to the opposite side of the directions pointed by all finger, and within a reasonable distance from the position of the fingertips

Inferences from finger orientation3
Inferences From Finger Orientation errors in total.

  • we calculate the intersection point I of the two straight lines aligned with their positions and orientations.

Inferences from finger orientation4
Inferences From Finger Orientation errors in total.

  • Enabling Orientation-Invariant Input

    • The orientation of the input gesture can be normalized by a compensated rotation determined by the average finger orientation while performing the gesture.


  • Multi-finger mouse emulation errors in total.

    • Matejka et al. presented SDMouse

    • By considering the orientation of the index finger, we can unambiguously associate fingers to buttons located in a reachable location regardless of the user’s position

Algorithm limitations
Algorithm Limitations errors in total.

  • Assumes an oblique touch

  • The orientation disambiguation step relies on the finger center displacement during the finger landing process and assumes that this displacement is caused solely by the deformation of the finger.

  • All fingers except thumb

Technology compatibility
Technology Compatibility errors in total.

  • Variety of other sensing technologies

    • Capacity-based sensing

    • Embedded optical sensor arrays

    • FTIR-based devices

结论 errors in total.

  • 1、没有针对触控技术的设计范式与专用用户界面。当前的触控技术依赖传统WIMP范式,没有针对触控技术提出有针对性的用户界面的设计与实现,触控技术采用何种范式可以最大程度地发挥出触控技术的优势仍急待研究,用户对触控技术的使用特别缺乏认知理论和方法的指导;


  • 没有系统的交互范式研究。 errors in total. 现有用户界面中主要依赖手势,交互中用户使用单或双手进行触摸,通过点击、移动等手势,手势基本依靠于触击所形成的坐标信息以及坐标信息的变化,缺乏一个形式化的描述模型。手势的设计全部取决于坐标,输入属性单一,人手还存在大量其它属性,如面积,形状,指向、压力等


  • 3 errors in total. 、交互体验较弱。操作流畅度不佳,过多的手的抬起与放下导致交互体验较差。对交互过程控制不足,无深入的用户交互体验研究,如无缝性(seamless interaction)、不可见性(invisibility)以及直接操作(direct manipulation)特性。


  • 4 errors in total. 、缺少有效的的评价办法。手势的设计缺少理论指导,无有效的评价办法,基本靠设计者主观设计;而在手势设计时,由于条件增多,单手单指,单手多指、多手多指等一系列情况的出现,设计的手势是否能满足自然、流畅操作的要求,在常规任务下手势的可用性等方面目前存在较多的未知因素。


  • Thanks for your attention. errors in total.

  • 欢迎到昆明!!

  • 昆明理工大学 云南省计算机技术应用重点实验室, 13700600260,

  • QQ: 1946899

  • Skype: cnwangfeng

  • E-mail: wangfeng@acm.org