Use of a Wii -mote as a real-time motion tracking device/controller

1. “Simulating the Visual Appearance and Physical Application of Automotive Surface Finishes” Pine Technical College, Johnson Center for Simulation PI: John Heckman, Co PIs: Gary Meyer, Robert Musgrove NSF Award #IIP- 0438693 3 Yr Award w/extension Start Date: 01 Feb 2005 Key Attributes of our Innovation Ecosystem: Brief Project Overview: This project has combined cutting-edge computer graphics research (University of Minnesota), the coatings and finishing expertise of the largest US manufacturer of automotive refinishing products (DuPont Performance Coatings) and simulation and virtual reality expertise (Pine Technical College) to produce valuable painter training tools, completely new approaches to visual simulation, and to conduct research into using COTS game devices as 3D trackers. Questioning & Curiosity: This project brought together people with a wide range of experiences, perspectives, backgrounds, and incentives, but they have some common elements, too. These include willingness to consider new possibilities, an inquisitive approach to learning and development, and the patience to listen and understand others. Risk Taking: Multiple aspects of this project required trying things to see if they worked. Some of these worked out well, and others require further research or were found to be not viable. Examples of viable concepts include the basic idea of projecting a realistic appearance onto a surface. A related idea that did not prove useful was the sufficiently-realistic projection of a new appearance upon an actual vehicle. Projector technology is not yet up to the task. • Program Activities: • Creation of a reflection model and its supporting software. • Importation and “painting” of complex and realistic virtual models into the existing Pine Technical College 3D real time simulation system. • Incorporation of reflection model into simulator. • Research into simulating the appearance of “orange peel” and other undesirable coating surface flaws. • Development of a compact and portable simulation hardware set to facilitate training of painters. • Development of a virtual air brush system that permits artistic application of paint coatings to virtual objects. • Research into the use of the popular and inexpensive Wii-mote controller as a rigorous 3D motion-tracking device. • Porting the Painting Simulation system to both Apple (OS-X) and to Linux (Ubuntu 9.10) Use of a Wii-mote as a real-time motion tracking device/controller • Top Contributions: • Video projectors were used to make surfaces realistically appear as if they had been painted with metallic automotive paints. • The previous simulation system was dramatically improved as a training tool, providing more-credible images and feedback along with faster performance. • A spray painting experiment was performed at DuPont Marshall Labs in Philadelphia to capture live the orientation and position of a spray gun operated by an expert painter, along with simultaneous video recording, and this data informed simulation development. • The project proved feasible the concept of a 3D real-time motion tracking system based on the inexpensive Wii-mote handheld game device. • The resulting spray paint simulation systems were successfully ported from Windows XP t, other operating systems including OS-X and Ubuntu 9.10 Openness: This project has been generally open inside the network of project collaborators, and with the resulting research. The biggest challenge to openness was obtaining access to and permission to use relevant but confidential industry intellectual property. Collaboration Across Fields: Fields of collaborators on this project included computer science, coatings science, simulation technology, business, automotive refinishing, and education and pedagogy. Placing Partners in “New Environments” & “Playgrounds”: Computer graphics graduate students, long-time coatings industry researchers, and simulation developers all found themselves in unfamiliar territory and inspired/informed by different perspectives and approaches to problems. At different times, various parts of our team were amazed by bureaucratic hurdles, allocation of resources, and efficiencies considered normal by other team segments. • Partners: • University of Minnesota • Jon Koniecsny • Clement Shimizu • Lijun Qu • Scott Johnson • Lehman Brothers Garage • Darrell Amberson • DuPont Performance Coatings • Allan Rodriuguez • Craig Shoup • John Moore • BYK Gardner (Geretsried Germany) • Konrad Lex • Gabi Kigle-Becker • Top Challenges: • Communication and coordination was difficult across the industry, research, and technical college collaborators. • Initially, in 2005, readily available computer hardware was not sufficiently powerful to support project applications. Improvements in technology have substantially solved this issue. • Industry partner involvement was difficult to sustain. Leading/Inspiring for Surprising or Unexpected Results The dynamic creative atmosphere of this project allowed consideration of such concepts as use of the Wii-mote as a tracking system. Initially assumed to be too closed, too limited, and too slow, the Wii-mote device was extensively explored and resulted in proof of the concept. The main limiting factor turned out to be the narrow (22º) field of view of the Wii’s on-board camera. . PFI . . National Science Foundation Partnerships For Innovation Grantee’s Meeting April 25-27, 2010 Arlington, VA