Grocery Shopping Assistant Carolina Galleguillos Pixel-café / June 2 2006
Description GroZi project (grocery shopping assistant) • Increase independence of people with low vision (specially blind) to perform grocery shopping in a supermarket or store. • Help to plan shopping list, walking path to the store and grocery shopping.
Motivation • 1.3 million legally blind people in the U.S • Grocery store are underselling to this market. • Blind people are “high cost” customers. • Advance research on object recognition for mobile robotics with constrained computing resources.
Motivation Characteristics Grocery Store: • Structured Environment (+). • Controlled Lightening (+). • Maintained by staff (+). • Well indexed (+). • People moving around aisles (-). • Huge amount of products (30K) (-).
Motivation Possible existing solutions: • Seeing-eye dog trained. • RFID tags (aisle, shelf, product). • Barcode scanning (shelf). • Help of sighted guide/customer service. • Memorize store layout. • Home delivery.
Why computer vision? • Limited ability of dogs. • RFID tags bring privacy concerns and heavy infrastructure. • Eye safety and mislabeling. • Independence. • Store layout changes constantly. • Autonomy.
Our Solution • Develop a handheld device that performs visual object recognition with haptic feedback. • Avail of complementary resources (RFID, Barcode scan, sighted guide) We are focusing on the computer vision aspects of this problem.
MoZi Box General purpose low-cost mobile system geared for computer vision applications. MoZi is a combination of the Mobile Vision System (MoVs) and ZigZag • Finite memory : Compact Flash (CF) cards ranging from 256 MB to 4 GB. • Processor speed: in the neighborhood of 60-400MHz • Frame rate: enough snapshots to cover the shelf with some overlap (as in panoramic stitching) (15fps instead of 30fps?). • Color Calibration: Macbeth color chart to calibrate the color space.
Use of the System • Creating a Shopping List. • Getting to the Grocery Store. • Navigating the Store.
Shopping List • Online Website: • Website stores data and images of different products. • Feedback from users. • Provides walking path. • Prepare shopping list. • Download information into Mozi Box.
On the way Separate project. • Mozi Box with GPS. • Visual waypoints. • Traffic/Street sign reading. • Use in addition to cane and asking sighted bystanders.
Inside the Store • Finding aisle (OCR, RFID, ask). • Avoiding obstacles (cane). • Finding products (sweep of aisle, spot product, barcode check). • Checking out (coupon and cash).
Obtaining training data • Online Images (Web). • Collecting from MoZi box (in situ). • Collecting from embedded camera near the barcode scanner (in situ). • Known databases (COIL-100,ETH-80, etc.)(more research oriented) • Synthetic examples. • Active learning.
Obtaining training data • Active learning problem: Find UPC for the corresponding image (labeling). • Semi-supervised. • Weakly labeled.
Obtaining training data • Sunshine Store @ UCSD. • Venue for pilot study. • 4K items in stock. • 1749 sq. ft. (assignable) • We want to scale to a bigger number of products (30K). • No bakery or vegetables.
Object Recognition 2 types of recognition (m:n, m<<n): • Detection (of objects). • Verification (objects detected are in that list). Algorithms: SIFT, AdaBoost cascade, Multiclass Adaboost, Probabilistic Boosting tree, Color histogram matching, etc.
Text Detection • Standard OCR is unlikely to be sufficient. • Low resolution and distortion are main problems. • Reading aisle signs, text on shelves.
Considerations • Occlusion and clutter of products (caused by people and shopping carts). • Multiple images of same shelf to perform “hole-fill-in”. • Cannot fit dominant plane to the front of product shelves. • Large number of items.
Acknowledgements People (UCSD/Calit2): • Serge Belongie • John Miller • Stephan Steinbach • Michele Merler • Tom Duerig Captions: Dennis Metz/ D. Stein [X. Chen and A. Yuille ]