SenseCam Work at Dublin City University

SenseCam Work at Dublin City University Alan F. Smeaton, Gareth J.F. Jones and Noel E. O’Connor (PIs) Georgina Gaughan, Cathal Gurrin, Hyowon Lee, Hervé Le Borgne (PostDocs)Aiden Doherty, Michael Blighe, Ciarán Ó’Conaire, Michael McHugh, Saman Cooray (PhD students) Barry Lavelle, Paul Reynolds (Masters students)Sandrine Áime (Summer student) … 15 people working on SenseCams in some way at DCU Center For Digital Video Processing,Dublin City University, Ireland

Overview • Our contribution to developing SenseCam work; • Automatic event segmentation - 3 approaches; • Application: generation of rolling weekly summary based on Addenbrook’s • Face detection and body patch matching • Arizona data • Using BT and other sensors for context • Alternative way to presenting SenseCam images

Our (DCU) Contribution • We do image/video analysis, indexing, summarisation, etc. and we apply this to SenseCam data; • We have no particular SenseCam application, we will develop underlying technology; • We’re keen to hear about the real problems of SenseCams in practice, and to offer … • We consider the typical full-day SenseCam images, do event segmentation and summarisation;

Event Segmentation Multiple Events Finishing work in the lab At the bus stop Chatting at Skylon Hotel lobby Moving to a room Tea time On the way back home Summarisation A day’s SenseCam images (3,000 – 4,000)

Automatic Event Segmentation • Task: automatically determine events from a collection of SenseCam image data; • Based around image-image similarity using MPEG-7 features where differences may indicate events; • Similar problem to shot bound detection in video but more challenging given the fish-eye view and lesser similarities within an event vs. a shot; • Several approaches can be taken:

Extract MPEG-7 descriptors for this image Extract MPEG-7 descriptors for this image • Scalable Colour • Colour Structure • Colour Layout • Colour Moments • Edge Histogram • Homogeneous Texture • Scalable Colour • Colour Structure • Colour Layout • Colour Moments • Edge Histogram • Homogeneous Texture : : Similarity Score Similarity Calculation between 2 Images

For each image... • Scalable Colour • Colour Structure • Colour Moments • Edge Histogram Extract MPEG-7 descriptors... ... to compare Similarity between... ... adjacent images ... adjacent blocks of 10 images ... pairwise 0.7 0.91 0.65 0.7 0.15 0.8 0.74 0.15 0.92 0.65 0.82 Event-segmented images of a day Event Segmentation: Approach I One Day’s Images ...... ...... ......

Stage 1: • comparison of adjacent images • Stage 2: • Comparison every 2nd image • Stage 3: • Comparison of blocks of images • Incorporation of a face detector

Preliminary Results Images from 1 day Number of pictures: 2685Manually detected events: 27 Lots more to do, including fusion of descriptors and optimising windowing

Event Segmentation II • Use similarity clustering, and time • Combine low-level content analysis and context information (i.e. metadata provided by the SenseCam and temporal data) • Generate a similarity matrix by fusing low-level and metadata information • Implement time constraints to constrain clustering • Simple hierarchical clustering of images into events

... to variate the number of Events For each image... 1 Event (whole set as 1 Event) .......... • Scalable Colour • Colour Layout • Edge Histogram • Homogeneous Texture .......... Extract MPEG-7 descriptors 2 Events Then apply Temporal constraints... + GPS meta-data ... • Light • Temperature • Accelerometer .......... 4 Events 8 Events : ... ... to calculate Similarity among images : Event-segmented images of a day (2 Events) Similarity matrix Event Segmentation: Approach II One Day’s Images

... to variate the number of Events For each image... 1 Event (whole set as 1 Event) .......... • Scalable Colour • Colour Layout • Edge Histogram • Homogeneous Texture .......... Extract MPEG-7 descriptors 2 Events Then apply Temporal constraints... + GPS meta-data ... • Light • Temperature • Accelerometer .......... 4 Events 8 Events : ... ... to calculate Similarity among images Event-segmented images of a day (4 Events) : Event-segmented images of a day (2 Events) Similarity matrix Event Segmentation: Approach II One Day’s Images

... to variate the number of Events For each image... 1 Event (whole set as 1 Event) .......... • Scalable Colour • Colour Layout • Edge Histogram • Homogeneous Texture .......... Extract MPEG-7 descriptors 2 Events Then apply Temporal constraints... + GPS meta-data ... • Light • Temperature • Accelerometer .......... 4 Events 8 Events : ... ... to calculate Similarity among images Event-segmented images of a day (8 Events) Event-segmented images of a day (4 Events) : Event-segmented images of a day (2 Events) Similarity matrix Event Segmentation: Approach II One Day’s Images

Approach II: Results

Approach III: Group Images into 3 Classes • Static Person • Person performing one activity • E.g. at computer, meeting, eating etc. • Moving Person • Travelling between locations • Static Camera • Sense Cam is put down • User is not wearing it

Features Used • Block-based Cross-Correlation • Spatiogram image colour similarity • Compares image colour spatial distribution • Accelometer motion • Feature-based training • Using Bayesian approach to classification • Viterbi algorithm used to smooth results • Applied to 1 day SenseCam images so far

Classify each image into 3 groups (Bayesian classification)... ...... Accelerometer (motion) For adjacent images, calculate... + Static Person Static Camera Moving Person Block-based Cross-correlation + ... then Smoothing (viterbi algorithm) Spatiogram Similarity SP MP SP MP SP SC Event-segmented (& classified) images of a day Event Segmentation: Approach III One Day’s Images

Accelerometer Data Example

Generation of Weekly Summaries • Assume events already segmented; • Calculate average values for events of low level features from all images; • Generate similarity matrix using the average value from each event; • Visually similar events can then be detected, and the time period (week) structured automatically into a short movie; • Why a movie week … Addenbrooke’s Cambridge application;

Mon Tue Clustering of similar Events Wed Thr Compare Event-Event similarity within a week Fri Sat Sun ... : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets

Clustering of similar Events ... : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden working on the desk Mon Tue Wed Thr Compare Event-Event similarity within a week Fri Sat Sun

Clustering of similar Events ... : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden waiting for bus Mon Tue Wed Thr Compare Event-Event similarity within a week Fri Sat Sun

Clustering of similar Events ... : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden at the office corridor Mon Tue Wed Thr Compare Event-Event similarity within a week Fri Sat Sun

Clustering of similar Events Unique Event 2 Unique Event 3 Unique Event 4 ... Unique Event 5 Unique Event 6 : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Mon Unique Event 1 Tue Wed Thr Compare Event-Event similarity within a week Fri Sat Sun

1 Week summary (on Sunday) Select images ... Mon : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden waiting for bus Mon Similar Events - Aiden at the office corridor Tue Similar Events - Aiden working on the desk Unique Events Wed Thr Compare Event-Event similarity within a week Fri Sat Sun

Select images (on Monday) ... Mon Tue : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden waiting for bus Mon Similar Events - Aiden at the office corridor Tue Similar Events - Aiden working on the desk Unique Events Wed Thr Fri Compare Event-Event similarity within a week 1 Week summary (on Sunday) Select images Sat Sun

... Select images (on Tuesday) Tue Wed : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden waiting for bus Mon Similar Events - Aiden at the office corridor Tue Similar Events - Aiden working on the desk Unique Events Wed Thr Fri 1 Week summary (on Sunday) Select images Sat Compare Event-Event similarity within a week Sun Select images (on Monday) Mon

... Wed Select images (on Wednesday) : Event-level Similarity matrix Generation of Weekly Summary Event-Segmented image sets Similar Events - Aiden waiting for bus Mon Similar Events - Aiden at the office corridor Tue Similar Events - Aiden working on the desk Unique Events Wed Thr Fri 1 Week summary (on Sunday) Select images Sat Sun Select images (on Monday) Compare Event-Event similarity within a week Mon Select images (on Tuesday) Tue

Preliminary Results Number of similar images to a known event, from top 10 retrieved

Face Detection & Body Patch Matching • Apply face detection software to detection the presence of a face in the SenseCam image • Body Patch Matching • Identify similar body patch by color to detect subsequent appearances within an event; • This works well for personal photos, but SenseCam images are lower quality;

Face Extraction Face Extraction Similarity Score Body Patch Extraction Body Patch Extraction Similarity Score Combined Similarity Score Similarity Comparison by Person Detection 5:03pm 30 May 2006 8:28am, 7 June 2006

Arizona State U. Data • ASU gave us some SenseCam data 2 weeks ago • Session rather than all-day images; • Applied automatic event detection using 4x MPEG-7 low-level feature descriptors • Both Color Structure and Color Moments outperform others • Face Detection software performs badly on this data • Blurred Images cause “standard” face detection software to fail

Event detection using ASU data: 28-June-2006 Number of pictures: 357 Manually detected events: 28

Event detection using ASU data: 28-June-2006 Number of pictures: 434 Manually detected events: 11

Using BT to provide context • Achieved by logging Bluetooth devices in close proximity to the SenseCam wearer; • May be useful in determining which individuals are present around each picture; • Application created to poll and log Bluetooth devices on phone; • Currently developing host application to interface with mobile device and retrieve log file • Next step: synchronize time-stamps between SenseCam images and Bluetooth log file

Use of Multi-Sensor Data • Concept : To determine whether “events” can be identified based on multiple sensor data • Data collected from: • GPS Device • BodyMedia Device • Heart Rate Monitor • SenseCam • Development of a framework to extract the relevant data from the different data sources • CSV files, XML files, text files, Excel files

Presenting SenseCam Images? E.g. intelligent summary of one day (playback for 1 minute) • ... watching the fast playback of image sequences is not an ideal interaction: • Intensive concentration required during playback • Event boundaries cannot be clearly presented • Sense of time is skewed (more #images of an ‘important’ event, even if it lasted only 1 minute; less #images of ‘unimportant’ regular events even if they last many hours during the day)

Turn sequential playback into an interactive, spatial browsing interaction (similar to the way we turn video playback into keyframe browsing) =>

31 May 2006 • Approach: • 1-page visual summary of a day • Each image represents each event • Size of each image represents the ‘importance’ or ‘uniqueness’ of the event • Timeline on top orientates the user about time when each event happened • Mouse-Over activated

This is the most unique event of the day Two unusual meetings that happened that day in the lab Repeating Events are listed as small size at the bottom 31 May 2006

Mouse-Over will start playback that Event, while highlighting the time of that Event: this event (meeting a friend in Skylon hotel lobby) happened in the evening, for about 1.2 hour 31 May 2006

Talking with Gareth happened only 10 minutes, in the morning 31 May 2006

Working in the main morning time: 1.2 hours 31 May 2006

31 May 2006 Then my last desk-work of the day (2 hours) just after lunch time

31 May 2006 My lunch break

31 May 2006 My dinner time

31 May 2006 • Conclusion: • More relaxed, interactive, inviting summary of the day than fast-forwarding, while still taking advantage of playback synergy effect • Playing each of the Events in its location might be also good (without having to Mouse-Over) • ‘Importance’ is not by playing more images in that Event (this skews time), but by larger image size

Papers written • “Exploiting context information to aid landmark detection in SenseCam images”, submitted to ECHISE - 2nd International Workshop on Exploiting Context Histories in Smart Environments: Infrastructures and Design to be held at 8th UbiComp, Sept. 2006, Irvine, CA, USA; • “Structuring a Visual Lifelog Diary by Automatically Linking Events”, submitted to 3rd ACM Workshop onCapture, Archival and Retrieval of Personal Experiences (CARPE 2006) October, 2006, Santa Barbara, California, USA. • “Organising a daily visual diary using multi-feature clustering”, submitted to SPIE Electronic Imaging, San Jose, January 2007;

Future Work EVERYTHING !

SenseCam Work at Dublin City University