labeling images for fun n.
Skip this Video
Loading SlideShow in 5 Seconds..
Labeling Images for FUN!!! PowerPoint Presentation
Download Presentation
Labeling Images for FUN!!!

Loading in 2 Seconds...

play fullscreen
1 / 54

Labeling Images for FUN!!! - PowerPoint PPT Presentation

  • Uploaded on

Labeling Images for FUN!!!. Yan Cao, Chris Hinrichs. How do you improve Learning systems?. Get more processing power. (Faster computers, more memory, more parallel.) Find a more sophisticated algorithm. Get lots and lots of quality data. Why Manually label Images?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Labeling Images for FUN!!!' - newton

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
labeling images for fun

Labeling Images for FUN!!!

Yan Cao, Chris Hinrichs

how do you improve learning systems
How do you improve Learning systems?
  • Get more processing power. (Faster computers, more memory, more parallel.)
  • Find a more sophisticated algorithm.
  • Get lots and lots of quality data.
why manually label images
Why Manually label Images?
  • A job that’s easy for humans but challenging for Computer Vision
  • Why? Acquire Ground Truth
    • Segmentation, i.e. object extraction from an image, is hard
    • Multiple poses and views of objects
    • Depth of objects, which one is in the front when there is an intersection
    • Relationships between objects and their parts. E.g., face and eyes, car and wheels
general idea to make computers do labeling supervised learning
General Idea to make computers do labeling – Supervised learning
  • Enough training data – Images with manually pre-assigned labels.
  • Classifiers which are trained by the training data and used to label the queried images.
  • If we want do segmentations on the queried images, the training images need to include the information about the boundaries of the inside objects.
who is willing to be volunteer
Who is willing to be volunteer
  • Manually Labeling numerous images is a tedious job
  • Motivations which can make humans do something
    • Money! You know you will be paid
    • Fun! You enjoy doing it
    • Gain respect from others
esp an image labeling game
ESP – an image labeling game
  • Rules
    • Server randomly arranges a partner to you (could be a “bot”)
    • The same image on the two partners’ screen
    • When the labels typed by the partners match each other, gain scores and move to the next image
    • There might be some taboo words which can not be the labels for the image
esp an image labeling game1
ESP – an image labeling game
  • Rules
    • Partners strive to agree on as many images as they can in 2.5 minutes
    • Partners can choose to pass images when they both click “Pass” button
    • The more images the partners agree on the labels, the higher the final scores they achieve
taboo words
Taboo Words
  • Gained from the game
    • When the image is shown the first time in ESP, there are no taboo words
    • If the image is used again, there is a taboo word which is obtained from last agreements
  • At most 6 taboo words for one image
  • Taboo words guarantee that each image has many different labels
good label threshold
Good Label Threshold
  • It is the threshold to include a label to the list of taboo words for an image
  • If threshold = 1, it means that once a pair of partners agree on a label, this label will be set as a taboo word
  • If threshold =10, when 10 pairs of partners agree on the same label for an image, it is set as a taboo word
image source
Image source
  • Randomly selected from the Web using a small amount of filters
  • From “Random Bounce Me”, which randomly returns images from Google database
  • Qualifications of images:
    • Large enough (>20 pixels on either dimension)
    • Aspect ratio between (1/4.5, 4.5)
    • Not blank/single color image
  • Are the labels relevant to the images?
    • Do a search within the labeled images in the ESP database
  • Are the players motivated by the game?
    • Do statistics on user log
  • How’s the labeling rate?
    • See how many images are labeled within a time period
accuracy of labels
Accuracy of Labels
  • 20 images are randomly selected from ESP
  • 15 participants are asked to label 20 images with 6 labels on each image, given no information about the taboo words.
  • When the labels made by the participants are compared with the labels obtained from the game, 83% of the labels match
  • For all images, the 3 most common words entered by the participants were contained by ESP labels
is it fun
Is it fun?
  • Over 80% users played the game on multiple dates
  • In 4 months, 33 players played more than 50 hours on the game
labeling rate
Labeling Rate
  • If there are 5000 users online all 24 hours (it is easy to reach for online games), within a month all images in Google database (425,000,000) will be labeled!
more than labeling
More than Labeling
  • What if the players tell more information about images, such as where the objects are in the images?
  • Peekaboom
    • An interesting game which is fun and at the same time, collects information other than labels
rules of peekboom
Rules of Peekboom
  • Pairs of partners randomly arranged by Server
  • One sees a whole image and its label (Boom side)
  • The other sees a blank screen and an input box at bottom (Peek side)
  • The boom partner clicks on the image and each click reveals an area with a 20-pixel radius to the peek partner
rules of peekaboom
Rules of Peekaboom
  • According to the revealed parts, the peek partner inputs labels until one matches the label shown on the boom side
  • The boom partner can give hints to help the peek partner get the right label
    • Ping the “key” parts in the images
    • Tell how the word is related to the image
rules of peekaboom1
Rules of Peekaboom
  • The partners switch between peek and boom alternatively
  • For images with a hard-to-guess label, the partners can choose to pass
  • The more images they correctly label in 2.5 minutes, the higher their score
  • To make the game more fun, bonus rounds are added and users are ranked by their scores
information collected by peekaboom
Information collected by Peekaboom
  • How the word relates to the image (from hints)
  • Pixels necessary to guess the word
  • The pixels inside the object, animal, or person (from pings)
  • The most salient aspects of the objects in the image (from the sequence of clicks)
  • Elimination of poor image-word pairs (passing)
applications based on the information
Applications based on the information
  • Improving Image-Search Results
    • images in which the word refers to a higher fraction of the total pixels should be ranked higher
  • Bounding boxes of objects
applications based on the information1
Applications based on the information
  • Using Ping data for pointing
  • Do people have fun?
    • More than 90% people play multiple times on different days
    • Players on the “Top Scores” all played over 53 hours
  • Accuracies of collected data
    • Bounding boxes. Participants VS Peekaboom. Overlap percentage 0.754
    • Accuracies of Pings. Participants VS Peekaboom. 100% accuracy!
label me
Label Me

Russel et. al. MIT CSAILab

improving on image captions
Improving on image captions
  • Many image DBs are available which have captions for every image, which say what is in the image.
  • LabelMe allows users to add their own bounding boxes around objects and label them directly.
  • LabelMe’s authors claim their pictures are taken from a wide variety of places. (They seem to be mostly street scenes, and other travel photos, and a few insides of houses.)
how do you participate
How do you participate?
  • Just go to the URL:
  • You are given an image, which may or may not have previously drawn boundaries. If you see an object which you can identify, draw a boundary, and when you close the polygon it asks for a label.
  • There are no rules on how to choose the labels, or on how closely to draw the boxes. They trust your judgment – but more importantly, it reflects peoples’ different ideas.
more general results
More general results:

25th, 50th, and 75th percentile by polygon count of come common object types.


We can learn something about the way people take pictures from the distribution of where objects are located. Generally, people are standing when they take pictures.

tying it in with wordnet
Tying it in with WordNet
  • Some words have synonyms: man/woman, person, pedestrian; car, automobile, cab, suv
  • Look up each label on Wordnet. The authors report 93% of labels found a matching WordNet entry, though some manual word sense disambiguation had to be done.
  • This allows queries to match at various levels of specificity in the WordNet tree, and more general queries.
dealing with occlusion simple rules
Dealing with occlusion: simple rules
  • If an object is completely contained, it is inside.
  • If it has more control points in the overlapping region is probably on top.
  • Can use features like color histograms to match the overlapping region with one region or the other, but this is expensive, complicated, and doesn’t work as well.
image search reranking
Image search reranking
  • Do segmentation on query image, extract features, compare with features of regions labeled with search terms, reorder by strength of correlation.
80 million tiny images
80 Million Tiny Images

Torralba et. al.

shrinking images
Shrinking images
  • How much information does an image need to contain in order to identify its contents?
  • Why not ask humans before asking computers?
  • Torralba et. al. looked for the minimum resolution that humans need in order to identify the contents of an image.

The humans did much better at 32x32 resolution than the best recognition algorithms did at full resolution.

Note that for color images, the humans’ accuracy levels off at 32x32. For grayscale, the same happens at 64x64.

32x32x3 dimensions for color images, 32x32x4 dimensions for grayscale with very nearly the same accuracy, so ~3000 dimensions needed for recognition.

next acquire a huge number of images
Next: Acquire a huge number of images
  • Where do you start? – even at reduced resolution, there are just too many images out there to get them all.
  • Start with WordNet. For each of the 75,062 concrete nouns in Wordnet, do an image retrieval search on many image search engines. They used, Google, Cydral, AltaVista, Flickr, Picsearch, and Webshots.
  • Then eliminate duplicates and solid-color images.
  • About 10% of the words were rare, and had no matching images.
finding nearest neighbors
Finding nearest neighbors
  • Need a distance metric to compare the tiny images. They examine 3: SSD(Sum of Squared Differences), Warp, and Shift.
  • Normal SSD is done by summing the squared difference over all dimensions.
  • Computing distance between all pairs this way is too expensive, so they used the top 19 Principal Components. They did some experiments to show that this works reliably.
warp shift
Warp: Just warp the image in some simple way, like flipping, scaling or translating, and see if that improves the SSD.

Shift: Allow each pixel to shift in a 5x5 window, and take the best SSD from that. (Crude approximation of general warping.)

Warp & Shift
effect of db size
Effect of DB size
  • As the DB grows, the quality of nearest neighbors noticeably changes, even up to ~100,000,000.
  • Object Recognition
  • Image retrieval reranking
  • Person Detection & localization
  • Image Colorization
  • Recognition is done by finding neighbors, and retrieving the Wordnet entry for each.
  • Each one corresponds to a unique leaf node in the WordNet tree, and gets a single “vote”.
  • Unify the branches into a tree, weighting internal nodes by how many branches pass through them.
  • Classify by following link to highest voted child node.
image search reranking1
Image search reranking

Do an image search on, say, “person”, on any image retrieval engine. Then find the correlation with the search term with the neighbor set of each image returned, and rank them based on the strength of the correlation with the original search terms.

person detection
Person detection

Images matched with the Wordnet node “person” and their nearest neighbors. Note that the neighbors match the part of the person shown in the query image, and their poses and color of clothing.

Here, the system only returns whether the best match passes through the “person” internal node.

The internet has a large bias towards images with people in them, so not all applications of this method will work with things that are not people.

person location
Person location

Given a portion of an image, we can find its neighbors, and measure the correlation with “person” in that set.

Extending this, we can find the portion of a query image whose neighbor set has the highest correlation with “person”. This region is very likely to have a person in it.


Given a query image, (grayscale,) find its neighbor set, and take the average color of the set. Then apply that coloring to the grayscale image. Surprisingly, this works, especially given that not all neighbor images are even of the same type of object!