MSc thesis presentation

1. Multi-Task Learning with Ontology for Food Analysis Author: Gerard Marrugat Advisor: Petia Radeva Co-Advisor: Eduardo Aguilar 09/09/2019

2. 2 Index 1. Food Analysis Context 2. Motivation 3. Multi-Task Learning Model with Ontology 1. Model Design 2. Ontology Layer 4. Experimental Results 5. Conclusions & Future Lines

3. 3 Index 1. Food Analysis Context 2. Motivation 3. Multi-Task Learning Model with Ontology 1. Model Design 2. Ontology Layer 4. Experimental Results 5. Conclusions & Future Lines

4. 4 Food Analysis Context ● Food Recognition ● Food Group Recognition Fruit, Miso soup Vegetables Image source: Food/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model ● Cuisine Recognition Image source: Recognition of Multiple-Food Images by Detecting Candidate Regions ● Ingredients Recognition Salt, Sugar, Flour, Milk, Vanilla, Oil, Egg Thai Image source. Food Ingredients Recognition through Multi-label Learning Image source:selfproclaimedfoodie.com

5. 5 Index 1. Food Analysis Context 2. Motivation 3. Multi-Task Learning Model with Ontology 1. Model Design 2. Ontology Layer 4. Experimental Results 5. Conclusions & Future Lines

6. 6 Motivation Food Analysis Problems Intra-class variability ● Ingredients Intra-class variability example: Apple. Image source: Recipes5k Inter-class similarity ● Inter-class similarity example: Tomato sauce and Curry sauce. Image source: Recipes5k Decreasement in Precision

7. 7 Motivation Food Analysis Problems Less Frequent Ingredients in dataset

8. 8 Motivation Food Analysis Problems Less Frequent Ingredients in dataset Difficult to detect

9. 9 Motivation Food Analysis Problems Less Frequent Ingredients in dataset Difficult to detect Low Precision

10. 10 Motivation Hypothesis Dataset with Multiple Task Labels Egg´s Benedict Dish Ingredients Eggs Parsley Image source: Recipes5k Toast bread Butter Hollandaise sauce Bacon

11. 11 Motivation Hypothesis Relation between Task Labels Egg´s Benedict Eggs ... Parsley Image source: Recipes5k ... Toast bread Butter Hollandaise sauce Bacon

12. 12 Index 1. Food Analysis Context 2. Motivation 3. Multi-Task Learning Model with Ontology 1. Model Design 2. Ontology Layer 4. Experimental Results 5. Conclusions & Future Lines

13. 13 Multi-Task Learning Model with Ontology Dish: sushi Multi-Task Learning Model Cuisine: japanese Categories: vegetables, seafood, rice Ingredients: salmon, avocado, cucumber, rice Image source: peasandcrayons Dish: tacos Cuisine: mexican Categories: ... vegetables, meat, bread Ingredients: beef, lettuce, cheddar cheese, tortilla Image source: cocinavital

14. 14 Multi-Task Learning Model with Ontology Multi-Task Learning Model Shared Base Layers Specialized Last Layers

15. 15 Multi-Task Learning Model with Ontology Multi-Task Learning Model Relations between tasks

16. 16 Multi-Task Learning Model with Ontology Ontology Layer Relations between classes of same and different tasks Image source: Applying Deep Learning for Food Image Analysis

17. 17 Multi-Task Learning Model with Ontology How to convert it into a Layer? Image source: ladamic

18. 18 Multi-Task Learning Model with Ontology Matrix #elements x #elements Relations ● Dish-Dish ● Dish-Ingredient ● Ingredient-Dish ● Ingredient-Ingredient

19. 19 Multi-Task Learning Model with Ontology Ontology ● Element values ● Structure ● Operation

20. 20 Multi-Task Learning Model with Ontology Ontology Egg´s Benedict ● Element values 1 Ontology made of 1´s and 0´s Eggs ... 1 0 Image source: Recipes5k 0 1 0 0 Chocolate 0 1 1 Rice Bacon 0 1

21. 21 Multi-Task Learning Model with Ontology Ontology Egg´s Benedict ● Element values pbenedict-eggs Ontology made of probabilities peggs-benedict Eggs ... 0 Image source: Recipes5k 0 pbenedict-bacon 0 0 Chocolate 0 pbacon-benedict price-baco n pbacon-rice Rice Bacon 0

22. 22 Multi-Task Learning Model with Ontology Ontology Egg´s Benedict ● Element values 1 Ontology made of 1´s and -1´s Eggs ... 1 -1 Image source: Recipes5k -1 1 -1 -1 Chocolate -1 1 1 Penalize no relation Rice Bacon -1 1

23. 23 Multi-Task Learning Model with Ontology Ontology Egg´s Benedict ● Element values pbenedict-eggs Ontology made of probabilities and “negative probabilities” peggs-benedict Eggs ... pneg pneg pbenedict-bacon Image source: Recipes5k pneg pneg pneg Chocolate pbacon-benedict price-baco n pbacon-rice Rice Bacon pneg

24. 24 Multi-Task Learning Model with Ontology Ontology Egg´s Benedict ● Element values pbenedict-eggs Ontology made of probabilities and “negative probabilities” pneg of concept i = -1/#concept i peggs-benedict Eggs ... pneg pneg pbenedict-bacon Image source: Recipes5k pneg pneg pneg Chocolate pbacon-benedict price-baco n pbacon-rice Rice Bacon pneg

25. 25 Multi-Task Learning Model with Ontology Ontology ● Structure Ingredient-Dish Dish-Ingredient

26. 26 Multi-Task Learning Model with Ontology Ontology ● Structure Dish-Ingredient Ingredient-Dish Ingredient-Ingredient

27. 27 Multi-Task Learning Model with Ontology Ontology ● Structure Dish-Ingredient Ingredient-Ingredient Full

28. 28 Multi-Task Learning Model with Ontology Ontology Dot Product ● Operation

29. 29 Multi-Task Learning Model with Ontology Ontology Min. Elem.-Wise Product ● Operation

30. 30 Multi-Task Learning Model with Ontology Ontology Min. Elem.-Wise Product ● Operation A + Min(.) A

31. 31 Multi-Task Learning Model with Ontology Ontology Avg. Elem.-Wise Product ● Operation

32. 32 Multi-Task Learning Model with Ontology Ontology Avg. Elem.-Wise Product ● Operation A + Avg(.) A

33. 33 Index 1. Food Analysis Context 2. Motivation 3. Multi-Task Learning Model with Ontology 1. Model Design 2. Ontology Layer 4. Experimental Results 5. Conclusions & Future Lines

34. 34 Experimental Results Datasets Recipes5k Size: 4.826 images Tasks: Dish and ingredients VireoFood-172 Metrics Size: 110.241 images Tasks: Dish and ingredients F1-score Precision Recall Dish Ingredients Accuracy

35. 35 Experimental Results Recipes5k Results Which Structures and Element values help

36. 36 Experimental Results VireoFood-172 Results DI-II Dish-Ingr Ingr-Ingr Best Performance

37. 37 Experimental Results Two Ontology Layers Not better than DI-II Element Wise Product

38. 38 Experimental Results MTL vs D-I I-I Ontology Model Image source: Applying Deep Learning for Food Image Analysis

39. 39 Index 1. Food Analysis Context 2. Motivation 3. Multi-Task Learning Model with Ontology 1. Model Design 2. Ontology Layer 4. Experimental Results 5. Conclusions & Future Lines

40. 40 Conclusions & Future Lines Conclusions For first time a food ontology is integrated into an end-to-end model Six different ontology structure types are defined Exclusivity relation between elements helps to the classification Our model improved MTL performance Future Lines ● Automatic food ontology construction Scalability to high number of classes and tasks ● ● ● ● ●

41. 41 Thank you! Author: Gerard Marrugat Advisor: Petia Radeva Co-Advisor: Eduardo Aguilar