1 / 15

Forgetting in Deep Graph Networks

Paper published at the Graph Learning Benchmark workshop within the Web Conference 2021

andcos
Download Presentation

Forgetting in Deep Graph Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Catastrophic Forgetting in Deep Graph Networks: an Introductory Benchmark for Graph Classification Antonio Carta Federico Errica Davide Bacciu Andrea Cossu Graph Learning Benchmarks Workshop, WWW2021.

  2. Outline CONTINUAL LEARNING BACKGROUND EXPERIMENTAL SETUP FUTURE DIRECTIONS 1 3 5 2 4 6 PAPER’S AIMS RESULTS Q&A 2

  3. Continual Learning setting DRIFT DRIFT DRIFT 0 0 0 WE STUDIED CATASTROPHIC FORGETTING 3

  4. Why bothering? Training on the entire data may even lead to superior results! However… Datasets may be huge → no training on the edge ? ? ◎ ◎ New data after deployment → retraining ◎ ◎ Expensive ? ? ○ Inefficient → most of the information is alreadyin the model ? ? ○ 4

  5. Paper’s Aim (1) Does CL work on graphs? ● Graph distribution drift GRL regularization strategies? ● node distribution drift Graph classification ● neighborhood distribution drift 5

  6. Paper’s Aim (2) Reproducible Research + Framework ● Foster future research ?CL Avoid common mistakes ● Handle boilerplate code https://github.com/diningphil/continual_learning_for_graphs ● 6

  7. CL + GRL techniques Elastic Weight Consolidation (EWC) Learning without Forgetting (LwF) Structure Preserving Regularization (REG) [Ref. 3] Replay Memory Naive Strategy 7

  8. Which Models? Structure-agnostic Baseline ● Impact of structure in CL MLP + Mean Global Pooling ● Generic and simple DGN ● Based on GraphSAGE Mean aggregator & Global Pooling ● MLP GRAPH CONVOLUTION(S) GLOBAL POOLING 8

  9. Datasets [Ref. 5] OGBG-PPA CIFAR10 (same preprocessing of MNIST) 9

  10. Evaluation Setup Class-incremental Scenario ● Monitor ● Hold-out + Hyper-param optimization for all models ● 10

  11. Results Baseline: Competitive! Are DGNs ignoring the structure? 11

  12. Results: LwF sensitivity to hyper-parameters 12

  13. The road ahead Better understand the role of graph distribution drift on forgetting ○ Design ad-hoc regularization ◎ ◎ Need more benchmarks! ◎ ◎ Study node classification and other tasks ◎ ◎ 13

  14. References 1. Cossu A., Carta A., Errica F., Bacciu D., “Catastrophic Forgetting in Deep Graph Networks: an Introductory Benchmark for Graph Classification”, Graph Learning Benchmarks Workshop, WWW 2021. 2. Bacciu D., Errica F., Micheli A., Podda M., “A Gentle Introduction to Deep Learning for Graphs”, Neural Networks, 2021 3. Kipf, T. N., Welling, M., “Variational Graph Auto-Encoders”, Bayesian Deep Learning Workshop, NIPS, 2016 4. Hamilton, W., Ying, Z., Leskovec, J., “Inductive representation learning on large graphs”, NIPS, 2017 5. Dwivedi, V. P., Joshi, C. K., Laurent, T., Bengio, Y., & Bresson, X., “Benchmarking Graph Neural Networks”, arXiv 2020. 14

  15. Questions? Thank you! You can reach me at: andrea.cossu@sns.it andreacossu.github.io CIML homepage: ciml.di.unipi.it PAI Lab: http://pai.di.unipi.it/ 15

More Related