Developing the Durham University Museum Artefact Collection

1. Developing the Durham University Museum Artefact Collection For machine learning applications in cultural Heritage Protection and documentation Matthew Roberts Department of Archaeology Department of computer science

2. Research Questions How well does a Machine Learning dataset created from a museum’s digital archive generalize to recognition problems in cultural heritage? How can computer vision and object recognition support efforts in heritage protection?

3. Durham UniversityOrientalMuseum Dataset • The Museum has developed a set of digital records for it’s archives since 2001 • Archaeology, Egyptian, Northumberland, Oriental organizational structure • Oriental Museum provided both original and thumbnail images • File folders grouped as above and by year acquired • Files semi-labelled – Accession Number, View Codes

4. Supervised Machine Learning • Iterative Development • Data Understanding/Data Preparation • Broad classification focusing on shape (Type) and color (Material) • Collect additional data • Balance training classes • Modeling • Pytorch Architecture • Transfer Learning using Resnet • Visualization: tSNE, Visdom, Matplotlib • Manual Selection vs Automated Train/Validation Split Resnet50

5. Asiatic Image Sub-categories

6. Egyptian Image Sub-categories

7. Broad Classification Dataset • All Resnet models converge to 100% validation accuracy within 1100 epochs

8. Evaluation: Test Set 1 (Display Cabinet Images) • Camera phone photos • Angle of photography dependent on cabinet structure and layout • Variable lighting • Glass reflections • Background clutter and occlusions • Blurring from movement and low light • Collected from several museums

9. Museum Display Case Numbers • Durham University Oriental Museum • Bristol Museum and Art Gallery • British Museum • Ancient Egypt • China and South Asia • Enlightenment Room (Room 1) • Victoria and Albert Museum

10. Oriental Museum Display Case Results Resnet34 Resnet18 Resnet50 Resnet152 Resnet101 • Best results from Resnet50; in all models lowest performance from Zoomorphic classes

11. Project Planning: Heritage Documentation and Protection (HeDAP) • Documentation and protection of Libya and Tunisian Cultural Heritage. • Project funded by Durham University and the Cultural Protection Fund (British Council). • In collaboration with major local partners Department of Antiquity of Libya and Institut National du Patrimoine de Tunisie (INP). • Mobile App for FAST RECORDINGof movable objects in open sites and indoor museums and storerooms across Libya and Tunisia. • In collaboration with Interpol to fight the illicit traffic of Antiquities from Libya and Tunisia. • RECORDING and IMAGE RECOGNITION

12. Evaluation: Test Set 2 (HeDAP Images) • Documentation photos acquired using the HeDAP application in the Oriental Museum photo lab • Variations of lighting and background • Followed the established OM method of object photo documentation from multiple angles • Limited to unphotographed figurines (placeholder images used for vessels)

13. HeDAP Tablet App Results Resnet50 Resnet34 Resnet18 Resnet101 Resnet152 • Resnet50 still performing well, though the model shows a bias towards EFZ

14. Future Perspectives • Future perspectives • Basic principles observed, technology concept formulated • Initial experimental proof of concept • Technical Challenges • Data availability and labelling (Small Data) • Research vs Production phase • Technology driven by Classification rather than Identification • Potential Applications in Heritage • Recognition of single objects for illicit traffic of antiquities • Automated monitoring of auction sites • Museum Content Management Systems • More work to be done before it is ready for deployment in operational environment

15. Thank you for your attention!Any questions?