Reading what Machines ‘Think’

1. Reading what Machines ‘Think’ Fabio Massimo Zanzotto and Danilo Croce UniversityofRome “Tor Vergata” Roma, Italy

2. Prelude Question This is a fascinating research problem. Can we find a more controlled setting where we can test if this is possible? Tom Mitchell, Invited Talk at the AssociationforComputationalLinguisticsConference 2007 BrainActivation Pattern Recognizer Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A.: Predicting human brain activity associated with the meanings of nouns. Science 320(5880) (May 2008) 1191–1195 chair

3. Idea ComputationalMachine Brain Cognitive task Observing a chair Sorting a vector Cognitive physicalobject Observed image

4. Motivation Why investigating the computer side is relevant? • Foundational perspective • Computers are becoming extremely complex. They are fastly approaching the complexity of human brain • Computers are controlled machines: their behavior and thier internal organization is known • Then, computers offer a way to estimate if the claim on the brain side is reachable: if we can read what machines think, we can hope to read what brains think.

5. Motivation Why investigating the computer side is relevant? • Applicative perspective Can we develop technologies that “read the computer mind”? This predictive model can have a wide variety of applications, e.g., detecting malicious software, detecting the intentions of hostile computers by looking at their activation patterns.

6. In the rest of the talk • Investigating the computer side:Long term research program • Sketching the overall observation activity • Virtual Observation of Processes • Experimental Investigation

7. Long-term research program VirtualMemoryDump (organized in processes) ProcessMemoryDump PhysicalMemoryDump PhysicalMemory Chip 011101111000010101010101111101010101110101101111110001... 011101111000010101010101 101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110001101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110000101010111010110111111000010101011101011011111100001010101110101101111110001… 011101111000010101010101111101010101110101101111110001... Physicaldeviceactivationimagecapturer Virtualactivationimagecapturer Virtualactivationimagecapturer Virtualactivationimagecapturer

8. Sketching the overall observation activity Sorting a vector ObservedPhenomena: Processes BrainActivation Pattern Recognizer ProcessActivation Pattern Recognizer Building imagesfromprocesses Definingfeaturespacesforimages Sorting a vector chair

9. Observed Phenomena: Processes

10. Building activation images from processes Given a cognitive activity, the procedure for extracting images from this activity is then the following: • running the process p representing the cognitive activity c • stopping the process at given states or at given time intervals • dumping the memory associated with the process M(p) • given a fixed height image and the memory dump, read incrementally bytes of the memory dump and fill the associated RGB pixel with the read values I(M(p))

11. Building activation images from processes Process memory in a given time interval Activation image of the process in a given time interval where is the RGB pixel definition of the image

12. Building activation images from processes Smoothing (more similartoreal chip observation) Process: Vector Sorter Initial State Process: Vector Sorter Final State

13. Defining feature spaces for images We used three major classes of features • Chromatic feaures • Capture the color properties of the image determining, an n-dimensional vector representation of the 2D chromaticity histograms • texture (OP - OGD) features • emphasize the background properties and their composition. • transformation features (OGD)

14. Experimental Evaluation • Experimental Set-up • Collection of activation images • Used Machine Learning algorithms • Experimental Results

15. Experimental Set-up • 3 different “cognitive tasks” (algorithms) • sorting, comparing two strings, visiting a binary tree • 3 different programming languages • c, java, php • for each pair algorithm-programming language • 20 different randomly generated input data • 3 snapshots (beginning, middle, end)

16. Experimental Set-up We randomly splited the final set (540 images) in: • Training: 270 images Testing: 270 images Two classification tasks: • Determining the programming language (3 classes) (lang) • Determining the cognitive task (3 classes) (algo) Used Machine learning Models: • Decision Tree Learners (DecTree) • Naive Bayes

17. Results Classificationaccuracy

18. Conclusion The parallelism between computer and brain/mind is not new in general • Cognitive psychology • Cognitive sciences We looked this parallelism from an other perspective

19. Future Work VirtualMemoryDump (organized in processes) ProcessMemoryDump PhysicalMemoryDump PhysicalMemory Chip 011101111000010101010101111101010101110101101111110001... 011101111000010101010101 101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110001101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110000101010111010110111111000010101011101011011111100001010101110101101111110001… 011101111000010101010101111101010101110101101111110001... Physicaldeviceactivationimagecapturer Virtualactivationimagecapturer Virtualactivationimagecapturer Virtualactivationimagecapturer

20. Thankyoufor the attention!