Teaching Machines to Learn by Metaphors Omer Levy & Shaul Markovitch Technion Israel Institute of Technology Concept Learning by Induction
Few Examples Transfer Learning Target (New)
Source (Original) Define: Related Concept Transfer Learning Approaches Common Inductive Bias
Common Instances Common Features Different Feature Space Example -3 -2
0 2 3 Example -3 -2
0 0 2 3
4 9 Example -3 -2
0 2 3 = 0
4 2 9 Common Inductive Bias
-3 -2 0 0 2 3
4 9 Common Inductive Bias -3 -2
0 0 2 3
4 9 Common Instances -3 -2
0 0 2 3 4
9 Common Features 3 2 4
9 -2 -3 =
2 New Approach to Transfer Learning Our Solution: Metaphors
Metaphors Target (New) Source
(Original) Source Concept Learner
Target Metaphor Learner
h +/-
h ( )= h ( ( ) ) is a perfect metaphor if: 1. is label preserving 2. is distribution preserving
Theorem If is a perfect metaphor - and is a source hypothesis with error - then is a target hypothesis with error
The Metaphor Theorem If is an -perfect metaphor - and is a source hypothesis with error - then is a target hypothesis with error
Redefine Transfer Learning Given source and target datasets, find a target hypothesis such that is as small as possible. Redefine Transfer Learning Given source and target datasets,
find an -perfect metaphor such that is as small as possible. Metaphor Learning Framework Concept Learning Framework
Search Algorithm Hypothesis Space h
Evaluation Function Data Metaphor Learning Framework
Source Search Algorithm Metaphor Space
Evaluation Function Target
Metaphor Evaluation Metaphor Evaluation 1. is label preserving 2. is distribution preserving
Metaphor Evaluation 1. is label preserving Empirical error over target dataset 2. is distribution preserving Statistical distance between and
Metaphor Evaluation ( ( ) , ) Metaphor Evaluation
Metaphor Evaluation ( (
), ) Metaphor Evaluation
Metaphor Spaces Metaphor Spaces General
Few Degrees of Freedom Representation-Specific Bias Geometric Transformations R
Dictionary-Based Metaphors cheese queso Linear Transformations
Which metaphor space should I use? Which metaphor space should I use? Automatic Selection of Metaphor Spaces
Which metaphor space should I use? Automatic Selection of Metaphor Spaces Occams Razor Which metaphor space should I use?
Automatic Selection of Metaphor Spaces Occams Razor Structural Risk Minimization Automatic Selection of Metaphor Spaces
1 2 3 4 Automatic Selection of Metaphor Spaces
1 1 2 2
3 3 4
4 Automatic Selection of Metaphor Spaces 1 1
60 % 2 2
90% 3 3 91 %
4 4 70%
Empirical Evaluation Reference Methods Baseline Target Only Identity Metaphor Merge
State-of-the-Art Frustratingly Easy Domain Adaptation Daum, 2007 MultiTask Learning Caruana, 1997; Silver et al, 2010
TrAdaBoost Dai et al, 2007 Digits: Negative Image Digits: Negative Image
( ) =1 Digits: Negative Image Digits: Higher Resolution Digits: Higher Resolution
Digits: Higher Resolution Wine
Wine Qualitative Results Transfer Learning Target Task Instance
Target Sample Size 1 Digits: Negative Image Digits: Higher Resolution
2 5 10
Discussion Recap Problem: Concept learning with few examples Solution: Metaphors Recap
Problem: Concept learning with few examples Solution: Metaphors Target Source Recap Problem: Concept learning with few examples Solution: Metaphors
Target Source Generic framework Recap Problem: Concept learning with few examples Solution: Metaphors Target Source
Generic framework Wide range of relations Recap Problem: Concept learning with few examples Solution: Metaphors
Target Source Generic framework Wide range of relations
Learn the difference What if the concepts are not related? What if the concepts are not related? Metaphors are not a measure of relatedness
Metaphors are not a measure of relatedness Metaphors explain how concepts are related Vision M ETAPH O R S
Explaining how concepts are related since 2012.