1: \begin{abstract}

2: Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks -- thus raising important questions about when and how meta-learning algorithms should be deployed. 

3: In this paper, we seek to clarify these questions by 

4: 1. proposing a novel metric -- the {\it diversity coefficient} -- to measure the diversity of tasks in a few-shot learning benchmark and 

5: 2. by comparing Model-Agnostic Meta-Learning (MAML) and transfer learning under fair conditions (same architecture, same optimizer, and all models trained to convergence).

6: Using the diversity coefficient, we show that the popular \textit{mini}ImageNet and CIFAR-FS few-shot learning benchmarks have low diversity. 

7: This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under a fair comparison. 

8: Specifically, we empirically find that a low diversity coefficient correlates with a high similarity between transfer learning and MAML learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD). 

9: To further support our claim, we find this meta-test accuracy holds even as the model size changes. 

10: Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly.

11: We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.

12: \end{abstract}

13: