Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. - Inar Timiryasov; Jean-Loup Tastet (CoRR, 2023)

BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data. - Jean-Loup Tastet; Inar Timiryasov (CoRR, 2024)