I don't see any of the episodes as filler—not even the body swap and grudgby episodes—because for me, those episodes beautifully demonstrated how magic works/what's possible with magic and showcased some of the culture/sports of the Boiling Isles.
Just my perspective—I know everyone has a different opinion, which I respect.