Deep Dive: Mixture of Experts Explained

How MoE LLMs differ from Dense LLMs?

Earlier this week, I talked about Databricks release of DBRX LLM and its implication on the enterprise landscape. DBRX also happens to be a Master of Experts (MoE) LLM built on MegaBlocks research and this type of LLM architecture has been gaining traction lately so I thought to publish a deeper dive into MoEs.

Let me know via comments if this deep dive…