How an open-weight model outsmarts frontier private models

Kimi K3 is no doubt a real breakthrough. I think this paper is important; a lot of people might think that in order to have a really great LLM, you need to have some kind of private weights. And it is kind of logical: if I had the weights of a Frontier model on my computer, I don't know if I would want to share them! But China is probably playing a different game. By sharing a model this big and clever, it is not only expanding its soft power, but it also reminds the Americans that others can build LLMs too, and that theirs are as good. Kind of the same trick as when DeepSeek got released, and Nvidia lost 23% or something like this in one day. And seeing how they handled energy loss to train Kimi K3, they really did a good job. Now let's go through 2 or 3 things that really caught my attention.

First: The Attention Residuals

What the Kimi Team did here is pretty clever: at each layer, instead of giving the full story of the previous blocks, they only give a summary of them. This allows for lower computational time.

Second: The MoE

More than the attention residuals, activating only a part of the experts during inference substantially lowers the computational time.

Third: The Kimi Delta Attention

Here again, we can see that they are less and less focused on the size of the model, and more on solving an optimization problem for a specific model.

I think this model should be one of the last of its kind. I would expect research to drift toward optimizing the compute/energy ratio, lowering the consumption of the whole algorithm instead of making it bigger again.