A family of finetuned and merged models that reached the top positions of the Open LLM Leaderboard. How did they do it?
How to reduce the cost of your model?
Photo by Alexander Mils on Unsplash
In recent years, model parameters have exploded to a huge number of parameters (540 B with PaLM). The question that has been asked is whether this number of parameters is necessary.
According to OpenAI, as models grow, there is an increase in performance. In addition, there is the appearance of emergent properties (properties that cannot be observed except at a certain scale).
This view has been challenged by the fact that actually more data, and thus scaling is limited by the number of tokens needed to train a model optimally. Moreover, even these emergent properties may not even exist.