Retrieval Meets Long Context Large Language Models

This paper aims to explore the performence of RAG and long context large language model. A very interesting point is that retrival can significantly improve the performance of LLMs regardless of their extended context window sizes. In addition, they also explore why retrieval enhancement fails to improve the performance of large models with long contexts for small numbers of parameters. This is because the bottleneck of the model at this point is not in the length of the context that the model can input, but rather in its own performance.