A terrific read with a lot of relevant food for thought for current problems and challenges. It's also just an accessible way to get to know Shannon's theory (it was the introduction to Shannon's the Mathematical Theory of Communication).
Cool technique for data selection with the goal of getting higher performance from fine tuning. Not sure I get the interpretability arguments, but a lot of excellent technical choices, and much to learn from.
A critique of the DELPHI paper. A lo of interesting food for thought. Some points are developed well, some less. The aspect of liability, an immediate issue with Ml models, could have been developed further.
Extending the DPO trick to advantage-based preference model, which is shown to be more sound in a recent work by Knox. What makes this paper a treat is the practical discussion of regularization, etc.
A pretty interesting result showing that removing components (without explicit optimization) from internal weight matrices in LLMs improves performance on some end tasks. Interesting connection to LORA, and much remain to outline boundaries of result.
A really nice and elegant connection between meta learning and in-context learning, solving some of the nested optimization difficulties of meta learning.
A very nice trick relying on predictable randomization that helps save memory in zero-order SGD. The focus in on LLMs, where reducing the memory needs has significant impact, but the method is pretty general.