LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training (“dominant priors”), can often “outweigh” information we give it

LLM inference is more accurate when we give them examples (demonstrations) rather than just describing what we want.

Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model. They will always be “driving in fog”, with local, short-range probabilities crowding out long-range ones. In case you were wondering why they suck at the “big picture” – probabilistically, it’s a blur.

I’d guess that what all this stuff adds up to is, sustainable use of LLMs as a coding tool for nontrivial projects calls for an entirely reworked set of software development practices to conform to its limitations effectively, but the people in charge really really want and believe it to be a drop-in efficiency boost, and a big mess results. This reminds me a lot of the articles and arguments I’ve read over the years about low level vs high level programming languages and frameworks. Probably will play out a similar way.