I think it’s more useful to look at the tasks models haven’t gotten better at over time, and the tasks that are hard for them get better at in principle. The two best examples of these are:

  1. Deep familiarity with the codebase
  2. Technical communication