How Much Memory Does Your Agent Actually Need?