Of course we need a more numerate country, but let’s get the politics of this right at the same time