Refugees have unfinished business with Labor