Refugees take fight for permanent visas to Labor’s doorstep