The NZ football association was formed in 1891 but the All Whites didn't play their first international until 1922 (a 3-1 win against Australia in Dunedin).
England have played them twice on an Oceanic tour in 1991 winning both games.
New Zealand have appeared in two World Cups but have gone home in the first round of both despite being the only team not to lose a game in the 2010 edition in South Africa.
NZL, 8th Jun 1991, 2 - 0 (FR)