Elise Nelson / Us Weekly: Breaking Down Hilary Duff's Alleged Feud With Sister Haylie Duff Amid New Song 'We Don't Talk' - Hilary Duff and sister Haylie Duff have been battling - and sometimes avoiding - feud rumors for years. - The Texas-born siblings grew up in the spotlight side-by-side after moving to Los Angeles with their mom in 1993.