Priscilla Presley has received lots of mail from fans of her late husband, Elvis Presley - and not all of it is positive. "They said he wouldn't have passed if I hadn't left him. It was a guilt trip, completely. I've gotten a few of those letters, which were very, very difficult to read," Priscilla, 80, told the crowd at the 92nd Street Y in New York City on Wednesday, September 24. "To have someone think those things ... I had nothing to do with his passing. I'm mourning the loss of him. I still mourn him to this very day." Priscilla was married to the late Elvis from 1967 to 1973. She wrote about leaving the musician in her new book, Softly, As I Leave You, which was released on Tuesday, September 23. (The former couple shared late daughter Lisa Marie Presley, who was born in February 1968. Priscila left Elvis when their daughter was 4 years old.) "He was an amazing father, an amazing human being," Priscilla added during Wednesday's appearance. "I couldn't find anyone who was as giving as he was." Priscilla Presley Reveals Why She Left Elvis in New Book: Biggest Revelations Priscilla explained in her new memoir that - contrary to popular belief - her affair with karate instructor Mike Stone was not the reason she ended her marriage to Elvis. "I did not leave Elvis because I no longer loved him. I loved him as much on the day I left as on the day we married," she wrote. "I left Elvis because I needed a life of my own. I was living his life. Just as Elvis was searching for the meaning of his life, I was searching for mine." Priscilla claimed that her and Elvis' "marriage had been crumbling for years" before she decided to leave. Her affair was "just one catalyst," while Lisa Marie's birth was the other. Manoli Figetakis/Getty Images "I became jealous of a nanny taking care of my daughter when I should be taking care of her," Priscilla wrote. "I felt guilty about having a nanny there when Lisa woke up in the middle of the night and started crying." She admittedly "struggled" during this time in her life. "It was painful trying to keep our family together," Priscilla wrote. "I felt like a failure as both a mother and a wife. Eventually, I had to choose." She recalled Elvis being "stunned" by her decision to leave. Despite their split, the exes stayed close. Elvis would even call Priscilla when she was (unknowingly) in bed with another man. How Priscilla Presley 'Freed' Herself From Being Known as Elvis' Wife When Priscilla was dating Robert Kardashian Sr., the rock legend called her in the middle of the night. "Robert was sound asleep in bed with me when the phone rang at 2 a.m.," she wrote. "Fortunately, he was a sound sleeper, so I leaped out of bed and grabbed the phone before it woke him. Elvis was calling from the hotel after a performance." She explained to readers that it "certainly never occurred" to Elvis that she might be with someone else. "Despite our divorce, he still couldn't wrap his head around my being with someone else," she added. Neither Elvis nor Kardashian knew about the phone call that night.