Photo: Alex Brandon/AP

John Kelly

John Kelly, the former White House chief of staff, has revealed what it was like to work withDonald Trump‘s family members who currently serve as the president’s advisors.

“They were an influence that has to be dealt with,” Kelly replied after being asked if it was complicated to have the president’s family working at the White House during his tenure as chief of staff.

Kelly immediately qualified his statement, adding that he was not referring to First LadyMelania Trumpwith his remarks.

“By no means do I mean Mrs. Trump,” he said. “The first lady’s a wonderful person.”

The White House did not immediately respond to PEOPLE’s request for comment.

Apparently, at one point while Kelly was chief of staff, Trump reportedly told him tofire Kushner and Ivanka.

According to Vicky Ward’s bookKushner Inc., which was based on 220 interviews, Trump told his chief of staff, “Get rid of my kids; get them back to New York.”

Bill Clark/CQ Roll Call/Getty

Speaker of the House Paul Ryan Weekly News Conference

“[The president] has mused to Kelly that he thinks Ivanka and her husband should perhaps return to New York, where they would be protected from the blood sport of Washington and less of a target for negative media attention, White House officials said,” theWashington Post‘s Ashley Parker and Philip Rucker wrote.

However, in response to Ward’s book, the White House denied her written account of Trump’s reported conversation with Kelly.

“It’s sad, but not surprising, the media would spend time promoting a book based on shady anonymous sources and false information instead of all the incredible work Jared and Ivanka are doing for the country,” Press Secretary Sarah Sanders said in a statement to PEOPLE.

Peter Mirijanian, who spoke on behalf of Kushner attorney Abbe Lowell, also denounced Ward’s account in a statement to theNew York Times, which initially preview Ward’s manuscript in March.

“Every point that Ms. Ward mentioned in what she called her ‘fact checking’ stage was entirely false,” he said. “It seems she has written a book of fiction rather than any serious attempt to get the facts. Correcting everything wrong would take too long and be pointless.”

source: people.com