Earlier in the interview, host Chuck Todd asked what was "going on" between him and Bannon.
"Can you and Steve Bannon still work together in this White House or not?"
Scaramucci said he didn't know and hasn't directly asked Bannon about his views.
Two senior Trump advisers - one inside the White House and another who recently departed - signaled Sunday that the knives are out for Steve Bannon, President Donald Trump's controversial chief strategist.
But Kelly's rise to White House chief of staff - he was given the role last month when former chief of staff Reince Priebus resigned - has reignited the churn of speculation as Kelly looks at revamping the press and communications team.
"The fact that [Steve Bannon] is sitting there inside the White House probably makes people think that [Breitbart] has an influence, and I think that's reason enough to probably make some changes".
Scaramucci - who held the top White House communications job for just 11 days - urged current administration officials to directly disagree with Trump if he's wrong, but hypothesized that the president responded in that way because he "likes doing the opposite of what the media thinks he's going to do", and he is "of the impression that there is hatred on all sides".
The ex-aide was speaking on the ABC News show This Week when he was asked by host George Stephanopoulos about the statement the U.S. president gave yesterday in which he condemned the violence but not neo-nazis specifically.
Scaramucci is still a supporter of Trump's, saying that the president "can be a tough coach at times" but "this guy's a victor". "The president has a very good idea of the people who are undermining his agenda that are serving their own interests", Scaramucci added.
"My feeling about the whole thing is that what happened was sort of meant to happen", he said. "He's got to move more into the mainstream". The former CEO of Breitbart News, Bannon is often credited with fanning Trump's pugilistic impulses, and boasted about his role in legitmising the alt-right. If he does that, he'll have a successful legislative agenda that he can execute.
"I believe he needed to be much harsher as it related to the white supremacism and nature of that", he declared, adding that the "moral authority of the presidency" demands that the one holding that office "call that stuff out". "If he doesn't do that, you'll see inertia".
"I think the President knows what he's going to do with Steve Bannon", Scaramucci said.