The Victorian government has condemned a violent attack on the Indigenous protest camp, Camp Sovereignty, in Naarm by extreme-right demonstrators.

Footage seen by National Indigenous Times and widely circulated online shows men dressed in black attacking people at the camp in King's Domain on Sunday during nationwide anti-immigration rallies.

A 16-minute Facebook Live video, posted shortly after 5 p.m., captures camp members calling for peace and urging the intruders to leave.

"Stop stop, no stop," one person can be heard yelling. "Please leave."

The men, including neo-Nazi figure Thomas Sewell, were seen verbally abusing camp members and stomping on the Aboriginal flag. Some protesters also chanted, "Aussie, Aussie, Aussie, oi oi, oi," before fleeing as police arrived.

In a statement from Camp Sovereignty, they argued: "This was not random violence. These racists launched their attack at the time we regularly hold community gatherings for weekly healing ceremonies."

"These ceremonies are dedicated to honouring our ancestors, promoting peace and unity, and working to heal the wounds of genocide and the deep rift imposed between First Nations peoples and settlers. To target us at this moment reveals that their true intent is to desecrate what is sacred and to violently disrupt our efforts at collective healing on behalf of racism, fascism and white supremacy."

Organisers said police arrived after the men had already left and confirmed no arrests had been made. Four camp members were reportedly injured.

Camp Sovereignty quotes an unnamed, 30-year-old school teacher, who said: "I had what looked like a 15-year-old boy rip my hair, throw me to the ground and smash into my face with his fists. He did it with a smile on his face. I couldn't believe it."

Co-founder of Camp Sovereignty and spokesperson for the Black Peoples Union, Keiran Stewart-Assheton, said the attack shows "what Australia has always been — a fascist, white supremacist colony built on genocide".

"They came armed with poles to attack a group of predominantly women at a peace camp, but our sovereignty cannot be broken — we remain armed with truth, culture, and community, and that is far stronger than their hate," he said.

Victorian Police Minister Anthony Carbines called the incident "disgraceful".

"Police will be investigating those matters," he told reporters.

"They'll also be in contact with members of Camp Sovereignty and a lot of our First Nations, people who are deeply distressed and upset with that activity yesterday, and this is what happens, isn't it, when you have bullies in the community who roam in packs to intimidate others, it's gutless, and it needs to be called out, and we will hold those people to account."

The protests across Australia were also condemned by Minister for Multicultural Affairs Anne Aly, who described them as "clearly racist".

"Let's be very clear. Let's not be coy about this. They weren't protesting immigration from white Western countries. They were very clearly protesting immigration from countries that have brown people," she told ABC Radio.

"One of the very clear calls to action that was listed there was anti-Indian immigration, against people coming from India. Now that, to me, is clearly racist when you target a specific ethnicity."