Australia's first Indigenous Senior Counsel, Tony McAvoy, said it was clear Peter Dutton would oppose the Voice after speaking with him, long before the Coalition made their position official.

As a panellist at Garrmalang Festival's Blak Talk discussion on the Voice in Darwin on Saturday, Dr McAvoy said his assessment was that Mr Dutton was "going through the motions" when he engaged in discussions with the Referendum Working Group.

Dr McAvoy was present in Referendum Working Group meetings with Mr Dutton and former shadow shadow minister for Indigenous affairs Julian Leeser.

The coalition's announcement against the proposal came after a special party room meeting in the week following their loss at the Aston by-election, won by Labor on April 1.

Mr Leeser resigned from the opposition front bench following the decision.

Senator Jacinta Nampijinpa Price, a vocal opponent of the Voice, was later appointed shadow minister of Indigenous affairs.

Dr McAvoy said the Coalition failed to engage with the proposal on its merits, and that the process was an attempt from Mr Dutton to "salvage something of his own reputation" as leader after losing the seat in Melbourne's east.

"(The short-notice party room meeting) told me he was never going to vote in favour of it (the Voice), and in fact, this was a political football for him to use for his own interest and for his party's interests and that they weren't engaging in the issues on their merits," he said on Saturday.

"It was a political knee-jerk response to the loss of Aston and for Mr Dutton to salvage something of his own standing within his party.

"He probably was at risk of losing his position as leader, and that's why that position was taken."

Dr McAvoy said the fallout was "emblematic of all Aboriginal Affairs policy in this country".

"We (Aboriginal and Torres Strait Islander people) are collateral damage to everything else that happens. And that's the absolute reason why we need a Voice to Parliament and the executive to stop that type of policy-making for reasons other than what we're asking and what's in our benefit," he said.

Dr McAvoy said he has experienced people change from "spouting this mantra" that the Voice lacks detail, to being more receptive to the proposal once he's explained to them it's something of concern and to be self-determined by First Nations people.

Fellow panellist, Senator for the Northern Territory and Assistant Minister for Indigenous Australians Malarndirri McCarthy, was less scathing about her contemporaries in parliament, but said Dr McAvoy's "passion about the direct interaction that he and the working group had with the Opposition Leader" sent a clear message.

"It think it shows the rest of the country…that we are trying to bring every single person on board," Senator McCarthy said.

"Whether it's the Opposition Leader or others with him - even knowing that they're toying with us, that they're playing with us, that we know that they're going to say no, we have to keep trying in the goodness of the Uluru Statement from the Heart."

The senator commended Mr Leeser's resignation to the backbench, Ken Wyatt's resignation from the Liberal Party and the support of Tasmanian Liberal MP Bridget Archer as "valued moments of principle".

Last week the Voice re-entered Parliament for debate following the Joint Select Committee's recommendation the Constitution Alteration Bill pass without alteration.

In his address, Mr Dutton labelled the Voice "regressive" and said it would result in racial division, despite having previously said he supported local and regional Voices.

Minister for Indigenous Australians Linda Burney slammed the Opposition Leader for spreading "every bit of disinformation and misinformation and scare campaigns that exist in this debate".

Speaking on Saturday, Senator McCarthy said she is hopeful of the bill and referendum ready the end of June.

She told National Indigenous Times "it will be interesting to see who votes where" in the lower house.

"We expect it to pass but we'd like to see who else votes 'Yes' from the other side," she said.

"I'm confident it will pass in our house. Absolutely."