Aboriginal communities whose supposed views were cited by Peter Dutton as grounds for his opposition to the Voice to Parliament have accused the opposition leader of taking their comments out of context.

Mr Dutton referred to the alleged views of Indigenous people of areas ranging from Laverton and Palm Island as the basis for his rejection of a Voice to Parliament.

However, Pakaanu Aboriginal Corporation chief executive Marty Seelander told the ABC the majority of the Laverton Aboriginal community is backing the Yes campaign.

"It's taken out of context because in the initial conversation we had on his visit, the opposition leader's visit to Laverton, it was around that we want constitutional change," Mr Seelander told the national broadcaster.

"But we as a community, we did mention that we would like to know from the referendum if it was voted yes, how we would be represented from our community on that council."

Mr Seelander, along with the Laverton and Leonora Shire mayors, recently met with both Prime Minister Anthony Albanese and Mr Dutton in Canberra.

He said the Voice was not a key issue of discussion with Mr Dutton.

"We were a bit confused by his comments, because that certainly wasn't discussed when we recently visited Canberra," he told the ABC.

"Ninety-nine per cent of the meeting was to do with the issues in our community and around the abolishment of the CDC card. There was not much time spent on gaining our perspective on the Voice."

Palm Island Bwgcolman Elder Elizabeth Clay also rejected Mr Dutton's representation of her community's views.

"He should [support the Voice] but we don't expect him to. He'll oppose everything," she told the ABC.

"[It makes me feel] like I've felt all my life, like we are nothing. It does hurt."

Ms Clay said Palm Island "needs a lot of help" and that she hopes the Voice can help tackle some of the community's biggest concerns.

"There are over 5000 of us here, and you still have 20 people to a house. It's hard sometimes; it causes a lot of friction and the young people don't have jobs," she told the ABC.

"All the kids are going to school and graduating, but there are no jobs. We know the answers to our problems, and we are hoping the Voice will help us get a handle on it."

Ms Clay also rejected Mr Dutton's suggestion that Aboriginal academics can't represent the wider Indigenous community.

"I believe the academics will represent us all, who cares if they are academics or not, they are Aboriginal too," she told the ABC.

She also noted that calling it "the Canberra Voice" as a criticism was ill-founded.

"It has to be a Canberra voice, because that's where Parliament House is," she said.

"We do need a Voice in parliament, but there needs to be more consultation… We all agree with the Voice … the idea of a Voice to Parliament means a lot to them [Palm Island residents]."