Prime Minister Anthony Albanese has clashed with 2GB host Ben Fordham in an at-times heated interview over the Voice to Parliament.

Mr Albanese accused the host of "reading from the No pamphlet" and urged him to "get on board" if he feared the Voice referendum would fail.

Recent polling has shown a significant decline in support for the Voice.

While the federal government has outlined Indigenous health, education, housing, and employment as key proprieties for the Voice, Mr Fordham quizzed the Prime Minister on comments from Yes campaigners implying the Voice would have an influence beyond that, mentioning changing the date of Australia Day and speaking directly to the Reserve Bank.

"I can't talk directly to the RBA board, and I'm the prime minister ... You need to not raise red herrings. You know full well that if this is successful and there's a Voice, it won't have a right of veto [over government decisions]," Mr Albanese said.

"I'm worried about it failing, PM," Mr Fordham replied, promoting Mr Albanese to urge the host to "get on board… (and) make a difference".

Mr Fordham raised the possibility of two referendums, one for Constitutional recognition of Indigenous people and one for the Voice.

"You'll romp it in, and you can legislate the Voice, and you can get the body that you want. It is a win-win ... Why be so stubborn and to keep the two of them together?" he asked.

The Prime Minister said the call for a Voice to Parliament did not originate from politicians but from Indigenous leaders who "said they don't just want the symbolism of recognition, they want something that will make a practical difference to their lives".

Mr Albanese said the polling showed a downward trend for the Voice because to date the debate has been held "between politicians in Canberra", and predicted the dynamic would shift in the months to come.