The 54 permanent delegates to the National Council of Provinces were sworn in by Chief Justice Pius Langa on Thursday afternoon.
Mninwa Johannes Mahlangu was elected as the chairman of the NCOP. His was the only nomination for the position.
"I am honoured and humbled by your confidence in me," Mahlangu said.
"Our first key task is to establish the broad key issues to focus on in this term."
He said one of the NCOP's important tasks was to enhance its working relationship with provincial legislatures.
"We need to roll up our sleeves and make sure things happen," said Mahlangu, who is commonly known as "M.J".
Fifty-four of the 90 NCOP's delegates are permanent members -six for each of South Africa's nine provinces. The remaining 36 are sent by the provinces to attend NCOP sessions when necessary and do not have voting rights.
Western Cape Premier Helen Zille was booed as she lambasted members for chanting "ANC" when Mahlangu was elected.
"Mr Chairman, we think that you have done a good job, that is why we did not oppose you," Zille said.
"We stood and applauded when you were elected. But we feel that it was completely wrong for members of the ruling party to chant for the ANC when you were elected.
"It shows that many members of this house have a lot to learn about the separation of party from the State."
Zille said the NCOP had to stick to its mandate as a conduit from the provinces to Parliament.
"Our mandate is to make the voices of the people from the provinces heard. Our mandate is not from Polokwane," she said of the ANC's last congress held in Polokwane in 2007.
Zille promised to work with the national government in the spirit of co-operation.
"I hope that every member of this house, regardless of party affiliation, will take our government to account when it falters," she said.