Zheng Zeguang was due to attend a Commons reception on Wednesday, hosted by the all-party group on China.
But after protests, Speaker Sir Lindsay Hoyle and Lord Speaker Lord McFall ruled this out.
The Chinese embassy said it was a "despicable and cowardly" decision that would harm both countries' interests.
The ban, which was first reported by the Daily Telegraph, comes at a time when tensions between the two governments are high.
In March, China imposed travel bans and asset freezes on five MPs and two peers whom it accused of spreading lies about the country.
This was in response to the UK's decision to impose its first sanctions against Chinese officials for human rights abuses in Xinjiang.
Despite this, the China parliamentary group still decided to invite Mr Zheng to its summer party on the Commons terrace pavilion overlooking the Thames.
Last week, the five Conservative MPs who were sanctioned - Sir Iain Duncan-Smith, Tom Tugendhat, Nusrat Ghani, Neil O'Brien and Tim Loughton - wrote to the Speaker voicing their concerns.
The two sanctioned peers - crossbencher Lord Alton and Labour's Baroness Kennedy - wrote to the Lord Speaker.
They said: "The sanctions imposed by the Chinese government represent an attack not just on members directly targeted but on Parliament, all parliamentarians, select committees, and parliamentary privilege.
"We should never allow our place of work to become a platform to validate and promote such sanctions.
"We know that this is a view shared by a great many Right Honourable and Honourable Members who will wish their protests to be heard if this visit is to go ahead."
They added: "It is unthinkable therefore that parliamentarians should have to suffer this infringement on our liberties whilst the prime representative of the Chinese government in the UK is still apparently free to come to Westminster and to use facilities here as a mouthpiece for his regime."
Nusrat Ghani welcomed the "unprecedented" ban on the ambassador, telling BBC News: "Sanctioning MPs is a direct threat to Parliament and our democracy.
"We will not be silenced nor intimidated by any regime, and our Parliament will not be become a tool of propaganda for Chinese Communist Party absurd sanctions."
Another sanctioned MP, Tim Loughton, tweeted: "If the genocidal Chinese regime think they can shut down free speech by parliamentarians in a democracy there are consequences and in this case it is that the Chinese regime must not have a platform in the Mother of Parliaments."
In a statement, Sir Lindsay said he held regular meetings with ambassadors to "establish enduring ties between countries and parliamentarians".
But he added: "I do not feel it's appropriate for the ambassador for China to meet on the Commons estate and in our place of work when his country has imposed sanctions against some of our members.
"If those sanctions were lifted, then of course this would not be an issue."
Lord McFall's spokeswoman said the "meeting should take place elsewhere, considering the current sanctions against members including two members of the Lords".
A Foreign, Commonwealth and Development Office spokesperson said: "Parliament is independent of government. It is for the Speaker to decide who is allowed on the parliamentary estate."
A spokesman for the Chinese embassy said: "The despicable and cowardly action of certain individuals of the UK Parliament to obstruct normal exchanges and cooperation between China and the UK for personal political gains is against the wishes and harmful to the interests of the peoples of both countries."