China's coastguard said the arrests were made on Sunday morning off the southern province of Guangdong, near Hong Kong.
Hong Kong media reports said those on board the vessel were trying to reach Taiwan to claim political asylum.
The reports said Hong Kong activist Andy Li was among those detained.
Mr Li, who was arrested earlier this month for alleged collusion with foreign forces and money laundering, was detained on suspicion of "unlawfully crossing the border", the South China Morning Post reported, citing police sources.
It was not immediately clear what those now in custody might be charged with. Attempts by people from Hong Kong to flee the territory by boat are considered to be rare.
Hong Kong has seen a wave of arrests of activists in recent weeks under a controversial national security law imposed by China in June.
The security law, opposed by many in Hong Kong, punishes what Beijing broadly defines as subversion, secession, terrorism and collusion with foreign forces with up to life in prison.
The number of arrests has raised fears that China will use its security law to undertake a broad crackdown on Hong Kong's pro-democracy activists and media figures.
Hong Kong was handed back to China from British control in 1997, but under a unique agreement - "one country, two systems" - that gives the territory freedoms not seen on the mainland.
But critics have accused China of eroding those freedoms, leading to protests in Hong Kong and political tensions between Beijing and the international community.
A social media post by the Guangdong Coast Guard on Wednesday said those arrested were being held on suspicion of unlawfully crossing the border.
It said investigations were under way, but gave few other details. Only two of those detained were partially identified by their surnames, Li and Tang.
The South China Morning Post said sources from police in Hong Kong and mainland China confirmed Andy Li was the Li referred to.
At least one other person on board had previously been arrested on charges related to last year's anti-government protests, the newspaper said.
Hong Kong police commissioner Chris Tang said on Thursday he was aware of the boat interception, but added: "For the time being, we do not have any information from the mainland relevant authorities."
Taiwan, a self-ruled island off China's south-east coast, has sought to help Hong Kongers fearful of a political crackdown by Beijing.
In July, Taiwan opened an office to enable people from Hong Kong to emigrate to the island. The office received more than 1,000 inquiries in its first month alone.
Taiwan has for all practical purposes been independent since 1950, but China regards it as a rebel province that must be reunited with the mainland - by force if necessary.
That has resulted in political acrimony, with Taiwan's President Tsai Ing-wen accusing China of trying to force the island to accept Chinese sovereignty.
President Ing-wen heightened those tensions on Thursday by warning of an increased risk of accidental conflict in the waters around China.
The president complained of Chinese military activities near the island, saying better communication was needed to prevent "miscalculations".
The US and China have also been conducting military exercises in the disputed South China Sea.
The sea has in recent years become a flashpoint for tensions between China and other nations which claim sovereignty over two largely uninhabited island chains, the Paracels and the Spratlys.
China reportedly launched two missiles into the South China Sea on Wednesday, a move analysts saw as a warning to the US.
The missile launches came on the same day the US announced sanctions on two dozen Chinese companies for their role in building artificial islands in the South China Sea.
Speaking in Hawaii, the US Defence Secretary, Mark Esper, said China and America were in a "great power competition".
Beijing's defence ministry, meanwhile, said China would not "dance to the US tune".
Change before you have to.