The Vietnamese accusation came days after U.S. President Barack Obama visited Asia to underline his commitment to allies there, including Japan and the Philippines who are themselves locked in territorial disputes with China.
Obama, promoting a strategic "pivot" toward the Asia-Pacific region, also visited South Korea and Malaysia, but not China.
Vietnam has condemned the operation of the deepwater drilling rig in what it says are its waters in the South China Sea, and told China's state-run oil company to remove it.
In Washington, U.S. State Department spokeswoman Jen Psaki told reporters: "Given the recent history of tensions in the South China Sea, China's decision to operate its oil rig in disputed waters is provocative and unhelpful to the maintenance of peace and stability in the region."
"These events point to the need for claimants to clarify their claims in accordance with international law, and reach an agreement ... about what types of activities should be permissible within disputed areas," she added.
Vietnam also protested the move.
"Vietnam cannot accept this, and resolutely protests this action by China," the foreign ministry said on its website, summarizing comments by Foreign Minister and Deputy Prime Minister Pham Binh Minh, who spoke to his Chinese counterpart by telephone on Tuesday.
"We request China pulls out the ... rig and all vessels from this area ... Vietnam will take all suitable and necessary measures to protect our legitimate rights and interests."
A ministry official said the two countries had been in direct talks about the issue since Sunday, but did not say how China had responded to Vietnam's requests. China has said the rig was operating completely within its waters.
Daniel Russel, Assistant U.S. Secretary of State for East Asian and Pacific Affairs, said the United States was looking into the matter, but urged caution from all sides.
"We believe that it is critically important for each of the claimant countries to exercise care and restraint," he told Reuters during a visit to Hong Kong ahead of a previously scheduled trip to Hanoi on Wednesday.
"The global economy is too fragile and regional stability is too important to be put at risk over short term economic advantage."
China claims almost the entire oil- and gas-rich South China Sea, rejecting rival claims to parts of it from Vietnam, the Philippines, Taiwan, Malaysia and Brunei. It also has a separate maritime dispute with Japan.
Its claims coincide with growing diplomatic and military influence in the region and have raised fears of possible conflict.
The Maritime Safety Administration of China (MSAC) announced on its website on Saturday that all vessels should keep one mile away from the rig, called the Haiyang Shiyou 981. It expanded that to three miles on Monday.
The $1 billion rig is owned by China's state-run CNOOC oil company and it had been drilling south of Hong Kong.
On Sunday, Vietnam said the coordinates of the rig put it in Vietnam's exclusive economic zone and on its continental shelf, about 120 nautical miles off its coast.
But, like other Asian nations involved in territorial disputes with China, Vietnam appears to have limited options when dealing with the emerging superpower.
The Philippines said last month that the United States had a treaty obligation to help in case of an attack on its territory or armed forces in the South China Sea, although Obama did not say categorically that Washington would do so.
In 1992, Vietnam sent naval vessels into an area where China signed a contract with a U.S. firm to develop oil and natural gas in what it said were its waters.
"From 1992 until now, I haven't seen any action from Vietnam stronger than that," said a Vietnamese academic who specializes in South China Sea affairs.
"My guess is either this action from China is to send a message to the United States after Obama's Asia visit, or to direct the community to this topic to distract them from the terror in Xinjiang."
China's nervousness about Islamist militancy has grown since a car burst into flames on the edge of Beijing's Tiananmen Square in October and 29 people were stabbed to death in March in the southwestern city of Kunming.
The government blamed militants from the far-western region of Xinjiang for both attacks.
China routinely sends patrols into the South China Sea, mostly involving the coast guard and civilian maritime protection force rather than the navy.
But the positioning of such a large structure in disputed waters was seen by some analysts as a significant escalation in the dispute.
Singapore-based South China Sea expert Ian Storey said the rig movement risked a "potentially very dangerous scenario."
"There have been standoffs with survey ships in the past, but this is something new," said Storey of the Institute of South East Asian Studies.
"There's been a great deal of speculation about how China would use this expensive new rig and it seems we now have the answer. It puts Vietnam in a very difficult position."
"They will have to respond to a challenge to their sovereignty, and when they do, China will be sure to make a counter move, so we are in a situation where a potentially very dangerous scenario could unfold."
China's Global Times, an influential tabloid published by the ruling Communist Party's official People's Daily, wrote in an editorial on Tuesday that China should show a "firm attitude" towards Vietnam.
"China follows a moderate policy. But no country can always show a smiling face to the world. China shouldn't be angered easily, but if its interests are infringed upon, a strong retaliatory move should be expected," it said.- Reuters