Panetta also urged Pakistan to go after the Haqqani militant network, one of the United States' most feared enemies in Afghanistan, and said Washington would exert diplomatic pressure and take any other steps needed to protect its forces.
"It is an increasing concern that safe havens exist and those like the Haqqanis make use of that to attack our forces," he said.
"We are reaching the limits of our patience for that reason. It is extremely important for Pakistan to take action to prevent (giving) the Haqqanis safe havens, and for terrorists to use their country as a safety net to conduct attacks on our forces."
The comments came as Washington appears to be looking to other allies in the region for help in the face of Pakistan's foot-dragging. Panetta arrived in Kabul after a visit to India, Pakistan's old enemy, where he urged New Delhi to take a more active role in Afghanistan.
NATO has signed an agreement with three countries to the north of Afghanistan for land routes as the U.S.-led alliance begins a withdrawal of its forces from the country next year.
NATO chief Anders Fogh Rasmussen said earlier this week the "reverse transit" deal was signed with Uzbekistan, Kazakhstan and Kyrgyzstan.
Pakistan closed the shorter and cheaper routes through its territory last year to protest a cross-border NATO air attack that killed 24 Pakistani soldiers. Discussions to reopen the Pakistan routes have stalled.
Other irritants in U.S.-Pakistan ties include drone attacks in the lawless areas of Pakistan near the Afghan border where several militant groups operate but seen by many Pakistanis as a violation of sovereignty.
Washington says the attacks are crucial to attack militants and four days ago, a U.S. drone strike in northwest Pakistan killed al Qaeda's second-ranking leader, Abu Yahya al-Libi. It was the biggest blow to the militant group since the killing of bin Laden.
Tensions between Washington and Islamabad have also flared because a Pakistani doctor accused of helping the CIA find bin Laden was jailed for 33 years for treason last month.
"We intend obviously to take whatever steps necessary to protect our forces. That's a principle we always stand by. For that to happen, we have to have Pakistan cooperation to take steps to control the Haqqani threat from their side of border."
Pakistan has strong traditional links with the Afghan Taliban and other militant groups. Islamabad denies allegations it uses them as proxies to gain leverage in Afghanistan ahead of any settlement to the war, or in case a civil war breaks out after most foreign combat troops leave in 2014.
The South Asian nation has said its military is too stretched fighting homegrown Taliban militants to take on the Haqqanis, and it alone would decide when to take action.
Islamabad's cooperation is considered critical to U.S. efforts to stabilize Afghanistan before most foreign combat troops withdraw. But Washington will find it difficult to persuade Pakistan to both go after the Haqqani network and re-open the supply routes.
Anti-American feelings runs deep over issues like drones, and with a general election expected in early 2013, no politician will want to be seen as soft on the Americans.
U.S. President Barack Obama, for his part, will have to look tough against militancy during an election year in the United States, where many people may ask why the United States backs a country where bin Laden was said to have lived for years.