The Ansar al-Sharia group said they set off a bomb at the base in the central town Bayhan before capturing soldiers.
The attack comes amid fears that Yemen is plunging into chaos after Shia rebels took over the capital.
Shia Houthis have been expanding south from their northern stronghold, prompting clashes with al-Qaeda in Yemen (AQAP) and other Sunni groups.
The UN has warned that Yemen is on the brink of civil war.
The base is in Shabwa province, an al-Qaeda stronghold. It is reportedly home to more than 1,000 troops.
The militants captured heavy weaponry when they took the base, one official told the AFP news agency.
Tribal mediators were trying to convince the group to withdraw, the official added.
Last week, the Houthis dissolved parliament and announced the formation of an interim government.
The move was widely condemned, with critics calling the actions a “coup”.
The US, UK and France have said they are closing their embassies in Yemen due to the deteriorating security situation there.
7AYHVDMBUBIImage source: Wikimedia Commons President-elect Donald Trump celebrated his election victory at the Ultimate Fighting…
Millions of voters across the US chose to return Donald Trump to the White House…
Donald Trump declares victory in the US election as he addresses jubilant supporters in Florida.…
Stocks around the world are rising as Donald Trump appears to be on the cusp…
Donald Trump has won Pennsylvania, North Carolina and Georgia and taken a lead over Kamala…
Quincy Jones, the celebrated musician and producer who worked with Michael Jackson, Frank Sinatra, Ray…