Another local official told Reuters some of the escapees were "suspected of belonging to Al-Qaeda," and that they disappeared amid heavy clashes between warring militias in the city.
Houthi fighters entered Taiz in March, in a southward push from their base in the capital Sanaa that drew military intervention from a Saudi Arabia-led coalition of regional powers. But three months of air strikes have yet to push back the group.
A security official said army forces linked to former President Ali Abdullah Saleh allowed the inmates to escape as groups of fighters, dubbed "popular committees" by their supporters, advanced on the prison.
"Heavy fighting took place near the central prison and the popular committees approached and seized control of the area, but Saleh's forces opened the prison doors," the official said.
Another group of Al-Qaeda fighters escaped from a prison in the eastern city of Mukalla in April after army forces suddenly abandoned their posts.
Al-Qaeda and other hardline Sunni groups condemn the Shia Houthis as apostates worthy of death, and the two groups are fighting each other in several areas in central Yemen.