The Gunners had much the better of the first half but could not make their dominance count and Joleon Lescott headed the hosts in front from a 40th-minute corner.
That looked like being the way it would stay as Arsenal struggled to find a way to goal, but Koscielny proved the unlikely hero with an unstoppable shot eight minutes from time.
The Gunners were taking on their opponents with real pace and in the 24th minute Carl Jenkinson did well down the right and pulled the ball back to Lukas Podolski, who skied his shot well over the bar.
Gervinho then fired just wide with a shot that rippled the side-netting and drew cheers from the section of Arsenal fans down the other end who thought it had sneaked in.
Still they had not really tested Joe Hart, though, and they were made to pay in the 40th minute when Lescott headed in David Silva's corner.
City could even have been two ahead at the break but Vito Mannone partially redeemed himself for his error on the opening goal with a fingertip save from Edin Dzeko's curling shot.
Arsenal were still passing the ball around neatly without really looking like scoring, and Gervinho's radar was off target again as he blasted high and wide in the 66th minute.
Arsenal were pressing hard but City almost killed the game off on the break in the 80th minute, Yaya Toure playing in Sergio Aguero for a shot that was smartly stopped by Mannone.
That proved to be a crucial moment as Arsenal went up the other end and Santi Cazorla drew the first real save out of Hart with a fine shot from the edge of the box. The Spaniard took the resulting corner and Lescott could only divert the ball to the feet of Koscielny, who smashed the ball into the top corner.
Source: PA
Source: PA