Koscielny's winner rebounded in off his elbow in the third minute of added time and Arsene Wenger conceded his side had been lucky to mark his 20th anniversary as manager with victory.
The Gunners had swept aside Chelsea and Basel in their previous two matches but were forced to dig deep at Turf Moor for a fifth straight Premier League success and Germany international Mustafi explained its significance.
"It was all about coming here and giving everything," the £35million summer signing from Valencia told Arsenal's official website.
"It suits us more to play against big teams because it's not always easy to come to places like this and change from style to steel.
"But we showed that we have good character and steel, and that's what gave us the three points in the end."
Burnley were forced to play on the counter for long periods such was Arsenal's mastery of possession, but the Londoners spurned numerous goalscoring chances.
Alexis Sanchez, Theo Walcott, Mesut Ozil and Alex Iwobi had all been guilty of dithering when well-placed, while Burnley went close to making them pay with their limited chances.
Johann Berg Gudmundsson's header was turned away at full stretch by Petr Cech and Michael Keane headed a 75th-minute corner against the woodwork, although replays appeared to show the goalkeeper got his hand to it.
"The great teams fight until the end because they know that there's still an opportunity to score a goal and we showed that," Mustafi added.
"It just showed us that we are capable of scoring from every situation, even in the last minutes
A win like this is important to us."
In all Wenger's 20 years and 1,130 games in charge, few will have had a more dramatic ending but Burnley boss Sean Dyche took the blow squarely on the chin.
He refused to go banging on referee Craig Pawson's door after the final whistle and took solace instead in his own side's performance.
Midfield pair Jeff Hendrick and Steven Defour looked more like the club's two record signings before fading, while as well as going closest to breaking the deadlock with his second-half header, centre-half Keane underscored his credentials for a senior England call-up.
"That's up to Gareth (Southgate)," Dyche said
"I've got open communication with him and if he calls me, Michael will certainly get my backing.
"There were a number of good performances and that's why there's frustration in the outcome of course, but I've got to look at my players and what they're giving me and giving each other.
"There's no question they give a lot, in their performance, their will and demand and that drive it takes to win games.
"We know we're not market leaders, we're the minnows of the division, you can see by our spending power compared to everyone else.
"But that doesn't mean we're a side that doesn't want to take on the challenge because I think you saw that in abundance."
Source : PA
Source: PA