Alexis Sanchez slotted home a last-gasp penalty as 10-man Arsenal moved into second place in the Premier League but Wenger was furious at referee Jonathan Moss' decision to award Burnley a stoppage-time spot-kick of their own.
Andre Gray stepped up and made it 1-1 and cancel out Shkodran Mustafi's second-half opener.
Wenger was given his marching orders by Moss and stopped near the tunnel to watch the rest of the match and Sanchez's winner, before getting into a heated discussion with Taylor and appearing to push him away.
The Arsenal boss went on to apologise for his actions during his post-match press conference but could face action from the Football Association.
"I regret everything
I should have shut up, gone in and gone home
I apologise for that," Wenger said.
When asked if he had spoken to Taylor since the incident, the Frenchman added: "Not yet
I have a press conference to make.
"Look, it was nothing bad
I said something that you hear every day in football
Overall nine times out of 10 you are not sent to the stand for that
If I am, I am and I should have shut up completely
I was quite calm the whole game, more than usual."
Wenger also insisted that Granit Xhaka needs to show greater control.
Xhaka was shown his ninth red card since 2014 - and second for the Gunners - after a two-footed lunge on Steven Defour in the 66th minute.
"He has to control his game and not punish the team with a lack of control in his tackling," the Wenger said.
"We don't encourage our midfielders to go down on tackles
We want them to stand up and not make these kind of faults
If it is a bad tackle, it is a red card.
"It was an intense game where we played against a team who made the game very uncomfortable for us but we had to be patient and not make defensive mistakes
With that we did quite well but once we were 1-0 up we made it even more complicated by going down to 10 men.
"In the end we still got an unlikely win, because when you are 10 against 11 with two minutes to go it is difficult to win the game but we are a team who keeps going no matter what happens and that is a good quality."
Burney manager Sean Dyche was left to rue Moss' decision to give Arsenal a spot-kick in the last of seven additional minutes after Ben Mee's high foot connected with Laurent Koscielny in the box.
In October, Koscielny was on hand to help Arsenal claim a 1-0 win at Turf Moor after his controversial last-minute goal denied the Clarets a point.
Dyche felt Koscielny had handled the ball in the first meeting between the teams and this time felt an offside decision went against his side.
"Two last-minute incidents against the same side, home and away, has cost us two points," Dyche said.
"That's an unfortunate twist of fate that's for sure
We worked diligently to soften their threat
They get a lad sent off and then we've had to work hard to get something.
"Then you can't believe the decision (to award Arsenal a penalty)
He (Koscielny) was offside, it's as simple as that
We can only hope over a season that we get something back."
Source : PA
Source: PA