Gunners boss Arsene Wenger named an experimental line-up, with defensive duo Tom Cruise and Kyle Bartley both handed first-team debuts.
The visitors enjoyed the better of the opening exchanges, but 15 minutes in Olympiacos sprang forward and Konstantinos Mitroglou fired a left-foot shot over the top from 15 yards.
Luciano Galletti fired a volley narrowly wide of targets six minutes later, while Arsenal's best openings came via the pace of Theo Walcott drifting in off the right flank.
Dudu was off-target with a couple of long-range efforts for the Greeks as a disappointing first half ended with no sign of a breakthrough at either end.
However, just two minutes after the restart the home side hit the front as the Gunners lost the ball in midfield and Leonardo raced clear down the left and slotted a cool finish under the advancing Lukasz Fabianski.
Four minutes later Aaron Ramsey led a quick counter-attack and his superb pass found Carlos Vela, but his shot was blocked by Antonios Nikopolidis.
Vela passed up another fine opening 20 minutes from time after Fran Merida's astute pass released the Mexcian, who could only poke his left-foot shot over the crossbar from 15 yards out.
Right at the death Arsenal almost conjured up an equaliser as a corner from the right deflected towards goal and goalscorer Leonardo made a fine block on the line to ensure that his side held on for the victory.