City had lost all seven previous meetings between the teams in the competition, but Tony Pulis' men produced a tremendous display to secure a richly deserved victory.
Arsene Wenger made nine changes from the team which beat Bolton in midweek, with only skipper Cesc Fabregas and Denilson retained.
As predicted, 35-year-old Sol Campbell made his second Arsenal debut - the best part of four years since his last appearance for the Gunners.
But Campbell and the rest of the visitors' defence were bystanders when Stoke took the lead after just 71 seconds.
Rory Delap's long throw from the right was speared towards the edge of the six-yard area where Fuller darted in to plant a firm header into the top left-hand corner of Lukasz Fabianski's net.
Arsenal's passing game was not up to its usual standard and they were making little impression against a well drilled home defence.
And Stoke threatened to extend their lead in the 25th minute when Fuller skinned Armand Traore to set up Mamady Sidibe, whose shot was blocked close to goal.
Campbell settled to put in some trademark challenges, but Fuller remained a threat and may well have earned a 34th-minute penalty if he had gone down under Mikael Silvestre's ill-timed challenge.
But Arsenal drew level three minutes before the interval after referee Martin Atkinson had penalised Dean Whitehead for handball at the corner of the penalty area.
Everyone expected Fabregas to shoot, but he rolled the ball square to Denilson and the Brazilian's low drive took a slight deflection off Whitehead and sped past the unsighted Thomas Sorensen.
The contest really opened up at the start of the second half with both teams going for it in what had developed into a cracking cup tie.
A Fabregas shot was tipped over by Sorensen, but Stoke were doing most of the attacking with Matty Etherington a consistent threat and Fuller belting a shot over after robbing Silvestre.
And Pulis' men were inches away from taking a 63rd-minute lead when Francis Coquelin almost diverted Sidibe's cross into his own net with Etherington waiting to pounce.
Wenger brought on Andrey Arshavin, Eduardo and Aaron Ramsey in the 68th minute in an attempt to turn the tide.
But Sidibe and Fuller, who were about to be substituted themselves, combined for Stoke's 78th-minute winner.
Sidibe hurtled down the right flank and delivered the perfect cross over Campbell for Fuller to cushion his header past the wrong-footed Fabianski from six yards.
And Whitehead made sure with a close-range tap-in from Etherington's inch-perfect cross four minutes from time.