Dortmund went through 5-4 on aggregate after winning the first-leg in St Petersburg 4-2 three weeks ago.
But the Zenit defeat was Dortmund’s third Champions League loss of the season after defeats at Napoli and home to Arsenal in the group stages and this was a below-par performance from last season’s finalists.
Brazil star Hulk, a constant threat on the left wing, gave Zenit an early lead with a superb strike before Dortmund drew level when veteran captain Sebastian Kehl headed their equaliser just before the break.
Then Venezuela striker Jose Rondon, who joined Zenit in January from Russian rivals Rubin Kazan, came off the bench to head home the second-half winner.
Zenit will unveil ex-Chelsea and Tottenham Hotspur boss Andre Villas-Boas as their new manager on Thursday.
The Russians must have impressed their new manager, having produced a spirited display and taken the game to Dortmund, nearly achieving the three goals they needed in Germany to progress.
Hulk breathed some life back into Zenit’s Champions League ambitions when he put the guests ahead with 16 minutes gone after he cut inside two defenders and his powerful shot curled away from goalkeeper Roman Weidenfeller into the top left-hand corner.
At the other end, Dortmund’s Poland striker Robert Lewandowski was booked on 19 minutes for a handball while leaping for a corner and the yellow card will have repercussions.
It was his third of the tournament and means he is suspended for the first leg of the quarter-final. Dortmund discover their opponents in Friday’s draw.
Dortmund’s left winger Kevin Grosskreutz then forced Zenit goalkeeper Vyacheslav Malafeev into a good save while Sokratis’ headed well over the bar as the hosts went on the attack.
Attacking midfielder Oleg Shatov nearly added Zenit’s second when he fired in a shot which forced Weidenfeller to palm away.
But Dortmund went into the break level when left back Marcel Schmelzer fired in a cross, which Kehl headed home after it bounced up off the turf on 38 minutes, even though Malafeev got a glove to it.
Hulk, not short on confidence after his goal, fired over with a fierce free-kick on 51 minutes.
Things became untidy in the second half with passes going astray.
Dortmund just shaded possession, but were punished for sitting back when Rondon got in behind the defence to poach the winner.
Italian left-back Domenico Criscito swung in a superb cross and Rondon slipped his marker Mats Hummels for a diving header to cause Dortmund some late nerves while Weidenfeller was forced into several saves.