Wayne Rooney scored twice, netting either side of the controversial sending off of City captain Vincent Kompany, with Danny Welbeck also on target to make it 3-0 before the break for the visitors
But City pulled a goal back early in the second half when Aleksandar Kolarov bent a free-kick round the wall from the edge of the box.
And 10-man City struck again when Sergio Aguero made it 3-2 in the 65th minute following a mistake by United substitute Paul Scholes, summoned out of retirement by manager Sir Alex Ferguson following an injury crisis.
Premier League leaders City, three points in front of champions United at the top of the table, pressed for an equaliser, with Kolarov going close from a second free-kick in stoppage time, but Ferguson’s men held on.
Defeat meant City’s hold on the Cup lasted just one match, as the third round is the stage where Premier League clubs enter the tournament.
“I think we made them (City) better than they were,” Ferguson told ITV.
“We were just so careless in the second half.
“At 3-0 up at half-time they thought they were through and took their foot off the peddle.
“It was a careless performance, we should have been home and dry.”
City boss Roberto Mancini, clearly unhappy with the 12th red card shown to Kompany by referee Chris Foy, praised his team’s character after going 3-0 down.
“It’s best that we talk about the game and not the referee.
“I am happy because I think we played a fantastic performance, the players did well in the second half.
“We scored two times, we had other chances to score in the end and we didn’t concede any chances for them to score.”
Rooney gave United the lead against the run of the play in the 10th minute when, after spreading play wide to Antonio Valencia, he sprinted into the box and then, from the winger’s cross, he powered in a header off the crossbar.
And, two minutes later, City — three points in front of champions United at the top of the table — were down to 10 men.
Kompany appeared to win the ball cleanly from Nani, who played on, after sliding in on a wet pitch but the City defender had gone in two-footed and referee Chris Foy produced a red card.
Welbeck made it 2-0 with a brilliant volley from the edge of the box in the 30th minute and 10 minutes later he forced a penalty after he was brought down in the box by Kolarov.
Rooney’s spot-kick was well saved by Costel Pantilimon, only for the England striker to head in from the rebound as United look to avenge their 6-1 league defeat by City at Old Trafford in October.
Then, with half an hour left, Scholes came on to replace Nani as his former United midfield colleague David Beckham watched from the stands.
But it was Scholes who set up City’s second goal when he gave the ball away to James Milner, whose cross was met by Aguero.
His initial effort was saved by Anders Lindegaard but Aguero scored on the follow-up in the 65th minute.
However, United held on for victory.
Sunday’s other ties see Chelsea face 2008 FA Cup winners Portsmouth while Sunderland look to continue their revival under Martin O’Neill at Peterborough and Sheffield Wednesday play West Ham.