Ministers say there were "significant technical flaws" in the way the risks for each bid were calculated.
They have also ordered two independent reviews into what happened.
FirstGroup, which beat current operator Virgin Trains to win the 13-year deal, said it had submitted its bid correctly and was "disappointed" at the news.
Writing on his blog, Virgin founder Sir Richard Branson welcomed the move, details of which emerged in the early hours, adding that he was hopeful ministers would "now accept that Virgin Trains should carry on running the West Coast Main Line".
The implications of the decision to scrap the deal go much further than just the West Coast Main Line.
There were about 15 rail franchises due to be decided before the next general election and the whole franchising process could now be thrown into doubt.
The August announcement that FirstGroup would
take over train services on the line - one of Britain's busiest - in
December had sparked a legal challenge from Virgin, which has run the
franchise since 1997.
The Department for Transport said because of the decision to
rerun the bidding process it would no longer be contesting the judicial
review launched by Virgin Trains in the High Court.And it said an announcement would be made about the suspension of Department for Transport staff while investigations were carried out.
Transport Secretary Patrick McLoughlin described the mistakes made by his department as "deeply regrettable and completely unacceptable".
He said: "A detailed examination by my officials into what happened has revealed these flaws, and means it is no longer acceptable to award a new franchise on the basis of the competition that was held.
http://www.dft.gov.uk/news/press-releases/dft-press-20121003a/