"There is never a perfect time for this type of transition, but now is the right time," Ballmer, 57, said in a statement released by the Redmond, Wash., company.
Microsoft did not name a successor, but it's certain that the person who takes the reigns will need to push the world's largest software company further into mobile devices and step up its competition with faster-moving rivals.
Microsoft said Friday that it's forming a search committee, which will include Gates. Ballmer will stay on until a replacement is found.
After the news broke, Microsoft's stock shot up as much as 9 percent shortly after the markets opened. They came within two dollars of their 52-week high.
Ballmer's announcement comes less than two months after the company unveiled a sweeping reorganization of its business in an attempt to catch up with Apple and Google.
In his statement, Ballmer noted that Microsoft is moving in a new direction and needs a CEO that will be there for the longer term.
Microsoft, he added, "has all its best days ahead."
Ballmer met Gates in 1973 while they were living down a dormitory hall from each other at Harvard University. He joined Microsoft in 1980 to bring some business discipline and salesmanship to a company that had just landed a contract to supply an operating system for a personal computer that IBM would release in 1981.
Ballmer, a zealous executive prone to arm-waving and hollering, did the job so well that he would become Gates' sounding board and succeed him as CEO in 2000. He has worked at Microsoft for 33 years, matching the tenure of Gates, who left the company in 2008.
"It's a tad surprising, but every other business head has been rotated out," said BGC Financial analyst Colin Gillis. "They swapped out all their segment heads over the past few years. The only one they haven't changed is the CEO."