The IRS also is implementing large parts of President Barack Obama's health law, including enforcing the mandate that most people get health insurance. Republicans in Congress abhor the law, putting another bull's-eye on the agency's back.
The animosity is reflected in the IRS budget, which has declined from $12.1 billion in 2010 to $11.3 billion in the current budget year.
Obama has proposed a 10 percent increase for next year; Republicans are balking.
Rep. Ander Crenshaw, R-Fla., chairman of the House subcommittee that oversees the IRS budget, called the request "both meaningless and pointless" because it exceeds spending caps already set by Congress.
Koskinen said he suspects some people think that if they cut funds to the IRS, the agency won't be able to implement the health law. They're wrong, he said.
The IRS is legally obligated to enforce the health law, Koskinen said. That means budget savings will have to be found elsewhere.
Koskinen said he can cut spending in three areas: enforcement, taxpayer services and technology. Technology upgrades can only be put off for so long, he said, so enforcement and taxpayer services are suffering.
Last year, only 61 percent of taxpayers calling the IRS for help got it. This year, Koskinen said he expects the numbers to be similar. To help free up operators, callers with complicated tax questions are directed to the agency's website.
"The problem with complicated questions is they take longer," Koskinen said.
Your chances of getting audited vary greatly, based on your income. The more you make, the more likely you are to get a letter from the IRS.
Only 0.9 percent of people making less than $200,000 were audited last year. That's the lowest rate since the IRS began publishing the statistic in 2006.
By contrast, 10.9 percent of people making $1 million or more were audited. That's the lowest rate since 2010.