The Internal Revenue Service hasn’t been doing enough to collect the taxes owed by hundreds of thousands of people who don’t file their taxes and owe billions of dollars, according to a new report.

The report, from the Treasury Inspector General for Tax Administration, estimated that approximately $39 billion of the $441 billion average annual tax gap, or 9 percent, can be attributed to nonfilers, that is, individuals who don’t file a tax return on a timely basis or pay the taxes due on their delinquent returns. High-income nonfilers who earn at least $100,000 a year, while fewer in number, contribute to the majority of the nonfiler tax gap, according to the IRS.

TIGTA analyzed the inventory of nonfilers for tax years 2014 through 2016 and identified 879,415 high-income nonfilers who didn’t satisfy their filing requirements, with estimated taxes collectively due of $45.7 billion.

Of those 879,415 high-income nonfilers, TIGTA found the IRS didn’t work on 369,180 of their cases, though the total estimated tax due amounted to $20.8 billion. Of those 369,180 high-income nonfilers, 326,579 were not placed in inventory to be selected for work and 42,601 were closed out of the inventory without ever being worked on by the IRS. The remaining 510,235 high-income nonfilers, with a total estimated tax due of $24.9 billion, are sitting in one of the IRS Collection function’s inventory streams and probably won’t be pursued as resources at the agency decline. The IRS has already removed high-income nonfiler cases from its inventory, resulting in 37,217 cases totaling $3.2 billion in estimated tax dollars that probably won’t be worked on by the IRS.

Thanks to the IRS’ policy of working on just single-tax-year cases no matter how many returns haven’t been filed by a taxpayer, the IRS is missing out on opportunities to bring repeat high-income nonfilers back into compliance, according to the report. TIGTA identified the top 100 high-income nonfilers for tax years 2014 through 2016 that the IRS didn’t address or resolve. They had estimated taxes due totaling $9.9 billion.

The IRS is trying to address the problem, but it’s still in the process of conducting testing on a new strategy of dealing with nonfilers. TIGTA acknowledged that the service’s new nonfiler strategy seems to approach nonfiling in a more strategic manner. “However, the strategy has not yet been implemented, and TIGTA identified that the new nonfiler program is spread across multiple functions with no one area being primarily responsible for oversight,” said the report. “In addition, more needs to be done to address high-income nonfilers.”

TIGTA made seven recommendations in the report, including designating a senior management official with appropriate resources and specific nonfiler duties to address nonfilers, not pausing the nonfiler program, working on multiple-tax-year cases, and not removing high-income nonfiler cases from the inventory without resolution.

The IRS disagreed with one of TIGTA’s recommendations, but it agreed with two others, and partially agreed with four more. The IRS agreed not to pause the Individual Master File Case Creation Nonfiler Identification Process in the future, unless there are unusual circumstances, but it disagreed with placing the nonfiler program under its own management structure.

Eric Hylton, commissioner of the IRS’s Small Business/Self-Employed Division, pointed out that the agency has faced staffing challenges that have made it difficult to spend as much time on enforcement. “The issues TIGTA found in this audit are reflective of the resource challenges the IRS has faced in recent years,” he wrote in response to the report. “Since FY 2010, the IRS has lost nearly a third of its enforcement personnel, including more than half of its revenue officers (the Collection employees who work the most complex cases). To address this trend of declining resources, our budget request included funds for new and continuing investments in expanding and improving the effectiveness and efficiency of our overall tax enforcement program.”

Sen. Ron Wyden, D-Oregon, the ranking Democrat on the Senate Finance Committee, blamed Republicans for years of budget cuts at the IRS. “As a result of Republicans’ years-long effort to gut the IRS and protect their donors from scrutiny, wealthy tax cheats stole nearly $46 billion in just three years by refusing to even file their taxes,” Wyden said in a statement Monday. “What’s worse, the IRS did little to nothing to pursue these tax cheats. Investments in health care, infrastructure and education will be perpetually short-changed if paying taxes is essentially voluntary for those at the top. The IRS needs historic investments to address this crisis.”