But even though it’s not an election year, the office is in full campaign mode, with volunteers working the phones and reviewing maps in anticipation of a new front of modern campaigning — the recall phase.

A handful of Democratic state lawmakers in Colorado face recall petition efforts in what looks to be the first wave of fallout over legislative votes to limit gun rights. In an era in which recall efforts are booming, from governor’s offices down to town councils and school boards, the Colorado efforts will serve as the first test of gun-rights groups’ ability to punish elected officials who expanded gun control laws after last year’s Aurora, Colo., and Newtown, Conn., shooting massacres.

In Colorado, gun-rights activists wasted no time seeking recalls to oust state Senate President John Morse and three other Democratic lawmakers. The targeted lawmakers weren’t necessarily the main advocates for ratcheting back gun rights, but all come from districts with enough Republicans to give opponents hope they can boot out the Democrats and replace them with lawmakers friendlier to guns. Colorado is the only state outside the East Coast to have adopted significant statewide gun controls this year.

“Colorado seems to be the testing ground for some of the gun measures, so this has national implications,” said Victor Head, a plumber from Pueblo who is organizing a recall attempt against a Democratic senator.

Two of four recall efforts in Colorado already have evaporated from lack of support. But in Colorado Springs, Morse opponents are piling up signatures in gun shops and outside libraries and grocery stores. The National Rifle Association sent a political mailer saying it was coordinating the recall effort with local groups, though the local recall petitioners have denied that. The NRA did not return calls for comment on their involvement in the Colorado Springs effort.

Morse has mounted a campaign to urge voters not to sign petitions. In an indication of the national stakes, that push is largely funded by a $20,000 contribution from a national progressive group called America Votes. The Morse campaign said the donation came through the group’s local Colorado office.

The recall group’s main funding comes from a $14,000 contribution from a nonprofit run by a local conservative consultant, Laura Carno. She said that contribution was made possible by some out-of-state donors.

“People in other states that are further down this road, like New York and Massachusetts, are calling up and saying `What can we do to help?”‘ Carno said. “This isn’t what Colorado stands for.”

In an interview, Morse seemed resigned to facing a recall vote after signatures are verified. He believes national gun-rights supporters are using his district to make a national statement about the political peril officials face if they take on gun control.

“That’s what’s going on here. They want to take out the Senate president,” Morse said.

The organizer of the Morse recall effort, Anthony Garcia, didn’t disagree. Garcia doesn’t live in Morse’s district but in the northern Colorado town of Brighton. Garcia said Morse was targeted not just because of his votes for gun control but because he’s a prominent Democrat from a competitive district.

“It’s as much about saying Colorado is angry as it is about getting one guy out,” Garcia said. “Legislators need to know when citizens are outraged that they can’t ignore the people.”

Immediate accountability seems to be a common thread in recall attempts, said Joshua Spivak, who tracks recall elections nationwide at the Hugh L. Carey Institute for Government Reform at Wagner College in New York. Technology makes it easier to organize, Spivak said, and modern-day voters watching political activity in real time on Twitter and TV aren’t content to wait until another election to show their displeasure when they feel ignored.

Spivak said at least 169 officials at all levels of government faced recalls last year, up from 151 the year before. The number this year could go even higher, he said.

Technology isn’t the only explanation.

“The other reason,” Spivak said, “is that they succeed.”

Most recalls actually fail, as in the case last year of Wisconsin Gov. Scott Walker, a Republican who survived a recall election after attacking collective bargaining rights for state employees. But compared with re-election campaigns, when incumbents face up to 75 percent likelihood of winning, Spivak said recall elections have a much lower rate of success for incumbents.

In Colorado last year, seven recall efforts made it to ballots, all local races, Spivak said. Of those seven, two officials were ousted and two more resigned.

Nationwide, 108 recalled officials last year lost or left office after a recall. That makes the recall a powerful tool — and one likely to be used more often, Spivak said.

Back in Colorado Springs, a couple of Morse opponents defended the recall attempt as the best way for citizens to keep their representatives accountable.

“I believe in gun rights. And he didn’t listen. He’s supposed to represent the people, and when he doesn’t do that, what are supposed to do? Nothing?” asked Bianca McCarl, a 40-year-old merchandiser who is supporting Morse’s recall.

Assuming the Morse recall goes to ballots, with an election to be held by late summer, the incumbent holds a slight party registration advantage in the district. He believes most voters liked his gun votes.

He’s counting on the support from voters like Joan Muir, a retiree who placed a pro-Morse sticker on her car bumper after seeing other cars carrying messages calling for his ouster. In an interview, Muir said she was dismayed by the recall campaign.

“I live here. I’m for gun control,” Muri said. “I don’t care for guns, period, so they don’t speak for all of us when they say Morse didn’t listen to the people.”

That means Lerner, head of the exempt organizations division, probably won’t answer any questions on what she knew about IRS agents going after Tea Party-related groups. That also means she probably won’t say why she sat on the information for so long before it became public.

Lerner’s attorney William Taylor III asked committee Chairman Darrell Issa, R-Calif., in a letter if she could skip Wednesday’s hearing since she would be pleading the Fifth.

Taylor argued in the letter that forcing Lerner to appear “would have no purpose other than to embarrass or burden her.”

Late Tuesday, the House oversight committee released a statement saying Lerner was still under subpoena and would be required to appear in the morning.

“Chairman Issa remains hopeful that she will ultimately decide to testify tomorrow about her knowledge of outrageous IRS targeting of Americans for the political beliefs,” committee spokesman Ali Ahmad said in a statement.

Other former or outgoing IRS officials have already testified, and will continue to give their testimony on Wednesday. But Lerner, who is the official who first acknowledged the IRS program, has faced significant scrutiny.

Since the Department of Justice has launched a criminal investigation into the IRS scandal and the House committee indicated it would question Lerner about why she provided incomplete information to the committee at least four times last year, Taylor wrote that his client would be invoking her Fifth Amendment right against self-incrimination.

The House committee is also scheduled to hear from Deputy Secretary of the Treasury Neal Wolin, among others, as the search for someone who will claim responsibility continues.

On Tuesday, outgoing IRS Commissioner Steven Miller, was back in the hot seat as he testified for the second time in two weeks on Capitol Hill.

Miller expressed regret for the agency’s decision to use a planted question to go public with the IRS’s practice of singling out conservative groups.

It was one in a series of missteps that have not only publicly marred the reputation of the IRS but also called into question what the White House knew about the scandal and when they knew it.

“We’re not looking for people to be evasive but we want people forthright and straightforward with us,” Rep. Joseph Crowley, a Democrat from New York, told Fox News.

While Crowley did not go so far as to say Lerner should be let go, he did say, “the bottom line is that you cannot lie to Congress, be evasive or mislead. You must answer the question and not mislead Congress.”

Separately, two Tea Party-related groups filed lawsuits against the IRS this week.

On Tuesday, Texas-based True the Vote claimed it was unfairly targeted by the IRS and demanded in court documents the government admit its mistake, grant the group tax-exempt status and pay for damages totaling more than $85,000.

On Monday, the NorCal Tea Party Patriots filed the first federal suit against the national tax agency. Like True the Vote, the northern California group says the IRS violated its constitutional rights when it held up its applications for tax-exempt status.

The NorCal lawsuit, filed in the U.S. District Court of Cincinnati,  seeks group status for “all conservative and libertarian groups targeted for additional scrutiny” between March 2010 and May 2013. It’s also seeking unspecified monetary damages for the alleged violation of its constitutional rights and the costs associated with trying to comply with IRS demands.

The lawsuit is being backed by Citizens for Self-Governance, a group launched by Tea Party Patriots co-founder Mark Meckler.

Meckler claims that IRS agents also demanded massive amounts of disclosure of information not authorized by the Internal Revenue Code or any other federal law. The suit alleges that the tactic was used to delay or dissuade conservative groups from going through with their applications.

The IRS acknowledged that employees at its Cincinnati office had targeted conservative groups, creating massive amounts of paperwork or rejecting applications altogether.

Fox News’ Chad Pergram and Fox Business Network’s Rich Edson contributed to this report.

But her case is receiving new attention after her congressman, Republican Texas Rep. Kevin Brady, recounted her ordeal during a high-profile House committee hearing on the IRS practice of singling out conservative groups. He claimed she was audited and visited by several different federal agencies — including the FBI — in the years after she formed her Tea Party group.

“She received four FBI inquires,” Brady said. “And her business received unsolicited audits, unscheduled audits.”

Engelbrecht is the founder of True the Vote, a Houston-based group that says its goal is to root out voter fraud. She’s claimed for years that the Internal Revenue Service was picking on her, asking her over-the-top personal questions, demanding binders full of paperwork and going after her family’s oil field machinery business.

After her case was spotlighted at Friday’s hearing, Engelbrecht explained the targeting in an interview with Fox News.

“At some point those questions cross the line,” she said.

Her troubles started, she claims, after she founded a Tea Party group called the King Street Patriots and worked as poll watcher in the local 2009 elections. She said she found major discrepancies in voting procedures and uncovered cases of political harassment and wrongdoing. She brought up the claims to local officials but they were never proved.

In July 2010, she applied for nonprofit status for two of her organizations: True the Vote and King Street Patriots.

Five months after she filed for 501(c)(4) status for KSP, Engelbrecht says the FBI Domestic Terrorism Unit called her about one of the people who had attended a KSP group meeting. Five months after that, the FBI called her again to ask “how we were doing?” A month later on June 2011, she received another FBI general inquiry, which was followed by two more in November and December.

In February 2012, the IRS asked her questions about her nonprofit application.

“The first contact we had from the IRS, in the first general round of questions, they wanted to see every Facebook posting we had made, every tweet we tweeted,” Engelbrecht told Fox News’ “Huckabee.”

Engelbrecht said IRS agents wanted to know every place she had ever spoken publicly, to whom she had spoken, what she said and her intent.

Engelbrecht said her fight for 501 (c)(3) tax exempt status for True the Vote was even more frustrating. She underwent five rounds of questioning from the IRS and still hasn’t received approval. What’s worse, she claims, is that the government also started going after her personal tax returns and those of her family’s business.

Engelbrecht says the Bureau of Alcohol, Tobacco, Firearms and Explosives also conducted an unscheduled audit of Engelbrecht Manufacturing in February 2012. That was followed by another unscheduled audit in July 2012 by the Occupational Safety and Health Administration, and another in November 2012 by the Texas Commission on Environmental Quality.

At times, Engelbrecht said she thought about not coming forward with her story.

“I really thought that the best way I could protect my family would be to keep my head down,” she said.

Both Republicans and Democrats have lashed out against the IRS over its systematic scrutiny of conservative groups during the past two election cycles.

Engelbrecht said the questions she had the biggest problems with were the ones that focused on her family.

Calls to the FBI and IRS for comment were not immediately returned.

But when news broke that the government was using taxpayer money to bail out Wall Street banks, he started paying attention and didn’t like what he was hearing.

So the Florida father and information technology specialist decided to form a group, Unite in Action, to educate people in his area about the issues, he said. It was originally formed as a corporation before Devereaux decided to apply for tax-exempt status from the IRS.

That was two years ago. It was never approved.

“It’s all but killed us,” Devereaux told FoxNews.com. “We could lose everything. Today, it’s me and my organization, but tomorrow it could be you.”

Devereaux is among a group of activists, being represented by the American Center for Law and Justice, who are preparing to sue the federal government for the practice of targeting Tea Party groups. ACLJ Executive Director Jordan Sekulow told FoxNews.com he’ll likely file the civil suits next Wednesday or Thursday on behalf of more than a dozen Tea Party groups who say they were singled out by the IRS and had their tax-exempt status severely delayed or denied altogether.

The suits, combined with congressional inquiries and an FBI probe, signal that the heated hearing on Capitol Hill Friday – with the outgoing IRS chief – was just the start of a protracted legal and political battle over the scandal.

Sekulow said the number of plaintiffs in the civil suit are growing as is the list of who his organization wants held accountable. It’s still unclear whether the organization will file as a class-action or individually in the 17 different states where the complaints originate.

Litigation could take months or years and for some like Devereaux, time isn’t on their side.

While initially waiting for IRS approval, Devereaux dipped into his own bank account, maxed out credit cards and even borrowed money from friends so his group could put on a civic-engagement training session at the Omni Shoreham hotel in Washington. His goal was to eventually set up a steady stream of revenue for a tax-exempt nonprofit.

The next time Devereaux heard from the IRS, they had requested details and credentials on every single speaker and all the educational materials provided in the 78 classes held at the hotel. The IRS also wanted information on all 45 vendors, their credentials and a donor list.

Devereaux refused.

Five rounds of IRS letters later, and United in Action’s tax-exempt status is still in limbo.

If they are denied, Devereaux’s group would owe the federal government “somewhere in the neighborhood of $70,000 in back taxes,” he said, referring to money he would owe the government on donations.

“It’s more than we have in our bank account,” he said.

He’s not alone.

Waco Tea Party President Toby Walker said her group applied for a 501(c)(4) status in July 2010. She’d call the IRS from time to time to check on the progress but was basically told, ‘Don’t call us, we’ll call you,’ she said.

Then in February 2012, the IRS finally made contact.

Walker said she was asked questions that went well beyond the purview of the agency’s authority. They wanted to know everything about the Waco Tea Party group, their relationships with public officials, lists of volunteers and every single news story the group had ever been mentioned in.

Walker said the request was so lengthy and intrusive that had she complied with the demands, she “would have needed a U-haul truck of about 20 feet.”

While Walker’s group was finally granted tax-exempt status in March 2013, she said a lot of damage has already been done. She said people were afraid to support her group financially because they had not received the IRS-stamped status.

Others were afraid that they might be targeted by the IRS if they supported Walker’s group publicly. Having one of the most powerful government agencies angry at them wasn’t a risk many people were willing to take. And so the group suffered, she said.

“We spent thousands of our own dollars fighting this,” she said. “If this happens to one organization in America, we should all be outraged.”

Allegations that the IRS had been targeting conservative groups that applied for tax-exempt status date back years but a government report released Wednesday backed up the claims. The White House has spent most of the week trying to contain the fallout from the scandal. “Americans are right to be angry about it, and I am angry about it,” President Obama said earlier this week.

By Friday, two of the agency’s top tax officials had been removed from their posts. One, outgoing acting IRS commissioner Steven Miller, was grilled Friday morning in the House Ways and Means Committee by Republican and Democratic lawmakers who demanded answers on why the unfair practice of targeting conservative groups was allowed to continue on his watch.