The Caller analysis of White House visitor logs showed former IRS Commissioner Douglas Shulman visited the White House at least 157 times under the Obama administration.

Even Attorney General Eric Holder, one of Obama’s closest allies, visited only 62 times according to the records.

The records may not reflect every single visit, as some officials do not have to sign in every time they come to the White House.

But they could lend weight to concerns voiced by lawmakers at a hearing last week about the frequency of Shulman’s White House contact. During the time period when the IRS was singling out Tea Party and other groups for extra vetting — as they applied for tax-exempt status — Shulman visited the White House 118 times.

Asked to explain the visits, Shulman gave lawmakers a list of possible reasons.

“The Easter Egg roll with my kids … questions about the administratibility of tax policy … our budget, us helping the Department of Education streamline application processes for financial aid,” he said.

According to the Caller analysis, no other top official logged more than 100 visits.

The official with the next-highest number of visits — close to 90 — was Rebecca Blank, former deputy secretary and now acting secretary of the Commerce Department. Next in line was Thomas Perez, a top Justice Department official who has since been nominated to lead the Labor Department.

Health and Human Services Secretary Kathleen Sebelius and Treasury Secretary Tim Geithner each logged fewer than 50 visits.

Former IRS officials have testified that the scrutiny of conservative groups, while inappropriate, was not politically motivated.

The 29-page lawsuit named Attorney General Eric Holder, Treasury Secretary Jack Lew and several IRS officials — including Lois Lerner, the division director who refused to testify before Congress last week. The suit claims the constitutional rights of 25 Tea Party and other conservative groups were violated when tax workers singled them out for a drawn-out vetting process.

The American Center for Law and Justice is arguing that the Obama administration overstepped its authority and violated the First and Fifth Amendments of the U.S. Constitution, the Administrative Procedure Act as well as the IRS’ own rules and regulations.

“The whole timeline and the whole narrative that the White House has put forth does not hold up to the truth,” ACLJ Chief Counsel Jay Sekulow told Fox News on Wednesday.

In its suit, the ACLJ wants the government to admit wrongdoing. The suit also seeks to protect the groups from future IRS retaliation as well as compensatory and punitive monetary damages.

“The IRS and the federal government are not going to get away with this unlawful targeting of conservative groups,” Sekulow said later in a statement announcing the lawsuit. “As this unconstitutional scheme continues even today, the only way to stop this flagrant and arrogant abuse of our clients’ rights is to file a federal lawsuit, which we have done.”

Sekulow says the suit is intended to “send a very powerful message to the IRS and the Obama administration.”

Emails to the White House and IRS for comment were not immediately returned. Administration officials have said that while the additional scrutiny was inappropriate it was not partisan and therefore no laws were broken.

Allegations that the IRS had been targeting conservative groups that applied for tax-exempt status date back years but a government watchdog report released this month backed up the claims.

The White House has spent most of the last two weeks trying to contain the fallout from the scandal. Multiple congressional panels are currently investigating the allegations. The Justice Department has also launched its own investigation into whether the IRS broke the rules.

By last Friday, two of the agency’s top tax officials had been ousted from the agency. One was outgoing acting IRS commissioner Steven Miller, who was named in the suit. Another official, Lerner, the director of the division that singled out the conservative groups, was placed on leave — apparently after she refused to resign. She, too, was named in the suit.

Lerner last week invoked her Fifth Amendment right not to testify.

Separately, last week two other Tea Party-related groups filed lawsuits against the IRS.

On May 20, the NorCal Tea Party Patriots filed the first federal suit against the national tax agency. The suit, filed in the U.S. District Court of Cincinnati, seeks group status for “all conservative and libertarian groups targeted for additional scrutiny” between March 2010 and May 2013. It’s also seeking unspecified monetary damages for the alleged violation of its constitutional rights and the costs associated with trying to comply with IRS demands.

The lawsuit is being backed by Citizens for Self-Governance, a group launched by Tea Party Patriots co-founder Mark Meckler.

Meckler claims that IRS agents demanded massive amounts of disclosure of information not authorized by the Internal Revenue Code or any other federal law. The suit alleges that the tactic was used to delay or dissuade conservative groups from going through with their applications.
The IRS acknowledged that employees at its Cincinnati office had targeted conservative groups, creating massive amounts of paperwork or rejecting applications altogether.

On May 21, Texas-based True the Vote, filed its own suit against the IRS and is demanding the government admit its mistake, grant the group tax-exempt status and pay for thousands of dollars in damages the group says it suffered.

Lois Lerner, the head of the exempt organizations division which oversaw the controversial targeting of conservative groups, caused confusion on Capitol Hill Wednesday morning when she — according to some lawmakers — tried to have it both ways. She pleaded the Fifth, saying she would refuse to answer questions from a House committee probing the IRS program. But before she did so, she delivered a defiant opening statement declaring she had done nothing wrong.

Rep. Darrell Issa, R-Calif., chairman of the House Oversight and Government Reform Committee, questioned whether she had “effectively waived” her rights, but ultimately dismissed her from the hearing room. But Issa and others are now strongly considering trying to call her back.

“If you could do it the way she wants to do it, then every defendant would come, say ‘I didn’t rob the bank, and I’m not going to answer the prosecutor’s questions,'” Rep. Trey Gowdy, R-S.C., told Fox News. “So we’d all in life like to get out our version without having to answer anyone else’s questions. It’s just not fair. And I don’t think it’s legal.”

Issa made clear by the end of Wednesday’s hearing that he was strongly considering calling Lerner back.

“I must consider this, so although I excuse Ms. Lerner, subject to a recall, I am looking into the possibility of recalling her and insisting that she answer questions in light of a waiver,” Issa said.

He said for that reason, the hearing would stand in recess, but not be adjourned.

In her opening statement, Lerner asserted her innocence.

“I have not done anything wrong,” she said. “I have not broken any laws. I have not violated any IRS rules or regulations, and I have not provided false information to this or any other congressional committee.”

Lerner is represented by lawyer William W. Taylor, who is noted for winning a dismissal of all charges against former IMF director Dominique Strauss-Kahn in a high-profile sexual assault case.

It’s unclear whether Lerner can avoid another round of questioning by the committee.

Rep. Elijah Cummings, D-Md., the top Democrat on the committee who was as tough as any Republican on the IRS witnesses Wednesday, said he thought Lerner was still in her right to refuse to answer questions.

“I’d like to see (hearings) run like a federal court. Unfortunately, this is not a federal court and she does have a right,” Cummings said Wednesday. “And we have to adhere to that.”

Former IRS Commissioner Doug Shulman, who stayed to testify Wednesday, endured a tough round of questioning for the second day in a row. Lawmakers were visibly frustrated as he struggled to explain why he didn’t notify Congress after learning of the practice last year.

But lawmakers are itching to question Lerner, having aired a string of complaints about her own failure to notify Congress.

Lerner touched off the public controversy when, at an American Bar Association conference this month, she apologized for the IRS’ practice of targeting conservative organizations for additional scrutiny. It was the first time the agency acknowledged the practice.

She said she hadn’t revealed the information sooner, because she was never asked. But just two days before the ABA conference, Lerner was specifically asked about the investigation.

Rep. Joseph Crowley, D-N.Y., who had asked her about it, later called her answer evasive.

“The bottom line is you cannot lie to Congress, and you cannot be evasive, you cannot try to mislead Congress,” he said.

Now under the protection of her lawyers and the Fifth Amendment, Lois Lerner is facing a maelstrom of controversy.

Members of Congress are calling her evasive, and question why she didn’t reveal the program sooner — plus her history at the Federal Elections Commission is coming under scrutiny.

Lerner touched off the public controversy when, at an American Bar Association conference earlier this month, she apologized for the IRS’ practice of targeting conservative organizations for additional scrutiny. It was the first time the agency acknowledged the practice.

“They used names like Tea Party or patriots … and they selected cases simply because the application had those names in the title,” she admitted.

She said she hadn’t revealed the information sooner, because she was never asked. But just two days before the ABA  conference, Lerner was  specifically asked about the investigation.

Rep. Joseph Crowley, D-N.Y., asked her if she could “comment briefly on the status on the IRS investigation into these political not-for-profits.”

She said: “Well there’s a questionnaire that began this discussion and there’s also a questionnaire out there that is seeking info from 501 c3,4,5 organizations.”

Crowley later called her answer evasive.

“The bottom line is you cannot lie to Congress, and you cannot be evasive, you cannot try to mislead Congress,” he said.

In the 1990’s, Lerner also served as chief of enforcement at the Federal Elections Commission.

Under her direction, the FEC undertook the largest enforcement action in its history — suing the Christian Coalition for violating campaign laws. The Christian Coalition won, but in one deposition, FEC lawyers asked a defendant if televangelist Pat Robertson prayed for him.

James Bopp, the Christian Coalition’s lawyer, said he was “shocked and appalled” by that.

“Both political activity and religious activity are specifically protected by the First Amendment,” he said.

When Bopp learned years later that Lerner had been promoted to an IRS position, he became concerned.

“She was in effect being promoted for what she had done at the Federal Election Commission and now was going to be expected … to replicate that at the IRS and now we know that’s exactly what happened,” he said.

Lerner is represented by lawyer William W. Taylor, who is noted for winning a dismissal of all charges against former IMF director Dominique Strauss-Kahn in a high-profile sexual assault case.

Lerner said at Wednesday’s hearing that she had done nothing wrong.

“I have not done anything wrong,” she said. “I have not broken any laws. I have not violated any IRS rules or regulations, and I have not provided false information to this or any other congressional committee.”

The lawmakers are expected to ask former IRS Commissioner Douglas Shulman why he didn’t tell Congress that agents had been singling out conservative political groups for additional scrutiny when they applied for tax-exempt status — even after he was briefed on the matter.

Shulman, who was appointed by President George W. Bush, left the IRS in November when his five-year term ended. He is testifying before the Senate Finance Committee, which has launched a bipartisan investigation into the matter.

The hearing comes after White House Press Secretary Jay Carney said Monday that the president’s counsel was told on April 24 about the preliminary findings of an IRS audit that showed tax officials unfairly targeted Tea Party groups applying for tax-exempt status.

Carney had previously said that White House counsel did not have any details about the IRS probe and was given a generic heads up that one was being conducted.

Senior legal counsel Kathryn Ruemmler was told about the audit on April 24, Carney said Monday. She then told Denis McDonough, Obama’s chief of staff and other senior officials about the investigation.

“It was the judgment of counsel this is not a matter she should convey to the president,” Carney said. “Her opinion that this is not the kind of thing that requires notification to the president.”

“No one in this building intervened in an independent investigation or anything that could be seen that way,” he said, adding that the misconduct had stopped in 2012, “almost a year before we knew about it.”

Carney also said while Ruemmler knew the subject of the investigation and potential findings, they were not given a draft of the report and understood details could change.

Ahead of the hearing, the committee Chairman Max Baucus, D-Mont., and Utah Sen. Orrin Hatch, the ranking Republican, sent a letter to the IRS Monday, asking for an explanation. The letter included 41 separate requests for information. They gave the IRS until May 31 to respond.

The two senators said the IRS had not been forthcoming about the issue in the past.

“Targeting applicants for tax-exempt status using political labels threatens to undermine the public’s trust in the IRS,” Baucus and Hatch wrote. “Lack of candor in advising the Senate of this practice is equally troubling.”

For more than a year, from 2011 through the 2012 election, members of Congress repeatedly asked Shulman about complaints from tea party groups that they were being harassed by the IRS.

Shulman’s responses, usually relayed by a deputy, did not acknowledge that agents had ever targeted tea party groups for special scrutiny. At a congressional hearing March 22, 2012, Shulman was adamant in his denials.

“There’s absolutely no targeting. This is the kind of back and forth that happens to people” who apply for tax-exempt status, Shulman said at the House Ways and Means subcommittee hearing.

The IRS has said Shulman did not know about the targeting at the time of the hearing.

The agency’s inspector general says he told Shulman on May 30, 2012, that his office was auditing the way applications for tax-exempt status were being handled, in part because of complaints from conservative groups. However, the inspector general, J. Russell George, said he did not reveal the results of his investigation.

George was also testifying at Tuesday’s hearing. So was Steven Miller, who took over as acting commissioner in November, when Shulman’s term expired. Last week, Obama forced Miller to resign.

George issued a report last week blaming ineffective management for allowing agents to inappropriately target conservative groups for more than 18 months during the 2010 and 2012 elections.

The agents were trying to determine whether the groups were engaged in political activity. Certain tax-exempt groups are allowed to engage in politics, but politics cannot be their primary mission. It is up to the IRS to make the determination, so agents are supposed to look for clues when reviewing applications for tax-exempt status.

In March 2010, agents starting singling out groups with “Tea Party” or “Patriots” on their applications. By August 2010, it was part of the written criteria for identifying groups that required more scrutiny, according to George’s report.

Agents did not flag similar progressive or liberal labels, though some liberal groups received additional scrutiny because their applications were singled out for other reasons, the report said.