(CNN) -- The bipartisan group in the House working on immigration reform suffered a blow late Wednesday after Rep. Raul Labrador, R-Idaho, left the team over policy disagreements.

The group was close to finalizing its legislation but Labrador said the House "Gang of Eight" could not agree on a provision involving health care for undocumented workers, according to a statement his office issued Wednesday night.

"I have tremendous respect for the members of the bipartisan group who have been working with me to fix our broken immigration system," Labrador said in a statement. "But after today's meeting, the framework of the bill has changed in a way that I can no longer support."

Many immigration reform advocates were banking on Labrador as a key player to bring on Republican votes. An outspoken conservative, Labrador repeatedly told reporters he wants Republicans to "find a way to get to yes" on major reform.

Labrador first threatened to leave the group two weeks ago when it faced gridlock over how to cover health care costs. GOP members worried state or local governments would be stuck with the costs and Democrats were concerned an undocumented worker who was seriously injured or diagnosed with cancer and couldn't pay for insurance would be deported.

The group ultimately reached an agreement two weeks ago and Labrador stayed on, though they didn't provide details of the compromise.

But it apparently fell apart Wednesday, as Labrador's office said the group "couldn't satisfactorily resolve the issue" and the congressman would not sign onto the legislation.

"I believe that health care is first and foremost a personal responsibility," he said in the statement.

He vowed to keep fighting for a bill that he believes can pass in the House, and two sources told CNN that Labrador informed colleagues he plans to introduce legislation on his own.

Another member of the House "Gang of Eight" insisted the group would move on without Labrador. Republican Rep. Luis Gutierrez said the remaining seven members are committed to crafting a bill that could win support on both sides of the aisle.

"I am sorry to see him pull out, but I think he will be tremendously helpful in getting immigration reform passed by the House and signed by the President this year when all is said and done," Gutierrez said in a statement.

"I guess they will stop calling us the gang of eight, now," he added. "I am hoping they start calling us the Magnificent Seven."

Nonetheless, the congressman's departure represents a new roadblock for those seeking immigration reform in the GOP-controlled House, as the House Judiciary Committee--which marks up immigration legislation--is already stacked with many conservatives skeptical of a deal with Democrats.

Some members in the Senate's "Gang of Eight" met with Labrador and other House members Wednesday as the upper chamber attempts to pass a bill that could have a future in the House.

Many House conservatives, however, made it clear the Senate bill, which includes a pathway to citizenship, is a total nonstarter with them, and cited the House Speaker John Boehner's position that he doesn't plan to bring it up for a vote.

Many, including House Judiciary Committee Chairman Bob Goodlatte, pushed for the House to proceed with a series of immigration bills that focused on border security, and criticized the Senate version for not being tough enough on the subject.

Labrador told reporters after the meeting that overall, House conservatives don't like the Senate proposal, but are in favor of parts of it, though he stressed "I don't think you get anything out of the Senate without strong border security and we definitely don't get anything out of the House without strong border security."

Speaking after the bicameral meeting Wednesday, Sen. Marco Rubio of Florida continued to voice support for immigration reform legislation but distanced himself from the Senate proposal he helped craft.

Rubio wants to add changes that bolster security provisions and while he didn't tell reporters on Capitol Hill that he would vote against it, he did say, "If the changes don't happen, the bill can't pass."

"We'll keep working," he added. "We won't abandon the effort, we'll keep working to ensure that we have a bill that can pass."

Noting that he was asked to join the immigration effort in part to help bring Republicans on board, Rubio declared, "I can tell you that the bill as structured isn't going to pass the House, and it's going to struggle to pass in the Senate."

CNN's Rachel Streitfeld and Ashley Killough contributed to this report.