The Edgartown Library building committee hit yet another bump in the road this week when the town historic district commission said it will not allow the Warren House to be torn down.

The building committee’s latest plan calls for razing the historic colonial-era house and replacing it with a parking lot for the expanded and renovated library at the Carnegie building on North Water street

But after meeting on Tuesday with the historic district commission, that plan, like others before it, now must be scrapped.

The decision by the historic commission, which was informal and did not include a vote, is a reversal from three years ago when the commission had agreed to allow the house to be torn down to make way for a new library addition.

But commission chairman James Cisek told the building committee this week that the library renovation plans have changed dramatically since then and the commission must now start from scratch in considering possible demolition of the Warren House.

“The original [library plan] was designed with the intent to somehow honor the Warren House,” Mr. Cisek said. “I think we would start out as a new submittal, and the Warren House is not slated for demolition unless we felt that it needed to be demolished,” he added. Commission member Mary Sullivan agreed. “We thought it [the library plan] was going to memorialize the Warren House; this is our first glimpse of this,” she said. “If this one is your preferred, parking lot in front, it doesn’t resonate with the original plans.”

Purchased by the town five years ago for $3.5 million, the Warren House was intended to be renovated into an annex to the Carnegie building. The house dates to the 1780s with Victorian-era additions; its most recent use was as an annex to the Daggett House, an inn that was converted to a private residence several years ago. Today the Warren House sits vacant and crumbling while the town struggles to come up with a plan for renovating its library.

Adding to the air of discontent, on Tuesday North Water street neighbors turned out to protest the demolition of the house.

“The row of houses, it’s a part of one of the finest rows of colonial houses in America, and to take one of them away . . . is a real problem,” said North Water street resident and architect Dudley Canada. “I think that it endangers all the houses in Edgartown if we take this house down. I think our street is one of the most important historic streets in America.”

And the historic district commission also firmly staked out its position. “As a historic district we need to focus on the history of the town and buildings and to demolish a historic part of a building is to demolish our history and our past,” Mr. Cisek said.

Building committee chairman Chris Scott expressed his frustration and appealed for town boards to find some consensus for the sake of improvements to the town library.

“Something has to give . . . the town boards need to recognize that,” Mr. Scott said. “We’re looking at a site full of restraints and we’re doing the best job we can.

“What would be great is if we could all get behind something. I don’t see this as compartmentalized. It’s the town’s library, let’s look at it together and come up with the best idea and get behind it and try and sell it to the state.”

Mr. Scott said later that architects Celia Imrey and Jeffrey Hoover now have four constraints to work into their design: the original historic Carnegie building, the front lawn on North Water street, parking required by the Massachusetts Board of Library Commissioners and the Warren House.

In addition, the design must meet a strict program size requirement by the state library board in order to receive a matching grant to fund the project. The program size is based on a population formula, and requires a certain number of parking spots. West Tisbury, which is also embarking on a library expansion project, faces a similar problem with state requirements for parking.

The building committee later Tuesday night met with the planning board, whose members echoed the historic commission’s sentiments but were reportedly open to some kind of minimum parking plan.

As a result of the latest setback, Mr. Scott said the committee may consider revisiting the old Edgartown School as a site option. New building codes prevent the possibility of keeping the structure intact, so the building would need to be torn down and replaced with new construction.

Architects also will now take another look at the site plan to include the Warren House, with possible parking between the house and the proposed addition.

And despite all the problems and the fact that a Jan. 27 due date looms for the town to submit a completed state grant application, Mr. Scott remained stubbornly upbeat.

“There’s no bottom line yet,” he said. “The process continues.”