No single event finished off whaling, of course. It was doomed from the moment in 1859 when geologists discovered oil in the crust of Pennsylvania. Then came the piracy and scuttling of whaling ships during the Civil War (including the Edgartown whaler Ocmulgee, sunk by the Confederate raider Alabama), the loss of most of the New England fleet to Arctic ice in 1871, and the transfer of investment by the richest Vineyarders from whaling to the resorts at Oak Bluffs and Katama during the post-war building boom.
But the death of whaling brought to an end the last all-Island industry that created wealth by refining and exporting a product to the mainland. From this point forward, the Vineyard would depend largely on the kindness of strangers.

From the Vineyard Gazette edition of November 10, 1871:

San Francisco, Nov. 5.
Steamship Moses Taylor, arrived from Honolulu and Australia, reports a terrible disaster of the Arctic whaling fleet, caught in the ice. Thirty-three vessels were crushed or abandoned. The captains of whalers Arctic, Gay Head, Minerva, Elizabeth Swift, Reindeer, Eugenia, and Portia, arrived at Honolulu, furnished the following details:
The vessels commenced arriving at Cape Thaddeus on the first of May. The first of June the ice opened and let the fleet up within sight of Cape Navarino. The fleet, working northward, found some whales crossing the sea of Anadyr, and in Behring Sea more and plenty, but experienced much trouble from ice; and when the fleet arrived at Cape Behring and Plover Bay, the whales had passed into the Arctic ocean, whither the fleet followed, meeting fair success until about the first of September, when ice floes and bergs, to a great extent, commenced drifting down; and by the tenth a number of vessels had been sunk and the bulk of the remainder ashore. 
On Sept. 2 brig Comet sunk; 7th, the Roman was drifted bodily out to sea by two floes, and crushed like an eggshell. The crew narrowly escaped. The Florida and Victoria, of San Francisco, were also crushed. Sept. 13, the captains of the fleet hemmed in between Point Barrow and Wainwright Inlet, held a meeting and resolved to abandon the vessels, in order to save the lives of the crews, which was done, and 1,200 sailors took refuge on board the remainder of the fleet, which had been fortunate enough to escape outside before the ice closed in the vessels.

November 10, 1871:

The news of the terrible disaster to the Arctic whaling fleet was received here by mail, Monday evening, and at once became the all-absorbing topic of conversation. For a time it seemed almost impossible to realize the magnitude of the catastrophe which has destroyed so much property, and deprived of business 1,200 men, including many of our most enterprising Vineyard whalemen. The one great cause for congratulation is that no lives were lost.
Edgartown will feel the loss greatly. It is a severe blow to her already diminished whaling fleet. 
The Mary and Champion are both reported lost, and our fleet is now reduced to four vessels. The Champion, Crafton N. Collins, agent, was valued at about $40,000, and had on board 300 barrels of sperm oil taken last winter, worth about $12,000, on which there was no insurance. The vessel and outfits were quite fully insured.
The Mary, William H. Munroe, agent, who owned half of her, was valued with her catchings, at $25,000. She was but partially insured. The Europa escaped and had taken about 300 barrels of oil this season. Among the number of men in the lost ships we recall the names of the following Vineyarders:
Capt. Jared Jernegan, of the Roman; Ariel Norton, of the Awashonks; Henry Pease Jr., of the Champion; Abraham Osborn, of the George; Messrs. Frank B. Osborn, Samuel S. Ripley, Ambrose Vincent and Henry M. King, of the George; John N. Norton, mate of the Fanny; Henry F. Worth, of the Gay Head; Benjamin B. Worth, of the John Wells; Henry G. Dexter, of the Reindeer, all of Edgartown.
Capt. Leander Owen, of the Contest; West Mitchell, of the Massachusetts; Valentine Lewis, of the Thomas Dickason; B. Dexter, of the Emily Morgan; Mr. Lot Luce Jr., and several others, whose names we cannot recall, of Tisbury. . . . 
No event has before occurred in the history of the business more calculated to affect the oil markets, and in Boston Monday the price of whale oil jumped from 60 cents to one dollar per gallon, a rise of 40 cents over the last sales a few days ago.
The Boston Journal says holders of the article are not in a hurry to sell even at this large advance, and no transactions are reported, both buyers and sellers not having as yet sufficiently digested the news to be able to enter upon negotiations.
The New Bedford merchants take a hopeful view of the case and already two ships, the James Allen and Aurora, intended to lay over, are being fitted for the North.
Samuel Osborn Jr., of this town, who owned quite largely in both of the lost ships from this port, we hear is fully insured.
It is stated that the disaster to the Arctic fleet was predicted by the Esquimaux, but no attention was paid to their warnings. A dispatch says the wrecked officers started from San Francisco for home last Tuesday. They will arrive early next week.

December 11, 1874:

It is worth another ride over the 18 miles of road along the south shore to see the pretty village of Edgartown, at the easternmost end of the Island. In a commercial sense it is a place far advanced in decay: of all its whale-ships, which got from the sea the hard-earned fortunes of its people, there is but one left. This lies upon the ways, stripped of its rigging, looking like a mere effigy of a living craft. 
But the thrift and cleanliness of the sailor is marked in every paving-stone and shingle of the village. As soon as a mariner comes to fortune his first effort is to get a comfortable home, a big, square, roomy house, which shall always be ship-shape and well painted. 
I never thought so well of white paint before I saw these handsome houses, actually resplendent with a hue which is so often merely garish in such uses. If there be a trace of an instinct of cleanliness, white paint is an excellent stimulant to its activity, for it makes dirt hateful by making it apparent. 
These comfortable homes, like those of New Bedford, mark a period of prosperity which has passed never to return. Little by little the population is drifting away; some houses stand empty, and the quick agent of decay which makes havoc with our frail New England houses will soon be at work at them, and even the Yankee thrift cannot keep it away.
- Professor N.S. Shaler