The town council said that after seeking legal advice it would not return the 1820 painting, Dedham from Langham, by the British artist John Constable (1776-1837).
The picture had been part of the collection belonging to Anna Jaffé, who died in Paris in 1942. Her entire property was seized by the Nazi authorities and sold at auction in Nice in 1943.
It was resold several times, and purchased by the Swiss Junod family in 1946. It came into the museum's possession, along with a number of other pictures, in 1986 as a bequest from Mrs Junod.
Under her will, the collection was to be housed in a special room and not to be split up.
The council says there is no doubt that the Junods bought the picture in good faith.
In 2006 a representative of the Jaffé heirs asked the museum to give it back to the family.
The town council consulted two legal experts and various relevant bodies and was told that under Swiss law it had no obligation to return the picture, or to pay compensation. The refusal is also in line with the principles of the Washington conference on Nazi-confiscated art.