The home was built in 1902 for Frank and Eleanor Atherton.
According to a surviving grandson who spoke from his Big Island home, his family donated it to the University of Hawaii 50 years ago, to be used as the president's residence, although the deed doesn't specifically say so.
"It was specifically given to the UH with the hope, or the desire that it be used as the president's house," said Frank Atherton II.
Atherton said UH unsuccessfully sought his blessing along with cousin Joan Rohlfing for a plan to use the 3-acre parcel as a meeting venue after UH President MRC Greenwood opted to live elsewhere.
"We said we would much prefer it to be a residence for the president of UH and because of what you are currently doing it is creating a lot of noise in the neighborhood that the neighbors don’t like," Atherton said.
Neighbors like Jeremy Lam, who lives next door, complained about the noise and crowds.
"The traffic just got crazy with the valet parking and police had to come and then they backed off on the number of events," Lam said.
Manoa Chancellor Tom Apple said earlier this month the options for College Hill, include turning it into an alumni house, faculty club or combined faculty and staff club.
"It's very distressing. We would like to have the community to have some input. we have a never been invited to any meeting to have some input," Lam said.
A co-founder of the Historic Hawaii Foundation is calling for the home to be returned as a residence for the president.
The simple grandeur of the home, Bob Fox said, is symbolic.
"It gives it a sense of permanence, the sense of quality and sense of history. I think that's very important," said Fox.
Preservationist David Cheever hopes now that the university is looking for a new president, UH regents will rethink College Hill's future.
"The way that could work, is obviously when the contract is written for the new president, that this is offered as part of the perks, because this is a pretty decent perk." said Cheever.
Recent renovations to the grand old home which KITV first reported have exceeded the million dollar mark.
Work was to have been completed last month, but now won’t be finished until the end of the year.