Now, a moving exhibition of blown-up, century-old photographs has been put together by French artist JR, allowing new generations to explore the ghosts of the past for the first time.
For his "Unframed, Ellis Island" project, JR visited the island's archives, selected black-and-white photographs and mounted them on the hospital's dilapidated walls and broken windows.
There are girls with a scalp disease, their heads wrapped in white cloth, nurses in uniform, surgeons, a newly arrived family, German physicians visiting in 1912 and photographs of the mentally ill.
JR has pasted the cut-out, life-size photographs of a couple and their child onto a window looking across at the Statue of Liberty -- the iconic beacon of America -- so close and yet so far.
The hospital, which opened in 1902 and closed in 1954, treated 1.2 million sick immigrants, around 10 percent of those processed through Ellis Island after crossing the Atlantic.
Around 350 babies were born. Though most patients eventually rejoined their families, around two percent were deported and 3,500 died without realizing their dream of a new life in America.
Walking through the abandoned hospital's corridors, JR told AFP that he carefully selected where to position the 30 photographs, saying he was guided by the energy of the place.
"I didn't want to paste all over," said the 31-year-old artist.
"I came many, many times, and tried many things," but it didn't always work. "You can feel the intensity in certain rooms, it is charged (with) history."
In a former bedroom with a cracked chest of drawers, a photograph of three sleeping people has been pasted on the decrepit wall.
The little girls with white bandages are stuck on a broken window, surrounded by overhanging foliage with the outside light making their faces look translucent.
A group of mentally ill patients are the only photographs mounted outside, stuck on the porch of the psychiatric ward -- the same place where the photograph was originally taken.
"For the autopsy room, I tried hundreds of things, but there was no image that was connecting," JR said.
He says he doesn't believe in ghosts, but admits "there are some rooms, you don't want to walk alone," in the run-down buildings of interminable, dark corridors.
In the former washing room, enormous machines sit rusting. There is a morgue, a succession of empty bedrooms with old sinks and the room used to sterilize mattresses.
The hospital is so dilapidated that visitors are given hard hats to wear. Groups of a maximum of 10 are allowed to visit for 90 minutes four times a day, four days a week.
JR's work has appeared on streets all over the world, and he is motivated by showing to people who would not ordinarily visit museums. He says he has never exhibited anywhere like the hospital before.
"It's as if there are still souls who are stuck... I still feel their energy," he said.
JR became fascinated with Ellis Island after discovering the hospital three years ago in a book by photographer Stephen Wilkes.
The project is sponsored by the Save Ellis Island association and it took him two months to put the project together this summer before it opened to the public in October.
He has no plans to dismantle the exhibition.
"It's a last homage to this place because they might disappear forever. You don't know how long (the buildings) are going to stand like this," he said.
The exhibition has proven popular and JR is working on a book about the project. He's also planning a short film about Ellis Island in collaboration with Robert de Niro.
"I expected an interest but not like that, this is totally overwhelming," said Antoni Mrozinski, a volunteer guide.
"It adds to the whole story of Ellis Island," he said.
"That part of the story could be told, but couldn't be felt. And opening this for people, you can feel the rest of the story."