Ho was a singer of Native Hawaiian, Chinese, Portuguese, Dutch, and German descent. He was born in the small Honolulu neighborhood of Kakaʻako to Emily (Honey) Leimaile Silva and James Ah You Puao Ho, but he grew up in Kāneʻohe on the windward side of the island of Oʻahu. He was a graduate of the Kamehameha Schools in 1949 and he attended Springfield College on a football scholarship in 1950, but returned home to earn a Bachelor's degree in sociology at University of Hawai'i in 1953. In 1954, Ho entered the United States Air Force doing his primary training at Columbus AFB, Mississippi and spent time flying C-97s with the Military Air Transport Service. Transferred to Travis AFB, California, he went to the local city of Concord and bought an electronic keyboard from a music store, and recalls, "That's when it all started."