Rathnayake was born on 18 February 1942 in a small village near Kadugannawa as the third of the family with 10 siblings. His father was Rathnayake Arachchilage Don James. He was popularly known as Rathnayake Veda Mahaththaya, was an apothecary and sang "noorthi gee," a type of Sinhala folk songs. His mother was Abeykoon Mayadunnelage Sumanawathi. Victor was educated at Kadugannawa Junior School and then attended to Kadugannawa Maha Vidyalaya for secondary education.

 

One of his brothers, Somapala Rathnayake was a renowned musician in Sri Lanka, who worked mostly in background music for many Sinhala films and television serials. He died in July 2016 at the age of 70.

 

Rathnayake has one daughter and two sons from first marriage with Chithra Rathnayake. His daughter Chandani is a dancer and a vocalist. His elder son Jayantha Rathnayake is a renowned musician and composer. Jayantha started his music career as the keyboard player in Sihashakthi musical band. Jayantha is married as has one daughter and one son. His younger son Lelum is also a musician and singer who worked in a musical band.[8] On 6 April 2020, his elder son Jayantha Rathnayake died at the age of 52 while receiving treatments at Apeksha Hospital, Maharagama.

 

Rathnayake is currently married to Hashini Amendra, 40 years younger than him. Rathnayake was 75 years old at the marriage. Hashini is a bank employee. In January 2018, Hashini was taken into custody over a jewelry theft. The incident took place while she was employed at a state bank. She was produced in Court and is detained in remand custody.