Brown was born on May 3, 1933, in Barnwell, South Carolina, to 16-year-old Susie (née Behling; 1916–2004) and 21-year-old Joseph Gardner Brown (1912–1993) in a small wooden shack.[15] Brown’s name was supposed to have been Joseph James Brown Jr., but his first and middle names were mistakenly reversed on his birth certificate.[16] He later legally changed his name to remove “Jr.” In his autobiography, Brown stated that he had Chinese and Native American ancestry and that his father was of mixed African-American and Native American descent, while his mother was of mixed African-American and Asian descent.[17][18][19] The Brown family lived in extreme poverty in Elko, South Carolina, which was an impoverished town at the time.[9] They later moved to Augusta, Georgia, when James was four or five.[20] His family first settled at one of his aunts’ brothels. They later moved into a house shared with another aunt.[20] Brown’s mother eventually left the family after a contentious and abusive marriage and moved to New York.[21] Brown spent long stretches of time on his own, hanging out in the streets and hustling to get by. He managed to stay in school until the sixth grade.
He began singing in talent shows as a young child, first appearing at Augusta’s Lenox Theater in 1944, winning the show after singing the ballad “So Long”.[22] While in Augusta, Brown performed buck dances for change to entertain troops from Camp Gordon at the start of World War II as their convoys traveled over a canal bridge near his aunt’s home.[22] He learned to play the piano, guitar, and harmonica during this period. He became inspired to become an entertainer after hearing “Caldonia” by Louis Jordan and his Tympany Five.[23] In his teen years, Brown briefly had a career as a boxer.[24] At the age of 16, he was convicted of robbery and sent to a juvenile detention center in Toccoa.[25] There, he formed a gospel quartet with four fellow cellmates, including Johnny Terry. Brown met singer Bobby Byrd when the two played against each other in a baseball game outside the detention center. Byrd also discovered that Brown could sing after hearing of “a guy called Music Box”, which was Brown’s musical nickname at the prison. Byrd has since claimed he and his family helped to secure an early release, which led to Brown promising the court he would “sing for the Lord”. Brown was released on a work sponsorship with Toccoa business owner S.C. Lawson. Lawson was impressed with Brown’s work ethic and secured his release with a promise to keep him employed for two years. Brown was paroled on June 14, 1952.[26] He would go on to work with both of Lawson’s sons and would come back to visit the family from time to time throughout his career. Shortly after being paroled, he joined the gospel group the Ever-Ready Gospel Singers, featuring Byrd’s sister Sarah.[2
