He was the second son of James Maccubbin Carroll and Sophia Gough. He married his cousin Eliza Ridgley in 1811 and had five children, only two of whom survived into adulthood. He inherited Perry Hall, which suffered a devastating fire in 1839. He managed family businesses including rental properties and a grist mill. He was a slaveholder. In 1855, he sold Perry Hall outside of the Carroll family and spent his final years in Baltimore.