He was the youngest son of Nicholas Maccubbin, a merchant and High Sheriff of Annapolis, and Mary Clare Carroll. Like his oldest brother, he was chosen as an heir to his uncle Charles Carroll, Barrister’s estate, provided he changed his surname to Carroll.

He married Sophia Gough in 1787. They had seven children, five of whom survived. They raised their children at Perry Hall, the estate of his in-laws Harry Dorsey Gough and his wife, Prudence Carnan Gough. James studied law and served in the lower House and Senate of the Maryland State Legislature from 1787 - 1799.

Although James received Mount Clare after his uncle died in 1786, he did not move in until his aunt passed away in 1817. By then, he was a widower with mostly grown children. The enslaved who had cared for Mount Clare under his aunt had been manumitted or sold, so James brought a new community of enslaved people with him. By 1820, 28 enslaved and nine free blacks lived at Mount Clare. James embraced the industries that were encroaching upon the neighborhood. In 1827, he started leasing Mount Clare’s fields to brickmakers to mine clay. The following year, he donated land and the right-of-way for the country’s first railroad, the B&O, to construct their first commercial rail line. He also derived income from leasing the Mount Clare mills and two additional agricultural plantations, as well as renting out various properties.