The Smith-McDowell House is one of the finest ante-bellum constructions in western North Carolina and Asheville's oldest brick residence. Situated on a portion of a land grant issued to Colonel Daniel Smith after the Revolutionary War, the c. 1840 Smith-McDowell House was built by Smith's son, James McConnell Smith. James Smith, born June 14, 1787, was the first child of European parents born west of the Blue Ridge in North Carolina. He was also one of the wealthiest landowners in the county during the early part of the 19th century and one of the leading businessmen of pre-Civil War Asheville and Buncombe County. In addition to building and operating a toll bridge over the French Broad River, he owned and operated a tannery, a mercantile business, and the Buck Hotel in downtown Asheville.

In 1849, Smith served as the second mayor of the City of Asheville. After his death, Smith's daughter, Sarah Lucinda, and her husband, William Wallace McDowell, lived in the house until 1880. McDowell was a banker and planter who served as a major during the Civil War, and the couple raised nine children while living here.

The three-story brick house was designed and built as a five-bay mansion in Adamesque and Federal styles. The double-tiered front porch provides a vista of the mountain ranges to the southeast. The interior of the house contains much of the original Greek Revival woodwork. By 1910, the house had been sold, and the new owners secured the architectural services of Smith & Carrier Architects to design additions and modifications to the house and hired the sons of Frederick Law Olmstead to design a landscape plan for the property.

The house remained a residence until it was converted to a school in 1951. The Western North Carolina Historical Association leased the house in 1974, and it was restored over a six-year period, reopening to the public as a house museum in 1981. Areas of the house have been restored to reflect different periods of the evolution and renovations made to this magnificent building.