The mansion was built as the home of Dr. James Harvey Baker, a prominent physician. Some accounts of Dr. Baker's activities during the Civil War report him as being aligned with the Confederacy, while others claim his neutrality.
Whatever his allegiance, his son, Abner, fought for the Confederacy. In 1864, Dr. Baker was at home treating wounded Confederate soldiers when Union soldiers gained entry into the home, shot and killed the physician.
At age 22, Abner returned home from the war and sought to avenge his father's death. He shot and killed the alleged informer, Knoxville Postmaster William Hall. Afterwards, several of the postmaster's accomplices ambushed and killed Abner. Today, a marker on the front lawn memorializes Abner for his courage and loyalty to his family.
Dr. Baker's house was later sold in the late 1800's to the Peters family.