From NKAA, Notable Kentucky African Americans Database (PIONEERS & PATRIOTS: Anderson County Military Servicemen, Revolutionary War - WWI)

James Lillard (records also show his last name as Lillan or Lillen) was born a slave in Anderson County about 1841. In 1864, the Federal government initiated a new policy to enroll regiments of colored soldiers for service in the war. As a result, thousands of slaves and former slaves flocked to recruiting stations to volunteer. Among these was James Lillard, who made his way from Lawrenceburg to Camp Nelson, where on September 9, 1864 he was mustered into Federal service as a private in Company E of the 5th U.S. Colored Cavalry Regiment. His enlistment record describes him as a 19-year old farmer living in Anderson County.

Although the 5th Colored Cavalry had not reached its full complement of men, those enrolled by the end of September were assigned to take part in Burbridge's Campaign into southwest Virginia where it was active at the First Battle of Saltville on October 2, 1864. In December, another attempt was made to destroy the salt works, and the 5th was part of Stoneman's Raid. On December 17-18, 1864, the Battle of Marion was fought, and the Confederate lead mines were captured. On December 20-21, the Federals captured and destroyed the salt works after the 2nd Battle of Saltville. 

In January 1865, Company E of the 5th Regiment was detailed to escort a herd of cattle from Camp Nelson to Louisville. As this column passed through Mercer and Anderson Counties, its progress was closely monitored by Confederate partisans operating in the area. On January 25, 1865, just outside Simpsonville in Shelby County, a band of Confederate guerrillas under Captain Isaiah "Zay" Coalter caught the Union soldiers completely off guard, and after a fifteen minute action all 35 men of the rear guard were down, either dead or left for dead, resulting in a complete massacre.

Local citizens gathered up the dead troopers and buried them in a mass grave, while the wounded were taken to Louisville. Although James was most certainly part of this detail, he is not listed among the casualties of this small action, so it can therefore be assumed that he was not part of the rear guard and was further ahead in the formation.

In August 1865 the regiment was transferred to duty in the Department of Arkansas, where it was actively engaged against renegade groups until March 1866. It was mustered out of the service at Helena, Arkansas on March 20, 1866.