The Paul Jennings Memoir in Context


Paul Jennings, an enslaved African American who served the Madison family both at Montpelier and in Washington, D.C., made the incredible journey from enslaved to free man to memoirist. His brief volume, entitled A Colored Man’s Reminiscences of James Madison, is considered the first memoir about life at the White House. It’s also a rich firsthand account of the relationship between enslaved and enslaver—even more valuable for its insight into a system that was at odds with the values its perpetrators professed. As you click on each footnote in this annotated version of the memoir, you’ll find brief essays and explanations to help you delve more deeply into the world Jennings described.


In the preface to A Colored Man’s Reminiscences of James Madison (1863), John Brooks Russell explained the project’s genesis: “as [Paul Jennings] was a daily witness to interesting events, I have thought some of his recollections were worth writing down in almost his own language.”

Paul Jennings could certainly recollect an eventful life. Born into slavery at Montpelier, he was a dining room servant at the White House and a body servant to Madison during the president’s retirement. Eventually, Jennings purchased his freedom and took a job in the Pension Office in Washington, where his anecdotes about the Madisons caught the attention of Russell, a colleague.

Jennings was literate. He could have written down his recollections himself, had he wished. Russell seems to have provided the catalyst. A published author of historical articles, Russell saw value in Jennings’s stories of the Madisons, and knew a forum where they could be presented to the reading public. The memories contained in his Reminiscences belonged to Jennings; the language was “almost his own.” Russell’s statement that “some of his recollections were worth writing down” implies that Jennings shared other anecdotes that Russell did not deem worthy of preservation. Jennings, too, silently played an editorial role, having already decided which of his memories to share with his white coworker.

Madisonian Memoir, Slave Narrative, or Oral History?

As the title makes clear, Russell’s focus was on Madison. Each anecdote gave an insider’s view of the president and his world. By specifying that these were the memories of a “colored man,” Russell indicated that they were not the memories of Madison’s colleague or equal. Jennings’s importance came from the insight he provided into the world of the great man; any glimpses into Jennings’s world were secondary. In the context of slave narratives, A Colored Man’s Reminiscences has less in common with works written to expose the terrible realities of slavery (such as Harriet Jacobs’s 1861 Incidents in the Life of a Slave Girl), and perhaps more in common with slave narratives produced by the Federal Writers Project of the Works Progress Administration in the 1930s. The majority of those narratives were recorded by white interviewers, a fact which may have subtly altered the answers given by the elderly, formerly enslaved people being interviewed.

The structure of the Reminiscences, in fact, resembles an oral history with its digressions, interruptions, and chatty narrative, rather than a carefully composed essay. During the discussion of the burning of the White House during the War of 1812, Jennings makes an allusion to the portrait of Washington “of which I will tell you by-and-by.” The narration covers events of the next several months before returning to the rescue of the portrait, repeating details already given. News of the peace treaty is interrupted by a three-paragraph digression on the Madisons through 1832. The narrative then resumes at the peace of 1815. Following an affecting description of Madison’s death and funeral, the Reminiscences concludes with an anti-climatic identification of pallbearers. This meandering structure suggests that Russell may indeed have written down Jennings’s anecdotes in the order that he told them, “in almost his own language.”

Insider Views of the White House

A Colored Man’s Reminiscences of James Madison was the first book published in the genre of White House memoir. While written from the perspective of an insider, Reminiscences presents the Madisons in a flattering light and reveals very little of Jennings’s personal life or the realities of enslavement in the early nineteenth century. Elizabeth Keckley, dressmaker to Mary Todd Lincoln, took a different approach when she published Behind the Scenes: Or, Thirty Years a Slave and Four Years in the White House (1868). Keckley devoted three chapters to her own experiences in slavery, describing beatings and sexual assault. Keckley intended to present the often-criticized First Lady in a positive light. Mary Lincoln, however, was offended by the revelation of personal information that she found embarrassing. Many readers at the time also took offense at the apparent breach of the First Lady’s confidence by an African American woman in a subservient role.

A number of White House staff members wrote memoirs in the twentieth century, including Thomas F. Pendel, who began his career as a doorman in the Lincoln administration (Thirty-Six Years in the White House, 1902); Elizabeth Jaffray, the first woman to head the household staff during the Taft administration (Secrets of the White House, 1927); and Lillian Rogers Parks, seamstress from the Hoover through Eisenhower administrations (My Thirty Years Backstairs at the White House, 1961). Like Jennings, these longtime staff members took pride in their association with presidents and in their status as insiders. As expectations of privacy for public figures have changed over time, memoirists have shared more personal details than Jennings or Russell probably ever contemplated revealing. 1

The Annotated Edition

The annotations for this edition of A Colored Man’s Reminiscences of James Madison provide context for Jennings’s narrative, using other contemporary accounts to corroborate or clarify his observations. These comparisons show Jennings’s memory to be consistently reliable, with the occasional disparities over dates and minor details that occur when any two people try to recall the same event. Jennings clearly paid attention to the conversations that took place while he served meals and attended to Madison and his guests. Jennings’s ability to give thumbnail descriptions of the members of Madison’s cabinet, for example, shows a keen insight into their personalities that goes beyond the mere retelling of Madison’s favorite anecdotes.

While details of Jennings’s life appear only on the periphery of his Reminiscences, the annotations insert elements of Jennings’s biography into the narrative. Jennings was more than the enslaved household servant seen by the Madisons and their contemporaries. He was a husband and father living apart from his family, separated sometimes by a few miles, sometimes by a few hundred miles. As a free man, he was an active member of the African American community of Washington, D.C. He supported the efforts of others trying to achieve freedom. By the end of his life, he was a homeowner in a Washington neighborhood that included others formerly enslaved at Montpelier.

Did Jennings Tell All?

Jennings’s few comments about slavery at Montpelier are surprisingly benign. Rather than give insight into the harsh realities of slavery, Jennings seems at times to feed into the myth of happy slaves contentedly serving a kindly master: “Mr. Madison, I think, was one of the best men that ever lived. … [The slaves] generally served him very faithfully.” In the text, Jennings refrains from mentioning attempted escapes, backbreaking field labor, or monotonous rations issued to the enslaved, particularly as they compared to the bountiful table set for the Madisons. There is no mention of the irony or the injustice of the Father of the Constitution’s denial of the most basic form of liberty to the more than one hundred African American people he owned as property. While Paul Jennings chose not to level such charges against James Madison, the reader should not take Jennings’s testimony as a reason to believe that Madison had somehow managed to make slavery a benevolent institution. The reader may instead probe beyond the words of the narrative to consider the choices Jennings made.

Despite the best efforts of historians, Jennings’s story remains incomplete. As a free person, a literate man, a government employee, and a homeowner, Jennings appears more frequently in documentary records than most of his African American contemporaries, particularly other members of the enslaved Montpelier community. His Reminiscences offer a provocative glimpse into his world, yet leave the reader with the feeling that Paul Jennings knew and experienced much more than he chose to tell.

Generous support for this research was made possible by the Richard S. Reynolds Foundation. For more on Paul Jennings, visit The Life of Paul Jennings




Among the laborers at the Department of the Interior2is an intelligent colored man, Paul Jennings, who was born a slave on President Madison’s estate, in Montpelier, Va., in 1799. His reputed father was Benj. Jennings, an English trader there;3 his mother, a slave of Mr. Madison, and the grand-daughter of an Indian.4 Paul was a “body servant” of Mr. Madison,5 till his death, and afterwards of Daniel Webster,6 having purchased his freedom of Mrs. Madison. His character for sobriety, truth, and fidelity, is unquestioned; and as he was a daily witness of interesting events, I have thought some of his recollections were worth writing down7in almost his own language.

On the 10th of January, 1865, at a curious sale of books, coins and autographs belonging to Edward M. Thomas, a colored man, for many years Messenger to the House of Representatives,8 was sold, among other curious lots, an autograph of Daniel Webster, containing these words:9 “I have paid $120 for the freedom of Paul Jennings; he agrees to work out the same at $8 per month, to be furnished with board, clothes, washing,” &c.

J. B. R.10


About ten years before Mr. Madison was President, he and Colonel Monroe11 were rival candidates for the Legislature. Mr. Madison was anxious to be elected, and sent his chariot to bring up a Scotchman to the polls, who lived in the neighborhood. But when brought up, he cried out: “Put me down for Colonel Monroe, for he was the first man that took me by the hand in this country.” Colonel Monroe was elected, and his friends joked Mr. Madison pretty hard about his Scotch friend, and I have heard Mr. Madison and Colonel Monroe have many a hearty laugh over the subject, for years after.12

When Mr. Madison was chosen President,13we came on and moved into the White House; the east room was not finished, and Pennsylvania Avenue was not paved, but was always in an awful condition from either mud or dust. The city was a dreary place.14

Mr. Robert Smith was then Secretary of State, but as he and Mr. Madison could not agree, he was removed, and Colonel Monroe appointed to his place. Dr. Eustis was Secretary of War–rather a rough, blustering man; Mr. Gallatin, a tip-top man, was Secretary of the Treasury; and Mr. Hamilton, of South Carolina, a pleasant gentleman, who thought Mr. Madison could do nothing wrong, and who always concurred in every thing he said, was Secretary of the Navy.15

Before the war of 181216 was declared, there were frequent consultations at the White House as to the expediency of doing it. Colonel Monroe was always fierce for it, so were Messrs. Lowndes, Giles, Poydrass, and Pope–all Southerners; all his Secretaries were likewise in favor of it.17

Soon after war was declared, Mr. Madison made his regular summer visit18 to his farm in Virginia. We had not been there long before an express reached us one evening, informing Mr. M. of Gen. Hull’s surrender. He was astounded at the news, and started back to Washington the next morning. 19

After the war had been going on for a couple of years, the people of Washington began to be alarmed for the safety of the city, as the British held Chesapeake Bay with a powerful fleet and army. Every thing seemed to be left to General Armstrong,20 then Secretary of war, who ridiculed the idea that there was any danger. But, in August, 1814, the enemy had got so near, there could be no doubt of their intentions. Great alarm existed, and some feeble preparations for defence were made. Com. Barney’s21 flotilla was stripped of men, who were placed in battery, at Bladensburg,22 where they fought splendidly. A large part of his men were tall, strapping negroes, mixed with white sailors and marines.23 Mr. Madison reviewed them just before the fight, and asked Com. Barney if his “negroes would not run on the approach of the British?” “No sir,” said Barney, “they don’t know how to run; they will die by their guns first.” They fought till a large part of them were killed or wounded; and Barney himself wounded and taken prisoner. One or two of these negroes are still living here.24

Well, on the 24th of August, sure enough, the British reached Bladensburg, and the fight began between 11 and 12. Even that very morning General Armstrong assured Mrs. Madison there was no danger. The President, with General Armstrong, General Winder, Colonel Monroe, Richard Rush, Mr. Graham, Tench Ringgold, and Mr. Duvall, rode out on horseback to Bladensburg to see how things looked.25 Mrs. Madison ordered dinner26 to be ready at 3, as usual; I set the table myself, and brought up the ale, cider, and wine, and placed them in the coolers, as all the Cabinet and several military gentlemen and strangers were expected. While waiting, at just about 3, as Sukey, the house-servant,27 was lolling out of a chamber window, James Smith,28 a free colored man who had accompanied Mr. Madison to Bladensburg, galloped up to the house, waving his hat, and cried out, “Clear out, clear out! General Armstrong has ordered a retreat!” All then was confusion. Mrs. Madison ordered her carriage, and passing through the dining-room, caught up what silver she could crowd into her old-fashioned reticule,29 and then jumped into the chariot with her servant girl Sukey,30 and Daniel Carroll,31 who took charge of them; Jo. Bolin drove them over to Georgetown Heights;32 the British were expected in a few minutes. Mr. Cutts,33 her brother-in-law, sent me to a stable on 14th street, for his carriage. People were running in every direction. John Freeman (the colored butler) drove off in the coachee34 with his wife, child, and servant;35 also a feather bed lashed on behind the coachee, which was all the furniture saved,36 except part of the silver and the portrait of Washington (of which I will tell you by-and-by).

I will here mention that although the British were expected every minute, they did not arrive for some hours; in the mean time, a rabble, taking advantage of the confusion, ran all over the White House, and stole lots of silver and whatever they could lay their hands on.37

About sundown I walked over to the Georgetown ferry, and found the President and all hands (the gentlemen named before, who acted as a sort of body-guard for him) waiting for the boat. It soon returned, and we all crossed over, and passed up the road about a mile; they then left us servants to wander about.38 In a short time several wagons from Bladensburg, drawn by Barney’s artillery horses, passed up the road, having crossed the Long Bridge before it was set on fire.39 As we were cutting up some pranks a white wagoner ordered us away, and told his boy Tommy to reach out his gun, and he would shoot us.40 I told him “he had better have used it at Bladensburg.” Just then we came up with Mr. Madison and his friends, who had been wandering about for some hours, consulting what to do. I walked on to a Methodist minister’s, and in the evening, while he was at prayer, I heard a tremendous explosion, and, rushing out, saw that the public buildings, navy yard, ropewalks, &c., were on fire.41

Mrs. Madison slept that night at Mrs. Love’s,42 two or three miles over the river. After leaving that place she called in at a house, and went up stairs. The lady of the house learning who she was, became furious, and went to the stairs and screamed out, “Miss Madison! if that’s you, come down and go out! Your husband has got mine out fighting, and d–you, you shan’t stay in my house; so get out!”43 Mrs. Madison complied, and went to Mrs. Minor’s, a few miles further, where she stayed a day or two,44 and then returned to Washington, where she found Mr. Madison at her brother-in-law’s, Richard Cutts, on F street.45 All the facts about Mrs. M. I learned from her servant Sukey. We moved into the house of Colonel John B. Taylor, corner of 18th street and New York Avenue, where we lived till the news of peace arrived.46

In two or three weeks after we returned, Congress met in extra session, at Blodgett’s old shell of a house on 7th street (where the General Post-office now stands). It was three stories high, and had been used for a theatre, a tavern, an Irish boarding house, &c.; but both Houses of Congress managed to get along in it very well, notwithstanding it had to accommodate the Patent-office, City and General Post-office, committee-rooms, and what was left of the Congressional Library, at the same time. Things are very different now.

The next summer, Mr. John Law, a large property-holder about the Capitol, fearing it would not be rebuilt, got up a subscription and built a large brick building47  (now called the Old Capitol, where the secesh prisoners are confined), and offered it to Congress for their use, till the Capitol could be rebuilt. This coaxed them back, though strong efforts were made to remove the seat of government north; but the southern members kept it here.

It has often been stated in print, that when Mrs. Madison escaped from the White House, she cut out from the frame the large portrait of Washington48 (now in one of the parlors there), and carried it off. This is totally false. She had no time for doing it. It would have required a ladder to get it down. All she carried off was the silver in her reticule, as the British were thought to be but a few squares off, and were expected every moment. John Susé (a Frenchman, then door-keeper, and still living)49 and Magraw,50 the President’s gardener, took it down and sent it off on a wagon, with some large silver urns and such other valuables as could be hastily got hold of. When the British did arrive, they ate up the very dinner, and drank the wines, &c., that I had prepared51 for the President’s party.

When the news of peace arrived, we were crazy with joy. Miss Sally Coles,52 a cousin of Mrs. Madison, and afterwards wife of Andrew Stevenson, since minister to England, came to the head of the stairs, crying out, “Peace! peace!” and told John Freeman53 (the butler) to serve out wine liberally to the servants and others. I played the President’s March on the violin,54 John Susé and some others were drunk for two days, and such another joyful time was never seen in Washington. Mr. Madison and all his Cabinet were as pleased as any, but did not show their joy in this manner.

Mrs. Madison was a remarkably fine woman.55 She was beloved by every body in Washington, white and colored. Whenever soldiers marched by, during the war, she always sent out and invited them in to take wine and refreshments, giving them liberally of the best in the house. Madeira wine56 was better in those days than now, and more freely drank. In the last days of her life, before Congress57 purchased her husband’s papers, she was in a state of absolute poverty,58 and I think sometimes suffered for the necessaries of life. While I was a servant to Mr. Webster, he often sent me to her with a market-basket full of provisions, and told me whenever I saw anything in the house that I thought she was in need of, to take it to her. I often did this, and occasionally gave her small sums from my own pocket, though I had years before bought my freedom of her.59

Mr. Madison, I think, was one of the best men that ever lived. I never saw him in a passion, and never knew him to strike a slave, although he had over one hundred;60 neither would he allow an overseer to do it. Whenever any slaves were reported to him as stealing or “cutting up” badly, he would send for them and admonish them privately, and never mortify them by doing it before others. They generally served him very faithfully. He was temperate in his habits.61 I don’t think he drank a quart of brandy in his whole life. He ate light breakfasts and no suppers, but rather a hearty dinner, with which he took invariably but one glass of wine. When he had hard drinkers at his table, who had put away his choice Madeira pretty freely, in response to their numerous toasts, he would just touch the glass to his lips, or dilute it with water, as they pushed about the decanters. For the last fifteen years of his life he drank no wine at all.

After he retired from the presidency, he amused himself chiefly on his farm.62 At the election for members of the Virginia Legislature, in 1829 or ’30, just after General Jackson’s accession, he voted for James Barbour, who had been a strong Adams man. He also presided, I think, over the Convention63 for amending the Constitution, in 1832.

After the news of peace, and of General Jackson’s victory at New Orleans, which reached here about the same time,64 there were great illuminations.65 We moved into the Seven Buildings,66 corner of 19th-street and Pennsylvania Avenue, and while there, General Jackson came on with his wife, to whom numerous dinner-parties and levees were given. Mr. Madison also held levees67 every Wednesday evening, at which wine, punch, coffee, ice-cream, &c., were liberally served, unlike the present custom.

While Mr. Jefferson was President, he and Mr. Madison (then his Secretary of State) were extremely intimate; in fact, two brothers could not have been more so. Mr. Jefferson always stopped over night at Mr. Madison’s, in going and returning from Washington.68

I have heard Mr. Madison say, that when he went to school, he cut his own wood for exercise. He often did it also when at his farm in Virginia. He was very neat, but never extravagant, in his clothes. He always dressed wholly in black — coat, breeches, and silk stockings, with buckles in his shoes and breeches. He never had but one suit at a time. He had some poor relatives that he had to help, and wished to set them an example of economy in the matter of dress.69 He was very fond of horses, and an excellent judge of them, and no jockey ever cheated him. He never had less than seven horses in his Washington stables while President.

He often told the story, that one day riding home from court with old Tom Barbour (father of Governor Barbour), they met a colored man, who took off his hat. Mr. M. raised his, to the surprise of old Tom; to whom Mr. M. replied, “I never allow a negro to excel me in politeness.” Though a similar story is told of General Washington, I have often heard this, as above, from Mr. Madison’s own lips.70

After Mr. Madison retired from the presidency, in 1817, he invariably made a visit twice a year to Mr. Jefferson–sometimes stopping two or three weeks71–till Mr. Jefferson’s death, in 1826.

I was always with Mr. Madison till he died,72 and shaved him every other day for sixteen years. For six months before his death, he was unable to walk, and spent most of his time reclined on a couch; but his mind was bright, and with his numerous visitors he talked with as much animation and strength of voice as I ever heard him in his best days. I was present when he died. That morning Sukey brought him his breakfast, as usual. He could not swallow. His niece, Mrs. Willis,73 said, “What is the matter, Uncle Jeames?” “Nothing more than a change of mind, my dear.” His head instantly dropped, and he ceased breathing as quietly as the snuff of a candle goes out. He was about eighty-four years old,74 and was followed to the grave75 by an immense procession of white and colored people. The pall-bearers76 were Governor Barbour, Philip P. Barbour, Charles P. Howard, and Reuben Conway; the two last were neighboring farmers.

Afterword to the Annotated Edition: Freedom Delayed, Freedom Denied, Freedom Achieved

Edward Coles, Dolley’s cousin and Madison’s former secretary, was “greatly astonished” to learn that Madison had not freed any enslaved people in his will. (Coles himself, a staunch advocate of emancipation, had freed the people he enslaved, after moving with them to Illinois in 1819.) Coles advised Madison in 1832, “it is due to the finale of your character & career, & to the consummation of your glory, that you should make provision in your Will for the emancipation of your Slaves. … it would be a blot & stigma on your otherwise spotless escutcheon, not to restore to your slaves that liberty & those rights which you have been through life so zealous & able a champion.” Coles believed that, in their conversations, Madison had “disclosed to me not only his wish but his intention to free his slaves in his will.” In 1849, Coles heard second-hand that Madison had told his lawyer that “he found many difficulties in the way of doing it, during his wife’s life, and had finally concluded not to free them in his will – saying … Mrs. Madison knew his wishes and views and would carry them into effect at her death.” In 1855, Coles wrote to Nelly Willis for confirmation from Madison’s “favourite Niece, who resided so near him and had better opportunities of knowing his opinions and intentions than any other of his near relations now living.” Willis’s son replied on her behalf that “There is one fact which induces my mother to think not only that Mr. Madison expected his wife to liberate his slaves but that he left with her written directions upon this subject.” Nelly Willis had sat with Dolley in her chamber during Madison’s burial, and saw that Dolley had taken “two sealed papers” from a drawer. One was Madison’s will, and the other was labeled “To be opened only by my wife should she be living at the time of my death.” According to Willis’s son, “nothing more was ever heard” about this second paper “which was thought to have contained written instructions on the subject of the slaves.” (Interestingly, the codicil to Madison’s will bears the notation “not to be opened till his death & then only by his [wife].” It is possible that the two papers Nelly Willis saw her aunt remove from the drawer were the will and the codicil. The codicil, however, did not request that Dolley should free the enslaved; it simply contained miscellaneous bequests and directives.)

Whether or not Willis had correctly guessed the contents of the second paper, the only legally binding provision Madison made for the enslaved community was in his will: “I give and bequeath my ownership in the negroes and people of colour held by me to my dear Wife, but it is my desire that none of them should be sold without his or her consent or in the case of their misbehaviour; except that infant children may be sold with their Parent who consents for them to be sold with him or her, and who consents to be sold.”

If Madison hoped that the people he had enslaved would eventually be free, those hopes were for the most part unfulfilled. Ben Stewart, formerly enslaved at Montpelier, recalled, “During the days of her poverty [Dolley Madison] sold off her servants one by one, and I remember I was bought by a Georgia man and taken to that state.” Paul Jennings, Sarah Stewart, Ralph and Catherine Taylor, and the Taylor children continued to be enslaved by Dolley after her move to Washington. Other members of the enslaved community were sold to the purchaser of Montpelier and to neighboring landowners, or transferred to John Payne Todd. Todd made the gesture of freeing enslaved people in his will, a provision that was questionable, given Todd’s debts at the time of his death in 1852. The Taylor family successfully sued for their freedom, charging that the administrator of Todd’s estate had unjustly kept them in slavery.

Ultimately, of the approximately one hundred people enslaved by Madison in 1836, only Paul Jennings, the Taylors, and Ellen Stewart (who attempted to escape on the Pearl) are known to have gained their freedom prior to the abolition of slavery. Incomplete evidence suggests that Sukey may also have become free. 77

Join the Conversation!