Phase 1 Transcription Complete! Huzzah! Huzzah! Huzzah!
On June 21, 2016 we embarked on Decoding the Civil War, an ambitious project to transcribe 12,921 handwritten pages from 35 ledgers. These ledgers were from the archive of Thomas T. Eckert, head of the Washington office of the United States Military Telegraph, Assistant Quartermaster of U.S. Army, and Assistant Superintendent of the U.S. Military Telegraph.
- Thomas T. Eckert
The 35 ledgers record the telegrams sent and received by the War Department during the American Civil War. About one-third of the recorded messages in the ledgers were written in code, and another third may have never been published in the The War of the Rebellion: A Compilation of the Official Records of the Union and Confederate Armies (often simply abbreviated to OR), and there are more than 100 communiques from Lincoln himself.
As of this morning, November 15, 2017, we can announce that all the ledgers have been transcribed by our Volunteer Corps! All the difficult work of parsing sentences, figuring out 19th Century hand and spelling, and even dealing with difficult to read, smeared, pages has been completed in under 17 months. An incredible achievement. It would have taken 2 staff members working nearly full time 3 to 4 years to reach this goal!
We have begun to publish those ledgers on the Huntington Digital Library; 16 are already available for researchers. As we work though the huge amount of data provided by our Volunteer Corps, the remaining volumes will be published. We are excited to provide this new trove of American Civil War material to the Public, Researchers, and Students. The work completed has already been used to create educational materials that will be published shortly.
We began this project hoping to be finished in six months to a year. We learned quickly that it would take longer, that the task was full of challenges, and that tools needed to be built to overcome some of those challenges. Through it all our loyal volunteers stayed with Decoding the Civil War. As we move forward with the next phase of our project we will call again on those volunteers to help.
Right now, those volunteers can rest. All who have worked on Decoding the Civil War deserve thanks.
Thank You.
Strike up the Band! A lively polka is in order!
Thank you also to the National Historical Publications and Records Commission (NHPRC), which funded a two year grant for this project and to our institutions who have provided support throughout: The Huntington Library, Art Collections, and Botanical Gardens; North Carolina State University; and Zooniverse with its spectacular team at the University of Minnesota. Finally, a thank you to Daniel W. Stowell, whose advice, support and enthusiasm has been fundamental since the beginning.
Decoding the Civil War: Phase 2, Two Work Flows, Your Choice
After a year of hard work by our volunteers on Decoding the Civil War, Phase 1, we are about ready to launch Phase 2, the marking of metadata within specific telegrams. There are two work flows to this task. The first work flow, Code Words, is marking the arbitraries, or code words, for those messages in code. These coded telegrams will then be fed into Phase 3, the final decoding of the telegrams. Having the marked arbitraries should make the process of decoding much faster, possibly aided by computer algorithms.
Example of highlighted code words (arbitraries).
The second work flow, Metadata, is a little more ambitious and complex, as we are asking our volunteers to work with individual telegrams, identifying specific metadata such as the sender, recipient, date sent, time received, etc. We are asking for metadata for a total of 10 fields; most telegrams have only a few; rarely do they have all 10. What we wish to accomplish is a way to provide simple metadata that will enable researchers to find all the telegrams to, say, Secretary of War Edwin M. Stanton, no matter whether it is in Ledger 2, or 6, or 22.
But did not Phase 1 enable full-text searching? Yes it did, and it is wonderful, but the transcriptions are accurate to the text as written in the ledger. Keeping with Stanton, if you typed in “Stanton” in the search box, you would get those pages where “Stanton” matches the search. But what if the telegram begins or ends with “EMS” or “Stantin” or “the Secretary of War”? The full-text search would ignore those pages. Furthermore, such a search looks at the whole message and returns results for any mention of “Stanton,” including other people named Stanton or places named Stanton. What if you want to look for Stanton only as the recipient? A search in a specific metadata field for “recipient” would enable that search and give you the correct results.
Example of highlighted metadata fields.
To aid in that search we will take the metadata tagged by the volunteers in Phase 2 and standardize the terms. So, continuing with Stanton, if the recipient is “EMS” and it is tagged as a sender or recipient, we will be able to take the consensus term and edit it to the standardized form of “Stanton, Edwin M. (Edwin McMasters), 1814-1869.” Once all the telegrams are tagged and the fields edited, if you do a specific search for “Recipient” as “Stanton, Edwin M. (Edwin McMasters), 1814-1869.” you will only get those telegrams to Stanton, not from or about him, and you will have those whether they are sent to him as “EMS,” “Stanton,” or “Stantin.”
The tagging of individual telegrams in the Phase 2 Metadata workflow will eventually enable specific searches to be done across the almost 16,000 telegrams. It will enable users to look for individuals or places or dates in specific fields. And the tagging of code words (arbitraries) in the Code Word work flow will help round out this project with the final decoding of encoded telegrams. An incredibly useful archive has been made available in Phase 1 of Decoding the Civil War. Help us leverage and categorize that hard-earned knowledge in Phase 2 to aid in the discovery of the American Civil War.
The Beta Test site for Phase 2 is here. The original site can be seen here.
Take up the Challenge
Today, April 17th, 2017, marks the first day of a two-week challenge, a challenge for not only our current volunteers but for all who would like to join in. The goal is a simple one: complete 10 ledgers in Decoding the Civil War between April 17 and May 1.
Our volunteers have been doing yeoman’s work turning out 200 classifications a day (a classification is equal to a page of transcription). However, we have fallen behind where we had hoped to be at this stage of our project. Thus, the challenge and the selection of 10 ledgers. We need roughly 425 classifications per day, a bit more than double our current number. We can do this and it will help get us back toward the long-term goal of having the majority of ledgers finished by June.
But why accept the challenge? The canard is often repeated that libraries and archives are dead, or if not dead, then they are simply morgues for outdated material. Our work, all our work, has demonstrated that active collaboration, research, and discovery are vital. Remember that the work that is being done on Decoding the Civil War brings together resources from four institutions — The Huntington Library, Art Collections, and Botanical Gardens; the Papers of Abraham Lincoln at the Abraham Lincoln Presidential Library and Museum; North Carolina State University; and the Zooniverse with its team at the University of Minnesota — and the hard work of over 3,000 volunteers. There is also the generous backing of the National Historical Publications and Records Commission (NHPRC). The collaboration of these groups has brought back to life telegrams from the Civil War, presenting the United States Civil War to the world in a continuous stream, not neatly packaged and organized.
Finally, Decoding the Civil War has created new and exciting paths of research—paths that have been cleared by the hard work of the citizen archivists, who have generously volunteered countless hours to this collaborative project. A hearty Thank You to them!
Starting today, let us see what new paths can be carved and cleared. To keep track of our progress we will be resetting the statistics page to reflect only the ten ledgers in the challenge. The numbers will not be set to zero as some work on the chosen ledgers has already be done. Rather, the numbers can be used as a base line to mark progress going forward. And we, as well as you, will be able to see the number of classifications per day clearly. Come back to our blog daily to see updates and new posts.
Go to our Decoding the Civil War project website, register as a new volunteer, or dive in!
Let us continue to prove that our work is vital! Take up the challenge: 10 ledgers in 2 weeks!
Decoding the Civil War 