Man versus Machine: The Ultimate Editing Challenge

17 April, 2012

By beating Gary Kasparov in 1997, Deep Blue proved that a computer could beat the best in the world at chess. But what about editing documents?

For years, the idea of computer-aided editing was little more than a joke. However, in the age of voice recognition and machine translation, have computer checkers reached the point where they can defeat professional editors at what they do best? One editor agreed to take the challenge.

The Challenge

Representing the machine world was Intelligent Editing's PerfectIt. PerfectIt is an MS Word add-in for professionals. It doesn't waste time on spelling, grammar or other conventional mistakes picked up by checking software. Rather, it focuses on difficult-to-find errors, such as ensuring:

• phrases are hyphenated consistently
• words are not capitalized in one place but lowercase in others
• abbreviations are defined
• capitals in headers are consistent
• bullets are punctuated and capitalized consistently.

Representing human editors was Pat Winfield. Pat is a leading professional editor and the owner of Bookworm Editorial Services. She's an advanced member of the Society for Editors and Proofreaders and her client list includes top publishers, government agencies, and businesses.

Keeping Score

Both software and human editors make mistakes. Sometimes they miss errors and sometimes they suggest false positives (i.e. they suggest something is an error when it really isn't). So to keep score, we awarded:

• one point to PerfectIt for every actual mistake found that the human editor missed
• one point to Pat Winfield for every false positive that the software suggested.

The challenge was carried out on four books due for publication. Each book had already been checked by its author and was to be edited by Pat Winfield prior to publishing. The four books ranged from 90 to 200 pages.

The score was tracked using PerfectIt's results. Each document was tested once before editing, then after human editing, and again after editing with PerfectIt. There was no test of spelling, grammar or any substantive editing. It was a straight copyediting challenge. PerfectIt tests 27 categories of mistakes, but for simplicity we limited the challenge to 15 of the most important. Those were split into four categories:

• Hyphenation
• Capital letters
• Abbreviations
• Bullets, Lists and Tables.

Round One: Hyphenation

PerfectIt checks consistency of hyphenation. For example, if the word 'pre-determined' appears in one location with a hyphen, it shouldn't appear elsewhere in that document without the hyphen. However, there are exceptions. For example, 'the dark-brown shoes' is correct (with a hyphen) but 'the shoes are dark brown' is also correct, even though there is no hyphen. PerfectIt shows both locations and leaves the user to decide which (if either) is wrong.

Adding up all four books, Round 1 was won by the human editor. There were a total of 113 possible inconsistencies left in the documents. PerfectIt found 16 that had slipped by the human editor. But there were 64 inconsistencies that were not actual errors. Table 1 shows the full results.

Round Two: Capital Letters

PerfectIt checks for consistency of capitalization. If a word is capitalized in one place, it usually should be capitalized in other locations. For example, capitalizing the word 'Internet' is a style choice. However, if it is capitalized, it shouldn't appear as 'internet' elsewhere in the same document.

Round 2 was also won by the human editor. However, despite the victory, a trend started to become clear. Both Round 1 and Round 2 saw a significant number of errors slip past the human editor. Even after the author had read through the work and an editor had checked it thoroughly, the software was still finding mistakes in every book. Figure 1 shows that PerfectIt scored lower than the human editor. But the fact that it found 17 errors (in addition to the 16 in Round 1) demonstrates its value.

Round 3: Abbreviations

In documents with abbreviations, it's important to define every abbreviation that might be unfamiliar to readers. For example, the first time the abbreviation 'NGO' appears in a document, it should be defined in full as 'non-governmental organization'. PerfectIt tests to make sure each abbreviation:

• is defined
• does not appear on its own before it's defined
• is not defined twice in the same document
• is not spelled out in full after it's defined.

The results from Round 3 were similar to Rounds 1 and 2. It was a victory for the human editor, but a number of mistakes slipped through that would have made it all the way to publication if it wasn't for the software. Adding together the errors for all four books, the results were:

• Total inconsistencies: 71
• Errors found by both: 11
• Errors found only by PerfectIt: 7
• False positives: 53

Round 4: Bullets, Lists and Tables

PerfectIt checks that bullets, lists and tables have consistent punctuation and capitalization. There are many different ways of capitalizing or punctuating a list. However, once an author has chosen a style, it's important to stick to it.

On bullets, lists and tables, the software won. Though human editing caught 17 errors, there were nine errors that slipped by, with only a couple of false positives (as shown in Figure 2). But at three rounds to one, and an overall win on points, the human editor won the challenge.

A Human Victory

It's hard to beat a human editor, and harder still to beat a top editor such as Pat Winfield. Software isn't about to make editors redundant. However, even though it lost the competition, PerfectIt more than proved its worth. These documents had been checked first by the author and then by the editor, but PerfectIt still found 49 errors that had got past both.

In just minutes, PerfectIt can check a book-length document and dramatically improve the quality of editing, even after a leading professional has been through it. So when it comes to delivering quality documents, the competition shows that the real victor is a mixture of human judgement and computing power. Software can't beat a human editor, but a human editor with software can deliver the best documents of all.

“I take pride in my work, so even though I can work without software checking, I always use PerfectIt at the end to ensure my clients get the highest-quality documents I can deliver.” — Pat Winfield, Bookworm Editorial Services

Try the Challenge

Do you work with reports, proposals, articles, books, contracts or other professional documents where quality matters? If so, are you ready to take the challenge yourself? Can you do as well as Pat? PerfectIt runs on PCs with MS Word. Click to try the free download now.