How Can You Analyze 12,000 Documents Quickly? It's Natural … Language Processing
Federal agencies do not carry out laws and conduct public policy alone. Instead, they rely on public comments to inform their actions. However, sometimes the agencies are overwhelmed with thousands public comments – too many for their staff members to read individually.
How do you quickly organize and analyze thousands of unique public comment documents? A successful quick turnaround project requires hard work, a great team, and innovative solutions.
Abt Global’s public comment management team recently quickly analyzed and reported on a set of 12,000 citizen comments related to a controversial federal action. Given only weeks to perform the analysis, we collaborated with Abt’s data scientists to use natural language processing (NLP) to help streamline the work and meet our tight deadlines. NLP offers a link between computer science and everyday language. With NLP, you can use machine learning and algorithms to quickly analyze text, identify patterns and concepts, and create meaning.
Helping Clients Make Important Decisions
The team determined which coding tasks were best handled with NLP and which required close human attention. NLP is good, but it isn’t 100 percent foolproof! We identified several comment categories that were well-suited for automation, including determining similarity between documents and identification of commenters’ geographic locations. Other tasks still required human attention, but using NLP for the more repetitive tasks allowed staff the time to closely scrutinize and accurately code the documents to prepare the final analysis.
This experience led me to explore ideas about how we can incorporate more NLP into our public comment management analysis process and Abt’s CommentCounts tool. Given the ever increasing volumes of public comments received by agencies, and often limited resources to perform the required analysis, I look forward to advancing NLP techniques to efficiently help our clients make important regulatory and policy decisions.
Moving forward, here are three innovations coming to Abt’s CommentCounts tool and our public comment management practice using NLP techniques:
1. A New CommentCounts Function to help analyze and code more complex and multifaceted public comments;
2. Robust Quality Assurance practices to validate the accuracy of a NLP analysis functions, given the high legal stakes for failure to adequately address critical public comments; and
3. Detection of duplicate comment submissions, comments received via bots – artificially generated comments using automated processes – and comments using false email addresses and the processes for handling these situations.
NLP is an exciting game-changer for many areas of our work! Please feel free to ask questions and share your NLP experiences as we explore new methods in our change-agile culture.
Read more about CommentCounts.
