To better understand the approaches to text analysis at higher education institutions in the U.S. and Canada, we conducted a research study on 84 institutional researchers and assessment professionals during a 4-week period from July 8 to August 5 this year. Our goal was to identify any challenges they face and how we can make the experience easier and more efficient.
In the sections that follow, we provide a summary of our findings and illustrate how NLP can improve your experience.
Surveys with the most valuable student feedback
The amount of time spent on text analysis
The approaches and pain points of text analysis
Different types of software used for text analysis
How NLP can help relieve your pain points
Which survey has the most valuable open-ended responses for your school?
This was the first question we asked respondents, and the most popular answer with 31 percent of responses was student satisfaction surveys, indicating that institutions care deeply about the satisfaction of their students. Student satisfaction surveys were then followed by course evaluations (23%) and campus climate surveys (17%) in terms of importance. We also found out that many schools administered campus climate and satisfaction surveys in the spring semester to assess the pandemic’s impacts on students, faculty, and staff.
In addition to the surveys shown below, several institutions also mentioned that they administer values-based surveys to measure where their academic communities stand in terms of their values. Diversity surveys was another survey administered by schools to find out how they can create a more inclusive educational experience.
How long is your institution taking on text analysis?
Our results showed that the majority of respondents (65%) took one week or less to analyze open-ended responses for their surveys with the most valuable responses. During this time frame, about 80 percent of respondents analyzed 2,500 or under responses, and they were quite efficient in handling the large number of responses.
Is your team using specialized software to analyze student feedback?
Our results showed that 77 percent of respondents used a manual approach while the remaining respondents used specialized software for text analysis. The manual approach involved reviewing all the comments and coding each response based on the themes identified, which was “time-consuming”, “tedious, and “labor-intensive” for many. The program that was used often for the manual approach was Excel, and although it was accessible and familiar, respondents faced difficulties in accurately narrowing down the themes and dealing with synonym issues.
“It takes forever” was one of the comments we received during our interviews. This interviewee was frustrated with a lot of manual labor involved with a software his team uses and with having to check whether the software is coding things properly.
This translated to an overall satisfaction rating of 5.3 out of 10 for respondents’ experience with text analysis, which indicated that although institutions are taking one week or less on text analysis, they are not very happy with their approaches.
We found that those who used specialized software (M=6.73, SD= 2.40) to code their open-ended responses were significantly more satisfied than those doing it manually (M=4.81, SD= 2.31). It is important to note that our research also found that using a specialized software does not always equate to additional time savings. In fact, many of them were still spending a lot of time on manual work. This could explain an average satisfaction score of just 6.73 out of 10. One challenge with generalized software is that a lot of domain specific language is often lost.
Inductive and Collaborative Approach to Text Analysis
We also found that institutions use a similar, inductive process in analyzing their survey comments manually. The approach involved respondents trying to extract the main themes by going through responses and coding each response based on the themes they identified.
Text analysis was also a collaborative task where teams of two to four worked together on coding, reviewing themes, and interpreting the results. One of our interviewees said that text analysis is a “campus effort”, and her team tries to include a variety of voices and experiences from different departments in interpreting the data. As a result, their findings are widely accepted across their campus, and her team can increase buy-in from their university.
“We engaged in a process where we tried to include a variety of different voices and experiences in interpreting the data. I think that’s one of the reasons why our summary was so widely used at the time, because we kind of had that buy-in”
To ensure consistency in analysis, institutions developed structured processes each team member could follow, and team members shared their findings in regular meetings. For example, one of our interviewees used a process involving multiple rounds in his department. In the first round, his team used coding schemes to develop themes, which was followed by the second round where they developed sub-themes based on the themes identified in the first round. In the third round, the team synthesized their findings and performed sentiment analysis.
Which tools or software do you use for analyzing student comments?
Python/R was the most popular answer. It was followed by Dedoose, Max QDA, NVIVO, and SPSS/SAS that had an equal share of 15 percent each. In our follow-up in-person interviews, Atlas.ti was another platform that is often used for qualitative research by doctoral students.
What respondents liked about these software were features such as the ability to customize codes, export data, and visualize the results (e.g. word clouds). But the drawbacks included a steep learning curve, a lot of manual work, and the software’ lack of ability to capture the essence of themes. Although such platforms were useful for coding, they lacked a full understanding of the context of higher education.
Lastly, it’s important to note whether the location where each of these companies stores your information is compliant with Canada or U.S. data privacy or storage (sovereignty) laws. For example, Dedoose hosts their data on Microsoft Azure's U.S. servers only. It is always best to review the privacy agreement. Even with desktop software, information can be shared to cloud servers during updates or crash errors.
As you've seen throughout this blog, analyzing students' comments is a complicated and time-consuming task, and the current tools available don't successfully address the pain points experienced by IR researchers and assessment professionals. However, text analysis can be easier and more efficient by leveraging natural language processing (NLP). Find out how Kai Analytics uses advanced NLP techniques to help institutions.
How does Kai Analytics make text analysis easier?
Save time and manual labor. By using advanced algorithms, our process can analyze around 5,000 comments per hour, so you can better prioritize your time.
Easier and simple. Our process is simple, and we identify the themes for you. There is no coding or programming experience required.
Minimize bias and human errors. Our process also helps minimize human bias and errors that can be introduced when individuals are sorting data.
Accuracy and depth of insight. Our models are trained to specially understand the context of higher education to accurately capture the themes. Moreover, they can identify associations between themes to give a better picture of your data.
Secure and Compliant. The privacy and integrity of your information is important for us. Therefore, we have safeguarded encrypted data centres in Canada and the U.S. depending on which country your institution is located in.
We hope this blog allowed you to get a glimpse into the approaches and pain points of text analysis in higher education and invite you to learn more about our services.
To find out how Kai Analytics can help your higher education institution, please contact us by filling out the form below.
Comments