Using GenAI tools to refine EAP assessment
There are several tools available to help English for academic purposes assess students’ writing skills. Here is some guidance on how to use a selection of them
Created in partnership with
Created in partnership with
Students’ essay writing and digital literacy skills have become popular topics in English for academic purposes (EAP) assessment research in recent months. Before the emergence of ChatGPT and other large language models, EAP practitioners like ourselves would pilot the writing coursework task and refine the brief by writing sample essays. When using artificial intelligence-generated sample essays, we can gauge their key features to refine the writing briefs.
Creating an EAP writing brief in Magic School AI
The Magic School platform provides AI-powered and fully customisable tools for educators. It features its own AI chatbot, Raina, inspired by leading large language models from OpenAI, Anthropic and Google.
Raina has the ability to customise and generate teaching-related materials. It provides a range of custom prompts after the user’s initial prompt to help with the rewriting process and to offer supplementary resources. After several rounds of revisions, we used Raina to generate an EAP writing brief on the topic of the advantages and disadvantages of AI in higher education. Figure 1 shows AI-generated examples of marking criteria for the essays.
Using Prepostseo to generate sample essays
Prepostseo is an online search engine optimisation and content writing toolkit. Its key features include a plagiarism checker, paraphrasing tool, grammar checker and text summariser, among other factors. We chose it because it shares some of the same features as Grammarly and QuillBot and used it to generate sample essays to show us what AI-generated essays look like.
As you can see in figure 2, the platform is easy to navigate. It offers 19 essay types which cover descriptive, persuasive, problem and solution, cause and effect and compare and contrast options.
It only took a few seconds to generate three sample essays of short, medium and long lengths. The main similarities between them were the structure and inclusion of uncited information. They all followed a block structure with body paragraphs about the advantages and drawbacks of artificial intelligence in higher education. There was information used as supporting detail in one of the generated essays that was irrelevant to the essay topic.
It is worthwhile to examine AI-generated sample essays so that we can better identify AI-generated content and include more requirements about essay structure, source integration and logic into our essay-writing briefs. By using GenAI to refine these briefs, we can refine assessment, ultimately building students’ language and critical thinking skills.
- Resource collection: AI transformers like ChatGPT are here, so what next?
- How to navigate the grey areas of AI ethics
- How comfortable are your students with AI and VR?
Tip for generating writing brief with Magic School AI
Identify the learning objectives: what specific skills or knowledge do you want the students to demonstrate? Are the learning objectives aligned with the whole module learning outcomes? Understanding this will help you craft a prompt that aligns with the identified learning outcomes.
Be clear and specific: use short and simple sentences and include the expected word count, structure requirements, reference system, source number and type and any specific requirements or guidelines you think students should follow.
Provide clear assessment marking criteria: include information about how the writing task will be evaluated, such as the grading rubric or assessment criteria. These documents could be uploaded as attachments into Magic School AI.
Describe your identity: Raina, the Magic School AI chatbot, can better understand and generate the end product if you let her know your identity. For example, “I am an EAP teacher from a Chinese university teaching year two undergraduate students. The writing task topic is ‘Benefits and drawbacks of integrating GenAI in higher education’. Students will be asked to write a discursive essay that discusses both sides of this topic in a relatively balanced manner. The word count is 300-500. The writing brief should include the essay topic, task description, format requirements, and structure suggestions. I attach a sample writing brief.”
Keep refining prompts: the first attempt might not generate satisfying results, so you need to be prepared to make multiple attempts to tweak and customise the brief by gradually inputting more specific instructions. Another possible way to keep refining your prompt is to rely on Raina’s automatically generated prompts for you. Each time, several prompts will be automatically generated to help users brainstorm possible instructions to improve the prompts (figure 3). You can also click on one provided prompt to generate further content in the chat box.
Mengqi Hu and Rui Xu (Tiffany) are language lecturers at the English Language Centre in the School of Languages at Xi’an Jiaotong-Liverpool University.
If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.
