How to make ChatGPT work as a teaching assistant: a case study in law

Interactive questioning is a tool that I employ in my commercial law classes – inviting queries from my students and posing questions to them. But often only a handful of students engage in this exercise; some feel uncomfortable speaking in front of their peers, and others are tired or distracted.

While the reasons for their reluctance vary, not wanting to embarrass themselves in front of the whole class is definitely one of them.

Many students would likely be more open to voicing their questions and responses if they had access to a private, one-to-one law tutor. So five years ago, I created a simple chatbot using pre-made software and deployed it on Facebook Messenger. I compiled a list of common questions regarding the law paper I was teaching and paired each question with an answer.

However, it quickly became clear that the chatbot wasn’t very useful. It occasionally matched the questions and answers incorrectly; the students’ questions frequently exceeded the scope of those in its list; and the chatbot couldn’t actively ask students questions to test their knowledge. After a while, I pulled the chatbot offline and waited for smarter software to appear.

• Resource collection: AI transformers like ChatGPT are here, so what next?
• AI text detectors: a stairway to heaven or hell?
• ChatGPT as a teaching tool, not a cheating tool

The wait ended when ChatGPT hit the market in 2022. ChatGPT’s ability to generate human-like responses and understand the nuances of natural language make it a versatile tool.

Nevertheless, in its original form, it can’t act as a law tutor for the courses I teach because its knowledge of New Zealand law is limited. If you ask ChatGPT a specific question on a topic such as New Zealand takeover laws, its answers can be vague, somewhat generic and sometimes plainly wrong. However, these problems can be fixed with a bit of work. And while my experience is specific to law, the lessons here could be applied across multiple disciplines.

Here’s how to turn ChatGPT into a helpful law tutor

The first step is to subscribe to ChatGPT Plus. OpenAI currently offers a free version of ChatGPT, which runs purely on GPT-3.5; but ChatGPT Plus, which has a monthly fee of NZ$37 (£19), also gives the option to use GPT-4.

There’s a big difference between GPT-3.5 and GPT-4. GPT-4 can pass a simulated US bar exam with a score around the top 10 per cent of test takers. By contrast, GPT-3.5 typically scores around the bottom 10 per cent.

The second step is to provide GPT-4 with a summary of the relevant country’s legal information on a selected topic, for instance, takeover laws. The responses it generates are based on prompts, previous user messages and previous responses. Therefore, providing GPT-4 with good prompts and the necessary information on takeover laws is crucial.

Currently, each prompt has a word limit of about 2,000 words. If the summary exceeds this, it needs to be broken down.

The first prompt could be along the lines of: “I want you to be a law tutor on New Zealand takeover laws. I will give a summary of the relevant laws in three parts. After that, you will answer questions based on the summary and your general knowledge. Please acknowledge.”

The next prompt could be: “Here is Part 1 of the summary. Please acknowledge by replying ‘Part 1 received’. Part 1 is as follows: [insert Part 1 of the summary here]”.

Asking GPT-4 to acknowledge the receipt of Part 1 of the summary is important because, without this step, it might start generating an answer after it receives Part 1.

Now GPT-4 is ready to answer user questions, pose questions and comment on the answers provided by the user.

To evaluate GPT-4’s effectiveness as a law tutor, I posed the following:

Target Ltd (“Target”) is a New Zealand company listed on the NZX. Bidder Ltd (“Bidder”), a US company, decided to take over Target. It sends a takeover notice to Target.

Target and Supplier Ltd (“Supplier”) have a long-term business contract. After receiving the takeover notice from Bidder, Target proposed to Supplier to add a “change in control” clause to the contract. Under this clause, Supplier is given a right, but not an obligation, to terminate the contract if Target becomes a subsidiary of a foreign company. Seeing no downside, Supplier accepts the “change in control” clause being added to the contract.

Advise the directors of Target whether they may have breached any provision of the Takeovers Code.”

The answer generated by GPT-4 accurately informed me that the directors of Target may have breached Rule 38(1) of the Takeovers Code by adding the “change in control” clause to the contract with Supplier Ltd after receiving the takeover notice from Bidder Ltd.

It summarised Rule 38(1) and said that by adding a “change in control” clause to the contract, the Target directors could potentially create uncertainty and discourage Bidder Ltd from proceeding with the takeover as Supplier Ltd would have the right to terminate the contract if Target becomes a subsidiary of a foreign company.

It also highlighted that the panel would need to examine the specific circumstances and effects of the “change in control” clause to determine if there is an actual breach of Rule 38(1).

Next, I requested that GPT-4 generate a question and provide feedback on my response. The question generated was:

In the context of New Zealand takeover law, what is the purpose of including a director’s recommendation in a target company statement?”

I then gave my answer to GPT-4:

The purpose is to help the bidder decide if they should make a takeover offer.

GPT-4 replied:

Not quite. The purpose of including a director’s recommendation in a target company statement is to provide the shareholders of the target company with the directors’ opinions on whether to accept or reject the takeover offer. This helps the shareholders make an informed decision based on the insights of the directors who are familiar with the company’s affairs and the potential implications of the takeover.

GPT-4 demonstrated exceptional proficiency as a law tutor, leading me to believe that it is one of the most valuable educational tools ever created. Its natural language programming has introduced a new dimension to teaching; by harnessing this tool, educators can create a more personalised and effective teaching approach that can help cater to students’ unique needs and learning styles.

Disclaimer: Parts of this article were written with assistance from ChatGPT.

Benjamin Liu is a senior lecturer in commercial law at the University of Auckland Business School.

If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.