AI and insurance: The emerging risk landscape in Australia

The Oxford Dictionary defines AI as “the capacity of computers or other machines to exhibit or simulate intelligent behaviour” and “software used to perform tasks or produce output previously thought to require human intelligence.” In essence, it is a form of technology that allows computers and machines to simulate traditional human intelligence.

At its logical extreme, AI may one day theoretically be able to learn and perform any task that previously required human intelligence to complete. AI technology is, perhaps thankfully, not yet at a stage of sophistication where it can simulate all, or even most, forms of human intelligence, but it is certainly taking great leaps forward at a rapid pace.

AI is already widely used in the economy and is being adopted by businesses at an astonishing pace. The pace of AI’s development has left governments and regulators rushing to keep up. The result is that, until relatively recently, there was very little, if any, specific regulation of AI in the major jurisdictions around the world, including in Australia.

The EU recently enacted a comprehensive AI regulation through the introduction of the Artificial Intelligence Act in August 2024 (EU AI Act). The EU AI Act classifies AI systems according to the following categories of risk:

• unacceptable risk: AI systems which pose a clear threat to people are prohibited, such as cognitive behavioural manipulation of people or specific vulnerable groups (eg voice-activated toys that encourage dangerous behaviour in children) and social scoring systems (eg classifying people by reference to behaviour) and biometric identification of people, including facial recognition (except by law enforcement);
• high-risk: AI systems that negatively affect safety or fundamental rights are considered high risk and subject to strict obligations before they can go to market;
• limited risk: AI systems that make it possible to generate or manipulate images, sound, or videos (including ‘deepfakes’) are subject to transparency requirements so that individuals are aware when they have been used; and
• minimal risk: AI systems that are already widely in use, including systems used in video games, spam filters and inventory management. These systems are not regulated, but a voluntary code has been suggested.

General purpose AI models (such as ChatGPT) are subject to transparency requirements and must provide technical documentation, instructions for use, comply with the EU’s Copyright Directive, and publish a summary about the content used for training. If a general purpose AI model exceeds certain computing power thresholds specified by the EU, the provider of the model must notify the EU and demonstrate that the model does not present systemic risks.

In 2024, the Australian Government conducted a consultation process on a “Proposals Paper for Introducing Mandatory Guardrails for AI in High-Risk Settings” which set out three proposed regulatory approaches to address the risks of AI:

• adopting AI guardrails within existing regulatory frameworks as needed;
• introducing new framework legislation to adapt existing regulatory frameworks; or
• introducing a new cross-economy AI-specific law (for example, an Australian AI Act).

The consultation completed in October 2024 and the government is presently considering the responses to inform next steps on AI regulation.

The government has in the meantime released a new Voluntary AI Safety Standard for use by Australian businesses. In step with similar actions in other jurisdictions – including the EU, Japan, Singapore, the US – the intention is that the standard will be updated over time to conform with changes in best practice.

On 29 October 2024, ASIC also released a report entitled “Beware the gap: Governance arrangements in the face of AI innovation” urging financial services and credit licensees to ensure their governance practices keep pace with their accelerating adoption of AI. AI risk is therefore likely to be a key focus area for regulators over the short to medium term.

The reason why new regulation has come about is that AI can be both unreliable and even potentially harmful if it is not properly supervised. For example, AI is well known to be susceptible to several limitations and issues including:

• Model drift: When an AI model's performance degrades over time due to changes in data distribution, the operating environment, or even the model's goals or objectives, requiring monitoring and updating or retraining the model;
• Hallucinations: AI models can generate false or inaccurate information in response to human prompts. For example, Air Canada was recently ordered to pay damages after its chatbot gave incorrect information about bereavement fares, leading to a customer’s financial loss. ChatGPT was also involved in a legal case where a lawyer submitted fabricated court cases to support a legal brief;
• Discrimination: When an AI model makes connections between data based on algorithms, this may result in bias/ discrimination against individuals or groups;
• Garbage in/garbage out: When an AI system is trained on poor quality input data, the output will inevitably be of equivalently poor quality. It is inherently susceptible to human error at the data input stage and in all aspects of its subsequent use; and
• Cyber crime: AI is also susceptible to exploitation by malicious actors. It has been used to increase the sophistication of phishing attacks making phishing emails more believable by improving their wording and grammar. AI has also been used to generate 'deepfakes', audio or video content which mimics a genuine actor in order to extract personal and financial details.

These are just a selection of some emerging issues arising in the use of AI which highlight how things can go wrong, even where there is human involvement in the process.

AI can give rise to a range of liabilities for businesses. In this section, we discuss the main types of claims by third parties that businesses may face arising from the use of AI:

• IP Infringement: central to almost any AI system is a large mass of data on which the system is trained. Although referred to as “data”, the training materials are frequently themselves original works, in which copyright can subsist. The use of GenAI systems therefore has the potential to result in copyright infringement by users. There is a complex statutory regime governing copyright in Australia, including defences of “fair dealing” for certain limited purposes, against which use of GenAI needs to be considered to understand infringement risk. Mis-use of confidential information through use of GenAI is also a relevant issue.
  
  Emma Iles, an HSF partner specialising in intellectual property disputes observes: “this is a new and complex area for Australian companies. While the potential risks of copyright infringement and mis-use of confidential information are real, there are a number of steps companies can take to reduce risk, including ensuring contractual arrangements with AI service providers are negotiated with protections in place in case things go wrong, implementing policies and training directed to minimising key risks and using ‘closed’ system AI products (to protect confidential information).”
• Customer claims: the use of AI in the provision of services (whether directly or even potentially in the background) has the potential to result in liabilities to customers or other third parties if they suffer loss or damage as a result. This could result in liability for negligence or contractual breaches if services are not provided with reasonable care or products that use AI technology do not comply with capabilities represented.

• Discrimination: the use of AI has the potential to result in unintended and unpredictable bias or profiling. Therefore, care needs to be taken when AI is used in facial recognition technology, the employment space or in determining eligibility for financial products such as mortgages.

• Regulatory action: regulators such as ASIC and the ACCC are increasingly focused on taking action to avoid harm resulting from AI. Whilst new codes and AI-specific legislation are under consideration, there are a number of existing laws relating to misleading and deceptive conduct and directors’ duties that could be used by regulators to seek civil penalties for AI misconduct in appropriate cases. For example, ‘AI washing’ (ie where a company claims to possess a level of service relating to its use of AI which it does not in fact have) may be one of the new frontiers of regulatory action (like recent cases of greenwashing in climate disclosure cases).
  
  Christine Wong, a partner at HSF specialising in regulatory investigations and enforcement, observes that: “statements about the use of AI can potentially be misleading in at least two ways: by falsely asserting capabilities that the relevant AI does not in fact have and/or by omitting to disclose key risks associated with the use of AI and how data about individuals is being used. Representations about AI can be complex, particularly where this is an area undergoing rapid technological advancement and there is currently a lack of standardised language and definitions concerning the use of AI.”

  Tania Gray, an HSF partner specialising in regulatory disputes also notes: “there are a multitude of existing causes of action that might apply to bad outcomes arising from the use of AI. Regulators will consider the range of tools in their arsenal to pursue action for conduct they consider worth pursuing.”

• Class Actions: Australian class action regime continues to be an avenue that plaintiff law firms and funders use to bring proceedings against both listed and unlisted companies involving allegations of loss to shareholders and consumers. To the extent a company is facing a regulatory investigation or suffers a substantial share price decline following the disclosure of adverse information to the market arising from an AI event, the possibility of a class action continues to loom large.

  Melissa Gladstone, a partner at HSF specialising in class actions observes “class action risk remains a key issue in corporate Australia. The rise of AI, which is rapidly advancing ahead of the pace of regulation, presents yet another category of emerging risk for boards and directors to grapple with.”

Whilst third party liability risk presents some of the most obvious and significant potential liabilities facing companies that use AI, there are also a number of ‘first party’ issues that can potentially occur causing loss to companies directly. For example:

• System damage: AI becomes increasingly embedded in IT systems, it is possible that IT malfunctions or programming errors could result in system outages that cause system damage and business interruption loss. The recent CrowdStrike outage that caused global system failures is an example of the type of issue that can arise and also highlights the risks in relying on critical third party service providers. A logistics company which relies on AI to optimise its delivery routes and storage capacities could also suffer first party loss if the system does not perform.
• Crime: as noted above, threat actors are increasingly using AI to improve the sophistication of phishing scams and enhancing their ability to breach data security protections. The rise of AI arguably increases the risk of theft of company assets (both financial and data). 
• Physical damage: given the extent of automation powered by AI already being used today, it is increasingly more likely that AI may cause physical damage. For example, by way of autonomous vehicles colliding with other property or even by way of an AI-powered thermostat malfunctioning and causing a fire in a factory.