New rules warn of AI data poisoning, attacks and theft

Australian businesses are being warned by the nation’s leading cybersecurity organisation about threats to privacy, property and attacks on their operation due to the use of artificial intelligence technology.

The Australian Signals Directorate released the AI guidelines on Wednesday in collaboration with foreign security agencies, including the US Federal Bureau of Investigation, the UK’s National Cyber Security Centre and Israel’s National Cyber Directorate.

The 15-page report notes that AI “presents both opportunities and threats” to Australian businesses and outlines five concerns about the technology that could put businesses at risk.

The guidelines arrive one week after the federal government released its Safe and Responsible AI interim report that outlined mandatory and voluntary regulations planned for using the technology.

The ASD’s Engaging with Artificial Intelligence report, which was designed for small, medium and large organisations as well as government agencies, detailed a series of AI risks.

They included “data poisoning” or manipulating training data to produce incorrect results, “input manipulation attacks” involving hidden commands to access more of an AI model than allowed, and generative AI “hallucinations” in which the technology delivered incorrect data.

The report gave the example of a case in which a New York lawyer created a legal brief using ChatGPT but found six cases in the documents had been “hallucinated” by the program.

“To take advantage of the benefits of AI securely, all stakeholders involved with these systems … should take some time to understand what threats apply to them and how those threats can be mitigated,” the report said.

The guidelines recommended businesses using AI hire qualified staff, conduct regular “health checks,” maintain data backups and question how its use will affect privacy obligations.

Australian Institute for Machine Learning director Simon Lucey welcomed the guidelines, saying the risks were real but, if they could be overcome, the technology could unlock significant economic benefits.

Professor Lucey said data poisoning and hallucinations could prove to be a significant threat and anyone using the technology should take care to choose a transparent AI model.

“One of the challenges that the technology has at the moment is that it has so much potential but it’s such an alien technology in the sense that previous technologies have given us a sense of how they operate, how they work,” he said.

“When AI makes a mistake, it’s often very difficult to trace back to find why that happened.”

University of the Sunshine Coast computer science lecturer Erica Mealy called the guidelines a “great first step” in helping businesses to understand generative AI technology, particularly as it was being adopted faster than expected.

“There’s definitely security risks involved in AI for businesses in terms of trademarks and intellectual property,” Dr Mealy said.

“We need to develop a global understanding of what it is good for and what it isn’t good for and we need to keep an eye on data ownership and privacy.”

Jennifer Dudley-Nicholson
(Australian Associated Press)