Common risks of AI
Accountability, liability and medico-legal risks
AI involvement may create uncertainty about responsibility when an adverse event occurs. Clear individual, organisational and vendor responsibilities must therefore be maintained for AI-assisted decisions and outcomes.
Automation bias, over-reliance and human oversight risks
Excessive reliance on AI may undermine human autonomy, clinical judgement and diagnostic vigilance. Risks include accepting AI-generated recommendations without adequate critical review, excessive trust in algorithmic outputs and the gradual de-skilling of clinicians.
Bias, discrimination, fairness and equity risks
AI systems may inherit or amplify biases in their training data, potentially producing unfair or unequal outcomes. These risks may affect groups based on race, gender, age, geographic location or other characteristics, particularly where populations are underrepresented in the data.
Clinical safety and quality risks
AI systems may produce inaccurate, incomplete, misleading or unsafe outputs that could adversely affect patient care. These may include incorrect or missed diagnoses, inappropriate treatment recommendations, hallucinated clinical information, unreliable predictions and performance degradation over time, particularly where systems are inadequately validated or monitored.
Governance, oversight and implementation risks
Poor implementation, inadequate evaluation and insufficient ongoing monitoring may create risks. These include deploying unvalidated tools, unclear approval processes, inadequate governance structures, failure to monitor performance and insufficient incident reporting.
Organisational and workforce risks
Organisations should recognise that introducing AI may affect workforce roles, responsibilities, skills, workload, workplace culture and administrative functions. Risks include inadequate training, unclear responsibilities, resistance to change, poor change management, workflow disruption, inequitable access to AI tools, displacement of some administrative roles and reduced capability to work safely when AI systems are unavailable.
Privacy, confidentiality, cybersecurity, and information security risks
AI systems may create risks relating to patient privacy, confidentiality, consent and information security. These include entering identifiable patient information into external platforms, unauthorised data sharing, inadequate consent processes, data breaches and secondary use of patient data.
Regulatory and compliance risks
Rapidly evolving AI technologies may create uncertainty regarding legal and regulatory obligations. Risks include using unregulated tools, failing to meet consent, privacy or record-keeping obligations, and not complying with applicable software-as-a-medical-device requirements.
Transparency and explainability risks
Some AI systems operate as “black boxes”, making it difficult to understand, audit or explain how outputs are generated. This may reduce clinician confidence and create challenges when reviewing decisions or explaining them to patients.
Common risks with AI scribe tools
Accuracy of the notes:
The doctor conducting the consultation is responsible for the accuracy of the medical record of that consultation. Any patient notes generated are deemed to be signed off/approved by you. Before entering the AI-generated record into the clinical record, the doctor must check and if necessary, edit the document to ensure accuracy, and that relevant content has been included (or not been excluded).
Consent:
Gaining consent from the patient before recording is critical and should be documented in the patient’s medical record. Providing information and signs in the practice about the AI technology, while helpful for expectation setting, does not substitute for consent. Have a system in place to seek and record the consent for the use of this technology (it may be inbuilt).
Security of content:
Determine whether the content is encrypted/redacted and if it is stored (even temporarily) on an overseas server (as this may breach Australian privacy legislation if specific consent is not obtained).
Privacy/Storage:
You will need to ensure any provider meets the obligations under the Privacy Act before utilising their services. The contract between the AI program provider and the user/doctor should cover information security and Australian privacy law. Is the AI program regulated by the TGA. and if so, approved by them as part of their regulation of software based medical devices?
