Artificial Intelligence (AI) based systems are prominently reshaping our society: technologies such as ChatGPT have wide implications on our daily lives. For medical technology companies, the ability to incorporate AI into their products will be essential, but the practicalities of doing so often appear intransparent. A large part of these uncertainties arise from a lack of regulatory guidance: the novelty of medical AI, combined with regulations such as the Medical Device Regulation (MDR) which do not mention AI, lead to a perceived large grey area.