AI is known to hallucinate, or generate misleading, inaccurate, or completely false information.
"A common AI hallucination in higher education happens when users prompt text tools like ChatGPT or Gemini to cite references or peer-reviewed sources. These tools scrape data that exists on this topic and create new titles, authors, and content that do not actually exist."
University Library, University of Illinois Champaign-Urbana, Introduction to Generative AI
AI has been used to create and spread disinformation, or information that is intentionally false in order to mislead, cause fear, or be destructive. This happens frequently on social media in the form of "bots," or fake social media accounts created to masquerade as humans and purposely generate fake information.
According to Ali Ünlü, a research scientist from the University of Virginia School of Education and Human Development:
"By flooding social media with posts on specific issues, bots can create a false sense of urgency or consensus, leading people to believe that certain topics are more significant than they are. This practice, known as “astroturfing,” can deceive not only everyday users but also the media and government agencies."
Q&A: Is That Real? Bots Make It Hard To Recognize Truth by Bryan MacKenzie, Oct. 31, 2024
Because generative AI is trained on data from all across the web, its output will reflect the human biases and stereotypes that already exist in information. Algorithm bias can amplify and reinforce racial, gender, and other stereotypes.
OpenAI (ChatGPT) collects any personal information you provide. This includes information you provide in your prompts as well as the information you give it to use the platform, like IP address, usage, device information, and cookies. The information is used to continue to train their datasets.
It's up to you to understand what data is being collected and how it's being used. Reviewing the privacy policies of the AI platforms you use is a good starting point.
AI tools are trained on material from across the web, which includes content that has been copyrighted and used without the creator's consent. Using output that contains copyrighted work puts your own credibility at risk.