I love how all the TV commercials for AI - whether it's Google's AI or Apple's or anybody else's - have tiny fine print that says you, the user, should verify everything the AI tells you because they can't guarantee that it's accurate.
When the large language models use the entirety of the internet - and the internet is full of BS - then much of what the AIs spit out will also be BS.
On a technical forum I participate in, one of the users asked a tech question to one of the AI services. The answer was complete garbage. The user then queried the AI to show the sources it used for the answer. It immediately returned a list of sources. Several of them were not even real. It made them up. It was like a kid getting caught lying and doubling down by lying some more.