[S]ecurity and academic researchers have found that AI code assistants invent package names. In a recent study, researchers found that about 5.2 percent of package suggestions from commercial models didn't exist, compared to 21.7 percent from open source or openly available models.
Running that code should result in an error when importing a non-existent package. But miscreants have realized that they can hijack the hallucination for their own benefit.
Not surprised about twitter's grok AI having a security hole (and the doge website). When you prioritise efficiency, it's difficult to distinguish devs who do things right from those that cut corners. In fact, those that cut corners might initially seem better to management. On the other hand, lots of slow devs hide behind "doing things right".
There is no easy answer. Pushing for efficiency isn't necessarily wrong - just that it's difficult for management to push in that direction without these kinds of issues. That cost (along with the reputation and likely maintenance cost) might be acceptable given the speed of shipping - although that balance changes significantly if personal/payment data are involved.