It is interesting that, even before real AGI/ASI gets here, that "the system wants what it wants", like capitalism + computing/internet creates the conditions for an infinite amplification loop.

I am amazed, hopeful, and terrified TBH.

Feedback gain loops have a tendency to continue right up to the point they blow a circuit breaker or otherwise drive their operating substrate beyond linear conditions.

This made me laugh and feel scared simultaneously.

I assume someone has already written it up as a sci-fi short story, but if not I'm tempted to have a go...