Emotional Impact

Due to the fast development of AI systems, their behavior, in some aspects, has become akin to humans. Especially with the introduction of large language models like chatGPT, it has become increasingly difficult to differentiate human and AI generated language. When coupled with human-like design, this cognitive ability has been found to lead to emotional attachment by users. While emotional impact cannot be prevented entirely, it is important to assess the extent to which the AI encourages human attachment and ensure clear signaling to the end-user that they are interacting with an AI.

You can find futher information about emotional impact of AI here