The Instrumental Convergence Thesis

Page 76:

"Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by many intelligent agents."

Comment:

The Instrumental Convergence Thesis could help us prepare for artificial agent behavior and make sure some unwanted behavior does not occur. In this text however, it almost seems nothing can be done and that a superintelligent artifical agent will follow these instrumental values without question. Whereas we might still have some control on the behavior of an artificial agent, if only we incoroporate it in our strategy of building it. An artifical agent might also be more flexible in its goals and approaches than how this text makes it out to be. If we reach the stage of a superintelligent artificial agent it might also be aware of certain values to uphold and different approaches that might be favored by these values. Its final goal might then also be changed slightly when it seems all favored approaches comes very close but do not fully reach its final goal. Lastly, an artifical agent might be able to ask for help or clarification in certain situation with have a drastic impact. If we start discussing future, hypothetical situation, why not also include ways in which artificial agent might be able to reach goals in favorable ways.