Since a superintelligence could rewrite itself to remove human tampering, isn Friendly AI impossible?
A3). Capability does not imply motive. I could, if I wanted to, take a knife and drive it through my heart right now, yet I do not choose to do so. This objection stems from the anthropomorphic assumption that a mind must necessarily resent any tampering with its thinking, and seek to eliminate any foreign influences. Yet even with humans, this is hardly the case. A parent’s tendency to love her children is not something she created herself, but something she was born with- but this still doesn’t mean that she’d want to remove it. All desires have a source somewhere- just because a source exists, doesn’t mean we’d want to destroy the desire in question. We must have a separate reason for eliminating the desire, such as a different, conflicting desire. There are good evolutionary reasons for why humans might resent being controlled by others- those who are controlled by others don’t get to have as many offspring than the ones in control. A purposefully built mind, however, need not have
