If I change the encryption type or the encryption password and the maximum number of recovery points are then reached, what happens?

The image consolidation during backups will continue as usual for images with the older password. When the remaining oldest image is the last Full Backup with the old password, that Full Backup will be deleted.