Enable the Categorymembers Module.
When Enabled, if the category is mapped to a group name - the category manager will no longer be able to add/remove users manually. The only other option is to change the role of the users that were added through the synchronization process.
Prevent the category manager from overriding users' roles, where the offline synchronization process controls both the users and their roles.
When enabled, the user suggestions in Add Member to Category also finds the groups.