After I had removed the groups I didn't want to refresh the group membership in, I ended up with a textfile called refresh.txt as input file.
I ended up using this command in a command prompt to re-add the users in the groups from this file.
for /F "delims=" %f in (refresh.txt) do dsget group %f /members | dsmod group %f /chmbr
Things you must take into consideration is that if you do this on many groups with many members, it will start a lot of replication. So, you might want to split the list into multiple smaller lists to not affect the network too much.