I've tried reproducing the issue you describe on one of my systems here, and I'm unable to do so. I tried with an Ubuntu 64-bit VM, made it a Shared VM, and even copied the rather unusual memory configuration you have set in your /etc/vmware/config, but everything still worked just as it should, and I have no trouble clicking the "Save" button when adjusting the memory size well above 8 GBytes. I can subsequently power on the VM with no trouble too.
So... There is some as-yet-undetermined factor at play. I tested with Debian 6.0.6 x64 on the host machine, not the Xubuntu 12.04 x64 you're using there, so it might be something distro-specific, perhaps. If you happen to observe anything else that triggers (or cures) this problem, please do let me know so we can work on tracking it down and fixing it.
Thanks,
--
Darius