I had the same problem on xubuntu hardy, using gdm as my display manager. The solution was to explicitly set my default session type to Xfce in the gdm screen, and log in again. https://bugs.launchpad.net/ubuntu/+bug/231311