A fate worse than death is something judged to be more terrible than dying. Torture is often considered worse than death, as it goes on longer. So is being allowed to live while all the things you hold dear are removed, killed, spoiled, or destroyed in some way. The result is that the victim's spirit is broken such that they no longer have the will to live.
In some canons, destruction or entrapment of the soul is possible, and is generally considered a fate worse than death, especially if a benign canonical afterlife exists. In others, contagious diseases or magical curses turn characters into vampires, Borg, zombies, or other such creatures who would happily kill (and possibly eat) their former friends. Some characters would consider rape to be a fate worse than death; others would not (but almost none would trivialise it). Being trapped or immobilized, but conscious — for example, as a statue — for long periods of time is another possibility. Many characters would also list witnessing the loss of their loved ones, the destruction of their home country/city/planet, or the failure of some important goal as a fate worse than death.
Sadly, subjecting Mary Sues to Fates Worse Than Death is simply not practical, as for canon to revert to normal, the Sue must die or be otherwise removed (though this usually involves the Sue dying, for obvious reasons). So, the usual solution is to give the Sue a taste of a fate worse than death, just to teach her a lesson, and then to kill her anyway.