I think it depends on the situation. We tried to play in a fivesome at Kiawah two weeks ago on vacation. Play was brutally slow, all the courses were packed. The guys in front of us were a 3 some, and I was supposed to be in a foursome, but there was an older couple originally with me and another guy who decided they wanted to play by themselves (she had an artificial leg, and he was a real old guy, I applaud them for still playing, but it was obvious they were going to be playing SLOOOOWWWW, so they did us a favor by dropping back by themselves)
Anyway, on the 2nd hole, a fairly short par 5, the threesome in front of us had not even teed off when we pulled up. So we agreed to play 5. And it worked great. Our five kept up to the players in front of us, and the people behind us never waited. Until a ranger showed up on hole no. 6 and made us break up again . . . whereupon we waited on damn near every shot until the back nine, where it picked up a bit. I can only surmise that some folks quit after 9.
I was PO'ed, but nothing I could do about it. I guarantee you we were not holding anyone up in a fivesome on that day at least.