Excellent question, too.
With that type of switcher, I usually recommend running a dedicated program display, a dedicated preview display, and one (or more) multiviewer displays. This way the director has a true representation of how everything looks on program to ensure focus and other things are accurate.
And, ideally, I recommend using true production displays instead of consumer TVs, but that obviously comes with a steeper price-point. Production displays are going to give you things like SDI inputs and more control over color/image.
One compromise that many people choose is to use a high quality production display for at least program, then cheaper displays for everything else.
As far as size, that basically comes down to budget and viewing distance. I would far prefer a higher quality (and brighter) smaller display compared to a lower quality (and dimmer) large display.