Is there a way to
accomplish the same thing (keep the browser running, but don't show a
browser window) without raising a conspicuous dialog?

Selenium seems to use xvfb as a virtual display.

There's also a new nsIAppShellService::createWindowslessBrowser(),

but I don't think that's ready.