It is possible to remove flash plugin for Safari, but keep it for Firefox. Just move it to Firefox profile directory.

cd ~/Library/Application Support/Firefox/Profiles/xxxxxxx
mkdir plugins
mv /Library/Internet Plug-Ins/Flash Plugin.plugin plugins
mv /Library/Internet Plug-Ins/flashplayer.xpt plugins

This boosts Safari startup speed a lot, and yet I can fallback to Firefox to visit those sites that requires Flash. For example, Amazon EC2 console panel.

Google Chrome does not load the flash plugin from /Library/Internet Plug-Ins directory. It actually has one under /Applications/Google Chrome.app/Contents/Versions/7.0.517.41/Google Chrome Framework.framework/Internet Plug-Ins directory.