If I have two browsers installed, do I have to configure both of them to use the proxy service?

No, but you will need to set up each browser that you plan to use with the proxy server. Configuring one browser will not automatically configure the other one.