Greetings,
I am using postfix on ubuntu for sending emails through ojs3. Ojs2.x has similar mechanism. It is worth to be note that my provider blocks port 25 for preventing spam. So I configured to use port 465 to send emails from server. Also for receiving emails I use my gmail adress. All output goes through gmail to. So it is there headache to filter spam. If you want to have full email server, you need additionally install dovecot, for example, and interface, like squirrel. It is no need to change something in ojs to make it work.
You can look here for example [HOW]OCS mail settings with GMAIL smtp server