Insure that the IIS role is installed on your server. Then, make sure that the SMTP Server feature is enabled.
The SMTP service should be started.
Open the IIS 6.0 Manager so you can manage the SMTP server settings.
Right-click to open the properties dialog.
Click the Authentication button. Ensure that anonymous access is selected (SharePoint requires this).
Use the Connection button to Add computer(s) that can connect to this SMTP server. Since it is anonymous, you want to be very selective.
Use the Relay button to add computers that can relay through the virtual server. Again, necessary for security.
Select the Delivery tab. We’re assuming you have to authenticate to your email provider to forward and send email through them.
Click the Outbound Security button. This shows using basic authentication with TLS encryption (pretty standard from what I’ve seen).
Click the Outbound Connections button. In many cases you’ll have to change the TCP port from the default of 25. My provider uses 587.
Clicked the Advanced button. In the Advanced Delivery dialog, enter the SMTP server in the Smart host field and a Masquerade domain. The Masquerade domain name replaces the local domain name listing in Mail From lines in the SMTP protocol (your SMTP server may be in a domain such as mail.yourdomain.com).
Set the Outgoing E-Mail Settings in SharePoint Central Administration. Notice there are no security settings.