So you did something like this?
http://www.chrislazari.com/2011/04/iis-75-installing-smtp-service-securing.html(http://www.chrislazari.com/2011/04/iis-75-installing-smtp-service-securing.html)
Have you confirmed that its actually working outside of osTicket? I mostly ask because this sounds like a server configuration issue not an osTicket issue. Also PHP has mail functions built in that you may be able to utilize see http://php.net/smtp(http://php.net/smtp)