No, the POP3 settings are not related to the problem you are having. This is only for retrieving emails, not sending.
When submitting a new ticket both the client and system users should get an email (at least when it's like this configured in the admin). Are you getting neither emails?
If autoresponders are off for a client, after opening a new ticket he will see something like this on the screen:
Thank you for contacting us.
A support ticket request has been created and a representative will be getting back to you shortly if necessary.
Support Team
otherwise something like:
Thank you for contacting us.
A support ticket request has been created and a representative will be getting back to you shortly if necessary.
An email with the ticket number has been sent to email@domain.com. You'll need the ticket number along with your email to view status and progress online.
If you wish to send additional comments or information regarding same issue, please follow the instructions on the email.
Support Team
Perhaps you can check what you are receiving. Another important thing to know is: are you running osTicket local?