Version 8.0 of Messages.app, shipped with OS X Mavericks, has a new feature which automatically inserts Emoji symbols when you type common text emoticons, such as replacing ":)" with a smiling Emoji face.
Personally, I prefer to send the actual text rather than an Emoji symbol, but I couldn't find anywhere in the app's preferences to turn it off. After a lot of searching, it turns out the answer is simple. Just click in the text field in Messages and deselect Edit → Substitutions → Emoji, and emoticons will no longer be replaced with Emoji symbols. You can also right click in the text field and deselect Substitutions → Emoji from the menu.