You can certainly use a scripting language to handle dialogue. Basically a script might look like this:
ShowMessage("Hello " + hero.name + ", how can I help you?")
choices = { "Open the door for me", "Tell me about yourself", "Nevermind" }
chosen = ShowChoices(choices)
if chosen == 0
if hero.inventory["gold key"] > 0
ShowMessage("You have the key! I'll open the door for you!")
isGateOpen = true
else
ShowMessage("I'm sorry, but you need the gold key")
end if
else if chosen == 1
if isGateOpen
ShowMessage("I'm the gate keeper, and the gate is open")
else
ShowMessage("I'm the gate keeper and you need gold key to pass")
end if
else
ShowMessage("Okay, tell me if you need anything")
end if
This is fine for most games. The scripting language can be simple and you can write more complicated logical branches. Your engine will have some representation of the world that is exposed to the scripting language. In this example, this means the name of the hero and the items in the inventory, but you could expose anything you like. You also define functions that could be called by scripts to do things like show a message or play some sound effect. You need to keep track of some global data that is shared between scripts, such as whether a door is open or a quest is done (perhaps as part of the map and quest classes).
In some games however, scripting could get tedious, especially if the dialogue is more dynamic and depends on many conditions (say, character mood and stats, npc knowledge, weather, items, etc.) Here it is possible to store your dialogue tree in some format that allows easily specifying preconditions and outcomes. I don't know if this is the way to do it, but I've once asked a question about storing game logic in XML files. I've found this approach to be effective for my game (in which dialogue is heavily dependent on many factors). In particular, in the future I could easily make a simple dialogue editor that doesn't require much scripting and allow you to simply define dialogue and branches with a graphical user interface.