Undefined behavior is the result of any action in a programming language that the language specification specifically avoids defining, without making any suggestions as to what might happen. An example in an esoteric programming language would be a program going off the right side or bottom of code space in 2L. Non-esoteric examples in C include using an uninitialized variable, overflowing an integer, and abusing sequence points - for example:
int b = 1; b = b++; // undefined int c = b++ * b--; // undefined
Code that exhibits undefined behavior may produce unexpected and inconsistent results, crash the program (or the interpreter), cause an error, etc. An implementation has no responsibility to try to do anything that even remotely makes sense when faced with undefined behavior, and in fact is perfectly free to make demons fly out of your nose if it wants to.
In the field of esoteric programming, undefined behaviour is often used as part of Turing-completeness proofs. Such proofs often work by showing that a language can emulate a more complex language, and often the more complex language has to be defined specifically for the proof. Frequently there will be some areas where the translation breaks down, meaning that some parts of the more complex language will act weirdly when used, perhaps causing the invariants on which the proof relies to fail and causing other commands to start malfunctioning. In order to get around these problems, the offending situations can be marked as undefined behaviour; as long as the complex language is still Turing-complete even despite the undefined behaviour, the proof will work, and more simply than it would trying to define the behaviour in these situations. (The Turing-completeness proof for the more complex language, in turn, has to ensure that the programs it generates never allow the undefined behaviour to be invoked, as there's always the possibility that it could do something that ruins the construction.)