Infinity is a concept of something beyond any finite number. An infinity can either describe size, where it's called a cardinal infinity, or place, where it's called an ordinal infinity. While there are multiple levels (infinitely many, in fact), the term "infinity" is often used to mean any of the infinities, especially when it doesn't matter which infinity is used.
Many esoteric programmers seem to have a torturous relationship with infinity. It could be said that mathematicians embrace infinity, engineers aggrieve it, and programmers, being both mathematicians and engineers, are obliged to ignore it lest their heads explode.
To please strict constructionists who object that they just can't afford that much tape, these structures may also be described as unbounded - meaning that they are finite, but may grow when needed without limit. This, of course, acts exactly the same as having an infinite tape to begin with, so the concepts are equated in the far, far, far majority of cases.
There also exist models of computation where the program is represented as infinitely many non-dynamic instructions. Such a program is termed transfinite. Note that such a program is equivalent to having a source file of infinite size, rather than simply having an infinite loop (which corresponds to infinitely many dynamic instructions).