Instability in systems is generally characterized by some of the outputs or internal states growing without bounds. Not all systems that are not stable are unstable; systems can also be marginally stable or exhibit limit cycle behavior.

In control theory, a system is unstable if any of the roots of its S-function has real part greater than zero. This is equivalent to any of the eigenvalues of the state matrix having real part greater than zero.