We study the best exponential decay in the (deterministic) blocklength of the probability of error that can be achieved in the transmission of a single bit over the Gaussian channel with an active noisy Gaussian feedback link. We impose an expected block power constraint on the forward link and study both almost-sure and expected block power constraints on the feedback link. In both cases the best achievable error exponent is finite and grows approximately proportionally to the larger of the signal-to-noise ratios on the forward and feedback links. The error exponents under almost-sure block power constraints are typically strictly smaller than under expected constraints. The error exponents achievable with active feedback are shown to be superior to those that are achievable with passive feedback. Some of the results extend to communication at arbitrary rates below capacity and to general discrete memoryless channels.