Is the only difference between ATP energy and ADP that ADP has one less phosphate?
Structurally, yes. Although this third phosphate is a high-energy phosphate bond, and when this bond is broken, the energy is utilized to drive other reactions. So chemically speaking, ADP is just one phosphate away from ATP. Biologically speaking, these two molecules are very different. Adding a phosphate to ADP does make ATP, and this is primarily done by an enzyme called ATP synthase that is driven by the electron transport chain.