Balance is an even distribution of weight, and this applies to objects in the physical realm, such as your body weight, architecture, dancing, yoga; flavors like spicy, salty, sweet, and savory; and of course it applies to the visual design as well, as elements like points, lines, and forms have visual weight that can either be balanced or unbalanced to achieved a particular mood or convey an idea.
Closely related to symmetry, objects can be balanced via symmetrical placement on the page as well as in relation to each other.

GDPE diagrams balance 08

But balance can also be less explicit, and require a little more work from the ol' visual cortex, such as the example of asymmetrical balance below:

GDPE diagrams balance 09

Similar to unity, balance can convey order and unity of forms, but it can also help guide the eye as it flows through the content. Balance creates a feeling of order, whereas imbalance can create a feeling of unease or movement in an otherwise still composition.

The below examples exhibit some kind of balance or imbalance. 

Achieving balance or imbalance

Scale: Perhaps the most obvious way of working with and achieving balance is through the principles of scale and proportion. Due to our real-world experience with such things, forms that are larger in size feel heavier. A large form can thus be balanced with a larger number of smaller forms.

Balancing objects of different scales: which way do you think the scale is going to tilt? Will it tilt at all?

Colors and values: colors carry with them their own weight. Reds and purples may feel heavier than light blue or yellow. Value is also closely related to weight, as black often feels much heavier than, say, white or light grey. One can use colors to achieve balance, but it is often more interesting to create imbalance through contrast color and value weights.

Balancing heavy and light colors and values

Thickness: similar to scale, lines can be thick, and thus feel heavy. Likewise, other lines can be thinner and feel much lighter. This contrast in line weight is particularly important in the world of typography, where even a single letterform will require within it a balance of thick versus thin lines:

Different letterforms with different contrasts between strokes.

You'll likely find that many typefaces have bottom-heavy letterforms.