The resolving power of an optical instrument is a measure of its ability to create individual images of objects, as opposed to a single merged image. If you look at two stars in the sky, for example, you can tell they are two stars if they're separated by a large enough angle. Some stars are so close together they look like one star to the naked eye, but you can often tell they are two stars by looking through a telescope. Light entering your eye passes through your pupil, which is a couple of mm across. The lens (or mirror) at the end of a telescope is much larger, which makes all the difference.

If you look at an object, light entering your eye creates a diffraction pattern on your retina. When two objects are separated by a small angle, the diffraction patterns overlap. You are able to resolve the two objects as long as the central peaks in the two diffraction patterns don't overlap. The limit is when one central peak falls at the position of the first dark fringe for the second diffraction pattern. This is known as the Rayleigh criterion. When the central peaks overlap the two objects look like one.

The size of the central peak in the diffraction pattern depends on the size of the aperture (the opening you look through). The smaller the aperture the more spread there is in the pattern.

The minimum angular separation for two objects to be resolved when viewed through a circular aperture is:

The closer you are to two objects, the greater the angular separation between them. As you get further from the objects, however, the angular separation decreases until they eventually merge to become one.

Definition and Graphics provided by webphysics 

Rayleigh Criterion

For a single slit aperture 

For a circular aperture 

Resolved Image

Two very distinct peaks from two different sources

At  Rayleigh Criterion

Two peaks are still distinct but with a great deal of overlap

Unresolved images

The two peaks are no longer distinct