The Hill sphere (or Roche sphere, not to be confused with the Roche limit) applies to objects such as planets that (1) are in orbit around a more massive object such as a star, and (2) are massive enough themselves that smaller objects (satellites) can be in orbit around it. For a planet, the Hill sphere is the imaginary sphere within which a satellite or moon can be in orbit around the planet, and outside of which the Sun or star's gravity prevents the smaller body from orbiting the planet. In other words, the radius of the Hill sphere (Hill radius) is the maximum distance a satellite can be from a planet and still orbit the planet.
As an example, since the Moon orbits Earth it must lie within Earth's Hill sphere. However, if the Moon were far enough away from Earth -- outside Earth's Hill sphere -- it would then orbit the Sun rather than Earth. The gravitational force from Earth must dominate that of the Sun in order for a satellite to orbit it, which only happens if the satellite is close enough to Earth.
For a planet orbiting a star in an elliptical orbit, the Hill radius (radius of the Hill sphere) rHill is given by
where a and e are the semimajor axis and eccentricity, respectively, of the planet's elliptical orbit, and m and M are the masses of the planet and star, respectively. In the above formula, the quantity a(1-e) is the distance of closest approach of the planet to the star.
For circular orbits, e is zero and a is the radius rorbit of the orbit. In this case, the Hill radius is
The formulas for rHill make intuitive sense for the following reasons. A larger rHill implies a greater tendency for satellites to orbit the planet rather than the star. It can be expected that this would happen for either a larger planet orbital radius (the planet is farther away from the star), a larger planet mass, or a smaller star mass. These all result in an increased influence of the planet's gravity relative to the star's. The formula is entirely consistent with these intuitive claims. Moreover, a highly elliptical orbit (greater e) brings the planet closer to the star for the same semimajor axis a, and so one would expect a smaller Hill radius for larger e, again consistent with the formula.