The key to understanding overlays, masks, transparency, and compositing is to understand the parent/child relationship between tracks. In general terms, the parent track is the highest track in a group of tracks (often only two) and the behavior of the child tracks (i.e. how they are composited together) is determined by the parent track.
Click the Make Compositing Child button on a parent track to make a child track.
Click the Make Compositing Parent button on a child track to make a parent track.
In this first example, there are three tracks. Track one contains a mask. The circular region in the mask is transparent, so the second track shows through. The third track is also a child, so it is displayed behind track two.
In this example, the third track is also a parent, so it is displayed behind the composite of tracks one and two.
You can also produce complex compositing by creating nested parent/child groups with compositing parents and children at various levels.
When you have multiple levels of parent and child tracks, clicking the Make Compositing Child button moves the track in one level and clicking the Make Compositing Parent button moves out one level.
Hold Ctrl while clicking the Make Compositing Child button to move a track and all its child tracks in one level.