Mask Groups are designed to help in controlling the areas in your adjustment thats are affected by allowing you to restrict those selections without compromising the mask.
Mask groups are typically used with Adjustment Layers and used to restrict the areas that are affected by your adjustments. There are three methods that you can use with Mask Groups, Selection, Feather and Paint. With the Selection method, you make a selection of any kind, selecting the areas that you only want affected. When you press the Selection button it will put the layer you are on, into a group and use the selection you made to reveal any adjustment in only that area. This is a hard line selection with no feathering, so typically you’ll want to use this if there are isolated elements in your selection.
The Feather button works exactly the same way as the Selection button. You need to make a selection before using it, but instead of a hard edge mask on the group, it feathers the mask based on your input. When you press the button a Gaussian Blur prompt window will appear asking how much you want to feather your selection by. It defaults to approximately 250px, which will work in the majority of full size images, but you can adjust this as you require.
With both the Selection and Feather buttons, you must first make a selection. If you press these without making a selection you will get a warning requesting that you make a selection before you proceed.
The Paint button works by dropping the adjustment layer into a group and putting a black mask on the group, hiding the entire adjustment. Then you use a white paint brush to reveal the adjustment in just the areas of interest.
The Apply button has been created to collapse any groups that you created into a single layer. When you create many groups with layers, your file sizes can get quite large, this tool take the mask on the group and combines it with the mask on the adjustment inside the group, leaving you with a single layer. All of the adjustment you make will still be there, but you will be left with one group, saving on file size.
The video below will show how to use the Mask Group tools in more detail.