Anchoring flag holder permanently requires the proper anchors, screw sizes and materials. Sleeve anchors are a possibility as well as anchoring cement.
The good news is there is more than one good solution to this problem. Lets examine what usually happens to the average homeowner using plastic anchors.
a. They use a screw that is too small a diameter for the anchor, or
b. They use a screw that is too short for the anchor.
Assuming you used proper screws and your bracket still pulled out of the house, it may be time to step up to sleeve anchors. One of the best features of a sleeve anchor is the harder you pull on them, the more they expand.
The proper size for troublesome flag brackets is the 5/16 x 1 1/2-inches hex nut sleeve anchor. The 5/16 anchor has a 1/4-inch center bolt that should fit your bracket.
To install, redrill your holes in the brick/mortar to 5/16, at least 2-inches deep. Then attach the anchors to the bracket and align the assembly into the holes. A gentle tapping with a hammer is allowed. If the anchors will not start, try to ream the openings of the holes with a 5/16 carbide drill until they do.
Once the bracket is against the wall, tighten the hex nuts on the ends of the anchors until they get firm. Make sure the bracket is tight to the wall because the nature of the sleeve anchors is not to draw stuff tight to a surface. They just hold it there.
Another great solution for oversized holes in a masonry surface is with anchoring cement. Coating an oversized hole with anchoring cement can tighten loose fitting plastic or lead anchors. This powered product is mixed with water and packed around the item to be anchored. Do not start mixing too far from the hole, because it sets fast. A cup of Rockite Anchoring Cement will set in five minutes.