If the bees are large, bumblebee size, they are probably Carpenter Bees. If so, they nest in the wood, specifically in a piece of wood where they can burrow in from the underside and then excavate their long tunnel to put the egg/s in.
Male Carpenter Bees do often defend a territory. They are trying to keep other Carpenter Bees away, not you specifically. They can be rather scary though, as they will hover at eye level inches away from you. My suspicion is that they see a reflection of themselves in our eyes.
Interestingly, male bees cannot sting. The stinger (which only females have) is a modification of the ovipositor. So the threatening male Carpenter Bees are harmless. Even the females are very rarely known to sting. (Honeybees, remember, are almost all females, except for the occasional drone. That's why we think all bees can sting.)
Personally I just swish at them irritably with my hat or hand and walk on by.
Now if they are smaller and have an oblong body, not a sort of round body, they might be wasps. Wasps can be nasty and sting hard and often. So I'd suggest looking up photos of carpenter bees before you swat at them!