I think that you'll see a lot of folks agreeing with you; I wrote of a similar occasion earlier in the summer. Most of the younger group that surf here have a tendency to share and share alike; it can be the body boarders who are inconsiderate (along with their parents...the "hey!!! we're on vacation, the hell with everone else" school of thought).
Surfers know that some fishermen will gladly chuck a 4 oz. weight their way, so they obey the 300' from the pier rule, but others......
Anyway, don't let it get you down; acccording to the time on your post, it happened about 1/2 hour ago. I say go back, go fishing, and move if necessary. But don't let them make you just sit in the room.