From the photo (thanks for providing), this is a juvenile swan, not an adult swan due to the color of its wings. It has brown/taupe grey on the back feathers, indicating that the juvenile has not turned completely white which will occur at approximately 1 year of age. This would explain why he/she keeps leaving the pond. The older parents are chasing this swan out of their habitat. This juvenile may have flown into the area of the family and as an outsider, is being told to get lost and find another pond. If you put him back into the pond, watch closely for several minutes. If he is being chased from the pond by the other swan(s) then this is the reason behind the movement away from the pond.

Now, if the photo is coming across as the wrong color, then, it could be the adult male leaving the pond, but it would not explain his being agitated with the female. This is what does not make sense. Yes, pairs (just like in any family) will sometimes get into an argument, but usually, the behavior is forgotten within a couple of hours. Nothing prolonged like you are describing. Try the blanket resolution, watch the swan and see if it is being chased from the pond by the other swans. If this is indeed the case and the other swans are chasing him from the area, you will need to find him another safe secure home free from predators. The Regal Swan