The swan parents will leave the nest periodically for short periods to take a rest, bathe, eat and drink. Once all eggs are laid and incubation begins, especially when hatching is imminent, the rest periods will be less and shorter. In fact, one parent will stay by the nest. So, more than likely, what you observed is the normal respite from the nest by the parents.

Now, having said this, if something is wrong with the eggs, they are infertile, etc., then the parents will leave the nest and not worrying about sitting near or on it. Nature will not allow the birds to continue sitting on non-viable eggs. The Regal Swan