There is a possibility that the swans did not like the leaves. Do you know what tree they might have originated? Some leaves and straw, such as pine, has a very caustic sap/resin that can harm bird feathers. But, swans do use leaves in their nest, but prefer more straw or reed/grasses. The leaves, being wet, may have also attracted pests such as ants, mites or other insects that are attracted to rotting vegetation. The smell alone from rotting leaves may have also acted as a deterrent. You might go out and remove the leaves in the event the swans return and see if they might use the nest if they do return to the area.

Also, if the swans think that someone or something has interfered with the nest, the birds may look elsewhere for a less open/obvious site. The Regal Swan