According to the announcement for the 2018 event, you can't stay at the beach during the "swim time", in order to make the shy participants feel comfortable. So you are right that it is only "after all is over" that you can spend the rest of the day there. During the event you may hang around the "festival area" where clothes are required.
To me, this event seems to be quite prudish, and this impression is fortified by the video from Daily Telegraph. Nudity only to an extent that nothing may be seen - this gives the wrong message IMO.