Do you think its fair that I have been suspended?

I work for a charitable run nursery, we have a deed of trustees. I asked my manager if I was able to swap shifts with another colleague, this meant that I was working just the morning and my colleague was working the afternoon shift. The manager was happy to do this so the shift was swapped. However when I returned to work the following day, I was met by one of the trustees with a letter, she told me to read the letter and then leave the premises, this was done in front of other colleagues. The letter stated that I was suspended from work due to leaving work when I was supposed to be working. I left the premises, then later that day I received a phone call from the colleague I swapped shifts with asking if I was ok, she informed me that the afternoon I left work due to swapping my shift, that another colleague phoned up the trustee and told them I had walked out of the nursery. (may I add this colleague is the daughter of the trustee). Where do I stand legally, I feel very upset by what has happened, and also the manager was not informed about any suspension of her staff.