Nope. Not how it works. The restrictions are Mon - Fri so nobody pays at the weekend. Your permit covers you during the restricted times. You knew this when you applied for it. You have no more right to park your car on a public road than anybody else.