Mandatory and optional participation are observed in E-R models between 2 different entities.
In the mandatory participation, for every instance of entity A, there must exist an instance of entity B and vice versa.
An example of Mandatory participation would be relationship between mother and child. The child entity would exist only if there were a mother and similarly a mother would exist only if there were a child
In optional participation, it is not necessary for all the instances of the entity to participate in a relationship. It may be that the number of instances participating for a particular entity may even be zero.
Optional participation is good for depicting relationships that are not compulsory or may be temporary i.e relationships that may change over a period of time.
An example of this is Entity A is married to Entity B. Not all instances partake in this relationship and the relationship may even change over time.