When a person is harmed because of the religion that he follows it is religious violence.
When those who follow one religion talks ill of the others who follow a different religion, when they are ill-treated or neglected, when no one talks to them or they are boycotted, it is religious violence.
On the other hand there can be people who do not believe in any established religion. They are harmed when they are discriminated.
Not tolerating people believing in other religion, criticizing their practices and rituals, is a way of harming them.