If "around" 40 students in a class is a surprise, you must have been teaching in language schools before. In my 5 universities the norm was 28-42 students, with classes of 60 - 100 as occasional outliers.

Question: are students entitled to quietly ignore their teacher? Did you ever do so as a university student?

All college oral English classes in China exhibit a 'bell curve' of ability. Most FTs teach solely to the top 10% of students. I tried to teach to the 'mean', but doing so requires significant use of Chinese.

If 1/3rd of your class is responsive, you are likely somewhat entertaining and involved. Moderate your expectations and enhance your enjoyment of the teaching process. Being a caring, attentive and cheerful teacher is often lesson enough to Chinese students who have suffered more than a decade with ppt Chinese 'teachers' droning.