The PolyDoctor SOP was created to take polygon surfaces and analyze and repair for simulation by the Cloth Solver. Clean geometry is essential for a high-quality cloth simulation. You can use the PolyDoctor SOP to fix all kinds of issues with polygonal geometry. It can detect a polygon with multiple vertices indexing the same point causing nasty wing edged geometry. The PolyDoctor SOP will detect these cases and it has automatic fixes for most of them. It will detect self-intersecting regions, which will cause issues for the Cloth Solver. It will also remove very small polygons and a host of other issues with polygonal geometry.

Increase the number of substeps on the solver until the problem goes away.

As a general rule, the thickness should be 1% of the size of the polygons on the cloth. You can use the parameters on the Visualization tab to check if there is a Thickness or Penetration problem.