I provided a technique that could prevent cracks in both cloth physics and displacement here:
https://www.reallusion.com/FeedBackTracker/Issue/A-crack-free-displacement-Hack-to-add-to-your-Human-Shader

Perhaps the last ideal solution to this would be that Displacement maps have the option to be used as physics weight maps and this would simply need a non destructive brightness/contrast filter/overlay. This way people have the option of using custom maps or linked to existing maps to lighten project sizes and resources.