I provided this info and more in another FT post and tech support stated it was fixed in V 7.7 - I don't know what was fixed, even the update notes did not state this Iray problem was fixed.

This issue has been ongoing since Iray came out. It seems to be a mesh resolution limit that is not being accounted for in the VRam when using tessellation.

I figured out a solution which I think should be a tip for the manual talking about texture tessellation and displacement when using organic models.

It seems I'm able to get displacement just having the model use smooth mesh by subdivision and use a displacement texture but don't use tessellation (keep tessellation at 0 or keep it to the bare minimum) This would be a recommendation for organic models. I've tried subdivision on its own and works fast in Iray.. I tried texture tessellation on its and its very slow trying to get the same level of detail and I'm using an RTX 2080 Ti! but using both with high tessellation settings likely makes the mesh size extreme.

So the tip may be this "when using Iray use smooth mesh by subdivision, set tessellation to 0 or very minimal or the model resolution will become too big for Iray to handle. Iray may need to allow for larger mesh or project triangle size. I has taken me a looooong time of trial and error to figure this out for organic models that need more detail and it should be a tip for both content creators and a tip in the manual for general users.

Ultimately there should be some way to see what the additional triangle count is when using tessellation -maybe artificially change/override the current model count doubling it with each tessellation level ... maybe the bug or lack of visual cues is that either Iray has a max triangle limit or triangle count isn't accounted for in the video Memory stats and gives a false impression to the user that the Video card isn't being fully utilized when in fact it is.

The information below is how I arrived for the solution above.

Tessellation and subdivision works fine by themselves although refreshing with tessellation itself is very slow and could be the problem. When using subdivision with tessellation my particular project triangle count is 214376

I tried baking the displacement into a new model which came to 48 MB (856744 project triangles) using 3.1 GB vram and imported it into iClone and loaded Iray.. it crashed but perhaps it was because I wanted to see if additional displacement using the same displacement texture would all me to push the quality a bit further (It didn't, I'd have to use smooth mesh by displacement again to prevent tearing. I opened up the project again, deleted the displacement textures, set iClone to min mode and made sure nothing else was enabled )unchecked smooth mesh, displacement, all reflection settings. I used the same 8K textures It was blazingly fast! no crashes. Still need to get subdivision working with tessellation.

Also in minimal mode with veiwport preview Iray doesn't refresh with any new angles but works properly when adjusting camera angles.