In V-Ray device selection, make sure to uncheck your 1080ti when using RTX mode. And you will be able to dedicate your 3080ti to your GPU rendering, you will be able to access all available VRAM on your 3080ti here which is great! This way Maya, Max and all other applications will only use your 1080ti for viewport performance. In this case here you can use your 1080ti for Display/Viewport performance, to do that go to Nvidia Control panel, select Manage 3D settings, and look for 'OpenGL rendering GPUīy default this option is set to Auto-select, change it to your 1080ti This way you will be able to have some spacing between the 2 GPUs, and they will run quieter and cooler, and they will boost frequencies higher

The 3080ti is much faster than your other 2 GPUs, I recommend to remove 1x 1080ti from this machine and keep 1x 1080ti + 1x 3080ti

In your case here it is best to stick to RTX mode, so V-Ray will be able to use dedicated ray-tracing hardware on your 3080ti to speed up your renderings

Vray benchmark scores