Start a new topic

GPU Rendering Animation Slowly Consumes Memory -> Crash

Hi,


I'm LOVING the GPU rendering, it's really enabled a lot more flexibility and quality from my output. Thanks for developing this!


KS 9.3.14 Pro

Windows 10

RTX 3080 GPU with 10GB video memory.


I'm rendering long sequences of animation (4000+ frames), and it keeps crashing part way through. Using MSI Afterburner to monitor the GPU, and in the render window of Keyshot, I can see the video memory footprint of the render is increasing with each frame produced. About 6MB per frame. (This is for 1920x1080 images)


Since I'm rendering thousands of frames, my video memory fills up and it crashes the render after ~1000 frames, as it's consumed an extra 6GB by that point. The crash produces no error code, it just appears to be a completed render until I go into the folder and find missing frames.


Additionally: I noticed that if I turn on GPU rendering, it consumes a chunk of video memory, but when I turn OFF GPU rendering, it doesn't release that memory. Reloading the file or opening a new blank scene doesn't release it either. I have to exit out of KS and restart in order to get the memory back. As you can imagine, this has the potential to crash large scenes unnecessarily.


3 people have this question

Hey Seth--I'm having the same issue on KS 10.1.79. It's a bummer after I developed custom cloudy dielectrics and metallic paint for the GPU mode. (Metallic paint material is bugged on GPU and causing crashes too!)

I'm on a 8gb P4000. Same issue all though I noticed it was running out of memory even just adding camera animations. My native scene consumes about 2.5gb after restarting KeyShot.

Hopefully we hear something soon!

Here's a annotated graph documenting the GPU memory consumption issue. KeyShot must be restarted to clear the GPU RAM.

To reproduce create a new animation and run it on the GPU. Small, low-res video sizes with low sample counts reproduce the issue most frequently. Frame images and/or meta data pile up in GPU RAM. This highly limits the amount of frames that can be rendered.

I'm on Windows 10 (19042.804) with a Quadro P4000 (27.21.14.6140) with KeyShot 10 Pro (10.1.79)

image


I'm having the same issue. Keyshot tech support blamed it on the latest NVIDIA drivers but when I switched back to 10.0 the problem went away. Lots of other interface bugs in 10.1 as well.

Same here!

I have gtx 1070 8Gb which is not rtx but renders very fast compared to my 6700k CPU. But, it leaks memory! Even though the scene is about 3Gb, when it renders animations it crashes after some 500-700 frames whith the warning that GPU doesn't have enough memory. It happens starting from Keyshot 9. So it is a really pain in the neck rendering animations.

The interesting fact that my laptop with gtx 1060 6Gb doesn't have the memory leak. Even though it renders much slower it doesn't crash.

I just installed the latest product driver 461.72 for my Quadro P4000, same memory leak issue. 

I wish KeyShot would advise which driver version should be used, the 441.66 that is recommended here by Luxion is no longer available from Nvidia. 

We where able to isolate the issue you are all having with the GPU memory leak.

We are releasing a hot fix very soon but to help all of you out I will post the hot fix update here for you and you can try it out to see if the issue is still present:


Make sure to perform a fresh installation of the software to remove any traces of the previous version:


here is the link to the hot fix: https://we.tl/t-aP6NGkpbbR 


Please post your feedback here if you are still having any issues.



Thanks,

Justin 

QA Support manager

Luxion Makers of KeyShot


Hi Justin--Thanks for the hot fix. I do recommend everyone deactivate their license as part of the uninstall, I didn't but fortunately was able to use my serial key to reactivate.

With GPU on, I scrubbed around on the timeline and the GPU memory held constant. Then using the animation preview render for ~460 frames there was some memory use increase as seen below, but much better than before. (I was away from the computer while this ran, not sure if the 0.9GB increase is to be expected.)


image

image


I deactivated the license, uninstalled, and installed this hotfix version. 

Now it gets stuck on the "Reactivate license" dialog.


image


What can I do to reactivate my license?

Hi Anindo--Maybe try Install a new license and use your license key?

Hey Anindo,


Just sent the License file through the support ticket you sent.

Let me know if you still cant activate.

Hi Justin, 


Thanks, the license file you sent has worked, thank you very much!



McIvan,  

Yes, I tried that by following the support FAQ. That too didn't work, unfortunately. So I raised a ticket, and kudos to the support crew for a super-quick response with a new license file! 


Now it's time to explore whether this hotfix has solved the GPU out-of-memory issue. 


Regards

--

Anindo

I'm back to report that while the 10.1.81 hotfix does slow down the GPU memory leak, the issue is not solved yet. 

Now, instead of crashing the GPU in an hour, my test scene crashed in 4 hours 39 minutes. 

image



This scene has no geometry nodes, not even a bump map, and has less than 400 triangles:


image


The GPU memory at the start was 3.1 GB, stayed there for around 4000 samples, then it crept up to 6 GB and then crashed. 

I would be happy to send over my Keyshot file and any settings if my doing so helps diagnose the issue and resolve it. 


I also was examining the GPU render yesteday. It started from 2.8GB and went to about 7.9 (8Gb 1070ti).

I launched GPU memory graph and it popped up very quickly then settled at about 95% of GPU memory and then started to grow very slowly to 97% and even get back to 95% and then again to 97%. It managed to render the animation of 181 frames without a crash, even though prior to the fix it used to crash at about 80 frames in this scene. After the render finished it didn't clean up the GPU memory, only Keyshot restart made the memory reset. It is the most serious issue for my since GPU render was implemented.

Thank you all for testing this.

We also continued to test and have an updated 10.1.82 that we released this morning.

head to: https://www.keyshot.com/resources/downloads/ and use the full installer.

Same as before make sure to perform a fresh installation of the software.


How to perform a fresh install:

  1. Uninstall KeyShot
  2. Make sure to delete the KeyShot  resources folder
    • note: move it to the desktop if you wish to back up your resources
    • Windows Default:
      • C:\Users\UserName\Documents\KeyShot or
      • C:\Users\Public\Documents\KeyShot
      • Delete C:\ProgramData\Luxion\KeyShot folder
      • Delete C:\Users\%username%\AppData\Local\Luxion\KeyShot folder
    • Mac Default:
      • /Library/Application Support/KeyShot
  3. Download and Re-install KeyShot using the links below.
    For this step, you’ll need the full installer(not the update installer).


Again thanks everyone for helping us figure this issue out.


1 person likes this
Login to post a comment