These views are expressed by the panelists and are not the position of the Visual Effects Society
Gen AI tools for VFX
Q: Arpita Venugopal – Anything for environments when you haven’t captured lidar?
A: @donalleniii – Blockade Labs Skybox HDRI generator
A: Laurence Cymet – There is a lot of interesting potential toolsets developing. Some examples include – Luma AI for example will generate a gaussian splat from video, which can derive a mesh. Tripo SR will generate 3d assets from single images using a multi-view generation (won’t be accurate). DiffusionLight will estimate a light probe from a single image, DUST3R will estimate camera positions and mesh from an arbitrary number of images.
Q: Devrishi Chatterjee – Does the depth data bake into the meta data for the filmed exr per shot , or is like per camera set up ?
A: Jim Geduldick – Yes it can be fused into custom metadata headers.
Q: Nikhil Kamkolkar – Is sensor fusing at a software or hardware level?
A: Jim Geduldick – It can use both software and hardware
Q: Tom Funk – How accurate was the LIDAR data? Was AI used in mapping / fusing it with the RGB data? Did you do a nodal offset type of calibration? Is the LIDAR data temporally aligned with the RGB? (same speed, genlock?) I’d built a triscopic rig (primary camera plus micro-4K’s on either side (with wider FOV; genlocked, of course), with nodal offset data for each) to accomplish similar – primarily depth data.
Further dev would include multiple fixed cameras on set. It would be interesting to compare how lidar+ capture data compares…
A: Jim Geduldick – LIDAR is going to be the most accurate way to get information paired with the RGB data and plates. The ideas is that we are able to scale down the need for a multi-point capture to less cameras sources needed.
Q: Pablo Izaguirre – To Jim, you showed a gaussian splat capture of a motorman, in what interface / software where were you previewing it? are you able to export them to a usable format?
A: Jim Geduldick – That particular one was PlayCanvas with 3DGS data. You can download the GIT or use the web interface
Q: Devrishi Chatterjee – AI generated depth models usually have lot of flickering and consistency when analyzing video footage..is it better when shooting live and generating the data?
A: Jim Geduldick – This is one approach. I find working with clean captured footage vs synthetic data is better or being able to fuse them.
Q: Anonymous attendee – As diffusion models improve, how do you see generative video playing into VFX? In the future could a portion of diffusion models be used entirely for VFX shots?
A: Jim Geduldick – That’s still a moving target but it could with very mature tools be assisted in the future. It’s going to be a creative decision choices as well as technical.
Q: Anonymous attendee – Lots of great information about off the shelf ai software tools, curious if you all are seeing larger studios delevoping their own advanced tools for specific tasks that are capable of cinema quality output?
A: Jim Geduldick – Yes there are a lot of custom solutions that have been and are being developed at studios and vfx/animation studios to solve problems and for custom pipelines.
Q: Anonymous attendee – Do you think AI overall and diffusion models specifically will take from certain parts of film budgets and place them elsewhere? In 2030 as diffusion models are even higher quality, how would this disrupt a typical film budget?
A: Jim Geduldick – The budgets are already moving the buckets around in terms of pre pro into post/vfx. Even the budgeting and scheduling is using more AI models to help streamline this beyond the creative and tech tools.
Q: Anonymous attendee – @Jim, are video generation models being used in creating studio content today? How will they be used in the future as workflows improve?
A: Jim Geduldick – Very grey area because of the legalities most are a “no” for using them or a portion of data being secure and trained locally.
Q: Anonymous attendee – What will VFX look like in 2030? How will diffusion models play a part?
A: Minta Carlson – At the rate these tools have accelerated the last 3 years, it’s truly hard to say!
Learning & Career Paths
Q: Raj Persaud – With AI tools becoming more popular, many students entering VFX are eager to use them for the heavy lifting, sometimes at the cost of learning foundational skills like lighting, 3D modeling, texturing, rendering, compositing, animation principles, mastering Photoshop, frame rates, and optimizing assets for game engines. While AI can be a great tool, does relying on it too much weaken critical thinking and problem-solving, especially in teamwork and troubleshooting? Should students focus on mastering these fundamentals instead of taking shortcuts? Do you think VFX programs should offer a dedicated course on AI that explores both its benefits and limitations rather than letting it replace essential industry skills?
A: Minta Carlson – Absolutely fundamentals are a MUST. These models are powerful but not inherently masterful. The artist remains the master and needs to know how to communicate their vision first and foremost.
A: Martin Nebelong – I would say basics become more important than ever. The time we save on doing polishing should be put into studying design and improving our basic art skills imo. That will lead to much more interesting outputs.
Q: Shaun Foster – Thanks for the great presentation! Question: Similar to a question above about fundamentals, and communicating ideas – are their “best in class” exercises that you know of that are great practice for students or even new-professionals?
A: Minta Carlson – There are so many tools, it’s hard to be able to take them all on. I would recommend getting a fundamental comfort level with prompting AI models, but more so than that, work backwards from use cases rather than target tools. That’s how you will get the best learnings, in my opinion. ComfyUI is a good toolkit that is totally opensource and has a learning curve, but gives you a lot of control.
Q: Stan Szymanski – What career paths would you recommend for Senior Digital Artist (of all types) who are entering the job market in a few months?
A: Minta Carlson – Personally – learn how to train/finetune models and learn comfyui at least at the basic level. I see needs for those skills in a digital art context a lot, and they are really great positions with a lot of creativity.
Q: Joey Morelli – Do you all have any career path suggestions for someone like me w/ 25 years of post production experience with edit / color / mograph / compositing?
A: Minta Carlson – I think that there are new careers emerging every day! I recommend trying to master a few tools that align with the workflow goals you have, because you may find a career springs from that.
A: @donalleniii – Use these tools to assist you with creative projects that you share online
Gen AI Toolsets
Q: Michael Gilday – How do you stay current with which tools are worth the time investment? Also do you do any open source in ComfyUI?
A: @donalleniii – I try to create and share something new with these tools everyday, and that has helped me learn the tools. I do use comfy to make my own pipelines
Q: Howie Muzika – For the Text to 3D asset gen, how many maps are generated? Is it just a single albedo, or do you get metallic, normal, displacement, ect?
A: @donalleniii – Its at least 4 maps I think
Q: Joey Morelli – As AI gets better, do any of you feel there will end up being a SUITE / PACKAGE of tools rather than having so many subscriptions with different companies? This currently seems very cost prohibitive for someone like me who is a Sole Proprietor.
A: @donalleniii – Start with the free stuff now so that you get familiar with how you can use it. and yes eventually there may be all encompassing tools too
A: Jim Geduldick – Some of the larger existing tools for VFX , post production and content creation are adding various AI modalities into their features from Nuke to After Effects and Blender you’ll see more suites of tools than just the one offs. Like plugins for a lot of the current workflows there will be gaps still left to be filled on specialized features.
Q: Jacqueline Cooper – Does Eleven Labs allow you to take a video with your voice for instance and enhance it?
A: Minta Carlson – Yes it enhances the voice to be more relatable, outgoing, and less hesitant.
A: @donalleniii – Not that I know of… I used Adobe Podcast for that. It enhances the quality not the performance.
Q: Michael Neely – Do you use generated images to add to your training model?
A: Minta Carlson – Yes, I often retrain using some generated images to refine.
Q: George Tesler – Question for Minta: what platform/software do you use to run your custom-trained model? Do you have to run it locally? Can you import to any could-based web UI?
A: Minta Carlson – I have used a lot of different platforms that work really well! I have spun it up myself using AI toolkit on cloud gpu like modal or runpod, and FAL or Replicate have much easier to use UI if you don’t want all that mess. You can use your model on those services as well, or locally using something like ComfyUi which is great but a little more of a learning curve.
Q: Devrishi Chatterjee – How do you select the model ..do you base on existing models?
A: Minta Carlson – I use Flux Dev because that’s my favorite to train, but I have tried as many as possible!
Q: Devrishi Chatterjee – What’s the comfyUI plugin for guided image generation ?
A: Martin Nebelong – https://github.com/NimaNzrii/comfyui-photoshop?tab=readme-ov-file this one 🙂 I also highly recommend Krita Diffusion.
Q: Winfred M – Please share link for Dreams Tool please
A: Martin Nebelong – https://www.playstation.com/games/dreams/
Q: Devrishi Chatterjee – How do you link Krea to photoshop ?
A: Martin Nebelong – It’s a bit of a “hack”.. involves a windows tool called windowtop that lets you overlay a cropped browser window over photoshop. So not really useful for other than fun speculative workflow examples. BUT you can feed photoshop into Kreay through their realtime tool where you have the ability to screenshare (the option is in the lower left part of the realtime screen). It’s not super useful as a proper workflow, more as a fun showcase. I use windowtop to overlay krea browser window directly over Photoshop and then paint through the browser window. If you want a more useful workflow I recommend the photoshop comfyui plugin (google it) or Krita Diffusion (highly recommend this)
Q: Jack Mcmann – Question for Martin: As AI video tools continue to get better, what do you think these workflows will be like in 2030? Additionally, what new types of content will be created (Will it be easier to create VR content)?
A: Martin Nebelong – Oh 2030.. at that point I expect us to be able to build 3d and see it transformed with 3d ai in realtime. So a simple brushstroke in space becomes a detailed three based on your conversation with the system. A new brushstroke defines the shape of a certain branch, another the shape of a leaf. Suddenly the whole forest adheres to that change. Sounds like Scifi doesn’t it? I don’t think it’s too farfetched actually
Q: Michael Neely – Fiver introduced a platform this week to facilitate artists being paid for their contribution to models with their music, illustrations, etc. Do any of you know of other platforms that provide residual income for artists whose work is being used for training models?
A: Minta Carlson – I believe Bria offers compensation for artists who help build their model
Q: Marty Havran – What are the best programs for 3D character model generation that would be useable for production or a turntable?
A: Martin Nebelong – I’d recommend Rodin 3d or Tripo3d.ai
A: @donalleniii – Rodin and Meshy have been the best so far..but still not at the quality of a trad DCC pipeline
Q: Kyle Carlin – This is more of a response but if you do modeling meshy is a great tool for a starter kit. You can then create around your model.
A: @donalleniii – Agreed! It’s a great start!
Q: Michael Neely – Martin, have you explored ZBrush and SetTop with Krea?
A: Martin Nebelong – Hey Michael, no but I did explore Adobe Substance 3d modeler as input which was quite fun. You can leave krea running and use it to explore new paths to take your 3d model in or even to generate more detailed 3d models by sending the resulting image into tripo3d.ai or Rodin 3d and then import that back into your 3d tool.
Training LoRAs
Q: Winfred M – Does the refinement data need to be similar to the one the model has been trained on? Can it differ?
A: Minta Carlson- I retrain the model entirely from scratch each time, and I will add or remove images based on what I am seeing in my test generations – for example, if a concept is really consistently present and I don’t want it to be, I see if it is in the dataset and I may balance it with a similar concept or remove it.
Q: Lee Sullivan – How many images at a minimum does it take to train this kind of model?
A: @donalleniii – The least I have seen in a LoRA training is 16 images
Q: Bruno Nicolas – How long does this whole training process take in terms of work for you and in terms of computer runtime to train?
A: Minta Carlson – It depends on if you run locally or on a cloud. Most cloud services it takes a couple hours at most. Locally takes quite a lot longer.
Q: Raj Persaud – Since AI learns from the patterns in your art, do you ever worry about losing control over your style? Once the model understands your unique approach, others might be able to generate similar work without you being compensated. Do you see this as a concern, or do the benefits of AI outweigh the risks for you?
A: Minta Carlson – Great question! No one else has access to these models, they aren’t connected to a larger model, they are a separate file that I control completely!
Q: Walt Liu – Hi Minta, When training a full-body character LoRA and a style LoRA, what’s the optimal number of images you’ve found effective for your training data? Thanks!
A: Minta Carlson – Hey there! I tend to aim for 20-30 images for a style and 15-25 images for a character. It depends a bit on how unique the character or style are from the existing model – very unique tends to take more nuanced images.
Q: George Tesler – When training a LoRA model for a character, is that a separate model from your overall style model? Or do all characters + style all go into one single model?
A: Minta Carlson – I usually train them separately because it’s easier to work with and cleaner, but I’m sure it’s possible to train together. You can also use multiple LoRA models (the type of model I train) at once.
Common Gen AI Challenges
Q: Anonymous attendee – The big problem that I have is CONSISTENCY for these gen AI platforms. What are your thoughts about progress in that field?
A: @donalleniii – If we start from a char turnaround, it creates a lot of consitency
Q: Howard Cabalfin – How do we use these tools with clients who require IP security?
A: @donalleniii – I don’t yet.. more for concepting and ideating and planning not final public facing pixels
Q: Zissis Papatzikis – When handling sensitive studio data, are you comfortable using cloud-based apps where data security and location are uncertain, or have you considered building in-house AI clusters using public models? If so, do you think an in-house cluster can realistically match the quality and performance of cloud solutions?
A: @donalleniii – Excellent question! This is an area of active exploration at the company I work at. I use cloud based ai models for Research and dev and for rapid prototypes, and simultaneously build local GPU clusters to run comfy nodes.
A: Jim Geduldick – A lot of the studios, brands, agencies and labels have varying degrees of security protocols of what can and can’t be used with the various tool sets of AI and platforms. Data integrity is a big topic on where data is trained on and where it is stored /processed.
Q: Conor Collins – Don, do you see a delta in the quality of work you make in Comfy vs online tools where the QA has been managed to produce a saas product?
A: @donalleniii – I believe quality is relative to the audience perception… an unpopular opinion. … the delta has not been an issue for me
Q: Michael Millspaugh – How does the state of US copyright law affect clients interest in using these tools in their workflows?
A: @donalleniii – This is still a work in progress. It’s a moving target.
A: Ben Lock – The issue at the moment is use of a model that has been trained on data without the permission of the owner of that data. You should check with your legal department if you need clients to indemnify you if choosing to use Gen AI tools in your output.
Q: Howie Muzika – What are the considerations for working with NDA material with cloud based AI Tools? How do I know the assets I’m feeding any given AI is secure and not being fed to a larger dataset that’s available to the public?
A: Jim Geduldick – This is very much dependent on the project and IP / Data you are working with. You and the team would need to do your diligence on what these tools, their legalities and use permissions are allowed for any given project. Some of these tools are not allowed to be used on projects scaling from feature work to digital likeness (digi double work)
A: Minta Carlson – For training models, you can run it entirely locally without uploading anything outside of your machine, but of course the rules will vary in terms of what a studio or company legal team require. It also requires a good GPU.
Q: Anonymous attendee – Do you guys know if any of these platforms output anything higher than 720p? That seems to be the upper limit of most of them.
A: Jim Geduldick – They vary on the platform. Some of them are limited in resolution, bit depth, frame rates etc.
A: Sebastian Sylwan – I have had very good luck with outputting 720p and then upscaling the results I like.
A: Laurence Cymet – Some models are capable of tiled sampling resulting in some higher resolution images. The common process seems to be iteratively upscaling.
Q: Conor Collins – Has the licensing around FLUX.dev ever come up as a limitation?
A: Jim Geduldick – The licensing issues come up quite a bit across various models.
A: Minta Carlson – Not in my work, but certainly I have heard it’s an issue for others.
Q: Reb Perry – How does it feel as a creator to partner with ai
A: Jim Geduldick – It’s a collaboration with the tools. Most of these tools are not exactly new under the hood AI has been around in visual effects, animation and production for a while now. Most of what I see in using the tools is being able to get ideas and decisions on creative made quicker.
A: Minta Carlson – It feels like using a tool 🙂
Q: Howard Cabalfin – Rather than having the AI create characters on green screen, could it output elements with alpha channel?
A: Jim Geduldick – Some platform outputs have a limitation on outputting alpha channels and the quality needed for compositing and other workflows.
A: Laurence Cymet – Some models are capable of outputting images with an alpha channel, see “Layer Diffusion”
Q: Kyle Carlin – When will we have an energy efficient Ai system where we don’t have to charge so much from resources using it?
A: Minta Carlson – It’s hard to say – but certainly we’re seeing advancements in computing on a regular basis, I’m hopeful!
A: Laurence Cymet – To add some detail here – training to develop a model is the most compute intensive requiring massive amounts of GPU cycles, while running image generation jobs using that model is similar to rendering – whether it happens locally or on a cloud machine, it’s just CPU or GPU cycles usually minutes per image. Some Gen AI jobs are more compute intensive than others.