Previs to Post: The AI Tools to Upgrade (and/or Automate) Your Workflow

The stage visuals for Madonna’s recent World Celebration Tour were created by artist Sasha Kasiuha using Runway. Cr: Runway/Sasha Kasiuha

READ MORE: 10 Best AI Tools for Video Editing & Production (SproutVideo)

AI is reshaping content creation at every stage bringing efficiencies in production by automating the more monotonous tasks and even helping to conjure new forms of storytelling.

As Conner Carey at SproutVideo puts it, there are “plenty of impressive and equally underwhelming platforms and tools to try out.” Here is a (far from exhaustive) list of AI tools that professional content creators are using today from previs to post-production.

3D Asset Creation

Meshy is a text-to-3D generator for 3D asset creation with a free version to get you started and paid tiers if you want to progress.

“You can choose from a number of different art styles ranging from realistic voxel, which is kind of that Minecraft – Lego block look, all the way down to realistic hand drawn and cartoon line art,” explains Tim Simmons of Theoretically Media. “I quickly generated a 3D spaceship without giving it too many details.”

Finessing designs further might require more 3D animation skills or at least the intuition to port the work in progress into software like Adobe Mixamo, a free auto-rigger for 3D characters.

Simmons does exactly this, showing how, in just a few minutes, a character can be completely rigged with total camera control.

“Meshi provides a really cool solution to generate assets, and then bring them into a 3D software package and start playing around with it.”

Transcriptions & Subtitles

Producing transcriptions and subtitles with AI is nothing new. Social media platforms have been using AI to automatically generate video subtitles for a couple of years.

For getting through volumes of foreign language content, Alex Depew at Adorama recommends AI translation platform Blanc. “Perfect for people who have courses, documentaries, educational information, and many other scenarios,” he says.

READ MORE: AI Tools That Can Upgrade Your Filmmaking (Adorama)

Subly is a transcription and subtitle tool with language translation for more than 70 languages. It can produce a video or audio file transcription within minutes, with a built-in editor for adding customized subtitles to existing video files.

According to Carey, who tried it out, the Subly editor provides additional options that allows users to differentiate between multiple speakers and add additional text to videos. Once you’ve finished adding subtitles to your video, it can be downloaded to your device for distribution.

Auris is an “exceedingly affordable” option compared to Subly, Carey reckons. It offers basic AI transcription and subtitle generation with language translation in 27 languages. While Subly is more fully-featured and faster, Auris provides 30 minutes of free transcription each month.

AI Voices and Voice Cloning

Dozens of platforms produce AI voices to various degrees of realism. Some platforms even make it possible to clone your voice. AI voices make it possible to reproduce the intonations and inflections you use when speaking.

Eleven Labs generates speech in voice-actor style from text, using a model trained to understand what the text says and adjust delivery accordingly. Users can select a pre-existing voice or create a model from a sample they provide. The company recently released an AI Speech Classifer feature, which analyzes uploaded tracks to identify if they contain audio generated using ElevenLabs.

An enhance-speech tool in Adobe Premiere Pro cleans up audio recordings and “makes poor-quality audio sound like it was recorded in a studio,” according to Rebecca Whitaker at Vidyard.

Phil Ranta, CEO of social platform Spree, told Business Insider that while voice-cloning tools could be dangerous for the potential scams that could result from them, they also make workflows more efficient for content that requires a voiceover.

A cheaper comparable alternative is Murf, which “holds its own with natural-sounding AI voices and an easy-to-use interface,” Carey says.

Descript is an audio and video editor that’s meant to feel “just like a word processor,” Andrew Mason, the company’s CEO and co-founder, told Business Insider. The tool transcribes audio and video and syncs the text to the sound so it can be edited like a Microsoft Word document.

Descript also offers features like voice cloning to create additional audio for a track by typing, automated removal of filler words, and a “regenerate” tool that automatically substitutes bad audio to make it sound cleaner. Creator economy insiders said Descript has significantly reduced their time editing video and audio.

READ MORE: 13 innovative AI tools that are changing content creators’ work processes, from ideation to production to editing (Business Insider, subscription required.)

Video Background Addition and Removal

Removing a video’s background with AI is a feature of many video generators and generally provides impressive results with low effort and a small budget.

“The main challenge with this feature is precision; therefore, look for a tool that allows you to fine-tune the results. It’s a red flag if no options are available for improving the final result,” says Carey. “An effective background removal tool will allow you to manually fix any mistakes the software makes.”

Runway, for example, includes a AI greenscreen that “currently proves to be the most finessed and practical feature for day-to-day editing needs,” rates Vidyard’s Whitaker. You start by creating a mask for the object. It offers multiple features for customizing the mask, allowing you to make precise changes to edit the new background, according to Carey who has tested it. Runway also includes an Inpainting tool for removing background elements.

Adobe Generative Fill, available in Photoshop, takes your existing video footage and extrapolates it. It can creates backgrounds or adds items within a shot which This feature is “game-changing for budget-conscious video professionals,” Whitaker says.

READ MORE: The Best AI Video Editors for a New Era of Video Creation (Vidyard)

“Generative Fill has made thumbnail generation twice as fast and 10 times better once you learn how to use it correctly,” Ranta told Business Insider.

Adobe Firefly includes the ability to “inpaint,” allowing users to generate a specific part of an image and replace the original using AI.

Storyboarding

Boords.com is designed to streamline the storyboarding process by making it easier for teams to plan shoots by creating and sharing storyboards efficiently. It’s a space to visualize ideas, iterate on concepts.

Krock.io is an all-in-one remote collaboration review and project management platform aimed at video creators in creative agencies, design companies, and freelance pros. It offers features such as online proofing, audio annotation and AI storyboard creation.

Storyboarder, made by German production company FYNAL, uses AI to automate storyboarding. Cartoon Brew’s Amid Amidi explains how users upload a PDF of their screenplay to the cloud-based platform. The software then analyzes the script to create a shotlist. The user then reviews the shot descriptions, and selects some general settings for each shot. The program then generates the images, which can be further refined for style and content.

READ MORE: Tech Bros Want To Replace Storyboard Artists With Artificial Intelligence (Cartoon Brew)

Storyboard Hero, from Singaporean tech company Ilathys, is for video agencies and content creators. It offers a hand-drawn image option. Plus, you can export polished PDF storyboards complete with voiceovers to simplify video planning and communication.

Creator Millie Pham appreciates that it continuously updates, always adding new features and improvements to streamline the creative process and enhance collaboration. “However, it’s not all smooth sailing,” she says. “There’s a bit of a learning curve, so you’ll need to invest some time to really get the hang of it.”

READ MORE: Storyboard Hero Review 2024: Best AI Storyboard Tool? (Millie Pham)

SDR-to-HDR Conversion

AI can now let you convert SDR color space video to HDR via a new feature for all of NVIDIA’s RTX GPUs. While this new feature might mostly be aimed at improving your gaming and YouTube-watching experience, it’s another sign of what AI is currently capable of and what it may be in the future,” explains Jourdan Aldredge at No Film School. “Watch out for how these new AI-powered HDR features might make their way to other areas of content and media, as AI HDR upscaling is shaping up to be the latest frontier for AI to conquer.”

NVIDIA is also developing AI techniques for its Deep Learning Super Sampling system to help improve frame rates and image quality in games and other forms of media.

“We’ve also seen NVIDIA launch an eye contact feature that can (very creepily, we might add) use AI to make it appear like the user is making eye contact during video calls,” Aldredge says.

READ MORE: You Can Use AI To Upgrade SDR Content to HDR with Nvidia (No Film School)

AI Video Enhancement

Footage too grainy or too low-resolution? There’s an AI for that. Topaz Labs provides photo and video quality enhancement that can help make, for instance, 720p footage stand up next to 4K.

“The product is designed for professional video editors, but it’s also useful to upscale grainy Zoom recordings and stabilize videos captured on smartphones,” says Vidyard’s Whitaker. “The results are impressive.”

SproutVideo’s Carey also rates Topaz Labs tech the best around, “because it focuses on doing a few video enhancements really well: deinterlacing, upscaling, and motion interpolation.”