@EHuanglu: My Workflow Unveiled: Text to ...

My Workflow Unveiled: Text to Cinematic Storytelling

Hey friends, I'm excited to share my workflow for crafting cinematic stories from simple text prompts! As many of you know, I often rely on image-to-video tools because it grants me greater control over framing, character design, environment, and lighting compared to video generation.

Here's the tech stack that fuels this process:

- Text-to-Image: @midjourney
- Image-to-Video: @Kling_ai
- Video-to-Video Lips Syncing: #liveportrait

In fact, I used these very tools to create the video you might have seen recently like this video I just created below with this workflow.

So, let's dive into the steps!

Thread 🧵

Step 1: Text to Image

This is where the magic begins! I generated this initial image using Midjourney.

Here's the prompt I used:
[Close-up of Elizabeth Olsen's face. She's wearing a winter hat and standing in an Icelandic landscape. Aim for natural and realistic skin with blue eyes, set against a lush green environment. Imagine a cinematic movie still.]

Tips: When writing your text prompt, ensure the character's face is in a frontal view. This makes lip animation in LivePortrait much smoother later on.

Step 2: Image to Video

Now, let's breathe life into that still image! Here, I turned to Kling to generate video based on this prompt:

[The woman is walking towards the viewer, with the camera tracking her.]

Remember: Keep your initial prompts concise and clear. You can always add complexity once you have a solid foundation.

Here is the video generated with @Kling_ai without voice

Step 3: Magic of LivePortrait

This is where things get truly exciting! LivePortrait takes centre stage:

Source Video: Use the video you generated in Step 2.
Driving Video: Find a separate video where someone is talking directly at the camera, ensuring the framing aligns with your generated video.

Top Tips:
- Crop your driving video to clearly showcase the character's expressions.
- Pay attention to "relative head rotation" and adjust "motion smooth strength (v2s)" accordingly. If your character is talking quickly or singing, lower this value.

Step 4: Polishing Your Final video (Optional)

Consider using a video scaler like Topaz Video AI to enhance the clarity and sharpness of your final video.

Remember: Scalers can't fix glitches or finger-like issues.

And That's a Wrap!

I hope this workflow empowers you to tell your own stories in a cinematic way.

If you have any questions, feel free to ask here or send me a DM – I'm always happy to help within my knowledge base!

Check out another video I created with same workflow.

View Tweet