Character Consistency in an AI Video Remaker: Why Face Swaps Fail and the Six Root Causes We Fixed in a Week
Recasting a film sounds like the easiest job you could give an image model. Take one frame per shot, tell the model "replace this person with the one in the reference image", and hand the stills to a video model to animate. We built an AI video remaking pipeline on exactly that idea. In the first batch, close-ups changed only the actor's hair colour and kept the original face. In two-person shots the change landed on the other person. And even when every still looked right on its own, the finished video flickered from one keyframe to the next like a slide deck.
This post covers character consistency. Between 26 October and 2 November 2026 the pipeline's repository took 144 commits, and the characters were the part we reworked most often. What follows walks through each failure in the order it showed up, with its root cause, the fix, and the numbers from the experiment that decided it. An earlier post, why AI video remakes are so unstable, surveyed the instability of agent-style video pipelines across the market; this one goes down into the hardest single component.
The pipeline
A remake runs in four stages: take the source apart, build the assets the brief asks for, generate shot by shot, and assemble. Character consistency lives in one step of stage three, the stills: for every camera shot we take one frame and edit it into the remade version, which then serves as a keyframe for the video model.
The setup that matters here:
- Every character gets a reference image first: a front-facing, full-body "base look" standing in a neutral pose. Each outfit is then edited from that base, so every costume is the same person.
- Stills are image edits by Seedream. The input is the source frame with subtitles, channel logos and stickers removed, plus a reference image for each person being replaced.
- Video comes from Seedance reference-to-video (R2V). It takes all of the shot's stills plus a depth video extracted from the source clip, which carries the original motion and timing.
- A frame with nothing to change is not redrawn. The cleaned frame becomes the still as is.
Almost everything the video model knows about a character comes from the stills. A wrong face in a still gets animated faithfully; stills that disagree make the video cut hard between keyframes. All six problems below are problems in the still layer.
Failure 1: redrawing on a depth map let the words win
The first design (committed 27 October) had Seedream redraw each still on a depth map: the depth map would hold the composition, text would describe who stands where, and reference images would supply the looks. The idea was to keep the source face from leaking through.
The stills-prompt experiment on 29 October killed it. Over 8 rounds and about 380 stills, each checked by hand, the depth-map design kept the composition in 0 of 18 stills in one set of shots. To the model a depth map is a few grey blobs; what actually drove the picture was the frame reading. In one shot the source shows the curve of someone's back, the reading called it shoulders and chest, and Seedream drew the person from the front.
Editing the cleaned frame directly, and describing only how the parts visible in that frame change, kept the composition in 16 to 18 of the same 18. Later rounds added three rules for the reading: describe only parts that differ from the frame, say colours in the design's own words, and use whole-person descriptions only where the whole person shows. One rule has a very specific origin. A reading said a character "becomes a white young man", and the model turned his hair white. Hair now has to be named by its colour, and a vague hair line is refused and read again. Replayed over 190 hair edits from earlier rounds, the check caught all 27 vague ones with no false refusals.
Failure 2: text alone draws a different face
The same experiment produced a counterintuitive number. Same reading, same prompt, the only difference being the reference image: the swap landed in 12 of 18 stills without references and 8 of 18 with them. References made the edits timid, so we switched them off.
The cost showed up the same day. With text alone each still draws "a face that fits the description", a different one in every shot, and the source face often survives with new colouring. Consistency means every shot shows the person in the reference image, and no description can guarantee that.
References could only come back once failure 3 was understood.
Failure 3: image models cannot see names
The reference-image experiment on 29 October found out why references had looked harmful. The edit instructions pointed at people by their names in the story: "turn X into the person in Image 2". The image model has no idea which person on screen a name belongs to, so it guessed from whatever positions the reading happened to mention. In one shot the change line carried no position at all, and the per-person edit put one character's design on the man standing next to him.
The per-person editing experiment (12 stills that each replace two or more people) shows the mechanism clearly:
| Approach | Fully correct (of 12) | Main failure |
|---|---|---|
| A: one edit, every change in one instruction | 8 | Skips the least prominent person (asleep, in profile) |
| B1: one person per edit, stacked, no references | 7 | Wrong person: given only a name, the model edits whoever stands out |
| B2: as B1, with references | 6 | Wrong person, and references get placed on the wrong body |
When all the changes sit side by side in one instruction, the model can tell the people apart by contrast. Split them up and each instruction holds a single name, so the model picks the most prominent person in the frame.
The fix gives every person a referring phrase built from what the frame shows, which we call a find. While reading the frame, Grok writes one phrase per person: where they are on screen, their hair, which side faces the camera, what they wear. No names, no two people sharing a phrase, both checked in code. The edit line becomes "
We hit the same wall a second time with outfit references. One man's outfit came out as the woman standing in front of him, because the edit described him by his source look and could not tell apart two men seen from behind. Reading the anchor frame for a find first got 9 of 9 outfits onto the right person. It is now a hard rule in the project: any edit that hands an image model a frame with people in it points at them by what the picture shows, and says outright what must not be taken from that frame (face, body, sex).
Failure 4: close-ups kept the source face
With pointing solved, close-ups still changed only the hair. The source face is so sharp in a close-up that when "keep the composition" and "use the reference's face" pull against each other, the model keeps the composition.
The face-mask experiment on 30 October compared four variants over 6 shots:
| Variant | Image 1 | Prompt | Result |
|---|---|---|---|
| B, baseline | Cleaned frame | Current | Close-ups change only the hair |
| M1 | Each face boxed black | As B | Black boxes left in multi-person stills ("keep everything else unchanged") |
| M2 | As M1 | Adds: each box is a face, draw it from the reference | Faces swapped, but laughing, screaming and lowered heads all turn into calm frontal faces |
| M3 | As M1 | M2 plus one line of expression and gaze per face, plus "hair outside the box changes too" | 6/6 right identity, expression and hair |
The boxes come from a face-detection service that returns one box per face, brow to chin. M3 went live the same day, with each face's expression now a required field in the frame reading. It had one side effect: with faces masked, the model printed a brand name on three plain T-shirts, so a rule went in that clothing with no print gets none.
Failure 5: black boxes moved and bent people
A day later, two-person close-ups went wrong. A solid black box hides the head's outline, angle and lighting, so the model guesses, and it moved people apart and changed the angle between their heads. The comparison on 31 October tried three masks over 9 close-ups, measuring drift as the mean difference from the source frame at 24×42 greyscale:
| Mask | Mean drift | Worst two-person close-up | A single close-up, head bowed |
|---|---|---|---|
| A: full black box | 25.4 | 50.7 (one person pushed back, the other shrunk) | 43.5 |
| S: shrunk black box (0.7 × 0.88 of the box) | 23.5 | 28.5 | 27.8 |
| R: Gaussian blur over the full box | 19.5 | 28.4 | 20.8 |
| B: blur inside the shrunk box | 18.0 | 27.2 | 19.0 |
None of the 9 shots kept the source face, so blurring did not let it leak back, and the blur kept head position, profile angle and a hand resting on a shoulder. Drift only compares variants against each other: a change of skin tone or hair colour raises the number on its own.
The shrunk blur went live and then showed its own weakness. With the box shrunk, the forehead, jaw and cheeks stayed sharp outside the blur, and the source face shape with them. The second version that day blurred a larger ellipse raised over the box: 1.3 times its width, 1.5 times its height, centre lifted by 0.05 of its height. We measured coverage against MediaPipe's face-outline landmarks on 88 faces:
- An ellipse inside the box (0.8 × 1.0) covered 42% of the face outline.
- The raised ellipse (1.3 × 1.5, lifted 0.05) covered 99%, with every outline point inside on 97% of faces.
- Rotating the ellipse by the face's roll or sliding it by its yaw (estimated from YuNet keypoints) gained nothing, since the detector's box already sits on the face it sees.
The blur radius is 3.5% of the frame width. A three-person shot taught two edge cases. When the detector boxed no face, the pipeline still sent the masked-face prompt and the model invented a face and lifted the head, so a frame with no box now takes the unmasked path. And only the boxes the service drew count; dark areas already in the scene (a curtain) do not.
Failure 6: every still graded differently, so the video played like slides
With the right people in place, R2V video still jumped between keyframes. Each Seedream edit applies its own colour grade, so the stills of one shot differ in brightness and cast, and the video model cuts between them.
The fix moves each redrawn still's colours back to its own cleaned frame: per-channel mean and standard deviation matched in Lab space, following the colour transfer of Reinhard et al. (2001), which worked in lαβ space; we use Lab. It applies only to redrawn stills, and not when the brief itself changes the style. Measured over 37 shots and 145 stills (134 redrawn):
| ΔE from source | Lightness spread within a shot | Colour-cast spread within a shot | |
|---|---|---|---|
| Source frames (floor) | 0.0 | 10.6 | 1.3 |
| Seedream stills, before | 9.9 | 13.4 | 1.6 |
| Full Lab match (shipped) | 0.4 | 10.6 | 1.3 |
The spread within a shot, the column that produces the slide-deck feel, went back to the source film's own level. In the worst single case an edit pushed lightness from 89 to 123 (ΔE 33.3); after matching, ΔE was 2.0.
The last gate: frame pre-reading and a still check
After those six fixes most remaining errors came from the frame reading itself. Two things went in on 1 November:
- Read every frame before generation. Each shot's still frame is cleaned and read ahead of stage three, and the reading names anything the shot table plainly gets wrong about that frame: a misnamed body part, the side facing the camera, who stands where. In one run of shots the table had read a person's back as their front, and the video model, prompted with that word again and again, turned her round. Now a mismatch makes Grok rewrite the shot's description from its first, middle and last frames; the frame is read again, and if it is still wrong the shot fails.
- Check every still. Each redrawn still goes to a lightweight DeepSeek vision model alongside its cleaned frame: pose, side facing the camera and crop kept, no face or body part added or lost, each redrawn person like their reference, no text. A failing still is made again once, told what went wrong; the verdict is stored on the still but never blocks video generation.
The six root causes at a glance
| Symptom | Root cause | Fix | Evidence |
|---|---|---|---|
| Composition redrawn, a back drawn as a front | Depth map carries too little; text wins | Edit the cleaned frame; describe only visible parts that differ | Composition 0/18 → 16–18/18 |
| A new face in every shot | Text description only | Every redrawn person's reference goes with the edit | Faces drifted between shots with references off |
| Change lands on someone else | Image models cannot see names | On-screen find phrases |
0 wrong of 13; outfits 9/9 |
| Close-ups keep the source face | Sharp source face beats the reference | Face mask plus per-face expression | 6/6 identity, expression, hair |
| People moved and bent | Black box hides head outline and light | Blur inside a raised ellipse | Drift 25.4 → 18.0; coverage 42% → 99% |
| Video plays like slides | Each edit grades its own way | Lab mean/std match | ΔE 9.9 → 0.4 |
My read: the third fix carries the most weight. Masking and colour matching patch weaknesses in the image model. The find phrase corrects a wrong assumption on our side: we had treated a name as a pointer we could pass to the model, and it never was one. The finding that references made things worse was itself a product of that assumption. Change how people are pointed at and it reverses.
Three lessons on experiment method
Change one variable at a time. The face-mask experiment changed only Image 1 and the opening of the prompt; references, model and resolution stayed fixed, so the differences between M1 and M3 can be attributed directly. The per-person experiment is the counterexample: its frame readings came from several versions, the older ones without screen positions, and every wrong-person failure in group B came from those. That discounts the conclusion that per-person editing is worse.
Every rule should name the shot it fixes. Each constraint in the project's notes carries a date and the shot that triggered it: the shot where the change went to the wrong person, the one with no face box, the one where a black box pushed a person back, the one where a back was drawn as a front. A rule that cannot point to its shot cannot be judged when it is time to decide whether it stays. We trimmed the R2V prompt the same way, A/B-testing each kind of cut on real shots first; three test shots went from 1163/892/927 words to 833/648/692, with cleaner lines and cuts closer to time.
n=1 shows "no harm", not "better". Most of the experiments above generated one image per cell. In the stills-prompt experiment the same code run twice gave one shot two correct stills once and two wrong ones the next time; in another shot the same man was identified as one character, then another. With 2 stills per shot per group, the difference between groups was smaller than the noise within a group. So we grade conclusions: with few samples we only claim the change did no harm, and the 9/9 on outfits is a 9/9 from a single run per variant. To compare fine-grained variants we now want at least 4 stills per shot.
Conclusion
Character consistency in AI video remaking is decided almost entirely at the still layer, and each failure traced back to an assumption about the model that did not hold: that a depth map locks the composition, that text can describe a face, that a name points to a person, that a black box hides only the face, and that each edit's colour is neutral. None of these held for Seedream in late October 2026. If you are building a similar pipeline, do two things first: edit the source frame instead of redrawing it, and point at people by what the picture shows. They are the smallest changes and they took us from wrong-person edits to none. Masking and colour matching come after; they fix close-ups and continuity.
For how other tools in this space handle characters, see the 2026 survey of open-source AI video generation tools; for how the image generation tooling itself has shifted, see three years of image generation tools.
References
- ByteDance Seed — Seedream official page (accessed 2026-11-02)
- ByteDance Seed — Seedance 2.0 official page (accessed 2026-11-02)
- ByteDance Seed — Seedance 2.5 official page (accessed 2026-11-02)
- Google — MediaPipe Face Landmarker
- OpenCV Zoo — YuNet face detection
- Erik Reinhard, Michael Ashikhmin, Bruce Gooch, Peter Shirley — Color Transfer between Images, IEEE Computer Graphics and Applications 21(5) (September 2001)
- This site — Why AI video remakes are so unstable: the 2026 landscape and two architectures (2026-09-19)
- Experiment data: the author team's internal experiment records, 29 October to 1 November 2026 (stills prompt, 8 rounds, ~380 stills; per-person edits, 12 stills; reference images, 13 stills; face mask, 6 shots; blur comparison, 9 shots; mask coverage, 88 faces; colour match, 145 stills; outfits, 9 sets)