Using AI for colouring process?

[mikdog]

So, I enjoy animating but one of the processes I abhor is the colouring process. It's not creative at all, just a slog.
I thought it was just me until I saw Tonino's video title lol:
https://youtu.be/CNqdWW4GtrE

I wanted to know if there were a way to eg colour the first few frames, or colour some key frames, and let an AI do the rest based on the reference images?
Surely AI's at a smart-enough level now where it can identify 'oh this is the eye, oh this is the skin, this is the pants' etc. and colour matte fills accordingly?

Would be amazing to not have to do the laborious process every time of colouring, especially when it should be evident even to even a 3 year old what parts of the image are what.

Wondering if anyone has any ideas how we can eg get ChatGPT or some kind of AI model to work for us in TVPaint? If anything I think doing the boring bits is a good use of AI.

Thank you.

[slowtiger]

1. CTG layers got that covered already.

2. You don't need any steenking AI to do this. The technique is well established for about 30 years now, I've used it in Animo and Toons.

[Peter Wassink]

So, I enjoy animating but one of the processes I abhor is the colouring process. It's not creative at all, just a slog.
I thought it was just me until I saw Tonino's video title lol:
https://youtu.be/CNqdWW4GtrE
...
Thank you.

i agree.
And, like i saw Tonino mention in the video, i have experimented quite a bit with EB-synth, but found its not very suitable for regular 2D (its more useful with life action footage.)
so
i 'm curious too.

[Peter Wassink]

1. CTG layers got that covered already.

Am i missing something?
CTG won't color your animation on the basis of a first frame.

[mikdog]

1. CTG layers got that covered already.

Am i missing something?
CTG won't color your animation on the basis of a first frame.

I'm curious too! Maybe I underestimated the power of CTG layers? Wouldn't be the first time I've misunderstood the function of a feature.

[slowtiger]

To be sure I just re-read the manual, it's mentioned here: https://doc.tvpaint.com/docs/colorize-t ... s/examples

[Peter Wassink]

Thats a no.
This example from the manual is a runcycle that happens to allow just a few strategically placed squiggle lines to color all frames correctly.
Its actually a bit of a cheat and not a real life coloring example.
Its good to be aware that this is a way to use it, just be aware that most animation will still require more traditional approach even in CTG.

CTG is a dynamic color system, though its pretty good at deciding where to put the boundaries between colors based on the line drawing, it doesn't color intelligently.
You still need to make sure there is a squiggle correctly placed on each frame.
If you are lucky, or place them cleverly you will be able to re-use and minimise the number of squiggles, but its still you that will have to judge and do this, its not automated.

[mikdog]

Mmm… I also glanced through the manual and didn’t see mention of AI, I think CTG is more of an algorithm that doesn’t understand what it’s filling or what is on-screen. It’s still a great tool and I use it to save time, but it’s not quite Ai in the sense of being able to interpret the image in any meaningful way, and like Peter mentioned, still requires a large degree of user input.

However, if you show a child a character and ask it point out the eyes, pants etc it will and can then colour an image accurately, even as that character moves and turns etc. which CTG currently can’t do.

I feel that this is a huge opportunity to create a tool to fill this gap that if often rote slog work which is why colouring is sometimes farmed out to juniors, because almost anybody can do it by following the rules.

Ai is good at following rules and from what I can gather, is now pretty good at identifying parts of an image.

In my mind, that 1+1 is just waiting to equal 2 with an easy to use tool where the user has a reference pic of their character coloured, maybe in a 4 pose turnaround that the Ai can reference, and based on that, the Ai can colour all the uncoloured linework without the user needing to lay down strokes or identify parts of the image, maybe only later in a clean-up pass in case there are mistakes.

I’m guessing just a matter of time before someone creates it, I’ll be stoked to try it!