Anti Aliasing problem needs fixing badly!

[David_Fine]

I have read a number of posts about this issue and I am writing again, here in the feature request area, because to me, it is perhaps one of the most important things that needs addressing in TVPaint. I have been working with the wonderful TVPaint for a few months now and happily using the tools to move and rotate elements, only to come to realize that they become blurrier and blurrier as I move through a sequence. I know, they are pixels and they will distort if changed repeatedly, but there are confusing settings that make this problem worse and now, are resulting in extra work for me having to retrace lines I thought were final artwork, but which I now realize are not usable because they have become fuzzy. As I worked, it was less apparent than later, when I examined the drawings.

Just moving an image without rotating or resizing in any way degrades the line quality and makes it more and more blurry. This, with the Anti Alias setting on BEST. BEST, as it happens, is not best. Why? I discovered by experimentation that I need to turn anti alias to NONE in order to avoid any degradation or blurry quality. Thing is, on NONE, when you resize or rotate, it pixelates and looks terrible, so you can't leave it on NONE all the time. SMART seems to do a better job for rotating, but why not BEST? I mean, shouldn't best be best? The manual shows the differences, but does not really explain how and when to use the different settings.

Why are these settings even necessary? Photoshop doesn't have these settings. When I resize or move or rotate, it just always figures out the best way to do it to retain quality. No different setting at all. Multiple rotations in Photoshop will also degrade or blur the image somewhat, but not as badly as in TVPaint. So for me, this is about the essence of the work itself, which is your drawings. Some have said they never ever use those manipulated drawings in their final work and make tracing, or clean up an additional stage, like old times with tracing from the paper drawings. I get that, but, of course, this is not the old times and I want to be able to manipulate my work and use that work. I know, it's not vector, but the quality loss sometimes is not what it should be. It seems to me that if Photoshop handles it reasonably well, TVPaint should be able to as well. I shouldn't have to trace my whole film because of quality loss like this.

Cut brush has the same issues, but different settings again and little info about which to use and when. Anti alias, sure. Optimize? Brush smooth? Which to use for which situation? Again, why not have no option and just have TVPaint do the best job, like in Photoshop?

Sorry for this rant, but I really trusted that one of the beauties of the application was the way I could work with my drawing, changing, moving, adjusting, but now I see that doing that degrades the quality. It shouldn't, or it shouldn't that much.

I wonder if you could comment or tell me if I am missing something obvious about avoiding this problem and being able to work quickly without having to constantly change that setting every time depending on whether I need to move, resize or rotate part of an image.

As a sub note, I realise that the problem is enhanced when the image is smaller in the frame. Would it be good to always work at a higher resolution in order to minimize this problem and then output at the final, HD resolution? Is there any downside to doing this, apart from larger file sizes? Does reducing the resolution down to HD cause any destruction or degradation that working natively in HD the whole time would not?

[Paul Fierlinger]

As a sub note, I realise that the problem is enhanced when the image is smaller in the frame. Would it be good to always work at a higher resolution in order to minimize this problem and then output at the final, HD resolution? Is there any downside to doing this, apart from larger file sizes? Does reducing the resolution down to HD cause any destruction or degradation that working natively in HD the whole time would not?

That's the ticket. My everyday format is 120% of 1080p. When I know I will be getting into some tricky resizing I go for 200% or more. This is why Sandra and I invest on a regular basis into updating our hardware as much as we update our software.

[NathanOtano]

Have you tried the smart mode? When you just move it stays non aliased and when you scale or rotate it uses the aliasing wisely.

I've done some tests and here they are :

For rotate : Photoshop/Smart Mode TVPaint/Original


For scale : photoshop, tvpaint best/smart, tvpaint medium that's not good for animation, original):


Anyway same here, i work with my resolution x2 (then it stays clean when reducing, 4 pixels for 1). It's more handy for cleaning and inbeetweening also, for animating it's not too slow.

[Peter Wassink]

"Medium" is the best option for manipulating lines
"Best"is the best option for manipulating full color images.
the "best" algorithm is more processor heavy and although the surface looks better then with the medium option, it also produces artifacts around the edges, which makes it less ideal for lines.

and indeed its generally a good idea to work in higher resolutions, i like to work double sized (if the greater filesize allows it) to prevent possible rounding errors when going back and forth between full and double.

[D.T. Nethery]

As a sub note, I realise that the problem is enhanced when the image is smaller in the frame. Would it be good to always work at a higher resolution in order to minimize this problem and then output at the final, HD resolution? Is there any downside to doing this, apart from larger file sizes? Does reducing the resolution down to HD cause any destruction or degradation that working natively in HD the whole time would not?

That's the ticket. My everyday format is 120% of 1080p. When I know I will be getting into some tricky resizing I go for 200% or more.

Anyway same here, I work with my resolution x2 (then it stays clean when reducing, 4 pixels for 1). It's more handy for cleaning and inbeetweening also, for animating it's not too slow.

This is one of those workflow tricks that everyone who invests any time in TVPaint learns along the way (either on their own or from another user on this forum ) , BUT speaking as someone who is a TVPaint "evangelist" , an enthusiastic supporter of TVPaint , I must say that it really bothers me whenever I read a discussion like this: it should not be that hard. . SERIOUSLY. (especially for the new user or the person who never checks these forums). ALL of these various tips and tricks should be clearly and comprehensively covered in the User Manual and in Video Tutorials on the TVPaint website. TVPaint has too many things that one has to figure out like a puzzle , guess about how to do it , get lucky if someone else on the forum tells you how to do it .

So for example: if people who use TVPaint professionally understand that to avoid image degradation when transforming/resizing drawings that you must work at 120% - to - 200% higher than your intended output resolution as a normal part of the workflow, then that should be made abundantly clear in the user manual and in the beginner tutorials. This shouldn't be something you only find out about after much frustrating trial and error , after you've found your way into the inner circle.

[Paul Fierlinger]

Maybe we should all put up a virtual collection box or an inside crowdfunder drive to collect enough money to hire someone knowledgeable to write and illustrate a completely new manual from scratch. I would vote for you, David to be the author and I'll put in $200 (100 for each of us) to get it started. Any thoughts from the community? Would you go for it David since you have already completed a lot of pages?

[D.T. Nethery]

Paul, it is on my "To Do" list (to make some tutorials) ... but my To Do list gets longer every day. Unfortunately my regular job takes too much of my time . I don't know when I would actually have the time to focus on just that . (and honestly , there are many areas of TVPaint where I still feel like a newbie -- many aspects of the FX Stack, KeyFramer, Multiplane, Particles Generator ... sometimes with those I am still just stumbling along by trial & error ). The person who I would contribute money to via a Crowdsource campaign to create tutorials would be Mark Chong. (who has of course , done a fine job with previous tutorials for TVP) . I suggested to the tutorial sites Lynda.com and DigitalTutors.com that they contact Mark about hiring him to doing a comprehensive set of TVPaint tutorials for their websites , but I believe they don't see the need for it , because in their view there are not enough TVPaint users ... What I usually end up doing when I am teaching is to create quick n' dirty screen capture tutorials (or I do a video conference with them via Skype with Screen Sharing) for individual students when the necessity arises, because they get stuck and can't figure out some function on their own or from searching this forum (which I frequently suggest to them ... ) , but I have nothing systematically planned out . I realize that creating tutorials is extremely time consuming , but I hope that part of the development process for the upcoming TVPaint 11 release has time and money earmarked to create a new user manual and new tutorials. One thing I try to remember to do is bookmark certain discussion threads here on the forum that have particularly helpful posts (usually from Sven or ZigOtto, to name two of the power users here) , but somehow all those posts need to be collected , edited , and compiled .

By the way, David Fine, I want to thank you for all the thoughtful questions you have posted here over the past several months. I think discussing these issues is going to help the software to get better. However, I didn't mean to derail your topic on the Anti-Aliasing issue into a broader discussion about "we need better tutorials" . (which I am aware is already a known issue to the TVPaint developers ... sorry , I probably should have just refrained from commenting about it again. I don't want my comments to be taken as a "complaint" or a criticism ...)

[David_Fine]

Great responses here. I love this forum for quick and excellent feedback. It took only a few hours to get all these comments. Thank you, D.T., for your comment about my posts. Sometimes I might worry that my many comments and concerns can sound like I am harping on about problems. That's why I always try to be positive too, but mentioning how great TVPaint is in so many ways. Having said that, there are some frustrating aspects and this one is a pretty big one to me. It is also related to your comment about the manual. I totally agree.

So for example: if people who use TVPaint professionally understand that to avoid image degradation when transforming/resizing drawings that you must work at 120% - to - 200% higher than your intended output resolution as a normal part of the workflow, then that should be made abundantly clear in the user manual and in the beginner tutorials. This shouldn't be something you only find out about after much frustrating trial and error , after you've found your way into the inner circle.

This nails it on the head for me. There are some things which should be clear right from the start, but which we discover by trial and error and from the good people on this forum. Manipulating the image is a huge feature of any digital drawing program, so understanding how best to do it to retain quality is vital. Really vital. So I a want to get some specific feedback on that. Paul says that his regular working method is 120% higher than 1080p, but why not 130% or 150%? What are other people's working resolution for better quality, even when final output is 1080p? Obviously higher for camera moves in, but I mean just a static frame.

As regards the manipulation tools, I would appreciate an overview of best working practices for that. So far, my trials with the Transform Tool seem to indicate that when moving an image and not manipulating, BEST is to be avoided always. It degrades the image each time. NONE works, but it sounds like SMART also works as it figures out to use NONE when needed and BEST when rotating or resizing. What is the purpose of MEDIUM? When is that better to use?

When rotating or resizing, is SMART better than BEST? When is BEST good for anything?

For the Cut Brush tool, we have another set of choices entirely different to the Transform Tool. Any suggestions for best practice there for retaining quality when either just stamping or resizing or rotating? Is it better to resize and rotate using the Transform Tool? Many use the brush as a way to pick up part of a drawing, manipulate it and then stamp it down. Sometimes it's just grab, move, stamp down. Which settings are best and in both cases, how can one easily switch between settings to maintain an efficient workflow?

As previously noted, it would be great if these settings worked best no matter what, without options to have to worry about, just as they do in Photoshop. I'm not meaning to compare to a huge app like Photoshop, but clearly that is working with bitmap and so I would expect TVPaint to be no less good in that vital aspect of the application.

NathanOtano, I am interested in your visuals, but you have posted them very small, so I can't see any detail. Can you please repost bigger? Thanks. Here is an image I made which shows what happens when I simply move the same image four times using BEST setting on the Transform Tool. No rotating or resizing. Notice how blurry it gets compared to the one on the left.

[Paul Fierlinger]

Paul says that his regular working method is 120% higher than 1080p, but why not 130% or 150%? What are other people's working resolution for better quality, even when final output is 1080p?

It's all based on trial and error and the line I typically draw with. My pen is at times Size 1.00, 1.50, 1.75, and 2.00. Then it depends on what kind of action goes into the particular clip -- will the people I intend to draw be small, medium, best or smart within the landscape? What also determines my chosen format is what goes on in all the clips that I plan to draw inside a project. I keep learning with every project which size works the best for me and passing this information on to others would be useless unless someone would draw exactly the way I do and that's impossible ( ) I blush, but without apologies.

[Elodie]

Sometimes I might worry that my many comments and concerns can sound like I am harping on about problems. That's why I always try to be positive too, but mentioning how great TVPaint is in so many ways.

David, it's always a pleasure to read your posts. Your comments are always interesting because they reflect some issues "newcomers" may have with TVPaint (anti-aliasing, "static image" layer, to quote the recent conversations).
Anyway, you keep telling things nicely, with humor and cleverness. A big "thank you" for this (Please, don't change ) !

We know our manual is not perfect (is there any perfect manual ?), but versions after versions, we improve it (according to many feedbacks we had) and add more and more contents. Btw, the next manual can be improved more easily than before and may even include tutorial videos.
David (Nethery) put his finger on a horrible truth : making tutorials is terribly time consuming and our team do not always have the time to record them (and there are so many subjects to cover...). I do hope in the near future, we'll have a full-time person working on making tutorials.

To focus on the subject, let me explain the difference between transform and position tool :
- the first one allows you to move an object without respecting full pixel values. So, you can move a drawing of 0,10% of pixels e.g. If you do so, an anti aliasing must be used to calculate the rendering. That's why you must hit "enter" to confirm this change : because you modify pixel values. And that's also why, the more you apply a change, the more your drawing will blur.
- position tool allows you to move an object pixel by pixel. So you never lose quality. That's why you don't need to confirm the change : because you keep your drawing's quality.

I arrive at the bus station : i'll continue within few minutes

[Elodie]

I'm back. So, where was I...

So, the real problem actually is that when you are using Transform tool, you can move an object without keeping full pixels value. If by default, it was always moving pixel by pixel, this discussion wouldn't even exist

---

Regarding custom brushes, maybe the Anti Aliasing names (none, medium, best) could be more precise ?
Respectively, it could be : sharped brush - blurred brush - image stamp ?

[D.T. Nethery]

To focus on the subject, let me explain the difference between transform and position tool :
- the first one allows you to move an object without respecting full pixel values. So, you can move a drawing of 0,10% of pixels e.g. If you do so, an anti aliasing must be used to calculate the rendering. That's why you must hit "enter" to confirm this change : because you modify pixel values. And that's also why, the more you apply a change, the more your drawing will blur.
- position tool allows you to move an object pixel by pixel. So you never lose quality. That's why you don't need to confirm the change : because you keep your drawing's quality.

See now ... I have just learned something new about TVPaint ! Thank you, Elodie, for this explanation. I should have known this , but for some reason I have never realized that the Panning position tool is completely different from the Transform and Warp tools ... many times I need to simply shift drawings over to the left or to the right (or up - or down) and by habit I use the Transform tool, with the understanding that if I use it several times that my line will start to get blurry ... but for a simple shift right-left- or up-down I should always be using the Panning tool .

Panning_Transform_Warp_Tools.jpg (32.89 KiB) Viewed 30321 times

.

[ZigOtto]

... many times I need to simply shift drawings over to the left or to the right (or up - or down) and by habit I use the Transform tool, with the understanding that if I use it several times that my line will start to get blurry ... but for a simple shift right-left- or up-down I should always be using the Panning tool .

I personnally use the transform tool for shifting as well, but only by dragging the mini-arrows to set the amount of pixels to move left, right, up or down,
doing like this, I always get entire pixel values for the offset, (1.00, 2.00, 3.00, ...., 21.00, 22.00, ...),
with AA set to None, no chance to get some blurrish effect, even after several occurences.

[David_Fine]

Wow, "panning". As DT says, who knew it works completely differently than transform? I didn't either. Thank you, Elodie, for the kind words about my comments. I feel good knowing you don't see a post and roll your eyes wondering why you allowed me to have a license. So, anyway, I am still unclear on some things and would like to ask you for more clarification.

So now I understand the pan tool. Perhaps more accurately, the move tool. Pan suggests a constant movement. At least that's how I use the word "pan", as in, panning shot. So to move something, you use the pan tool and the pixels are preserved perfectly with no recalculation.

As regards the transform, when I need to resize or rotate, I am still unclear on the best setting. All I know is "best" is not the best as it degrades. I suppose "smart" is likely the real best thing, but I wonder if you could confirm. When using transform or warp, which setting will preserve the quality the best? I still don't understand why there are choices here though. Who would want anything less than the very, very best that can be rendered. As I mentioned, Photoshop does not give such choices, it just does the job and if you move an item a hundred times, it looks the same as the first because it treats that move as a "pan" without having to make that choice. So simple and logical that it suggests to me that there must be some other reason why TVPaint does it this way with all these options, which I can't understand why anyone wants, because, as I say, I just want the best always.

For grabbing artwork with the Cut Brush, there are a number of other choices.

screenshot_383.jpg (5.96 KiB) Viewed 30292 times

So really, again, my question is, for the best quality when resizing or rotating, how do I use these options and why are there options when I really only want best always? I can't find anything in the manual about this. I suspect it's there, but searching turns up nothing for me.

Thank you!

[Fabrice]

... many times I need to simply shift drawings over to the left or to the right (or up - or down) and by habit I use the Transform tool, with the understanding that if I use it several times that my line will start to get blurry ... but for a simple shift right-left- or up-down I should always be using the Panning tool .

I personnally use the transform tool for shifting as well, but only by dragging the mini-arrows to set the amount of pixels to move left, right, up or down,
doing like this, I always get entire pixel values for the offset, (1.00, 2.00, 3.00, ...., 21.00, 22.00, ...),
with AA set to None, no chance to get some blurrish effect, even after several occurences.

or you can your mouse with a zoom factor of 100%.