So when it comes to plant seeds, we're able to combine them to max out the grade... but with special ingredients, we can only average them, right? And there's often no point in averaging; why would you mix an item with a good grade with an item with a bad grade if it ruins the good grade? Does that mean that special ingredients with very low grades are basically just useless/a dead weight, since they'll heavily drag down the grade of any potion?

@Rin

What Schnecco said, and also a common scenario to run into while brewing a potion that requires x2 of a plant product is having two different grades of that plant product, and since they have to be the same grade to add them to the brew, you would use the averager. This happens a lot with the Potion of Evolution especially, because players tend to buy their Evolvulus Flowers from different sellers so they end up with different grades.

My opinion of the averager in general is that it's a glass half full/half empty situation. In addition to what Schnecco said, it also frees up inventory space.