You could create a subfolder for each color in your inventory and put your go-to color in the main folder.
When you want to pick a color different from your standard color, use the "Wear with options" button instead of the quick dress icon to put the item on (the t-shirt with the gear on it). This will show a list of subfolders, so you can pick your color.
If you have the Wardrobe Premium Subscription, you'll also see a folders icon below the item, which will retrieve the item's subfolders, which you can click to put it on your avatar.