Now onto my suggestion. Everything are still sellable, but the price is decided based on what the merchant have in stock. In this scenario, a Iron helmet might be worth it's while dragging to town and sell off if the Merchant doesn't have any helmets at all. However if the merchant have a decent number of better quality helmets, the merchant are still willing to buy the helmet, but to a really low price.
But then, wouldn't it me more profitable to sell a Iron helm to the general store, and some cheese to the armorsmith?
However, I find it hard to think off a system that would prevent you from selling whatever you'd like to any merchant. How would the AI know if it's "inprofitable" to buy something? And how can the AI refuse to buy something? (and how do you tell the player that a particular merchant don't want to buy it?)
Easiest thing I would think, is that: Shopkeepers only buy things that's relevant for their store, hence, the general store wouldn't be your helm, and the smith won't buy your cheese.
How the AI would know whatever something is profitable or not, will most likely be easier to fix with a behind the scenes formula, rather than a dynamic system (Will most likely go faster to code, hence more time on more "important" parts. And if done right, the player wont notice the difference)
I'm not sure what you mean with "And how can the AI refuse to buy something?". Either, some items wont show up when in the "sell tab", or they're greyed out, or simply when trying to sell, the NPC says that he's not interested in that particular item.
About the idea of being able to trade things, instead of getting money.
Trading a Sword for a pig
This have been done in many RPGs before, and it's not really that complicated.
Like someone mentioned, every item would just get a specific value.
So when you want to buy the Steel Sword (Worth 80coins) you can trade in your Steel dagger (Worth 15coins in used condition) a Iron helmet (Worth 20coins) and give 45coins.
Here the player trades a Beer bottle (35gold) for 4 Carrots (16gold) The player could also, if so he chooses, take another 19 gold, or 3pieces of bread + 16 gold.