Hi Steven, great point. I look at it like this. Since vegans are at most maybe 3 percent of the population, most crops are eaten by non-vegans. Additionally, when you shop for plant foods, you are indeed gambling whether or not it caused someone harm in the farming process. We are moving away from harmful models, but as you point out, not quick enough. But, when you eat animal products you can be 100% sure that animals were exploited.
Your particular charge is a common one against veganism as if most crops are not eaten by non-vegans. Also there is the point that in reality most crops grown are given to animals that are then eaten by non-vegans. Its a sticky issue for sure!