It is unfortunately quite telling that much of Christianity cannot state with authority why Abel's sacrifice was looked upon with favor, and Cain's was not. It is true, of course, that the language of Genesis is somewhat more effusive about Abel's sacrifice than about Cain's. Abel "brought of the firstlings of the flock and the fat thereof," while it is stated simply that "Cain brought of the fruit of the ground." On the other hand, Cain as the progeny of cursed Adam would not be thought necessarily more successful than his father at wrenching a crop from the cursed ground festooned with thorns and thistles--"fruit of the ground" might have been precious indeed. And this is all before Noah's blessing (curse?) from God to the effect that the animals would know fear and dread of humanity. Abel's unsuspecting flock cannot be thought to have been particularly hard to manage.
And so the commentators are led often to admit that Cain's bloodless sacrifice should have been perfectly acceptable (especially insofar as the commentators--with indifferent warrant--are reflecting the Pentateuch's rules back onto the start of Genesis.) Some theory must be devised, then, why Cain's offering was not accepted, and this usually boils down to some notion that Cain's disposition about the whole thing was not correct (an especially important consideration when we strive to remember that the two brothers were not in a zero-sum contest.)
A key to a wholesome understanding of the matter can come indeed from looking at how sacrifice as an institution and a practice appears in the rest of the Old Testament. One observation should rear up above all others, if the element of sacrifice is to be viewed properly as bearing on the ministry of Jesus. This is the observation to the effect that the solemnity and (presumably) the efficacy of a sacrifice is largely independent of its size or its cost. As the Scriptures progress, an element develops in which the magnitude of a sacrifice is taken as emblematic of its value, and yet the most solemn of the sacrificial proceedings (think of the scapegoat) involve the fewest victims. There has been no shortage, of course, of Jewish and Christian thinkers who have lamented the tendency of sacrifices to become celebrations of the givers' piety or generosity--when of course such things are beside the point.
We must consider, then, that sacrifice is properly about one of only two things when sacrifice is in its true forms. Sacrifice is about either thanksgiving or expiation. If "thanksgiving" is the matter at hand in the "Cain and Abel" story, then there is no defensible reason to contend that Cain's offering was inferior to Abel's. No serious commentator will contend that Cain should have bartered with Abel for some "firstlings" to slaughter--much thanks that would be, for Cain to give in appreciation for the hard-wrung produce of the soil.
But what about expiation? Other than labored contrivances on the part of some Christians to contend that Cain was somehow expected to know that the fruitful-vine Christ of the Gospels was to be truly foreshadowed only by the shedding of blood, there is precious little abroad in Christianity as regards examination of the tenuous decades after the Fall. Many children's-book representations of Adam and Eve and their First Family are essentially indistinguishable from the last century's museum dioramas of Primitive Man or Stone Age Man, with father and adolescent sons lugging around spears, and with mother and destined-for-motherhood daughters hunching over the fire. In the (usually explicitly anti-evolutionist) Christian representations, the father can be called Adam, and the mother can be called Eve, and yet the same cast of timeless routine overshadows both the religious and the humanist representations.
Yet these are people (if Genesis is really taken seriously) who have undergone great traumas--traumas all the greater in that these people's sufferings have been attended by the imposition upon them of tendencies to the greatest of moral failings. This may seem an impious explanation of the Genesis-generations' evil, but there is no escaping the fact (again, if Genesis is really taken seriously) that the sovereignty of God has overshadowed the moral decline of humanity: "I will put enmity between thee and the woman . . . . thy desire shall be to thy husband . . . he shall rule over thee . . . ," and all because the man "hast hearkened unto the voice of thy wife." It is no wonder that Paul has to wrestle in Romans with the idea of people being "given over" to evil tendencies they possessed already.
All of this was raw and fresh to the generations after the Fall, and they--unlike us at so great a distance--were in a position to remember (or have related to them first-hand) the very words and the very settings in which humanity first came to be. As I have written before, the expectation must be that neither Abel's offering nor Cain's was bloodless. Abel must have seen innocent creatures trodden under his own feet and those of his complacent flock, and Cain must have seen innocent creatures writhing in agony when falling under his earth-splitting implements. This would never have happened in their father's Eden-tending years, and in those blessed years Adam was doing his duty to care for Creation. Every agony of every beast was to be lain at the feet of Adam and his progeny.
This, then, is the magnitude and the scope of the question of expiation in the "Cain and Abel" sacrifice story. Humanity had failed God and Creation and each other, and the picture is presented to us of the first generations of humanity presenting themselves with the picture of expiation. Abel takes unsuspecting and presumably unalarmed creatures, and then he takes them out of the misery of their existence (including disease and old age) to which Abel's sin and those of all humanity had condemned those blameless creatures. Much as Jesus tells us that divorce has its roots in Moses' troubling transmutation of God's desires, so also might we consider that the less-than-instantaneous blood-letting death of Temple sacrifice victims was a post-Genesis artifice. Surely no one imagines that Abraham intended to slit Isaac's throat and let him bleed out--neither need we consider that Abel made his flock's deaths unnecessarily painful (as we remember that the Noahide abstention from blood is still in the future.)
The only reasonable explanation of the acceptability of Abel's offering is found when considering the sacrifices to be expiatory. Abel's disposition was correct, and Cain's was not. This is made explicit in Cain's infamous question of whether or not he was his brother's keeper--the "offering story" is just a backdrop. Our earth is not an ordered system in which we can obtain without guilt material blessings that arise without suffering. The "fruit of the ground" is not ours for the taking--and neither can we believe that our existences from moment to moment are leveraged upon other than the suffering of other creatures. Ultimately, the only logical route of our following the path of expiation is through the teaching of Jesus to the effect that our very lives are the sacrifices.