In this post, I will be operating under the assumptions that there is a literary dependency between the "synoptic gospels" and further that Mark was written first, with both Matthew and Luke using it as a source. This is known as "Markan Priority". I won't be defending those assumptions, simply using them. Also, I will be using the traditional names of the Gospel writers, though they were originally anonymous, because saying "the gospel attributed to Mark by later tradition" gets old pretty fast, and so know that's what I mean when I just say "Mark".
Secondly, this will be extremely useful for looking at the example that I am going to use to illustrate my point (the NIV is a little clearer though). Scroll down a bit for the english versions, and look at the key at the top to figure out the color scheme. That site is useful for any look at the Synoptic problem, I would recommend it.
Redaction Criticism is a way of building on the findings of the Synoptic problem in order to discover more about the goals of the people who wrote the gospels. Specifically, it looks at what the gospel writers have changed from their source materials, and then tries to find out if there is any sort of reasons the author makes the changes he does. When you look at parallel passages from the gospels, you can see that there are similarities and differences. Redaction criticism tries to answer why the differences are there.
All of the gospel writers have certain themes that they want to bring out in their accounts of the life of Jesus. A cursory reading of Matthew will show that he has a huge interest in tying Jesus' mission to fulfilling old testament scriptures, portraying Jesus as something of a second Moses. Looking at John we can see his strong desire to show that Jesus is equivalent with YHWH and how he structures his writings to demonstrate that. Mark has been described as a "passion narrative with an extended beginning" and we can see when we read it his goal to show that Jesus' crucifixion is an integral part of the plan all along.
Sometimes all these things can be seen more clearly if we look at how an author is using his sources, to see if there are things he wants to bring out more clearly, or things he wants to leave out because they don't portray what he wants to portray. I was going over this because of a conversation with somebody who mentioned James as a sceptic in the bible that later believes in Jesus, based on the fact that he is seen in Mark doubting Jesus, then is later a major church leader.
A major theme in the book of Mark is that Jesus is misunderstood by those that are closest to him. Constantly his disciples are missing the point of what he is saying, and seem clueless to what his actual mission is. As the reader we are priviledged to know what is happening, and we can see how clueless everybody else is. A great example of this is when Jesus family comes to get him, in chapter three (20-21, 31-35 is what we are looking at). His own family comes to get him because they think he is out of his mind. Jesus uses this incident to show those around him what it means to be part of the family of the kingdom (what we would later call the "christian family"). At this point you should probably follow the link at the top and read the actual texts in all 3 gospels.
In the context of Mark, the section on his family's unbelief are key for the overall structure of the gospel. This passage, however, is one that is taken over by both Matthew and Luke (triple tradition), and it is interesting to see what they do to it in their versions (again, I am assuming that Mark was written first, and the other evangelists had access to it). I am not making any judgements on the historicity of the passages, simply looking at what the text says, and what the literary reasons for the differences might be.
You will notice if you followed the link that neither Matthew or Luke have included the bit about his family thinking he is out of his mind. It is skipped over while the rest of the section is very similar. I find this to be quite interesting. Why would that particular verse be skipped in this passage? What is it about his family not believing in his mission that would cause the other evangelists to omit that bit of information? Without reading the Markan account, could you infer the reason that Jesus' family was there in Matthew and Luke?
I think that the reason for this omission has to do with another part of the Jesus story, this one that is found in Matthew and Luke, but not in Mark. I am referring, of course, to the Virgin birth. Because Mark's gospel begins with Jesus as an adult, we have no way of knowing whether he knew of the interesting circumstances of Jesus' birth and chose not to mention them, or if he hadn't heard that story. Matthew and Luke relate the same basic facts about Jesus' birth, but with little overlap as far as the specifics are concerned. Both of them relate the story in such a way as there is no question as to whether or not it is a very miraculous occasion.
I think the fact that the birth narratives play such a major role in their gospels give Matthew and Luke a strong incentive to leave out any mention of his family (his MOTHER no less) thinking he is out of his mind. They both go through great pains to show how Jesus' mission and calling were set since before he was born. Luke in particular shows Mary's joy in the specialness of her son, "treasuring all these things in her heart". How would it look if she then, shortly after he began to fulfill his calling, came to get him because she thought he had gone crazy? That section just doesn't fit in with the narrative that Luke is constructing. Same thing goes for Matthew. He shows clearly how Jesus' birth and childhood fulfilled a bunch of OT prophecies, in a way that it isn't possible without miraculous intervention. In both of these cases, it wouldn't make sense to have the kind of doubt from his family that we see in the Markan account.
I think those are some specific reasons to leave out that verse, but there are some more general reasons as well. They have to do with the overall portrait that the Gospels are trying to paint. I have given some suggestions to this effect above, but I feel it's much more enriching to actually try to discover such things for yourself, so I will offer no more conjectures. Read through the Gospels, trying not to let them influence each other, and look for what makes each of them unique. Take a section found in 2 or more of them and compare them. The birth narratives in Matthew and Luke are a great starting point, just don't skip the genealogies.
Look at the Gospels not in terms of the history they contain, but in terms of literary works, written to convey a message by authors who were artistic in their presentation of their material. It can make for a very different experience, and you may learn a lot from the experience.