The role that madness plays in The Spanish Tragedy and in Hamlet, indeed in all revenge tragedies, is a vital one; it provides an opportunity for the malcontent to be converted by the environment into the avenger.
In almost all revenge tragedies, the malcontent takes the form of a renaissance man or woman who is confronted with a problem - the deed to be avenged. This crime, and the criminals that perpetrated it, effect that surroundings to such an extent that it is impossible to remain unchanged by them. At this point, the malcontent is addressed with the question that Hamlet asks:
To be, or not to be, that is the question:
Whether 'tis nobler in the mind to suffer
This drives him to emotion and irrationality. Confronted with this cessation of reason, the malcontent is driven into some form of madness. When the malcontent finally discovers the answer to the question of activity or inactivity, this madness ends and reason returns. This return to reason reveals itself in the rather cold and cynical attitude that the avenger takes prior to their death. This change can be seen in the language and style that Hamlet uses on his return; he no longer uses soliloquy, and has ceased all pretence and posturing. The dilemma is that it is wrong to kill someone, but it is also wrong to go on living when the criminal remains alive and unpunished. This desire not to kill someone else leads on to the desire to kill oneself, to escape from the previous dilemma. This, however, creates another problem: people who kill themselves do not get proper funerals, and therefore do not go to heaven. These two dilemmas can be combined and reduced: to live and kill and be damned, or to die and be damned. The outcome is the same in both cases, so the more satisfying choice is to take the first option, to kill and be damned. This dilemma is illustrated by Hieronimo, holding a dagger and a noose; the proper tools for either a revenger to interact with the world, or for the malcontent to leave it.
The second question that the malcontent is faced with prior to his or her conversion into an avenger is about the legality of revenge. Being set in what we can see to be renaissance civilisations, civilisations that have developed laws and rules governing behaviour, the right and correct response to the original crime would be to allow the legal system to take over, and to suitably punish those that have broken the law. In both Hamlet and The Spanish Tragedy, the malcontents attempt to right their wrongs through the law, and, in doing so attempt to sublimate their baser urge for revenge. However, for differing reasons, this fails. In Hieronimo's case it is because Lorenzo continually blocks his access to the king. In Hamlet's case, the reason for not going through the legal system is different; it is because it is the king himself that is the criminal. When this happens, when justice is blocked or corrupted, then the malcontent is left with no choice about what action he or she must take. It is in doing this, in turning away from the Christian view of revenge (God being the ultimate revenger), and taking the more active classical view that constitutes the decision "to be or not to be":
At tamen absistam properato cedere letho,
Ne mortem vindicta tuam tum nulla sequatur
The Spanish Tragedy, 2 i 79-80
This is a distortion of the Christian view of revenge as outlined in the psalms to one that has a more blood-thirsty classical view to it.
The above discussion of reason and emotion doesn't however adequately answer the question. There is no doubt that avengers are to some extent truly mad, but they are however not...