The answer lies in
conservation of energy.
A nucleus will decay if there is a set of particles with lower total
mass that can be reached by any of the above types of decay process or
simply by fission, a process in which a massive nucleus splits into
two less massive ones. Alpha decay is also a type of fission, common
because the alpha particle is a particularly low energy arrangement of
two protons and two neutrons.
The mass of a nucleus is determined by the sum of the energies of
all its constituents. The energies of the constituents depend on their
masses, their motion, and their
interactions.
In chemistry we talk of the energy levels, or states of
electrons,
in an atom. Electrons fill energy levels because
there is a rule of electron behavior (derivable from the quantum
theory) known as the
Pauli Exclusion Principle.
This principle applies to all
fermions.
The principle states that only one electron can occupy any possible
state in an atom. Each energy level has only a fixed number of states
in it -- and can contain no more than that number of electrons.
Let us consider a Helium 4 nucleus. The two protons occupy the two
lowest possible energy states for protons and the two neutrons occupy
the two lowest energy states for neutrons. This fills the lowest
energy levels for both types of particles.
Their interactions are such that the mass of this nucleus is less
than the mass of a helium three nucleus plus a free neutron, so it
cannot decay into that combination.
If one of the neutrons could beta decay it would produce a Lithium
4 nucleus (3 protons and one neutron) plus an electron and an
anti-electron type neutrino. But the sum of these masses is greater
than the Helium 4 mass so this decay is forbidden too.
But why is Lithium 4 more massive than Helium 4 even though a free
neutron is more massive than a free proton?
The reason is that a third proton cannot be put into as low an
energy state in the nucleus as occupied by the second neutron. Just
as for electrons in an atom, the lowest energy level in the nucleus
has only two states for protons and two states for neutrons.
The pattern of stable nuclides thus consists of nuclei with roughly
equal numbers of protons and neutrons (or a few extra neutrons because
electrical repulsion between protons makes the energy levels for
protons slightly higher than the equivalent levels for neutrons).
Nuclei with excess protons decay via beta-plus emission while nuclei
with too many neutrons decay by beta-minus or electron emission.
Beta decay and gamma decay often occur as steps in a chain of
radioactive decays that begins with the fission of some heavy element.
The fragments which appear after this fission have the right number of
neutrons and protons to be some nucleus, but they are not arranged in
the right energy levels because they just split off in whatever
arrangement they happened to find themselves in. Secondary
transitions in which a proton moves from a higher level to a lower one
with emission of a photon are then common, as are beta-emission
transitions in which either a proton or a neutron moves to lower
energy level (and changes type). Only when all the fragments have
settled down to their lowest mass (energy) forms does the decay chain
end. Different steps in the chain may have very different half-lives.