Can you learn self defence by itself?

No.
We cannot teach self defence as an independent concern.

Self defence is part of the taijiquan syllabus.
Without a wide variety of inter-related kung fu skills, self defence simply will not work.
We teach self defence in the manner in which we approach combat - 'internally'.