In Chinese martial arts the title sifu is also an informal one.
It is akin to calling your father 'dad'.
You would not call anyone else's father by this name, only your own.
Calling an instructor sifu denotes a friendly, familiar relationship.
You are both part of the same school and this is your teacher.