So I configured a FAST Search Connector. But It didnt work quite allright so I tried to delete it. No luck! There were no options in Central Administration to delete a Search Service Application after creating it.
After a quick search on the internet I find that you can remove the SSA by using
stsadm -o deleteconfigurationobject -id *GUID*
command. This command works by using object id(GUID). In order to get the object ID; the GUID of SSA, I just copied it from IE, central administration Search Service Application URL.