My drone is drifting forward and left continuously if i counter act its hovering, i calibrated the escs, it didn't help. didn't mount the gps yet. i guess we can't switch between gps and altitude modes with my flysky ct6b.

Yes you can,
Set sw1 for GPS & sw2 alt.hold
Please check your flight controller is it level properly?? <- It can be the cause of your drifting issue.

i tried changing the flight controllers place, nothing seems to change. The flight is not stable at all. I recalibrated the escs and the imu status says basic calibration is not necessary. Kindly help me.

Please check your transmitter's trims and sub-trims

Post some snaps of your quad setup.

 

 
Build video