Hello RCI,
I am building a quadcopter with these specs:
Naze32 10DOF
DYS BX2212 980KV motors
20A Blheli_s ESCs
5000 mAh 3s LiPo
Gemfan 1045 props

When I am using a 3000 mah 3s battery, my quad flies well with 9443 props but 1045 props make motors very hot.
When I use the 5000 mah lipo with the 9443 props, the quad barely takes off, but with the 1045 props it flies but the motors get extremely hot.

Online threads say that 1045 props work well with 980KV motors. What am I doing wrong? Should I switch to APM instead of the Naze32?

Changing the flight controller won't help you.

A certain Kv cannot be associated with a certain prop size. You need to take in consideration the thrust and wattage of the motor too.

If the motors you are using are the ones in the link below, you need to reduce prop size to 8x4.5. Your quad seems to be overweight (for the given power system).

https://www.banggood.com/DYS-BX2212-KV920-Brushless-Motor-CWCCW-For-RC-Multicopters-p-936355.html

As said changing the Flight Controller will not have much impact on your quad since you are not flying any racing quad.

The motors are designed as clones for dji phantom 1 which can take maximum of 9443 prop and not suitable for 1045 which is the reason for heat production.

It is also the weight of the battery that makes your quad heavy and so your motors are doing some extra work lifting your quad.
If 3000mah lipo can provide sufficient amount of power then try to fly with 3000mah and 1045 props. However it is not recommended to mount fly with 1045 props.

What is the weight of your quad?? Even it may have impact on motors. Heavy quad more power and so does the power consumption.

aren`t these motors rated upto 5s with 8045?

I just put the specs in ecalc and i could not find anything fishy.