If I have a Hard Charm attached to Talonflame and use its "Jet Shoot" attack, then next turn my opponent does a simple attack that does 20 damage, can I pick the order of when to apply the Hard Charm's -20 reduction and "Jet Shoot's" +40 recoil? If I use Hard Charm to reduce the net damage to zero (20-20), could I then negate the "Jet Shoot" recoil?
No, it doesn't work like that. The check for effects on the defending player's Active Pokemon, such as Hard Charm and Jet Shoot, both happen at the same step in the damage calculation process. So you don't stop damage calculations mid-step just because the calculation was less than or equal to zero. The net effect of Hard Charm & Jet Shoot result in +20 damage at that step in the calculation process.
Source: TPCi Rules Team (2015-12-10)