These are some quick observations from a decently scripted MoP server I played on before starting Wowaddicts, data from their DB and calculations to see if their scripting at least roughly matches the scarce info I got from online research on the topic of player XP from pet battles. You can ignore this part because it's not exactly well organized and might get you confused (especially the part with quest XP). Overall these observations roughly agreed with the online info, so I combined them with the info to write the formulae below.

@ lvl 43 ... rested (1.5x) ... + 3x server XP boost
4785 XP qst (lvl 39/40 ... 4395 XP ... 4350 XP) ... 14355 XP
5075 XP qst ... 5582.5 XP
2-pet enemy
3583 @ +2 lvl
3418 @ +1 lvl
3257 @ 0 lvl ... 4186.875
2607 @ -4 lvl
2442 @ -5 lvl
0 @ -6 lvl

@ lvl 44 ... -||-
4785 XP qst (lvl 39/40 ... 4395 XP ... 4350 XP) ... 14355 XP
5315 XP qst
2-pet enemy
3593 @ +1 lvl

@ lvl 69 ... rested (1.5x) ... + 3x server XP boost
12320 XP qst
3-pet enemy
29925 @ +1 lvl
28500 @ 0 lvl
29782 @ -1 lvl

@ lvl 89
196000 XP qst ... 215600
3-pet enemy
~161700 @ 0 lvl


~~~


3-pet enemy team and 0 lvl difference between highest player team pet level and highest enemy pet level (3/4 of qst XP reward for appropriate lvl):
<XP for standard quest for character's level ... quest level == char level, NOT required level == char level> * 0.75

2-pet enemy team and 0 lvl difference between highest player team pet level and highest enemy pet level (3/4 of XP from battle against 3-pet enemy team, same conditions):
<XP for standard quest for character's level ... quest level == char level, NOT required level == char level> * 0.75 * 0.75

1-pet enemy team and 0 lvl difference between highest player team pet level and highest enemy pet level (1/2 of XP from battle against 3-pet enemy team, same conditions):
<XP for standard quest for character's level ... quest level == char level, NOT required level == char level> * 0.75 * 0.5

for every 1 lvl of difference between highest player team pet level and highest enemy pet level add/subtract 5% of base 0 lvl difference XP for appropriate number of pets on enemy team:
@ +5 lvl or more ... +25% (for example highest player pet lvl 15, highest enemy pet lvl 20 or more)
@ +4 lvl ... +20%
@ +3 lvl ... +15%
@ +2 lvl ... +10%
@ +1 lvl ... +5%
@ -1 lvl ... -5%
@ -2 lvl ... -10%
@ -3 lvl ... -15%
@ -4 lvl ... -20%
@ -5 lvl ... -25% (for example highest player pet lvl 25, highest enemy pet lvl 20)
@ -6 lvl or more ... no player XP

Note 1:
The final results should be processed with round/floor/ceil functions to always get integers. Since I don't know which one of those functions you prefer or if you perhaps prefer another way of removing fractions, I left the results in the example below as they came from the calculation.

Note 2:
Any applicable XP bonuses (server XP rates from .xp, heirlooms if they're supposed to apply to pet battle XP, rested if it's supposed to apply to pet battle XP, etc.) should be applied to the results of these basic calculations, so to the rightmost numbers in the example below.

Note 3:
I think the biggest issue will be determining the base (quest) XP for the calculations for each player level because from what I saw on that MoP server's wowhead-like DB site, the XP for a certain quest level isn't always the same - for example there are quests that are marked as lvl 44 that give 537 XP, 1345 XP, 5315 XP and 6625 XP (check attached screenshot). Unless there are some special flags that would help identify correct reference quests or calculate correct reference XP in a different way, a somewhat reasonable solution might be finding all quests of player's level, putting them in an array, ordering them by XP and picking the median XP from that (not average because that could result in less XP on some higher levels than on some lower levels depending on how many quests of given level there are overall and how many of them are simple low XP quests like "go to this other quest NPC" and bam, here you go a few 100 XP at lvl 60). Also, I'm not sure if there are quests or every level in the DB and how many there are for each level, so for some levels other methods might be necessary to put the XP somewhere between XP for the previous and next level.

Note 4:
Pet battles against pet trainer NPCs are supposed to give more XP according to some online info, but unless you manage to find more info on those battles specifically, I'd just go with the same formulae below for both wild pet battles and trainer NPC battles. As for pet battle duels against other players, I think those are not supposed to give any XP, just like normal PvP duels.


~~~


Complete example for 10000 XP per quest of player's lvl (for quick script check purposes):

3-pet enemy team (15 <= lvl <= 25 for wild pet teams):
@ +5 lvl or more ... 7500 * 1.25 = 9375
@ +4 lvl ... 7500 * 1.20 = 9000
@ +3 lvl ... 7500 * 1.15 = 8625
@ +2 lvl ... 7500 * 1.10 = 8250
@ +1 lvl ... 7500 * 1.05 = 7875
@ 0 lvl ... 10000 * 0.75 = 7500
@ -1 lvl ... 7500 * 0.95 = 7125
@ -2 lvl ... 7500 * 0.90 = 6750
@ -3 lvl ... 7500 * 0.85 = 6375
@ -4 lvl ... 7500 * 0.80 = 6000
@ -5 lvl ... 7500 * 0.75 = 5625
@ -6 lvl or more ... 0

2-pet enemy team (6 <= lvl <= 14 for wild pet teams):
@ +5 lvl or more ... 5625 * 1.25 = 7031.25
@ +4 lvl ... 5625 * 1.20 = 6750
@ +3 lvl ... 5625 * 1.15 = 6468.75
@ +2 lvl ... 5625 * 1.10 = 6187.5
@ +1 lvl ... 5625 * 1.05 = 5906.25
@ 0 lvl ... 10000 * 0.75 * 0.75 = 5625
@ -1 lvl ... 5625 * 0.95 = 5343.75
@ -2 lvl ... 5625 * 0.90 = 5062.5
@ -3 lvl ... 5625 * 0.85 = 4781.25
@ -4 lvl ... 5625 * 0.80 = 4500
@ -5 lvl ... 5625 * 0.75 = 4218.75
@ -6 lvl or more ... 0

1-pet enemy team (1 <= lvl <= 5 for wild pet teams):
@ +5 lvl or more ... 3750 * 1.25 = 4687.5
@ +4 lvl ... 3750 * 1.20 = 4500
@ +3 lvl ... 3750 * 1.15 = 4312.5
@ +2 lvl ... 3750 * 1.10 = 4125
@ +1 lvl ... 3750 * 1.05 = 3937.5
@ 0 lvl ... 10000 * 0.75 * 0.5 = 3750
@ -1 lvl ... 3750 * 0.95 = 3562.5
@ -2 lvl ... 3750 * 0.90 = 3375
@ -3 lvl ... 3750 * 0.85 = 3187.5
@ -4 lvl ... 3750 * 0.80 = 3000
@ -5 lvl ... 3750 * 0.75 = 2812.5
@ -6 lvl or more ... 0