Objective
Encourage realistic player and NPC behaviour =>
Discourage 'cheating' through grinding (= position yourself as to enable hitting enemies without them not being able to hit you) =>
Reduce success rate of NPCs running into grinding traps =>
1. Improve chance of NPCs recognizing they cannot hit the player (or NPC) while they are being hit (before they get killed).
2. Make (most) NPCs realize they have suffered near-lethal injury.
3. Make NPCs retreat to safe distance (either because the player is out of reach to them, or because they need to recover), and make them stay there for a while to either await a new opportunity to attack the player (or NPC), or to recover from their wounds before they re-engage in combat (or follow their leader: hired NPCs).
4. If NPC's fail to end an attack on the same target for 3 times in a row, the NPC's AI should switch to 'full retreat' and make the NPC move out of attack range form target and reset his/her designated 'target'.
Assumptions
The experience orb system will remain the main source of experience, whatever the new enchantment (, smithing and/or trait) system may entrail.
I assume that the grinding mechanism in itself is contradictory to the gameplay objectives as laid out in the mod and the official server set-up and rules. I hope this helps in achieving these objectives, but I do realize many players are rather used to grinding a lot, so quite a few won't be in favour of this or even start a little rant against this. :)
Background info / references
Following through on a discussion we had in this thread, I listed a number of possible measures to discourage grinding behaviour. When reconsidering the mechanism of grinding (see definition above), I realized the most effective measure will be to improve the NPC AI. Apparently it is possible to make ranged unit NPCs flee when 'they realize they're in danger' ... so, it should also be possible with all vanilla MC coding options at hand, to code for AI that covers the 4 objected AI functionalities mentioned above.
I did a bit of research for earlier suggestions related to this, and found these: flee from mortal danger, take some time to heal and nothing more really. So I assume this aspect hasn't been thoroughly touched upon (at least publicly).
There's quite some references available on designing AI for units, so I won't assume it will be hard to find a suitable code ... just the work ... should be enjoyable for someone. ;)
Additional specifications
Not all NPCS should obey the basic AI to avoid mortal danger. I propose to have any unmanned Beasts and Bomb carrying and Berserker type units ignore these AI rules. It may also be considered to let NPCs that have strength and speed potion effects ignore them, until the effect is removed. As a general rule I propose to let NPCs retreat when their health level is at or below 6 hp. This parameter value may also be unit dependent.
When an NPC realizes (s)he cannot hit a player (whether from melee strikes or ranged attempts), after completing a path-finding routine towards the player, the NPC should retreat to a position at a safe distance. When they reach a safe position (distance), a cooldown will start and, dependent on the NPC type, they may start a new attempt to kill the player when they are partially or fully healed or after a certain cooldown time has ended. As a general rule I propose that heavy armored, high hp and/or generally unintelligent units will resume an attempt after they have either partially healed (up to 80% of full health) or when they are fully healed. Weak and/or ranged units and 'smarter' units like villagers should only resume an attack after a serious cooldown period (and when the player is still within range, of course). If the player directly attack the NPC during the time the NPC is set 'in retreat', the NPC will immediately respond to the attack by either fleeing or counter-attacking the player.
Player weapon enchants
The proposed AI will not be effective against players that are able to prevent NPCs from getting any hit chance against them. Therefore, in line with the measures proposed on enchantments here, I wish to stress the relevance of disabling a number of high level enchantments like Infinity and Knockback III or higher. Of course, this will depend on the system that will ultimately be adopted and implemented for enchanting anyway, of which we can still only speculate.
Hired NPC behaviour
In case of hired units blindly following their leader, but ending up with lethal injury through fall damage, suffocation, burns, etcetera, they should simply obey the rule to simply 'stay put' until they have healed to full health. After full recovery they can resume to follow their ignorant leader.
NPC vs NPC
Whether an NPC targets a player or another NPC, would not have to make any difference. I assume if the code for this intended AI is created, the NPC AI uses a defined 'target' for which the AI decision making sequence is valid. If the NPC is attacked by another entity than it's actual target, I assume the AI now redefines the NPCs 'target' to the entity that attacked him/her most recently. The decision making sequence will then just restart and reiterate to one of the possible modes 'attack', 'flee', 'seek refuge and heal', etcetera. The NPC can be hired or not and the target can be a hired NPC, a player, or another NPC (like during invasions, borderland fights or orc brawls fights).
Repeating loop prevention
A potential problem may be that sequences of 'attack', 'retreat to heal', 'attack again', etcetera, may lead to endless fighting without a conclusion when an (NPC or player) remains out of reach of another NPC fighting him/her. Possibly such scenario's can already take place with the current AI, but it will certainly become more likely to happen when the AI as proposed here will be implemented. This can not only be weird but also become a burden to CPU and memory when chunks remain loaded. This must be countered with some measure that ends the sequence. In any case I think a counter should be implemented on the number of times an attack attempt was made on the actual target. I suggest that if that counter reaches 3 and the NPC's AI decides to stop yet another fruitless attempt to 'finish him off', the NPC's AI should activate another new mode: 'full retreat'. This mode should bring the NPC to a distance that is guaranteed out of attack range of the latest target and reset the NPC's target, leading to an NPC in the field without any target, awaiting orders.
NPC Respawners
These were added in Public Beta 28 and offer an opportunity for players to build NPC grinders. This can be prevented by combining them with a protection banner that has a protection zone large enough to prevent building of effective grinding traps around the banner and offers NPCs the opportunity to effectively retreat, as described above. This implies NPC Respawners should not be placed in a closed confined space where NPCs do not have the ability to retreat while being fired at by a player that is target locked by the NPC. This may require some more, dedicated pondering, but this may do for now.
Considerations
I had a quick look at some relevant AI currently in the mod and it seems to me the way the mod is set up should easily allow for addition of well thought-through AI rules. I realize the amount of work to code for this may not be the most appealing aspect of this improvement option, but would like to point out the opportunity of having the community assist in this by providing possible draft code formulations during the conceptual design process. I'm sure there's dozens of players out there with either command block or java experience who can easily prepare a draft concept of the sequence of checks, conditions and actions to be included in this.
Now shoot folks! (... be gentle and don't hide in grinding position, face it like a man :P)
I will improve this OP as my own insight progresses and based on any constructive feedback/criticism.