Good idea, sofar. Thought, I would always sugguest to implement a general approach in combination with a concrete usecase (a game actually). This avoids a lot of refactoring later on
Nevertheless, here are some thoughts from my experiences, because I use a similar system:
1. Multithreading and scripting languages like lua are really hard. Lua do not really support multithreading, therefor consider this in your design early on. Remember, that the state of a single VM can't be really shared between different threads as long as the VM doesnt support it , which is the case with lua (I use multiple stateless VMs to cover multi-core support).
2. You will need a lot of communication between engine and AI. The go-to command is a good idea and the right direction, but you will need to react to external events too. E.g. the entity movement is blocked by an other entity etc. Therefor use an asynchroniously , delayable message/event system.
3. There are some basic AI algorithm which are really performance critical and should be part of the engine:
- pathfinding (your have some navigation map/waypoint system already in the engine).
- sensory data/scanning (build a query system to send async scan requests, you will need this a lot)
- steering (your go-to command which will result in the steering of the AI, including common steering behavior).
4. It is really important for an AI to be aware of the surrounding, its state, and current events. A good way to handle this is the use of a stimuli system. External events, internal states etc. can trigger a time restricted stimuli on your entity, which will help the AI to make further decisions. Scanning the surround all the time is just to performance demanding and scanning it just every X ms is too slow. The latter will result in clearly visible delay in the decision making process.
Eg a sound from a gun could propagate a 'gun_sound' stimuli in all close npcs. The npc ai can then decide what to do with it (one ai is running away from it, an other will approach the last know position of the sound, an other entity will be more careful as long as the stimuli is active etc.)