MicroPython MicroMLP: How do I reward the program based on state?

Question

I have been trying to use MicroMLP to teach a small neural network to converge to correct results. Ultimately, I want to have three outputs, one which is high priority (X must be as close to XTarget as possible; Y and Z must be within bounds, and should approach YTarget and ZTarget as best as possible). Right now, I'm just trying to get convergence of one variable to understand this library.

The below code works for xor, but I don't really understand why it works, or how to extend this to reward behavior:

from microMLP import MicroMLP
import utime
import random
import machine
import gc

DEPTH=3
mlp = MicroMLP.Create( neuronsByLayers           = [DEPTH, DEPTH, 1],
                       activationFuncName        = MicroMLP.ACTFUNC_GAUSSIAN,
                       layersAutoConnectFunction = MicroMLP.LayersFullConnect )

nnFalse  = MicroMLP.NNValue.FromBool(False)
nnTrue   = MicroMLP.NNValue.FromBool(True)

led = machine.Pin(25, machine.Pin.OUT)
tl = 0
xor = []
c=0
for i in range(5000):
    if not i % 100:
        led.toggle()
        gc.collect()
        print(" Iteration: %s \t Correct: %s of 10" % (i,c))
        c = 0
    xor.clear()
    xorOut = nnFalse
    for j in range(DEPTH):
        if random.random() > 0.5:
            xor.append(nnTrue)
            xorOut = nnFalse if xorOut == nnTrue else nnTrue
        else:
            xor.append(nnFalse)
    p = mlp.Predict(xor)
    mlp.QLearningLearnForChosenAction(None, xorOut, xor, 0)
    if p[0].AsBool == xorOut.AsBool:
        c += 1

led.off()

print( "LEARNED :" )

c = 0
tries = 0
for i in range(100):
    led.toggle()
    gc.collect()
    xor.clear()
    xorOut = nnFalse
    for j in range(DEPTH):
        if random.random() > 0.5:
            xor.append(nnTrue)
            xorOut = nnFalse if xorOut == nnTrue else nnTrue
        else:
            xor.append(nnFalse)
    tries += 1
    p = mlp.Predict(xor)
    c += 1 if mlp.Predict(xor)[0].AsBool == xorOut.AsBool else 0
 
print( "  %s of %s" % (c, tries) )

del mlp
print(gc.mem_alloc())
gc.collect()
print(gc.mem_alloc())

I'm trying to achieve two goals, first for me to understand, second for the machine to do useful work.

Goal #1: learn to adjust a value properly.

Inputs:

• Target (0,1)
• Value (0,1)

Outputs:

• An adjustment toward Value (-1,1)
  • Possibly this has to be (0,1) so I've considered using the adjustment as adjustment - 0.5 to put it into the (-0.5,0.5) range

I want to reward the thing based on the degree to which value comes closer to the target. (As a special case, if it's impossible to adjust that far given the output, I want to maximize its reward for making the maximum adjustment.) I don't want to know the value adjustment should target; I only want to know that whatever value it gave produced a state I like better, and what that value was. If I can know the correct output, I don't need deep learning.

Goal #2, the later one I expect to be able to do myself if I can get one variable working, is to have several inputs and three outputs. These inputs relate to the current targets and the deviation from those targets. One of these is of the highest priority to track toward a target value; the other two should track toward a target value, but are allowed to deviate by some amount with no harm done. If I can just figure out how to use the neural network, I should be able to assemble that.

Does this sound reasonable? Is this the correct tool, or is Q-Learning wrong for this?

Feel free to suggest a better package for regular Python as well, although MicroMLP is the only usable one of which I'm aware for the platform I'm targeting. I'll likely want a much more powerful one that I can use with extra available hardware if present.

If I get something I can work with, I'll write documentation and submit a PR to the MicroMLP repo so nobody has to ask this again.