8-bit microcontrollers (such as the AVR on the nano) are unlikely to do much in the way of inference. One could in theory enlist/write a matrix library that worked with 8-bit floats but it's unlikely to be particularly efficient.
Modern ARM microcontrollers (such as those on the Nano 32/33) with 32-bit cores and with a floating point unit are much more likely candidates. ARM very helpfully provides the CMSIS NN library to run pre-trained models. Size of model will be limited compared to desktop/server deployed models but some stuff can be successfully run. There are bleeding edge microcontrollers with neural processing units (NPUs) at the time of this writing.
Another option is an FPGA. That is a whole other skill set and the learning curve is steep but they can be much higher performance than a microcontroller.
On the largest side of embedded, application cores with Mali GPUs/etc can do a bit more work. There are USB powered NPUs for processors without a GPU (or powerful enough GPU). We're even beginning to see application cores with built in NPUs.