As a result of volunteer work after a 2017 disaster relief project, some lessons were learned about trash recognition. One thing clear is that human visual and mental effectiveness in trash recognition is not normally very good. It is definitely an acquired skill even for those with a few decades of light experience. We were able to see items in the grass and dirt we had missed before we gained more intensive experience.
Lighting is a major factor too. Some items are best identified under predominantly omnidirectional light. Others appeared best with a strong single light source. This leads to an adjunct engineering approach that will improve training and execution for nearly any AI approach. An LED that strobes on every $x$ frames would provide an additional dimension to training data and could potentially double aspects of effectiveness, such as speed and thoroughness. Also, training must be done during times of direct sunlight and clouded daylight, since the edges patterns in one lighting condition may not resemble the edge patterns of another.
Model-free training may suffice, more specifically that the only model necessary may that of be a CNN network that has some model of built in capability to recognize arbitrary features in the edges of objects in focus.
The coordination of grasp and multiple visual source angles is probably the most complex part of the AI design, especially if trash removal must include items caught in taller vegetation or fences, considerably widening the 3D range for both vision and mechanical grasping.
Even ground trash can be challenging in that the angle of approach for the grasping mechanism is a major factor in the reliability of the grasping action. This means that the output of the recognition network must include the 3D location of the grab point and the angle of approach for the grabber, which is a compound angle.
If the AI that controls the robotic grabber had the center pixel of the trash object or even estimated the center of mass, the robot could try grabbing some items a hundred times before getting a firm grasp. It is the way the grabber is pointing and the mechanical feature of the trash it closes upon that produces a reliable grip on the item. Therefore, training on the basis of grabber success is surely the way to train. No such database yet exists.
Distinguishing a cigarette from a straw or a bottle from a can or a banana peel from a dead mouse does not, in practice, matter much. Object recognition is overkill and of little value. The robot is looking for a holding point. The type of intelligence required features the artificial equivalent of hand-eye coordination, the cerebellum and the visual centers of the brain. Only if there is a long cord, string, or wire that is or that is attached to the trash, is anything beyond that required. Focus on the functional aspects of grasping, not what the item once meant to people before they tossed it out.
Train for eight dimensional output.
- Distance from the base joint of the grasping arm to the grasp point on the trash item
- Pitch from grasping arm to grasp point
- Yaw from grasping arm to grasp point
- Pitch of the grabber relative to the arm
- Yaw of the grabber relative to the arm
- Banking of the grabber relative to the ground
- Grasp starting opening size
- Grasp ending opening size
Best accuracy and reliability can be achieved with three layers of learning. Each more specific layer builds on top of learning achieved at the more general layers, further tuning the parameters for specificity.
- Factory installed recognition of basic forms of the most common types of ground trash
- Regional trash pattern learning
- Street and neighborhood specific adjustments, remembered between deployments to specific trash removal routes
Strobe a bright LED light in the direction the camera and syncronized with its frame rate. Also, an infrared channel in the camera that produces an IR nibble in the pixels to accompany three-color will improve learning speed and execution accuracy. If these recommendations are followed, there will be five independent dimensions of data and one dependent dimension at the input of the network to be trained.
- Frame pair
- Strobe on vs off
- Vertical index
- Horizontal index
- Pixel depth
- Brightness value
This device must traverse the grounds, streets, and sidewalks like a sweeper or lawnmower and could conceivably be mounted on the front of such existing devices. Full automation would require an automated grounds vehicle, which, as of this writing has not yet been developed for municipal or private use. A utility grounds vehicle or golf cart mounting with a human driver would currently be required.
One of the things we noticed doing the volunteer work is that if a commercial grounds keeping company came by, they would mow the trash into pieces unless otherwise instructed by their managers. The idea of the workers may have been that the trash would biodegrade. For paper and cigarettes, this is true, but aluminum, steel, glass, plastic, foam, and some other materials, it is not. Steel can also ruin the mower and become shrapnel if hit by a blade. Glass and nails can pose dangers for years.
When a blade hits any one of these things, the complete removal of the trash takes ten to a few hundred times longer, depending on the material.