I want to build a semi autonomous robot/machine that will clean up trash in cities. For this to be possible it needs to recognize 'trash'. As trash can be all sorts of things (think ciggaret buts, plastic, bottles or anything that humans leave behind) I was curious if it would be possible for a AI to learn this broad concept with current technologies? And if so what would I need to build it? I was thinking Tensorflow and a dataset with different kinds of trash.

Thank you for taking the time to help!

  • $\begingroup$ Yes, it is. You can take martin-thoma.com/object-detection as a starting point and then fine-tune the model. $\endgroup$ – Martin Thoma Nov 29 '18 at 6:34
  • $\begingroup$ The first step is to create an ontology of trash. That is a mindmap in which the socio-economic impact is made visible. Trash can be there in cities, on the street, in the home or in the garden. It can contains of empty pizza boxes, old newspapers and glasses. Trash can be metal, plastic or clothes. Before it's possible to train a neural network with all these different sort of waste there is a need to classify the subject. In it's easiest case an ontology consists of an XML file, which stores the words in a hierarchical way. $\endgroup$ – Manuel Rodriguez Nov 29 '18 at 9:33

As a result of volunteer work after a 2017 disaster relief project, some lessons were learned about trash recognition. One thing clear is that human visual and mental effectiveness in trash recognition is not normally very good. It is definitely an acquired skill even for those with a few decades of light experience. We were able to see items in the grass and dirt we had missed before we gained more intensive experience.

Lighting is a major factor too. Some items are best identified under predominantly omnidirectional light. Others appeared best with a strong single light source. This leads to an adjunct engineering approach that will improve training and execution for nearly any AI approach. An LED that strobes on every $x$ frames would provide an additional dimension to training data and could potentially double aspects of effectiveness, such as speed and thoroughness. Also, training must be done during times of direct sunlight and clouded daylight, since the edges patterns in one lighting condition may not resemble the edge patterns of another.

Model-free training may suffice, more specifically that the only model necessary may that of be a CNN network that has some model of built in capability to recognize arbitrary features in the edges of objects in focus.

The coordination of grasp and multiple visual source angles is probably the most complex part of the AI design, especially if trash removal must include items caught in taller vegetation or fences, considerably widening the 3D range for both vision and mechanical grasping.

Even ground trash can be challenging in that the angle of approach for the grasping mechanism is a major factor in the reliability of the grasping action. This means that the output of the recognition network must include the 3D location of the grab point and the angle of approach for the grabber, which is a compound angle.

If the AI that controls the robotic grabber had the center pixel of the trash object or even estimated the center of mass, the robot could try grabbing some items a hundred times before getting a firm grasp. It is the way the grabber is pointing and the mechanical feature of the trash it closes upon that produces a reliable grip on the item. Therefore, training on the basis of grabber success is surely the way to train. No such database yet exists.

Distinguishing a cigarette from a straw or a bottle from a can or a banana peel from a dead mouse does not, in practice, matter much. Object recognition is overkill and of little value. The robot is looking for a holding point. The type of intelligence required features the artificial equivalent of hand-eye coordination, the cerebellum and the visual centers of the brain. Only if there is a long cord, string, or wire that is or that is attached to the trash, is anything beyond that required. Focus on the functional aspects of grasping, not what the item once meant to people before they tossed it out.

Train for eight dimensional output.

  • Distance from the base joint of the grasping arm to the grasp point on the trash item
  • Pitch from grasping arm to grasp point
  • Yaw from grasping arm to grasp point
  • Pitch of the grabber relative to the arm
  • Yaw of the grabber relative to the arm
  • Banking of the grabber relative to the ground
  • Grasp starting opening size
  • Grasp ending opening size

Best accuracy and reliability can be achieved with three layers of learning. Each more specific layer builds on top of learning achieved at the more general layers, further tuning the parameters for specificity.

  • Factory installed recognition of basic forms of the most common types of ground trash
  • Regional trash pattern learning
  • Street and neighborhood specific adjustments, remembered between deployments to specific trash removal routes

Strobe a bright LED light in the direction the camera and syncronized with its frame rate. Also, an infrared channel in the camera that produces an IR nibble in the pixels to accompany three-color will improve learning speed and execution accuracy. If these recommendations are followed, there will be five independent dimensions of data and one dependent dimension at the input of the network to be trained.

  • Frame pair
  • Strobe on vs off
  • Vertical index
  • Horizontal index
  • Pixel depth
  • Brightness value

This device must traverse the grounds, streets, and sidewalks like a sweeper or lawnmower and could conceivably be mounted on the front of such existing devices. Full automation would require an automated grounds vehicle, which, as of this writing has not yet been developed for municipal or private use. A utility grounds vehicle or golf cart mounting with a human driver would currently be required.

One of the things we noticed doing the volunteer work is that if a commercial grounds keeping company came by, they would mow the trash into pieces unless otherwise instructed by their managers. The idea of the workers may have been that the trash would biodegrade. For paper and cigarettes, this is true, but aluminum, steel, glass, plastic, foam, and some other materials, it is not. Steel can also ruin the mower and become shrapnel if hit by a blade. Glass and nails can pose dangers for years.

When a blade hits any one of these things, the complete removal of the trash takes ten to a few hundred times longer, depending on the material.

  • $\begingroup$ The concept of using features like “distance from base point” or “pitch from grasping point” are on the first look a good strategy. They help to reduce the problem down to a smaller one and fits perfectly to deeplearning algorithms. The shortage of features is, that they're connected to a mathematical space, which is a matrix. All the items are stored in an array as numerical values and this describes the grasping task. Is a matrix with ten floating point numbers the right choice for modeling complex situations? No, it won't replicate human thinking but would result into a broken markov chain. $\endgroup$ – Manuel Rodriguez Dec 3 '18 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.