2
$\begingroup$

I'm working in a company that opens restaurants in enterprises. Every day at lunch, we want our clients to be able to scan their trays, sothat the food is detected automatically thanks to AI / image recognition.

Technically speaking, we have a number of food items that grows over time in our database, but everyday there are about 30 items available at the same time in the restaurant. About 5 items are changing each day (for example, the main dish changes, but the bottle of water is always the same).

This means, when the client go to the till to pay, the client will place the tray by himself, the camera will take photos of the tray and will try to identify different items separetely among the 30 items available this day. Clients pay per food article, which means we don't need to track the weight or the quantity in the main dish.

I have absolutely no experience in AI/ML and don't know how to start for my need, I'm a web developer.

  • Which tool should I look at first?
  • Which skill do I need to acquire? I mean, are there easy to use high level libraries , or do I need to learn ML from scratch?

First I was thinking of Amazon Recognition or Google Vision but It seems to be made for recognizing ANY food item among their own database. My need seems easier since I just need to recognize several items on a tray among 30 known items.

Thanks a lot for your help.

|improve this question
$\endgroup$
  • $\begingroup$ you should take a model that recognise objects (not necessary food) like Amazon Recognition, google vision or Yolo for example, and train it from scratch with the food you have in database. You only need to create a dataset to train the model, so you need to acquire some experience in this regard, but there are plenty of tutorials on internet for how to train a network and so on. $\endgroup$ – Jérémy Blain Nov 20 '18 at 15:58
  • 3
    $\begingroup$ Not necessarily from scratch, but the last few layers (transfer learning). See martin-thoma.com/object-detection as a starting point. $\endgroup$ – Martin Thoma Nov 20 '18 at 16:07
  • 2
    $\begingroup$ This seems like an ambitious project. How much time have you allocated to this, and what is your output - a feasibility report, a prototype, an actual working installation? Computer vision projects are not in the "install a library and go" category yet, you need some understanding of the internals and the language of data science - not to expert level, but at least some - which will take some dedicated time to obtain. $\endgroup$ – Neil Slater Nov 20 '18 at 16:11
  • $\begingroup$ @NeilSlater Thanks! I'll guess this will be my main objective in 2019, so let's say 6 months full-time. I'm a senior Python/Django web dev so I guess this can be useful. I will dedicate any time I need to level up on this topic. I just need to start the right way :) $\endgroup$ – David D. Nov 20 '18 at 19:19
  • 1
    $\begingroup$ @DavidD. I don't know Amazon rekognition, but if you have access to the model that you can load on any machine learning framework like keras or pytorch, then you can retrain the model with your own dataset, you just need to change the output layer with the number of class you need. Search for transfer learning or fine tuning like Martin Thoma said in his comment. (But 30 data for each class would be not enough I think, but you need to test it !) $\endgroup$ – Jérémy Blain Nov 21 '18 at 8:32
1
$\begingroup$

For a midsize corporation running multiple cafeterias, an AI tool may be feasible, provided sufficient time and resources are invested well in advance of system use. Selling as a full strategy an AI tool without a corporate commitment to a strategy which includes costs listed below is unwise. As of this writing and for the foreseeable future, there are no drop-in AI systems that will recognize a command such as, "Learn these foods and their prices and then keep learning whenever we adjust the menu due to cycling of main dishes, buying considerations, or substitutions made due to shortages. By the way, when one food obscures another, either by design of the buyer to avoid paying for items or by accident, detect that and respond appropriately."

You cannot assume that the foods in the tray will look like a single photo of that food in a tray either. How many frames of them will be required is variable and some domain specific research may be required to size the project. A smaller investigative project will need to be completed before the corporation can decide whether to invest all the way.

The best approach is to set up multiple camera angles focused on a target area clearly outlined between the existing human cashier and the tray holder. The cashier involved in training the AI must ensure the trays are in the outlined target area and charge for the food on the tray accurately. The data from cameras and point of sale systems must be merged to produce videos with itemized lists of items on the trays.

A sufficiently deep LSTM network could be trained, tested, and verified on a sample of that data. Theft detection would need to be designed into all elements of the strategy. The current database schema would need to link to image files. Thirty frames per second would not be necessary. Two or three FPS might be sufficient. Some theoretical investigation and subsequent experimentation as part of the initial investigation would be be wise.

These are probable costs and considerations associated with transition and continued use of the AI system, and this is not meant to be an exhaustive list.

  • Initial research to size the project properly
  • At least two cameras at each location, usable for both training and execution
  • Extension of networking equipment
  • Software to tie itemized lists of items at the point of sales with the associated video feed and indicate to the point of sales system what to charge the tray-holder once the items can be recognized
  • Database storage to support low frame rate, medium resolution video storage
  • Human personnel training time and materials so that they can participate in the machine training as new items are added (so that the recognition of the items in the trainers is transferred to the machine trainee)
  • An off line cashier station for training so that items can be added to the database without interrupting cafeteria revenue generation
  • Purchase and shipping of items to the simulation station
  • Training programs for food buyers and cooks
  • A plan for how to transition the workforce in accordance with legal and ethical standards
  • Associated customer and public relations
  • Equipment to indicate probable theft to the food purchaser or to the appropriate security personnel if the food purchaser will not comply
  • Plan for dealing with cash or excluding it from payment options
|improve this answer
$\endgroup$
-1
$\begingroup$

An autonomous food recognition system won't work, especially not for commercial purposes. Even if cloud based machine learning has a recognition rate of 99% this isn't enough.

The problem is that many AI engineers believe they can take the human out of the loop and are then surprised when their system doesn't work. A better idea would be to combine technology and humans. That means, for example, scanning the tray image but letting a human employee recognise the picture.

The reason is that a normal restaurant is mostly driven by manual labour. The only way that advanced technology will be accepted by people is if this technology will have human-level-capabilities. That is currently only possible if some kind of human-operator is in the loop who decides, rejects frauds, and is able to laugh about problems.

A semi-autonomous vision system would involve showing the picture in a GUI application, and having a human employee guess what is on the table. It is some kind of Mechanical Turk game, called “guess what is on the picture”. Such a kind of human-machine-interaction has the advantage that it is able to work in all circumstances.

|improve this answer
$\endgroup$
  • 3
    $\begingroup$ I don't think this answers the question as posed (it also could do with separating into paragraphs for readability). However, OP does need to keep this at the front of their mind. Ideally the work involved is a feasibility study or prototype which may or may not confirm the reservations in this answer. Perhaps for instance confidence levels can be included so that combined machine+human intervention success rate is 99.9%, at fraction of cost of full staffing $\endgroup$ – Neil Slater Nov 21 '18 at 9:21
  • 1
    $\begingroup$ "Even if cloud based machine learning has a recognition rate of 99% this isn't enough." is not enough for who ? you ? I think your requirement aren't the same as the OP (How could you know ?). Answer the question instead of answering another one. $\endgroup$ – Jérémy Blain Nov 21 '18 at 10:40
  • $\begingroup$ @JérémyBlain The OP is about a computer vision project in a restaurant environment. The first impression might be, that a detection rate of 99% will solve the problem. That means, the picture is taken, the software prints out that it's item #12 and the task is done. The reality is much more complicated. The recognition rate is not the same like the project success rate. That means, even if the vision algorithm works great, the overall restaurant project will fail, because the vision application should replace human workers who have done the same task manually in the past. $\endgroup$ – Manuel Rodriguez Nov 21 '18 at 10:53
  • $\begingroup$ The OP IS asking about a tool that replace human workers for automated tool... So I think the resturant project would success with such technology, actually... $\endgroup$ – Jérémy Blain Nov 22 '18 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.