Image Captioning is a hot research topic in the AI community. There are considerable image captioning models for research usage such as NIC, Neural Talk 2 etc. But can these research models be used for commercial purpose? Or we should build much more complex structured ones for commercial usage? Or if we can make some improvements based these models to meet the business applications situation? If so, what improvements should we take? Are there any existing commercial Image Captioning applications can be referenced?


Thinking Commercially

A commercial solution will need to be able to ascertain, continuously verify, and utilize the best of options for captioning learning models.

Each fairly successful image captioning learning models could be placed in an adapter to provide a common training, optimizing, testing, evaluation, and usage interface. The hot swappable container-addon mega pattern used in peripheral device installation, RAID, J2EE containers, browsers, and other containment sub-systems can be applied.

System Description

The system acceptance criteria being as follows.

  • A new model could be added without stopping or starting the system if tauted as successful by others
  • Any model can be deleted without starting or stopping the system if it has not been performing well
  • Each possible state of each model added can be given some percentage of the total system resources
  • Various training processes and hyper parameter tuning processes can be applied to any of those model-state combinations
  • A-B testing can be done on any model-state combination
  • Stats can be displayed on any model-state combination any model overall or any state across all added models
  • The interface for each model requires that its associated model produces an array of caption-reliability pairs with zero ore more pairs
  • The reliability measure that is included in the pair with the caption is an indication of the model's assessment of appropriateness to the image
  • The most appropriate caption is picked from the proposals of the various models

States can be idle or any of these.

  • Training
  • Testing
  • Evaluation
  • Hyper parameter tuning
  • In use

For example, for the two learning models usable in captioning systems suggested in this question, NIC and Neural Talk 2 we could have a system resource allocation like this:

  • 30% NIC Training
  • 5% NIC Hyper parameter tuning
  • 45% NT2 In use
  • 15% NT2 Evaluation
  • 5% NT2 Hyper parameter tuning

Samples may be pulled from a pool of samples that have been vetted. That pool may be augmented by real images passing through the system, filtered in accordance with security criteria to avoid external attempts at control.

In the assignment of resources, the sample pool selection criteria must be specified. If the system is already at 100%, the model-state combinations from which the resources shall be drawn must also be specified.

Handling Multiple Output Options

Since there may be more than one model in use and each model may have zero, one, or multiple caption suggestions for each image, each with a reliability measure, the outputs must be analyzed to provide the best choice to associate with the image being analyzed. Additional system criteria must cover this process scenario. For any given image, final evaluation must follow the following general guidelines.

  • If multiple models produce similar or exactly the same captions, they most be weighted higher in final evaluation.
  • If the model's reliability is proven in actual use based on feedback from end users, the model's output must be weighed higher accordingly
  • Re-entrant models (such as reinforcement learning network models) must have access to the end user feedback for additional learning even in in-use state
  • Clear winners are chosen
  • Close races are disambiguated through trained functionality
  • Exact ties are broken through pseudo random index generation

Another artificial network may be placed at the output and appropriate encoding and normalization may be applied before training so that a properly trained network, converged using a quantification of the above additional criteria, can select the best caption from the options for each image.

Phased Development Approach

Phase one of such a system would likely require manual handling of model-state allocation. Phase two would be semi-automation. The location of new models would still require expert attention. Perhaps further in the future a hunt for new models could be automated too.

  • 1
    $\begingroup$ What about the dataset? MSCOCO is just for research usage, right? How to collect a dataset for commercial use? What about the dataset's quantity, quality and collecting method? $\endgroup$ Sep 12 '18 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.