Is private research with low funds still feasible in the field of AI?

Question

My question assumes that a private researcher doesn't have access to anything stronger than a modern PC with a high end GPU to implement his projects. He can also use cloud computing but with limited funds as well.

Is it still feasible to do research with those restrictions? AlphaGo used 1,202 CPUs and 176 GPUs to beat Lee Sedol. Is this enormous power only required to achieve the final optimizations or have we already reached a state where high end research can only be done with larger funding?

Please include in your answer examples of recent research results and the required infrastructure that was used to create them.

Answer 1

Yes, you can do research in Artificial Intelligence with low funds (but you need a lot of time!). Notably, because AI is not the same as applied machine learning (indeed running ML programs on big data requires a lot of computer power). For example, knowledge representation and reasoning or natural language processing (both are subfields of AI) generally don't require a lot of computer power. Even when interested in machine learning (which is not the same as AI, just a subfield of it among others), you can use a powerful PC or laptop for that.

A lot of recent papers (probably most of them) in journals like Artificial Intelligence, or in conferences like IJCAI (see their past proceedings) are somehow theoretical, and when something is implemented, it runs on a laptop or desktop. Notice that both AI journal and IJCAI conferences are peer-reviewed with a very selective process.

^{Actually it is difficult and rare to find a research paper in AI mentioning that costly equipment was needed to do the research. Costly supercomputers used for research are generally not used by AI researchers (but by researchers in physics or bioinformatics), and AI researchers often don't even have access to such facilities.}

For examples, recent IJCAI2016 papers such as Coco: Runtime Reasoning About Conflicting Commitments, Interdependent Scheduling Games, Control of Fair Division, Verifying Pushdown Multi-Agent Systems against Strategy Logics, etc. don't mention any costly computation. Actually, it is likely that most recent papers don't use and don't need large scale costly cloud computing. And some of that research might not have been implemented (perhaps by some intern) in anything more than a toy prototype.

---

Please include in your answer examples of recent research results and the required infrastructure that was used to create them.

^{Here are some recent research publications, they all explicitly mention the needed computer equipment, follow the link to read them:}

Event recent IJCAI experimental papers like A Multicore Tool for Constraint Solving, Compiling Constraint Networks into Multivalued Decomposable Decision Graphs, External Memory Bidirectional Search, Multiple Constraint Acquisition, Completion of Disjunctive Logic Programs, Eliminating Disjunctions in Answer Set Programming by Restricted Unfolding and On the Empirical Time Complexity of Random 3-SAT at the Phase Transition mention at most a multi-core workstation (e.g. at most a dual Xeon socket workstation, often a cheaper laptop or desktop). I guess that most authors need their GPU only as a graphics card, to use their display screen. BTW, papers in other journals, such as JAIR (like Improving the Efficiency of Dynamic Programming on Tree Decompositions via Machine Learning), when they mention computer equipment, gives similar kind of machines (laptop or at most high-end desktop).

^{(Actually, I don't remember having read an AI paper mentioning costly computing equipment; and I believe the reason for that is that in the academic community access to supercomputers is nearly reserved to research in other domains: physics, bioinformatics, etc. For an AI researcher gaining such access is difficult and uncommon; BTW, in H2020 European research grants computing cost above 15% of the labor cost needs to be dully justified, so is exceptional)}

However, you'll better publish your software as free software or open-source, and you need a lot of time (preferably full-time, or at least half-time) to do the research work, publish it, and follow outside progress in your area. BTW, contacting a nearby university could be helpful (you could attend some seminars, etc.)

You can find interesting blogs (e.g. the one of J.Pitrat) of AI researchers working on just a laptop or a desktop.

So, most research in AI (even in Machine Learning, read papers from JMLR) are done by researchers working on a laptop or desktop. I am not even sure you need a powerful GPU to do the research. You certainly can do interesting research with a desktop computer (probably running Linux, with 32Gb RAM, some AMD Ryzen or Intel i5 or i7) costing in 2017 less than 1500€ or maybe 2500€ (and sometimes a low end laptop is enough). In rare cases (a small minority of papers), you might need a dual-socket workstation (something costing perhaps 5000€).

Notice that Big Data is not considered as a field of Artificial Intelligence. I guess (but don't really know) that research in Big Data requires access to more computer power than a PC. BTW, I am not sure that AlphaGo is really a research project, it is more an industrial demo.

PS. See also http://norvig.com/21-days.html (notice who is the author!) and this Bismon draft report (developing programming techniques, in a GPLv3+ alpha-stage software prototype, notably frame-based or semantic-network related ones, with reflection and introspection, reusable and relevant to AI), and the RefPerSys project.

Answer 2

It is totally practical and encouraged to do research on even common laptops. Many questions about AI can be addressed using this architecture and for a reference, look at every AI grad student which may not have access to such a super computer (most of them). AI research is done on laptops. It's the presentation of a system on a difficult problem which may require a supercomputer. It's also the reason why every domain on OpenAI gym can easily run on a laptop, and as a case study, Deepstack is a new AI system which can beat professional poker players and runs on a laptop.

Answer 3

A lot of research can be performed on some pretty basic computers. Not all machine learning research is based on very large neural networks. There is a lot of machine learning research that goes on with other more simple algorithms such as k-means clustering, and softmax regression. These algorithms are pretty basic, and so can run very fast, but they do not require massive super computers to train. Also, a decent high range GPU can train some pretty large neural networks. You can reasonably train some fairly large convolutional neural networks on a high end GPU. While training very large neural networks on millions of examples is a huge task and requires a lot of processing power, not all of machine learning is based around deep learning, and a lot of algorithms run extremely fast when they are GPU accelerated.

Answer 4

The field of AI is vast that there’s always room for small scale research and inquiry. Utility of AI is key, but the potential applications are broad, and intelligence is a spectrum.

Fundamental Combinatronics, a collective with no current funding, is engaged in a project to develop “adaptive AI” for a set of consumer-oriented, combinatorial game products. The requirements are distinct from real-world applications.

We can’t compete with the major players in terms of resources, and we’re late to the party in terms of Machine Learning and Neural Networks, and, because the AI is for a consumer, mobile game which carries significant restrictions in terms of the bounding rationality (networking cannot be assumed; software volume is measured in megabytes; memory is restricted to lowest-common-denominator consumer-grade devices with non-specialized processors.) For these reasons, we re going the opposite direction of current industry trends--the good-old "boring stuff".

Because the automata only need to outperform the average/above-average human player, an old-school, heuristic approach is feasible. (Fun also, because it involves solving non-trivial, partisan Sudoku games in a Combinatorial Game Theory sense, a type of research all on its own. Although the context is ultimately intractable, it is a context automata are well suited for.) Old-school is beneficial in that it’s nice to have an app product with a decent AI that is under 7mb. (No barrier to download or strong incentive to delete from the device. While the new iPad has up to 128gb, only a small subset of players will be willing to devote significant volume for strong AI, and these players represent a distinct, secondary market segment.) It’s not optimal for an AI take up any more volume than is strictly necessary for a given product.

Fuzzy logic should also be useful for its efficiency in terms of applicability under what would today be considered severe computational restrictions.

[M] games are economic so the model is interesting from a Game Theory standpoint in providing a novel, compact, intrinsic and highly mutable mathematical model based positional valuation in n dimensions in conjunction with stability states in a causal/temporal framework. The combinatorial nature of [M] is ideal for quantitative analysis, and the games involve blocking factors (sudoku) and symmetry breaking (even order gameboards). For players > 2 coalitions also become a factor.

The focus of the procedural research is currently in four main areas and what we’re terming “Adaptive AI” :

Dynamic Strength: Sheer strength is not the goal. We’re working on AI that tailor their strength to their human player’s strength and preferences. For most humans, we don’t want the automata to win more than 2/3 games because always losing is no fun and makes the product less "sticky". Even if the human player desires an automata it cannot beat, the automata should only be sufficiently strong to almost always beat their human. AI strength can be limited by restricting rationality (time and memory), which carries an added benefit of energy conservation (less bits flipped), but the rules-based approach is useful in that rules can be recombined combinatorially to produce automata of different strengths and preferences. Automata play against each other to determine strength hierarchies, and identify poor heuristics to be weeded out of stronger automata.

General Intelligence: The automata have to function on an array of related games, where equilibria can be altered in numerous ways without adding mechanics. Additionally, mechanics can be added without altering the nature of the games, such as introducing Graeco-Latin squares. This presents a problem if each configuration has to be learned through intensive self-play because the automata must be able to play at a respectable strengths immediately. Thus the goal is not sheer strength, but consistent strength across the widest array of contexts. (“Respectably weak” and “semi-strong” automata have utility value in that those categorization may be said to describe the majority of human player base.) The idea is an “axiomatic intelligence” that can be extended to include an ever increasing array of contexts.

Counter-Intuition: The automata should not be prone to repetitive play. Initially we’re using limited monte-carlo for positional selection tie-breaking, and the scope can be extend to larger arrays of positions with varying degrees of perceived optimality, up to rational but “counter-intuitive” decisions, which can be subsequently evaluated. This may be useful in adapting to new, dominant strategies that emerge in allowing the automata to experiment with less obvious choices. In situations where the automata is consistently winning, there is incentive to experiment, "investing in loss" in the sense that mistakes are useful from an experience/learning standpoint.

"Genetic" Evolution: The eventual goal is to implement some form of local reinforcement where the automata learn through play against their human, and self play in restricted contexts, such as between turns when playing against their human. With networking enabled, the automata can play against such automata, with the idea of producing strong automata with human play characteristics. (It will be fun when we eventually put these automata up against pure deep learning algorithms in a wide array of [M] contexts with distinct mathematical properties. My money would be on the ML and NN algorithms in sequential games, but in asynchronous games where there is no turn order, it will be interesting to see if the "axiomatic systems" can produce desirable outcomes by making sound decisions faster than smarter, more complex automata;)