Can current trends and tools, in the field of machine learning, replicate the complexity of financial market? If yes, then what are the tools available in this domain.
Q. I am trying to build a model to infer results from stock market using the concept to create a graph on the companies enlisted. Can anyone suggest me approaches to do so?
In one way the answer is NO. You can't incorporate all the necessary details in an AI program to correctly predict the financial market, at least not in currently. A particular event is a result of many actions and events from the past. Everything is like a chain.
You might have heard of the ButterFly Effect. It simply states that small things can have very large effects. Like flap of butterfly's wings leading to a hurricane. You can read more about it here.
Now back to your question. I believe you are somewhat familiar with the financial system and stocks. Close your eyes and try to imagine all the data that is being churned into machines. Apart from that, there are many decisions that lead to changes in the financial market directly. Like policies made by the government deals bagged by the company related to imports and exports, death of the main employee (like CEO, founder), and other many things. Apart from that, there are uncountable things that indirectly affect the market. I want you to think about it for a while. Now consider doing this for all the stocks present. After even stock values of one are dependent on the others in one way or another.
This amount of data, in other ways considering all these things, is not possible for current generation computers to process. You might have a hit with quantum computers coming though.
Now what you can do, is try to make a Deep Network (with Reinforcement Learning) that can predict the behavior and shift in the direction of some stocks. For example, whether it will go up for down or try to keep a constant line on the graph.
Selection of Features is always an important key to the success of Machine Learning algorithms. Another problem that stock market-related AI programs face is that Economics is itself an emerging and novice field of science. It's not as mature as physics, chemistry or biology. Thus finding out and deciding over the choice of features is another difficult job.
I found this paper online. Haven't given a look to it but, I believe this might help you a bit.
Reviewing the Question
There are multiple questions contained within this posted question. (One of the sentences end with a period, but it is clearly intended to be a question.) All are good questions and fairly easy to answer, assuming that the word 'replicate' can be replaced with 'model' or 'simulate'. (Because the financial world is chaotic, any meaningful replication would likely require a quantum level reproduction of earth and everyone and everything on it.)
The kind of modelling, analysis, and visualization of results is done all the time in the research we do. Approaches and proof of concepts have been provided to insurance, banking, and health organizations along these lines and can be discussed here in general terms within the constraints of any confidentiality agreements.
Restating the Questions
It is best if I restate the questions the way I understand them from the information available in the original post to ensure I understand what the questioner wishes.
Please indicate if my restatement distorts any of the intent contained in the original three questions.
A graph comprised of vertices and edges commonly represents relationships between legal entities. Applying this visualization to represent probable relationships between legal entities from matrices of historical market data is quite possible and may have many uses for financial analysis. Such a graph can be visualized using GraphViz, Mathematica, Matlab, or various libraries available for use from programming environments of Python, C++, Java, LISP, JavaScript or other languages.
Vertices
Instead of vertices representing legal entities registered as tax entities, as in many of the web services that display graphs from public records and purchased aggregated corporate data, the vertices in the graph presumably envisioned by the questioner would represent tradable securities. The attributes of such vertices might be.
Edges
Edges represent the probable strength of financial connectedness between any two vertices representing two tradable securities.
Because the nature of relationships and the associated details between the corporations offering tradable securities and the mindsets of all the trading agents are obscured, relationships must be inferred naively (without factual knowledge of causality), perhaps using the probability relations of Rev. Thomas Bayes (1701 – 1761) or other more sophisticated methods (some of which cannot, for legal reasons, be detailed here).
Relational models must be created (likely more than one) to capitalize on identifiable features in the trading metrics of one of two selected vertices and match that feature with the same or another feature in the other of the two vertices. The correlation must be statistical and designed in such a way as to be resistant to effects outside the relationship between the two legal entities associated with the two tradable securities.
Naive Bayesian classification, other statistical approaches, FFTs, or neural nets may apply to assist with achieving a functional correlation value. Windowing the data in a loop will be necessary to implement sensitivity to single events sparsely spaced in the time domain of the historical data.
To attempt to guess causality, you will need to apply different temporal shifts to see if the feature of one preceded the feature of the other and by how much. (If event A in security B preceded event C in security D, and this pattern repeats over a range of months or years, then there is a probability greater than zero that event C was caused by event A.
The science (and perhaps the art) of creating a set of potential mathematical models of how various corporate and trading relationships may have impacted historical trade metrics between any two securities is the first hurdle in this proposed best approach.
Using various known methods, a probability distribution of the single dimensional or multi-dimensional strength of the relationship, for each of the proposed models, can be calculated from the historical data of the two entities between which one of the many inter-vertex analyses is occurring.
Statistics of these distributions would then be the attributes of the edge shown between two vertices. For more intuitive usability, the following attributes would need to be available via point and click drill down for each edge and each model tried between the two tradable securities connected by the edge.
Measures to Make Computation Time Practical
To accomplish the above each vertex would ideally be compared with each other vertex, for each model, iterating through temporal parameters of the model to determine relationships involving time delays.
If there were a hundred tradables to consider, ten probabilistic relationship models, a thousand temporal permutations that must be tried to converge on a good fit between each model and the historical data, a hundred iterations to converge for each temporal window, a window of a thousand temporal observations, ten thousand windows to cover the entire range of historical data, and a thousand cycles for each test of fit, the primary computations would be 100 x 99 x 10 x 1000 x 100 x 1000 x 10,000 x 1000 = 99 x 10^18 CPU cycles.
(The number 99 comes from the fact that, without some permutation elimination scheme, the histories of each of 100 vertices must be compared with those of the other 99.)
Several methods may be applied together to reduce this set of expanded permutations to permit batch process completion after the close of the market in NY or Hong Kong and before the time zone dependent dawn.
Prediction
Once models are generated and functional and some of the visualized constructs can be verified, then the relational models can be used to predict probable events before they occur. This may seem like science fiction to some, but we predict physical, social, and economic events all the time.
In the case of the profitability in relation to markets, if such predictive tools were to be distributed to eleven other traders, the ability to use the tool to generate profit would almost immediately deteriorate to one twelfth in monetary value.
In fact, this is probably the state of the market today. Only those with automated tools are probable winners, funneling money from those without tools.
The AI Research Perspective
Although the above does not seem like AI the way it is described, often what is conceived as an intelligent agent and appears intelligent in behavior after deployed, refined, and tuned, appears like straight software engineering when one gets into the details of implementation.
Furthermore, if the method for interfacing with the models used to match features in the history of two tradables is generalized so that arbitrary models can be added or modified at will without damaging the effectiveness of job execution, one can build some sort of analogy of a genetic algorithm to search for models that exhibit higher correlations and therefore progressively enhance predictive capabilities.
Meta Modelling
At this point in development, model development is still largely up to the researcher. However, once a model interface, perhaps employing the bridge and facade design patterns, is developed, it is possible to generalize the concept of historical feature correlation between two tradables as models with a set of mutation operations and develop concurrent processes that employ an automated experimental test fixture to develop new models without programmer intervention.
Although the details of such meta-modelling cannot, for legal reasons, be detailed here, the meta-model design options naturally become apparent after some experience is gained after implementing and deploying the above approach in a real scenario with actual tradable historical data.
Using Off the Shelf Code, Libraries, and Frameworks
Obviously, there is appreciable monetary value to this type of development, therefore it is unlikely that anyone will post (or even sell) code specific to this domain. However using super-computing platforms, basic analysis algorithms such as FFT functions, and statistics packages with correlation coefficient routines, naive Bayesian capabilities, and convergence detection support will certainly assist in reducing the development effort required to implement and test this approach or others like it.
You can actually build a statistical model (not exactly and AI) to predict the stock market trend. But the prediction accuracy will be very bad.
The accuracy of your model depends on the following things.
NOTE: This is not in anyway a correct answer to your question. I just wanted to state that anything and everything in this universe can be modeled given that your are aware of all the variables which affect it and how they interact with each other.
