Reviewing the Question
There are multiple questions contained within this posted question. (One of the sentences end with a period, but it is clearly intended to be a question.) All are good questions and fairly easy to answer, assuming that the word 'replicate' can be replaced with 'model' or 'simulate'. (Because the financial world is chaotic, any meaningful replication would likely require a quantum level reproduction of earth and everyone and everything on it.)
The kind of modelling, analysis, and visualization of results is done all the time in the research we do. Approaches and proof of concepts have been provided to insurance, banking, and health organizations along these lines and can be discussed here in general terms within the constraints of any confidentiality agreements.
Restating the Questions
It is best if I restate the questions the way I understand them from the information available in the original post to ensure I understand what the questioner wishes.
- What are the current trends in modelling and the production of predictive results from historical market data?
- To what degree can the complexity of financial landscape be simulated?
- What are the best approaches to create a graph that represent aspects of the financial market given a list of tradable securities?
Please indicate if my restatement distorts any of the intent contained in the original three questions.
A graph comprised of vertices and edges commonly represents relationships between legal entities. Applying this visualization to represent probable relationships between legal entities from matrices of historical market data is quite possible and may have many uses for financial analysis. Such a graph can be visualized using GraphViz, Mathematica, Matlab, or various libraries available for use from programming environments of Python, C++, Java, LISP, JavaScript or other languages.
Vertices
Instead of vertices representing legal entities registered as tax entities, as in many of the web services that display graphs from public records and purchased aggregated corporate data, the vertices in the graph presumably envisioned by the questioner would represent tradable securities. The attributes of such vertices might be.
- Exchange
- Unique exchange ID (symbol)
- Name
- An array of vectors of historical trade metrics (with each vector containing a UTC time stamp)
Edges
Edges represent the probable strength of financial connectedness between any two vertices representing two tradable securities.
Because the nature of relationships and the associated details between the corporations offering tradable securities and the mindsets of all the trading agents are obscured, relationships must be inferred naively (without factual knowledge of causality), perhaps using the probability relations of Rev. Thomas Bayes (1701 – 1761) or other more sophisticated methods (some of which cannot, for legal reasons, be detailed here).
Relational models must be created (likely more than one) to capitalize on identifiable features in the trading metrics of one of two selected vertices and match that feature with the same or another feature in the other of the two vertices. The correlation must be statistical and designed in such a way as to be resistant to effects outside the relationship between the two legal entities associated with the two tradable securities.
Naive Bayesian classification, other statistical approaches, FFTs, or neural nets may apply to assist with achieving a functional correlation value. Windowing the data in a loop will be necessary to implement sensitivity to single events sparsely spaced in the time domain of the historical data.
To attempt to guess causality, you will need to apply different temporal shifts to see if the feature of one preceded the feature of the other and by how much. (If event A in security B preceded event C in security D, and this pattern repeats over a range of months or years, then there is a probability greater than zero that event C was caused by event A.
The science (and perhaps the art) of creating a set of potential mathematical models of how various corporate and trading relationships may have impacted historical trade metrics between any two securities is the first hurdle in this proposed best approach.
Using various known methods, a probability distribution of the single dimensional or multi-dimensional strength of the relationship, for each of the proposed models, can be calculated from the historical data of the two entities between which one of the many inter-vertex analyses is occurring.
Statistics of these distributions would then be the attributes of the edge shown between two vertices. For more intuitive usability, the following attributes would need to be available via point and click drill down for each edge and each model tried between the two tradable securities connected by the edge.
- Median relationship strength
- Mean relationship strength
- Standard deviation of relationship
- Direction of causality
- Median delay in causality
Measures to Make Computation Time Practical
To accomplish the above each vertex would ideally be compared with each other vertex, for each model, iterating through temporal parameters of the model to determine relationships involving time delays.
If there were a hundred tradables to consider, ten probabilistic relationship models, a thousand temporal permutations that must be tried to converge on a good fit between each model and the historical data, a hundred iterations to converge for each temporal window, a window of a thousand temporal observations, ten thousand windows to cover the entire range of historical data, and a thousand cycles for each test of fit, the primary computations would be 100 x 99 x 10 x 1000 x 100 x 1000 x 10,000 x 1000 = 99 x 10^18 CPU cycles.
(The number 99 comes from the fact that, without some permutation elimination scheme, the histories of each of 100 vertices must be compared with those of the other 99.)
Several methods may be applied together to reduce this set of expanded permutations to permit batch process completion after the close of the market in NY or Hong Kong and before the time zone dependent dawn.
- Filtering and then decimating (removing redundancy) the historical data
- Truncating the historical data to analyze only the recent (and therefore the most relevant) historical data
- Widening the error margin to only what is displayed (such as two significant figures)
- Optimizations of algorithms, the mathematics behind them, the machine instruction representation of computations, or the mapping of values to data types
- Distributing analysis processes to take advantage of parallel computing
- Limiting the list of tradable securities using a narrowing set of inclusion criteria
- Early elimination of possible edges between unlikely relational candidates using heuristics, model simplifications, neural nets, or fuzzy logic
Prediction
Once models are generated and functional and some of the visualized constructs can be verified, then the relational models can be used to predict probable events before they occur. This may seem like science fiction to some, but we predict physical, social, and economic events all the time.
In the case of the profitability in relation to markets, if such predictive tools were to be distributed to eleven other traders, the ability to use the tool to generate profit would almost immediately deteriorate to one twelfth in monetary value.
In fact, this is probably the state of the market today. Only those with automated tools are probable winners, funneling money from those without tools.
The AI Research Perspective
Although the above does not seem like AI the way it is described, often what is conceived as an intelligent agent and appears intelligent in behavior after deployed, refined, and tuned, appears like straight software engineering when one gets into the details of implementation.
Furthermore, if the method for interfacing with the models used to match features in the history of two tradables is generalized so that arbitrary models can be added or modified at will without damaging the effectiveness of job execution, one can build some sort of analogy of a genetic algorithm to search for models that exhibit higher correlations and therefore progressively enhance predictive capabilities.
Meta Modelling
At this point in development, model development is still largely up to the researcher. However, once a model interface, perhaps employing the bridge and facade design patterns, is developed, it is possible to generalize the concept of historical feature correlation between two tradables as models with a set of mutation operations and develop concurrent processes that employ an automated experimental test fixture to develop new models without programmer intervention.
Although the details of such meta-modelling cannot, for legal reasons, be detailed here, the meta-model design options naturally become apparent after some experience is gained after implementing and deploying the above approach in a real scenario with actual tradable historical data.
Using Off the Shelf Code, Libraries, and Frameworks
Obviously, there is appreciable monetary value to this type of development, therefore it is unlikely that anyone will post (or even sell) code specific to this domain. However using super-computing platforms, basic analysis algorithms such as FFT functions, and statistics packages with correlation coefficient routines, naive Bayesian capabilities, and convergence detection support will certainly assist in reducing the development effort required to implement and test this approach or others like it.