The aggregation of multiple independently acquired supplier databases is a common problem that results from either multiple feeds or acquisitions. The goal is to identify real legal entities and their physical addresses and methods of contact across the set of databases with potentially differing schemas and data entry forms and personnel. The commonality is the actual relationships between corporations, sites, addresses, personnel, phone systems, email systems, and other related business and communications features.
A supplier [may] be common across [independent databases]
That the businesses are all suppliers narrows the domain somewhat, but may not affect the approach much.
We have analysts go through the system manually to try identify common suppliers by name and address.
The intention appears to be to train an artificial network to replace the manual operations identifying common suppliers or at least assist and reduce the cost of such operations. Training examples are critical input to the training process of such an artificial network.
If the before state presented to the manual operatives along with their identifications of matched identities have been stored, that is a good training example set. If overturns of those choices were made and those overturns were stored, that is another even more exacting set of labels that can be used for network training.
Most applications of artificial networks to this task require a few operations that, if done efficiently, can drastically reduce the cost of aggregating the databases into one or linking between them if the choice is to leave them independent for the short term. These operations are required for all such aggregation, whether for suppliers, vendors, customers, or other business roles. There is also much in common between business aggregation and the aggregation of records from multiple sources on people.
These are the typical front operations.
- Converting input forms into a common character encoding and schema of one or more comprehensively representative record types across all data sources
- Data cleaning, which includes the elimination of useless or duplicate records, correction of obviously broken records, and standardization of the representation of values of a particular feature value
- Identity likelihood measurements based on exact or approximate matches of text, IDs (such as EIN), primary web domains, phone number ranges, or other key identifying fields
- Determination of likelihood of identity between two records in light of the full array of matching fields and their individual likelihoods
Each of these operations involves transformations that can be parameterized. Finding as close to an optimal state for these parameters and building mapping tables to assist in matching can produce a reliable approximation of human matching. In some cases, the automated version can exceed the manual version of this process in terms of reliability. Furthermore, the automated process can be real time and consume few computing resources, removing the need for any batch processes either manual or automatic.
In greater detail, the following lower level processes can be accomplished through artificial networks, especially in combination with fuzzy logic theory (where syntactic transformations, perhaps regular expression search and replace operations, can be have probabilities of equivalence).
- Equivalence of addresses in different syntactic forms
- Overlapping zip codes or phone number exchanges
- Repair of records that do not fit into standard forms
- Likelihood that a match of a specific features infers full entity match
- Likelihood that a partial feature match infers entity match
There are several other functions that can be parameterized and tuned from good labeled data from the manual processes. If the manual process was not monitored properly and saved, it is also possible to train the various fields based on some highly reliable matching parameter or set of them. These are a few examples.
- Tax authority and ID
- DUNS ID
- International Suppliers Network ID
- ISO 9362 Business ID
- Central office address, suite number, and date