1
$\begingroup$

I have a dataset about (240000,23). For my task, I have to use an unsupervised learning method and apply it on every single column separately in order to detect anomalies that might exist. I have pre-processed the data and I am visualizing the TimeElapsed vs the parameter in Python (The graphs look like the one shown in an earlier post by me here).

I am wondering if there is a way wherein after the graphs are plotted, the graphs are compared with each other and then clustered together based on their similarities.

Example: If I have temporal data of about 200 products, I plot the graphs of all the 200 products (as can be seen in the link provided above) and these 200 graphs must be compared with each other and must be separately plotted as 200 different points on a scatter plot (by using some unsupervised learning techniques) based on how similar are the graphs of different products to each other.

I don't know if the code that I have would be helpful in guiding me, but the code that I have is here:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1234)

dataset = pd.read_csv('Temporal_DataTable.CSV', header = 0)
dataset != 0
(dataset != 0).any(axis=0)
dataset = dataset.loc[:, (dataset != 0).any(axis=0)]

dataset['product_number'] = pd.factorize(dataset.Context)[0]+1
dataset['index'] = pd.factorize(dataset.Context)[0]+1

cols = list(dataset.columns.values)
cols.pop(cols.index('StepID'))
cols.pop(cols.index('Context'))
cols.pop(cols.index('product_number'))
cols.pop(cols.index('index'))
dataset = dataset[['index','product_number','Context','StepID']+cols]
dataset = dataset.set_index('index')

max_time_ID_without_drop = dataset.groupby(['product_number'])['TimeElapsed', 'StepID'].max()
avg_time_without_drop = np.average(max_time_ID_without_drop['TimeElapsed'])

dataset_drop = dataset.drop(index = [128, 133, 140, 143, 199])

max_time_ID_with_drop = dataset_drop.groupby(['product_number'])['TimeElapsed', 'StepID'].max()
avg_time_with_drop = np.average(max_time_ID_with_drop['TimeElapsed'])

dataset = dataset.drop(columns=['TimeStamp'])
dataset_drop = dataset_drop.drop(columns=['TimeStamp'])

grouped = dataset.groupby('product_number')
ncols = 4
nrows = int(np.ceil(grouped.ngroups/40))
for i in range(10):
    fig, axes = plt.subplots(figsize=(12,4), nrows = nrows, ncols = ncols)
    for (key, ax) in zip(grouped.groups.keys(), axes.flatten()):
        grouped.get_group((20*i)+key).plot(x='TimeElapsed', y=['Flow_Ar-EDGE'], ax=ax, sharex = True, sharey = True)
        ax.set_title('product_number=%d'%((20*i)+key))
        ax.get_legend().remove()
        handles, labels = ax.get_legend_handles_labels()
        fig.legend(handles, labels, loc='upper center')
plt.show()

Thanks in advance for the help.

$\endgroup$
  • $\begingroup$ Do you have the vectors x,y that describe those graphs? you can perform clustering at y vector (if x ocuppies the same time period and in the same density/sampling rate in all graphs). $\endgroup$ – Pedro Henrique Monforte Mar 26 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.