need a large human-labeled training set
brittle (doesn't work well with examples that are in a different genre from the training set)
only requires a small set of labeled data (seed relations)
complex iterative process
It's perfectly reasonable to apply 'traditional' Deep Learning approaches to try and learn an adjacency matrix (a matrix is just a vector of vectors, which can be flattened into a single output vector) but you might need a lot of training data as N gets larger.
Your outputs could certainly have the form of an adjacency matrix, as you describe. Whether it's ...