As part of this semester assignment , I'm working on a project that aims to to represent the knowledge in "PMBOK 6th edition, section 11: Project Risk Management (page 395 -> 458)" and the knowledge in "PMI standard for Risk Management " (128 pages) as a knowledge graph. The generated knowledge graph will be used later to build recommendation system to infer real-time personalized recommendations.
I have been reading on how to convert unstructured text into a knowledge graph in research papers and articles and I have found mainly 3 ways to do this:
1/ Using a joint of Named-entity recognition (NER) and Relation Extraction (RE) to extract the entities and the relations from your unstructured text. 2/ taking advantage of the linguistic knowledge of Transformer models and fine tune a transformer model (BERT, T5) for the task of extracting entities and relations. I could find some pretrained models like REBEL :https://github.com/Babelscape/rebel. 3/ use prompt engineering (LLM (GPT)) to generate the knowledge graph.
I could not find any of the three approaches as good as I wanted:
1/ The majority of the resources I have found that tackles the first approach (NER & RE) showcase simple tasks where the named entities and relations are very straightforward. Example this article here: https://freedium.cfd/https://medium.com/mlearning-ai/building-a-knowledge-graph-for-job-search-using-bert-transformer-8677c8b3a2e7 where the entities are [skills, Diploma, Major, Year of experience] and the relations [DEGREE_IN, EXPERIENCE_IN..]. In this case, training NER and RE models will be easy. But in my case, determining entities and relations is very complex. Annotating the corpus manually is incredibly tedious and labor-intensive (Could not even determine what are the entities and the relations) . You can grab a feeling of how complex it is by looking at how big our dataset is (191 pages of knowledge) and how complex the knowledge is in the corpus (many definitions, a lot of terminology...)
2/ I have used the pretrained REBEL model but the results looked weird. (redundant relations, sometimes the extracted relations make non sense). And So I wanted to fine tune BERT for this specific task on my custom data (PMBOK, PMI) but I really could not understand how to do this (what should be the data format to train and test the transformer model?, how to evaluate the model ?...)
3/ The fact that LLM are stochastic models, a lot of variations in the generated graphs for each prompt (sometimes huge differences) and this lead to huge ambiguity because I cannot evaluate how good the graph is in representing the knowledge.
I'm open to any other resources and any other inspirations or approaches to tackle this project. Thank you in advance.