A CriticalAssessmentof State-of-the-Art in EntityAlignment
MaxBerndorf LdwigWack adEvgeniyFam
Ludwig-Maximilians-Universitit Minchen Munich Germany{berrendorf faernan}@dbs.ifi .1mu.de 1.wacler@canpus . lnu.de
Abstract. In this work we perform an extensive investigation of two state-of-the-art (SotA) methods for the task of Entity Alignment inKnowledge Graphs. Therefore we first carefully exasine the bench- marking process and identify several shortings making the resultsreported in the original works not always parable. Furthermore we suspect that it is a mon practice in the munity to make thehyperparameter optimization directly on a test set rexducing the infor-mative value of reported performance. Thus we select a representative sample of benchmarking datasets and describe their properties. We alsoexasmine different initializations for entity representations since they are a decisive factor for model performance. Furthermore we use a sharedtrain/validation/test split for an appropriate evaluation setting to evalu- ate all methods on all datasets. In our evaluation we make several inter-esting findings. While we observe that most of the time Sot A approaches perform better than baselines they have difficulties when the dataset contains noise which is the case in most real-life applications. Moreover in our ablation study we find out that often different features of SotA method are crucial for good performance than previously assumed. Thecode is available at httpe://mberr/ea-sota- parison.
Keywords: Knowledge Graph - Entity Alignment - Word Embeddings
1Introduction
of information. Knowledge Graphs (KGs) often serve as such data structure [6]. The quality of information retrieval crucially depends on the accessible storageMoreover to satisfy diverse information needs a bination of multiple data sources is often inevitable. Entity Alignment (EA) [2] is the discipline of align-ing entities from dlifferent KGs. Once aligned these entities facilitate informationtransfer between knowledge bases or even fusing multiple KGs to a single knowl- edlge base.
identify which factors are essential for its performance. Although papers often use In this work our goal is to analyze a SotA approach for the task of EA andthe same dataset in the evaluation and report the same evaluation metrics theselection of SotA is not a trivial task: as we found out in our analysis the usage of different types of external information for the initialization or train/test splits
of different sizes² makes the results in different works inparable Therefore factors among strongly performing methods in multiple works: while still guided by the reported evaluation metrics we identified these mon
They are based on Graph Neural Networks (GNNs). GNNs build the basis of the most recent works [16 9 12 21 22 4 23 25 10 14 7 17 20 18 19].They utilize entity names in the model. Supported by recent advances in word embeddings these attributes provide distinctive features.They consider different types of relations existing in KGs. Most GNNs ignorediferent relationship types and aggregate them in the preproceing step.
Given these criteria we selected Relation-aware Dual-Graph Convolutional Net- work (RDGCN) [17] as it also has demonstrated impressive performance inlished Deep Graph Matching Consensus (DGMC) [7] method in our analysis for recent benchmarking studies [15 24]. Additionally we include the recently pub-two reasons: the studies mentioned above did not include it and the authors sap po s po ad po sms poa make use of relation type information.
tializations based on entity names. Although both methods utilize entity names We start our studly by reviewing the used datasets and discussing the ini-the actual usage differs. For parison we thus evaluate both methods on alldatasets with all available initializations We also report the zero-shot perfor- mance i.e. when only using initial representations alone as well as a simpleoptimization. Related works often do not discuss how they chose hyperparam- GNN model baseline. Furthermore we adress the problem of hyperparametereters and e.g. rarely report validation splits. So far this problem was not ad-dressed in the runity. In the recent prehensive survey [15] the authors use cross-validation for the estimation of the test performance. The models arelected by not reported procedure. Also in the published code of the investigated either evaluated with hyperparameters recormmended for other datasets or se-approaches we could not find any trace of train-validation splits raising ques-tions about reproducibility and fairness of their parisons. We thus create a shared split with a test train and validation part and extensively tune theensure that they are suficiently optimized. Finally we provide an ablation study model's hyperparameters for each of the dataset/initialization binations tofor many of the parameters of a SotA approach (RDGCN) giving insight intothe individual ponents² contributions to the final performance.
2Datasets &Initialization
Table 1 provides a summary of a representative sample of datasets used forbenchmarking of EA approaches. In the following we fist discuss each datasetsproperties and in the second part the initialization of entity name attributes.
Table 1. Summary of the used EA datasets. We denote the entity set as & the relationset as R the triple set as T the aligned entities as A and the exclusive entities as .
DBP15k datasert subset graph [R| (7) [ 4| [X]zh-en ja-en 日 xh 19 814 19 388 19 572 1 701 1 323 1 299 70 414 95 142 77 214 15 000 15 000 15 000 4 572 4 388 4 814fr-m 19 780 19.661 1 153 903 105 998 93 484 15 000 15.000 4 780 4 661WK3115k en-de en 19 993 15 126 1 841 1 208 209 041 115 722 15 000 9 783 5 343 4 993en-fr en fr 15 169 14 603 15 393 2 228 2 422 596 203 356 144 244 169 329 10 021 7 375 7 284 7 794 4 582 8 109OpenEA en-de 15.000 15 000 169 96 84 867 92 632 15 000 15 000 0en-fr fr en 15 000 15 000 193 166 96 318 $0 112 15 000 15 000 0dy d-w d 15 000 15 000 15 000 167 72 21 68 063 60 970 73 983 15 000 15 000 15 000 0 0W 15 000 121 83 365 15 000 0 0
2.1 Datasets
of EA approaches. It has three subsets all of which base upon DBpedia. Each DBP15k The DBP15k dataset is the most popular dataset for the evaluationsubset prises a pair of graphs from dlifferent languages. As noted by [2] therediffering in the number of exclusive entities in each graph. The alignments in exist multiple variations of the dataset sharing the same entity alignment butthe datasets are always 1:1 alignments and due to the construction method for the datasets exclusive entities do not have relations between them but onlyto shared entities. Exclusive entities plicate the matching process and inreal-life applications they are not easy to identify. Therefore we believe that this dataset describes a realistic use-case only to a certain extent. We foundtory² having a different set of aligned entities This is likely due to extraction another different variant of DBP15k as part of the PyTorch Geometric reposi-of alignments from data provided by [20l via Google Drive² as described in theirGitHub repository.d As a result the evaluation results published in [7] are not directly parable to other published results. In our experiments we use theit is the predominantly used variant. (smaller) JAPE variant with approximately 19-20k entities in each graph since
DBPedia YAGO and Wikidata obtained by iterative degree-based sampling to s esopnd spvu gmatch the degree distribution between the source KG and the extracted subset.
Table 2. The statistics about label-based initialization in the OpenEA codebase:tributes". id denotes initialization with the last part of the entity URI. For dy this aribute denotes initialization via attribute values for a predefined set of *name at-basically leaks ground truth whereas for Wilkidata the URI contains only a numeric identifier thus rendering the initialization “label" useless.
subaet d d side via attribrutel 0 15 000 via id via id (%) 100.00%d-y 3 15 000 2 883 8 391 12 122 7 301 0 48.67% 50.81% 0.00%Y
The alignments are exclusively 1:1 matchings and there are no exclusive enti-ties Le. every entity occurs in both graphs. We believe that this is a relatively unrealistic scenario. In our experiments we use all graph pairs with 15k entities(15K) in the dlense variant (V2) i.e. en=de15k=v2 en=fr15k=v2 dy15k=v2 d=v=15k=v2.
WK3I15k The Wk31 datasets are multi-lingual KG pairs extracted from Wikipedia.graphs contain aditional exclusive entities and there are m:n matchings. We As in [2] we extract additional entity alignments from the triple alignments. Theonly use the 15k variants where each graph has approximately 15k entities. There are two graph pairs en=de and en=fr. Moreover the alignments in thedataset are relatively noisy: for example en=de contains besides valid alignmentssuch as ("trieste “triest”) or ( frederick i holy roman emperor" *friedrich 1. (hrr)") also ambiguous ones such as (1" “1. fc saarbricken") (*1" “1. fcnoise aggravates alignment it also reflects a realistic setting. schweinfurt 05°) and errors such as (*1" *157" ) and (“101" *100°). While the
2.2 Label-Based Initializations
Prepared trunslations (DBP15k) For DBP15k we investigate label-based initial-izations based on prepared translations to English from [17] and [7] (which in turn originate from [20]. Afterwards they use Glove [11] embeddings to obtainan entity representation. While [17] only provides the final entity representationvectors without further describing the aggregation [7] splits the label into words (by white-space) and uses the sum over the words’ embeddings as entity repre-length. sentation. [17] additionally normalizes the norm of the representations to unit
Prepared RDGCN Embeddings (OpenEA) OpenEA [15] benchmarks a large vari- ety of contemporary entity alignment methods in a unified setting also includingRDGCN [17]. Since the graphs DBPedia and YAGO collect data from similarsources the labels are usually equal. For those graph pairs the authors pro- pose to delete the labels. However RDGCN requires a label based initialization.
Thus the authors obtain labels via attribute triples of a pre-defined set of “name-attrilbutes**: skos:prefLabel for DBPedia-YAGO and . org/entity/P1476 for DBPedia-Wikidata.
is not found via attribute the last part of the entity URI is used instead. For However when investigating the published code we noticed that if the labelDBPedia/YAGO this effectively leaks ground truth since they share the samesince their labels are the Wikidata IDs e.g. Q3391163. Table 2 summarizes the label. For DBPedia/Wikidata this results in useless labels for the Wikidata sidefrequency of both cases. For du DPBedia entities always use the ground truth label. For 49% of the Wikidata entities useless labels are used for initialization.For d-y YAGO entity representations are always initialized via an attribute triple. For DBPedia in 81% of all cases the ground truth label is used. We storethese initial entity representations produced by the OpenEA codebase into a fileand refer in the following to them as Sun initialization (since they are taken from the implementation of [15]).
Mult-lingual BERT ( WK3l15k) Since we did not find related work with entityembedding initialization from labels on WK3l15k we generated those using aFollowing 5] we use the sum of the last four layers as token representation since pre-trained multi-lingual BERT model [5] BERTBase Multilingual Cased.it has parable performance to the concatenation at a quarter of its size. To summarize the token representations of a single entity label we explore sum mean and max aggregation as hyperparameters.
3Methods
We evaluate two SotA EA methods RDGCN [17] which we reimplemented andpdepe qm uoeu poa ejao a pas aa m nog [ evaluation. In the follwing we revisit their architectures and highlight differ- ences between the architecture described in the paper and what we found in thepublished ocde.
Similarly to all GNN-based approaches both models employ a Siamese ar-chitecture. Therefore the same model with the same weights s applied to bothgraphs yielding representations of entities from both KGs. Given these entity representations the EA approaches pute an affinity matrix that describesthe similarity of entity representations from both graphs Since the main differ- ence between methods is the GNN model in the Siamese architecture for brevitywe only describe how it is applied on a single KG g = ( R 7).
3.1 Relation-aware Dual-Graph Convolutional Network (RDGCN)
Architecture The RDGCN [17] model prises two parts performing message-passing processes applied sequentially. The message passing process performed
