Network representation learning:A macro and micro view
Xueyi Liu Jie Tanga b *
Dpor f Cp Scle nTec T sty Be 10004 Ch i Nl L If S Tt ML g 4
A R T I C L E 1 N F O
Abstract
works lke th ranptatwk socil and acadmic nwek anbe resentd by gphs ecent yr Graph is a univere data structure that is widely used to organize data in real-world. Various real-word net-have witnessed the quick development on representing vertices in the network into a low-dimensional vector u osp uu mu pdsreprestati ing Existing alorithm can e cateoized int theegroups shalomedding modes algoriths n th grapdta In ths suvey wendutacmhensverevewf currentleatue nkheterogenes netwok embdding mdels graph neral etwork base mdels. We review atef-thart a gorithms for each category and discuss the essential differemces between these algorithms. One advantage of thesuvey s thte sstematially stdy thdelying thetiafuati detyin th difnt atgris ofalrit whf deght fetning th dt of th w eelearing field.
Neteoek eprnsentation lesening KrywondcGraph neural networks Graph spectral they
1. Introduetion
N is the number of nodes in the network). It is also hard to design an efficient algorithm based just on the adjaoency matrix Taking mu-nity detection as an example most existing algorithms will involve calculating the spectral deposition of a matrix (Fraglkiskos andwith respect to the number of vertices. Existing graph analytical Vazirgiannis 2013) whose time plexity is always at least quadraticmethods like distributed graph data procesing framework (e.g.. GraphX (Gonzalez et al. 2014) and GraphLab (LowYet al 2012) sufferfrom high putational cost and high space plexity. This plexity makes the algorithms hard to be applied to large-scale net-works with millions of vertices.
networks exist in the real-world like the social networks (Aggarwa] Graph is a highly expressive data structure based on which various2011; Myers et al. 2014) citation networks (Sen et al. 2008) biological networks (Marinka et al. 2019) chemistry networks (Martins et al..2012) traffic networks and others Mining information from real-world networks plays a crucial role in many emerging applications. Forexample in social networks clasifying pele int social ommunites according to their profile and social connections is useful for manyrelated task like social recoemmendation target advertising user search (Zhang et al. 2018) etc. In munication networks detecting -oq u uosnp o psan dq uo snnns unlogical networks predicting the role of protein can help us reveal the mysteries of life; predicting molecular drugability can promote new drug development. In chemistry networks predicting the function ofmolecules can help with the synthesis of new pound and new ma-supply effctive analysis. For example the only structural information terial. The way in which networks are generally represented cannotqau sl isn s apou ao jmqe xeo uaoe[pe ue woaj sa ueo am bours and the weight of the edges between them. It is not informativeenough with respect to the neighbourhood structure and its role in the graph and also of high space plexity (ie. O(N) for one node where
Recent years have seen the rapid development of network repre- o s sond a sle u es mative and low-dimensional representations for network vertices whichcan preserve the network structure vertex features labels and otherauxiliary information (Cai et al. 2018; Zhang et al. 2018) as Fig 1 il- lustrates. The vertex representations can help design efficient algorithmseasily applied to vertex representation vectors. since various vector based machine learning algorithms can thus be
the proposed algorithms were part of dimensionality-reduction tech- Such works date back to the early 2000s (Zhang et al. 2018) whenniques (e.g Isomap (B Tenenbaum et al. 2000) LLE (Roweis and Lawrence 2000) Eigenmap (Belkin and Niyogi 2002) and MFA (Yan
Fig. 1. A toy eample for network embedding task Vertices in the netwrk lying in the left part are embedded into d-dimensional vector sace where d is muh q po eu s sg qo pe oq es ens ae as aq qm so au a sapou po aquu e q u skept in the embedding space (e.g Structurally similar vertices E and F are embedded closer to each other than structurally dissimilar vertices C and F).
et al. 2007). These algorithms firstly calulate te affinity graph (e.g. k-nearest-neighbour graph) for the input of high-dimensional data.Then the affinity graph is embedded into a lower dimensional space. However the time plexity of those methods is too high to scale toetal. 2014;Zhangetal2016;Caetal2016focusingondeveloping large networks. Later on there is an emerging number of works (Bryaneffcient and effective embedding method to assign each node a low dimensional representation vector that is aware of structural informa-tion vertex content and other information. Many efficient machine learning models can be designed for downstream tasks based on thelearned vertex representations like node classification (Zhu et al. 2007; BlagatGrabam and Muthukrishnan 2011) link prediction (Linyuan andZhou 2011; Gao et al. 2011; Liben-Nowell and Kleinherg 2003) (Liu et al 2018) visualization (Tang et al 2016) clustering (Fragkis-2015). Fig. 2 shows a brief summary of the development history of graph u) qs qde8 a8poux pue “(10 su8A pue soyembedding models.
techniqoes. what the survey focuses on beyond reviewing existing graph embedding
related information which can belp readers get a fast glimpse of existing Table 1 lists some typical graph embedding models and some of therlations. Shllow embedding modes can be roughly grouped into two main graph embedding models their inner mechanisms and underlying re-categories shallow neural embedding models and matrix factorization based models. Shallow neural embedding models (SN) are character-ized by embedding look-up tables which are updated to preserve various proximities lying in the graph. Typical models are DeepWalk(Bryan et al. 2014) node2vec (Grover and Leskovec 2016) LINE (Tang et al. 2015b) and so on. Matrix factorization (S-MF) based models aim tofactorize matrices related with graph structure and other side informa- tion to get high-quality node representation vectors. Based on shallowembedding models designed for homogeneous networks embedding techniques (e.g. PTE (Tang et al. 2015a) metapath2vec (Dong et al. 2017) GATNE (Cen et ol. 2019) are designed for heterogeneous net-Different from shallow embedding models graph eural networks (GNNs) works and we referthese models to heterogeneous (SH) embedding models.are kind of techniques characterized by deep architectures to extract meaningful structural information into node representation vectors Inaddition to the discussion of the above-mentioned types of models we also focus on their inner connections advantages and disadvantages optimization methods and some related theoretical foundations.
In this survey we provide a prehensive up-to-date review ofnetwork representation leaming algorithms aiming to give readers a macro covering the some mon basic insights under different kindsof embedding algorithms and the relationship between them as well as a micro lacking no details of different algorithms and also theories behindthem view on previous effort and achievements in this area. We group existing graph embedding methods into three major categories based onthe development dependencies among those algorithms from shallow embediding models whose objects are basic homogeneous graphs (Def. 1)models mos of whose basicideas are iherited frm shallow embdding with only one type of nodes and edges ” to heterogeneous embeddingjects expanded to heterogeneous graphs with more than one types of models designed for homogeneous graphs with the range of graph ob-nodes or edges and also often node or edge features then further to gruph neuraf network based models many of whose insights are able to be foundin shallow embedding models and heterogeneous embedding models like the inductive leaming and neighbourhood aggregation (Cen et al..2019) spectral propagation (Dommat et al 2017; Zhang et al. 2019b) and so on. Though it is hard to say the ideas of which methods are inspired by whose thoughts the similarity and connections between them can help us understand them better and also always offer some interesting rethinking of the mon field they belong to which are also
Finally we summarize some existing challenges and proposepossible development directions that can help with further design. We organize the survey as follows. In Section 2 we first summarize someuseful definitions whch cahlp rees understand the basiccoeps and then propose our taxonomy for the existing embedding algorithms.falling into those three categories I Section 7a we disuss some Then in Section 4 5 and 6 we review typical embedding methodsrelationships within those algorithms of different categorizes and related optimization methods. We then go further to discuss someproblems and challenges of existing graph embedding models in Section 9. At last we discuss some further development directions for network representation learming in Section 10.
2. Preliminaries
We summarize related definitions as follows to help readers under-stand the algorithms discussed in the following parts.
First we introduce the definition of a graph which is the basic datastructure of real-world networks:
Matrix Factorizatice Based Models; Right Channel: Grasph Neural Network Based Models. Fig. 2. A brief summary of the development of network embedding techniques. Left Channel Shallow (Heterogeneous) Neural Embedding Models; Mid Chanel:
Definition 1. (Graph). A graph can be denoted as G = ( 7 2) where7 is the set of verties and ≥ is the set of edges in the graph. When associated with the node type mapping function Φ : 7'→ mappingeach node to its specific node type and an edge mapping function 甲 : → mapping each edge to its corresponding edge type a graph G canbe divided into two categories: bomogeneous graph and heterogeneous graph. A homogeneous graph is a graph G with only one node type andgraph when |] [x| >2. one edge type (i.e. /| 1 and || 1). A graph is a heterogeneous
works like transportation network (Ribeiro et al. 2017) social net- Graphs are basic data structure for many kinds of real-worid net-q se suoq q po aq ue au uo os pue works academic networks (Scott et al. 2016; Yang and Leskovec 2015) those networks In the survey we use graph embedding and network neous graphs based on the knowledge we have n nodes and edges × inrepresentation learning alternatively both of which are high-frequency61g e1 u qto 7e 1a uqz) a a u paeadde s Bryan et al 2014; Grover and Leskovee 2016; Yang et al. 2020b) anddimension for nodes in a graph or a network. When we use the term both denote the process of generating representative vectors of a finitegraph embedding we focus mainly on the basic graph models where we simply care about nodes and edges in the graph and when we usenetwork representation learning our focus is more on networks in real-world.
modeling vertex proximities we briefly summarize the proposed vertex uo posq suuuogle Bupaqua jo Jaqumu ae[ e sj auoq aoussimilarities as follows (Zhang et al. 2018):
Definition 2. (Vertex Proximities). Various vertex proximities can exist in real-world networks like first-order proximity second-orderproximity and higher-order proximities The first-order proximity can measure the direct connectivity between two nodes which is usuallydefined as the weight of the edge between them. The second-orderHigher-order proximities between two vertices v and u can be defined as tween the distributions of their neighbourhood (Wang et al. 2016).*je s uz) n xaμa o a xauaa wog Aqeqod uogsuen das- au 2018).
Definition 3. (Structural Similarity). Structural similarity (Ribeiroet al 2017; Henderson et al 2012; Donnat et al 2017; Lorrain and White 1977; Pizarro 2007) refers to the similarity of the structural rolesof two vertices in their respective munities although they may not connect with each other.
Definition 4. (Intra-munity Similarity). The intra-munity
similarity originates from the munity structure of the graph andmunity. Many real-life networks (e.g social networks citation net- denotes the similarity between two vertices that are in the same -works) have munity structure where vertex-vertex connections in a munity are dense but sparse for nodes between two munities.
embedding algorithms based on graph spectral or adopt the graph u ps a s ueqeaspectral way which is also a crucial development direction for embed. ding algorithms we briefly introduce them as follows:
Definition 5. (Graph Laplacian). Following notions in (Houng andMaelhara 2019) L = D A where A is the adjacency matrix D is the corresponding degree matrix is the binational graph laplacian Z I DAD- is the normalized graph Laplacian I =I D′A is the random walk graph Laplacian. Meanwhile let A = A el denotes the augmented adjacency matrix then L L are the augmented graphLaplacian augmented normalized graph Laplacian augmented random walk graph Laplacian respectively.
3. Overview of graph embedding techniques
In this section we will give graph embedding techniques of eachBupuesapun aaaq e aa sapau dpuq o uonpou jagq e oae of the overall architecture of this paper. Fig. 3 shows a panoramic viewof existing embedding models and their connections.
Shallow Embedding Models. These models can be divided into twomain streams: shallow neural embedding models and matrix factoriza- tion based models. Though there are some differences between thosetwo embedding genres it has been shown that some shallow embedding based models especially those adopt random walk to sample vertexa suppaqa xaas a on ppou uu-dps opd pue sauanbasspeife close connections with matrix factorization models: they are actually implicitly factorizing their equivalent matrices (Qfu et al. 201Ba) to be
Besides matrices being factorized by shallow embedding models alsohave close relationship with graph spectral theories. Apart from models like GraphWave (Donnat et al. 2017) which are based on graph spectraldirectly (see Sec. 4.4) other models like DeepWalk (Beyan et al. 2014) node2vec (Grorer and Leskovec 2016) LINE (Tang et al. 2015b) canalso be proved to have close relatioship with graphspectralby peving that their equivalent matrices are filter matrices (Qiu et al. 2018a).
methods like matrix factorization and spectral embedding models can Then the explicit bination of traditional shallow embeddingbe seen in the embedding model ProNE (Zhang et al. 2019b) where
Table 1vestex labels; “eter." ~ heterogemeous networks Abbreviations used: F-O" S-O" *H-0" -C' ST' refer to First-Ordes econd-Order High-Order Intra-Com- An overview of network representation leaming algorithms (selected) Symbols in some fommulascam refer to Def. 5. Forothers “A* ~ w/o vertex atributes;*L* ~ w/Factorization Based models and Shallow Spectral mdels -S denotes optimzatonsraegls I which PS" NS"refert positive sling and negativ smling mnity and Structural similaities *SN' SHN MP' S" refer to Shallow Neural Embedding model Shallow Herogeneous Network Embedding Model Matrix
“(r)-(t)SVD° refers to (randomized)-(truncated) singular vale depositioe “SGNS* refers to *Skip-Gram with Negative Sampling”’; *lter-Update” refers to. PysatitityModel Type Neural Heter. L 0-$ Matris Filternode2vec (Grver al lsk Deepwalk (Bryan et al. 2014) S-N x x SGNS G SGNS G HO Table 2 DeepWalk Table 2 node2vee (r)xDif2vee (Rermbereki snd Salur 2016) PSWalkets (Rryan et al 2016) SGNS GRel2Vee (Ahdtpan e al. 208) LUNE (Tang et al 2015b) PS NS D G SGNS G F-0 5-0 1C Table 2 LINE b(z) 1pRBM [138]UPP-SNE (Zhang et al 2016c) SGNS G PS G HO F-0(610 *Te so nM) aNRIL DDRW (Liet al 2016) PS NS SVM D G SGNS SVM D G F0 S-0 HO M T L Tablte 2 LINE = （r）g -=（rGraphGAN (Tamng es al 2015b) G D F-0 struet2vec (Natarajan and Indedlit PIE (Tang etal. 2015a) 2014) 'SNDS PS NS G 5-0 ST Eq. 13HIN2ve (Ahmmdlyas et al. 2018) metapath2vec (Deng ct al. 2017) SGNS SGNS HO HOGATNE (Cen et al. 2019) HERe (Shi et al. 2019) SGNS MF SGNS HO HO user rating matris RHeGAN (HuYum mnd Shi 2019) HeeRec (Wang et al 2019b) L. SGNS G D HO F-OM-NMF (Warng et al. 2017b) MF ter-Update F0 5-0 1C 5 = g5 NetMF (Qiu et al. 2018s) tSVD H-O Table 2 DeepWalkProNE (Zhang et al. 2019b) GraRep (Can et al. 2015) JDSVD[Hochstenboch r-5VD tSVD HO F-0 Fq.5 Eq.6TADW (Cheng et a 2015) HOPE (0= et al. 2016) 2009] 1-MF H-O(withoet HO kstep transition GeneralHSCA (Zhang et al 2016b) A. ter-Update homephily) HO k-step transition matris MPrmNE (Zhang et al. 2019b) S-SS HO stgA)- matrix M[ײ1j g(x) =Graphzoom (Deng et al. 2020) HO A(2) = (1 x)bGraphWave (Denmrt et al. 2017) x ST 8() =eGCN (Kip and Welling 2017) NND A L SGD HO b2AD2 h(2) 1 xGrapiSAGE (Hamilton et al. 2017) FasGCN (Chen e al. 2018) A L A L SGD SGD HO HO Depend on A-M = h(z) 1 xASGCN (Hng e al. 2018) A L SGD HO h(z) = 1 xGAT (Velicknit et al. 2018) A L SGD SGD HO HO PAag(S) -AIsfNN (Gutmann and Hyviriren. GIN (Varwani et al. 2017) A L A L SGD HO 02Ab 2) A(b) = (1 x)bA L SGD HO h(μ) = (1 x)*ACR-GNN (Perg et al. 2020) Carkasingh 2020 A L. SGD HO [0 2AD 2]RGCN (Yan et al. 2007) A L SGD HO co
Table 1 (contised)
Model TypeNeural Hee. A/ L 0-5 Proximity Matrix Filter9Yz.gDropfidge (Fm-etsl 2018) BVAT (Comen et al. 2001) A L A L Adversarial SGD SGD BO HO sA-[90HetGNN (Wang et al. 2017a) R-GCN (Bryan etal. 2016) GralSP (JIn et al 2020) SGNS SGNS SGD H-0 ST HO HO
Matrfees that are Implikitly factorized by DeepWalk LINE and node2vec same Table 2with (Qiu et al. 2018a) DW' refers to DeepWalk n2v* refers to node2vee.
Model MatrisDWLINEn2v x2x)
vertex embeddings are firstly obtained by factorizing a sparse matrixand then propagated by band-pass filter matrix in the spectral domain. Moreover such close connections can also be seen int the university ofspectral propagation technique peoposed in ProNE which is proved to be a universal embedding enhancement method improving the qualityof vertex embeddings obtained by other shallow embedding modelseffectively (Zhang et al. 2019b).
Such associations enable some basic ideas of those shallow embed-ding models can be regarded as the basis of GNN models.
Heterogeneous Embedding Models. Based on shallow embeddingmodels many embedding models for heterogeneous networks can be developed by some techniques like metapath2vec (Dong et al. 2017) which applies certainty constrictions on the random sampling process and PTE (Tang et al. 2015a) which splits the heterogeneous graph intoseveral homogeneous graphs.
Moreover various graph coetent in heterogeneous models likevertex and edge features and labels evokes the thoughts on how to effectively utilize graph content in the embedding process and also howto bee inductive when being applied on dynamic graphs which is a
4. Shallow embedding models
4.1. Neural based
Fig 3. An overview of existing graph embedding models and their correlation.
mon feature of real-world graphs. For example the proposedembeding model GATNE (Cen et al 2019) applies attention mecha- nism on vertex features during the embedding process and try to learnthe transformation function applied on vertex contents to make themodel bee inductive (GATNE-I).
Such design ideas can be seen as basic models for Graph NeuralNetworks.
Graph Neural Networks. Different from above mentioned shallowembedding models Graph Neural Networks (GNNs) are some kind of deep inductive embedding models which can utilize graph contentsbetter and can also be trained with supervised information. The basic idea of GNNs is iteratively aggregating neighbourhood information fromvertex neighbours to get a succesive view over the whole graph structure.
focusing on developing enhancement techniques (Huang et al. 2018; Yu Based on vanilla GNN models there is a huge amount of workset al. 2019b; Feng et al. 2020; Hong et al 2020) to improve the eff- ciency and effectiveness of GNN models.
lems lying in GNN architecture with also methods proposed to solve Despite the advantages of GNN models there are also many prob.such problems most of which focus on graph regularization Deng et al. 2019; Verma et al 2019) basic theories (BareelEgor et al 2020) selfsupervised learning (Qiu et al 2020; Hu et al. 2020a) architecture search (Zhou et al. 2019) and so on.
embeding tables containing node embeddings as row or column vec- dn-Supoop kq pazuapere si seq pou jo pup e s aotors which are treated as parameters and can be updated during the
