Faster normalizeSimilarity() and networkPlot(), OpenAlex ids in cocMatrix(), normalized Co-citation in Biblioshiny

.Rbuildignore

@@ -40,3 +40,4 @@ CRAN-SUBMISSION
 ^graphify-out$
 ^raw$
 ^revdep$
+^dev$

.gitignore

@@ -36,3 +36,4 @@ revdep/library/
 revdep/checks.noindex/
 revdep/library.noindex/
 revdep/data.sqlite
+dev/

NEWS

@@ -1,6 +1,10 @@
 bibliometrix V.5.5.0.9000 (development version)
 
 Bug Fixes and Improvements
+- networkPlot() is four to sixteen times faster on dense networks: two steps worked one edge at a time. Colouring the edges (the colour of the community inside a community, grey between communities) ran an apply() over the edge list that read the vertex attribute again for every edge, and the weights set by community.repulsion were computed in a loop that looked up both end points with which() over the vertex names. On the co-citation network of the 1,000 most cited references of management (120,836 edges) they took about three quarters of the 23 seconds the function needed. Both are now computed on the whole edge list at once. Measured on management and scientometrics over co-citation of references (1,000 nodes) and of authors (500), bibliographic coupling (400 documents), keyword co-occurrence (300) and collaboration (200), each with Louvain, Walktrap, Leiden and no clustering, with and without normalization and repulsion: 23-31 seconds become 5-6 for the co-citation of references, 13-16 become 1-2 for co-citation of authors and coupling, and about 1 second becomes 0.15 for keywords. The graph with every vertex and edge attribute, the layout, the cluster table, the modularity and the Pajek graph compare identical() before and after in all twenty cases. What remains is igraph's own work: the layout, the betweenness centrality and the ten runs of Louvain or Leiden. Every network in Biblioshiny is drawn through networkPlot(), and so are couplingMap(), thematicMap() and thematicEvolution().
+- Biblioshiny: the Co-citation network can be normalized: the Method Parameters of the Co-citation network gain the Normalization Method selector that the Co-occurrence and Collaboration networks already have (None, Association, Jaccard, Salton, Inclusion, Equivalence; default Association, as there). The option had been left out because of the cost of normalizeSimilarity(), which is now computed on the non-zero cells only. As in the other networks, the normalized similarity drives the layout of the nodes; clusters and edge widths are still computed on the co-citation counts. On the co-citation networks of 1,000 references, cited authors and cited sources of management, normalizing adds one to three seconds.
+- cocMatrix() keeps the short OpenAlex work ids in CR: in an OpenAlex collection the cited references are work ids (W followed by digits), and cocMatrix(Field = "CR") drops every reference of 10 characters or fewer as not congruent, a filter written for reference strings. Ids of older works are 7 to 10 characters long, so they were discarded: on a collection of 585 documents downloaded through the OpenAlex API, 911 of the 53,078 references (1.7%), i.e. 784 distinct cited works, disappeared from every analysis built on CR (bibliographic coupling, co-citation). The filter now spares a reference that is an OpenAlex work id. Collections from other databases are unaffected: the matrices over management, scientometrics and two Scopus collections (old and new reference format), with and without short = TRUE, compare identical() before and after.
+- normalizeSimilarity() no longer builds dense matrices: every index divides the co-occurrence c_ij by a function of c_ii and c_jj, but it was computed with outer() on the diagonal, which allocates dense n x n matrices, and the inclusion index called min() once per cell through mapply(). On the co-citation network of the 9,999 most cited references of management the association strength took 2.9 seconds and a peak of 4.85 GB; the inclusion index took 3.6 seconds on 1,918 keywords. The indices are now computed on the non-zero cells of the sparse matrix only: 0.03 seconds and a peak of 335 MB (the collection included) on the same network. Values and returned class are unchanged: for all five indices, over coupling, co-citation, co-occurrence and collaboration networks from WoS, Scopus and OpenAlex collections, a base matrix input and a matrix with an item that never occurs, the result compares identical() with the previous one. An unknown type, which failed with "object 'S' not found", is now refused by name, and so is a non-square matrix.
 - No more igraph deprecation warnings: the network functions still called igraph functions deprecated since igraph 2.0/2.1 ("was deprecated in igraph 2.0.0", seen in thematicEvolution() through networkPlot()). They are replaced by their current names with identical results: layout.auto/circle/star/sphere/mds/kamada.kawai/fruchterman.reingold() by layout_nicely(), layout_in_circle(), layout_as_star(), layout_on_sphere(), layout_with_mds(), layout_with_kk(), layout_with_fr(); page.rank() by page_rank(); get.edgelist() by as_edgelist(); decompose.graph() by decompose(); delete.vertices() by delete_vertices(); write.graph() by write_graph(); hub_score()/authority_score() by hits_scores(); cluster_leiden(resolution_parameter = ) by resolution = ; eigen_centrality() no longer passes the deprecated scale and options = arpack_defaults. Each replacement was checked identical() against the old call with the same seed; unused deprecated imports are dropped and igraph (>= 2.1.0) is now required. Biblioshiny still passes layout = "layout.norm" to visNetwork::visIgraphLayout(), which calls that igraph function itself: a soft deprecation inside visNetwork, never shown to users.
 - applyReferenceMatching() joins each citation string once: normalize_citations() returns one row per occurrence of a reference, and joining it back on the citation string multiplied a reference cited k times into k x k rows, with dplyr's "Detected an unexpected many-to-many relationship" warning. The duplicates were collapsed afterwards, so the results were right, but on 2000 Web of Science records the intermediate table held about 2.55 million rows instead of 130,264, and a single reference cited 1160 times accounted for about 1.35 million of them. Occurrences of the same string share their match (only the ids of the filtered references differ, and those rows are dropped right after), so the matches are now reduced to one row per string before the join. On the Scopus fixture, a 530-document Scopus collection and the 2000-document WoS collection, all four elements of the result compare identical() before and after.
 - dominance() retains productive co-authors with zero first-authorships and bounds k (#694): in dominance(), the author pool was initialized solely from names(table(results$FirstAuthors[results$nAUperPaper > 1])), omitting prolific co-authors who have multi-authored publications but zero first-authored papers (FAA = 0). Under Kumar & Kumar (2008), their Dominance Factor is strictly DF = FAA / MAA = 0 / MAA = 0.0. Slicing with D[1:k, ] without clamping k to nrow(D) also produced spurious NA rows with false ranks when k exceeded the available author count. dominance() now evaluates all authors with multi-authored papers (MAA > 0), assigns DF = 0.0 when FAA = 0, clamps k to the number of eligible rows, and returns an empty data frame when no multi-authored paper exists.

NEWS.md

@@ -1,6 +1,10 @@
 # bibliometrix (development version)
 
 ## Bug Fixes and Improvements
+- **`networkPlot()` is four to sixteen times faster on dense networks**: two steps worked one edge at a time. Colouring the edges (the colour of the community inside a community, grey between communities) ran an `apply()` over the edge list that read the vertex attribute again for every edge, and the weights set by `community.repulsion` were computed in a loop that looked up both end points with `which()` over the vertex names. On the co-citation network of the 1,000 most cited references of `management` (120,836 edges) they took about three quarters of the 23 seconds the function needed. Both are now computed on the whole edge list at once. Measured on `management` and `scientometrics` over co-citation of references (1,000 nodes) and of authors (500), bibliographic coupling (400 documents), keyword co-occurrence (300) and collaboration (200), each with Louvain, Walktrap, Leiden and no clustering, with and without normalization and repulsion: 23-31 seconds become 5-6 for the co-citation of references, 13-16 become 1-2 for co-citation of authors and coupling, and about 1 second becomes 0.15 for keywords. The graph with every vertex and edge attribute, the layout, the cluster table, the modularity and the Pajek graph compare `identical()` before and after in all twenty cases. What remains is igraph's own work: the layout, the betweenness centrality and the ten runs of Louvain or Leiden. Every network in Biblioshiny is drawn through `networkPlot()`, and so are `couplingMap()`, `thematicMap()` and `thematicEvolution()`.
+- **Biblioshiny: the Co-citation network can be normalized**: the Method Parameters of the Co-citation network gain the *Normalization Method* selector that the Co-occurrence and Collaboration networks already have (None, Association, Jaccard, Salton, Inclusion, Equivalence; default Association, as there). The option had been left out because of the cost of `normalizeSimilarity()`, which is now computed on the non-zero cells only. As in the other networks, the normalized similarity drives the layout of the nodes; clusters and edge widths are still computed on the co-citation counts. On the co-citation networks of 1,000 references, cited authors and cited sources of `management`, normalizing adds one to three seconds.
+- **`cocMatrix()` keeps the short OpenAlex work ids in `CR`**: in an OpenAlex collection the cited references are work ids (`W` followed by digits), and `cocMatrix(Field = "CR")` drops every reference of 10 characters or fewer as not congruent, a filter written for reference strings. Ids of older works are 7 to 10 characters long, so they were discarded: on a collection of 585 documents downloaded through the OpenAlex API, 911 of the 53,078 references (1.7%), i.e. 784 distinct cited works, disappeared from every analysis built on `CR` (bibliographic coupling, co-citation). The filter now spares a reference that is an OpenAlex work id. Collections from other databases are unaffected: the matrices over `management`, `scientometrics` and two Scopus collections (old and new reference format), with and without `short = TRUE`, compare `identical()` before and after.
+- **`normalizeSimilarity()` no longer builds dense matrices**: every index divides the co-occurrence `c_ij` by a function of `c_ii` and `c_jj`, but it was computed with `outer()` on the diagonal, which allocates dense *n* × *n* matrices, and the `inclusion` index called `min()` once per cell through `mapply()`. On the co-citation network of the 9,999 most cited references of `management` the association strength took 2.9 seconds and a peak of 4.85 GB; the inclusion index took 3.6 seconds on 1,918 keywords. The indices are now computed on the non-zero cells of the sparse matrix only: 0.03 seconds and a peak of 335 MB (the collection included) on the same network. Values and returned class are unchanged: for all five indices, over coupling, co-citation, co-occurrence and collaboration networks from WoS, Scopus and OpenAlex collections, a base `matrix` input and a matrix with an item that never occurs, the result compares `identical()` with the previous one. An unknown `type`, which failed with *"object 'S' not found"*, is now refused by name, and so is a non-square matrix.
 - **No more igraph deprecation warnings**: the network functions still called igraph functions deprecated since igraph 2.0/2.1, which R reports as *"was deprecated in igraph 2.0.0"* warnings (seen in `thematicEvolution()` through `networkPlot()`). They are replaced by their current names with identical results: `layout.auto/circle/star/sphere/mds/kamada.kawai/fruchterman.reingold()` by `layout_nicely()`, `layout_in_circle()`, `layout_as_star()`, `layout_on_sphere()`, `layout_with_mds()`, `layout_with_kk()`, `layout_with_fr()`; `page.rank()` by `page_rank()`; `get.edgelist()` by `as_edgelist()`; `decompose.graph()` by `decompose()`; `delete.vertices()` by `delete_vertices()`; `write.graph()` by `write_graph()`; `hub_score()`/`authority_score()` by `hits_scores()`; `cluster_leiden(resolution_parameter = )` by `resolution = `; and `eigen_centrality()` no longer passes the deprecated `scale` and `options = arpack_defaults`. Each replacement was checked `identical()` against the old call with the same seed; unused deprecated imports are dropped and `igraph (>= 2.1.0)` is now required. Biblioshiny still passes `layout = "layout.norm"` to `visNetwork::visIgraphLayout()`, which calls that igraph function itself: a soft deprecation inside visNetwork, never shown to users.
 - **`applyReferenceMatching()` joins each citation string once**: `normalize_citations()` returns one row per occurrence of a reference, and joining it back on the citation string multiplied a reference cited *k* times into *k* × *k* rows, with dplyr's *"Detected an unexpected many-to-many relationship"* warning. The duplicates were collapsed afterwards, so the results were right, but on 2000 Web of Science records the intermediate table held about 2.55 million rows instead of 130,264, and a single reference cited 1160 times accounted for about 1.35 million of them. Occurrences of the same string share their match (only the ids of the filtered references differ, and those rows are dropped right after), so the matches are now reduced to one row per string before the join. On the Scopus fixture, a 530-document Scopus collection and the 2000-document WoS collection, all four elements of the result compare `identical()` before and after.
 - **Local citation analyses of a Scopus collection work after reference matching** (#696): applying `applyReferenceMatching()` in Biblioshiny rebuilds `CR` with a join on the normalized references, so a document none of whose references survives the matching gets `CR = NA`. `CR_SO()` passes every reference to `convert_scopus_new_to_classic()`, where `str_detect(NA, ...)` is `NA` and the `if` stopped *Most Local Cited Sources* and the *Co-citation* network with *"missing value where TRUE/FALSE needed"*. Converting the references a second time was not the cause: a reference already in the classic layout is returned unchanged. `convert_scopus_new_to_classic()` now returns a missing or empty reference as it is, so such a document gets `CR_SO = NA`, and a Scopus collection that carries a missing `CR` for any other reason is covered too. Documents with references are unaffected. Reported by @BehradSa1999; the cause was traced by @GabrielBaiano.

R/cocMatrix.R

@@ -132,8 +132,10 @@ cocMatrix <- function(M, Field = "AU", type = "sparse", n = NULL, sep = ";", bin
   }
   Fi <- lapply(Fi, trim.leading)
   if (Field == "CR") {
-    Fi <- lapply(Fi, function(l) l <- l[nchar(l) > 10])
-  } ## delete not congruent references
+    # delete not congruent references, but keep OpenAlex work ids (W + digits),
+    # which can be as short as 7 characters
+    Fi <- lapply(Fi, function(l) l <- l[nchar(l) > 10 | grepl("^W[0-9]+$", l)])
+  }
 
   ## Scelta dell'informazione contenuta in CR da utilizzare (Reference, Autore, Affiliation, ecc.)
 

R/networkPlot.R

@@ -552,18 +552,15 @@ clusteringNetwork <- function(bsk.network, cluster, seed = NULL, n_runs = 10) {
   El <- as.data.frame(as_edgelist(bsk.network, names = F))
 
   colorlist <- colorlist()
-  # apply() over a zero-row edge list still calls the function once, on an
-  # empty row, to learn the type of its result -- and the comparison inside
-  # stops with "argument is of length zero". A graph without edges has no edge
-  # to colour.
-  if (nrow(El) > 0) E(bsk.network)$color <- apply(El, 1, function(x) {
-    if (V(bsk.network)$community[x[1]] == V(bsk.network)$community[x[2]]) {
-      C <- colorlist[V(bsk.network)$community[x[1]]]
-    } else {
-      C <- "gray70"
-    }
-    return(C)
-  })
+  # An edge inside a community takes the colour of the community, an edge
+  # between communities is grey. Computed on the whole edge list at once: an
+  # apply() over the edges read the vertex attribute again for every edge. A
+  # graph without edges has no edge to colour.
+  if (nrow(El) > 0) {
+    community <- V(bsk.network)$community
+    c_from <- community[El[, 1]]
+    E(bsk.network)$color <- ifelse(c_from == community[El[, 2]], colorlist[c_from], "gray70")
+  }
   E(bsk.network)$lty <- 1
   E(bsk.network)$lty[E(bsk.network)$color == "gray70"] <- 5
 
@@ -606,28 +603,15 @@ switchLayout <- function(bsk.network, type, community.repulsion) {
       original_weights <- E(bsk.network)$weight
     }
 
-    # Apply new weighting scheme with gradual growth
-    new_weights <- numeric(nrow(row))
-
-    for (i in 1:nrow(row)) {
-      node1 <- row[i, 1]
-      node2 <- row[i, 2]
-
-      comm1 <- membership[which(names(membership) == node1)]
-      comm2 <- membership[which(names(membership) == node2)]
-
-      if (comm1 == comm2) {
-        # INTRA-COMMUNITY Edge
-        # Moderate increase with sub-linear growth
-        multiplier <- 1 + (repulsion_strength^0.7) * 1.5
-        new_weights[i] <- original_weights[i] * multiplier
-      } else {
-        # INTER-COMMUNITY Edge
-        # Gradual reduction with attenuated exponential function
-        divisor <- 1 + exp(repulsion_strength * 1.2) - 1
-        new_weights[i] <- original_weights[i] / divisor
-      }
-    }
+    # Apply new weighting scheme with gradual growth: intra-community edges are
+    # strengthened, inter-community edges weakened
+    comm1 <- unname(membership)[match(row[, 1], names(membership))]
+    comm2 <- unname(membership)[match(row[, 2], names(membership))]
+    # INTRA-COMMUNITY Edge: moderate increase with sub-linear growth
+    multiplier <- 1 + (repulsion_strength^0.7) * 1.5
+    # INTER-COMMUNITY Edge: gradual reduction with attenuated exponential function
+    divisor <- 1 + exp(repulsion_strength * 1.2) - 1
+    new_weights <- ifelse(comm1 == comm2, original_weights * multiplier, original_weights / divisor)
 
     # Apply new weights
     E(bsk.network)$weight <- new_weights