Agglomerative Hierarchical Clustering Algorithm

An Agglomerative Hierarchical Clustering Algorithm is a Hierarchical Clustering Algorithm in which each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy.

• AKA: Bottom-Up Hierarchical Clustering, HAC.
• Context:
  • It can range from being an Agglomerative Soft Clustering Algorithm to being a Hard-Clusters Agglomerative Clustering Algorithm.
  • It can be implemented into an Agglomerative Hierarchical Clustering System.
  • …
• Example(s):
  • a CURE Clustering Algorithm.
  • a Single-Linkage Clustering Algorithm.
  • a Complete-Linkage Clustering Algorithm.
  • a UPGMA Clustering Algorithm.
  • …
• Counter-Example(s):
  • a Divisive Hierarchical Clustering Algorithm,
  • a Partitional Clustering Algorithm,
  • a Density Based Clustering Algorithm,
  • a Grid Based Clustering Algorithm,
  • a Model Based Clustering Algorithm.
• See: Unsupervised Machine Learning Algorithm, Clustering Task, Hierarchical Clustering Tasks.

---

---

References

2019a

• (Wikipedia, 2019) ⇒ https://en.wikipedia.org/wiki/Hierarchical_clustering Retrieved:2019-5-19.
  • In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis which seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two types:
    • Agglomerative: This is a “bottom-up” approach: each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy.
    • Divisive: This is a “top-down” approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.
  • In general, the merges and splits are determined in a greedy manner. The results of hierarchical clustering are usually presented in a dendrogram.

    The standard algorithm for hierarchical agglomerative clustering (HAC) has a time complexity of [math]\displaystyle{ \mathcal{O}(n^3) }[/math] and requires [math]\displaystyle{ \mathcal{O}(n^2) }[/math] memory, which makes it too slow for even medium data sets. However, for some special cases, optimal efficient agglomerative methods (of complexity [math]\displaystyle{ \mathcal{O}(n^2) }[/math] ) are known: SLINKK for single-linkage and CLINK for complete-linkage clustering. With a heap the runtime of the general case can be reduced to [math]\displaystyle{ \mathcal{O}(n^2 \log n) }[/math] at the cost of further increasing the memory requirements. In many programming languages, the memory overheads of this approach are too large to make it practically usable.

    Except for the special case of single-linkage, none of the algorithms (except exhaustive search in [math]\displaystyle{ \mathcal{O}(2^n) }[/math] ) can be guaranteed to find the optimum solution.

    Divisive clustering with an exhaustive search is [math]\displaystyle{ \mathcal{O}(2^n) }[/math] , but it is common to use faster heuristics to choose splits, such as k-means.

2019b

• (Wikipedia, 2019) ⇒ https://en.wikipedia.org/wiki/Hierarchical_clustering#Agglomerative_clustering_example Retrieved:2019-5-19.
  • For example, suppose this data is to be clustered, and the Euclidean distance is the distance metric.

    The hierarchical clustering dendrogram would be as such:

    Cutting the tree at a given height will give a partitioning clustering at a selected precision. In this example, cutting after the second row (from the top) of the dendrogram will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering, with a smaller number but larger clusters.

    This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements {a} {b} {c} {d} {e} and {f}. The first step is to determine which elements to merge in a cluster. Usually, we want to take the two closest elements, according to the chosen distance.

    Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described in the single-linkage clustering page; it can easily be adapted to different types of linkage (see below).

    Suppose we have merged the two closest elements b and c, we now have the following clusters {a}, {b, c}, {d}, {e} and {f}, and want to merge them further. To do that, we need to take the distance between {a} and {b c}, and therefore define the distance between two clusters.

    Usually the distance between two clusters [math]\displaystyle{ \mathcal{A} }[/math] and [math]\displaystyle{ \mathcal{B} }[/math] is one of the following:

    • The maximum distance between elements of each cluster (also called complete-linkage clustering):

[math]\displaystyle{ \max \{\, d(x,y) : x \in \mathcal{A},\, y \in \mathcal{B}\,\}. }[/math]

• The minimum distance between elements of each cluster (also called single-linkage clustering):

   :::: [math]\displaystyle{ \min \{\, d(x,y) : x \in \mathcal{A},\, y \in \mathcal{B} \,\}. }[/math]

• The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA):

[math]\displaystyle{ {1 \over {|\mathcal{A}|\cdot|\mathcal{B}|}}\sum_{x \in \mathcal{A}}\sum_{ y \in \mathcal{B}} d(x,y). }[/math]

• The sum of all intra-cluster variance.
• The increase in variance for the cluster being merged (Ward's method)
• The probability that candidate clusters spawn from the same distribution function (V-linkage).

• In case of tied minimum distances, a pair is randomly chosen, thus being able to generate several structurally different dendrograms. Alternatively, all tied pairs may be joined at the same time, generating a unique dendrogram . One can always decide to stop clustering when there is a sufficiently small number of clusters (number criterion). Some linkages may also guarantee that agglomeration occurs at a greater distance between clusters than the previous agglomeration, and then one can stop clustering when the clusters are too far apart to be merged (distance criterion). However, this is not the case of, e.g., the centroid linkage where the so-called reversals (inversions, departures from ultrametricity) may occur.

2016

• (Saket J & Pandya, 2016) ⇒ Swarndeep Saket J, and Dr Sharnil Pandya. (2016). “An Overview of Partitioning Algorithms in Clustering Techniques.” In: International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), 5(6). ISSN:2278 -1323
  • QUOTE: Hierarchical clustering method seeks to build a tree-based hierarchical taxonomy from a set of unlabeled data. This grouping process is represented in the form of dendrogram. It can be analyzed with the help of statistical method. There are two types of hierarchical clustering methods. They are 1) Agglomerative Hierarchical Clustering and 2) Divisive Clustering [7]. In the agglomerative approach which is also known as ‘bottom up approach’, Hierarchical algorithms always result into what is called ‘nested set of partitions’. They are called hierarchical because of their structure they represent about the dataset. Divisive and Agglomerative strategies are two important strategies of hierarchical clustering. In case of divisive approach, popularly known as ‘top down approach’, ‘all data points are considered as a single cluster and splitted into a number of clusters based on certain criteria [8]. Examples for such algorithm are BRICH (Balance Iterative Reducing and Clustering using Hierarchies), CURE (Cluster Using Representatives). The most important weakness of hierarchical clustering technique is that it does not scale properly because of time complexity. In addition to this, it is difficult to alter ones the process of analysis has already started.

2008

• (Manning et al., 2008) ⇒ Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze. (2008). "Hierarchical agglomerative clustering". In: "Introduction to Information Retrieval".
  • QUOTE: Hierarchical clustering algorithms are either top-down or bottom-up. Bottom-up algorithms treat each document as a singleton cluster at the outset and then successively merge (or agglomerate) pairs of clusters until all clusters have been merged into a single cluster that contains all documents. Bottom-up hierarchical clustering is therefore called hierarchical agglomerative clustering or HAC . Top-down clustering requires a method for splitting a cluster. It proceeds by splitting clusters recursively until individual documents are reached. See Section 17.6 . HAC is more frequently used in IR than top-down clustering and is the main subject of this chapter.

2002

• (Zhao & Karypis, 2002) ⇒ Ying Zhao, and George Karypis. (2002). “Comparison of Agglomerative and Partitional Document Clustering Algorithms.”
  • QUOTE: Hierarchical agglomerative algorithms find the clusters by initially assigning each object to its own cluster and then repeatedly merging pairs of clusters until a certain stopping criterion is met. The various clustering criterion functions that we are considering in this paper can be used to determine the pairs of clusters to be merged at each step in the following way.

    Consider an [math]\displaystyle{ n }[/math]-document dataset and the clustering solution that has been computed after performing [math]\displaystyle{ l }[/math] merging steps. This solution will contain exactly [math]\displaystyle{ n-l }[/math] clusters, as each merging step reduces the number of clusters by one. Now, given this [math]\displaystyle{ n-l }[/math]-way clustering solution, the pair of clusters that is selected to be merged next, is the one that leads to an [math]\displaystyle{ n-l-1 }[/math]-way solution that optimizes the particular criterion function. That is, each one of the [math]\displaystyle{ (n-l)\times(n-l-1)/2 }[/math] pairs of possible merges is evaluated, and the one that leads to a clustering solution that has the maximum (or minimum) value of the particular criterion function is selected. Thus, the criterion function is locally optimized within the particular stage of the agglomerative algorithm. Depending on the desired solution, this process continues until either there are only [math]\displaystyle{ k }[/math] cluster left, or when the entire agglomerative tree has been obtained.

1984

• (Day & Edelsbrunner, 1984) ⇒ William H.E. Day, and Herbert Edelsbrunner (1984). "Efficient algorithms for agglomerative hierarchical clustering methods.” In: Journal of classification 1, no. 1: 7-24.

---