Genetic relatedness is a central concept in genetics, underpinning studies of population and quantitative genetics in human, animal, and plant settings. It is typically stored as a genetic relatedness matrix, whose elements are pairwise relatedness values between individuals. This relatedness has been defined in various contexts based on pedigree, genotype, phylogeny, coalescent times, and, recently, ancestral recombination graph. For some downstream applications, including association studies, using ancestral recombination graph-based genetic relatedness matrices has led to better performance relative to the genotype genetic relatedness matrix. However, they present computational challenges due to their inherent quadratic time and space complexity. Here, we first discuss the different definitions of relatedness in a unifying context, making use of the additive model of a quantitative trait to provide a definition of "branch relatedness" and the corresponding "branch genetic relatedness matrix". We explore the relationship between branch relatedness and pedigree relatedness (i.e. kinship) through a case study of French-Canadian individuals that have a known pedigree. Through the tree sequence encoding of an ancestral recombination graph, we then derive an efficient algorithm for computing products between the branch genetic relatedness matrix and a general vector, without explicitly forming the branch genetic relatedness matrix. This algorithm leverages the sparse encoding of genomes with the tree sequence and hence enables large-scale computations with the branch genetic relatedness matrix. We demonstrate the power of this algorithm by developing a randomized principal components algorithm for tree sequences that easily scales to millions of genomes. All algorithms are implemented in the open source tskit Python package. Taken together, this work consolidates the different notions of relatedness as branch relatedness and, by leveraging the tree sequence encoding of an ancestral recombination graph, provides efficient algorithms that enable computations with the branch genetic relatedness matrix that scale to mega-scale genomic datasets.