Co-occurrence Analysis of Values Based on Group and Time

Co-occurrence (Matrix) of Values Based on Group and Time

The problem presented is a classic example of a collaborative filtering task, where we want to analyze the co-occurrence matrix of values based on group and time. In this post, we will delve into the details of how to solve this problem using data manipulation and analysis techniques.

Background

Collaborative filtering is a technique used in recommendation systems to predict user preferences based on their past behavior. In this case, we want to analyze the co-occurrence matrix of values for different groups over time. The co-occurrence matrix is a matrix that shows the frequency of co-occurrence between pairs of values.

Problem Statement

We are given a dataset with three columns: ID, Group, and Time. We want to calculate the total number of collaborations per individual, the total number of different individuals each person has collaborated with, and the total number of repeat collaborations in the existing team.

For example, if we have two projects, Trx1 and Trx2, where A, B, and C collaborate on Trx1, and E and B collaborate on Trx2, we want to calculate:

The total number of collaborations per individual: In 1980, two projects occur (Trx1 and Trx2). In the first one, A, B, and C collaborate, in the second one, E and B do.
The total number of different collaborators for each person: For every IDi we want, the total number of IDj(j != i) with whom IDi collaborates on a Trx project within z (say 3) years before the focal project. Also, we want the number of different collaborators.
The total number of repeat ties.

Solution

To solve this problem, we will use data manipulation and analysis techniques.

library(data.table)
library(magrittr)
options(stringsAsFactors = F)

# Read the dataset
dat <- read.table(text="Group ID Time
Trx1 A 1980
Trx1 B 1980
Trx1 C 1980
Trx2 E 1980
Trx2 B 1980
Trx3 B 1981
Trx3 C 1981
Trx4 C 1983
Trx4 E 1983
Trx4 B 1983
Trx5 F 1984
Trx5 B 1984
Trx5 C 1984
Trx6 A 1986", header=T) 

# Convert the data to a data.table
dat <- as.data.table(dat)

# Define the number of prior years
priorYears <- 3

# Get unique IDs
unqIDs <- unique(dat$ID)

# Initialize an empty data.table to store results
results <- data.table(ID = character(), year = numeric(), total = numeric(), diff = numeric(), repeatSum = numeric())

# Loop through each row in the data
for (i in 1:nrow(dat)) {
  # Get the end and start years
  endYear <- dat$Time[i] 
  startYear <- endYear - priorYears
  
  # Filter dates to only include projects within z years before the focal project
  subset.DT <- dat[dat$Time >= startYear & dat$Time < endYear]
  
  # Keep only groups where the current ID collaborated
  groupsToKeep <- unique(subset.DT$Group[subset.DT$ID == dat$ID[i]]) %>% .[. != dat$ID[i]]
  subset.DT <- subset.DT[subset.DT$Group %in% groupsToKeep,]
  
  # Calculate the total number of collaborations per individual
  total <- length(which(subset.DT$ID != dat$ID[i]))
  
  # Calculate the total number of different collaborators for each person
  unqMembers <- unique(subset.DT$ID) %>% .[. != dat$ID[i]]
  currentMembers <- unique(dat$Group[dat$Group == subset.DT$Group[1]]) %>% .[. != dat$ID[i]]
  
  diff <- length(unqMembers)
  repeatSum <- sum(table(subset.DT$ID)[currentMembers], na.rm = T)
  
  # Add results to the data.table
  results <- rbind(results, data.frame(ID = dat$ID[i], year = endYear, total, diff, repeatSum))
}

Explanation

The solution consists of three main steps:

Filtering dates: We filter the dates to only include projects within z years before the focal project.
Keeping groups where the current ID collaborated: We keep only groups where the current ID collaborated on a Trx project.
Calculating co-occurrence matrix: We calculate the total number of collaborations per individual, the total number of different collaborators for each person, and the total number of repeat ties.

The results are stored in an empty data.table called results, which is populated with each iteration of the loop.

Example Use Case

Suppose we want to analyze the co-occurrence matrix of values for two groups: Trx1 and Trx2. We can use the following code:

# Filter data for Trx1 and Trx2
Trx1 <- dat[dat$Group == 'Trx1',]
Trx2 <- dat[dat$Group == 'Trx2',]

# Loop through each row in the data
for (i in 1:nrow(Trx1)) {
  # Get the end and start years
  endYear <- Trx1$Time[i] 
  startYear <- endYear - priorYears
  
  # Filter dates to only include projects within z years before the focal project
  subset.Trx1 <- Trx1[Trx1$Time >= startYear & Trx1$Time < endYear]
  
  # Keep only groups where the current ID collaborated
  groupsToKeep <- unique(subset.Trx1$Group[subset.Trx1$ID == Trx1$ID[i]]) %>% .[. != Trx1$ID[i]]
  subset.Trx1 <- subset.Trx1[subset.Trx1$Group %in% groupsToKeep,]
  
  # Calculate the total number of collaborations per individual
  total <- length(which(subset.Trx1$ID != Trx1$ID[i]))
  
  # Calculate the total number of different collaborators for each person
  unqMembers <- unique(subset.Trx1$ID) %>% .[. != Trx1$ID[i]]
  currentMembers <- unique(Trx2$Group[Trx2$Group == subset.Trx1$Group[1]]) %>% .[. != Trx1$ID[i]]
  
  diff <- length(unqMembers)
  repeatSum <- sum(table(subset.Trx1$ID)[currentMembers], na.rm = T)
  
  # Add results to the data.table
  result <- rbind(result, data.frame(ID = Trx1$ID[i], year = endYear, total, diff, repeatSum))
}

This code loops through each row in the data for Trx1 and calculates the co-occurrence matrix with Trx2. The results are stored in an empty data.table called result.

Last modified on 2024-03-03