Multi-Similarity Loss

Image for post
Image for post

Multiple Similarities:

1. Self Similarity:

Image for post
Image for post
x1 = anchor, x2 = positive, x3,x4 = negatives
Image for post
Image for post
Sᵢₖ= cosine similarity between pairs, λ = similarity margin, α,β = hyperparameters
Image for post
Image for post
x1 = Anchor, x2,x3 = positives, λ = margin
Image for post
Image for post
x1 = anchor, x2,x3 = negatives, λ = margin

2. Negative relative similarity:

Image for post
Image for post
Weight assigned to a negative pair in ms loss, this is derived by derivative of the MS Loss with respect to a single pair.
Image for post
Image for post
weight for x1-x2, Case 1 > weight for x1-x2, Case 2> weight for x1-x2, Case 3

3. Positive relative similarity:

Image for post
Image for post
Weight assigned to a positive pair in ms loss, this is derived by derivative of the MS Loss with respect to a single pair.
Image for post
Image for post
weight for x1-x2, Case 1 > weight for x1-x2, Case 2> weight for x1-x2, Case 3

Mining Hard Positives And Negatives

Image for post
Image for post
A = anchor, P = positives, N = negatives

i) Hard Negative Mining:

Image for post
Image for post

ii) Hard Positive Mining:

Image for post
Image for post

Understanding The Code:

class MultiSimilarityLoss(nn.Module):
def __init__(self, cfg):
super(MultiSimilarityLoss, self).__init__()
self.thresh = 0.5
self.margin = 0.1
self.scale_pos = cfg.LOSSES.MULTI_SIMILARITY_LOSS.SCALE_POS
self.scale_neg = cfg.LOSSES.MULTI_SIMILARITY_LOSS.SCALE_NEG
def forward(self, feats, labels):
# feats = features extracted from backbone model for images
# labels = ground truth classes corresponding to images
batch_size = feats.size(0) sim_mat = torch.matmul(feats, torch.t(feats))
# since feats are l2 normalized vectors, taking
its dot product with transpose of itself will yield a similarity matrix whose i,j (row and column) will correspond to similarity between i'th embedding and j'th embedding of the batch, dim of sim mat = batch_size * batch_size. zeroth row of this matrix correspond to similarity between zeroth embedding of the batch with all other embeddings in the batch.
epsilon = 1e-5
loss = list()
for i in range(batch_size):
# i'th embedding is the anchor

pos_pair_ = sim_mat[i][labels == labels[i]]
# get all positive pair simply by matching ground truth labels of those embedding which share the same label with anchor
pos_pair_ = pos_pair_[pos_pair_ < 1 - epsilon]
# remove the pair which calculates similarity of anchor with itself i.e the pair with similarity one.
neg_pair_ = sim_mat[i][labels != labels[i]]
# get all negative embeddings which doesn't share the same ground truth label with the anchor
neg_pair = neg_pair_[neg_pair_ + self.margin > min(pos_pair_)]
# mine hard negatives using the method described in the blog, a margin of 0.1 is added to the neg pair similarity to fetch negatives which are just lying on the brink of boundary for hard negative which would have been missed if this term was not present.

pos_pair = pos_pair_[pos_pair_ - self.margin < max(neg_pair_)]
# mine hard positives using the method described in the blog with a margin of 0.1.
if len(neg_pair) < 1 or len(pos_pair) < 1:
continue
# continue calculating the loss only if both hard pos and hard neg are present.
# weighting step pos_loss = 1.0 / self.scale_pos * torch.log(
1 + torch.sum(torch.exp(-self.scale_pos * (pos_pair - self.thresh))))
neg_loss = 1.0 / self.scale_neg * torch.log(
1 + torch.sum(torch.exp(self.scale_neg * (neg_pair - self.thresh))))
# losses as described in the equation loss.append(pos_loss + neg_loss) if len(loss) == 0:
return torch.zeros([], requires_grad=True)
loss = sum(loss) / batch_size
return loss

Resources:

About Me:

ML Engineer/Research Head at Greendeck.