During online group learning, students constantly encounter socio-emotional challenges that can trigger negative social emotions. These emotions weaken their psychological safety, disrupt communication, strain peer relationships, and ultimately lower productivity and contribution to group tasks. As a result, their attitude towards group learning may suffer. Emotion regulation is essential to mitigate the negative effects of socio-emotional challenges. The aims of the current study were to investigate: 1) the socio-emotional challenges students faced during online group learning, 2) the social emotions these challenges evoked, 3) how students regulated their emotions in these situations, 4) how students’ attitudes towards group learning relate to these social emotions, and 5) how the different emotion regulation modes correspond to positive and negative emotions. A mixed-method sequential explanatory design combined questionnaires and interviews to achieve the aims. The findings showed that unsuccessful collaborations were marked by socio-emotional challenges including differing task understanding and unequal participation among group members. These challenges were linked to negative social emotions, particularly disappointment and dislike. Conversely, successful collaborations fostered positive emotions such as respect and sympathy. These emotions arose from shared goals, aligned standards, productive working styles, and a supportive group climate. Students used a variety of strategies to regulate their emotions, including situation selection, situation modification, attentional deployment, cognitive change and response modulation. Furthermore, negative social emotions during the group learning were positively correlated with negative attitudes towards group learning. And, self-regulation of emotion was associated with negative emotions, while socially shared regulation was associated with positive emotions. The findings provide a foundation for developing interventions to strengthen emotion regulation and enhance online group learning effectiveness.