This paper proposes a new approach for rumor detection in Twitter using word embedding with pre-trained language models (PLMs). We exploit the power of a SOTA PLM called RoBERTa for the task of rumor detection. However, we focus on the general issue of fine tuning and adapting PLMs for this task. That is why, we compare RoBERTa with several other PLMs. Furthermore, we integrate data augmentation and tuning class distribution techniques to enhance results. Our experiments show that our approach outperforms SOTA approaches using standard datasets and metrics.