This paper systematically discusses the security and privacy concerns for e-learning systems. A five-layer architecture of e-learning system is proposed. The security and privacy concerns are addressed respectively for five layers. This paper further examines the relationship among the security and privacy policy, the available security and privacy technology, and the degree of e-learning privacy and security. The digital identity attributes are introduced to e-learning portable devices to enhance the security and privacy of e-learning systems. This will provide significant contributions to the knowledge of e-learning security and privacy research communities and will generate more research interests.