Fleece blankets, fleece gloves, etc make my skin very dry, uncomfortable?

At first I thought it must be quite common to get dry skin from fleece, but I haven't heard of anyone else who has this problem. Whenever I even touch fleece, it makes my skins very dry. the only time i can stand to use it is if i'm so cold i don't notice. even if i have just put lotion on, my skins gets very dry when touching fleece. Is this normal or has anyone else had problems like this? Or have any possible solutions?