Silk shirt & jeans? Definitely yes, for a refined casual look in the spirit of "Working class girl goes Lady"
My Style Tip: Oversized shirts are always to be combined with slim bottoms, otherwise you might look like Alice shrunken in Wonderland. It's important to use the right "accents" in order to define your shapes, despite the XL top. I like the combination with a statement necklace to put an accent on the neckline. When not wearing anything on top of the shirt, I always choose rolling up the sleeves to reduce some of the "oversize effect" and define the silhouette in the upper part of the body. Use belts or tuck it in to define your waistline.
Sweater Jacket: Mango, Shirt & Bag: Vintage, Sun Glasses: Gucci; Boots: BBUp