Clothing articles like sports jerseys that are "supposed" to be baggy make us feel comfortable. It's nice, it feels great; like hiding in plain sight. Let us be frank: they make you look bigger than you are.
The billowing extra fabric does not hide your girth, it accentuates it. We think the big guy mind associates being big with clothes feeling tight and therefore loose fitting clothes must be how skinny people feel. That is not the way to go. Generally avoid clothes that add bulk or create artificial shapes wide-leg pants, bulky shoes, and overly large or puffy jackets and jumpers.
Also, if you have lost some weight, don't continue wearing your out-sized clothes; any "bigness" in clothes not supported by your body hangs down as length which distorts your proportions, and creates bagginess, particularly under your arms.