Still Dropping Some Weight (Why to trust your clothes versus your scale)

So for the past three months I've been trying to loose weight, as of 3 weeks ago I've gone down about 15 pounds, I haven't weighed in since then but I think I'm down another 5-7 pounds as of right now. It's great to see the change in the scale but personally I never go by that. My doctor has stated to me that I should've dropped more weight based on the diet he has me on but I feel the scale is deceiving at times so I always go by my clothes.

Pants that used to fit me tight; as in leave a mark on my waist from how tight they were, are now about an inch too big in the waist. I've gone from a 38 to a 36 and now the 36 now fit loosely as well. My chambray shirt which I don't have a picture of, went from fitting me okay, to fitting me so tight it was pulling (which is when I knew I needed to shed some weight). Now it fits me loose. Below are a couple of pictures of me trying on my linen blazer and my jacket from MAB suit. For prior reference to how they fit before refer to thestyleblogger.com post.

Sorry for not a proper fit post, felt like complete crap today and didn't leave the house. I wanted to see if I had gone up or down with my weight turns out it's down. First picture is how it fits without me adjusting it, second of me adjusting it with my hands.

Linen Blazer by Tasso Elba, Size XL, $100 at Macy's 

Notice it's a bit loose. 

Not a great example but ideally I need it brought in about an 1-1.5 inches.

Michael Andrews Bespoke Wool Blazer Part of a Suit: $$$

Actually debating if this one needs to be tweaked, my sides got smaller!

If I do tweak it it's just a bit

Opinions???