Had the exact same issue. The problem is the AC clutch. It wears out over time. When you turn on the AC and the pressure builds, the clutch slips so you lose your AC. This is very common on Subaru's, tons of threads on the other forums, lots of videos on YouTube. The XV is somewhat new so not as many have had this happen yet but it is just a matter of time. The "clutch air gap" should be between .016" and .025" (.4mm-.7mm). Mine was around .025" and I was having issues with it slipping.
You have several options all of which I obtained from the other forums. The easiest and what will test to see if this is indeed your issue is to shim the stock clutch. Don't laugh but it actually works. What you need to do is find yourself some shim material. I made mine out of a heavy plastic wiper fluid jug. One guy used a CD case. You cut 3 rectangular pieces and place them between the 3 rubber bumpers and the front plate of the clutch. After inserted, you must verify that the clutch can still spin free and that you didn't use too thick of a shim. You don't want the clutch to drag. When the car is started and the AC or Defrost is off, the clutch should not be turning. This worked like a charm. It also proves there is nothing wrong with your compressor or charge. Just be careful of the shim size and also when inserting that you don't cut the rubber bumpers.
The second option is to remove a factory shim from the clutch. You take the center bolt of the clutch out and them remove the clutch. Behind it there are generally 2 shims. This is how the factory controls the air gap. You can remove one and most likely obtain the correct gap. Lots of videos how to but ours is even easier than the other Subaru's in that the clutch is not pressed onto the shaft. Remove the center screw and the clutch comes right off (at least on the DKV10Z it does). You do however have to use a tool to prevent it from spinning when you loosen and tighten the bolt. I bought the tool, it's triangle shape, 3 pins engage holes in the clutch to prevent it from spinning. Was $10 or so.
Lastly, Subaru does not offer the clutch by itself. They will only sell the entire compressor. You can however buy one brand new on Ebay for around $100-120. You have to check the sticker on the compressor to be sure you get the correct model. There are cheaper ones on Amazon and Ebay but they are not the correct part (at least in my case). Don't go by it saying it fits a XV, use the model on the compressor sticker. Mine was a DKV10Z Compressor with the grey plug not the DKV10R that is listed in the cheaper listings. This is a little more involved in that in addition to removing the clutch, you also have to remove and replace the pulley. So in addition to the tool to prevent it from spinning, you will need some kind of gear puller. A lot of kids use a hammer in the videos but that is very bad and can blow out your seals.
I did buy a replacement clutch but the shims worked so well, I just haven't installed it yet. I had planned on doing a write up with pictures but was going to do it with the clutch install as well. When I asked here a few months back about this issue, the reply's were to take it to the dealer. Yeah right, $1500+ vs free or $120 for a new clutch. As I mentioned, this is a common issue with Subaru's.There is a lot of info out there about it which I tried to sum up in this post.
Here is a video that shows where the shims go. His (not my video) shims are very crude but it gives an idea where to place them. As mentioned, be careful you don't cut the rubber bumpers when slipping them into place. Mine have been in there for several months and not moved at all.