I'm not sure if @Knine9 is coming back to the forum or not, but I guess I will describe the solution for this issue anyway, in case another XFCE user stumbles on this thread while searching after the same problem.
In Settings, open Settings Manager -> Session and Startup -> General and make sure "Automatically save session on logout" is not checked.
A couple tabs down in that same menu is the "Saved Sessions" tab (if you have any saved sessions. If you do not, the tab disappears). Go ahead and delete them if you have any in there.
Next, find your log out button in the launcher and click it:
In the menu that opens up, untick the checkbox that says "Save session for future logins".
Optionally, reboot or log out/log back in to confirm the issue is resolved.