It's not that bad. P90X is good if you're not in tip top shape already. It'll help you get there. Insanity is definitely a bit rough, but also doable. You don't have to be in peak condition already, just do what you can and take breaks as needed.
If you find yourself breaking down and skipping workouts because you're too tired, you need to change your diet. With these types of workouts, you likely won't be cutting carbs out of your diet or on an extremely low calorie meal plan. You'll need to fuel yourself for the workouts appropriately. I'm not saying go throw down two cheeseburgers right after the workout, but make sure you're getting the right meals for the amount of work you put into it. When I did P90X, I quickly learned that my low calorie diet for trimming wasn't exactly helping my workouts, so I changed it around a bit. I still burned more calories than I consumed, but nowhere near as much and I changed my ratios around. My workouts got better, and I still managed to lose some fat.