First, let's review why training improves your maximum performance as measured by VO2max, and your endurance which we will define as your ability to maintain that high level of performance over a number of hours.
When you cut back on training, these adptive changes reverse themselves at different rates.
So there are some risks with a rest - be it for an injury or to pursue other personal interests. And it doesn't appear that it makes any difference if one trains three days a week near their VO2max or 3 days at a slower pace as demonstrated in the following study. Two groups decreased their training volume by 50%. One trained at 68% VO2mx 3 x per week, and the other varied above and below lactate threshold for 10 minute intervals and averaged 83% VO2ax. Both maintained equal performance levels for 21 days. This offers hope that if there is an injury, a reduced training program can be undertaken with minimal risk for 3 weeks.
But there can be benefits from a brief rest as well. This is demonstrated nicely in a study which linked a 50%, single-step reduction in HIT to an approximately 6% improvement in simulated 100 km time-trial performances after 2 weeks. Thus a planned taper can be used to ones benefit.
If an injury requires you to slow down for two or three weeks, a 3 day a week, slow and steady program can do a lot to postion you for a comeback after your recovery. And even if you have to abandon the bike for a month or two (a fracture for example), you should be able to look forward to retraining more quickly than the first time around.
Bottom line: The benefits that occur in your muscles and cardiovascualr system from regular training decline with a half-time of approximately 12 days. This means that after two weeks of inactivity, you'll go halfway from your trained state to the level you'd be at if you hadn't trained at all. Take another two weeks off and you'll lose half of what's left, and so on. But just three (and perhaps even two) intense one-hour rides (on a trainer indoors, or out) each week will safeguard much of your aerobic fitness and muscle strength for a midwinter (or injury related) break.