For what it's worth, I read the OP's post as having nothing to do with the strength of crossover opponents or being scared. He simply said he'd prefer to play other teams more often.
That being said, since others brought up the issue of having an imbalance in the strength of crossover opponents, I've always felt there was one solution that could satisfy almost everyone. It's been stated on the board many times. That solution is, your qualification for the ACCCG is based on division record only. In other words, if we sweep the Coastal, it doesn't matter if we lose both crossover games. We won our division with a record of 6-0. Just because VT, or Pitt, or whoever goes 7-1 based on beating Wake and Louisville, it doesn't matter. They weren't the best team in our division.