Can a screech owl occupying a nest box prevent a squirrel from taking-over the box? I've often wondered about this, but in seven years of continuous observation, I have never had the opportunity (or, perhaps, misfortune) to witness such an encounter. However, here are some relevant facts: Even the smallest adult fox squirrel, Sciurus niger, weighs more than three times as much as the largest screech owl (adult fox squirrels range from 600-1,300 grams, while, according to Gehlbach, the mean weight of nesting female eastern screech owls, Megascops asio, is 168-183 grams), so that's not a fight the owl is going to want. If memory serves, the The Book of Owls states that eastern screech owls will prey on creatures that weigh as much as they do, where most other owls won't tangle with anything that's more than half their weight. Sallie, the raptor rehabilitator, assures me that screech owls would only tackle prey as large as themselves if they were desperate, but, either way, we're talking about an owl with the metaphorical heart of a lion. Add to that the biological imperative to protect her nest, and I think we can assume that most nesting females would put up some fight against an invading fox squirrel. To continue speculating, I think we can also assume that this would not be enough to deter a determined fox squirrel (in the worst case, it'd be a lot like a 150 lb. person taking on an 1,100 lb. grizzly bear armed only with two fistfuls of knives), but it might well put off one that is merely conducting a casual investigation.
Will the fox squirrels learn to avoid the screech owl nest box after repeated evictions? Yes. In my experience, a fox squirrel must be evicted three times before it will give-up on a nest box, but after that it will avoid the box for about a year. Possibly telling us something about fox squirrel memory, I find that a fox squirrel will return to a nest box from which it has been evicted after a bit less than a week. So the complete eviction cycle requires three evictions over a period of about two weeks.
If the owls keep finding the squirrels in the nest box in between evictions, will it cause the owls to choose another nest site? In the nine years since I put up my first screech owl nest box, I believe I've had to evict squirrels every single year, but the local screech owls have nested in my box every one of those years. This may mean that screech owls are tolerant of these occasional intrusions, or that there are no other suitable nesting cavities in my neighborhood (therefore they're too desperate to be choosy), or both. In any case, as far as I know, it's not a major problem.
What about species of squirrels other than fox squirrels? Beats me. Fox squirrels are the only squirrels I've observed in my area. However, I'd guess that the situation would be much the same even with the smaller gray squirrels, Sciurus carolinensis.