When this happens it is best to separate the tracking into two.
Create the first tracking and adjust the length so that everything is working properly.
Save the 1st one, and create the second tracking right afterward.
Just like sketches, you can create multiple tracking graphics in one video clip. This is one way to handle players going out of frame while being tracked.