Gray squirrels don't spend one continuous stretch of time building their nest. They add a little to it each day. It may have to do with where the nests are located. Usually, they are 25-30 feet straight up. Bringing in nest building materials would be something like a man having to carry all the materials from the road a quarter mile to the construction site by hand. Even though they are adept at climbing, all those trips up and down the trees has to take its toll of energy.
They start by selecting suitable dry leaves.
Taking a mouthful, they will head for the nest.
When the nest is this far along, the leaves are taken inside where they take some time weaving them into place. Sticks are also used in the nest's construction. If a nest was just a pile of leaves, it would quickly be blown away, but they have to be able to withstand both wind and, occasionally, snow pack.
I am not certain whether both the male and female participate in building the nest. I tried to determine that the other day but, since they look alike, I was having trouble keeping them separated. They do both sleep in the nest overnight. I have often watched them both climb to the nest and enter just at dusk.