I'd have to agree with Tomcat. While I was quite warm at Whitney Portal and did not zip up my bag, once at Trail Camp I would prefer the comfort of a real, although light-weight sleeping bag and needed it. As is, I had trouble sleeping at 12,000 feet and you really want to be peppy and rested (somewhat) when you add that next 2,500 feet of altitude to get to the top of Whitney.
Also, if you have inclement weather on the way there, you might want to put on that clothing and it might then be wet.
It also is only about 5.5 miles and you have all day. I also went slowly. About 4.5 hours up to Trail Camp from the Portal. Going it alone, I did have to deal with the tent (3.5 lbs Sierra Designs Flashlight) and stove, fuel and pot by myself. But I also took running shoes and a book for camp comfort.
But, there are people who go very light and as you suggest, do without the full comfort of a bag. Clearly, it is a matter of choice.
I left out the filter and dealt with iodine tasting water and also went with lighter foods. I also left out my wind mittens and wound up using a clean pair of socks and was very happy to have them on the west side of Trail Crest where the breeze and shadows really chilled the hands/fingers.
Have a great trip up.