Are the solar panels fragile?

No. While a solar panel is made of tempered glass, it is quite strong. Many manufacturers conduct impact tests to withstand up to golf-sized hail. Furthermore, the solar panels are regularly installed in extreme conditions such as those found in Arctic and Antarctic regions.