You must use local guide who knows where to go which is not on the tourist map and knows how to make you safe during ice cave visit.
On the first day of my trip in Iceland, I drove from KEF airport straight toward Jokulsarlon (Glacier Lagoon) through snow storm that made my trip so delay for seeing sunrise at the beach on time. Thereafter, after morning came, the weather was perfect for the Ice Cave Tour. That was a beautiful sunny day with temperature below zero (around -10 celcius). Temperature below freezing is required if you want to visit ice cave.
Local guide took me and my friends off the main road near Jokulsarlon by 4WD toward the edge of Glacier. The dirt road became disappear while we approaching our destination. We had to hike for about 2 km to reach this crystal cave.