Rav Yitzchak Zilbershtein, rav of Ramat Elchanan neighborhood of Bnei Braq, was asked by an avreich who is a baal teshuva if it is ok to let his children go to visit the grandparents who are not religious, and specifically if they can eat by the grandparents - even though the food is kosher, but because the grandparents are not religious the kids might hear inappropriate discussions at the table.

The father asking the question indicates he is ok with his kids visiting to say hello and to show respect, but is it ok to go for longer visits and to have meals there?

Rav Zilbershtein responds with an answer based on an explanation of the story of Yaakov and Lavan that he heard from Rav Mordechai Eliyahu. He shows how when Yaakov had to eat with Lavan, he himself had to but he made sure to send his kids away so they would not eat with them, but generally at other events the children were able to participate (see the original article for the more detailed explanation of the psukim).

Rav Zilbershtein says from that that people should be careful to not create a situation in which the non-religious grandfather will cause harm to the grandchildren. When eating together a certain bond is created that in such a situation can cause a cooling off of the children's Yiras Shamayim and irreparable harm can be caused. Therefore, the children should not be allowed to go visit the non-religious grandparents to eat with them.
source: Actualic