Visiting the Grave on the Yahrzeit

My parents passed away three years ago, a few weeks apart. They were very, very loving grandparents and very attached to their children, grandchildren and great-grandchildren. They supported their children and grandchildren financially in a rare way. My question is about going to the grave on the anniversary. We, brothers and sisters, come, but the numerous grandchildren (without the evil eye) rarely come. Some of them didn’t even come once, and each yahrzeit we’re forced to count the men so that there be enough for a minyan (quorum). It hurts me a lot, but I don't know if or what to say to them, because I don't know what is halacha, what is custom, and what is acceptable to do in this case.

Dear …!

Your pain is very understandable.

Visiting the graves of one’s relatives on Yahrzeit is mentioned in Halacha, but there is no specification as to who is included in this custom.

The actual customs differ greatly from family to family. Therefore, it is difficult to determine what the 'accepted' custom is.

If it’s important for the father and the mother that their child visit the grave of his grandparent, then, by doing so, the child fulfills the commandment of honoring his parents, which is a Torah obligation. This is far more important than visiting a grave, which is only a custom.

[Some authorities write that there is no point in gathering a minyan; the main thing is that family members come to the grave.]

Shulchan Oruch, section Yore Deah, chapter 344, §2
“Halichos Shlomo”, ch. 18, footnote 74