No, there is absolutely no downside if you join, only the 7 best scoring players count, there is no average. So every single point you score, takes away points from the other teams and keeps opponents of other teams away from scoring. Would be great if you could join!