Assuming you have good USA speed cloth, the cloth is treated with a Teflon coating. Any abrasive type of use - be if shuffling chips, picking up cards, brushing it with a billiard brush - will take it's toll in the long run, but you're talking about taking fractions of fractions of that coating off with each use. Shuffling the cards is probably doing twice as must damage to the cloth as brushing it off or using a vacuum on it. Typically the cloths I've replaced all show signs of wear at the middle of the table wear you're shuffling & spreading the cards the most (10x so if the game has a dedicated dealer) . I believe the specs on USA speed cloth are 30k+ double rubs, which puts it into a heavy duty commercial fabric category. The Teflon coating will start to wear off before the fabric shows signs of excessive wear, but the coating soaks into the fabric so it should retain some of it's water resistance even when it's getting towards the end of it's life.
Keeping the cloth clean will extend it's life, as any dust or crumbs left on the table will get ground up and become like sandpaper on the cloth with use. I typically vacuum my clothes with a brush attachment before each game, and wipe them down with a damp cloth if they get a little dirty.
