Early 20th Century Book on a Mixed Race Family

I only vaguely remember being told about this book.

From what I remember, there is a very well to do and respected white family living peacefully. The father, learns that he is actually part African American by some tiny fraction. (1/64 or something like that)

The family is then ostracized and nearly run out of town. I believe the last line was a cop or a neighbor telling someone to “Keep Moving”