Move RefPtr refcount into a struct