Dan Stevens, the actor, wanted out, and didn't want to appear even in first episode of season 4, and Julian Fellowes didn't want to recast or have a contrived issue with him being "out" or "away" all the time. At least that's what I've been reading, with interviews and such. So as we say here, Matthew is a "gone pecan". Sad.