To have Duplicate Check work together with Record Validation, use the RV page layout for RV Entry, then apply a Duplicate Check scenario to RV Insert and RV Update.

1. Apply RV Entry

You can only configure one Entry page. To use both Duplicate Check and Record Validation when creating or editing a record, do not use DC Entry.
Instead, set up the RV Entry page as described in How to deploy the Record Validation Entry page (in Salesforce Lightning)‍. 

2. Apply a DC scenario to RV Insert and Update

1. In Duplicate Check, go to tab DC Setup.
2. On the left-hand side, at Object Setup, click the Object you are using Record Validation for.
3. At the Scenarios tab, find the Scenario‍ you want to use to detect duplicate records in Record Validation.
4. In the scenario configuration, at Apply To, select RV Manual Insert Prevention and RV Manual Update Prevention.
5. Repeat for other Objects if needed.

Apply a Duplicate Check scenario to Record Validation Insert and Record Validation Update

Selecting RV Manual Insert Prevention will trigger Duplicate Check's prevention feature when creating a new record.
Selecting RV Manual Update Prevention will trigger DC prevention when updating an existing record.