You have two sets in two way 

1. catalog_category_product.xml set 2column-left layout

<referenceContainer name="content">
  <referenceBlock name="category.products.list">
       <action method="setDefaultDirection">
           <argument name="dir" xsi:type="string">asc</argument>
       </action>
  </referenceBlock>
</referenceContainer>

2. catalog_category_product.xml set 1column layout

<referenceBlock name="product_list_toolbar">
     <action method="setDefaultDirection">
         <argument name="dir" xsi:type="string">asc</argument>
     </action>
</referenceBlock>