Storage

Today I am excited to share some improvements that have just been released for our CSV Export and XML Export plugins! The big ticket item for these releases is a refactor of the way that exports are stored. You won’t see any visible differences in these latest changes, but if you look under the hood, you’ll see that these plugins just got a major upgrade. 🍻

Disclaimer This post may get a bit technical, so go ahead and pour that extra cup of coffee! It’s ok, I’ll wait. ☕

Let’s Talk About WordPress Hosting…

Recently, I engaged in some extremely scientific research and found that there are approximately eleventy billion different WordPress hosts on that complex series of tubes we call the internet. Each of those hosts, it turns out, also seems to do things just a little bit differently. These small differences in configuration normally do not affect the compatibility of our products, but we have observed a trend that has been preventing some people from using our export plugins, and we just can’t have that.

The Problem

Some web hosts do not allow WordPress to have write access to the filesystem. Sometimes this is a security precaution to prevent unauthorized or malicious file creation, but it can also be a bit more complex than that.

Consider, for example, a high-traffic site with multiple web nodes behind a load balancer. In this instance, if they allowed files to be written to one host, those files are not guaranteed to be available on subsequent requests since that web node may go down or the load balancer may reroute traffic to a different node. There are multiple ways to handle this particular issue, but some hosts choose to handle it by blocking write access to the filesystem altogether.

When it comes to our CSV and XML export plugins, this presented a problem, because the only way to store and download the export files was to write them to the filesystem. We heard from merchants who wanted to use our plugins, but simply weren’t able to because of the restrictions of their hosting environment. (Notably, the WordPress Swag Store was one that reached out specifically to request this feature!)

With the release of Customer/Order CSV Export 4.5 and Customer/Order XML Export Suite 2.4, this is now possible!

Data Stores

We decided to rebuild the storage handling of these plugins from the ground up, and implemented a data store layer to handle storage. In short, we rewrote the plugin to perform the exports without necessarily having to store them in any one particular way.

Previously when an export was created, the plugin would first create the output file, and then process each row one-by-one, and write each row to the export file. With our new changes, the plugin now just interacts with the general concept of a data store, and this now allows the site owner to swap the data store out at will!

Along with the data store changes, we have included two native data stores with these plugins — the legacy filesystem data store, and an all-new database data store, which allows exports to be saved completely to the database. This allows hosts with no write access to the filesystem to be able to create exports that are stored in the database instead, and this happens seamlessly — there are no changes to the interface of the plugin. If you didn’t look in the database to notice the new table, or in your exports folder to see missing files, you’d never know it had changed.

Seriously, you’re storing exports in the database?

Well, yeah! At first it felt a little weird to us, too, but it’s actually pretty nice. Remember, first, you do not have to use the database. You can always filter the storage method to prefer the legacy filesystem data store, or even create your own. Second, we have found that when downloading large exports, streaming them from the database is faster and uses less memory than serving them from the filesystem. Finally, our export plugins will automatically clean up exports more than 2 weeks old, so they won’t just pile up in your database indefinitely.

The Power of Data Stores

The true power of data stores lies in the flexibility they bring. We have only included two data stores, but the possibilities are limitless, and the opportunity is there for total customization to a particular setup or workflow.

Here are just a few examples that are now possible with a custom data store:

  • Store exports in a different MySQL database than what WordPress is installed on
  • Store exports in a completely different database engine such as PostgreSQL, SQLite, MongoDB, etc.
  • Store exports with a cloud storage service such as Amazon S3
  • Extend the filesystem or database data stores to modify the default behaviors, for example, to enforce at-rest encryption on all the stored exports.

Custom Data Store

Now, let’s get down to the nitty-gritty. Here’s how you could create a custom data store. The example code linked is written for Customer/Order CSV Export, but can be easily applied to XML Export if you change filter names and class names to match.

Extend the abstract data store class

  1. Create a new class for your data store that extends the base data store class:
    CSV Export: WC_Customer_Order_CSV_Export_Data_Store
    XML Export: WC_Customer_Order_XML_Export_Suite_Data_Store
  2. Implement and/or override each method in the abstract class
    View Custom Data Store Example →

Register your custom data store

When an export is created, the Data Store Factory is responsible for creating and returning a Data Store object that can handle the storage of that export:
CSV Export: WC_Customer_Order_CSV_Export_Data_Store_Factory
XML Export: WC_Customer_Order_XML_Export_Suite_Data_Store_Factory

Inside the factory’s create() method, the following filter is executed:
CSV Export: wc_customer_order_csv_export_custom_data_store
XML Export: wc_customer_order_xml_export_suite_custom_data_store

This will give you a slug that corresponds with a data store, and expects an instantiated Data Store object in return.

The two built-in data store slugs are database and filesystem, but if the slug given differs from those, it will check this filter for a valid custom data store before bailing.

Here, you’ll want to listen for your custom data store slug, and instantiate and return your data store if it is called.

Override the default storage method

Just before an export is created, the following filters are executed:
CSV Export: wc_customer_order_csv_export_start_export_args
XML Export: wc_customer_order_xml_export_suite_start_export_args

These filters allow you to override an array of arguments which creates an export. I’d encourage you to look a little closer at this code to see all the arguments here, but the one argument I want to focus on is storage_method. The default storage method is database, but can be overridden here with filesystem for the legacy filesystem data store, or with our new slug for our custom data store:

And that’s it!

We are excited to see how this new flexible data store system gets put to use. If you do something interesting with it, please let us know! We’d love to hear about it!

Published by Jared Burke

Jared is a PHP engineer at SkyVerge, focused on maintaining and building new features for our WooCommerce plugins. When he's not solving code puzzles, he solves real puzzles as a bonafide Rubik's cube master.

Leave a Reply

Your email address will not be published. Required fields are marked *