If you’re new to App Engine, then the file system model may be a bit different from what you might have experienced using other hosts.
With an App Engine application, you cannot write to the file-system where your application is deployed. Your application can
read any files from the deployed directory structure, but it
can’t write to that file-system. Instead, the application can use
Google Cloud Storage (GCS) for both reading and writing files.
To convert an app to use GCS for writable files, here are the primary things that you will typically need to do:
Another implication of the read-only file system is that if your app has a ‘plugin’ model, like WordPress, you can’t install or update plugins from your
deployed app. Instead, you will need to install or update any plugins locally, then redeploy the app.
You can find lots more info on all of the following in the
documentation for the PHP runtime.
The Application FilesystemWith App Engine applications, the ‘local’ filesystem — the project directory tree, which is uploaded with your deployed app — is
read-only. This constraint is imposed for scalability and security reasons; sometimes it’s referred to as ‘sandboxing’. (And, many common framework
vulnerabilities are indeed blocked by App Engine apps).
You can read any file in this deployed directory structure, for example your application might read deployed template or configuration files. (If in your app code you want to to read files you’re also serving
statically, you can use the
application_readable directive in your app.yaml file).
Importantly, your application can’t create or modify files in this local file system — e.g., it can’t write cache files there.
Because of this sandboxing, if your app has a ‘plugin’ model, like WordPress, you can’t install or update plugins from your
deployed app. Instead, you’ll need to install or update them locally, in your development environment, by running your app using App Engine’s
development server. You can then test the update locally, if you like, and then redeploy the app with the updates. If you’re comfortable doing so, you can also just download and unzip the plugin files directly, move them to the proper directory, and redeploy; rather than installing them via the WordPress administration console.
(Note that if you make any database-stored changes from the development server, like setting plugin options, these of course won’t propagate to your deployed app, which is using a different database. Once you re-deploy your app with the new plugin files, you can configure the plugin in the deployed app).
Google Cloud Storage For a read/write filesystem, your application can instead use Google Cloud Storage (GCS) buckets[1]. Google Cloud Storage is a RESTful service for storing and accessing your data objects on Google’s infrastructure. GCS is fast, scalable, and highly available, with multiple layers of redundancy; and stored objects can be terabytes in size.
For the App Engine PHP runtime, there is a GCS stream wrapper that supports the majority of standard file-system-related functions. This stream wrapper lets you use PHP file and directory functions such as file_put_contents, file_get_contents, fopen, and opendir. File and directory information can be retrieved for GCS objects using functions like file_exists, is_writable, filesize, is_dir, etc.
You point to a GCS directory or file using syntax like this:
gs:///path/to/file
So, you can write app code like this:
$fp = fopen("gs://my_bucket/some_file.txt", "w");
fwrite($fp, "Hello");
fclose($fp);
or:
$options = [ "gs" => [ "Content-Type" => "text/plain" ]];
$ctx = stream_context_create($options);
file_put_contents("gs://my_bucket/hello.txt", "Hello", 0, $ctx);
See the
documentation for more detail and examples.
(Before code like this will work in your app, you will need to set up GCS in a Google Cloud project and authorize your app engine app for that project, as described
here).
For the most part, once you’ve redefined your paths to gs:// URIs, your file-system calls will just work as before. You may want to check config settings for your app before you deploy it, to ensure that write paths use a GCS bucket path instead of a local filesystem path. E.g., you may need to point cache directories to GCS.
However, one way in which GCS is not like a ‘usual’ filesystem is that GCS does
not support file (object) appends. Instead of appending to a file, you must write a new version of that file that has the updated content.
One context in which this can come up is logging, and a good alternative is to simply use syslog for logging. All of your app’s calls to syslog, and the standard error stream, will be
included in your app’s
Admin Console logs, where you can search them. You can also use the
Logs API to access the logs programmatically. (We will look at the Logs API in more detail in a subsequent post, and talk about how you can use it to, e.g., email periodic reports, and to store logs in GCS). You can also download the logs using the appcfg.py request_logs
command.
‘Including’ Files From GCSYou can
include or
require files from GCS in your application. This, e.g., allows you to write/read cached or compiled templates. For security reasons, you need to explicitly enable this feature. You do this by specifying which buckets contain includable files, using the google_app_engine.allow_include_gs_buckets
directive in your php.ini file, e.g.:
google_app_engine.allow_include_gs_buckets = "bucket_1, bucket_2"
You can use this feature to use templating libraries like
Twig or
Smarty with your PHP App Engine app.
Uploading Files to GCS From Your AppYour app can’t use the read-only filesystem to store uploaded files. Instead, it can use GCS to store uploads (and, GCS is great at handling large files and lots of them). App Engine makes this process very straightforward.
To upload files from your app, you generate a special
upload URL, specifying a result handler as a ‘callback’, and use this special URL in your upload form. Once the file is uploaded, the ‘callback’ handler receives a POST request, from which the upload can be
processed. See the
documentation for the details.
If you are running
WordPress on App Engine, we have built a
plugin which (amongst other features) supports uploading to and accessing media from GCS, so that all you need to do is install the plugin.
Similarly, for other apps, you will likely need to modify existing upload forms so that they use this generated upload URL, which will then call back to the existing handler (which can process the temp file just as before).
Once your app is succesfully writing files to GCS, there are a number of ways for your app to access them and serve them, in addition to simply using the GCS stream wrapper. We’ll describe those in a follow-on post.
A RecapWith App Engine apps, your app can read from the deployed directory structure, but it can’t write to it. Instead, it can use GCS for that. There are three main things you will probably have to do, in order to convert an app to use GCS:
Finally, the ‘sandboxing’ of the app filesystem dictates that you can’t modify this filesystem from your deployed app. So, for WordPress plugins and similar, you’ll need to install/update plugins locally using the development server, and re-deploy your app.
In following posts, we’ll describe the various ways that your app can access and serve the files that are in GCS, and talk more about the Logs API and how you can use it.
-Posted by Amy Unruh, Developer Programs Engineer
1. Your app also has access to
memcache, and to persistent stores like
Google Cloud SQL and
Cloud Datastore, but we won’t cover these in this post.
Follow