BEGO, IT Consulting, Nikola Begedin

Persisting state between page loads on the backend with Elixir/Phoenix

Nikola Begedib — Tue, 09 Oct 2018 00:00:00 GMT

What's the idea?

Say you have a signup process that's more onboarding than account creation.

The user submits information in multiple steps, where the next step depends on the contents of the previous step, etc.

You also don't have an advanced frontend. It's just a basic server-based application, so each step means navigating to a different page

What if the user gives up half way through? How do you deal with that?

Suddenly, you are left with a partially created account.

Maybe that partial data is useless to you and you want to clean it up? Maybe you want to do something about it? Send an email? Perform some task?

Ideally, you don't save anything into the database yet, because why waste resources. However, you do keep it somewhere where it's cheap and easy to retrieve.

Then, when the user is all done, you save the data. When you detect the user gave up, you do something about it without having to sanitise anything in the database.

Do it with a `GenServer`!

Elixir has a GenServers and they're simple to use for any of those tasks you want to do. In fact, they even make it easy to create a multi-step process in the first place!

How?

Let's generate a basic app called "wizard", where a user creates their account and workspace.

mix phx.new wizard --no-ecto

We don't need a repository for the purposes of demonstrating the concept, thus, --no-ecto.

Next up, we add some routes for our onboarding flow

get "/step-1", WizardController, :step_1
post "/step-1", WizardController, :next_step
get "/step-2", WizardController, :step_2
post "/step-2", WizardController, :submit_steps
get "/doublecheck", WizardController, :doublecheck

And then we also add our controller.

I could've used a generator for all this, but I didn't, and you can copy paste, so it's fine!

# lib/wizard_web/controllers/wizard_controller.ex

defmodule WizardWeb.WizardController do
  use WizardWeb, :controller

  alias Wizard.Onboarding

  def step_1(conn, _params) do
    conn |> render("step_1.html")
  end

  def submit_step_1(conn, params) do
    conn.remote_ip |> Onboarding.submit_account(params)
    conn |> redirect(to: conn |> wizard_path(:step_2))
  end

  def step_2(conn, _params) do
    conn |> render("step_2.html")
  end

  def submit_step_2(conn, params) do
    conn.remote_ip |> Onboarding.submit_workspace(params)
    conn |> redirect(to: conn |> wizard_path(:doublecheck))
  end

  def doublecheck(conn, _params) do
    {account_params, workspace_params} =
      conn.remote_ip |> Onboarding.get_data()

    conn
    |> render(
      "doublecheck.html",
      account_params: account_params,
      workspace_params: workspace_params
    )
  end
end

The idea is, the step_1 action renders a form. The user then submits the form, calling the submit_step_1 action.

submit_step_1 sends the submitted parameters to our GenServer, which stores them.

Similarly, step_2 renders another form, which submits another set of parameters with submit_step_2, which also stores those parameters to the GenServer.

Then, we end up on redirected to the doublecheck action, where we retrieve both sets of parameters and render our template with those as the assigns.

For demo purposes, the template does nothing else other than rendering the parameters.

All 3 calls to the GenServer send conn.remote_ip as the first argument. The server uses the IP as a key to stare the data on a per-user basis.

Here's what the GenServer looks like:

# lib/wizard/onboarding.ex
defmodule Wizard.Onboarding do
  @name __MODULE__

  use GenServer

  # API

  def start_link() do
    GenServer.start_link(@name, [], name: @name)
  end

  def submit_account(ip, params) do
    GenServer.cast(@name, {:submit_account, ip, params})
  end

  def submit_workspace(ip, params) do
    GenServer.cast(@name, {:submit_workspace, ip, params})
  end

  def get_data(ip) do
    GenServer.call(@name, {:get_data, ip})
  end

  # Callbacks

  @impl true
  def init(_) do
    {:ok, %{}}
  end

  @impl true
  def handle_cast({:submit_account, ip, params}, %{} = state) do
    # keep old workspace params, update account params
    new_state =
      state
      |> Map.update(ip, {nil, nil}, fn {_, workspace_params} ->
        {params, workspace_params}
      end)

    {:noreply, new_state}
  end

  @impl true
  def handle_cast({:submit_workspace, ip, params}, %{} = state) do
    # keep old account params, update workspace params
    new_state =
      state
      |> Map.update(ip, {nil, nil}, fn {account_params, _} ->
        {account_params, params}
      end)

    {:noreply, new_state}
  end

  @impl true
  def handle_call({:get_data, ip}, _from, %{} = state) do
    {:reply, state[ip], state}
  end
end

Again, emphasis on simplicity for this demo.

Out state is a map and we use the IP as the key for storing individual bits of data.

The value for each IP is a tuple, the first element of which are the account params submitted in step 1 and the second element the workspace params submitted in step 2.

Note that using the IP as the data key probably isn't ideal, but again, this is about the idea, not the real life execution.

For the sake of getting this running as soon as possible, this is our view and respective templates:

# lib/wizard_web/views/wizard_view
defmodule WizardWeb.WizardView do
  use WizardWeb, :view
end

Nothing special about that.

# lib/wizard_web/templates/wizard/step_1.html.eex
<h3>Step 1: Your personal info</h3>
<%= form_for @conn, @conn |> wizard_path(:submit_step_1), fn f -> %>
  <div class="form-group">
    <%= label(f, :email) %>
    <%= text_input(f, :email, class: "form-control") %>
  </div>
  <div class="form-group">
    <%= label(f, :full_name) %>
    <%= text_input(f, :full_name, class: "form-control") %>
  </div>
  <div class="form-group">
    <%= submit("Next", class: "btn btn-primary pull-right") %>
  </div>
<% end %>

# lib/wizard_web/templates/wizard/step_1.html.eex
<h3>Step 2: Your workspace info</h3>
<%= form_for @conn, @conn |> wizard_path(:submit_step_2), fn f -> %>
  <div class="form-group">
    <%= label(f, :name) %>
    <%= text_input(f, :name, class: "form-control") %>
  </div>
  <div class="form-group">
    <% back_path = @conn |> wizard_path(:step_1) %>
    <%= link("Back", to: back_path, class: "btn btn-default") %>
    <%= submit("Doublecheck", class: "btn btn-success pull-right") %>
  </div>
<% end %>

# lib/wizard_web/templates/wizard/doublecheck.html.eex
<h3>This is the data you submitted:</h3>
<h4>Account</h4>
<pre>
  <%= @account_params |> Kernel.inspect(pretty: true) %>
</pre>

<h4>Workspace</h4>
<pre>
  <%= @workspace_params |> Kernel.inspect(pretty: true) %>
</pre>

<% back_path = @conn |> wizard_path(:step_2) %>
<%= link("Back", to: back_path, class: "btn btn-default") %>

As I said, the steps are forms, while the final step is a basic page that renders the data we have.

Where to from here?

In a real world scenario, each step would do something more. For example, individual submit steps could cast changesets or process the data in some way.

The controller could cast data into changesets and render validation errors, storing valid changesets into the GenServer, instead of raw parameters.

As a final step, we would also have some sort of call which actually persists the data, but as I said, this is about the idea.

The possibilities should be clear, though. It's trivial to manage state during user interaction and between page navigations using a GenServer.

Going even further...

LiveViews are soon to become a thing.

In the meantime, you could use the same GenServer, in conjunction with phoenix channels, to inform the user about any long running tasks in the onboarding process in real time.

Who says you need to use this in a primarily server-based app? Imagine the possibilities!

About this site

Nikola Begedib — Sat, 23 Jun 2018 00:00:00 GMT

BEGO, It Consulting and Application Development

Trade business registered in Croatia

Original registration info

Full name: Bego, obrt za IT savjetovanje i razvoj aplikacija
Address: Zrinskih i Frankopana 8, 42000 Varaždin, Croatia
Owner: Nikola Begedin

Setting up an Elixir/Phoenix project for acceptance testing with Cypress

Nikola Begedib — Thu, 21 Jun 2018 00:00:00 GMT

What is Cypress?

Cypress is this awesome new tool that works great with APIs and is as easy to setup as anything if you're mocking out your backend.

I've recently moved from Ember to React and I've been missing the test setup for acceptance (or end to end) tests a lot.

What Ember does with ember-cli is that it, right out of the box, give you a command which runs test in an actual browser environment (used to be PhantomJS by default, but is now headless Chrome), where you perform/simulate actual user interaction and make assertions on what happens both in the UI and in the background.

Cypress is an app, with a UI which does something similar, except it works with any framework. Further more, it provides you with an infrastructure which allows you to do even more.

But what about data?

With Ember, we usually ended up using either a fake server with fake data, or a mock server layer with mock responses to requests. ember-cli-mirage usually did the trick for us.

That means that, technically, only our frontend app was acceptance/e2e tested, but there was still no guarantee it works correctly with our backend.

However, now that I started using Cypress and an Elixir/Phoenix backend, I realised I could've and now can easily do more than that.

Ecto Sandbox mode

Ecto, the data layer/library Phoenix uses, has this thing called a Sandbox mode.

To elaborate, each test in Elixir/ExUnit runs in a separate erlang process. The database layer runs a pool of connections those processes to use to perform actions with the database.

To allow easier management of the data each test accepts, and to allow easier running of these processes in parallel, Ecto provides a thing called a Sandbox mode.

Effectively, each process, or each test, get's it's own sandboxed database, which runs in a single transaction, isolated from other processes, and is canceled when the process exits.

That means that, within the test, we perceive data as changing, but as soon as the test ends, any changes are rolled back.

This isn't on by default, but the process to set it up is relatively simple

# in config/test.exs
config :my_app, MyApp.Repo, pool: Ecto.Adapters.SQL.Sandbox

# in test_helper.ex
Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, :manual)

# before each test, or in an ExUnit.Case
setup do
  # Explicitly get a connection before each test
  :ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
end

Really, it's explained quite well in the Ecto documentation.

The problem is, this only works with ExUnit, so it only works with backend tests written in Elixir. If your app is a server-rendered backend app, with little to no client code, that's fine. Most of the time, it's not.

Another problem is that the database adapter we're using needs to support this. The two Ecto comes prepackaged with, Ecto.Adapters.MySQL and Ecto.Adapters.Postgres both support it, with MySQL supporting sandbox mode, but not with concurrent processes. For third party adapters, we'll have to look into the documentation, possibly open tickets, or even contribute to an open source project to get it working.

Ecto Sandbox Mode with a Frontend app

A progressive web application, a dynamic client application, a javascript app, a single page application, call it whatever you like, but most web sites or web apps have quite a lot of client-written behavior, which needs to be tested to and, ideally, those tests need to make sure it works with the current backend, which could easily be a separate project, making any syncing even harder.

Fortunately, phoenix_ecto has us covered with a plug called [Phoenix.Ecto.Sql.Sandbox](https://hexdocs.pm/phoenix_ecto/main.html#concurrent-browser-tests)

You should read that documentation. In fact, you should read the documentation of any Elixir library you use. The amount of info there is usually sufficient to de anything.

But to elaborate, here's how it works.

First, I create a separate mix environment for running the dev server in sandbox mode. This will have to run in a background process, or in a separate terminal tab, since cypress prefers not to be in charge of starting and shutting down the server.

I like to call this one e2e, so we add config/e2e.exs:

# config/e2e.exs

# it's basically a dev environment, with a minor change added to it
import_config "dev.exs"

# this is the minor change
config :my_app, sql_sandbox: true

Then, I add a couple of lines to the MyAppWeb.Endpoint module:

# lib/my_app_web/endpoint.ex:

use Phoenix.Endpoint, otp_app: :sift

if Application.get_env(:my_app, :sql_sandbox) do
  plug(Phoenix.Ecto.SQL.Sandbox, at: "/sandbox", header: "x-session-id", repo: MyApp.Repo)
end

What that does is, if I run my server with MIX_ENV=e2e mix phx.server, the router will support two additional actions

POST /sandbox checks out a sandbox connection
DELETE /sandbox checks in a sandbox connection

It also adds a plug layer to all other routes which checks for an x-session-id header and assigns the handler for that request the checked out connection.

Get where I'm going with this?

Our frontend testing tool can now

make a POST request to /sandbox to check out a sandbox connection and get back an id token at the start of the test
sign all further requests in that test with an x-session-id header, the value of which is that token, to use the checked out sandbox connection
make a DELETE request to /sandbox, with an x-session-id header, to check in the same sandbox connection at the end of the test

The end result is, our frontend test runs isolated from other frontend tests, while at the same time being able to access your actual backend, instead of using a fake one.

Setting up Cypress to use a Phoenix backend in Ecto sandbox mode

I'm assuming you've already followed the Getting Started section of the cypress docs. The base API url has been set and everything is ready. Ideally, you already have a test running and hitting our MIX_ENV=e2e mix phx.server backend running in another tab, causing that backend to error out due to duplicate date (because Cypress is not triggering sandbox mode).

First thing to note is that even if you set it up now, the tests will still likely fail. You have to do a MIX_ENV=e2e mix ecto.reset to clean up the database first.

Cypress with a Phoenix/Absinthe backend and a React Apollo frontend

This is setup where I've taught myself this procedure.

For checking out and checking in connections, I've created a pair of cypress commands

Cypress.Commands.add('checkoutSession', async () => {
  const response = await fetch('/sandbox', {
    cache: 'no-store',
    method: 'POST',
  });

  const sessionId = await response.text();
  return Cypress.env('sessionId', sessionId);
});

Cypress.Commands.add('dropSession', () =>
  fetch('/sandbox', {
    method: 'DELETE',
    headers: { 'x-session-id': Cypress.env('sessionId') },
  }),
);

For signing each request made by the app with an x-session-id header, things get a bit tricky. What we need to do is to modify window.fetch slightly, since that's what Apollo uses. To do that, we make use of Cypress' ability to inject code into the browser session it runs the test in.

const buildTrackableFetchWithSessionId = fetch => (fetchUrl, fetchOptions) => {
  const { headers } = fetchOptions;
  const modifiedHeaders = Object.assign(
    { 'x-session-id': Cypress.env('sessionId') },
    headers,
  );

  const modifiedOptions = Object.assign({}, fetchOptions, {
    headers: modifiedHeaders,
  });

  return fetch(fetchUrl, modifiedOptions)
};

Cypress.on('window:before:load', win => {
  cy.stub(win, 'fetch', buildTrackableFetchWithSessionId(fetch));
});

What we did is we wrapped our window.fetch into a function which adds the necessary header to each request.

Now, this is a tad complex. Ideally, we would make use of cy.server somehow, but I was unable to figure out out yet. If I do, or if someone else does, I'll be sure to post an update.

There's another caveat we need to solve here that makes it slightly more complex.

Our cypress test has no idea when it can end, so it also needs to wait for all outgoing requests to resolve or fail before starting checking the session back in. Otherwise, we might encounter trouble.

This works automatically for Cypress with XHR requests, but for requests made through the Fetch API, it fails, so we need to make our code a tad more complex.

const increaseFetches = () => {
  const count = Cypress.env('fetchCount') || 0;
  Cypress.env('fetchCount', count + 1);
};

const decreaseFetches = () => {
  const count = Cypress.env('fetchCount') || 0;
  Cypress.env('fetchCount', count - 1);
};

const buildTrackableFetchWithSessionId = fetch => (fetchUrl, fetchOptions) => {
  const { headers } = fetchOptions;
  const modifiedHeaders = Object.assign(
    { 'x-session-id': Cypress.env('sessionId') },
    headers,
  );

  const modifiedOptions = Object.assign({}, fetchOptions, {
    headers: modifiedHeaders,
  });

  return fetch(fetchUrl, modifiedOptions)
    .then(result => {
      decreaseFetches();
      return Promise.resolve(result);
    })
    .catch(result => {
      decreaseFetches();
      return Promise.reject(result);
    });
};

Cypress.on('window:before:load', win => {
  cy.stub(win, 'fetch', buildTrackableFetchWithSessionId(fetch));
});

What we do is keep a counter of active Fetch API requests in Cypress.env. We have a special command we then use to wait for all of them to run their course before checking the session back in.

Cypress.Commands.add('waitForFetches', () => {
  if (Cypress.env('fetchCount') <= 0) {
    return;
  }

  cy.wait(100).then(() => cy.waitForFetches());
});

The command waits in 100 ms increments to see if the fetch count dropped to zero. I found it completely reliable for my usage, but tweaks may be required.

The easiest way to put this command to use is to rewrite our dropSession command.

Cypress.Commands.add('dropSession', () =>
  cy.waitForFetches().then(() =>
    fetch('/api/sandbox', {
      method: 'DELETE',
      headers: { 'x-session-id': Cypress.env('sessionId') },
    }),
  ),
);

As an alternative for request waiting, we could have the ApolloClient not use Fetch in a test environment, but then we're forced to scatter our custom test-code around, while this approach can keep it in one place.

Ideally, at some point Cypress will know to wait for outgoing Fetch API requests to and we can just remove this complexity.

Cypress with a Phoenix JSON API backend and a React REST frontend

Really, the approach here is near identical to the GraphQL approach, except, if the network layer of the app uses XHR instead of Fetch, we don't need to maintain a fetch counter and tests will wait properly by default.

In that case, instead of overriding fetch, we would override the XMLHttpRequest.prototype.open function.

I didn't get a chance to actually test this, but the general approach would be something like:

const overrideOpen = original => {
  return function openWithHeaders() {
    original.apply(this, arguments);
    this.setRequestHeader('x-session-id', Cypress.env('sessionId'));
  };
};

Cypress.on('window:before:load', win => {
  cy.stub(
    win,
    'XMLHttpRequest.prototype.open',
    overrideOpen(XMLHttpRequest.prototype.open)
  );
});

Tip: Pre-creating data for tests to use

We can use different approaches here. One is to use cy.task() to execute outside code and seed the database that way. Another is to simply use the data layer our app likely already has and make direct requests for data creation to the API, after we've checked out the session, but before the actual test started running.

For example, with GraphQL, I created a cy.mutate(mutation, variables) command:

Cypress.Commands.add('mutate', (mutation, variables) => {
  const client = buildClient({
    'x-session-id': Cypress.env('sessionId')
  });
  return client.mutate({ mutation, variables });
});

My buildClient function is the same I use in my app to build an ApolloClient. It allows me to pass in custom headers for all requests to use, which is what I do here.