ExVCR is an awesome elixir package which helps you create repeatable integration tests. It works by saving the requests and responses you make to external services, in json files called cassettes. These cassettes are created the first time you run the tests. Here is an example from one of my projects:
defmodule X.AuthControllerTest do
The first time I run this test it creates a file at
This contains the request and response data from this test. Now, whenever you rerun
this test it just loads this request and response data and replays the response if the request
matches the saved request. So, your tests will pass even when you don’t have network connectivity!
This is all nice as long as you have small responses. However, when you need to do this for responses whose size is in MBs you run into a few issues:
- Your tests become slower.
- You don’t want to haul around MBs of fixtures in your source control (you must check in the cassettes i.e. the
I recently ran into this situation with one of my projects where I was downloading a gzipped csv file which was ~50MB. The way I fixed it is:
- Let ExVCR record the cassettes with all the large data.
- Download the same response from outside using curl.
- Take a tiny sample of the response data
- Replace the
response.bodyin the cassette with my sample response data
This fixes the slow tests and also reduces the size of cassettes.
Another thing to remember is if your response is binary and not a valid string, ExVCR stores it differently. Its json serializer runs binary data through the following functions
So, if you have gzipped (or any other binary) data, load it into IEx from step 2 above and run it through
the same functions as above and use this data to replace the
response.body. You can also skip 2
and use the data from your large cassette before trimming it down by running it through:
cassette = File.read!("fixture/vcr_cassettes/cassette.json") |> Poison.decode! |> hd
P.S My editor froze while trying to edit such a large cassette, so I used jq to remove the large body keys from the response using the following command:
jq 'walk(if type == "object" and has("body") then del(.body) else . end)' mycassette.json > cassette_without_body.json
To get the above working you need jq > 1.5 and the following in your
# Apply f to composite entities recursively, and to atoms