When Pew Research Center moved its long-running trend questions from phone to online, “don’t know” and other unprompted answers on one question fell from 10% to 1%. Same question, same month, different setup. Pew’s own analysis put the gap down to how the survey was delivered, not to anything people believed.
Moving a tracker from one survey platform to another is a smaller change than phone to web. It runs on the same mechanism, though. A tracker script migration breaks in three places: what the code says, what the platform decides when the code says nothing, and what the respondent sees on screen. Most teams check the first one. Wave 15 usually goes wrong in the other two.
We’ve moved a lot of trackers in our Bright MR and Kalever practice. The rule we work by is simple. We prove parity in the data, not on the screen. If the new build can’t reproduce last wave’s data file, code for code, the migration isn’t done.
Layer 1: What the Script Says
Trackers drift even when they stay put, one copy-over at a time. We covered that in Nobody Changed the Tracker. So Why Doesn’t Wave 7 Match Wave 1? A move adds three new places for things to break, all in the same week.
This is the part every tracker script migration plans for. Every platform writes logic its own way. So a filter, a loop or a piece of piped text has to move into the new system’s language, one by one.
The usual suspects:
- Question types. A ranking question or a slider on the old platform may have no exact twin on the new one. The nearest match stores answers in a different shape. Your tables still expect the old one.
- Masking. One programmer writes “Show only brands selected at Q3” as one rule. Another writes it as twelve. Rewrite it by hand and one brand slips out of the mask.
- Custom code. Old trackers carry scripts nobody remembers writing. A JavaScript fix from Wave 6 still runs every wave, and it rarely has a direct match on the new platform.
Layer 1 errors are the friendly ones, because they usually fail loudly. The survey breaks in testing, someone fixes it, and everybody moves on.
Layer 2: What the Platform Decides When the Script Says Nothing
This is where Wave 15 starts to drift. A script only says what it needs to. Everything else is a platform default, but defaults don’t move with the script.
Four defaults we check on every move:
- Required answers. One platform makes every question required by default. Another leaves them optional. Forcing answers changes the data. A 2015 study by Décieux and colleagues found that requiring every answer in an online survey raised dropout and lowered answer quality.
- Unticked boxes. In a multi-select, an unticked option can be stored as 0, as blank, or not stored at all. Pick the wrong one and your base changes. Then every percentage on that question moves with it.
- Answer codes. Some platforms number answer lists 1, 2, 3 by default. Your tracker needs Brand H at code 8 and “Don’t know” at 99, because that’s where five years of trend data live.
- Exclusive answers. “None of these” clearing the other ticks is often a setting, not a line of code. Copy the code, lose the setting.
Nothing on this list throws an error in testing. You find it later, as a step in the trend, so we check defaults one by one before the first test complete.
Layer 3: What the Respondent Sees
The last layer is the one nobody can read in a script. A question that looks different gets answered differently.
Pew’s 10% to 1% is the extreme case. On the phone, people could say “I don’t know” or “both” without anyone reading those options out. Online, with no such option on screen, almost everyone picked a side. A new platform can do the same thing on a smaller scale. People use a “Don’t know” they can see as a button. They ignore the same option when it sits behind a small link.
Pew also notes that people filling in a survey on their own lean toward options near the top of a list. Change the order, the layout or the direction of a scale, and the answers shift with it. Then there’s mobile, where a grid may turn into a stack of single questions without anyone asking it to. We covered the screen-by-screen checks in the tracker migration checklist.
Where a Tracker Script Migration Breaks, at a Glance
| Layer | What breaks | How it shows up | How to catch it |
|---|---|---|---|
| 1. The script | Question types, masks, custom code | Errors in testing, missing columns | Code review and full route testing |
| 2. Platform defaults | Required answers, unticked boxes, codes, exclusives | A step in the trend, a changed base | Compare test data files code by code |
| 3. The screen | “Don’t know” display, answer order, scale direction, mobile layout | Answers drift with no error anywhere | Side-by-side screenshots, soft launch vs. last wave |
How We Handle a Tracker Script Migration
A migration is a QA job with some programming attached. So we built our process around proving parity, and we let the technology do the retyping.
- Start from the source. The platform reads the existing script and the last wave’s data map. Nobody retypes 300 questions from a Word document.
- Convert automatically. The platform rebuilds the logic, codes and variable names in the new system’s format. That covers most of Layer 1 before a person opens the file.
- Validate by hand. Our ops team goes through the converted survey line by line. They check every Layer 2 default against the old build, because defaults are where software can’t guess intent.
- Compare the data, not the screens. We run the same test respondents through both builds, export both files and compare them variable by variable. We explain or fix every difference before anything goes further.
- Carry the history. The change log, the code frames and the known trend breaks move with the project. The next programmer shouldn’t inherit a script with no memory.
Our rule for step 4 is short. 1:1 parity, or it doesn’t ship.
Run a Parallel Wave When the Stakes Are High
You can shrink Layer 3 effects, but you can’t always remove them. So for a tracker where the client can’t risk any doubt, field the old build and the new one side by side for one wave and measure the gap. Research Live, published by MRS recommends exactly this when a tracker changes supplier or platform. It costs one extra wave of sample. A trend break you can’t explain costs the client’s trust.
Where to Go Next
For the checks you can run yourself, start with the tracker migration checklist.
Got a tracker with 14 waves on it and a contract renewal coming up? Talk to us before anyone scripts Wave 15.
And for one email a month on trackers, QA and the work behind clean data, sign up for the CodexMR newsletter.
