NativeScript Testing: End to End

 

keyboardI got to give a talk on NativeScript testing to a great group of people at NativeScript Developer Days. 20160920_190247

Despite my silly technical problems at the beginning, we managed to get back on track and I was able to present all the critical information that was needed to get everyone up and running.  I did manage to miss my last slide; so I will make sure to have all that information at the end of each of these posts.

For those who are interested in doing testing and weren't there; or those who were in the talk, and just want to get a quick refresh, or if you just prefer the information written down; then this set of blog posts is for you.   I plan on doing a multi-part series on this as I want to cover what I did in my talk in the first two blog posts; but then expand the series to show Angular Native unit testing, how to unit test controls, and how to write unit tests for the NativeScript core.

You can view the #NSDevDay video here .  You can download my slide deck from the talk here.  And you can download the entire app that I wrote for my talk, including all the tests from here.

The posts currently planned in this series are:

Now as you might have guess even though you can do a lot of tests via unit testing, there are some limitations to unit testing.  You are typically only testing pieces of the whole.  So you really need to make sure the whole app works and the code you used to tie together all the pieces works.  And this is called End to End testing.

In my book, "Getting Started with NativeScript",  I discuss using Appium, for end to end testing.   Since the point where my book was published, one of the NativeScript engineers Hristov Deshev has actually created a really neat plugin.  It actually wraps up all the same steps that I came up for in my book.  Since it is a plugin, it is way easier to use, since it handles all the configuration and installation for you.  You just type tns install appium and it will install everything you need for end to end tests.   In this case all the tests for Appium will be stored in the root /e2e-tests folder since they don't actually need to be part of the Application itself.  Appium uses Mocha (which is the primary reason why I use mocha for my normal unit tests, I like consistency.).  It also uses Chia for the Asserts; so the tests are created the exact same way as I described in the unit testing post; with only a couple minor  changes.

So lets look at your /e2e-tests/carderAppium.js test that I provided in my sample testing application:

var nsAppium = require("nativescript-dev-appium");

The first thing you will notice is that I don't require any of the code from the app; the end to end testing does not run from inside your app; it is 100% external to your app.  So, you are at this point requiring the Appium setup and control driver library that Hristov wrote to wrap the configuration complexity.

describe("Carder app tests", function () {
    this.timeout(100000);
    var driver;

    before(function () {
        driver = nsAppium.createDriver();
    });

    after(function () {
        return driver
        .quit()
        .finally(function () {
            console.log("Driver quit successfully");
        });
    });

We still use the describe; but we have to actually setup the driver (that controls the device) in the before function.  And we tear it down (or close it down) in the after function.   Then we actually do all of our tests...
it("should find the card title", function () {
        return driver
            .elementByAccessibilityId("message")
            .text().should.become('Back of Card');
});

So in the first test, what we have to do is use the driver we created in the before function; pretty much everything uses the driver.  It is the communication channel to the device being tested.   Then the command we use is .elementByAccessibilityId('message').   This command will search the iOS or Android layout for any element in the UI that has the "message" accessibility id attribute set.   Now, in NativeScript this is actually set using the automationText property.  So if you look at my main-dynamic.xml or main-page.xml file; you will see:
<Label row="1" id="message" automationText="message" text="{{ text }}" class="message" tap="scrollOff" textWrap="true"/>

Then once the driver finds this element, it looks at the .text() value and that value should become "Back of Card".   When the app starts up; the first card chosen is the "Back of Card".  So if my app is actually running properly, this test will succeed.

Lets skip to the last test; as understanding it, will explain all the other tests in between. So lets figure out how it works.   Now first thing to understand is in the carder app; the numbers, letters and pips on the card are actually using a font.  So if you were to switch to Arial as the font, the actual underlying character is a "r" for the hearts pip.  So that is why I have "r" and "q" letter used in the tests...

// 'r' is the hearts pip
var heart = 'r';
        
// Setup our xpath
var xpath =  "//"+nsAppium.xpath("Label")+"[@text='"+heart+"']";

// Lets run our checks
return driver.elementByAccessibilityId("prior")
    .should.eventually.exist
    .tap().tap()
    .elementsByXPath(xpath).should.eventually.have.length(2);

As you might have noticed we do another elementByAccessibilityId("prior") -- we are looking for the prior element which is a button in this case (xml is:
<Button automationText="prior" id="prior" text="Prior" tap="prior"/>).

Then once it exists we tap it twice.  As you can see you can keep stacking commands; so in this case we actually stacked tap twice.  This is important to know, because frequently you will want to do different tests to the same element; or multiple actions to the same element.  You can easily chain them.

Next up, we are using the elementsByXPath which searchs the UI for anything that matches the xpath and returns it.  And finally we check the number of elements found.

Appium allows you to set/get values of fields, emulate taps & gestures, act like typing in on a keyboard, or just act exactly like an end user would, and then you can verify the results.  This allows you to build complex tests for your UI that test the entire "user" exposed functionality.

Now lets go into some specific details on some of these commands that you need to be aware of in Appium and NativeScript testing.   The Appium web driver actually has a ton of different selectors; however since Appium was initially developed for the web, only a couple selectors in the Appium documentation actually work for your NativeScript mobile apps.   The two selectors you can use reliably is element(s)ByAccessibilityId and element(s)ByXpath.  "element" returns only the first element found.  "elements" (note the added 's') returns all elements found.    As discussed earlier, AccessibilityId uses the NativeScript automationText value to find item(s).

XPath actually allows you to drill down into the UI and find specific items that may have a specific hierarchy and/or certain parents.  For example; rather than search for all buttons, you can limit the search to buttons that are inside a GridLayout which is inside a StackLayout area.  However, the biggest downside with xpath is that it expects you to have the actual native android or native ios control type name.   For example; on Android the NativeScript Button class is actually using the android.widget.button.  The native class on iOS it is actually using an UIButton.   Now that makes XPath really, really hard to be cross platform test, doesn't it?     So to solve that issue; I have written a cool wrapper to help with this issue. It allows you to pass in your NativeScript class name and it will, depending on the platform you are testing against, will return the real underlying native component name.   So in this specific test case the xpath was "//" = any level of items, we aren't giving any specific parents (so find this anywhere in the layout).  Then my helper class nsAppium.xpath("Label") will give me the actual underlying UI name of a NativeScript Label component, and then finally "[@text='r']" means that element must have a "r" as the text field value.    In the case of the card it should find, the two pips on the edge of the card which should be a "r".  So this test would pass as long as the prior button worked to bring you to a King of Hearts card...

The next thing you need to be aware of in Appium is that you MUST return the driver results. You will see every one of my Appium tests does a return.   In all reality, the entire chain that we are doing is actually a promise chain.  So for the test to actually run and then pass/fail, the final result of the promise chain must be evaluated.  So ALWAYS return the promise chain, or your tests will say they passed without actually knowing for sure that it actually passed or failed.  This is CRITICALLY important you return the final promise!

The final gotcha in Appium is to know is at the top of the test file, the "this.timeout(100000);" is actually very important.  Appium can take a while to actually startup the DRIVER to communicate with the device/emulator.   And you really do not want the test to timeout (which = failure) before it actually starts running it.  So make sure at least for android, that this is a very large value...

A couple notes; Appium launching the driver can be extremely slow.  You have to wait a while before it actually appears to be doing anything.   Second; If you are using my NativeScript-LiveEdit plugin, the watcher now has a cool ability to be able to launch Appium when you hit the "a" button in the watcher window.

Now all of this can be automated and is highly recommended to be automated in something like local git hooks, or some other CI environment.   That way when you commit a change; Unless you have a beefy machine, I would recommend you set it to run on like every 3-5 commits (depending on how frequently you commit, it might be higher). Because Appium is fairly slow to get the whole test started.  At worst case I would recommend you run a Appium at least once a day, several hours before you go home...

If you need help setting up a automatic testing and/or CI environment or you would like some training, please contact me.

Resources:

 

6 thoughts on “NativeScript Testing: End to End

  1. Joe

    This is very cool stuff! I have been working today to get Appium working with my NativeScript iOS app at work. After hacking it a bit to support iOS 10/XCode 8, I was finally able to run my end-to-end test!

    I would really like to use the XPath trick you demonstrated in the video and referenced above, but unfortunately I can't find where the helper is defined. At runtime I get "TypeError: nsAppium.xpath is not a function". I have searched your code, the nativescript-dev-appium repository, and everywhere else I can think of, but I'm not finding the XPath helper. Where is that coming from, and how can I use it?

    Reply
    1. Nathanael Anderson Post author

      Awesome getting your stuff up and running. I must apologize; I totally forgot to get my pull request submitted to the ns-appium repo... I can send you a replacement file if you want with it in it (send me an email); and I'll also try and get the pull request submitted to the repo in the next day or so... (Lots of things on my todo list...)

      Reply
      1. Joe

        No worries, I can wait until your PR hits the ns-appium repo if it is in the next few days. I'm watching the repo, so I'll eagerly await the PR!

        Thanks again for putting the time into figuring this stuff out! It's been a huge help!

        Reply
        1. Nathanael Anderson Post author

          I posted the two pull requests that fix windows support and create the function. Sorry about the delay...

          Reply
  2. Nicolas

    Hello,

    Very nice post, thank you!

    Did you already have this kind of issue:

    I have 2 tests (exactly the same!). The first always fails and the second passes. I suspect that it doesn't wait for an element to be displayed... Even if I ask to. My first page of the app is a simple redirection to login or "connected homepage". Do you have any idea?
    Here is the code:

    it("should find the button! (always fail 🙁 )", function () {
    return driver
    .waitForElementByAccessibilityId('loginPage', 50000)
    .elementByAccessibilityId("loginButton")
    .text().should.become('Login / Join a team');
    });

    Reply
    1. Nathanael Anderson Post author

      One thing to narrow things down would be to switch the two tests around. Does the old test that was 2nd (and now is 1st) start failing? Or does the same test still fail?
      If the same test is failing than maybe your issue is that it isn't becoming that text.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *