Tag Archives: Testing

NativeScript Testing: Developer Days Video

I just noticed my video from my presentation was released.  This was the presentation that I did at #NativeScript Developer Day in Sept of 2016.   The subject was Testing your NativeScript app and dealt with both Unit Testing and End to End testing.

I saw a couple of "um" mistakes.    I "um" need to "um" work on "um" removing some "um"'s.  😉

Here are the questions at the end (since like an idiot, I also totally spaced about repeating them) are:
1. What about testing Angular2 NativeScript apps.
2. What about Code Coverage .

Slides Deck:
http://fluentreports.com/blog/wp-content/uploads/2016/09/NativeScript-Testing.pdf

Demo "Carder" App I wrote and used, includes all the tests:
https://github.com/NathanaelA/carder

Detailed Blog posts about this subject covering the video:
Part 1 - Unit Testing
Part 2 - End to End Testing

NativeScript Testing: End to End

 

keyboardI got to give a talk on NativeScript testing to a great group of people at NativeScript Developer Days. 20160920_190247

Despite my silly technical problems at the beginning, we managed to get back on track and I was able to present all the critical information that was needed to get everyone up and running.  I did manage to miss my last slide; so I will make sure to have all that information at the end of each of these posts.

For those who are interested in doing testing and weren't there; or those who were in the talk, and just want to get a quick refresh, or if you just prefer the information written down; then this set of blog posts is for you.   I plan on doing a multi-part series on this as I want to cover what I did in my talk in the first two blog posts; but then expand the series to show Angular Native unit testing, how to unit test controls, and how to write unit tests for the NativeScript core.

You can view the #NSDevDay video here .  You can download my slide deck from the talk here.  And you can download the entire app that I wrote for my talk, including all the tests from here.

The posts currently planned in this series are:

Now as you might have guess even though you can do a lot of tests via unit testing, there are some limitations to unit testing.  You are typically only testing pieces of the whole.  So you really need to make sure the whole app works and the code you used to tie together all the pieces works.  And this is called End to End testing.

In my book, "Getting Started with NativeScript",  I discuss using Appium, for end to end testing.   Since the point where my book was published, one of the NativeScript engineers Hristov Deshev has actually created a really neat plugin.  It actually wraps up all the same steps that I came up for in my book.  Since it is a plugin, it is way easier to use, since it handles all the configuration and installation for you.  You just type tns install appium and it will install everything you need for end to end tests.   In this case all the tests for Appium will be stored in the root /e2e-tests folder since they don't actually need to be part of the Application itself.  Appium uses Mocha (which is the primary reason why I use mocha for my normal unit tests, I like consistency.).  It also uses Chia for the Asserts; so the tests are created the exact same way as I described in the unit testing post; with only a couple minor  changes.

So lets look at your /e2e-tests/carderAppium.js test that I provided in my sample testing application:

var nsAppium = require("nativescript-dev-appium");

The first thing you will notice is that I don't require any of the code from the app; the end to end testing does not run from inside your app; it is 100% external to your app.  So, you are at this point requiring the Appium setup and control driver library that Hristov wrote to wrap the configuration complexity.

describe("Carder app tests", function () {
    this.timeout(100000);
    var driver;

    before(function () {
        driver = nsAppium.createDriver();
    });

    after(function () {
        return driver
        .quit()
        .finally(function () {
            console.log("Driver quit successfully");
        });
    });

We still use the describe; but we have to actually setup the driver (that controls the device) in the before function.  And we tear it down (or close it down) in the after function.   Then we actually do all of our tests...
it("should find the card title", function () {
        return driver
            .elementByAccessibilityId("message")
            .text().should.become('Back of Card');
});

So in the first test, what we have to do is use the driver we created in the before function; pretty much everything uses the driver.  It is the communication channel to the device being tested.   Then the command we use is .elementByAccessibilityId('message').   This command will search the iOS or Android layout for any element in the UI that has the "message" accessibility id attribute set.   Now, in NativeScript this is actually set using the automationText property.  So if you look at my main-dynamic.xml or main-page.xml file; you will see:
<Label row="1" id="message" automationText="message" text="{{ text }}" class="message" tap="scrollOff" textWrap="true"/>

Then once the driver finds this element, it looks at the .text() value and that value should become "Back of Card".   When the app starts up; the first card chosen is the "Back of Card".  So if my app is actually running properly, this test will succeed.

Lets skip to the last test; as understanding it, will explain all the other tests in between. So lets figure out how it works.   Now first thing to understand is in the carder app; the numbers, letters and pips on the card are actually using a font.  So if you were to switch to Arial as the font, the actual underlying character is a "r" for the hearts pip.  So that is why I have "r" and "q" letter used in the tests...

// 'r' is the hearts pip
var heart = 'r';
        
// Setup our xpath
var xpath =  "//"+nsAppium.xpath("Label")+"[@text='"+heart+"']";

// Lets run our checks
return driver.elementByAccessibilityId("prior")
    .should.eventually.exist
    .tap().tap()
    .elementsByXPath(xpath).should.eventually.have.length(2);

As you might have noticed we do another elementByAccessibilityId("prior") -- we are looking for the prior element which is a button in this case (xml is:
<Button automationText="prior" id="prior" text="Prior" tap="prior"/>).

Then once it exists we tap it twice.  As you can see you can keep stacking commands; so in this case we actually stacked tap twice.  This is important to know, because frequently you will want to do different tests to the same element; or multiple actions to the same element.  You can easily chain them.

Next up, we are using the elementsByXPath which searchs the UI for anything that matches the xpath and returns it.  And finally we check the number of elements found.

Appium allows you to set/get values of fields, emulate taps & gestures, act like typing in on a keyboard, or just act exactly like an end user would, and then you can verify the results.  This allows you to build complex tests for your UI that test the entire "user" exposed functionality.

Now lets go into some specific details on some of these commands that you need to be aware of in Appium and NativeScript testing.   The Appium web driver actually has a ton of different selectors; however since Appium was initially developed for the web, only a couple selectors in the Appium documentation actually work for your NativeScript mobile apps.   The two selectors you can use reliably is element(s)ByAccessibilityId and element(s)ByXpath.  "element" returns only the first element found.  "elements" (note the added 's') returns all elements found.    As discussed earlier, AccessibilityId uses the NativeScript automationText value to find item(s).

XPath actually allows you to drill down into the UI and find specific items that may have a specific hierarchy and/or certain parents.  For example; rather than search for all buttons, you can limit the search to buttons that are inside a GridLayout which is inside a StackLayout area.  However, the biggest downside with xpath is that it expects you to have the actual native android or native ios control type name.   For example; on Android the NativeScript Button class is actually using the android.widget.button.  The native class on iOS it is actually using an UIButton.   Now that makes XPath really, really hard to be cross platform test, doesn't it?     So to solve that issue; I have written a cool wrapper to help with this issue. It allows you to pass in your NativeScript class name and it will, depending on the platform you are testing against, will return the real underlying native component name.   So in this specific test case the xpath was "//" = any level of items, we aren't giving any specific parents (so find this anywhere in the layout).  Then my helper class nsAppium.xpath("Label") will give me the actual underlying UI name of a NativeScript Label component, and then finally "[@text='r']" means that element must have a "r" as the text field value.    In the case of the card it should find, the two pips on the edge of the card which should be a "r".  So this test would pass as long as the prior button worked to bring you to a King of Hearts card...

The next thing you need to be aware of in Appium is that you MUST return the driver results. You will see every one of my Appium tests does a return.   In all reality, the entire chain that we are doing is actually a promise chain.  So for the test to actually run and then pass/fail, the final result of the promise chain must be evaluated.  So ALWAYS return the promise chain, or your tests will say they passed without actually knowing for sure that it actually passed or failed.  This is CRITICALLY important you return the final promise!

The final gotcha in Appium is to know is at the top of the test file, the "this.timeout(100000);" is actually very important.  Appium can take a while to actually startup the DRIVER to communicate with the device/emulator.   And you really do not want the test to timeout (which = failure) before it actually starts running it.  So make sure at least for android, that this is a very large value...

A couple notes; Appium launching the driver can be extremely slow.  You have to wait a while before it actually appears to be doing anything.   Second; If you are using my NativeScript-LiveEdit plugin, the watcher now has a cool ability to be able to launch Appium when you hit the "a" button in the watcher window.

Now all of this can be automated and is highly recommended to be automated in something like local git hooks, or some other CI environment.   That way when you commit a change; Unless you have a beefy machine, I would recommend you set it to run on like every 3-5 commits (depending on how frequently you commit, it might be higher). Because Appium is fairly slow to get the whole test started.  At worst case I would recommend you run a Appium at least once a day, several hours before you go home...

If you need help setting up a automatic testing and/or CI environment or you would like some training, please contact me.

Resources:

 

NativeScript Testing: Unit Testing

ns-road-signsI got to give a talk on NativeScript testing to a great group of people at NativeScript Developer Days.  20160920_190247Despite my silly technical problems at the beginning, we managed to get back on track and I was able to present all the critical information that was needed to get everyone up and running.  I did manage to miss my last slide; so I will make sure to have all that information at the end of this post.

For those who are interested in doing testing and weren't there; or those who were in the talk, and just want to get a quick refresh, or if you just prefer the information written down; then this set of blog posts is for you.   I plan on doing a multi-part series on this as I want to cover what I did in my talk in the first two blog posts; but then expand the series to show Angular Native unit testing, how to unit test controls, and how to write unit tests for the NativeScript core.

You can view the #NSDevDay video here .  You can download my slide deck from the talk here.  And you can download the entire app that I wrote for my talk, including all the tests from here.

The posts currently planned in this series are:

Unit Testing

So our first type of testing we will discuss is Unit testing.   Unit testing, for those of you that are un-aware of what it is -- it allows you to take a part of your code (typically a class or module) and you test it.  The best way is to treat it like a black box.  You don't care how it is implemented internally; you only care about the interface it exposes (i.e. inputs and the results).  By treating it as a black box; this allows you to re-implement/rewrite/refactor/fix it or do any other internal things to the class without the rest of your real or test code caring about the actual internal implementation.  Because your class is self contained it makes your code a lot more de-coupled and stand alone.  This typically means the code is better engineered and more maintainable.  In a nutshell, unit testing, is basically throwing as many different types of inputs and then verifying all the outputs and/or execution of your class is correct.

Now some people like to develop the tests before the code; this way they think of the scenarios that they need to handle; create all the tests (which of course at this point will all fail) and then implement the code.   Others build the code and build the tests along side the code, or build them after the code is complete.   There are pro's and con's for all sides; and that is really a whole different blog post.   In my opinion it is better to just make sure you do it; not the when of doing it.  One other recommendation is that when you get a bug report; you write a test for it. This way once you fix it; then in the future that specific bug will never happen again as your tests will always be checking for it.  Remember the idea is to test for everything you can think of, if a bug slipped through -- that means you failed to test everything and so by adding those "bug" tests, now hopefully you are actually testing everything the class can do...

The really cool thing is NativeScript actually has built in unit testing.   The neat thing about the built in unit testing, is it actually runs fully inside your NativeScript application.   This allows you to test anything that you have in your application including any added plugins, or custom code.   To install it, you just do tns test init.  This will ask you which testing framework to use and will install it.   I prefer using Mocha, and so the tests examples will all be mocha based; but again the framework you use is your choice.   One thing to note is that the assert functionality is actually provided by the Chia library in each of the testing frameworks supported by NativeScript.  If you really prefer a different assert library there are ways you can change it, but that is out side of the scope of this blog post.

For all your unit tests in NativeScript, they are all located in the /app/tests folder.  To then run all your cool tests  you will type tns test android or tns test ios.  TNS will then build your app and launch it in test mode.   Once each test completes; your console will tell you which tests passed/failed and then finally how many passed/failed like this.

NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 0 of 6 SUCCESS (0 secs / 0 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 1 of 6 SUCCESS (0 secs / 0.005 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 2 of 6 SUCCESS (0 secs / 0.007 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 3 of 6 SUCCESS (0 secs / 0.008 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 4 of 6 SUCCESS (0 secs / 0.009 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 5 of 6 SUCCESS (0 secs / 0.083 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 6 of 6 SUCCESS (0 secs / 0.085 secs)
NativeScript / 19 (4.4.4; Samsung Galaxy S5 - 4.4.4 - API 19): Executed 6 of 6 SUCCESS (0.217 secs / 0.089 secs

Lets get to the actual creating of tests in Mocha.  In my sample carder app; I provided a sample test file. So lets look at the unit test file I provided. The file is located /app/test/carderUnitTest.js.  The first thing you will see is that I include the class I plan on testing.

var cards = require('../cards.js');

Pretty hard to test it, without requiring it.   So in case it isn't obvious you still need to require the file(s) that contain any plugins/classes/module you need to test.

Then the next part of the code you will see is;

describe('Cards', function() {
    describe("current card", function() { 
        it ('should return ace of spades', function() {

The first line of this, we use the describe command to give the test a name, this allows you to know which test group passed or failed.  You can also nest describe's inside of describe's to allow you to have sub-groups of tests.   Finally you will see the it command, which is the actual test. You can have as many it's as you need in a group to test the full functionality.  So walking specifically through this test; we are saying the main test group is Cards (I name it after the class I'm testing); the following describe is I'm testing functionality I'm testing the current card function; and finally what my expectation of my test, which in this case is that it should return ace of spade card.

Then next we add our add my actual test code;

// Reset Card deck to known state
cards.resetDeck();
            
// The get "Current Card" function
var result = cards.getCurrentCard();
            
// My Expectations, that it should be
// a Ace of Spades on a reset card deck
assert.equal(result.suit, "Spade"); 
assert.equal(result.value, "Ace");

If either of the last two lines are false, then the test will fail and it will then mark this test as a failure in the log.

As you can see creating this test was fairly simple; Reset deck, grab the card and then verify it.  If you scroll down in the test file to the end of it; you will see the random cards test.  As you can see that test is a bit more complex.  Your tests can be as simple or complex as you need.  Typically the simpler your tests are the better, as then it is less likely that your test is actually the broken part of the code if it fails.

One other feature of Mocha is you can actually do ASYNC tests; so if your code is promise or callback based; you can also test it.  By doing this:

it ("Test promise", function(done) {
  call().some().cool().func()
  .then(done).catch(function() { done(false); });
});

// or

it ("test callback", function(done) {
  someFunc(function(err, results) {
     if (results) done()
     else done(false);
  }
});

Notice in the it function, it has a done parameter now.  The done() function is passed in for you to call when the async code has completed.  The mocha unit test detects if you tell it to have a done parameter and will switch that test to be fully async.

Note: If you are using my NativeScript-LiveEdit plugin, the watcher has had the ability to start these tests instantly on Android (Without having to use the slow `tns test android` command) for quite some time.  So anytime you change any of the /app/tests/ files; it will automatically launch the tests and you will see the results almost immediately.  However, I have recently added a couple new features to the watcher; you can hit inside the watcher window a "t" to start testing immediately; or the "m" to switch modes (i.e. default-auto-test, always test, and never test), and the "n" to force it to launch the app in normal mode.

Now all of this can be totally automated and it is highly recommended to be completely automated in something like local git hooks or some CI environment so that when you commit a change the unit tests are done.  The sooner your tests run the quicker you find any breaking code after you make any changes.  It is always very re-assuring when you have to fix a bug in your class and then  re-run your tests and everything passes...

Depending on your CI environment and how frequently you commit, you might consider having the heavier tests run once every three commits or based on a keyword put in the commit or in a issue tracking system.  Again, the idea is the more tests you can do and the more testing coverage you have the better off you are.

If you need help setting up a automatic testing and/or CI environment or you would like some training, please contact me.

Resources: